6. Extend the benchmark

Ordinary functions and compact declarations cover small one-shot comparisons. Use more explicit interfaces when the benchmark needs additional data formats, worker state, or observations.

Add components

Scaffolding adds a component and an adapter to an existing manifest:

metewand scaffold dataset observations --language python --dry-run
metewand scaffold dataset observations --language python
metewand scaffold problem target --language python
metewand scaffold implementation candidate --language python --problem target

The adapters go under datasets/, problems/, and implementations/. Fill in their functions, declare result and metric fields, and add an experiment that selects them. Scaffolding does not add experiments or install dependencies. It preserves existing manifest comments and refuses to overwrite adapter files. Use --dry-run to inspect the proposed paths first.

--manifest PATH selects a different manifest. --language r selects R functions; --language rust generates native worker adapters. Rust adapters need to be added to a Cargo project and built. Copy each executable beside its adapter with the same basename and no extension. The crates currently require local path dependencies; scaffolding does not create a Cargo project.

Use R or native workers

R functions have the same roles as Python:

generate <- function(context) {
  set.seed(context$seed %% .Machine$integer.max)
  list(values = as.list(sample.int(10L, context$parameters$size, replace = TRUE)))
}

solve <- function(data, context) {
  list(total = sum(unlist(data$values)))
}

evaluate <- function(data, result, context) {
  list(absolute_error = abs(result$total - sum(unlist(data$values))))
}

Select them with references such as function: benchmark.R:solve. The generator expects a declared integer parameter size. Return named lists for JSON objects and unnamed lists for arrays. Use metewand::evaluation(metrics, verdict) when scientific rules require an explicit verdict. The R starter supplies a complete manifest.

The function context supplies parameters for the current role, problem_parameters for implementations and evaluators, seed for generators and implementations, and semantics for evaluators. Evaluators have no supplied random seed. Install dependencies in each selected worker environment.

Native Rust helpers run without a Python or R host. Other languages can implement the process protocol directly. Start with the Rust example or raw worker example.

Separate preparation and solving

Lifecycle workers let you load data in prepare, run a method in execute, convert its output, and reset state before reuse. They also support custom artifact writers and configuration-dependent applicability checks.

The Python package provides OneShotImplementation and serve_implementation. The R package provides serve_implementation with callbacks. Native Rust workers implement the Implementation trait and use the same serving lifecycle. The runnable Python, R, and Rust examples show complete adapters and declarations. The package source and R help pages describe their callback interfaces.

Include source and data deliberately

Function declarations automatically include the selected source file. Add imported modules, data files, and lockfiles through sources:

function: benchmark.py:gd
sources: [benchmark.py, solvers.py, requirements.txt]

Metewand hashes declared source content. Workers receive private copies of their bundles, and undeclared sibling modules are not automatically available. A lockfile in sources records its bytes; the local runner does not install or provision its dependencies.

Function workers exchange JSON objects. For large matrices, sparse structures, or native formats, use a materializer and custom result writers. Write all files under the assigned output directory, close them, and return a complete manifest with paths, media types, sizes, and hashes. The runtime validates the inventory and schemas before publication. See the generated-dataset example.

Move from final answers to profiles

Use a profile when the question concerns progress toward a common accuracy target. Profile selection is explicit and requires lifecycle workers with the corresponding capability.

Approach	Use it when	Recorded coordinate
Fresh sequence	You can rerun a solver from clean state at a finite schedule of iteration limits or tolerances.	Executor-observed time for each independent point.
Buffered profile	You can instrument one trajectory and capture bounded local measurements and result snapshots.	An instrumented clock with a problem-authorized scope, plus executor time.

Fresh sequences work with black-box solvers and stop according to independent evaluation and configured limits. Buffered profiles retain provisional local trace values and independently evaluated snapshots separately. Instrumented time and wall time remain different measurements.

The fresh-sequence example contains Python and R adapters, explicit and geometric schedules, repetitions, and resume commands. The buffered-profile example contains Python and native Rust workers, bounded traces, selected snapshots, and exports. Start from these complete manifests rather than adding profile fields to a function worker.

Find further integrations in examples, or read the design for the underlying model.