Experiments & Reproducibility

This page lists the current supported reproduction entry points for v0.6.0. It separates auditable benchmark workflows from archived paper-era scripts and exploratory examples.

For paper-exact reproduction, pin zeroproofml==0.4.3 or use the matching v0.4.3 release tag. v0.6.x is the active development line; paper-era bundles should be interpreted under their recorded schema-v1 metadata rather than reinterpreted as current schema-v2 provenance bundles.

Quick Reproduction Path

The paper-facing replay contract lives in artifacts/paper_2026/. It records commands, configs, expected outputs, tolerance bands, checksums, and pinned container recipes.

From a clean checkout:

python -m pip install -r requirements-paper.txt
python -m pip install -e .
make reproduce-paper

Run one domain:

make reproduce-dose
make reproduce-rf
make reproduce-ik
make reproduce-reference-robotics

Override output root or device:

make reproduce-dose REPRO_ROOT=results/tmp_repro REPRO_DEVICE=cpu

For paper-grade reruns, retain:

the git commit
artifacts/paper_2026/
the produced run directory
run-local manifest.json and provenance.json

Scientific Claim Benchmarks

Run smoke mode locally:

python -m zeroproofml.benchmarks dose --mode smoke --device cpu --seeds 1
python -m zeroproofml.benchmarks rf --mode smoke --device cpu --seeds 1
python -m zeroproofml.benchmarks ik --mode smoke --device cpu --seeds 1

Resume a paper-mode run:

python -m zeroproofml.benchmarks dose --mode paper --device cpu \
  --out-root results/benchmarks/dose/<run_dir> --resume

Regenerate a report:

python -m zeroproofml.report benchmark results/benchmarks/dose/<run_dir> --html-report

Expected output root:

results/benchmarks/<domain>/run_<timestamp>_<sha>/

Core artifacts:

seed_*/per_seed_result.json
manifest.json
provenance.json
resume_state.json
RUN_REPORT.md
optional RUN_REPORT.html
aggregated/summary.json
aggregated/paired_stats.json
aggregated/benchmark_metrics.jsonl
CLAIM_AUDIT.md
regenerated figures under figures/

Domain Notes

Domain	Focus	Extra artifacts
DOSE	Censoring, operating points, direction-aware bottoms	`dose_operating_points`, `dose_pareto_front`, diagnostics, direction-head summaries, provenance-weighted bottom-cost signal when fault/semantic splits are present
RF	Resonator response, peak retention, extrapolation	signal traces, frequency-response SVGs, qualitative failure packs
IK	Robotics inverse kinematics near singularities	workspace heatmaps, determinant-stratified metrics, fallback route plots

Use generated artifacts for papers and reviews. Avoid hand-curated notebook summaries when a benchmark report can be regenerated from stored JSON.

Reference Robotics Deployment

End-to-end reference path:

python scripts/reference_robotics_deployment.py --device cpu --epochs 1 --n-samples 2000

Expected artifacts under results/reference_deploy_robotics/:

output_contract.json
inference_summary.json (includes hybrid_path_metrics, fallback_family_benchmark, and provenance-aware routing comparisons and materiality when provenance diagnostics are present)
strict_inference_audit.json
bundle/model.onnx
bundle/metadata.json
bundle/VALIDATION_REPORT.md
bundle/VALIDATION_REPORT.summary.json

Importable path:

from zeroproofml.reference_robotics_deployment import (
    ReferenceRoboticsDeploymentConfig,
    load_reference_robotics_deployment_artifacts,
    run_reference_robotics_deployment,
)

artifacts = run_reference_robotics_deployment(
    ReferenceRoboticsDeploymentConfig(device="cpu", epochs=1, n_samples=2000)
)
same_run = load_reference_robotics_deployment_artifacts(artifacts.out_root)

Provenance-aware routing (fault → analytic fallback, semantic → reject) is a stable path in v0.6.x. Reference summaries record provenance_routing_comparison, provenance_routing_materiality, and provenance_benchmark_evidence when the diagnostics are available.

Trajectory Evaluation

Generate a stratified RR IK trajectory dataset:

python scripts/generate_reference_robotics_trajectory_data.py \
  --n-trajectories 48 --steps-per-trajectory 16

Evaluate a policy:

from zeroproofml.reference_robotics_trajectory_eval import (
    evaluate_reference_robotics_trajectory_policy,
    make_reference_robotics_dls_policy,
)

summary = evaluate_reference_robotics_trajectory_policy(
    "results/reference_robotics_trajectory_eval/rr_trajectory_eval_dataset.json",
    make_reference_robotics_dls_policy(damping=0.05),
)
print(summary["aggregate"]["mean_tracking_error"])

The returned summaries include tracking error, fallback rates, joint-limit violations, chattering events, and latency-budget violations. When policies tag route kinds with fault or semantic, the summary also exposes provenance-aware splits.

Downstream Pipeline Simulator

Use the downstream simulator to test whether reject flags, provenance labels, and direction labels survive multi-step handoffs:

from zeroproofml.downstream_pipeline import (
    DownstreamPipelineReferenceSample,
    build_downstream_pipeline_simulator,
    compare_downstream_pipeline_strategies,
    write_downstream_pipeline_report,
)

simulator = build_downstream_pipeline_simulator(
    "5-step",
    drop_reject_flag_probability=0.05,
    bad_downstream_behaviors=("json_roundtrip", "aggregate_mean"),
)

comparison = compare_downstream_pipeline_strategies(
    [
        DownstreamPipelineReferenceSample(
            decoded=(-3.0,),
            should_reject=True,
            provenance="semantic",
            direction_label="below",
            sample_id="censored_low",
        ),
    ],
    simulator,
)

write_downstream_pipeline_report("artifacts/composability", result=comparison)

This is an experimental harness for composability evidence, not a stable core SCM API.

Examples Inventory

Comparing Runs

Compare a new run with one or more baselines:

from zeroproofml.benchmarks import compare_benchmark_runs, load_benchmark_run

run = load_benchmark_run("results/benchmarks/dose/<new_run>")
comparison = compare_benchmark_runs(
    run,
    ["results/benchmarks/dose/<baseline_run>"],
)
print(comparison.to_dict())

Each run records git commit, package versions, arguments, dataset fingerprints, hardware metadata, checkpoint hashes, discovered bundle directories, and dirty-worktree state. Resumed runs preserve attempt history.

Upgrade Notes

Most v0.6.0 changes are additive, but strict-mode flattening can change masks at singular edge cases and can refuse expressions that field-rational simplification previously accepted. When upgrading, re-check:

examples/fru_strict_check_demo.py
the RR-IK reference deployment
the DOSE matrix and artifact path

Existing exported bundles continue to validate under their recorded schema and metadata. They are not silently reinterpreted under the hardened v0.6.0 schema.

Archived Workflows

Older paper-era scripts remain in the repository for historical reference, but current docs and reproducibility claims should use the benchmark harness, reference deployment, and paper bundle above.

Docs