[ezutfen]

Canonical issue: Zutfen-LLC/inferswarm#2 — ROADMAP Phase 0, "Establish reproducible RTX 3060 FreeToken baseline".

Rules implemented here are fixed in advance by InferSwarm's Phase-1 success criteria (§1.1, §2, §3, §9, §10, §13) and its benchmark contract. Neither document is duplicated into this repo; benchmarks/README.md links to them and covers usage only.

No results are claimed by this PR

This PR claims no benchmark numbers of any kind. It was developed without the target RTX 3060, so nothing here was measured on the declared hardware, and no result directory is created — in this repo or in InferSwarm. The only committed text fixture anywhere in the diff is a smoke-test prompt explicitly labelled as not a workload.

This is tooling and instrumentation. No distributed execution, no second-GPU expert path, no Phase-1 candidate.

Review fixes (commit 50a91b9)

Review found that the harness could still produce an apparently canonical, COMPLETE, MEASURED artifact while a precommitted requirement had failed. The invariant it now enforces:

A canonical-looking Phase-0 artifact exists only when every prerequisite, held-constant rule, workload-shape rule, instrumentation requirement, provenance requirement and repetition requirement was actually satisfied. Anything else is explicitly INVALID, INCOMPLETE or NON-CANONICAL.

Campaign validity is now a first-class answer, separate from execution completeness (inferswarm_phase0/validity.py). run.json carries execution_status (COMPLETE / INCOMPLETE) alongside validity (VALID / INVALID / NON_CANONICAL) and a structured campaign_invalidations list with stable reason codes (bench_bw.failed, workload.prompt_shape_violation, prefill.ambiguous_records, gpu.mismatch, held_constant.mismatch, …). The bare canonical: true boolean is gone — it was the field a reader would mistake for a verdict. MEASURED now labels an observation: repetitions that really happened keep it whatever the campaign verdict is, and that never promotes the campaign to a valid baseline. Both run.json and the first screen of SUMMARY.md lead with one of VALID CANONICAL CAMPAIGN / INVALID CANONICAL ATTEMPT / NON-CANONICAL DEVELOPER RUN / INCOMPLETE RUN, with the reasons immediately below — keyed off the overall campaign state, not the repetition protocol alone (--allow-missing-provenance leaves the protocol untouched and still makes a run non-canonical).

ft bench bw is a session-level prerequisite, not a B2-local side effect (inferswarm_phase0/bench_bw.py). B2 resolves its per-step fetch split from the profile and B3's --moe-backend auto reads the same profile (engine._adjust_config → load_backend_recommendation), so the refresh runs once, before the sweep traversal, in either direction — a reversed session runs B3 first and would otherwise consume a stale profile. --no-bench-bw is refused for a canonical sweep; a failed command, an unreadable profile, or a profile benched on another card aborts before any server starts. The record pins the command, both timestamps, the return code, the selected GPU UUID, the resolved profile path, the profile's contents and its sha256, and checks the profile's own gpu.uuid against the campaign's card.

Physical-GPU provenance is proven, not asserted (inferswarm_phase0/gpu.py). --gpu is required for a canonical run and is resolved to a stable UUID through freetoken.gpu_select — the same selector ft serve and ft bench bw use, so no second policy exists — with the UUID and the nvidia-smi index recorded. The resolved UUID is what every child process receives: the servers, the bandwidth refresh, and both microbenchmarks. After each server is ready, the UUID the engine reports for itself (/v1/stats gpus, filled from its own bound device) is compared with it, and a mismatch invalidates. The nvidia-smi provenance query now asks for index, without which a numeric selector cannot be correlated with a row.

Workload shape and output length are contract, not commentary. A prompt outside its frozen class rule and a completion_tokens that is not the requested max_tokens each invalidate the canonical block; the observation is preserved in full and the prompt is never rewritten or truncated. The criteria state W1/W2 as bounds and W3/W4 as ""; "" is not machine-checkable, so the tolerance is frozen in version control (±15 % / ±25 %) and reproduced verbatim in every artifact rather than chosen after seeing a fixture's token count.

Prefill instrumentation is a validity gate. Records are attributed by request uid — the response id is chatcmpl-<uid> and every instrumentation record is stamped with the same uid — rather than "newest record above a sequence floor". Where the id shape is unrecognized the sequence rule still applies, but more than one candidate is reported ambiguous instead of resolved by taking the newest, which would attribute another request's interval to this one. Unavailable endpoint, disabled instrumentation, no fresh record, ambiguity, shared batch and unusable timing each carry a stable code and each invalidates a canonical measured repetition. Warmups are exempt: §10 discards them by construction. For --dev-smoke, an explicit null plus reason remains acceptable and the run stays NON_CANONICAL.

CORRECTNESS_REFERENCE forces greedy request sampling on the same frozen prompt fixture, and records the override alongside the manifest's own values. --sampling-defaults none was not sufficient: the manifest states temperature / top_p / top_k explicitly in every request body and a request-level value beats a server default, so a manifest whose frozen performance sampling is deliberately realistic would have made the correctness reference sampled — with no seed available to make that reproducible (§5.3). The performance sweep keeps the manifest's frozen sampling untouched.

The single-expert microbenchmark now measures a single expert. top_k = 1, one routed expert per call, timed directly. The grouped top_k step is retained as a separately named grouped/batched diagnostic and is never divided by top_k: experts inside a grouped call execute concurrently, so step_ms / top_k is an amortized throughput-like quantity the hardware never produced. Both it and the bandwidth benchmark bind the process through FreeToken's own gpu_select path and record the UUID they actually bound, refusing rather than misattributing — the old code built torch.device("cuda") and merely recorded the requested selector.

Device (VRAM) memory bandwidth is measured (--device-bandwidth), which issue #2 asks for and ft bench bw does not provide — its ceilings are host DRAM and the PCIe link. A device-resident D2D copy over a working set far beyond L2 (512 MiB per buffer), CUDA-event timed after warmup, every repetition reported individually, byte accounting stated both ways (read+write and read-only), labelled MEASURED and diagnostic-only.

Dirty-tree and model-pin safeguards. A canonical run refuses a dirty FreeToken checkout and names the modified paths — a dirty tree cannot be reproduced from its commit SHA, and recording only the filenames does not make it reproducible. --inferswarm-commit must be a full 40-hex SHA. If the checkpoint path is a Hugging Face .../snapshots/<sha> layout, that SHA and the models--<org>--<name> cache entry are reconciled against --model-revision / --model-repository and a disagreement is a refusal; a non-snapshot path records "cannot cross-check" rather than guessing.

Held-constant and resolved-configuration audit. The runtime report gained the §2.3 fields it was missing — the resolved --max-prefill-length and --cache-type (both live on SchedulerConfig, and --cache-type is rewritten by _adjust_config for SWA/GDN models) and the resolved hybrid fetch fraction. A missing or null required resolved field invalidates, as does a held-constant value differing across arms, or B3 resolving to a MoE backend — or an NVFP4 expert path — that coincides with neither B1 nor B2. The resolved cache slots and KV page count are deliberately not held constant: §3 rule 2 asks for them to be reported, not equalized.

--dry-run now reports what would refuse the run, so a plan cannot print CANONICAL while it would abort on a dirty tree or an unresolvable --gpu.

What is now mechanically reproducible

• The exact B1–B5 sweep, with every value the criteria fix passed explicitly. --nvfp4-backend is stated on every arm because EngineConfig.nvfp4_backend defaults to "triton", not auto — a row that only said "auto" in prose would silently run Triton and duplicate B4. --moe-cache-auto is stated for the same reason: the CLI applies it when no sizing flag is given, the dataclass does not.
• Resolved configuration read back off the running engine, not re-derived: which MoE backend auto selected, whether --nvfp4-backend was inert for the executing expert path (decode_target="cpu" under cpu/hybrid, and under an offload run whose layers _auto_cpu_layers locked), whether _auto_cpu_layers fired, resolved cache slots/bytes and coverage, the resolved hybrid fetch fraction, the resolved --max-prefill-length and --cache-type, and whether the Marlin 992-slot cap applies and whether it bound (established by re-solving the same pure plan uncapped, not by guessing from equality with 992). Served by a new read-only GET /v1/instrumentation.
• B1/B4 equivalence is representable, not an error. They differ only in --nvfp4-backend; if auto resolves to triton on the rig, the two are recorded as observations of one resolved configuration. Nothing installs or forces Marlin to make them distinct.
• Prefill throughput measured where prefill happens. CUDA events bracket the prefill model forward in Engine.forward_batch and are summed per request across chunked prefill (Scheduler._accumulate_prefill). prompt_tokens / TTFT is never computed — TTFT also contains transport, tokenization, template rendering, queueing, sampling, detokenization and the SSE hop. A multi-request prefill batch is marked shared_batch with token counts withheld rather than split by an invented model.
• The precommitted protocol: 2 discarded warmups + 10 measured generations per (arm, class), every repetition preserved with its full inter-token gap list, recorded execution order, distinct session ids, and a reversed traversal for the second session. Overrides require --dev-smoke and stamp the run NON-CANONICAL in run.json, in SUMMARY.md, and in the --dry-run banner.
• Provenance with explicit nulls-plus-reason throughout; a canonical run refuses to start with missing required provenance, a dirty checkout, an unresolvable --gpu, a --model-revision that is not a 40-hex commit SHA, an --inferswarm-commit that is not one, or a local snapshot that contradicts either. Credential-shaped env names are excluded by name; the local model path stays out of the runtime report. The resolved expert weight format is backfilled from what the arms actually reported, and per-arm disagreement invalidates the campaign (§3 rule 4) rather than being smoothed into one value.
• Artifacts that cannot flatter themselves: execution_status is derived from expected-vs-observed repetition counts plus the failure list, and validity is derived independently from the invalidation list, so neither a lost repetition nor a broken prerequisite can read as success. The runner computes no cross-configuration ratio and selects no baseline — CANONICAL_PERFORMANCE_BASELINE is chosen by a human from the completed campaign (§2.2), and mid-campaign ratios are prohibited (§10).
• CORRECTNESS_REFERENCE as a separate subcommand, so it can never be conflated with the performance baseline: fixed --moe-cache-size, explicit resolved --nvfp4-backend, --moe-cpu-layers 0, --sampling-defaults none, forced-greedy request sampling, full output text retained. Its self-consistency check is two runs under different session ids compared on the recorded output_sha256.
• Hardware profile subcommand: GPU/VRAM identity, driver, compute capability, PCIe link generation and width (current and max), nvidia-smi topo -m, CPU/RAM/OS, the ft bench bw host-DRAM/PCIe measurements, an optional device (VRAM) memory bandwidth benchmark, and an optional true single-expert NVFP4 decode-GEMV latency measurement plus its grouped-step diagnostic. Every microbenchmark block is marked diagnostic-only and is never mixed into a throughput claim.

What still requires the RTX 3060

Everything that is a number. Specifically: pinning the exact nvidia/Qwen3.6-35B-A3B-NVFP4 upstream revision (§1.1 — inventing a SHA is a fabricated provenance record, so none is guessed here); the B1–B5 sweep itself and therefore the selection of CANONICAL_PERFORMANCE_BASELINE; the CORRECTNESS_REFERENCE capture and its greedy self-consistency check; the hardware profile, the VRAM-bandwidth and expert microbenchmarks; and the resolved-configuration facts that only exist at runtime — what auto picks on sm_86, whether vLLM's Marlin donor symbols are usable, whether the 992-slot cap binds, whether _auto_cpu_layers fires on the host, and what ft bench bw actually recommends for NVFP4 experts on this card.

W1–W4 fixtures

Coordinated with Zutfen-LLC/inferswarm#3, which supplies the realistic routing fixtures. This PR ships the container: the manifest schema, the validator, and the freezing helper (phase0_baseline.py hash). It deliberately invents no W1/W3/W4 content — the only example manifest declares canonical: false, holds one synthetic prompt labelled "SMOKE TEST FIXTURE — NOT a Phase-0 workload", and is refused by a canonical run.

Tests run

New, hardware-independent (subprocess / server / GPU / HTTP all mocked):

tests/benchmarks/                           294 passed   B1-B5 command construction, explicit NVFP4
                                                         values, manifest validation + frozen shape
                                                         rule, warmup/repetition sequencing,
                                                         provenance + preflight refusals, the
                                                         session-level bench-bw prerequisite and its
                                                         ordering, campaign-validity gate, prefill
                                                         attribution, GPU resolution/verification/
                                                         binding, both microbenchmarks, CLI dry run
tests/daemon/                                31 passed
tests/engine/test_runtime_report.py          23 collected (see below)
tests/scheduler/test_prefill_measurement.py   8 collected (see below)
tests/server/test_instrumentation_doc.py      8 collected (see below)

tests/benchmarks/ grew from 125 to 294 tests. Regression coverage was added for every review finding, including: canonical --no-bench-bw is impossible; a failed bench-bw prevents any server from starting; the refresh precedes B3 in a reversed session; a stale/unreadable/wrong-GPU profile cannot yield a canonical run; a dirty tree is rejected; --inferswarm-commit must be a full SHA; an HF snapshot/revision mismatch is rejected; the GPU UUID is required, resolved and verified against the engine; a runtime GPU mismatch is rejected; the microbenchmark binds the requested GPU; the single-expert benchmark uses top_k=1; the grouped result is never called single-expert; a non-greedy manifest cannot make CORRECTNESS_REFERENCE sampled; prompt-shape and completion-length violations invalidate; stale/missing/ambiguous/shared prefill invalidates; a missing resolved-configuration field invalidates (parametrized over the whole required list); expert-quant and held-constant disagreements invalidate; an unexpected B3 resolution invalidates; the summary uses overall campaign validity; a canonical campaign cannot finish VALID while any invalidation exists; and the device-bandwidth output is correctly labelled and structured.

The full suite could not be run in the container this commit was produced in. pytest tests/ --ignore=tests/e2e aborts at collection there with 77 collection errors, all ModuleNotFoundError for torch, huggingface_hub and fastapi: the container has no CUDA toolchain and torch is not installable through its proxy (download.pytorch.org is blocked, and the PyPI wheel installs but cannot load libcublasLt.so). Only tests/benchmarks/ (294) and tests/daemon/ (31) collect and run there, and both are green. Consequently the previously reported "3 flashinfer-environment failures, identical on the base branch" could not be re-verified in this session — that comparison stands from the earlier run but is not re-attested here, and the full suite should be re-run on a machine with torch before merge. The tests/engine/, tests/scheduler/ and tests/server/ additions in this commit are hardware-independent by construction (the runtime report is built from a fake engine's attributes, as its module docstring explains) but were not executed here.

Remaining gaps, stated rather than papered over

• Ordering. §10's A/B/A/B interleaving governs the Phase-1 candidate-vs-baseline comparison, where both arms are alive at once. Phase 0's five arms are five server configurations, so the sweep is config-major with one ft serve per arm; interleaving would mean a model reload per generation. The realized order is recorded, and --reverse-order provides the reversed second-session traversal. Called out in protocol.py.
• Analysis is not in the runner, by design. Bootstrap CIs, R_c/R_agg, and the §10 noise-floor guard (baseline CV > 5 %) belong after the campaign completes; the runner emits per-block min/median/max/IQR/CV over the preserved raw repetitions and nothing comparative. A separate analysis step is still needed before a Phase-1 verdict.
• Correctness depth. This captures a reproducible deterministic fixture (forced greedy, ignore_eos, output hashes, full text). It is not the C1/C2/C3 Phase-1 instrumentation — per-layer MoE outputs, router selections at decode step 0, step-0 logit vectors — which is not reachable over HTTP and is not attempted here. The CLI says so on every reference run.
• Anti-starvation is enforced structurally, not proved. The sweep arms carry --moe-cache-auto and the harness offers no way to lower the cache below it; --memory-ratio / --kv-reserve-tokens are held constant across arms and that agreement is now checked against the resolved configuration rather than assumed from the flags. §3 rule 2's supplementary --num-tokens-pinned baseline run is a Phase-1 concern and is not implemented.
• Marlin cap binding is knowable only for an auto-sized cache; with an explicit --moe-cache-size the report says "unknown, with reason" rather than guessing, because the cap rejects an over-large size instead of clamping it.
• FREETOKEN_INSTRUMENT_PREFILL changes the served process (two extra CUDA events per prefill batch). It is off by default, the harness enables it identically on every arm, and each arm now confirms from the server that it actually took effect — so it cannot advantage one configuration. It is recorded in the run's provenance, and a campaign comparing against numbers taken without it would not be like-for-like.
• --allow-missing-provenance exists as an escape hatch and forces the run non-canonical; it is not a way to publish a baseline.
• The W3/W4 prompt tolerances are the harness's own reading of "~", frozen before any measurement and recorded in every artifact. If InferSwarm later fixes an exact rule, changing them is a reviewable change to a frozen contract — not a per-run accommodation.

Follow-up benchmark-integrity fixes (0b3f88e)

• Canonical sweeps are locked to --bench-bw-dtype nvfp4; alternate formats remain available only to explicitly NON_CANONICAL developer smoke runs.
• The fresh profile must positively match the selected GPU and contain machine-usable NVFP4 CPU-MoE/PCIe measurements. The harness validates the exact captured file with FreeToken's own backend-recommendation and hybrid-fraction readers before any server starts. B2 also invalidates if hybrid reports the 0.0 fixed-cap fallback instead of a positive resolved fraction.
• /phase0-runs/ is ignored, and the documented hardware profile now writes to phase0-runs/hardware-profile.json, so session 1's own artifacts cannot dirty session 2's clean-tree preflight.

Verification: tests/benchmarks/ 310 passed; tests/daemon/ 31 passed (FastAPI supplied to the lightweight test environment). No Phase-0 measurements or result directories were produced. The remaining work is the real RTX 3060 hardware profile and B1-B5 validation after discovering the GPU UUID and pinning the actual upstream Qwen checkpoint revision.