Repository navigation
eval(decision): Bonsai 2 quality, calibration and latency without generated reasoning #587
Description
Parent: #583
Depends on: #585
Use #586 for the final optimized comparison; collect the reference results before that optimization.
Planned branch: codex/decision-evaluation.
Problem / motivation
Removing generated tokens can yield a large speedup, but Bonsai's published thinking-mode scores do not establish its immediate-decision quality. We need measured results before selecting classifier training or making Jev-like speedup claims.
Scope
Confirmed rollout: start on Apple Silicon (cpu_neon and Metal), with MMLU first. Other devices/backends follow later. Pin the MMLU version/split and prompt/answer-token construction, and declare the generative quality baseline and non-inferiority margin before any quality-matched speedup comparison. The synthetic #586 timing matrix is performance-only and supplies no MMLU accuracy evidence.
Build on the current Bonsai implementation, existing eval_geist/MMLU adapters and the protocol established in #585. Keep reusable runtime evaluation mechanisms here; application policy and domain datasets belong to the consumer.
Compare on identical held-out questions:
- Direct answer-token scores without generated reasoning.
- An optimized single-answer-token generative baseline.
- Ordinary generative/reasoning decisions, counting every generated reasoning and answer token.
- The selected-row optimization once perf(decision): compute only requested output rows behind the scoring API #586 is ready.
A hosted Jev run is optional only when account access and benchmark spending are explicitly authorized. Published provider speedups are contextual references, not local measurements or acceptance thresholds.
Acceptance
- Pin Bonsai checkpoint/hash, runtime revisions, tokenizer/template, backend/device, numerical modes, candidate ordering, input lengths, question count, cache policy and concurrency.
- Recheck same-token runtime parity against a pinned Prism reference using the existing coverage from Bonsai-27B: MMLU 0.44 against the PrismML fork's 0.76 on identical tokens #432; include longer/high-entropy cases, not only a short greedy smoke test.
- Use separate development, calibration and held-out test data with documented split policy. Include representative negative/ambiguous cases and label-order sensitivity.
- Report accuracy or macro-F1, log-loss/Brier score and reliability/calibration results, plus p50/p95 latency, throughput, memory and cold/warm behavior with raw samples.
- Any post-hoc temperature scaling is fitted only on the calibration split and tied to the checkpoint, precision/backend policy and prompt construction. Uncalibrated option probabilities remain clearly identified.
- Define the quality target/non-inferiority margin before comparing speedups. Report speedup at that target; report any quality loss instead of trading it away silently.
- Count preprocessing, all model calls and any abstention/fallback route in the reported end-to-end policy.
- Conclude whether token scoring is adequate, a trained head is justified, or backbone adaptation is needed. Record negative findings too.
Existing dependencies / history
PQ2_0 and Hadamard support are already implemented. #432 and #561 are closed fixes. Link any newly reproduced backend issue to its existing owner rather than duplicating Bonsai support. Reuse #364 for performance methodology.
Activity
bench(decision): record Apple projection savings and query latency li…
geisten commentedon Oct 4, 2026
Infrastructure continuation: draft PR #608 on
codex/decision-evaluation(head403167e). #592 and #599 are merged.Implemented offline pinned classic-MMLU preparation with explicit overlap policy and separate development/calibration/test, actual GGUF template/token-boundary validation, filtered benchmark arms with complete generation IDs/text/stop data, strict final-answer scoring, conditional log-loss/Brier/reliability and bound temperature scaling. The runner freezes the declared policy before inference and rejects pilot/subset masquerading, invalid/capped reasoning baselines and changed deterministic outputs for quality-matched claims.
Validation: format gate, release/ASan unit suites (each 70 pass / 45 supported-platform or fixture skips / 0 failures), full Python suite including 17 new cases, optimized GCC 16 driver compile, plus serial CPU NEON and Metal synthetic runner integration with exact selected/DENSE logits. These are implementation tests, not Bonsai MMLU quality or useful latency measurements.
Still open before acceptance: user-selected MMLU variant, margin and campaign scope; downstream overlap/cap policy; native pinned Prism same-token parity; representative negative/ambiguous controls; actual calibration and held-out campaigns; a measured conclusion about trained-head/backbone adaptation. No acceptance checklist is marked complete by the framework alone. Published Bonsai Redux/thinking scores are not substituted for classic five-shot greedy results. No hosted Jev service run is authorized or attempted.
geisten commentedon Oct 4, 2026
Confirmed campaign requirements are now recorded in
docs/DECISION_CAMPAIGN_PLAN.jsonon PR #608: classic MMLU5-shot, maximum 2 percentage points accuracy loss with a conservative paired 95% interval, small development pilot first.The runner now supports deterministic label-independent subject limits (
--per-subject 1 --subject-limit 8), and rejects bounded populations from full held-out quality runs. A development-only hypothetical BOS/TOK audit of the proposed eight-question selection checked all 96 actual candidate boundaries across 24 prefixes; prompt lengths are 399–695 tokens. No score or generation commands were used. Planning estimates with those lengths are 18–26 minutes on Metal and 88–124 minutes on CPU for two trials/all eight arms/cap512 if all reasoning outputs reach the cap; setup and shared-host load are excluded, and these are not measured campaign times.Still awaiting downstream choices: explicit duplicate/overlap policy and exact bounded generation/pilot policy. Proposed: exclude and report cleaned subset; eight questions from eight subjects, cap512, Metal first. No MMLU quality result yet, and the broader external parity/calibration/held-out acceptance items stay open.
The first CI head had 22 successful checks, one intentional coverage skip and a Linux-arm64 Python-fixture failure from intercepted CPU discovery. This is fixed and the complete local Python suite passes, including 18 new cases; updated CI is queued/running for head
97fb9c4. C/runtime behavior is unchanged by that test correction.geisten commentedon Oct 4, 2026
The user approved all remaining pilot choices; the frozen development pilot is now complete on Metal and CPU NEON in #608.
cais/mmlurevc30699e8356da336a370243923dbaf21066bb9fe, 5-shot, exact overlap exclusion, eight deterministic development questions/eight subjects, rotation0. This is a cleaned subset, not a stock-MMLU or MMLU-Redux score.3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1; Apple M1 Max 64 GiB, six threads, concurrency1, FP32 KV, greedy, cap512, two total trials/arm. Metal first, then CPU; 128 model calls per backend.Evidence: writeup, compact JSON, raw bundle and SHA-256 index.
These measurements bind clean source
13261ac73fb2841ef565bcabda5473eacd1e8ce0, with post-run model/binary/tokenizer/evaluator hash checks. The result was recorded before integrating newer main merges. Newer KV sizing/residency/snapshot changes are unmeasured by this artifact and require a newly bound acceptance run.This completes only the approved development pilot, not #587. Next: native Prism same-token parity with full recurrent-state reset, then a development-only cap study that yields complete reasoning answers. Freeze the revised policy before calibration/held-out evaluation. Ambiguous/negative controls, label sensitivity, calibration and the full cleaned held-out quality run remain open; #588 training remains gated. No Jev-equivalence claim or production rollout follows.
geisten commentedon Oct 5, 2026
The accepted pretrained-model comparison is implemented and measured in draft PR #608 on
codex/decision-evaluation. ModernBERT-Large-Instruct and Laya Typed Decisions are modular, default-off Python reference adapters; native geist C encoder architectures are not implemented by this step. No training, paid compute, calibration fitting or held-out inference ran.Apple M1 Max 64GiB, FP32 PyTorch MPS, six threads, concurrency one; all 778 cleaned classic-MMLU development questions:
Rejected inputs count as incorrect; no prompt is silently shortened. The larger context was explored after the initial four results. Its continuous MPS process ran out of memory after 614 questions, although the failing 1,703-token request succeeds in a fresh process. The remaining 164 questions were explicitly evaluated in a second process with identical checkpoint/source/dtype/prompt policy; the original failure remains recorded. *Recovered latency excludes OOM/restart/reload and does not demonstrate continuous-service reliability. Process RSS substantially understates total driver memory in this case.
The eight-question CPU/MPS matrix has matching choices. Native ModernBERT mask-head/full-head controls and Laya SDK controls agree on candidate argmax/choices; all measured within-policy repeats are stable. The writeup links the compact result, pinned provenance, p50/p95, raw and shipped-temperature calibration metrics, and the 99-file checksum-bound evidence archive. Its verifier regenerates metrics from raw outputs without ML dependencies. Format/Python checks and release plus ASan/UBSan pass; each C unit suite has 76 passes, 47 platform/fixture skips, zero failures/errors.
Recommendation from development evidence: retain ModernBERT zero-shot as the next reference baseline. The five-shot geometry adds little accuracy here and is slower, with an unresolved MPS memory-growth issue. Laya remains useful for testing typed choice/score/noul interfaces, but this MMLU result does not justify treating it as a general Bonsai replacement.
This does not close #587 or release #588's training gate. Native Prism same-token controls, representative negative/ambiguous inputs, label rotations, calibration and held-out evidence remain outstanding. Different prompts/runtime policies and the earlier eight-question Bonsai binary prevent a quality-matched speedup claim. Jev equivalence and native C encoder support remain unestablished.
geisten commentedon Oct 8, 2026
Status (sweep 2026-10-08): still valid. This is the Apple-first decision campaign: the confirmed rollout is M1 Max with cpu_neon and Metal, continued in draft PR #608. Nothing on main blocks it. #588 and #589 stay deferred until its findings. Labelled
needs-hardwarefor the M1 Max runs.Generated by Claude Code
Germar12 commentedon Oct 16, 2026
Independent numerical check of Bonsai 2 27B decisions, and its accuracy without generated reasoning on JevBench's public items.
Setup (Apple M1 Max 64 GB, Metal, 14–15 October 2026, every run under the machine lock)
Ternary-Bonsai-2-27B-PQ2_0.gguf(SHA-2563907dc16…), profilebonsai2-27b-pq2-v1.b9de7a2, built withDECISION=1, geistlib engine3841fa6, Python package 0.3.4; raw readout (one prefill, key-token probabilities), no option rotation, no calibration, context 8192.7625ecb, the 231 public items (easy 48, original 72, hard 111); choice, yes/no and score items mapped to fixed options.01ae597e3(CMake Release, Metal),llama-server -c 8192 -ngl 99 -np 1, fed geistr's exact prompt token IDs;n_predict1,temperature-1 (plain softmax),n_probs100,cache_promptfalse; option probabilities renormalized over the option-key tokens.Agreement geistlib vs. Prism fork (same tokens)
Accuracy without generated reasoning (JevBench's own scorer)
Of the 146 items where the readout gave the top option ≥ 0.9, 99 % were right.
Limits: public items only (no sealed split, not a board score); the items are public, so training-data overlap cannot be ruled out; one machine, Metal only; no MMLU run (this issue's MMLU-first plan is not addressed by this). Two runs were interrupted and completed in parts; details in the data README.
Data, scripts and method are in the geisten webapp repository under
benchmarks/jevbench_geistr_2026-10-14/(not yet published; they go online with the accompanying article).