Repository navigation
feat(decision): optional scoring API and reproducible reference baseline #585
Copy link
Copy link
Closed
Labels
decision-inferenceOptional decision scoring, classifier heads, calibration and quality-matched inference benchmarksOptional decision scoring, classifier heads, calibration and quality-matched inference benchmarksenhancementNew feature or requestNew feature or request
Description
Parent: #583
Planned branch: codex/decision-scoring (created from main d2641c9).
Problem / motivation
Consumers can score alternatives through peek_logits, and tools/eval_geist.c already implements SCOREALT, but there is no dedicated, optional numeric decision interface with explicit ownership, score semantics and a reproducible baseline.
Proposal / scope
- Add an EXPERIMENTAL API in a dedicated header (proposed
include/geist_decision.h) with a decision handle/options and a per-instance session/workspace. Reuse immutable model weights and current architecture dispatch. - Start with tokenized input and distinct single-token candidate IDs. Applications supply their own templates and map candidate IDs to domain labels. Multi-token labels must be rejected or handled by a separately explicit sequence-scoring API, never truncated to their first token.
- Establish the implementation using the existing prefill + dense-logits path; optimization is a follow-up.
- Define selected logits and probabilities normalized over supplied candidates separately from full-vocabulary probabilities. Neither is advertised as calibrated correctness confidence.
- Add a default-off decision build feature (proposed
DECISION=1) and explicit instance-level mode selection. Keep a documented disabled-build contract withGEIST_E_UNSUPPORTEDentry-point stubs/capability reporting. - Specify reset, borrowed-buffer lifetimes, model/handle destruction order and thread-safety. No shared mutable default mode and no model-weight mutation.
- Create the reusable reference protocol/harness from existing scoring tools; application-specific policy and datasets stay outside the library.
Acceptance
- A small supported model scores explicit candidate IDs through public headers; a test consumer needs no private headers.
- Dense reference scores and candidate normalization match independently checked
peek_logitsresults, including model-specific softcaps. - Invalid/duplicate token IDs, empty alternatives, capacity overflow, unsupported model/backend and non-finite scores have explicit behavior and defined failure outputs.
- Repeated queries reset all relevant state, including qwen35 recurrence; interleaved instances cannot contaminate each other.
- Feature-on/off builds are exercised; feature-off use fails explicitly. Existing generation, embedding and logits access retain their contracts.
- Workspace is sized with checked arithmetic and allocated at creation, not per token/layer/block. API parameter contracts follow AGENT.md.
- The baseline records model/runtime hashes, tokenization, prompt/candidate lengths, backend/hardware, warm/cold cache conditions, concurrency, p50/p95, quality and raw measurements. Include an optimized one-token/score baseline alongside longer generated answers.
- Public additions receive CHANGELOG/API-contract notes and the applicable format, release, sanitizer and optimized gcc checks.
Dependencies
No Bonsai reimplementation is required: main already supports PQ2_0 and prism.hadamard. #548 is relevant to later shared-prefix branching but must not block independent-query scoring.
Implementation: draft PR #592 (69e91ca). All 23 executed CI checks pass; local real-GGUF parity covers Qwen3, Gemma4 softcaps and Bonsai 2 27B. Raw smoke measurements and their limits are included in the PR. The issue remains open until merge.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
decision-inferenceOptional decision scoring, classifier heads, calibration and quality-matched inference benchmarksOptional decision scoring, classifier heads, calibration and quality-matched inference benchmarksenhancementNew feature or requestNew feature or request
Activity
feat(decision): add isolated opt-in candidate scoring