Skip to content

feat(decision): optional scoring API and reproducible reference baseline #585

Closed

Description

@geisten

Parent: #583
Planned branch: codex/decision-scoring (created from main d2641c9).

Problem / motivation

Consumers can score alternatives through peek_logits, and tools/eval_geist.c already implements SCOREALT, but there is no dedicated, optional numeric decision interface with explicit ownership, score semantics and a reproducible baseline.

Proposal / scope

  • Add an EXPERIMENTAL API in a dedicated header (proposed include/geist_decision.h) with a decision handle/options and a per-instance session/workspace. Reuse immutable model weights and current architecture dispatch.
  • Start with tokenized input and distinct single-token candidate IDs. Applications supply their own templates and map candidate IDs to domain labels. Multi-token labels must be rejected or handled by a separately explicit sequence-scoring API, never truncated to their first token.
  • Establish the implementation using the existing prefill + dense-logits path; optimization is a follow-up.
  • Define selected logits and probabilities normalized over supplied candidates separately from full-vocabulary probabilities. Neither is advertised as calibrated correctness confidence.
  • Add a default-off decision build feature (proposed DECISION=1) and explicit instance-level mode selection. Keep a documented disabled-build contract with GEIST_E_UNSUPPORTED entry-point stubs/capability reporting.
  • Specify reset, borrowed-buffer lifetimes, model/handle destruction order and thread-safety. No shared mutable default mode and no model-weight mutation.
  • Create the reusable reference protocol/harness from existing scoring tools; application-specific policy and datasets stay outside the library.

Acceptance

  • A small supported model scores explicit candidate IDs through public headers; a test consumer needs no private headers.
  • Dense reference scores and candidate normalization match independently checked peek_logits results, including model-specific softcaps.
  • Invalid/duplicate token IDs, empty alternatives, capacity overflow, unsupported model/backend and non-finite scores have explicit behavior and defined failure outputs.
  • Repeated queries reset all relevant state, including qwen35 recurrence; interleaved instances cannot contaminate each other.
  • Feature-on/off builds are exercised; feature-off use fails explicitly. Existing generation, embedding and logits access retain their contracts.
  • Workspace is sized with checked arithmetic and allocated at creation, not per token/layer/block. API parameter contracts follow AGENT.md.
  • The baseline records model/runtime hashes, tokenization, prompt/candidate lengths, backend/hardware, warm/cold cache conditions, concurrency, p50/p95, quality and raw measurements. Include an optimized one-token/score baseline alongside longer generated answers.
  • Public additions receive CHANGELOG/API-contract notes and the applicable format, release, sanitizer and optimized gcc checks.

Dependencies

No Bonsai reimplementation is required: main already supports PQ2_0 and prism.hadamard. #548 is relevant to later shared-prefix branching but must not block independent-query scoring.

Implementation: draft PR #592 (69e91ca). All 23 executed CI checks pass; local real-GGUF parity covers Qwen3, Gemma4 softcaps and Bonsai 2 27B. Raw smoke measurements and their limits are included in the PR. The issue remains open until merge.

Activity

  1. added
    enhancementNew feature or request
    decision-inferenceOptional decision scoring, classifier heads, calibration and quality-matched inference benchmarks
    on Oct 3, 2026
  2. added a commit that references this issue on Oct 3, 2026
    5cbdce1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    decision-inferenceOptional decision scoring, classifier heads, calibration and quality-matched inference benchmarksenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions