Skip to content

Add nativegen provider adapter and factory wiring - #1736

Closed
chris-beckman wants to merge 192 commits into
feat/native-generation-enginefrom
feat/native-generation-provider
Closed

chris-beckman wants to merge 192 commits into
feat/native-generation-enginefrom
feat/native-generation-provider

Conversation

@chris-beckman

@chris-beckman chris-beckman commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Depends on #1734 → #1811 → #1812, then #1735 and #1737. All other open work must land first. Base: feat/native-generation-engine. Downstream order: #1736 → #1846 → #1761 → #1762.

The native provider shares one bounded MLX/Vulkan runtime across registry views and model roles. The Go host owns artifact leases, physical-device admission, exact rendered request budgets, cancellation and recovery. Generation and Bialy remain resident; synchronized execution steps share device admission and, on MLX, explicitly charged transient workspace. Whole bounded Bialy requests receive priority without routinely unloading either model.

Context admission inspects native allocation plans and selects the largest fitting 256-token cache block, retaining the exact configured cap when it fits. Six-head, full-length MLX decision probes support a 512 MiB two-row workspace reservation; 256 MiB failed the enforced bound. The current native 48K context and six-head decision plan fit the logical 8 GiB Mac partition. This accounting fixture does not establish physical-8GB performance.

Complete prompts are preserved. Native token measurement supplies request budgets; utility calls preserve the retained conversation checkpoint. The provider accepts bounded exact continuation reuse as well as static-prefix reuse. The existing budget target exports complete static request fixtures for native measurement without tuning prompt content or removing schemas.

First use imports an authenticated packaged seed into device-persistent storage. Retry and repair preserve reader leases until reaping, verify payloads and warm the replacement. Readiness reports host-owned revision, admitted context and recovery actions; #1762 supplies the UI. Normal Mac and Nix build paths stage verified runtimes, shaders, libraries, notices and explicitly supplied signed seeds. No final trained release artifact is selected.

The Linux driver adapter serves inventory, generation and Bialy. It preserves the closure's Mesa ICDs and loads version-matched system NVIDIA components only into inference children. Temporary ICDs live until their readers exit. Corrected Vulkan normalization restores useful F16 Bialy scores; matched prototype scores differ from MLX by at most 0.0036 for ranks and 0.0012 for probabilities in the measured fixture. Persistent compilation-cache preparation uses the approved decide:prepare target.

Verification:

  • Current provider continuation fixtures: 83 results passed, including invalid acknowledgement bounds. The 48K local-model/provider batch passed 103 results; six-head workspace changes passed 93 decision/local-model results.
  • Normal staged M4 and AMD hosts passed cancellation, verified-payload repair and clean shutdown. Composed overlapping-generation decision trials completed without abstention on M4 (1.413/1.153 s) and NVIDIA (1.269/1.043 s). These are prototype observations, not timing guarantees.
  • Nix native builds and AMD/NVIDIA lifecycle probes passed. Fresh NVIDIA driver-cache warm-up removed the observed 19.781 s request stall; the slowest measured request step was 0.894 s after 33.657 s startup. The shared NVIDIA service was restored healthy after every window.
  • Store/export/signing fixtures and authenticated seed staging passed in the composed store work. The separate stop-settlement fix fix(session): settle late prompt completion as an intentional stop #1846 passed its browser stop/follow-up check.

A normally staged M4 application completed two turns in 18.897 s and 3.723 s. The follow-up reused 8,610 of 9,191 prompt tokens, using static-prefix reuse because earlier prompt content changed. Native Vulkan 35K-token cold/reuse/disk-restore/replay now has exact output parity and bit-identical teacher-forced logits; current composed AMD verification follows separately.

Idle context preparation accepts the assembler's explicit standing/history boundaries, with foreground preemption and synchronized partial retention. Registry projection and noninteractive screening share the normal policies. Known history ends on the foreground prefill chunk grid; the bounded decoder separates valid Gemma thought-channel tokens from answer/tool output and rejects malformed boundaries. Eleven focused preparation/parser results pass. Native preparation/cancellation and fixed-seed cold/prepared comparisons are retained.

Final composed AMD and M4 builds pass. Immediate and interrupted application preparation complete on both platforms. The refreshed NVIDIA Nix closure passed seven stages: 35K prompt parity, application continuation, overlapping Bialy decisions, cancellation/repair, completed preparation, immediate submission and partial preparation. Native cold/reuse/restored first token measured 14.117 / 2.821 / 3.441 s, with 98–104 tokens/s decode. A stable application follow-up reused 10,245/10,920 tokens with 251 ms first output. Cancellation retained the process; repair replaced it and returned ready. The shared NVIDIA service was restored and verified healthy.

Limits: Bialy instruction changes invalidate prefixes; cold long prompts and first-use utility work can remain slow. One M4 prototype sample emitted an unmatched channel closer and failed. Reconstructed greedy and fixed-seed cold/prepared outputs matched, but do not reproduce the original random sample. Physical 8 GB/Intel execution, final trained-model quality, production signing inputs and signed installed-release qualification remain unqualified. Only prototype training exists. Cross-launch checkpoint persistence is explicitly deferred. No merge or release is requested.

@chris-beckman
chris-beckman marked this pull request as ready for review October 30, 2026 17:00
@chris-beckman
chris-beckman force-pushed the feat/native-generation-provider branch from 8e20185 to 5646359 Compare October 30, 2026 19:45
Introduce modelstore: content-addressed artifact staging with hash
verification, manifest activation gated on verification, and an
append-only journal where the newest entry per (model, device) wins.
A torn final journal line is recoverable.

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Stage enforces the declared size while copying, validates digest shape
before any path use, and syncs the artifact and its directory before
publishing. Activation requires a signed manifest, recovers a torn
journal tail under an exclusive advisory lock, and syncs the journal
before returning. Corruption before the tail is an error, not history.

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
@chris-beckman
chris-beckman marked this pull request as draft October 30, 2026 23:48
- Stage/Verified traversal through os.OpenRoot; symlinked prefix directories
  cannot redirect reads or publishes outside the store (fixture included).
- recoverJournal validates every complete record torn or not; Active rejects
  corrupt or invalid complete records instead of falling back to older state.
- Stage propagates directory open/sync/close errors; the journal's directory
  entry is synced after append so a first activation survives power loss.

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
…eadiness

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
The provider supervises the nativegen service over stdio using the same
framed protocol as the engine crate. Frame demux dispatches replies to
waiting callers; Complete/Stream stay unimplemented until the chat
template projection lands.

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Runs only when NATIVEGEN_BIN and NATIVEGEN_MODEL point at a compiled
service and a checkpoint, so ordinary CI stays hermetic.

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Request frames are flat bodies matching the service's serde shape; the
reader tags service events by arrival for demultiplexing. The hello and
load round trips run against the compiled binary through the provider's
own codec.

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
omitempty dropped an empty manifest that load supplies, so the Rust
service rejected every load with 'missing field manifest'.

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
@chris-beckman
chris-beckman force-pushed the feat/native-generation-provider branch from 0c43d03 to 965a51d Compare October 31, 2026 00:10
@chris-beckman
chris-beckman marked this pull request as ready for review October 31, 2026 00:10
@chris-beckman
chris-beckman marked this pull request as draft October 31, 2026 04:30
…ntary leases

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
…l-store' into feat/native-generation-provider

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
…elstore checkpoints

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
…ases

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
…eat/native-generation-provider

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
…ir route

Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
@chris-beckman

Copy link
Copy Markdown
Contributor Author

Delivered on main through #1987 (ad462d8629) from this exact source head 5d6192567533. The source branch is retained.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant