Repository navigation
Conversation
Add preliminary native v0.1.26 results, raw run JSON, scrubbed guest hardware facts, and the local API harness. Prompt and memory topology limitations are documented.
On the RTX 5070 / 64 GB PC the GPU computing a share of the misses over PCIe measured no faster than the CPU: 7.30 and 7.44 tok/s with 24 and 16 GiB of a 40 GiB budget registered, against 7.05-7.74 unpinned, and registering costs startup time and driver memory. The default budget mode is now locked-only; the routing-aware prefetch (multi-run A/B: 6.93 -> 7.91 tok/s mean, identical answers) stays on with the GGUF in place. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… the mapped mode without experts.bin, --resident-budget-gib, the tier counters - docs/UNSLOTH_Q4.md: what UD-Q4_K_XL is (Hugging Face revision 38bb39e, the four shards with sizes and SHA-256), disk/RAM/GPU needs (111.3 GB download, 1.4 GB pack, no experts.bin), the iq_pack.py command and why --compat-bf16, the numerical policy (native experts/embedding/head/projections/PLE; the converted small tensors listed in conversions.json), the MTP draft layer, a server config (--native shard 1, --ple-gguf found by name, --resident-budget-gib N with N = RAM minus 20-24 GB), measured speed on an RTX 5070 / 64 GB PC (7-8.5 tok/s at 40 GiB; ~60 s to the first answer), the switches (STRATA_LOOKAHEAD=0, STRATA_KQ256=1, STRATA_PARTIAL_PIN=1, STRATA_FETCH_THREADS), scope and validation, credits (eddoursul Niko1221#245, gopinath87607 Niko1221#255, jagsan-cyber Niko1221#247). Experimental, not in setup. - docs/DETAILS.md: the mapped low-RAM mode reads a native pack's experts from the GGUF when it has no experts.bin (checked, identical answers on the Coder); --resident-budget-gib and the routing-aware prefetch; where the tier counters are (--stats, the server log, /metrics' ram_blobs / file_blobs / file_mb); a pointer from "Which model?". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…plit GGUF, Q4_K/Q5_K/Q5_1/Q8_0 experts, RAM budget + SSD tier with routing prefetch), the low-RAM mode reads GGUF experts in place, Windows GGUF load 2x (Niko1221#230), server status race (Niko1221#266) and tokenizer (Niko1221#268) fixes, truncated tool calls (Niko1221#211/Niko1221#231), AMD intrinsics (Niko1221#262); setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ATA_HIP_WMMA=1, gfx12 only) The v2 tensor-core design with gfx12's v_wmma_f32_16x16x16_f16: one block per (query, KV head), wave w owns dims [64w, 64w+64) (int8 scale group w), q and p split into FP16 hi + lo parts, int8 codes exact as FP16, scales in FP32, the four groups' q.k partials added in a fixed order, online softmax over 32-cell chunks. Masked (non-resident) cells get no weight, as in the decode kernel. FP32-level accuracy, not bitwise equal to the default kernel, hence opt-in. hip_prompt_attn_wmma = qsa_prompt_attn_parity built for HIP (skips off gfx12). NVIDIA builds unchanged (all new code is under __HIPCC__ / the HIP test section). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ame (default on HIP; STRATA_HIP_ROUTER_OLD=1: old) On RDNA4 the router took 39 us per call in decode (16% of the decode GPU time): 512 dependent FP64 adds, ten selection passes with 20 block barriers, a renormalisation re-reading its weights from global memory. The sum's only use is inv = (float)(1.0 / sum): all terms are positive with the largest exactly 1, so the serial sum and a tree sum are within 2^-43 of each other; when the bracket tree*(1 -+ 2^-40) maps to one float, that float is the serial sum's (1/x and the cast are monotone), otherwise thread 0 runs the serial sum as before. The selection runs on wave 0 from registers (the argmax with the lowest index on ties is unique), the renormalisation from registers in rank order. hip_router_fast checks ids and weights bitwise against the portable kernel (65,536 rows with ties, near-ties, NaN rows, the forced serial path; and 256 experts / k 8). CUDA builds unchanged (HIP-only code). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… unterminated Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…n's logits STRATA_DUMP_FIRST_LOGITS=<file>: the logits of the first window's first position (the token the prompt predicts), n_vocab float32, written once per run. Two engine variants run on the same prompt then give a first-token KL divergence: how far a prompt-path change moves the model's distribution, not only whether its top token changed. On a layer split the last stage's verifier writes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
…t; STRATA_EMULATE_CC for tests - generate.cpp: before the expert arena loads, the engine names the card and its compute capability and checks that the build has code for it (device_code_error: cudaFuncGetAttributes on a kernel). A binary built for other GPUs used to fail at the first kernel, after the whole arena had loaded; now it stops in ~0.3 s and says why. - include/strata/core/emulate.hpp: STRATA_EMULATE_CC=75|80|86|89 makes the host's device queries (compute capability, shared memory per block) answer as that generation, for a PTX build of it JIT-compiled on a newer card - ggml's MMQ config, the QSA kernels' choice, fused_gr's shared memory. Tests only. - STRATA_QSA_WARP=1|select|attn: the pre-sm_80 QSA kernels on any card (A/B). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
…ery fails; HIP builds compile (no device_code_error probe) - generate.cpp: the startup line names "(an unnamed GPU)" when cudaGetDevice/cudaGetDeviceProperties fail (the error is cleared), and the "no code for" message uses the same name; a HIP build prints the gfx architecture instead of a CUDA compute capability. - device.cu: device_code_error() returns "" on HIP - cudaFuncAttributes/cudaFuncGetAttributes have no mapping in hip_compat, and gpu_arch_problem() already stops a HIP run on an architecture the build lacks. - fused_gr.cu: STRATA_EMULATE_CC also answers the two compute-capability queries there (small tile, the sm_70 capacity rule), as the PR's description promises; with the variable unset nothing changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… STRATA_REQUEST_LINES, a relative exe Found by running Claude Code against the server (claude -p, three tool turns) and by starting a config with a relative engine path: - POST /v1/messages/count_tokens answered 404; Claude Code asks it for its context figures. It now renders and tokenizes the request exactly as /v1/messages would, without running the model. - A /v1/messages request with no "thinking" (and no effort) got the template's default, which thinks: Claude Code's helper calls (a ~60-token session title) came back with a thinking block and no text. Anthropic's thinking is opt-in, so such a request now renders with enable_thinking false. A config's reasoning_effort still applies; "thinking" set, or an effort, behave as before. - STRATA_REQUEST_LINES=1: a thread follows the engine log and re-states each request's summary line on stdout, for supervisors that only see the server's stdout (a tray, llama-swap). Off by default. - A relative "exe" in the config failed in Windows' CreateProcess (WinError 2): it is made absolute against the config's cwd, as the engine's cwd already is. Tests: count_tokens equals the input_tokens /v1/messages reports for the same request; the request line pattern parses the engine's summary. python -m unittest serve.test_server: 44 OK. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
…; its time field is prompt_read, not ttft - StrataEngine.restart() runs __init__ again (after a crash, an idle unload's reload), which started one more STRATA_REQUEST_LINES follower each time and printed every request line once per restart: a follower is started once per log path (the log is opened for append, so the first follower keeps reading the same file). - The field the follower printed as "ttft" is the engine's prompt read time (the new tokens only, without the queue, the template, the first decode step): it is named prompt_read. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…no warm-up, a thread per core - --flash-attn on|off|auto (mtmd's flash_attn_type): flash attention keeps K and V in FP16. On the NVFP4 fork the vision tower's output was 11.2% off with it on the GPU; FP32 attention (CPU) 0.1%. - On the CPU a CUDA build keeps the GPU unseen (CUDA_VISIBLE_DEVICES=-1 before the runtime starts): it otherwise opened a context there, 0.4-0.7 GB of VRAM the expert cache loses. - On the CPU without --threads: one thread per core (mtmd's default is 4); no warm-up encode (~6 s of start). - serve/server.py: relative exe/mmproj/model paths in the vision config are the config's cwd's, as the engine's are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
…e Cyrillic script The shipped draft subsets hold 142 of the vocabulary's 18,580 Cyrillic tokens, so an answer in Ukrainian, Russian, Bulgarian or Serbian drafted almost nothing. data/draft_vocab_cyrillic.bin is draft_vocab_en.bin plus every token whose text is Cyrillic (58,963 ids), built with the repository's own tool: python tools/draft_vocab.py --gguf <IQ2_XS shard 1> --base data/draft_vocab_en.bin --add cyrillic - tools/draft_vocab.py: a "cyrillic" script (U+0400-052F, 1C80-1C8F, 2DE0-2DFF, A640-A69F). - setup.py: --draft-vocab cyrillic; docs/DETAILS.md. On the NVFP4 fork: 1.4 -> 2.1 tokens per round, 83 -> 109 tok/s in Ukrainian answers, English unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
…he model's earlier choice The full setup rewrites the model's config and refreshed the MTP folder's subset with `a.draft_vocab or "cjk"`, so running it again without the flag put the CJK subset back over a cyrillic (or en) choice, and dropped draft_vocab from the config. saved_draft_vocab() reads the choice the config holds; the flag still wins. tools/test_setup_draft_vocab.py. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
An existing experts.bin was kept whenever its size matched the layout. A different checkpoint with the same geometry (a re-quantization or a fine-tune packed into the same folder) has the same size, so the engine silently served the old experts with the new model's dense weights. Before keeping the file, the first and last blobs of the first, middle and last layers are now compared with the GGUF; a mismatch rewrites it. The check reads six blobs, so an unchanged pack is still reused at once. Test: test_experts_bin_is_kept_only_with_this_ggufs_blobs packs two GGUFs of the same geometry into one folder (it fails on the previous code). Rebased onto 0.1.31 (Niko1221): 0.1.31 already refuses an experts.bin whose sidecar (experts.bin.src.json: shard names and sizes, the hash of native_experts.txt) does not match, so the six-blob comparison runs only when the sidecar says "reuse" - it catches the case the sidecar cannot (a re-quantized or fine-tuned checkpoint under the same file names and sizes). It reads each of the six expert blobs on its own instead of concatenating the whole layer. The test packs the second model under the first one's file name, so it reaches the new check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
… hc-read second half stops overwriting the next layer's base stamp kProfPer 32 -> 33: fused_gr_read_multi writes three stamps per half and the second half's base was 30, so its up-stamp landed at slot 32 = (l+1)*32 + 0, the next layer's stage-0 base. Derived columns (gap/head/hc0) also wrapped to ~1.8e19 ns whenever a slot was unstamped, printing ~1e14 ms per window. A delta now counts only when both stamps are nonzero and monotonic; the hc0 split replaces the single hc-read0 column only when the split chain is complete. Metrics-only: no computed model value changes. Reason recorded in PORTING.md 19.
Isolated benchmark (no inference-path changes) comparing the production bf16_gemv warp kernel against an mma.sync.m16n8k8 tensor-core variant with BF16->FP16 conversion, on the real decode GEMV shapes (n_in = 2560). Three TC variants (W preconverted / inline conversion / fused output) and a full k-split occupancy sweep; parity against an fp64 reference. Result: the TC variant is 1.3-2.2x slower on every real shape even at its best config - the warp kernel is already bandwidth-bound (374 GB/s) and a single-token GEMV wastes 15/16 of the m16 MMA tile. Keeping the warp kernel; the benchmark is checked in so this does not have to be measured twice.
… coupled speculative sampling - Add set_coupled_draft() and coupled_draft_state() in include/strata/core/coupled_draft.hpp - Add --coupled-draft and --no-coupled-draft CLI options to src/program/generate.cpp - Default option to STRATA_SPEC_COUPLED environment variable for full backward compatibility - Update usage() help text with option descriptions
park_current dropped the retained K/V whenever can_fit failed and fell back to a full snapshot capture. The gate duplicates what make_room decides immediately after, with harsher arithmetic: the slot term fires when make_room enforces the same cap, and the byte term compares the with-reuse estimate against headroom that ignores the parked image replacing the retained buffers. make_room already implements the stated policy - pressure drops the optional retained storage before evicting any parked conversation (the priority the cache tests pin). With a full cache the fallback buys nothing: the fresh estimate forces the same or more eviction, plus a whole-snapshot recapture instead of the growth - hundreds of ms per park at ~100K tokens where one long conversation round-trips while short-lived conversations keep the cache full. The with-reuse estimate stays uncapped: it counts the retained buffers' capacity and directories, and put() charges that same true size, so a capped figure would under-evict and overfill the budget. Co-Authored-By: Claude Code <noreply@anthropic.com>
…03 "starting" (Niko1221#344) restart() set ended=False before __init__, which zeroes max_context and blocks until READY, so a request in that window skipped load() and failed with a 400 "leaves no room to answer in the context (0)". The engine is now marked alive by __init__ once READY; prepare() answers 503 "the engine is starting" when max_context <= 0. The output thread is bound to its own process and queue, so a killed engine's thread no longer marks its successor dead or writes into its queue. Tests: RestartWindow (a fake strata --serve with a held READY). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…udget_tokens still thinks "thinking" absent meant no thinking, also for a request that gives reasoning_budget_tokens (Niko1221#123), which asks for thinking with a cap (serve.test_server's test_anthropic_stream failed: no thinking at all). Such a request renders as before. test_anthropic_thinks_only_when_asked pins the five cases. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…d X-Accel-Buffering for SSE - Implement do_OPTIONS (204 No Content) and add Access-Control-Allow-* headers to allow web-based clients (Open WebUI, LibreChat, browser extensions) to query endpoints. - Update _own_page to recognize reverse proxies and tunnels (Cloudflare Tunnel, Tailscale Funnel, Nginx) via X-Forwarded-Host, CF-Ray, and forwarded headers, preventing 403 CSRF false-positives when serving remotely. - Add X-Accel-Buffering: no to SSE streaming responses to prevent reverse proxies from buffering real-time token generation. - Add unit tests in serve/test_server.py covering CORS preflight, CORS headers, and reverse proxy origin checks. Co-authored-by: Gemini <gemini@google.com>
… own-page bypass for proxy headers
- The Origin check of /settings and the MCP tools let any page through when the request carried X-Forwarded-For,
CF-Ray or CF-Connecting-IP (headers every reverse proxy or tunnel adds), and compared hosts without their ports
(a page on another local port passed): any web page could then change the shared settings or run MCP tools with the
user's rights. Removed. The Origin must be this server's host:port, or an origin in the config's new
"trusted_origins" (the web app behind a proxy or tunnel); a proxy header alone opens nothing.
- CORS is opt-in: "cors_origins" in the config (an origin, a list, or "*"). Only /v1/* answers with
Access-Control-Allow-Origin (the origin itself, or "*"), never /settings, /unload or the app's files; the
preflight echoes the headers asked for ("*" does not cover Authorization). OPTIONS is answered 204 either way.
- X-Accel-Buffering: no on SSE stays. origins_of() checks both lists at start (scheme://host[:port], no paths;
"*" only for cors_origins). docs/DETAILS.md; tests for the bypasses, the trusted origin, CORS off/on, the header.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… EngineStuck instead of a catch-all RuntimeError - StrataEngine.close() (now also the unload and the restart): QUIT and EOF, then terminate, then kill, each with 20 s as the unload had (Niko1221#333 went from QUIT straight to kill after 10 s; an engine unpinning tens of GB can need more). A process that outlives all three raises EngineStuck and stays owned. - do_POST caught every RuntimeError as a 503 - EngineDied, GpuBusy and ModelBusy are RuntimeErrors too, and any other failure was turned into a vague 503. Only EngineStuck is caught now (also around POST /unload, and the idle-unload thread survives it); EngineDied keeps its own answer. - test_lifecycle: terminate before kill, EngineStuck, an EngineDied on load answered as such. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
A web page of another site could point its own name at 127.0.0.1 and reach
the server as the same origin: without an API key it could read /status,
/metrics and /api/requests, change settings and run MCP tools. The server now
answers only Host names it knows: localhost (and *.localhost), any IP address
(rebinding needs a name), the bind address and, bound beyond this PC, the PC's
name, name.local, its LAN addresses and host.docker.internal; any port. No
Host header (HTTP/1.0) passes. "allowed_hosts" in the config and
STRATA_ALLOWED_HOSTS add names (".example.com" for a domain, "*" turns the
check off); trusted_origins' hosts count too. A refusal is a 403 naming the
setting and one log line.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
/unload and /load took a plain form POST from any site: they now take JSON from Strata's own page only, like /settings (_own_page). Without an API key, a POST to /v1/* that carries an Origin must come from Strata's own page, a page on localhost or an allowed host name, an origin in trusted_origins or cors_origins, or a non-web scheme (browser extensions, desktop apps), and then send JSON; others (and Origin: null) get 403, so a web page can no longer burn GPU time with a text/plain POST. Requests without an Origin (curl, the SDKs) are unchanged; with an API key, the key decides. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
It keeps 256 of the 512 experts, chosen on code data, so it is weaker outside code and in other languages; for general use, or when RAM allows, MODELS.md now points to the full models. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
A rebinding page cannot send the key, so with one configured the check only broke tunnels and proxies that pass their own name on (cloudflared does). Keyless setups keep it, with allowed_hosts; the 403 now also says that setting an API key turns the check off. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ing or writing it
… their own device
…uns the server's engine arguments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C72PYQJouk92CHXH5hbjL1
The split fixture records on which device each copy and drain happened (OnDevice), and the test defines __wrap_cudaGetDevice/__wrap_cudaSetDevice for that, but the link only wrapped cudaMemcpy, cudaDeviceSynchronize and cudaGetLastError. On a machine without a GPU the real cudaSetDevice fails, the device never changes, and the test failed on its own harness: 'stage 1's buffers were written on its own device after that device was drained'. With both wrapped it passes (1608 checks), and it still fails when the per-stage OnDevice scopes in conversation_state.cpp are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er its own id A request for the conversation that already holds the session fell back to a checkpoint (or token 0) without a word when its prompt no longer continued the live end; only a switch between conversations logged the divergence. Say where, with the ids around it, so a client that re-renders its own prefix shows up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C72PYQJouk92CHXH5hbjL1
--conversation-disk-dir replaced the RAM conversation cache, so a request without a conversation id was never parked: whenever another request took the session in between, it was read from 0 again. Behind an agent app that was 816 of 840 short background prompts in one night, ~3 s each. Now both run side by side: a conversation with an id parks on disk as before, a session without one parks in the RAM cache (--conversation-cache-mib), and a request is looked up only where it belongs. The parity tool gets a 'mixed' scenario: A with an id, B and C without, interleaved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C72PYQJouk92CHXH5hbjL1
…ted run A prompt read in one go and one continued from its live end differ in state (the kernels batch it differently), so a baseline that interleaves conversations without a cache reads every return from 0 and cannot be compared with a restore. The baseline now plays A and then B through on their own; the candidate interleaves them with C between. PASS on IQ3_S, RTX 5060 Ti + RTX 2000 Ada, layer split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
|
Heads-up: today we cleaned up this repository's commit history, and GitHub closed this PR automatically when GitHub cannot reopen it, because its branch is built on the old history. If you want to keep it going: start a new branch from the new Sorry for the noise, and thank you for the contribution. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #716 (parking with --layer-split); the first 12 commits are that PR.
What
Parking in RAM (#57) needs host RAM that a PC running this model has little of: the experts take it. A long agent
conversation is 1-2.5 GiB of snapshot, so with a few sub-agents and side calls the budget is gone, and every switch
back reads the whole prompt again.
--conversation-disk-dir DIRparks on disk instead.X-Strata-Conversation: <id>(or the body fieldstrata_conversation) namesthe conversation; the engine gets it as
conv=. The id only picks the files: the prompt must still continue whatwas parked, otherwise the request reads from the deepest checkpoint it shares.
checkpoints; pages the files already hold are kept.
--conversation-disk-gib N(default 32) bounds the directory;the oldest conversation goes first.
restores there, and never pushes a keyed conversation out. A session holding a keyed conversation is never parked
in RAM too, and a keyed request is looked up only on disk.
does not continue its live end: diverges at N of M (prompt P); live=...|... prompt=...|..., with the token idsaround it. That is how a client that re-renders its own history (a template, a tool-call parser, a tokenization
the model did not sample) can be found; without the line the request just read from a checkpoint or from 0.
tools/conversation_cache_parity.py:--scenario disk(A with an id, B, A again, a request without an id, arewind to a checkpoint) and
--scenario mixed(A with an id, B and C without, interleaved), each against a runwith no cache at all.
Measured
RTX 5060 Ti 16 GB + RTX 2000 Ada 16 GB (
layer_split 30), Ryzen 7 9700X, 64 GB RAM, Samsung NVMe.On our own branch (v0.1.38 merged with this code plus unrelated patches of ours; the parking code is the same as
here), IQ3_S,
--conversation-cache-mib 1024:--scenario disk: PASS (A/B/A, delta parks, a rewind to a checkpoint; tokens and main-model state byte-exact).~2K-token conversation: the first park writes 254.5 MiB in ~0.36 s, a later one 113-226 MiB while keeping 141-255
MiB already on disk; coming back reads its running state in 0.2-0.4 s and restores 28.5 MiB in 50-75 ms.
--scenario mixed: PASS. Only A (with an id) parked on disk; B, B+, C and B++ (without one) parked in RAM, and B+ andB++ came back from RAM, each turn byte-exact against its conversation played through without a cache.
Earlier, IQ2_XS with the 0.1.34-based version of this code:
plus 0.4-1.1 s of running state.
--scenario disk: tokens and main-model state byte-exact.it again took ~90 s; two sub-agents alternating per call switched in ~1.5 s each.
Developed with an AI coding assistant; all numbers measured on the machine above (Linux, CUDA 13).