Research: native cross-slot prefill/TG phase exclusion

Current status — deployed and accepted (2026-10-01)

The owner authorized implementation, matched host/RPC build and deployment into the current proxy backend binary paths, end-to-end tests and closure. PR #10 is merged as 98b3e092d34b99222d9144f90e388767f02ec117; both installed manifests and running native/library maps verify this source. Applicable maintained and live acceptance is complete.

The fork default now runs compatible PP concurrently, seals cohort membership when the first member completes PP, retains final logits until all members finish, and drains their TG before admitting later PP. Unequal multi-chunk Qwen/MTP and late-arrival live checks plus two proxy8082 streams passed.

Final deployment and acceptance evidence. The earlier pending investigation/design/delivery statements below are historical and superseded by this accepted endpoint. Separate Settings/Requests cancellation work belongs to novkien/llama-proxy#506.

Historical investigation and intake

Goal and requested behavior

Prepare a native llama-server mechanism that separates prompt processing (PP/prefill) and token generation (TG/decode) across slots sharing the same inference engine:

While any slot in the admitted prefill group is still prefilling, slots that finish their prompts early must wait before starting TG.
While requests are in the TG phase, newly arrived requests must wait before starting prefill, even if a physical slot is free.
Multiple slots may prefill together, and multiple slots may generate together. This is phase exclusion, not a single-request concurrency limit.

The goal is to avoid prefill/decode interference and make phase admission authoritative inside the native scheduler. This is a feature-preparation issue, not a claim of a reproduced correctness defect or measured performance gain.

Current source evidence

Inspected fork master revision: 95e6565970621aa508e80fd805dcf73af413093c (2026-09-30).

In server_context_impl::update_slots():

Existing SLOT_STATE_GENERATING slots are selected for generation at lines 2931-2945, and their sampled/drafted tokens enter the batch at lines 3045-3048.
Pending prompts can then join that batch under params_base.cont_batching || batch.size() == 0 at lines 3058-3083.
A completed prompt becomes SLOT_STATE_DONE_PROMPT at line 3511. The result path promotes that slot to SLOT_STATE_GENERATING at lines 3736-3755 without a group-wide prefill completion barrier.

These paths support mixed-phase scheduling; they do not express the requested bidirectional exclusion. --no-cont-batching is an existing baseline to evaluate, but disabling prompt insertion into a nonempty generation batch alone does not establish that every admitted prompt completes before any member begins TG.

Related proxy work: novkien/llama-proxy#186 gates new streaming admission after observing generated tokens, includes a periodic bypass, and excludes nonstream requests. That issue was closed by owner decision with live scheduling acceptance unverified. It is useful history, but proxy token observation cannot provide an atomic native prefill completion barrier. Do not inherit its 60-second bypass or streaming-only scope implicitly.

Proposed design work

Evaluate a small scheduler-level phase policy using existing slot/task state and batch construction, rather than a new external coordinator:

Define the admitted prefill group and its boundary. Once the group is sealed, later arrivals wait for a subsequent group so arrivals cannot indefinitely delay ready slots.
During PP, process group members until all remaining members finish or terminate. Retain completed members' KV, sampler and required logits/result state while they wait. Do not emit generated tokens or perform speculative generation before the barrier opens.
During TG, permit generation for the eligible group and hold pending prefill work. Finish or explicitly yield the generation phase according to the agreed fairness policy.
Preserve cancellation, disconnect cleanup, EOS/output limits, slot release, prefix/RAM cache reuse, context shifts, parent/child tasks, incompatible batching groups and speculative/MTP state.
Apply the policy to native generation tasks regardless of streaming transport. Define how embedding, rerank and multimodal preprocessing interact with the same engine; do not leave an unexamined route able to violate the exclusion.

A first token sampled from final-prefill logits is a boundary case: the design must state whether sampling itself is delayed and how the output/logits survive the wait. GPU work and token emission must match the chosen phase definition, not merely a displayed slot label.

Decisions to resolve before implementation

Configuration/API surface and default behavior. An opt-in server policy is a candidate to preserve existing continuous-batching users; it is not yet an approved default.
How to seal a group, assign late arrivals and handle slots with incompatible adapters/batching constraints.
Strict drain-to-completion versus bounded phase switching for long or unlimited generations. Any fairness mechanism must preserve phase exclusion; it must not insert prefill while TG continues.
Cancellation/deadline semantics and useful status/metrics for the active phase and blocked requests.

Validation and acceptance

A design records the phase invariant, group membership, transitions, configuration and fairness choices.
Two prompts of unequal length, both requiring multiple prefill chunks: the shorter prompt cannot start TG while the longer member is still in PP.
A request arriving during TG starts no prefill until the permitted phase transition; same-phase concurrency remains available.
Staggered arrivals cannot keep an existing prefill group open forever; queued work progresses under the selected fairness policy.
Cancellation/error of a waiting or active member releases the barrier correctly, with no deadlock, busy loop, leaked slot or invalid logits/KV state.
Streaming and nonstreaming generation, cached continuations, and supported speculative/MTP paths obey the same invariant. Additional task types are covered or explicitly constrained.
Compare the proposed policy with current continuous batching and --no-cont-batching on the same disposable workload. Record PP throughput, TG throughput/inter-token gaps, TTFT, queue wait and total makespan; report tradeoffs rather than assume a speedup.
Reuse maintained server tests and isolated fixtures. Do not run fault injection or workload tests against production services while preparing this issue.

Search, authority and next action

Searched all fork issues (2 returned) and all current/historical fork PRs (2 returned); neither owns this phase policy. Searched upstream issues/PRs for prefill/decode. ggml-org/llama.cpp#29175 reports mixed-workload decode stalls on other builds; it is related external evidence, not a reproduction on this fork. Disaggregated prefill/decode work is a separate architecture.

The owner directly requested creation of this preparation issue in Codex on 2026-09-30. This task creates the issue only; no implementation, build, deployment or service action is authorized by this record. Next: review the scheduler design and resolve the listed decisions before starting implementation.

Current authorized delivery (2026-10-01)

The owner now explicitly requests implementation of fork #5 and #6, paired backend build/deployment into the llama.cpp-fork path selected by the current proxy, end-to-end tests for both, and closure after acceptance. At model-use stages force unload and proxy model-request blocking are authorized. Earlier research-only/diagnosis-only scope paragraphs above are historical. Codex root coordinates separate isolated source implementers from published base e58b770; implementation/review/build/deployment/acceptance remain pending. The latest implementation claim comment owns the current handoff.

Investigation: live two-request baseline and real native harness

Scope/authority: Codex investigation for issue #5. The owner requested journal/source investigation and this comment, then clarified that the comparison must run C++ code using the llama.cpp engine before selecting an implementation plan. The owner also requested evaluating one prefill owner at a time if parallel prefill is inefficient. This authorizes the standalone research harness; production server implementation remains pending owner review. NO_SOURCE_MUTATION: no fork/proxy source, live configuration, service state, cache or resource ownership was changed.

Deployed context and measured PP throughput

Inspected the llama-bridge.service user journal on jarvis-llm, beginning with the two live requests at 2026-09-30 22:28:33 +07. Startup journal identifies build 11274, commit e58b770f3. The physical checkout is still 95e656597...; its relevant server-context source matches e58b770f3c2346deab036d9e4c3b2ea61ca0bbc5. Do not infer the deployed revision from that checkout or its stale generated build-info file.

The original Qwen-27B instance on 8083 used four V100s, four slots, -b 2048 -ub 1024, Q8_0 KV, unified KV, MTP draft maximum 2, and CPU multimodal projection.

Slot 3/task 1293: 34,576 prompt tokens, starts 22:28:33.
Slot 2/task 1338: 212,252 prompt tokens, starts 22:28:48.

Aggregate observed PP progress rate, using near-aligned samples on a common wall-clock interval:

Interval (+07)	Slot 3 processed tokens	Slot 2 processed tokens	Combined delta / wall time
22:29:36 to 22:39:23	14,336 -> 34,572	32,768 -> 169,957	157,425 / 586.91 s = about 268 tokens/s
22:38:55 to 22:39:23	33,213 -> 34,572	167,710 -> 169,957	3,606 / 28.04 s = about 129 tokens/s

Sample boundary skew is at most 0.5 s. These are observed processed-counter rates, not a pure GPU compute measurement. Do not add the independently reported per-slot rates: at 22:29:36, slot 3's displayed 964.32 tokens/s still used about 14.87 s of last-updated PP time, while almost 63 s had elapsed since its prompt started. Progress printing happens before the next batch, and async completion/timing updates can lag. Image encoding, MTP processing, checkpoints and growing attention length confound a simple parallel-versus-serial comparison.

Slot 3's final journal summary at 22:45:06 reports 650.434 s PP / 34,576 tokens = 53.16 tokens/s, then 342.685 s TG / 314 output tokens = 0.91 tokens/s. At 22:44:17 slot 2 was still reporting PP, while a read-only slot snapshot during that period showed slot 3 had generated 58 tokens. This confirms overlapping stages, not merely a possible source path. Once both slots reached TG, individual short-window rates rose to roughly 9-15 tokens/s. These observations show interference/waiting; they do not quantify the speedup of an untested policy.

What the scheduler actually does

At the deployed/source revision:

iterate() visits slots in vector order.
pre_decode() first inserts generating/drafted tokens, then fills remaining batch capacity with pending prompts. Each prompt can fill all remaining capacity; this is not equal-share parallel prefill. Slot 2 can monopolize the 2,048-token batch and slot 3 advances mainly at breaks/residual capacity.
post_decode() immediately promotes each completed prompt and samples/emits its first token, independently of other prompts.
metrics_post_decode() and server_slot_stats explain the processed counters and last-updated timing.

Increasing -b alone may improve PP in some workloads, but with this greedy loop it can also give the earlier slot a larger uninterrupted share. -ub limits the physical microbatch, so a larger logical batch does not itself enlarge the GPU microbatch. Four-slot TG with small MTP drafts is far below b=2048; logical batch capacity is not an established TG bottleneck. Test b and ub separately and include wait/TTFT, not just throughput.

Standalone C++ harness completed

Created and compiled native-harness.cpp against the fork's actual libllama, with headers exported from matching e58b770f3.... It builds real token batches with sequence IDs/positions and calls llama_decode plus synchronization; it is not an HTTP-only test or simulated inference timing. No production source patch was written.

Four experimentally selectable policies:

mixed: generated tokens first, then greedy slot-order PP; individually completed prompts start TG immediately.
barrier-greedy: greedy PP, then group-wide TG.
barrier-fair: share PP batch capacity across unfinished members, then group-wide TG.
barrier-serial: exactly one slot owns PP until completion; ready members wait, then generate together.

The exclusion policies copy and retain final-prefill logits across subsequent decode calls. They do not sample the first token until the group barrier opens. Batch traces and assertions check no mixed PP/TG work, no early first token, and one PP owner for serial mode. Reported metrics include per-slot PP completion, TTFT/barrier wait, synchronized batch time, makespan and output hashes. Workload construction/model loading are outside makespan.

Checks: warning-clean C++ compilation, dry-run scheduling checks, and 72 real-engine CPU smoke runs. A 1.19 MB public ggml-org/models Stories260K fixture used one CPU thread/F16 KV, prompt lengths 16,96 and reversed, (b,ub)=(32,16),(64,16),(64,32), eight output tokens/slot and three shuffled repetitions. All 54 exclusion runs passed; output hashes matched across all policies/configurations/repetitions for each prompt order. Tiny CPU timings validate operation and logits preservation; they do not rank production V100 performance. The fixture's head width requires F16 rather than Q8_0.

The harness and complete CPU traces are available on jarvis-llm under /home/jarvis/native-prefill-research-20260930; durable Codex artifacts are under /home/jarvis/.codex/artifacts/prefill-research-20260930 on jarvis. The directory includes C++ source, a shuffled native matrix runner, a journal/SSE observer, README, live baseline and CPU summaries. To reproduce the CPU matrix from the remote directory:

python3 native-matrix.py --binary ./native-harness --model ./stories260K.gguf \
  --backend-path /home/jarvis/llama.cpp-fork/build/bin --output-dir ./cpu-repeat \
  --gpu-layers 0 --threads 1 --cache f16 --lengths 16,96 --predict 8 \
  --config 32,16 --config 64,16 --config 64,32 --repeats 3

Fixture: https://huggingface.co/ggml-org/models/resolve/main/tinyllamas/stories260K.gguf; SHA-256 270cba1bd5109f42d03350f60406024560464db173c0e387d91f0426d3bd256d.

Remaining evidence before choosing an implementation plan

The native runner currently isolates target-engine batching: no MTP/draft context, multimodal chunks, server checkpoint/cache admission, cancellation or late-arrival groups. It synchronizes each batch, unlike the server's async pipeline. Extend/validate these relevant paths after the initial target-only comparison. A policy winner on the tiny fixture or a target-only workload is not the production optimum.

Next GPU comparison: same target model/four GPUs/tensor split/KV/thread/context settings; unequal multi-chunk prompts and reversed slot order; repeat/shuffle all four policies at (b,ub)=(2048,1024),(4096,1024),(2048,2048) if allocation permits. Then include representative MTP/image cases. Compare PP/TG work, TTFT, ready-slot wait, makespan and memory failures before selecting parallel versus serial PP.

Current resource dependency: the original 8083 model was independently replaced by another active llama-server on 8086. Latest read-only inspection showed that process and GPU allocations still present. No target GPU matrix was run, and this task did not cancel workloads, unload a model or restart a service to make room. The concrete next step is to establish an available target-GPU window, run the prepared matrix, then resolve policy/admission/fairness choices and present an evidence-backed implementation plan. Production implementation approval is still pending.

Owner scope update: compare only concurrent versus serial PP

The owner clarified directly in Codex on 2026-10-01 that the decision is only whether PP A and B together or PP A followed by PP B minimizes total elapsed time. Configuration exploration is outside this comparison.

The active standalone native runner now compares exactly two policies: barrier-fair (both unfinished prompts contribute to each PP batch) and barrier-serial (one PP owner until completion). Model/hardware/prompts/batch/microbatch/threads/KV and the subsequent TG barrier are held fixed. The primary metric is elapsed time from group PP start until both prompts finish; full-request makespan with identical TG work is supplemental. Repeated runs are shuffled, and the prompt order is reversed to expose order effects. No tuning sweep or four-policy contest remains in the active runner.

Checked the narrowed runner with 12 additional real-engine CPU smoke runs: exactly two policies, one fixed configuration, both prompt orders, three repeats each; phase invariants and total-PP reporting passed. Tiny CPU timings do not establish the production GPU winner. The initial 72-run experiment remains historical and is reproducible with the archived native-matrix-initial.py; the active native-matrix.py uses fixed --batch and --ubatch arguments.

Latest read-only inspection shows Qwen-27B back on 8083, occupying all four V100s with active utilization. The prepared target-model comparison remains pending an available GPU window; no workload was stopped and no server/configuration was modified. Production implementation is still pending owner review. Next: run the fixed-setting two-policy comparison on the target resource, then select the faster total-PP path from that evidence.

Research checkpoint: fixed-setting GPU comparison started

The owner directly reported the GPU window available in Codex and explicitly authorized an agent Keep awake lease for this benchmark. Read-only checks confirmed zero active native/proxy requests; the previously warm child subsequently exited and all four GPU allocations returned to zero. A normal targeted proxy unload attempt returned model is not loaded; no hard restart, request cancellation or state override was used.

The prepared C++ runner is now executing real inference on Dirk Qwen3.8-27B UD-Q5_K_XL and four V100s. Only the PP scheduling policy changes: barrier-fair versus barrier-serial. Fixed settings: tensor split 31/31/30/8, the same UUID-to-CUDA order and selective P2P pairs as the original child, b=2048, ub=1024, Q8_0 KV, 20 threads, context 400000, four-sequence capacity. Synthetic prompt lengths are 8192 and 32768 tokens, reversed as well, with three shuffled repetitions per policy/order (12 runs). Eight generated tokens per slot provide a small, identical post-barrier sanity check; selection uses total group PP completion time.

This first comparison is target-engine-only, as already documented: MTP/image/checkpoint paths are excluded. No production source/configuration was changed. The authorized lease is renewed only while benchmarking and released in supervisor cleanup. Next: inspect all batch/slot evidence, compare repeated total-PP timings, record the result and its applicability limits before proposing production changes.

Research checkpoint: target PP results, MTP validation continues

All 12 fixed-setting target-engine GPU runs completed. For the same total 40,960 prompt tokens (8192 + 32768), pooled median total PP time was 42.257 s serial and 42.583 s concurrent. Serial saved 0.326 s, approximately 0.77% of the concurrent time. This is a small difference; the experiment does not show a substantial PP speedup from batching the two prompts together.

Per-order medians, three repetitions each:

Prompt lengths by slot	Concurrent PP	Serial PP
8192, 32768	42.451 s	42.326 s
32768, 8192	42.629 s	42.192 s

Both modes completed exactly the same prompt token work and maintained the group barrier. Model loading/tokenization were outside this wall-clock measurement. No configuration sweep was performed.

A supplemental harness path now runs the fork's actual MTP prompt hook: target nextn embeddings plus common_speculative_process and the MTP context, with the same main/draft KV settings and output limits as the server. The initial supplemental attempt failed with CUDA OOM because the harness incorrectly requested vocabulary logits at every prompt position and used the unrestricted default output reservation. The inspected server actually requests completion logits only at the final prompt position; MTP nextn embeddings are independently unmasked. Its target output limits are 12 total/3 per sequence, draft 4 total/1 per sequence, and recurrent snapshot counts 2/0. The scratch harness was corrected to those existing native semantics; no fork source was changed. This was an experimental harness error, not a demonstrated production failure.

The failed-run supervisor released its authorized lease and the existing idle timer powered the host off. The normal managed load API subsequently woke the host and loaded Qwen-27B successfully; a safe, non-force managed unload saved/released that idle model before resuming the corrected MTP experiment. The owner-authorized agent lease is active only during the continuing benchmark and will be released/read back at completion. No hard restart, alarm reset or persistent setting change occurred.

Next: complete the same two-policy comparison with the MTP PP hook and report whether the nearly equal target-only result persists. Eight target-only output tokens and one MTP-path output token are merely bounded sanity checks after the common barrier; the selection metric remains completion of both prefills. Multimodal input and checkpoint/cache replay are not part of these synthetic text fixtures.

Research result: GPU PP comparison and V100-focused engine candidates

Owner scope: compare concurrent versus serial PP with fixed settings, then research repositories specifically targeting V100/SM70. No production implementation is approved.

Completed real GPU comparison: all 12 target-engine runs finished, with 8192 + 32768 synthetic prompt tokens and reversed slot order, three repetitions per policy/order. Hardware/settings were fixed as recorded in the preceding checkpoint. Aggregate median total PP completion:

Concurrent (barrier-fair): 42.5830 s.
Serial (barrier-serial): 42.25655 s.
Serial advantage: 0.32645 s / about 0.77%; effectively near parity for this measured workload, not a universal conclusion.

Per-order medians: short-first concurrent 42.4511 s versus serial 42.3258 s; long-first concurrent 42.6292 s versus serial 42.1921 s. All phase invariants passed. Detailed individual/summary evidence is in the durable Codex artifacts gpu-two-policies-runs.json and gpu-two-policies-summary.json, with complete native traces on jarvis-llm under native-prefill-research-20260930/gpu-two-policies.

MTP limit: an additional standalone MTP-prefill runner compiled but aborted on its first target decode with CUDA error: out of memory, device 3, VMM allocation. No valid MTP comparison exists. This is an experimental harness/resource failure, not a demonstrated production defect. No fork/proxy source patch was made. Model loading is outside the timed PP interval; the target-only run excludes image/checkpoint/MTP costs.

Cleanup: the owner-authorized agent lease was verified released after the failed experiment; no benchmark process remains. Automatic idle policy powered the host off afterward. Managed Qwen-27B load job bd89f862d5c741aba47f74160941c18c subsequently succeeded; Keep awake readback is inactive. No alarm reset or Linux service hard restart was used. The owner then steered work to online repository research.

Primary-source repository findings (verified 2026-10-01):

mistrjirka/llama.cpp, v100-optimized: closest GGUF/source-compatible candidate. SM70 attention/quantization and long-context PP optimizations. Its matched upstream benchmark reports Qwen3.8-27B PP 298.47 -> 440.08 tokens/s on a V100 for a restored 100k prefix plus 1k new tokens. This is cached-suffix PP, not cold 101k PP and not a result on our four-card host.
1CatAI/1Cat-vLLM: SM70-first serving fork with dedicated FlashAttention-V100, quantized linear paths, tensor parallelism and MTP/DFlash. The README includes four-V100-16GB Qwen3.8 results with model/context/precision and quality qualifications.
dnv2003/v100-skinny: hand-written W4A16/W8A16 kernels built around Volta FP16 tensor instructions, running the published mixed NVFP4/FP8 Qwen3.8 representation on four V100-SXM2-16GB. Its 219.1 tokens/s AIME decode figure uses a particular NVFP4 artifact and MTP depth 7; it is not a same-weight speedup over our GGUF workload.
geoffwatts/ninfer-v100: narrow C++/CUDA Qwen engine, single GPU and custom .ninfer artifacts. Main benchmark hardware is V100-PCIe-32GB; not a direct four-16GB deployment replacement.
poisonxa16/pxa: Pascal/Volta engine using PXQ/PXQN formats with V100 benchmarks. Format/model changes prevent direct comparison with our existing GGUF.
humanjesse/vllm-v100: 1Cat-derived fork adding GGUF/compressed-tensors/MoE SM70 paths; its last pushed source is older than current 1Cat, so exact Qwen/version support needs confirmation before a trial.

These are repository claims and source candidates, not host-verified gains. Next research priority: inspect mistrjirka's relevant PP kernels for architecture/source fit, or prepare an isolated 1Cat comparison if the owner chooses an engine trial. Issue #5 remains open; no final native implementation plan or production repair is claimed.

Owner decision and research handoff: concurrent PP, then group TG

Decision/authority: The owner directly selected this flow in Codex on 2026-10-01: PP A and B run concurrently; a member finishing early waits; TG begins only after both members finish PP. The owner then explicitly requested no further testing. This supersedes the earlier fallback discussion/recommendation of serial PP. Stop the research tests here. No production runtime patch, publication or deployment was performed in this task.

Completed evidence

The real native-engine comparison used the same Dirk Qwen3.8-27B model, four V100s, fixed b=2048/ub=1024, thread count, tensor split, Q8_0 KV and context capacity. Only PP scheduling changed. Each successful condition had six runs, including reversed slot/prompt order, for the same 8192 + 32768 = 40960 synthetic prompt tokens.

Native PP path	Concurrent median total PP	Serial median total PP
Target engine	42.583 s	42.257 s
Target plus actual MTP PP hook	45.718 s	45.463 s

Both policies are close on this workload: serial's advantage was approximately 0.77% and 0.56%, respectively. All 24 successful GPU runs passed equal-token-work and phase-barrier checks; output hashes matched across policies for each prompt order. Loading/tokenization are excluded from these wall-clock PP intervals. These synthetic text fixtures do not establish performance for images, cache/checkpoint replay or other prompt lengths.

After the shorter prompt completed and waited at the barrier, MTP traces show the longer member's measured per-batch throughput increasing from approximately 511-513 tokens/s while sharing PP to 823-827 tokens/s while alone. Thus the barrier harness did not reproduce a persistent half-speed allocation. This is distinct from the current continuous-batching server, where the finished member may enter TG. The proposed isolated native-server overlap/release reproduction was prepared but not executed; the owner's stop instruction ends that investigation without claiming a production reproduction or confirmed defect.

Reviewed implementation direction for the next authorized step

Use concurrent same-phase prefill for the admitted group. Keep completed members' KV/sampler/final logits and MTP per-sequence carry state valid while they wait. Do not sample/emit the first generated token or initiate speculative TG before the group completion barrier. Once every remaining member completes PP, transition the group to TG. Cancellation/error must remove that member from the completion condition. Preserve the existing issue's requirement that later PP work does not join an active TG phase; group boundaries and fairness still need concrete treatment in the implementation design.

The owner selected the behavior; an implementing session must read this decision, the current issue discussion and existing source before source work. The earlier owner boundary was research/comment first and production implementation for later approval. Do not infer authorization to deploy from this research handoff alone.

Cleanup and artifacts

All native comparison processes exited. The supplementary reproduction script was not started. Keep awake was released and read back inactive; the expired private receipt from subsequent testcase preparation was removed only after confirming no active hold. No owner GUI hold, alarm, persistent idle setting or published source was changed. The physical fork checkout remains clean at 95e656597..., and the tested native libraries/headers correspond to build 11274 / e58b770f3....

Artifacts on jarvis: /home/jarvis/.codex/artifacts/prefill-research-20260930, including both GPU summary/run JSON sets, both compressed raw-evidence archives, C++ runners and README. Full remote evidence remains under /home/jarvis/native-prefill-research-20260930; corrected MTP runs are in gpu-mtp-two-policies-fixed, and the initial failed scratch MTP attempt is kept separate.

The final read-only cleanup snapshot reported nvidia-smi device-handle errors for physical GPU1 and GPU3, while no benchmark process remained. This snapshot does not establish the cause or a production repair; no further diagnosis/test or alarm reset was performed after the owner requested a stop. Do not claim all GPUs healthy/free from this handoff.

Status/next: Research and requested comparison are complete; issue remains open for implementation design/owner approval and eventual delivery. No additional test is scheduled or running.

Implementation claim and owner authorization

The owner explicitly authorized issue #5 and #6 implementation, end-to-end deployment into the actual llama.cpp-fork binary stream currently selected by the proxy, model testing for both issues, and closing only after acceptance in Codex on 2026-10-01. At the model-use stage the owner explicitly authorizes force unload and blocking proxy model requests. This supersedes the earlier research-only/no-more-research boundary; it does not authorize unrelated source, alarm resets or a hard proxy restart.

Accountable coordinator: Codex root task. Source implementer for #5: /root/issue5, isolated worktree /home/jarvis/.codex/worktrees/llama-fork-issue5, branch codex/issue5-phase-barrier, clean base e58b770f3c2346deab036d9e4c3b2ea61ca0bbc5. Source implementer for independent #6 is in a separate worktree. Current PR list has no competing open fork PR.

Implement the selected concurrent PP cohort and completion barrier, then grouped TG with later PP held; preserve stream/nonstream, cancellation, cache, parent/child and speculative/MTP state. The owner-selected behavior must be active in the actual deployed fork backend. Next: review the bounded native scheduler candidate, run focused disposable checks, publish linked reviewed source, build hash-matched native/RPC packages, activate the current backend path and verify real behavior through the current proxy before closure. No acceptance is claimed yet.

Work report — integrated candidate and native build

Owner and scope: Codex root task, issues #5/#6, authorized binary delivery. Candidate is based on published e58b770f3c2346deab036d9e4c3b2ea61ca0bbc5, isolated branch codex/issues-5-6-delivery.

Done: The native scheduler now admits compatible prompts concurrently while PP is open, seals membership when the first member finishes PP, retains owned final logits, and starts first-token sampling/speculative TG only when every remaining member finishes PP. TG drains that cohort before later requests can prefill. This is the fork default for streaming and nonstreaming, with no timed bypass. Compatible PP members share logical batch capacity; incompatible task/adaptor groups still obey the phase barrier. Cancellation releases members and orphaned completion children. Five focused cases extend the maintained completion test file, including staggered early PP admission and successful parent/child generation. Documentation records queue/TTFT tradeoffs and unlimited-generation consequences.

A combined candidate builds llama-server, test-batch-alloc, test-rope and test-sampling on jarvis-llm with GCC 13 and the existing architecture-specific GGML dependency package. An initial apparent successful build produced empty outputs and was discarded; a clean rebuild produced valid ELF files. Native compile errors in the new sampler local declaration and variadic log calls were corrected in the integrated candidate. No installed binary has been replaced. Focused CPU server/regression tests are now running; independent review is in progress.

Runtime boundary: The explicitly authorized model gate is blocked. Force unload signalled cancellation to 30 pre-existing nonstream requests, but the proxy still retains their admissions despite no native model process and empty GPU memory. The retained force operation is waiting for drain. We are investigating supported reconciliation without clearing queues, alarms or restarting the proxy without recovery authority.

Remaining/next: Focused tests, paired reviewed build/PR/merge, exact installed-bin backup and deployment to both host/RPC guest, real model and proxy end-to-end acceptance. Issue stays open until those checks pass. Root owns delivery and acceptance.

Work report — reviewed combined candidate ready for publication

Owner/scope: Codex root, native fork #5/#6. Integration branch codex/issues-5-6-delivery, base e58b770f3c2346deab036d9e4c3b2ea61ca0bbc5. One linked PR will carry the related native binary changes; both issues remain open through deployment/acceptance.

Done/evidence: Independent source review approved the final integration after correcting a discovered incompatible-LoRA PP deadlock: DONE_PROMPT waiters no longer seed the next batch group. Native GCC13 build of server and focused test binaries passed. CPU-only allocator passed53 tests/345 assertions; RoPE temporal-only text equivalence and sampler tests passed. All five new completion phase cases passed, four existing n_probs variants passed, speculative with/without-draft parity passed, existing per-request LoRA passed, and the new incompatible-LoRA barrier case passed after sizing its fixture within the model's256-token training context. Earlier fixture failures (oversized prompts and mistaking tokens:[0] in progress events for TG) were corrected in maintained tests, not masked by the runner. git diff --check and Python syntax checks pass. Tests use disposable state/local fixture files/loopback ephemeral ports with CUDA hidden. Cached continuation and stream/nonstream equivalence checks are additionally running.

Runtime/recovery: Owner approved exceptional proxy Cold restart after force-unload cancellation stalled. The restart cleared accepted/queued stale admissions;8082 reports zero active requests and model gate blocked. This is recovery, not deployment acceptance. No alarm reset or host power cycle occurred. Native installed binaries remain unchanged.

Remaining: Publish/review/merge exact candidate; build and install matching host/RPC packages at the existing proxy-selected binary path, verify manifests/process maps, then real model/proxy acceptance (unequal PP/TG barrier, legacy Nex image/cache/text). RPC VM startup currently fails with kernel AppArmor DENIED exec on the existing root-owned PCI-fix helper; a bounded local allow-rule decision is pending. Proxy Settings cancellation is separately authorized in novkien/llama-proxy#506 and builds on merged ggml-org#408/PR505. Neither a passing CPU suite nor merge closes #5/#6.

Next: Delivery delegate publishes this reviewed content unchanged; root retains paired build/deployment and all remaining acceptance.

Work report — focused acceptance complete; combined PR prepared

Owner and scope: Codex root task, fork issues #5/#6; publication delegate for the unchanged reviewed integration branch codex/issues-5-6-delivery at base e58b770f3c2346deab036d9e4c3b2ea61ca0bbc5.

Why: Record the completed CPU/native checks before publishing the paired implementation. Issue #5 admits compatible prompts into an open concurrent PP cohort, seals that cohort when a member first finishes, retains its final logits, and starts TG only after every remaining member finishes or terminates; later arrivals wait through cohort TG. This keeps the phase boundary authoritative while bounding group admission. The tradeoff is that a short prompt can wait for a long prompt, and an unlimited generation can hold later PP until the cohort drains.

Done and evidence: The reviewed 14-file candidate builds llama-server and focused allocator, RoPE, and sampling binaries with GCC 13. Focused CPU checks passed: allocator (53 cases/345 assertions), RoPE temporal-only text parity, sampler, five phase/barrier cases, four n_probs variants, speculative parity with and without a draft, per-request and incompatible-group LoRA cases, and cached continuation checks (four borrowed-snapshot cases, stream/non-stream equivalence, and cache/no-cache control). The cache/no-cache case was unskipped only in its out-of-source test wrapper; no repository skip or source change was made for it. Python syntax and git diff --check passed. All checks used disposable local fixtures with CUDA hidden; they do not establish paired V100/RPC behavior.

Remaining: The combined PR and merge, matched host/RPC binary installation, process/manifest verification, and live unequal-prompt PP/TG acceptance remain pending. Issue #5 stays open until deployed behavior is verified. The paired #6 image/cache/text model acceptance also remains pending.

Next: Publish the reviewed head and hand off the exact PR revision to the root coordinator for paired build, deployment, and native/proxy acceptance.

Work report — combined PR opened; type-check correction pending

Owner and scope: Codex root task, issue #5; combined PR #10 carries the reviewed 2664ff5f1b1a83386a7678dcba8b1e8d768845a7 candidate. See the exact-head pre-merge handoff.

Done and evidence: The PR links this issue and #6. The focused native CPU/server checks and independent review passed for this head. GitHub type-check found two new test calls passing an assertion message as positional strict; the precise files/lines and result are recorded on the PR. Other workflow jobs were still running at handoff.

Remaining: Correct and re-review those assertions, let required checks pass, then merge and complete paired host/RPC installation plus live PP/TG and Nex acceptance. The issue remains open until deployed behavior is verified.

Next: Root coordinator owns the exact-head CI repair and remaining delivery/acceptance.

Work report — type-check correction pushed to combined PR

PR #10 is updated at exact head 6dbd31a697ea6c402dd47f476a92e912103f4712; see the head-specific report. The two test failure paths now raise AssertionError directly instead of passing a message as pytest.fail's second positional argument. The CI-version ty 0.0.84 check on the affected files, latest ty, and git diff --check passed. GitHub checks for the new head are pending/running. Merge, paired binary deployment, and live phase acceptance remain pending; this issue stays open.

Work report — exact-head handoff ready to merge

PR #10 is reviewed at 6dbd31a697ea6c402dd47f476a92e912103f4712; the pre-merge handoff records scope, tests, checks, and remaining acceptance. Focused native/CPU tests passed, and current-head GitHub type-check is green. Broad Linux/Windows CI remains optional and in progress; no branch protection or repository ruleset gate is configured. Root owns merge and then paired binary deployment plus live PP/TG acceptance. This issue stays open until deployed behavior passes.

server : isolate PP phases and infer DFlash M-RoPE (#5, #6)

Work report — scheduler candidate merged; live acceptance pending

PR #10 merged at reviewed head 6dbd31a697ea6c402dd47f476a92e912103f4712 as commit 98b3e092d34b99222d9144f90e388767f02ec117; see the merge report. Focused native/server tests and exact-head type-check passed. Optional CPU Linux and Server CI jobs remained in progress/queued at merge. No installed binary was changed here. Root owns paired host/RPC deployment, manifest/process verification, and live phase acceptance. This issue remains open until deployed PP/TG behavior passes.

Work report — immutable paired build and acceptance window

Owner/scope: Codex root, fork #5/#6, merged PR #10 at 98b3e092d34b99222d9144f90e388767f02ec117. Both issues stay open for binary deployment and real model/proxy acceptance.

Done/evidence: The host's new isolated release build includes server, RPC server and existing runtime tools. Its complete34-entry manifest validates, reports 98b3e092…:e3b0c442… (clean source), and the staged server reports build11275/commit98b3e092/GCC13. Only unchanged architecture-specific GGML libraries are reused from the verified e58 dependency package; all affected llama/common/server libraries are rebuilt. Original installed binaries remain e58.

The RPC VM was off. Owner separately approved (1) a local AppArmor read/execute allow for the existing root-owned PCI helper and (2) repairing the QEMU hook after its callback into virsh deadlocked libvirt. The reviewed replacement validates the guest XML against sysfs GPU/audio identities without re-entering libvirt; valid PCI passes, mismatched PCI fails. The helper remains unchanged. Exact old hook/profile backups remain local. The VM starts successfully and the guest-only build is running against exact merged98b3e092 with its own GGML package.

Shared GPU tasks caused host/control reboots and interrupted agents/build preparation. Root retained the published/tested source, resumed the guest stage and obtained the owner's explicit decision to prioritize fork #5/#6 deployment/tests before releasing the GPU. The normal proxy force-unload of the now-renamed Qwen-Translator returned ok:true,status:unloaded,force:true; the model gate is blocked, no native child remains, and all four V100s are empty. This is the current acceptance window, not permission to change other model configurations or reset alarms/power-cycle the host.

Remaining/next: Verify/stage guest manifest matching98b3e092, snapshot exact installed bins on both nodes, install the full pair at the actual /home/jarvis/llama.cpp-fork/build/bin paths, then confirm loaded executable/library maps and affected real model behavior through8082. Functional CPU phase/LoRA/sampler/speculation/cache checks already passed; no GPU speedup is asserted from those tests. Separate Settings/Requests cancellation work continues in proxy ggml-org#506. Root owns remaining delivery/acceptance and closure.

Work report — matched native pair installed; real model acceptance started

Owner/scope: Codex root, authorized delivery of #5/#6 from merged PR #10, exact revision 98b3e092d34b99222d9144f90e388767f02ec117.

Installed evidence: Both actual /home/jarvis/llama.cpp-fork/build/bin directories now validate the same clean source identity 98b3e092…:e3b0c442…. Host manifest has 34 entries; RPC guest manifest has 28. Each architecture retains its own verified GGML dependencies. A guarded same-filesystem atomic directory exchange preserved the exact previously installed e58 binaries in build/bin.before-issues5-6-98b3e092-20261001 on each node. No physical source checkout or model artifact was overwritten.

The normal controller woke the host after idle shutdown. An intermediate manual-load job correctly failed on the deliberately temporary e58/98 pair mismatch; no model inference was accepted from that mismatched pair. After both installs and manifest verification, a fresh Qwen-27B load job 76caed3df38b44c59eddda388f228ca5 succeeded. Native /props reports b11275-98b3e092d. Its running PID5749 uses the actual installed executable and all nine mapped project libraries hash-match the host manifest; RPC PID3846 uses the installed guest executable on50052. The requested model gate is blocked.

Remaining/next: Real unequal multi-chunk PP cohort/TG barrier and late-arrival checks with MTP, two synchronized proxy8082 requests, followed by legacy Nex+DFlash cold-image/media/cache/text acceptance and unchanged original-weight hashes. Functional native CPU tests already passed; no new GPU performance claim is made. These issues remain open until their applicable live behavior is verified. Root owns acceptance and closure; Settings/Requests cancellation continues separately in proxy ggml-org#506.

Acceptance and closure — deployed #5/#6

Owner/scope: Codex root. The owner explicitly authorized implementation, paired build/deployment into the backend path used by the current proxy, end-to-end tests and closure. Related PR #10 is merged as 98b3e092d34b99222d9144f90e388767f02ec117.

Deployment: Both actual host and RPC-guest build/bin manifests validate the same clean98b3 source identity, with34 and28 entries respectively. Exact prior e58 installed directories are backed up on each node. Normal proxy loads succeeded for Qwen-27B and Nex. Running executable and all mapped project libraries match the installed manifests; /props reports b11275-98b3e092d. Original model/route settings were preserved. No proxy source activation is claimed by this native binary deployment; separate cancellation feature ggml-org#506 still needs its own delivery.

#5 live acceptance: Unequal Qwen prompts processed8,339 and4,247 tokens in9 and7 distinct PP progress positions. Neither emitted a real TG token before both PP totals completed. Each then generated64 tokens, with MTP draft counts50/49. A live native slot snapshot observed A/B in TG (n_decoded=1) while admitted C was processing with zero PP tokens; C progressed and completed after the cohort drained. Two synchronized OpenAI streams through8082 both returned HTTP200, generated reasoning deltas and [DONE]. The initial observer's prompt-field assumption was corrected to explicit native slot IDs, and the proxy fixture counts reasoning as well as content. Maintained isolated tests separately cover cancellation, parent/child, incompatible LoRA, cache, sampler, stream/nonstream and speculative behavior. Earlier real-engine comparisons informed the owner's choice; no new performance tuning or speedup claim is included.

#6 live acceptance: The original Nex target, mmproj and legacy DFlash draft now process the deterministic32x32 cold image with HTTP200,20 generated tokens and positive DFlash activity (draft_n=64) instead of the previous position-validator500. The binary logs temporal-only M-RoPE compatibility inference before draft-context initialization. A/B/appendA restores1,036 cached tokens from the1,040-token image prefix and processes7 new tokens; cache-disabled reference reports1,043 processed/zero cached. Text A/B/appendA controls match the cold reference. The actual inline PNG vision request through8082 succeeds and emits generated output. SHA256 of all three original files matches the pre-test record; no GGUF/tensor derivative or configuration replacement was required.

Retained output observation: The first strict harness reported one extra failure: warm media tokens differed from the cold reference. Bounded controls repeated both paths: cold repeats equal each other, warm repeats equal each other, with one different generated token at index11. No numerical or cache-state root cause is isolated. This is retained as evidence against the owner's existing Nex RAM acceptance decision, which explicitly removes exact Nex warm/cold token equality as a closure criterion. It does not negate #6's cold-position/media/cache/text/proxy acceptance, and no equality or new numerical fix is claimed.

Disposition: Applicable acceptance for #5 and #6 is complete on the installed matched pair. Close both as completed under the owner's requested endpoint. Unlimited or long TG still delays later PP as documented by the selected drain-to-completion policy; Nex warm/cold variation retains its prior accepted boundary. Root continues the separately authorized Settings/Requests cancellation delivery in proxy ggml-org#506.

Research: native cross-slot prefill/TG phase exclusion #5

Description

Current status — deployed and accepted (2026-10-01)

Historical investigation and intake

Goal and requested behavior

Current source evidence

Proposed design work

Decisions to resolve before implementation

Validation and acceptance

Search, authority and next action

Current authorized delivery (2026-10-01)

Activity

novkien commented on Sep 30, 2026

Investigation: live two-request baseline and real native harness

Deployed context and measured PP throughput

What the scheduler actually does

Standalone C++ harness completed

Remaining evidence before choosing an implementation plan

novkien commented on Oct 1, 2026

Owner scope update: compare only concurrent versus serial PP

novkien commented on Oct 1, 2026

Research checkpoint: fixed-setting GPU comparison started

novkien commented on Oct 1, 2026

Research checkpoint: target PP results, MTP validation continues

novkien commented on Oct 1, 2026

Research result: GPU PP comparison and V100-focused engine candidates

novkien commented on Oct 1, 2026

Owner decision and research handoff: concurrent PP, then group TG

Completed evidence

Reviewed implementation direction for the next authorized step

Cleanup and artifacts

novkien commented on Oct 1, 2026

Implementation claim and owner authorization

novkien commented on Oct 1, 2026

Work report — integrated candidate and native build

novkien commented on Oct 1, 2026

Work report — reviewed combined candidate ready for publication

novkien commented on Oct 1, 2026

Work report — focused acceptance complete; combined PR prepared

novkien commented on Oct 1, 2026

Work report — combined PR opened; type-check correction pending

novkien commented on Oct 1, 2026

Work report — type-check correction pushed to combined PR

novkien commented on Oct 1, 2026

Work report — exact-head handoff ready to merge

novkien commented on Oct 1, 2026

Work report — scheduler candidate merged; live acceptance pending

novkien commented on Oct 1, 2026

Work report — immutable paired build and acceptance window

novkien commented on Oct 1, 2026

Work report — matched native pair installed; real model acceptance started

novkien commented on Oct 1, 2026

Acceptance and closure — deployed #5/#6

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions