Repository navigation
Feature request: serve concurrent requests (cross-request batching) #465
Description
With several clients on one GPU (orchestrator agent + sub-agents, a chat bot) everything queues: /v1/status reports concurrency.serving: 1, and DETAILS.md lists "one request at a time" as a v1 limit.
Request: batch decode steps across a few concurrent requests (even 2-4 slots), like llama.cpp --parallel.
Setup: RTX 3090 24 GB + 62 GB RAM, IQ3_S, 262K ctx, OpenCode multi-agent pipeline. Thanks!
Activity
Niko1221 commentedon Oct 2, 2026
Thanks for the request. Strata runs one sequence at a time by design: decode is limited by reading experts from RAM and over PCIe, and the engine keeps one sequence's state (KV, the draft layer, the conversation checkpoints) per process. Batching requests across users would mean rewriting the decode loop for several slots, with a shared expert cache and per-slot drafting. That is a large change, with limited gain while expert reads are the bottleneck. So it's not planned for now; requests queue and run in order. Closing; the request stays here for later.
#465 engine: batch slots for one GPU - warn instead of refuse, window…
#465 server: "parallel": N, slots in /metrics, one status per request…
#465 setup: --parallel N, and the count it recommends for the card
#465 tools: batch_interleave_test.py (a prompt read beside decoding s…
#465 a request left alone in a slot goes back to the solo path; docs:…
#465 engine: INFO slot_cache=0|1 beside batch_slots (the server sends…
#465 docs: the final build's RTX 5070 numbers; the turn-checkpoint ex…
#465 serve/test_parallel: the first request decodes long enough to ov…
#465 setup: recommend "parallel" only where the experts mostly fit in…
Niko1221 commentedon Oct 4, 2026
Planned after all: 0.1.39 has it as an opt-in.
"parallel": Ninstrata-<model>.json(or setup--parallel N) decodes up to N requests together in the engine's batch slots; more wait for a free slot. A request alone still runs the usual path with MTP drafts (and goes back to it when the others have finished); with two or more, each slot decodes one token per window and the weights are read once for all of them. A new prompt is read chunk by chunk with the other slots decoding in between, a long prompt gives way to a much shorter waiting one, and each slot keeps its conversation for the next turn. Each slot's session (0.56 GiB at 32K) comes out of the expert cache, so setup recommendsparallelonly where the experts mostly fit in VRAM (about 24 GB and up on one card for Q2_0, or a multi-GPU split); elsewhere it stays at one at a time, and--parallel Nis still honoured as asked.Measured on an RTX 5070 (12 GB) with Q2_0, 4 requests at once: the last one starts after 1.8 s instead of 11.2 s, but together they decode at 63 tok/s against 71 one after the other, and a request alone is 11-24% slower (the slots' VRAM comes out of the expert cache). On a card where most experts run on the CPU the slots buy waiting time, not speed. A card that holds more of the experts in VRAM may do better; we have not measured one yet. Details and limits: docs/BATCHING.md. The engine part is PR #559 by blange48.
Merge upstream Niko1221/Strata main (v0.1.39) into ac922
Merge Hardin22/Strata-DualGPU (v0.1.38-dual) onto 0.1.39 with the for…
serve: the Prometheus /metrics with "parallel" slots (one status per …
41 remaining items
replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): include/strata from ac922
replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): src/core from ac922
web: show each parallel slot as its own Model state card
web: show each parallel slot as its own Model state card
web: show each parallel slot as its own Model state card
web: show each parallel slot as its own Model state card
serve, web: a busy slot card can name the client that owns it
serve, web: a busy slot card can name the client that owns it
serve, web: a busy slot card can name the client that owns it
serve, web: a busy slot card can name the client that owns it
web: show each parallel slot as its own Model state card
serve, web: a busy slot card can name the client that owns it
docs(CONCURRENCY-RADIX): review-2 - concurrency is SHIPPED production…
docs(CONCURRENCY-RADIX): review-4 - reconcile with the feature's own …
ds41: DSpark and batch slots together (--spec dspark --batch N)