--batch: request priority (a free slot and the next prompt read go to the highest priority waiting)

[brocktice]

Why

We serve Qwen3.8-Flash-Next IQ3_S on two RTX PRO 6000 (one engine per card, parallel: 4, --batch-mtp, all experts in VRAM) to several clients at once: an interactive coding agent, a log-alert summarizer that must answer within seconds, and an evaluation battery that sends thousands of requests a day.

With first-come-first-served waiting, a burst of battery requests filled all 8 slots and the alert summarizer queued behind it, so its calls timed out and our monitoring paged us. More slots don't fix it, because a burst can always fill them. Under SGLang we used --enable-priority-scheduling (a per-request priority), and we lost that when we moved to Strata.

We've been running the same idea in a router in front of Strata for several days. The alert summarizer's slowest call over a full day was 4.6 s (1,210 calls) while the battery kept 8 requests in flight. This PR moves it into the server, where it can see the slots and the #656 give-way path.

What it does

"priority" in the request body is an integer, higher goes first. It's the same field SGLang and vLLM read, so gateways like LiteLLM already pass it through. If the field is absent or not an integer, it counts as 0.

With --batch:

• A free slot goes to the highest-priority request waiting for one.
• The control lines (the next prompt read or admission) go to the highest-priority request waiting for them.
• serve: cooperative prefill preemption at safe chunk boundaries (--prefill-preempt) #656 give-way: a long read gives way to a waiting request of a higher priority, whatever its length. It never gives way to a much shorter request of a lower priority. Within one priority level, the old rule applies (a prompt under half as long). YIELDS_MAX still caps how often a read gives way.
• No preemption: a running request is never paused.

With every request at 0, all checks reduce to the existing code paths.

Deadlock note: requests waiting for a slot never hold back the control lines. A read that gave way holds a slot and needs the lines to continue and free that slot, whatever its priority; otherwise long/long/short with mixed priorities would deadlock (the ENGINE_REVIEW finding 1 shape). A request that gave way and is letting others go first (after_epoch) also doesn't outrank anyone until its epoch passes.

Not covered: the single-request path without --batch (the server's fifo lock). That lock is shared with the vision encoder and settings changes, so changing it seemed like a separate discussion.

Tests

New tests in serve/test_parallel.py, using the existing fake engine:

• test_a_free_slot_goes_to_the_highest_priority_waiting: both slots busy, a priority-0 request waits, then a priority-5 one arrives; the first free slot goes to the priority-5 request.
• test_a_long_read_gives_way_to_a_higher_priority: two prompts of the same length; the later, higher-priority one makes the first read give way.
• test_no_way_given_to_a_lower_priority: a short priority-0 prompt does not make a priority-5 read give way.
• Priority.test_request_priority and test_outranked_skips_a_request_letting_others_go_first (unit).

All three service tests fail on main (checked three times each) and pass with the change. serve/test_parallel.py passed 5 runs in a row. The full serve/ suite passes except test_responses.NoLeakedSampler.test_server_close_ends_the_sampler, which also fails on unmodified main in 2 of 3 runs here (a sampler thread from an earlier test is still shutting down).

Real engine: v0.1.43 on one RTX PRO 6000 (Blackwell), Qwen3.8-Flash-Next IQ3_S, parallel: 4, --batch-mtp. The same request mix was run with and without the field:

Scenario	Without priority	With priority
4 slots decoding 700-token stories, 8 more waiting (priority 10); a short request arrives 2 s in. First token:	45.3 s / 39.1 s (10 and 8 of the waiting requests went first)	12.2 s / 12.0 s (the first slot to free, nobody ahead of it)
A ~84K-token read under way (priority 10); an equally long request arrives 3 s in. Its first token:	29.5 s (waited for the whole first read)	16.0 s (the first read gave way at 16,384 tokens, parked in slot 2, and finished at 33.3 s)

No errors; every request finished.

docs/BATCHING.md has a short paragraph describing the behavior.

Happy to change the shape: a config default per API key, /metrics waiting counts per priority, or the non-batch path.

🤖 Generated with Claude Code

[brocktice]

Rebased on main (v0.1.44.1, 296b59d) after #2170, as its description suggested: the priority now goes into the condition inside #2170's give_way() and ask(), so a read gives way to a higher priority at a PP line and on the quiet poll alike. Nothing else in this change moved.

New test: test_no_way_given_to_a_lower_priority_between_far_apart_progress_lines (--pp-every 80, a much shorter request of a lower priority gets no BYIELD). It fails when give_way() / ask() leave the priority out and passes with it.

Tests: serve/test_parallel.py 36 of 36. The whole serve suite: 791 run, 11 skipped, one failure in test_responses.NoLeakedSampler.test_server_close_ends_the_sampler that passes 3 of 3 alone, on this branch and on main alike (timing under the full run, not this change).

🤖 Generated with Claude Code