[Engine] serve: the RAM gate releases the heap and stops evicting when it cannot help (rebased #1394) by alanthinker · Pull Request #2253 · Niko1221/Strata

alanthinker · 2026-10-13T09:45:38Z

Problem and result

The physical-RAM gate of the parking path assumes an evicted parked conversation is visible to /proc/meminfo before the next check reads it. That assumption is wrong for the allocations this cache actually makes: ConversationBuffer is a list of 16 MiB segments, and glibc's mmap threshold is dynamic - the first generation of segments is mmap'd, freeing one raises the threshold past 16 MiB (capped at 32 MiB), and every later segment comes from the main heap. Freeing one of those only moves the bytes onto the allocator's free list: RSS, and with it MemAvailable, does not move.

On today's main (0.2.0-beta) the gate added for #1347 evicts parked conversations one at a time and re-reads the figure after each. In steady state the figure never moves, so the loop can empty the whole cache and still refuse the snapshot - worse than the refusal it was meant to remove: every parked conversation is lost and every later request re-reads its whole prompt (measured in #1347: 270k tokens, 96 s, 32 times in one session).

After this change a release (malloc_trim(0)) runs before the first re-read and after every eviction, so a dropped conversation shows up in the next reading; and where the missing bytes exceed what the parked conversations could give back, the gate gives up before evicting anything.

Related discussion

#1347 (the issue, reproduced on an RTX 4080 SUPER with 91 GB RAM), #1394 (the original PR by @Arthur031221, unmergeable after #879 / #1601 / #1573 landed), and this thread:

@qiuzan2001's point about the two contradicting comments is fixed in the second commit.
@alfre2v reproduced the bug independently on an RTX 3090 / 64 GB box (glibc 2.35, sm_86) and tested this branch there: unpatched v0.1.44.1 failed in every run (8 refusals in one, the engine heap reaching 6.9 GiB, a 214K-token snapshot re-read in 90 s), and this PR admitted the snapshot in all three of its runs with no cost on fresh 4K-32K prompts. Their numbers also answer the trim-timing question raised above: on a heap holding up to 5.5 GiB of freed segments a trim took 37-482 ms (median about 111 ms, roughly 80-90 ms per GiB returned), 12-14 trims per 19-minute run, about 2.4 s in total - against a 90 s re-read. They also measured two side effects (the heap stays large between trims; long prompts read about 2% slower right after a trim) and one residual case outside this PR, the post-capture floor check dropping a freshly captured snapshot in 1 of 3 runs.

Approach

conversation_release_freed_memory() (malloc_trim(0) under __GLIBC__, a no-op elsewhere) is called before the first re-read and after every eviction.
ConversationCache::admit_ram() gives up before evicting anything when the missing bytes exceed evictable_bytes() - what evict_oldest() may actually take, i.e. entries not holding a pinned shared prefix - and re-checks the figure after each reading, so losing telemetry ends the loop.
The reading stays the memory guard's (sample_host_memory(): MemAvailable, the cgroup limits above the engine, --memory-limit-mib) and the refusal line keeps ram_note().

Validation

Rebased on main at e88ee39d (0.2.0-beta, engine 0.1.45.3); branch tip 3dbc9aca. Built and tested by me on the serving box from #1347: Linux, RTX 4080 SUPER (32 GB), 94 GB RAM, CUDA 13.2, sm_89.

Passed:

cmake -B build -DSTRATA_BUILD_CONVERSATION_TESTS=ON -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89 && cmake --build build --target conversation_cache_test conversation_memory_test
conversation_cache_test: 4829 checks passed
conversation_memory_test: 24 checks passed
the cache tests main added since: session_save_reclaim_test 119 checks, conversation_file_test 201 checks, conversation_split_failure_test passed
the full engine target strata compiles (CUDA 13.2, sm_89)

Failed: none.

Not run: running the rebuilt 0.2.0-beta engine on the GPU (this box still serves on the deployed 0.1.x build; the binary compiled, it was not started); HIP, SYCL and Metal builds.

Separate from my checks: @alfre2v ran this branch (at ab963c96 + f09887f6) end-to-end on an RTX 3090 / 64 GB box, results in Related discussion; their soak numbers are theirs, not mine.

Limits

The -0.3 MB / 127.7 MB figures in the first comment were measured on the v0.1.43.1-era serving heap; I have not reproduced that heap shape on 0.2.0-beta, so I quote no new trim numbers of my own - @alfre2v's trim timings above are the measured answer.
malloc_trim(0) returns what has coalesced to an arena's top; a mid-heap segment that cannot coalesce stays invisible even after the release. The loop still converges: further evictions eventually coalesce, and evictable_bytes() bounds the case where even emptying the cache could not help.
The trim runs only when a reading comes out short, so between parks the freed segments stay in the heap (@alfre2v: up to 8.9 GiB on their box); other programs on the machine see less free RAM in between, and long prompts read about 2% slower right after a trim (fresh pages fault in).
The post-capture floor check keeps its own "freed heap is visible at once" assumption and can still drop a freshly captured snapshot (@alfre2v saw it in 1 of 3 runs); this PR does not touch it.
--session-save-reclaim (serve: reclaim host caches and wait for SAVE memory admission #1573) keeps the same assumption between its steps; left alone here to keep this PR to the parking gate.

…ot help The physical-RAM admission gate of the parking path (generate.cpp) evicts parked conversations one at a time and re-reads the host's free RAM after each, on the assumption that dropping a conversation is visible to /proc/meminfo at once. That assumption is wrong for the allocation this cache actually makes. ConversationBuffer is a list of 16 MiB segments (segment_bytes) and each segment is one operator new. glibc's mmap threshold is dynamic: the first generation of segments is mmap'd and munmap() returns it, but freeing an mmap'd block larger than the threshold raises the threshold to that size (up to 32 MiB), so every later segment comes from the main heap instead. Freeing one of those only moves the bytes onto the allocator's free list: unless the chunk reaches the top of the heap, RSS - and therefore MemAvailable - does not move at all. Reproduced on this machine (a stand-alone test that parks and drops 8 x 16 MiB segments in two generations): gen2 parked (8 x 16 MiB) : RSS 129 MiB gen2: evict ONE mid-heap : RSS 129 MiB <- the reading does not move gen2: after malloc_trim(0) : RSS 113 MiB <- only now is it visible So the loop can evict conversation after conversation while the figure never moves, and in the worst case empty the cache and still refuse the snapshot - worse than the refusal it was meant to remove, and it throws away the prompt read that built it. This is Arthur031221's Niko1221#1394, rebased onto the current main: - conversation_release_freed_memory() (malloc_trim(0) under __GLIBC__, a no-op elsewhere) is called before the first re-read and after every eviction, so a dropped conversation shows up in the next reading. - admit_ram() gives up before evicting anything when the missing bytes exceed evictable_bytes(), so a shortfall the parked conversations cannot cover no longer empties the cache for nothing, and it re-checks the figure after each reading so losing telemetry ends the loop. Adapted to what main has since gained: the reading stays the memory guard's (sample_host_memory(): MemAvailable, the cgroup limits above the engine, --memory-limit-mib) and the refusal line keeps ram_note(); evictable_bytes() counts only the entries evict_oldest() may actually take, i.e. those not holding a pinned shared prefix (Niko1221#879). Tests are Niko1221#1394's: a simulated host that returns half, a fifth, or none of what was dropped, another process taking RAM between evictions, a host already under the floor, an empty cache, and no telemetry at all. (136 check() sites here against the 109 in main's current gate.) conversation_cache_test: 4276 checks passed conversation_memory_test: 24 checks passed (-DSTRATA_BUILD_CONVERSATION_TESTS=ON) Supersedes Niko1221#1394 (unmergeable against main now that main has the pinned prefixes, sample_host_memory() and --session-save-reclaim).

…p and stops evicting when it cannot help (rebased Niko1221#1394) Source: Niko1221#2253 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…up robustness, conversation cache memory) Serve: Niko1221#2290 Niko1221#2284 Niko1221#2285 Niko1221#2170 Niko1221#2140 Niko1221#2278. Setup: Niko1221#2161 Niko1221#2287. Engine: Niko1221#2253 Niko1221#2265 Niko1221#2247 Niko1221#2217 Niko1221#2243. New opt-in engine flags: --session-state-codec (enabled in the local configs), --conversation-cache-park-gain and --drop-file-cache (left off: no measured benefit on the RX 7600 XT box). Bench: bench/results/upstream-prs-2026-10-13. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

qiuzan2001 · 2026-10-15T04:40:43Z

Small consistency point from reading the branch: the comment above evict_oldest() in include/strata/core/conversation_cache.hpp (line 281 on this branch) still says each 16 MiB segment is "far above glibc's mmap threshold, so dropping an entry returns the whole footprint" to the kernel, while the new admit_ram comment a few lines below (line 304) explains that the heap keeps it once the mmap threshold has risen, which is the reason for this PR. Updating the older comment would keep the two from contradicting each other. Also, the description gives what malloc_trim(0) returned (127.7 MB) but not how long it took; since it now runs on the request path after a refusal and after each eviction, its time on a serving heap would complete the numbers.

…view)

alanthinker · 2026-10-15T05:00:11Z

Good catch - fixed in f09887f: the evict_oldest() comment now says what the measurements showed (the threshold is dynamic; once it has risen past 16 MiB, a dropped segment only reaches the allocator's free list, and the reading moves only after the release). On the trim timing: the 127.7 MB figure came from the serving heap at the moment of the refusal; I have not been able to reproduce that heap shape since, so I cannot honestly time the call.

Niko1221 · 2026-10-17T07:18:52Z

Thanks. The glibc mmap-threshold finding is a useful one, but it is a fix for the RAM gate rather than a speed change, so it is outside this round. We would take it as a bug fix.

alfre2v · 2026-10-17T16:47:22Z

I hit this bug independently on a single RTX 3090 box and tested this PR there, against unpatched v0.1.44.1 and against a GLIBC_TUNABLES workaround. The short version:

Unpatched v0.1.44.1 reproduces the bug in every run of the same sequence. [Engine] serve: the RAM gate releases the heap and stops evicting when it cannot help (rebased #1394) #2253 fixes the RAM gate in all three of its runs, with no cost on fresh 4K-32K prompts.
@qiuzan2001's question about malloc_trim(0)'s time on a serving heap: on a heap of 5.9-9.7 GiB holding up to 5.5 GiB of freed segments, a trim took 37-482 ms (median about 111 ms, roughly 80-90 ms per GiB returned). There were 12-14 trims per run, about 2.4 s in total over a 19-minute run.
Two side effects and one residual case are worth knowing before it lands (details below):
- the heap stays large between trims;
- long prompts read about 2% slower right after a trim;
- in 1 of 3 runs, the post-capture floor check (generate.cpp:8362-8367, which this PR doesn't touch) threw away a 214K-token snapshot after capturing it.

@Niko1221, since you'd take this as a bug fix, I hope the numbers help.

Setup

Hardware: RTX 3090 24 GB, Ryzen 9 3900X, 64 GB RAM (62.7 GiB usable), Ubuntu 22.04, glibc 2.35, CUDA 13.0 (sm_86). The engine was built locally; experts are resident (46.84 GiB).
Model and config: Qwen3.8-Flash-Next IQ3_S, --max-context 262144 --kv int8 --kv-grow --conversation-cache-mib 8192 --conversation-cache-slots 4 --conversation-cache-min-tokens 2048 --conversation-cache-min-free-mib 3072, with STRATA_KV_GROW_HOLD=1 and STRATA_ADAPT_OVERLAP=1.
Builds, with identical CMake settings:
- v0.1.44.1 (296b59d);
- v0.1.44.1 + this PR (ab963c96 + f09887f6), which merges without conflicts. For measuring only, I added one log line around the malloc_trim(0) call (diff at the end).
The PR's tests on Linux: conversation_cache_test 4,276 checks and conversation_memory_test 24 checks pass, as the description says.
Each run, on a freshly started server:
1. a ladder of fresh prompts at 2K / 32K / 64K / 120K / 250K, 3 times each, plus a needle at 250K;
2. tools/strata_bench.py --long --runs 3;
3. an agent-style soak: one main conversation growing by about 20K tokens per turn to 214K over 10 turns, alternating with two subagents (+4K per visit), 128 tokens out, temperature 0.
The ladder parks a conversation at every switch, so by the time the soak starts, the heap already holds GiBs of freed segments. That's the state a fresh server never reaches.

Unpatched v0.1.44.1: the bug, every run

Same box, same sequence, on two different days. Run 1:

strata serve: conversation cache: skip parking (physical RAM admission; need 1483 MiB plus 3072 MiB floor, 4501 MiB available, source meminfo; evicted 3, 0 still parked)
strata serve: prompt 214234 tokens = 0 reused + 214234 read in 90425 ms (2369.2 tok/s), ...

What it did: after evicting all three parked conversations, the reading was still only 4,501 MiB, 54 MiB short of what it needed. So the gate refused, and the main conversation's next turn re-read all 214K tokens.
The scale: 8 refusals in that run. The engine's [heap] reached 6.9 GiB. In an identical earlier run, it still held 5.3 GiB at the end with at most one conversation parked; a fresh engine's is about 1 GiB.

Run 2 (the same-day control for this PR): at 250K the gate refused twice in the ladder. In the soak, the decisive park was admitted only after evicting 3 conversations (evicted 3 parked conversations to admit this snapshot (1483 MiB plus 3072 MiB floor)), where the patched build needed at most 1. One of the three was a subagent's, so its last visit started from zero (0 reused + 20207 read).

This PR: results

	v0.1.44.1 (same day)	+ #2253 (3 runs)	v0.1.44.1 + `GLIBC_TUNABLES` 4 MiB (same day)
Soak: main turns that resumed their context	9/9	9/9, 9/9, 9/9	9/9
Soak: subagent revisits that resumed	7/8	8/8, 8/8, 8/8	8/8
`skip parking (physical RAM admission …)` per run	2	0, 0, 0	0
`strata_bench` prefill, 4K / 16K / 32K (tok/s)	2,077 / 2,618 / 2,695	2,095 / 2,611 / 2,711; 2,154 / 2,616 / 2,714; 2,080 / 2,622 / 2,705	1,988 / 2,590 / 2,611
Decode (ladder, 2K-250K)		within ±5% of unpatched	within ±5%
`[heap]` max	7.0 GiB	8.7-8.9 GiB	2.1 GiB

The third [Engine] serve: the RAM gate releases the heap and stops evicting when it cannot help (rebased #1394) #2253 run also carried a test-only extra check (below) that never triggered, so that run is [Engine] serve: the RAM gate releases the heap and stops evicting when it cannot help (rebased #1394) #2253's behaviour too.
On this box, run-to-run spread on the 4K prompt is about 3.5%, so the three [Engine] serve: the RAM gate releases the heap and stops evicting when it cannot help (rebased #1394) #2253 runs and the unpatched one match.
The workaround's cost shows clearly at 4K and 32K.

`malloc_trim(0)` timings

From the test-only log line (RSS before → after, and mallinfo2() read before the timer starts):

Run	Trims	Median	Max	Total
1	14	111.1 ms	417.0 ms (returned 5,357 MiB)	2,402 ms
2	12	114.5 ms	481.9 ms (returned 5,330 MiB)	2,379 ms
3	13	111.2 ms	434.0 ms (returned 5,588 MiB)	2,396 ms

The time tracks the bytes returned: 40 ms for 0.5 GiB, about 110 ms for 1.4 GiB, 210-230 ms for 2.6-2.8 GiB, 330-480 ms for 4-5.5 GiB.
The largest trims came in the 250K part of the ladder, where each park frees about 4.5 GiB of segments.
In the soak (parks at up to 214K), trims returned 0.5-0.8 GiB in 37-67 ms.
For scale: a 214K-token re-read costs about 90 s here, and a park 0.1-0.6 s.

All trims of run 1

malloc_trim #1 took 40.1 ms (returned memory); RSS 55904 -> 55403 MiB (500 MiB released); heap before: 5880 MiB, 538 MiB of it free
malloc_trim #2 took 75.3 ms (returned memory); RSS 57386 -> 56434 MiB (952 MiB released); heap before: 7613 MiB, 1004 MiB of it free
malloc_trim #3 took 111.6 ms (returned memory); RSS 56322 -> 54940 MiB (1382 MiB released); heap before: 7613 MiB, 2385 MiB of it free
malloc_trim #4 took 111.1 ms (returned memory); RSS 57400 -> 55986 MiB (1414 MiB released); heap before: 7843 MiB, 1457 MiB of it free
malloc_trim #5 took 169.7 ms (returned memory); RSS 55650 -> 53450 MiB (2199 MiB released); heap before: 7843 MiB, 3656 MiB of it free
malloc_trim #6 took 212.0 ms (returned memory); RSS 56131 -> 53455 MiB (2676 MiB released); heap before: 7620 MiB, 2756 MiB of it free
malloc_trim #7 took 417.0 ms (returned memory); RSS 56277 -> 50919 MiB (5357 MiB released); heap before: 7619 MiB, 5404 MiB of it free
malloc_trim #8 took 362.1 ms (returned memory); RSS 55501 -> 50924 MiB (4577 MiB released); heap before: 7623 MiB, 5295 MiB of it free
malloc_trim #9 took 365.0 ms (returned memory); RSS 55507 -> 50924 MiB (4583 MiB released); heap before: 7624 MiB, 5296 MiB of it free
malloc_trim #10 took 365.0 ms (returned memory); RSS 55492 -> 50921 MiB (4571 MiB released); heap before: 7624 MiB, 5299 MiB of it free
malloc_trim #11 took 48.9 ms (returned memory); RSS 56999 -> 56393 MiB (606 MiB released); heap before: 8380 MiB, 647 MiB of it free
malloc_trim #12 took 39.8 ms (returned memory); RSS 57261 -> 56776 MiB (485 MiB released); heap before: 8700 MiB, 550 MiB of it free
malloc_trim #13 took 36.9 ms (returned memory); RSS 57653 -> 57190 MiB (463 MiB released); heap before: 9378 MiB, 534 MiB of it free
malloc_trim #14 took 47.1 ms (returned memory); RSS 57815 -> 57232 MiB (584 MiB released); heap before: 9652 MiB, 644 MiB of it free

Side effects worth knowing

The heap stays large between gates. The trim runs only when a reading comes out short, so between parks the freed segments stay in the heap: up to 8.9 GiB here, against 2.1 GiB when every segment is its own mapping.
- In the last minute of the soak, MemAvailable was about 1.1 GB lower than with the 4 MiB threshold (3.5-3.9 GB vs 4.6-4.9 GB).
- The gate's decisions are right; other programs on the machine just see less free RAM in between.
Long prompts read about 2% slower right after a trim. In all three runs, the first 120K prompt (before any trim) read at the unpatched speed: 2,481 / 2,499 / 2,481 tok/s, against 2,474-2,508 unpatched. The 120K prompts after the first trims read at 2,417-2,449, and the 250K medians were 2,286-2,291 (unpatched: 2,341).
- My reading: the trim hands the heap's free pages back, so the next prompt's large buffers fault fresh pages in. That's the same thing that makes a fixed low threshold cost prefill everywhere, here paid only after a trim.
- In the soak, the main turns' total prompt time matched unpatched (100.0-100.2 s vs 100.9 s).

A residual case: the post-capture floor check (1 of 3 runs)

In run 1, while parking the 214K main conversation at the end of the soak:

strata serve: conversation cache: malloc_trim #14 took 47.1 ms (returned memory); RSS 57815 -> 57232 MiB (584 MiB released); heap before: 9652 MiB, 644 MiB of it free
strata serve: conversation cache: skip parking (RAM floor after capture: 3072 MiB needed, 3011 MiB available, source meminfo)

What happened:

the gate admitted the snapshot after one trim, with no eviction and all four slots still parked (7.59 GB);
the snapshot was captured;
then the separate check after the capture found free RAM 61 MiB under the floor, and dropped the freshly captured snapshot. A next main turn would have re-read 214K tokens.

Why the other two runs didn't hit it: in both, an eviction at the previous park had left three conversations parked (6.71 GB), so the 214K capture had about 0.9 GB more room.

My reading: the gate passed with almost no margin, and the capture then took slightly more than the gate allowed for (or something else used RAM in between; the gate's additional figure isn't logged when it admits without evicting, so I can't tell which). This check never fired in 11 other runs of this sequence, unpatched or with a fixed threshold of 4-32 MiB.

Ideas, untested:

Release once before the post-capture reading too, and read again before refusing. I built this as a test-only change (the third run above). Its reading never came out short in that run, so it never acted: no harm, but no evidence either.
Make the post-capture check evict the oldest other parked conversation (with a release) instead of discarding the new snapshot, as the gate already does. This handles the overshoot whatever its cause, and keeps the newest snapshot.
A small safety margin at the gate.

A workaround for anyone waiting

Until a fix is released, one line in the run config's env block (the server passes it to the engine) avoids the gate failure:

"GLIBC_TUNABLES": "glibc.malloc.mmap_threshold=4194304"

Thresholds tested (same sequence, one run each):
- 4 MiB and 8 MiB fix it.
- 16 MiB fixes it only partly. The smaller pieces (each buffer's last segment, and the 64 KiB → 16 MiB growing pieces of re-parks) still go to the heap, and glibc serves later 16 MiB requests from that free heap memory before it ever consults the threshold.
- 32 MiB and unset fail.
Cost: about 4-9% prefill at 4K, about 3% at 32K, about 2-3% at 120-250K. The engine's own 8-16 MiB working buffers lose heap reuse. Decode is unaffected.
This PR avoids that cost on short prompts.

Not tested

HIP and SYCL builds, Windows, macOS, layer split, --batch, --session-save-reclaim, and models other than Qwen3.8-Flash-Next IQ3_S. One machine, CUDA 13.0 on Linux only.

The test-only timing patch (on top of this PR; abridged: the #include <chrono> and a glibc ≥ 2.33 guard around mallinfo2() are left out)

 void conversation_release_freed_memory() {
 #if defined(__GLIBC__)
-    malloc_trim(0);
+    static std::atomic<unsigned> trims{0};
+    auto rss_mib = [] {
+        long pages = 0, resident = 0;
+        if (FILE* f = std::fopen("/proc/self/statm", "r")) {
+            if (std::fscanf(f, "%ld %ld", &pages, &resident) != 2) resident = 0;
+            std::fclose(f);
+        }
+        return (double) resident * 4096.0 / (1024.0 * 1024.0);
+    };
+    const struct mallinfo2 mi = mallinfo2();
+    const double heap_mib = (double) mi.arena / (1024.0 * 1024.0), free_mib = (double) mi.fordblks / (1024.0 * 1024.0);
+    const double before = rss_mib();
+    const auto t0 = std::chrono::steady_clock::now();
+    const int released = malloc_trim(0);
+    const double ms = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - t0).count();
+    const double after = rss_mib();
+    std::fprintf(stderr, "strata serve: conversation cache: malloc_trim #%u took %.1f ms (returned %s); RSS %.0f -> %.0f MiB "
+                         "(%.0f MiB released); heap before: %.0f MiB, %.0f MiB of it free\n",
+                 ++trims, ms, released ? "memory" : "nothing", before, after, before - after, heap_mib, free_mib);
 #endif
 }

I analyzed all the reports with my own bloodshot human eyes and guided the agent in this exploration. I used Claude (Anthropic) to help me run these measurements and co-write this report. This work started with me looking for a good parking strategy for Strata that works well for agentic coding workflows (the OpenCode harness), within the hardware constraints described above. All runs, logs and numbers come from my machine.

Niko1221 · 2026-10-19T17:13:43Z

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

…ot help The physical-RAM admission gate of the parking path (generate.cpp) evicts parked conversations one at a time and re-reads the host's free RAM after each, on the assumption that dropping a conversation is visible to /proc/meminfo at once. That assumption is wrong for the allocation this cache actually makes. ConversationBuffer is a list of 16 MiB segments (segment_bytes) and each segment is one operator new. glibc's mmap threshold is dynamic: the first generation of segments is mmap'd and munmap() returns it, but freeing an mmap'd block larger than the threshold raises the threshold to that size (up to 32 MiB), so every later segment comes from the main heap instead. Freeing one of those only moves the bytes onto the allocator's free list: unless the chunk reaches the top of the heap, RSS - and therefore MemAvailable - does not move at all. Reproduced on this machine (a stand-alone test that parks and drops 8 x 16 MiB segments in two generations): gen2 parked (8 x 16 MiB) : RSS 129 MiB gen2: evict ONE mid-heap : RSS 129 MiB <- the reading does not move gen2: after malloc_trim(0) : RSS 113 MiB <- only now is it visible So the loop can evict conversation after conversation while the figure never moves, and in the worst case empty the cache and still refuse the snapshot - worse than the refusal it was meant to remove, and it throws away the prompt read that built it. This is Arthur031221's Niko1221#1394, rebased onto the current main: - conversation_release_freed_memory() (malloc_trim(0) under __GLIBC__, a no-op elsewhere) is called before the first re-read and after every eviction, so a dropped conversation shows up in the next reading. - admit_ram() gives up before evicting anything when the missing bytes exceed evictable_bytes(), so a shortfall the parked conversations cannot cover no longer empties the cache for nothing, and it re-checks the figure after each reading so losing telemetry ends the loop. Adapted to what main has since gained: the reading stays the memory guard's (sample_host_memory(): MemAvailable, the cgroup limits above the engine, --memory-limit-mib) and the refusal line keeps ram_note(); evictable_bytes() counts only the entries evict_oldest() may actually take, i.e. those not holding a pinned shared prefix (Niko1221#879). Tests are Niko1221#1394's: a simulated host that returns half, a fifth, or none of what was dropped, another process taking RAM between evictions, a host already under the floor, an empty cache, and no telemetry at all. (136 check() sites here against the 109 in main's current gate.) conversation_cache_test: 4276 checks passed conversation_memory_test: 24 checks passed (-DSTRATA_BUILD_CONVERSATION_TESTS=ON) Supersedes Niko1221#1394 (unmergeable against main now that main has the pinned prefixes, sample_host_memory() and --session-save-reclaim).

…view)

alanthinker · 2026-10-19T23:55:32Z

/reopen

github-actions · 2026-10-19T23:55:41Z

The check passed, but GitHub would not reopen it (Validation Failed: ). If the branch was deleted, restore it or open a new pull request and link this one.

…le for /reopen

alanthinker · 2026-10-19T23:57:25Z

/reopen

github-actions · 2026-10-19T23:57:37Z

Reopened, thank you. A maintainer will pick it up from here. If it conflicts with main, rebase it first; that speeds up review.

alanthinker mentioned this pull request Oct 13, 2026

serve: evict parked conversations to pass the physical-RAM gate instead of dropping the snapshot #1394

Closed

qiuzan2001 mentioned this pull request Oct 15, 2026

[Engine] evict redundant holders while preserving pinned prefixes #2443

Open

serve: the evict_oldest comment matches the raised mmap threshold (re…

f09887f

…view)

alfre2v mentioned this pull request Oct 17, 2026

[Bug]: Long sessions: parking evicts every conversation and still refuses (freed memory stays in glibc's heap) #2934

Closed

Niko1221 closed this Oct 19, 2026

alanthinker and others added 2 commits October 20, 2026 07:29

serve: the evict_oldest comment matches the raised mmap threshold (re…

3dbc9ac

…view)

alanthinker changed the title ~~serve: the RAM gate releases the heap and stops evicting when it cannot help (rebased #1394)~~ [Engine] serve: the RAM gate releases the heap and stops evicting when it cannot help (rebased #1394) Oct 19, 2026

Merge the pre-rebase branch tip so the closed PR's head stays reachab…

3ee9597

…le for /reopen

github-actions Bot reopened this Oct 19, 2026

Navigation Menu

Repository navigation

[Engine] serve: the RAM gate releases the heap and stops evicting when it cannot help (rebased #1394) - #2253

[Engine] serve: the RAM gate releases the heap and stops evicting when it cannot help (rebased #1394)#2253
alanthinker wants to merge 5 commits into
Niko1221:mainfrom
alanthinker:conversation-cache-ram-gate-release

alanthinker commented Oct 13, 2026 •

edited

Loading

Uh oh!

qiuzan2001 commented Oct 15, 2026

Uh oh!

alanthinker commented Oct 15, 2026

Uh oh!

Niko1221 commented Oct 17, 2026

Uh oh!

alfre2v commented Oct 17, 2026

Uh oh!

Niko1221 commented Oct 19, 2026

Uh oh!

alanthinker commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

alanthinker commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

alanthinker commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem and result

Related discussion

Approach

Validation

Limits

Uh oh!

qiuzan2001 commented Oct 15, 2026

Uh oh!

alanthinker commented Oct 15, 2026

Uh oh!

Niko1221 commented Oct 17, 2026

Uh oh!

alfre2v commented Oct 17, 2026

Setup

Unpatched v0.1.44.1: the bug, every run

This PR: results

malloc_trim(0) timings

Side effects worth knowing

A residual case: the post-capture floor check (1 of 3 runs)

A workaround for anyone waiting

Not tested

Uh oh!

Niko1221 commented Oct 19, 2026

Uh oh!

alanthinker commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

alanthinker commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alanthinker commented Oct 13, 2026 •

edited

Loading

`malloc_trim(0)` timings