[frank-2077]

Problem

The current CUDA graph cleanup mechanism in PR #21611 introduces a timing issue that causes unnecessary performance degradation in high-concurrency scenarios.

Current behavior in ggml-cuda/common.cuh:1467-1488:

// sweep every 5s, evicting cuda graphs unused for >=10s
if (time_now - last_graph_eviction_sweep >= 5'000'000) {
    last_graph_eviction_sweep = time_now;
    for (auto it = cuda_graphs.begin(); it != cuda_graphs.end(); ) {
        if (time_now - it->second->last_used_time >= 10'000'000) {
            it = cuda_graphs.erase(it);  // ← Can evict graph we're about to use!
        } else {
            ++it;
        }
    }
}

The issue:
If a graph's last_used_time is >10s ago, it gets deleted even when we're about to use it in the same function call. This forces expensive re-capture operations (warmup → capture → instantiate).

Solution

Exclude the currently requested graph from the cleanup sweep:

if (it->first != first_node_ptr && time_now - it->second->last_used_time >= 10'000'000) {

This simple check prevents deleting the graph we're actively requesting. The graph's last_used_time gets updated later in the function, so subsequent cleanups will preserve it correctly.

Code Change

File: ggml/src/ggml-cuda/common.cuh

-                if (time_now - it->second->last_used_time >= 10'000'000) {
+                if (it->first != first_node_ptr && time_now - it->second->last_used_time >= 10'000'000) {

1 line changed - adds a check to skip the currently requested graph during cleanup.

Impact on Production Deployments

Our deployment characteristics make this fix critical:

• Long-running service with intermittent 10s+ idle periods per graph
• High concurrency on single GPU, pushing hardware limits
• Expensive capture/instantiate operations under contention

Profiling Data - Capture & Instantiate Latency Growth (High Concurrency)

Under concurrent load (1→8 concurrent requests), re-capture costs become prohibitive:

Module	Capture Time (ms) 1→8 concurrent	Instantiate Time (ms) 1→8 concurrent
LLM	0.88 → 1.73 (p50: 0.95 → 1.18)	1.03 → 2.08
TTS-LLM	0.45 → 1.00	0.77 → 3.60
VPM	1.96 → 19.9 (p50: 3.1)	1.53 → 26.4
T2W-flow	13.6 → 50.2	20.4 → 198 (p95: 549ms)
T2W-vocoder	2.39 → 5.44	4.70 → 15.0

Key observation: Some modules see 10-27× latency increase under concurrency, likely due to CUDA driver contention. Avoiding unnecessary re-captures is essential for maintaining throughput.

Benefits

1. Eliminates pathological eviction - Graphs in active use never get evicted
2. Preserves original cleanup intent - Truly stale graphs (>10s genuinely unused) still get cleaned up
3. Minimal change - Only 1 line modified, easy to review and verify
4. Zero overhead - Single pointer comparison added to existing loop
5. Works for bursty workloads - Common pattern where same graph is used repeatedly with >10s gaps

Testing

• Verified that graphs requested within 10s of last use are not evicted
• Confirmed currently requested graph is never evicted regardless of idle time
• Confirmed cleanup still works for genuinely unused graphs
• Stress-tested under high concurrency with intermittent access patterns

---

This fix applies to any high-concurrency deployment where graph access patterns have occasional >10s gaps. We observed this in production serving multimodal workloads with bursty request patterns.

Note: Our workload uses a custom model architecture (Omni multimodal model) on a forked llama.cpp, but this fix applies to any high-concurrency deployment where graph access patterns have occasional >10s gaps.

[ggml-gh-bot]

Hi @frank-2077, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[am17an]

Seems like an okay fix, but does it really fix the issues you outlined? If not then please create an issue with a reproduction and someone will look into it.

[frank-2077]

@am17an Hi, I open a new PR. Let’s go over there to talk.