[am17an]

Overview

CUDA graphs scale with the size of the graph. We can amortize this cost by running smaller chunks. This idea was given to by @ORippler. This provides a 1-2% TG speed-up.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES

[aendk]

I proposed this a few years ago here #11867.
With CUDA graphs more stable, this makes more sense to merge now.

I can run Windows perf numbers later this week due to GTC, unless somebody else can help us with this.

In the meantime, my AI flagged testing as a possible area for improvement. Can you build a test which ensures that kernel fusion is respected by your new logic? Also, are there backend tests which we can run with GGML_CUDA_GRAPH_CHUNK_SIZE=1 and the default values, to compare the outputs?

[am17an]

I did test this across a bench of devices and models - however I don't know what a good test to add to the CI would be as we don't have any existing test suite for checking CUDA graph regressions either.

[ORippler]

1. Did we test multi-GPU behavior? AFAIK we enqueue already split subgraphs on GP0 -> GPU1 -> reduce -> repeat so there should be no perf deltas, as we likely we won't be hitting the 512 magic number.
2. Should evaluate the magic number 512 from perf perspective, but this can be done in follow-up work. We should smoke-test Windows (x86 & RTX Spark) against perf regressions, but someone from nvidia can do this.

however I don't know what a good test to add to the CI would be as we don't have any existing test suite for checking CUDA graph regressions either.

I feel we should add dedicated fusion tests and separate them out from test-backend-ops (we currently don't enforce that fusion actually happens for the patterns we want to test fusion for for example). As part of a backend-specific operator test-suite we should evaluate cudaGraphs as well (this is beyond the scope of this PR imo)

[ORippler]

Suggested change

	size_t num_nodes = 0;
	std::vector<cudaGraphNode_t> nodes;

unrelated clean-up we could do

[ORippler]

On what HW/Model/OS config was that magic number determined? Wondering if it makes sense to distinguish between Windows/Linux potentially. Can be a follow-up PR though

[am17an]

I tried 128,256,512 and 1024. Multi-GPU has no improvement in my tests.

[ORippler]

Perf gains seem to be minor on RTX Spark (it's not much, but it's honest work I guess)

Measurement	PR~.json	PR.json	PR reduction
Average latency	9.9381s	9.9047s	0.34%
Average prediction speed	40.564 t/s	40.744 t/s
Total request latency	874.557s	871.615s	2.942s
Total prompt processing	37.423s	37.321s	0.101s
Total generation	619.543s	616.908s	2.635s
Remaining request time*	217.591s	217.386s	0.206s

[ORippler]

@aendk feel free to also take a look, from my side it's good