[am17an]

Overview

After #22789, it is not necessary that inputs to a split be bounded by a number as it can grow dynamically. This helps in graphs which require a lot of inputs + spec-dec (e.g. DSV4-Flash with recurrent rollback). Using this and #26610, using DSV4 + Dspark I'm able to maintain a single split across the Meta backend (which does not cache uid except the last one) enabling ~30-35 toks/s of sustained throughput.

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO

[ggerganov]

For my understanding, how does enabling spec-dec contribute to the increased number of inputs?

[am17an]

It adds the "roll-back" inputs

llama.cpp/src/llama-graph.h

Lines 600 to 605 in 7d04ce4

	ggml_tensor * state_persist_src_idxs = nullptr; // I32 [n_state_persist]
	ggml_tensor * state_persist_dst_idxs = nullptr; // I32 [n_state_persist]
	ggml_tensor * state_restore_src_idxs = nullptr; // I32 [n_state_restore]
	ggml_tensor * state_restore_dst_idxs = nullptr; // I32 [n_state_restore]
	ggml_tensor * state_snapshot_src_idxs = nullptr; // I32 [n_state_snapshot]
	ggml_tensor * state_snapshot_dst_idxs = nullptr; // I32 [n_state_snapshot]