[ggerganov]

cont #13746 (comment)

Overview

• Remove virtual llama_kv_cache::update()
• Remove virtual llama_kv_cache::defrag_sched()
• Add virtual llama_kv_cache::init_update()
• llama_kv_cache_unified::defrag_prepare() is now const

The logic for shifting and defragmenting the KV cache is now implemented using a memory state (i.e. llama_memory_state) for consistency with the decoding states that were introduced in #13746. The idea is that calling init_update() will check if any updates have to be performed without mutating the KV cache (a.k.a. the memory). We can then apply the created memory update state to perform the necessary updates:

llama.cpp/src/llama-context.cpp

Lines 451 to 461 in 503dda2

	const auto kv_state = kv_self->init_update(this, optimize);
	if (kv_state->get_status() == LLAMA_MEMORY_STATUS_NO_UPDATE) {
	// no updates need to be performed
	return false;
	}
	if (!kv_state->apply()) {
	LLAMA_LOG_ERROR("%s: failed to apply memory update\n", __func__);
	}
	}

This change generalizes the concept of updating the memory module. So far we have been doing KV cache shifts and defrags, but in the future we can do additional operations through this mechanism.

We also start to avoid the explicit "defrag" term as it is too specific for the unified KV cache. Instead, the init_update() method takes a bool optimize flag that can mean different things depending on the underlying memory implementation.

Next PRs

• Move the llama_kv_cache::init_* interface to llama_memory_i (see kv-cache : refactor + add llama_memory_state_i #13746 (review)) PR: memory : migrate from llama_kv_cache to more generic llama_memory #14006