v0.6.2

Small additive release — groundwork for KV-cache quantization, seq_id branching, and an early-stop knob on the streaming callback. Full backwards compatibility: every new parameter is optional with a default that preserves prior behavior.

Added

• KV-cache quantization on LlamaCppConnector — create_inference_context() accepts type_k and type_v (GGML quantization type integers). Pass type_k=8, type_v=8 for Q8_0 — roughly 50% KV-cache memory savings, indistinguishable accuracy on 7B models. Pass type_k=2, type_v=2 for Q4_0 — ~75% savings, model-dependent quality drop. Defaults to None (FP16). Existing callers see no change.
• n_seq_max on create_inference_context() — Maximum number of distinct sequences (seq_ids) the context can hold. Defaults to 1 (the prior implicit behavior). Set higher to enable seq_id branching: copying KV state between sequences and pruning one without affecting another.
• seq_id on generate_on_context() — Generate against a specific sequence in the context. Defaults to 0. Pairs with n_seq_max>1 for branched generation.
• Early-stop support in token_callback — The streaming callback can now return True to break out of the sampling loop immediately. Useful when the consumer detects completion (e.g. a full JSON tool call) before the model emits EOS. Backwards compatible: existing callbacks return None, which is not is True.

Compatibility

PROTOCOL_VERSION (0x01) and AVP_VERSION_HEADER ("0.4") are unchanged — no wire-format or transport-binding changes since v0.6.1. The handshake avp_version string moves to 0.6.2 to track the SDK feature surface.

Install

pip install --upgrade 'avp[ollama]>=0.6.2,<0.7'

Notes

This release groundworks the KV-cache transfer pipeline used by SubtextChain (multi-agent framework with same-process KV transfer) — specifically Q8_0 KV quantization and the seq_id surface needed for post-MVP worker → main return. Both knobs are optional; SDK consumers who only use think() / generate() won't notice anything different.