research(nightly): hierarchical-cluster-rag — coherence-weighted two-level cluster tree for agent memory RAG

[ruvnet]

Summary

• Adds ruvector-cluster-rag: a zero-dependency Rust crate implementing a two-level cluster-summary index for agent memory RAG, inspired by RAPTOR (ICLR 2024) and classical IVF.
• Three benchmarked variants: FlatBrute (ground truth), ClusterSearch (IVF-style), CoherenceTree (coherence-weighted cluster scoring).
• Per-cluster cohesion score λ·sim(q,c) + (1-λ)·cohesion(c) routes queries toward tight, semantically concentrated clusters rather than all-or-nothing brute force.
• Adds ADR-298 recording the design rationale and production path.
• Adds nightly research document with full benchmark methodology, forward-looking thesis, and ecosystem integration notes.

Real benchmark results (x86_64 Linux, release build)

Config: n=10,000, dim=128, nq=500, k=10, k_clusters=40, nprobe=20, lambda=0.70

Variant	Mean µs	p95 µs	QPS	Recall@10	Memory
FlatBrute (ground truth)	1490.9	1567.7	671	1.000	4.9 MB
ClusterSearch	1034.9	1270.1	966	0.779	5.0 MB
CoherenceTree	981.4	1070.6	1019	0.776	5.0 MB

Acceptance gate: PASS — both cluster variants ≥ 0.70 recall@10. Memory overhead: 2.0% above raw leaf storage.

Honest finding: on uniform random data, CoherenceTree ≈ ClusterSearch in recall (expected — all clusters have similar cohesion on random data). The coherence advantage emerges on structured corpora where cluster tightness varies. Measuring this on real embeddings is the primary next step.

Files changed

• crates/ruvector-cluster-rag/ — new crate (6 source files, 1 binary, Cargo.toml)
• docs/adr/ADR-298-hierarchical-cluster-rag.md — architecture decision record
• docs/research/nightly/2026-08-07-hierarchical-cluster-rag/README.md — full research document
• docs/research/nightly/2026-08-07-hierarchical-cluster-rag/gist.md — SEO-optimised public technical article
• Cargo.toml — workspace member registration

Test status

cargo test -p ruvector-cluster-rag
# 14 passed; 0 failed
cargo run --release -p ruvector-cluster-rag --bin benchmark
# ACCEPTANCE PASS: ClusterSearch recall 0.779 ≥ 0.70
# ACCEPTANCE PASS: CoherenceTree recall 0.776 ≥ 0.70
# All acceptance criteria met. Benchmark complete.

Ecosystem connections

• ruvector-agent-memory: drop-in accelerated backend for growing memory corpora
• ruvector-coherence: cluster cohesion reuses the cosine-sim primitive
• ruvector-temporal-coherence: cohesion decay → ruFlo re-cluster trigger
• ruvector-mincut: mincut boundary detection could sharpen cluster seeds
• Edge/WASM: zero dependencies, 2% overhead, fits WASM heap at 10K–100K scale

---

Generated by Claude Code