Plan: ApexStore rollout as primary storage (caching + diff + structured data)

[ElioNeto]

Summary

Plan and execute the phased rollout of ApexStore (packages/teamcode/src/storage/apex-store/) as the primary storage engine, replacing / complementing the current dual system (SQLite via Drizzle ORM + JSON filesystem).

Context

• ApexStore is an LSM-Tree KV engine written in Rust, running as a sidecar HTTP server (default port 8080)
• A pre-built binary lives at packages/teamcode/bin/apexstore-server (~11.6 MB)
• The TypeScript client (client.ts) and sidecar manager (sidecar.ts) exist but are completely unused — zero imports from the rest of the codebase
• enabled defaults to false; init() silently swallows errors with console.warn
• Source repo: github.com/ElioNeto/ApexStore

Current Storage Architecture

Tier	Technology	Data	Limitations
SQLite (Drizzle ORM)	bun:sqlite	Sessions, Messages, Parts, Todos, Permissions, Shares, Events (~6 tables)	Single-file lock contention; no caching layer; WAL overhead for small KV lookups
JSON filesystem (Storage service)	Flat .json files	Session diffs (session_diff/{id}.json), legacy data	Filesystem scan for listing; no atomic writes; no compression

Why ApexStore

• LSM-Tree engine (Rust) — optimized for write-heavy KV workloads and prefix scans
• Block-level LZ4 prefix compression — ideal for repetitive LLM prompt templates and context blobs
• Configurable memtable/block cache — tune memory vs throughput per workload (defaults: 4 MB memtable, 64 MB block cache)
• WAL durability — crash recovery built-in; better than raw JSON files
• REST API via sidecar — decoupled process with independent lifecycle; crash does not take down the main app

Phased Rollout Plan

Phase 0 — Foundation (Est. 1-2 weeks)

Goal: Make ApexStore usable from Effect services with proper lifecycle

• Create Effect service layer (ApexStore.Service) with Layer.effect
  • Wrap existing ApexStoreClient (plain async/await) into Effect
  • Handle sidecar lifecycle via Effect.acquireRelease
  • Integrate with RuntimeFlags for per-instance enable/disable
  • Circuit breaker: if sidecar is down, fall back to current storage transparently
• Add ApexStore config to the config system (src/config/)
  • apexStore.enabled, apexStore.port, apexStore.memtableSize, apexStore.blockCacheSizeMb
  • Default: disabled (opt-in during rollout)
• Initialize during bootstrap (src/project/bootstrap.ts)
  • Start sidecar alongside other services when enabled
  • Graceful shutdown on SIGTERM/SIGINT
  • Dynamic port allocation (same pattern as Critical: Second application instance freezes when Go core is enabled (port conflict on :43001) #1077) to avoid port conflicts
• Write Effect integration tests
  • Real sidecar binary; test set/get/delete/list round-trips
  • Test failure modes: binary missing, port conflict, crash recovery
  • Test concurrent operations (existing test does 50 concurrent ops)

Phase 1 — Cache Layer (Est. 1 week)

Goal: Use ApexStore as a hot cache in front of SQLite/JSON (no migration risk)

• Cache LLM prompt assembly (src/session/prompt.ts)
  • Cache compiled system prompts + context templates
  • Key: prompt:{agent}:{hash(template)} → compiled prompt JSON
  • TTL-based invalidation (5 min) — templates rarely change within a session
  • Expected: 40-60% reduction in prompt assembly time
• Cache provider model catalog (src/provider/provider.ts)
  • Cache models.dev API response to reduce startup latency
  • Currently has 60-min TTL in memory but lost on restart
  • ApexStore persistence makes cache survive restarts
• Cache resolved config (src/config/config.ts)
  • Cache parsed opencode.json[c] / teamcode.json[c] by directory
  • Avoid re-parsing on every session start or worktree switch

Phase 2 — Diff Storage (Est. 2 weeks)

Goal: Replace JSON filesystem for session diffs

• Implement Storage adapter wrapping ApexStoreClient to implement Storage.Interface
  • Key schema: diff:{sessionId}:{snapshotHash} → compressed JSON blob
  • All existing Storage operations (read, write, list, delete) map to KV equivalents
• Shadow mode — write to BOTH JSON filesystem and ApexStore; read from ApexStore
  • Log divergence between the two backends
  • Feature flag: FLAG_apexstore_diff_shadow=true
• Shadow routing infrastructure already exists (packages/core/src/router/flag.ts) — reuse the same pattern
• Migration script — one-time bulk copy of existing JSON diffs into ApexStore
  • Run on first startup with ApexStore enabled
  • Keep JSON files as fallback during transition period
• Cutover — make ApexStore the primary diff backend
  • Remove JSON filesystem path after monitoring period (e.g. 1 week without divergence)

Phase 3 — Structured Data Evaluation (Est. 3-4 weeks)

Goal: Evaluate ApexStore as SQLite alternative for session/message metadata

This phase is conditional — only proceed if benchmarks justify it.

• Benchmark — compare read/write latency, memory, throughput
  • ApexStore (LSM-Tree + HTTP) vs SQLite (in-process + memory-mapped)
  • Workloads: session list by directory (prefix scan), message cursor pagination, tool call history write-heavy
• Design KV schema for structured data
  • session:{id} → full session metadata JSON blob
  • msg:{sessionId}:{createdAt}:{id} → message + parts blob
  • idx:session:byDirectory:{dir} → set of session IDs (secondary index)
  • idx:msg:bySession:{sessionId} → ordered set of message IDs
• Build secondary index layer — either in ApexStore or application-managed
  • Must support: list by directory, order by createdAt, cursor pagination
• Dual-write + compare — write to both SQLite and ApexStore
  • Shadow-read from ApexStore; compare results; log discrepancies
• Decision gate — only proceed to cutover if ApexStore matches/exceeds SQLite
  • If ApexStore is slower, keep SQLite + ApexStore cache only (Phases 1 + 2 are still valuable independently)

Risks & Mitigations

Risk	Likelihood	Impact	Mitigation
Sidecar process crashes	Low	Data loss (unflushed writes)	WAL durability; health check every 3s; auto-restart with exponential backoff; circuit breaker to SQLite fallback
Port conflict with other instances	Medium	Startup failure	Dynamic port allocation (port 0 → OS assigns free port); same pattern as Go core fix (#1077)
Rust binary not bundled	Low	Feature gracefully disabled	Check binary exists in resolveBinary(); clear error message suggesting cargo build
LSM-Tree read amplification	Medium	Higher random-read latency	Tune BLOCK_CACHE_SIZE_MB (default 64MB); bloom filters already enabled (BLOOM_FALSE_POSITIVE_RATE=0.01)
Migration data loss	Low	Session diffs unrecoverable	Keep JSON files until verified; shadow mode with divergence logging
ApexStore repo maintenance	Low	Unsupported upstream	Binary is self-contained; can fork/build in CI; documented build process
Rust compilation in CI	Medium	CI pipeline slowdown	Cache target/ directory; pre-build binary in base image

Success Criteria

1. Phase 1: Prompt assembly time reduced by ≥40% (measure via performance.now() or Effect metrics)
2. Phase 2: Diff read/write p95 latency ≤ current JSON filesystem baseline
3. Phase 3 (if pursued): ApexStore throughput within 20% of SQLite for session/message workloads
4. All phases: Zero data loss during migration; automatic fallback on failure; no user-visible changes

Files Reference

File	Purpose	Status
packages/teamcode/src/storage/apex-store/index.ts	Module entry, lifecycle, cache convenience API	Unused
packages/teamcode/src/storage/apex-store/client.ts	HTTP REST client for Rust sidecar	Unused
packages/teamcode/src/storage/apex-store/sidecar.ts	Sidecar process spawner + health check	Unused
packages/teamcode/src/storage/apex-store/sidecar.test.ts	Integration tests (7 tests, skipped if binary missing)	Partial
packages/teamcode/src/storage/storage.ts	Current JSON filesystem storage (333 lines)	Active
packages/teamcode/src/session/session.ts	SQLite+Drizzle session storage	Active
packages/teamcode/bin/apexstore-server	Pre-built Rust binary (11.6 MB)	Pre-built
packages/core/src/router/flag.ts	Shadow mode infrastructure (reusable pattern)	Active

Dependencies

• Rust toolchain (for building ApexStore from source in dev)
• actix-web + actix-web-httpauth (ApexStore's HTTP framework)
• No external npm packages — everything is HTTP calls + child process