fix(conversation-memory): move vitest.config.ts to plugin directory to reduce resource usage - #462
Conversation
Design document for conversation-memory v3 based on analysis of claude-mem, SimpleMem, Mem0, LangMem, and Zep. Key changes: rule-based observations (no per-event LLM), flat text schema, legacy exchange system removal, token-budgeted injection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit implements Task 1 of the V3 redesign: creating the new database schema with a clean slate approach. **New Schema:** - pending_events: Temporary storage for tool events before LLM extraction - id (INTEGER PRIMARY KEY AUTOINCREMENT) - session_id, project, tool_name, compressed (TEXT NOT NULL) - timestamp, created_at (INTEGER NOT NULL) - observations: Long-term storage for extracted insights - id (INTEGER PRIMARY KEY AUTOINCREMENT) - title, content, project (TEXT NOT NULL) - session_id (TEXT, nullable) - timestamp, created_at (INTEGER NOT NULL) - vec_observations: Vector embeddings for semantic search - id (TEXT PRIMARY KEY) - embedding (FLOAT[768]) - observations_fts: Full-text search index - title, content (TEXT) **Implementation:** - initDatabaseV3(): Creates all tables with clean slate (deletes old DB) - insertPendingEventV3(): Stores compressed tool events - insertObservationV3(): Stores observations with optional embeddings - getPendingEventsV3(): Retrieves pending events by session - deleteOldPendingEventsV3(): Cleanup old events - searchObservationsV3(): Search with filters (project, session, date, FTS) - getObservationV3(): Get single observation by ID - deleteObservationV3(): Delete observation with cascade to vec/fts - getObservationCountV3(): Count observations by project - getPendingEventCountV3(): Count pending events by session **Testing:** - 36 comprehensive tests covering all functions - Tests verify schema creation, constraints, indexes - Tests cover CRUD operations, search, filters - Tests validate FTS and vector embedding storage **Breaking Changes:** - Old tables removed: exchanges, vec_exchanges, tool_calls, session_summaries - No migration from old schema (clean slate) - Existing database file will be deleted on first run with new schema Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ostToolUse hook Implement compressToolData function for compressing tool output/results into concise strings for storage in pending_events table. NO LLM calls - pure rule-based compression. Compression formats: - Read -> "Read /src/auth.ts (245 lines)" - Edit -> "Edited /src/auth.ts: old_string → new_string" - Write -> "Created /src/auth.ts (120 lines)" - Bash -> "Ran `npm test` → exit 0" or "Ran `npm test` → exit 1: error summary" - Grep -> "Searched 'pattern' in /src → 5 matches" - WebSearch -> "Searched: query text" - WebFetch -> "Fetched example.com" Skipped tools (return null): Glob, LSP, TodoWrite, TaskCreate, TaskUpdate, TaskList, TaskGet, AskUserQuestion, EnterPlanMode, ExitPlanMode, NotebookEdit, Skill Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rations - Add observations.v3.ts with simplified title + content schema - Implement create(), findById(), findByIds(), findByProject(), searchByVector(), deleteObservation() - Use db.v3.ts functions for database operations - Use sqlite-vec KNN queries for vector similarity search (WHERE embedding MATCH ? AND k = ?) - Generate embeddings using EmbeddingGemma (title + content) - Add comprehensive test suite (20 tests, all passing) This replaces the old 10+ field schema (type, title, subtitle, narrative, facts, concepts, files_read, files_modified, etc.) with just title + content for simplicity. Sources: - [KNN queries | sqlite-vec](https://alexgarcia.xyz/sqlite-vec/features/knn.html) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename deleteObservation to delete (reserved word workaround) - Remove createdAt from Observation interface (internal field only) - Make ObservationWithSimilarity internal (not exported) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix two TypeScript errors: - Rename deleteImpl to deleteObservation since 'delete' is reserved - Update test imports to use deleteObservation instead of 'delete' The 'delete' keyword is reserved in JavaScript/TypeScript and cannot be used as an identifier or export name. Changed function to deleteObservation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… hook This commit implements Task 5 of the v3 redesign: batch extraction prompt for the Stop hook. The new module provides efficient LLM-based observation extraction from compressed tool events. Key features: - buildBatchExtractPrompt: Creates structured prompts with 10-20 events - parseBatchExtractResponse: Robust JSON parsing with markdown handling - extractObservationsFromBatch: Main extraction function with graceful degradation - Previous observations context (last 3) for deduplication - Empty array support for low-value batches - Comprehensive test coverage (16 tests, all passing) Technical details: - Uses Gemini API via existing gemini-provider.ts - System prompt guides LLM to extract meaningful insights - Handles malformed responses gracefully (returns empty array) - Follows design doc requirements for title/content length limits The batch extraction approach reduces LLM costs significantly compared to per-event extraction while maintaining observation quality. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Simplified LLM configuration for Stop hook batch extraction:
- Removed round-robin provider wrapping (no longer needed for v3)
- Changed LLMConfig interface: single apiKey instead of provider/apiKeys array
- createProvider() now returns direct GeminiProvider instead of RoundRobinProvider
- Updated config schema documentation and examples
Configuration now uses simple format:
{
"apiKey": "your-gemini-api-key",
"model": "gemini-2.0-flash" // optional
}
The round-robin provider is scheduled for deletion in Task 12 (legacy removal).
Marked package is kept - still used by show.ts for MCP read tool.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ccess Implements Task 9: Add Layer 3 progressive disclosure for reading raw conversation transcripts. **Changes:** - Add src/core/read.ts with readConversation() and formatConversationAsMarkdown() * Reads from legacy DB (exchanges table) for backward compatibility * Falls back to JSONL file reading for V3 conversations * Supports startLine/endLine pagination with 1-indexed line numbers * Returns markdown formatted conversation transcript - Add comprehensive test suite (19 tests) in src/core/read.test.ts - Update MCP server to use new read module **Key Features:** - DB-first reading for legacy data (exchanges table) - JSONL file fallback for V3 conversations - Proper sidechain message grouping with markers - Tool use and result formatting - Token usage display - Metadata extraction (session ID, git branch, cwd, version) **Test Coverage:** - DB and JSONL reading paths - Pagination (startLine/endLine) - Sidechain handling - Tool use/results formatting - Empty/missing conversation handling Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove readConversationFromDb() function and update readConversation() to only read from JSONL files, as per V3 design spec. Changes: - Remove readConversationFromDb() function that read from exchanges table - Remove DbExchange interface - Update readConversation() to only try JSONL file reading - Update tests to remove DB-related test cases - Keep formatConversationAsMarkdown() for JSONL file reading - Keep pagination support (startLine/endLine) - Keep markdown formatting This aligns with V3 architecture where raw conversation data is stored in JSONL files, not in the database. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nversation This completes the removal of legacy DB code from read.ts: - Removed unused Database import (better-sqlite3) - Removed unused db parameter from readConversation function - Fixed fs import to use namespace import - Updated all test callers to remove db parameter - Updated MCP server to call readConversation without db All tests pass after these changes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add search.v3.ts with simplified search using only observations
- Remove exchange-based search logic (no exchanges, vec_exchanges tables)
- Remove multi-concept (array query) search path
- Support filtering by: projects, date range (after/before), mode (vector/text/both)
- Return compact observations: {id, title, project, timestamp}
- Implement recency boost for vector search results
- Handle FTS5 special characters in text search (e.g., dots in "passport.js")
Tests:
- Comprehensive test coverage for all search modes (vector, text, both)
- Test ISO date validation
- Test recency boost calculations
- Test filtering by project and date range
- Test result ordering and structure
- Test edge cases (empty DB, special chars, unicode)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove `similarity` field from CompactObservationResult interface
- Remove `applyRecencyBoost` function and related constants (BOOST_FACTOR, BOOST_MIDPOINT)
- Simplify result handling (no complex similarity sorting)
- Update tests to remove similarity/recency boost test cases
- Fix test assertion that referenced removed `content` field
Specification requires compact observations to return only: {id, title, project, timestamp}
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove recency boost functionality and similarity field from search results to align with V3 design simplification: - Remove BOOST_FACTOR and BOOST_MIDPOINT constants - Remove applyRecencyBoost function - Remove similarity field from CompactSearchResult and CompactObservationResult interfaces - Update searchConversations to not apply recency boost or sort by similarity - Update searchObservations to not include similarity scores - Update formatResults and formatObservationResults to not display similarity percentages - Update searchMultipleConcepts to use timestamp sorting instead of similarity - Update formatMultiConceptResults to not display similarity percentages - Update all related tests to reflect these changes This change simplifies the search implementation by removing complex recency boosting logic and similarity scoring, making results purely based on vector distance or text matching. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements Task 2 of the v3 redesign: PostToolUse hook that stores compressed tool data in pending_events table. Key features: - Uses compress.ts for rule-based compression (no LLM calls) - Skips low-value tools (Glob, LSP, Task*, etc.) - Stores in pending_events table with session_id, project, tool_name, compressed, timestamp - Async/non-blocking execution - Comprehensive test coverage (19 tests) This replaces the old observer-based approach with a simpler hook-based system that queues events for batch processing later. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The implementation was overly complex with: - CLI entry point with stdin/stdout handling - Database initialization logic - Enhanced error handling and logging - Complex project name fallback logic Simplified to core requirement: - Simple hook function that stores compressed tool events - Synchronous function (non-async for simpler hook call) - Takes explicit parameters (db, sessionId, project, toolName, toolData) - No CLI entry point (hooks.json will call this function) Updated tests to match simplified implementation: - All 18 tests pass Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement Task 3: Stop hook for batch LLM extraction from pending_events. Implementation: - Collects all pending_events for the session - Skips extraction if < 3 events (too short to be useful) - Groups events into batches of 10-20 events (default 15) - For each batch, calls Gemini with: - The compressed event data - Previous batch's all observations (for deduplication context) - Extraction prompt to produce title + content observations - LLM may return empty array for low-value batches - Stores extracted observations with embeddings - Runs async (non-blocking) - Handles errors gracefully (continues with next batch) Files: - hooks/stop.ts: Main implementation with handleStop function - hooks/stop.test.ts: Comprehensive test suite (13 tests, all passing) - core/db.v3.ts: Added getAllPendingEventsV3 for batch extraction Tests cover: - Minimum threshold (skip if < 3 events) - Batch creation and processing - Previous observations context carryover - Empty LLM response handling - Error handling and graceful degradation - Session isolation - Custom batch sizes - Integration with observations.v3.ts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add explicit type annotation for allEvents variable
- Add explicit type annotation for event parameter in map callback
- Both use Array<PendingEventV3 & { id: number }> type
This resolves TypeScript errors:
- Line 87-89: 'event' is of type 'unknown'
- Ensures type safety throughout the batch processing logic
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements Task 6: SessionStart hook that injects recent observations at session start within a token budget. Features: - Reads config (maxObservations, maxTokens, recencyDays, projectOnly) - Queries recent observations for the project - Formats as markdown with project header - Respects token budget (stops when maxTokens reached) - Returns formatted markdown for injection Output format: # [project-name] recent context (conversation-memory) - observation title: content - ... Implementation details: - Uses simple token approximation (chars / 4) - Filters by project when projectOnly is true - Filters by recencyDays to get recent observations - Returns empty result when no observations exist or budget exceeded - All 13 tests passing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement V3 3-tool architecture as per spec: - search: Single query string, returns compact observations - get_observations: Full details by ID array - read: Raw conversation from JSONL Progressive disclosure: - Layer 1: search() returns compact observations (~30t) - Layer 2: get_observations() returns full details (~200-500t) - Layer 3: read() returns raw conversation (~500-2000t) Changes: - Removed legacy multi-concept search - Removed response_format parameter (always JSON now) - Removed types, concepts parameters from search - Updated to use V3 database and search modules - Added comprehensive V3 tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Issues fixed: 1. CRITICAL: Add files parameter to search.v3.ts and implement filtering logic 2. Minor: Remove count field from search response (not in spec) 3. Minor: Fix tool descriptions to remove type/concepts filter mentions Changes: - search.v3.ts: Add files?: string[] to SearchOptions interface - search.v3.ts: Implement files filtering by checking if file paths appear in content - search.v3.ts: Add content field to SELECT queries for filtering - search.v3.test.ts: Add comprehensive tests for files parameter - server.ts: Pass files parameter from search handler to searchV3 - server.ts: Remove count field from search response JSON - server.ts: Fix tool description to remove type/concepts mentions All 88 tests passing for search.v3 and server.v3. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix mock functions to return proper Promise types - Fix sessionId null/undefined type mismatch - Add eslint-disable for unused 'files' variable The mock functions for initEmbeddings and generateEmbedding now properly return Promise<void> and Promise<number[]> respectively. Changed all mockGenerateEmbedding assignments to use async arrow functions to match the expected Promise<number[]> return type. Changed sessionId default from null to undefined to match the expected type signature. Added eslint-disable comment for 'files' variable which is used in the matchesFiles closure but TypeScript doesn't recognize this usage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove type annotations from vi.mock arrow functions to fix TypeScript syntax errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add comprehensive test to validate hooks.json structure and ensure all three required hooks are properly registered: - SessionStart hook for context injection - PostToolUse hook for tool event storage - Stop hook for session summarization This test verifies the hooks.json conforms to the hooks-schema.json and includes all required v3 hook registrations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit removes all legacy V2 architecture code as part of the V3 redesign. The new V3 architecture uses a simplified schema with only pending_events and observations tables, and eliminates the complex exchange-based indexing system. Deleted files: - Core: parser.ts, sync.ts, tool-compress.ts, observer.ts, observation-prompt.ts - LLM: round-robin-provider.ts (no longer needed with single Gemini provider) - CLI: inject-cli.ts, observe-cli.ts, observer-cli.ts, search-cli.ts, sync-cli.ts - Core: db.ts, inject.ts, observations.ts, search.ts, indexer.ts, session-summary.ts, verify.ts - Tests: All corresponding test files for deleted modules - MCP: Removed DB integration tests (V3 reads conversations directly from JSONL files) Updated files: - hooks/hooks.json: Removed sync and observer start commands from SessionStart - cli/index-cli.ts: Simplified to minimal CLI (most functionality via MCP) - llm/index.ts: Removed RoundRobinProvider export - llm/index.test.ts: Removed RoundRobinProvider tests - server.test.ts: Removed DB integration tests and multi-concept search tests - Test files: Updated to use in-memory database for isolation The V3 architecture is now cleaner and simpler: - 3 MCP tools: search, show, stats, read - 3 hooks: SessionStart, PostToolUse, Stop - 2 DB tables: pending_events, observations - Rule-based tool compression (no LLM for PostToolUse) - Batch LLM extraction at session end (Stop hook) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…o reduce resource usage Root cause: vitest.config.ts was in worktree root with include: ['**/*.test.ts'] causing vitest to scan and include 144 test files from node_modules/zod. - Move vitest.config.ts from root to plugins/conversation-memory/ - Update exclude pattern from ['node_modules'] to ['**/node_modules/**'] Performance improvements: - Collection time: 10.60s → 1.90s (5.6x faster) - Total duration: 1.53s → 0.64s (2.4x faster) - Max memory: 203MB → 170MB (16% reduction) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Caution Review failedThe pull request is closed. 📝 WalkthroughWalkthroughIntroduces a comprehensive v3 redesign of conversation-memory from an observer-polling daemon to a hook-based async batch processing architecture. Replaces legacy components with new modules: v3 database schema (pending_events, observations, vector embeddings, FTS), compression utilities, LLM batch extraction, three hooks (PostToolUse, Stop, SessionStart), and a simplified MCP server with three tools (search, get_observations, read). Removes extensive legacy code (observer, indexer, sync, search, parser, and associated CLI commands and tests). Changes
Sequence Diagram(s)sequenceDiagram
participant User as User/Claude
participant PostTool as PostToolUse Hook
participant DB as Database<br/>(pending_events)
participant Stop as Stop Hook
participant LLM as LLM Provider
participant Obs as Database<br/>(observations)
participant Start as SessionStart Hook
participant Context as Context Injection
User->>PostTool: Tool invocation completes
PostTool->>PostTool: compressToolData()
PostTool->>DB: insertPendingEventV3
DB-->>PostTool: event stored
rect rgba(100, 150, 200, 0.5)
Note over Stop,Obs: End of session
Stop->>DB: getPendingEventsV3(sessionId)
DB-->>Stop: pending events list
Stop->>Stop: createBatches (10-20 events)
loop For each batch
Stop->>Stop: Build batch prompt w/ prev observations
Stop->>LLM: extractObservationsFromBatch()
LLM-->>Stop: extracted observations
Stop->>Obs: insertObservationV3 (with embedding)
Obs-->>Stop: observation stored
Stop->>Stop: Accumulate for next batch context
end
end
rect rgba(100, 200, 150, 0.5)
Note over Start,Context: New session starts
Start->>Obs: searchObservationsV3(recent)
Obs-->>Start: observations (last 7 days)
Start->>Start: Apply token budget
Start->>Start: formatObservation() to Markdown
Start->>Context: Inject markdown context
Context-->>User: Context available in session
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary
**/node_modules/**Problem
Vitest was consuming excessive resources when running tests:
Root Cause
include: ['**/*.test.ts']exclude: ['node_modules']only excluded root node_modules, not plugin's node_modulesSolution
['**/node_modules/**']Performance Improvement
Test plan
🤖 Generated with Claude Code
Summary by CodeRabbit
Release Notes
New Features
Database Changes
Refactoring