Integrate LanceDB PoC alongside Turso

Parent Epic: #451

Overview

Integrate the existing LanceDB PoC from nodespace-data-store into the NodeSpace desktop application as a parallel, experimental database backend. This integration maintains full backward compatibility with Turso while enabling validation of LanceDB capabilities.

Self-Contained Scope

• Copy and adapt LanceDB PoC code into desktop app backend
• Wrap LanceDB operations in feature-flag controlled service layer
• Initialize both Turso and LanceDB connections at app startup
• Zero disruption to existing Turso implementation
• Environment variable controlled: EXPERIMENTAL_USE_LANCEDB=true

Implementation Approach

Phase 1: PoC Integration

• Create packages/desktop-app/src-tauri/src/datastore/lance/ module structure
• Copy core files from nodespace-data-store/src/
• Adapt connection initialization for Tauri environment
• Add LanceDB dependencies to Cargo.toml

Phase 2: Service Layer

• Extend NodeService with optional LanceDB connection
• Implement feature flag logic (EXPERIMENTAL_USE_LANCEDB env var)
• Ensure both datastores initialize successfully
• Add connection health checks for both backends

Phase 3: Verification

• Confirm LanceDB initializes without errors
• Verify Turso continues working unchanged
• Test feature flag toggle (on/off states)
• Document connection configuration

Acceptance Criteria

• LanceDB PoC code integrated into src-tauri/src/datastore/lance/
• LanceDB dependencies added to Cargo.toml
• NodeService extended with optional lance_db: Option<Arc<LanceDataStore>>
• Feature flag EXPERIMENTAL_USE_LANCEDB controls LanceDB initialization
• Both Turso and LanceDB initialize successfully when flag enabled
• Turso-only mode works unchanged when flag disabled
• Connection health checks pass for both backends
• Integration documented in code comments

Technical Specifications

Source Files to Integrate:

• nodespace-data-store/src/lance_data_store_simple.rs → Core LanceDB operations
• nodespace-data-store/src/lib.rs → Public API surface
• nodespace-data-store/Cargo.toml → Dependency references

Target Structure:

packages/desktop-app/src-tauri/src/
├── datastore/
│   ├── mod.rs (updated)
│   ├── turso/ (existing)
│   └── lance/ (NEW)
│       ├── mod.rs
│       ├── store.rs (adapted from lance_data_store_simple.rs)
│       └── types.rs
└── services/
    └── node_service.rs (updated)

Service Layer Pattern:

pub struct NodeService {
    turso_db: Arc<TursoConnection>,
    lance_db: Option<Arc<LanceDataStore>>,  // NEW - parallel
    use_lancedb: bool,  // Feature flag from env
}

impl NodeService {
    pub async fn new(app_handle: &AppHandle) -> Result<Self> {
        let turso_db = Arc::new(TursoConnection::new().await?);
        
        let (lance_db, use_lancedb) = if env::var("EXPERIMENTAL_USE_LANCEDB")
            .unwrap_or_default() == "true" {
            let lance = Arc::new(LanceDataStore::new().await?);
            (Some(lance), true)
        } else {
            (None, false)
        };
        
        Ok(Self { turso_db, lance_db, use_lancedb })
    }
}

Dependencies to Add (Cargo.toml):

[dependencies]
lancedb = "0.18"  # Or latest version from PoC
arrow = "54.0"    # Match PoC version
arrow-schema = "54.0"

Resources & References

Existing PoC:

• /Users/malibio/nodespace/nodespace-data-store/ - Complete working implementation
• /Users/malibio/nodespace/nodespace-data-store/README.md - PoC documentation
• /Users/malibio/nodespace/nodespace-data-store/SCHEMA.md - Schema documentation

NodeSpace Patterns:

• packages/desktop-app/src-tauri/src/services/node_service.rs - Service layer pattern
• packages/desktop-app/src-tauri/src/datastore/turso/ - Existing datastore structure

Epic Context:

• LanceDB Parallel Implementation for Database Evaluation #451 - Parent epic for LanceDB evaluation

Definition of Done

• LanceDB code integrated and compiles successfully
• Feature flag controls LanceDB initialization
• Both backends initialize without errors (flag enabled)
• Turso continues working unchanged (flag disabled)
• No breaking changes to existing Turso implementation
• Code follows Rust formatting standards (cargo fmt)
• Integration passes cargo clippy with zero warnings
• Documentation added explaining dual-backend architecture

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com