Skip to content

LanceDB Parallel Implementation for Database Evaluation #451

Closed
Closed
LanceDB Parallel Implementation for Database Evaluation#451
Assignees
malibio
Labels
backenddatabaseepicexperimentalfoundation

Description

@malibio
malibio
opened on Nov 10, 2025

LanceDB Parallel Implementation for Database Evaluation

Overview

Implement LanceDB database operations in parallel with existing Turso implementation to validate feasibility of migration. This evaluation focuses on critical capabilities: automatic versioning, JSON scalar indexes for dynamic schema support, and atomic batch operations. The parallel approach enables risk-free validation while maintaining production stability.

Self-Contained Scope

  • Parallel LanceDB implementation alongside existing Turso (zero disruption)
  • Experimental Tauri command endpoints with _lance_experimental suffix
  • JSON scalar index validation (Option B3 from architecture analysis)
  • Feature flag controlled: EXPERIMENTAL_USE_LANCEDB environment variable
  • Comprehensive evaluation framework with side-by-side comparison
  • Data-driven recommendation based on real testing

Implementation Approach

Phase 1: Foundation

  • Integrate existing LanceDB PoC (nodespace-data-store) into desktop app
  • Implement parallel CRUD operations with experimental endpoints
  • Backend batch coordinator for atomic operations

Phase 2: JSON Index Validation

  • Validate JSON scalar index creation and performance
  • Test multi-level nested property filtering (address.street.zip)
  • Test filtered/partial indexes scoped to node types
  • Document sparse property behavior

Phase 3: Evaluation

  • Performance benchmarking framework (LanceDB vs Turso comparison)
  • Feature parity validation (all Turso operations work in LanceDB)
  • Developer testing UI for side-by-side exploration

Phase 4: Decision

  • Compile evaluation report with performance data
  • Migration recommendation with risk assessment
  • Path forward (commit to LanceDB or optimize Turso)

Sub-Issues

Acceptance Criteria

  • LanceDB PoC integrated and running alongside Turso
  • All CRUD operations work via experimental endpoints
  • JSON scalar indexes validated with real data
  • Performance benchmarks completed
  • Feature parity confirmed
  • Developer testing UI functional
  • Evaluation report delivered with clear recommendation
  • Zero disruption to existing Turso implementation

Technical Implementation Notes

Architecture Decision Context:

  • Primary requirement: Automatic versioning/time-travel (LanceDB provides built-in)
  • Critical need: Dynamic schema support (users add properties at runtime)
  • Key validation: JSON scalar indexes (GitHub issue #4516, PRs #4566, #4577)
  • Risk mitigation: Parallel implementation enables data-driven decision

Key Technologies:

  • LanceDB (columnar vector database with automatic versioning)
  • Apache Arrow (columnar in-memory format)
  • JSON scalar indexes (late 2024 feature for nested property filtering)
  • Backend batch coordinator (atomic operations creating single version)

Feature Flag Design:

pub struct NodeService {
    turso_db: Arc<TursoConnection>,
    lance_db: Option<Arc<LanceDataStore>>,
    use_lancedb: bool,  // EXPERIMENTAL_USE_LANCEDB env var
}

Experimental Endpoint Pattern:

#[tauri::command]
async fn create_node_lance_experimental(node: Node) -> Result<String>;

#[tauri::command]
async fn compare_backends(operation: String, data: Value) -> Result<ComparisonResult>;

Resources & References

Architecture Documents:

  • /Users/malibio/Desktop/nodespace-lancedb-vs-tursodb.md - Original decision rationale
  • /Users/malibio/Downloads/nodespace-persistence-architecture.md - Atomicity analysis
  • /Users/malibio/nodespace/nodespace-data-store/ - Existing LanceDB PoC

LanceDB Features:

  • JSON scalar indexes: GitHub issue #4516, PRs #4566, #4577
  • Native JSON type with json_extract() support
  • Automatic versioning via append-only architecture

NodeSpace Patterns:

  • docs/architecture/development/process/epic-and-subissue-workflow.md
  • Single epic branch: feature/issue-451-lancedb-evaluation
  • All sub-issues work on same branch

Definition of Done

  • All 8 sub-issues completed and closed
  • LanceDB capabilities validated with real data
  • Performance comparison data collected
  • Clear recommendation delivered (migrate or optimize Turso)
  • If recommending migration: migration plan documented
  • If recommending Turso: optimization strategy documented
  • Zero regressions in existing Turso implementation

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

Activity

  1. malibio
    added
    foundation
    epic
    backend
    database
    experimental
    on Nov 10, 2025

  2. malibio commented on Nov 10, 2025

    @malibio
    malibio
    on Nov 10, 2025
    CollaboratorAuthor

    Test Baseline (2025-11-10)

    ✅ All tests passing: 769 passed, 31 skipped

    • Unit tests: All passing
    • Integration tests: All passing
    • Performance tests: All passing
    • Browser tests: All passing

    Test execution time: ~9 seconds (in-memory mode)

    No test failures to track. Starting implementation with clean baseline.

  3. malibio
    self-assigned this
    on Nov 10, 2025
  4. malibio
    added a commit that references this issue on Nov 10, 2025

    WIP: LanceDB integration and parallel CRUD operations complete (#452, #…

    a59ab16

  5. malibio commented on Nov 10, 2025

    @malibio
    malibio
    on Nov 10, 2025
    CollaboratorAuthor

    Progress Update - Phase 1 Complete ✅

    Completed Sub-Issues:

    Key Achievements:

    1. Dependency Resolution

      • Upgraded to LanceDB 0.20 with arrow 55
      • Resolves half version conflict (now compatible with Rust 1.91.0)
      • Full compilation success with zero warnings

    2. Integration Architecture

      • Both backends (Turso + LanceDB) run in parallel
      • Feature flag control via EXPERIMENTAL_USE_LANCEDB environment variable
      • Graceful degradation if LanceDB initialization fails
      • LanceDB path auto-determined from Turso DB location

    3. Experimental Commands

      • create_node_lance_experimental
      • read_node_lance_experimental
      • update_node_lance_experimental
      • delete_node_lance_experimental
      • compare_backends - Primary validation tool for side-by-side comparison

    4. Test Status

      • ✅ All frontend tests passing (85 files, 1538 tests)
      • ✅ All Rust tests passing (494 tests)
      • ✅ Code quality checks passing (cargo fmt + clippy)

    Architecture Highlights:

    • Zero changes to existing Turso implementation
    • ExperimentalState managed via Tauri state system
    • Error conversion properly maps LanceDB → NodeServiceError
    • Complete CRUD parity between backends

    Next Phase:
    Ready to proceed with JSON scalar index validation (#454) and benchmark framework (#455).

    Commit: a59ab16

  6. malibio
    added a commit that references this issue on Nov 10, 2025

    WIP: JSON scalar index validation and performance benchmarking comple…

    fd113e9

  7. malibio commented on Nov 10, 2025

    @malibio
    malibio
    on Nov 10, 2025
    CollaboratorAuthor

    [Phase 2 Retest] LanceDB v0.22.3 Evaluation - CRITICAL FINDINGS

    Upgraded from v0.20 → v0.22.3 to test if JSON querying capabilities improved.

    Result: ❌ JSON Querying BROKEN - Situation WORSE than v0.20

    Critical Discovery

    LanceDB v0.22.3 has BROKEN the json_extract() functionality that worked in v0.20:

    Phase 2 (v0.20) Status:

    • ✅ json_extract(properties, '$.field') worked with Utf8 string storage
    • ✅ Basic JSON querying functional (though not indexed)

    v0.22.3 Regression:

    • ❌ json_extract() signature changed: requires LargeBinary (JSONB format)
    • ❌ Cannot cast Utf8 to LargeBinary in SQL - no migration path
    • ❌ Requires JSONB encoder/decoder implementation
    • ❌ Breaking API changes: 'text' type removed, must use 'string'
    • ❌ All Phase 2 limitations PLUS new blockers

    Technical Details

    JSONB Format Requirement:
    LanceDB v0.22.3 expects JSONB binary format (like PostgreSQL), not UTF-8 JSON strings.

    Error Encountered:

    Failed to select values from path '$.status': InvalidJsonb

    Migration Attempt:

    • Changed schema: DataType::Utf8 → DataType::LargeBinary
    • Updated arrays: StringArray → LargeBinaryArray
    • Result: Compilation succeeded, but runtime InvalidJsonb errors

    To Support v0.22.3 Would Require:

    1. Implement JSONB encoder (convert JSON → JSONB binary)
    2. Implement JSONB decoder (convert JSONB → JSON)
    3. Update all storage operations
    4. Estimated effort: 2-3 days development + testing
    5. Still no scalar indexes after all this work

    Test Results

    All v0.22.3 tests FAILED:

    • ❌ Single-level property filtering
    • ❌ Multi-level nested property filtering
    • ❌ Deep nesting (5 levels)
    • ❌ Performance benchmarks
    • ❌ Scalar index creation

    Updated Recommendation

    🔴 STRONGLY DO NOT MIGRATE to LanceDB v0.22.3

    Phase 2 "DO NOT MIGRATE" recommendation now STRENGTHENED:

    Aspect v0.20 (Phase 2) v0.22.3 (Current)
    JSON Querying ✅ Works ❌ Broken
    Scalar Indexes ❌ Not supported ❌ Not supported
    Query Performance ⚠️ Full scan ❌ Can't query
    Migration Effort Medium Very High
    Recommendation Do Not Migrate STRONGLY Do Not Migrate

    Key Takeaways

    1. API Instability: LanceDB is undergoing frequent breaking changes without backward compatibility
    2. Lost Functionality: v0.22.3 removed working features from v0.20
    3. No Benefits: Still no scalar indexes on JSON fields after upgrade
    4. High Migration Cost: Would require custom JSONB implementation

    Turso Advantages (Reinforced)

    • ✅ Native JSON/JSONB support with SQLite's json_extract()
    • ✅ Scalar indexes work on nested JSON fields
    • ✅ Mature, stable API with backward compatibility
    • ✅ No breaking changes between versions
    • ✅ Excellent query performance with indexes

    Files Modified

    • Cargo.toml: lancedb 0.22.3, arrow 56.0
    • src/datastore/lance/store.rs: Schema migration attempt (incomplete)
    • src/datastore/lance/tests/json_index_tests.rs: Syntax updates
    • src/datastore/lance/tests/json_index_v22_retest.rs: New test suite
    • docs/json_scalar_index_validation_v0.22.md: Comprehensive findings report

    Commit: 8034a9d

    Next Steps

    Main agent should proceed with final evaluation report (#459) using these strengthened findings. The Phase 2 blocker not only persists but has worsened in v0.22.3.

  8. malibio
    added a commit that references this issue on Nov 10, 2025

    CRITICAL: LanceDB v0.22.3 JSON querying fundamentally broken

    0a51ad2
  9. malibio
    closed this as completedon Nov 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

backenddatabaseepicexperimentalfoundation

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions