Skip to content

Create performance benchmark framework #456

Closed

Description

@malibio

Create performance benchmark framework

Parent Epic: #451

Dependencies

Overview

Build comprehensive benchmarking framework to compare LanceDB and Turso performance across all operations. Framework enables data-driven decision making by measuring real-world performance differences.

Self-Contained Scope

  • Benchmark harness for CRUD operations
  • Side-by-side LanceDB vs Turso comparison
  • Multiple dataset sizes (100, 1k, 10k, 100k nodes)
  • Operation categories: reads, writes, queries, structural operations
  • CSV/JSON export of benchmark results
  • Zero impact on production Turso operations

Implementation Approach

Phase 1: Benchmark Harness

  • Create benchmark runner infrastructure
  • Implement operation timing utilities
  • Setup dataset generation for various sizes
  • Implement result collection and aggregation

Phase 2: Operation Benchmarks

  • Benchmark single node CRUD operations
  • Benchmark batch operations (structural changes)
  • Benchmark query operations (filters, sorting)
  • Benchmark version operations (LanceDB time-travel)

Phase 3: Comparative Analysis

  • Run identical operations on both backends
  • Collect timing data and memory usage
  • Generate comparison reports
  • Visualize performance differences

Phase 4: Report Generation

  • Export results to CSV for analysis
  • Generate markdown summary reports
  • Create performance visualization charts
  • Document performance characteristics

Acceptance Criteria

  • Benchmark framework implemented and functional
  • All CRUD operations benchmarked
  • Dataset sizes tested: 100, 1k, 10k, 100k nodes
  • Side-by-side comparison for all operations
  • Results exportable to CSV and JSON formats
  • Markdown report generation working
  • Memory usage tracked alongside timing
  • Framework reusable for future benchmarking

Technical Specifications

Benchmark Framework Structure:

pub struct BenchmarkRunner {
    turso_db: Arc<TursoConnection>,
    lance_db: Arc<LanceDataStore>,
    dataset_sizes: Vec<usize>,
    results: Vec<BenchmarkResult>,
}

pub struct BenchmarkResult {
    operation: String,
    backend: String,  // "turso" or "lancedb"
    dataset_size: usize,
    duration_ms: f64,
    memory_mb: f64,
    throughput_ops_per_sec: f64,
    success: bool,
    error: Option<String>,
}

impl BenchmarkRunner {
    pub async fn run_all_benchmarks(&mut self) -> Result<()> {
        for size in &self.dataset_sizes {
            self.benchmark_create_operations(*size).await?;
            self.benchmark_read_operations(*size).await?;
            self.benchmark_update_operations(*size).await?;
            self.benchmark_delete_operations(*size).await?;
            self.benchmark_query_operations(*size).await?;
            self.benchmark_structural_operations(*size).await?;
        }
        Ok(())
    }
}

Operation Benchmark Pattern:

async fn benchmark_create_operations(&mut self, dataset_size: usize) -> Result<()> {
    let nodes = generate_test_nodes(dataset_size);
    
    // Benchmark Turso
    let turso_start = Instant::now();
    for node in &nodes {
        self.turso_db.create_node(node).await?;
    }
    let turso_duration = turso_start.elapsed();
    
    // Benchmark LanceDB
    let lance_start = Instant::now();
    for node in &nodes {
        self.lance_db.create_node(node).await?;
    }
    let lance_duration = lance_start.elapsed();
    
    // Record results
    self.results.push(BenchmarkResult {
        operation: "create".to_string(),
        backend: "turso".to_string(),
        dataset_size,
        duration_ms: turso_duration.as_secs_f64() * 1000.0,
        throughput_ops_per_sec: dataset_size as f64 / turso_duration.as_secs_f64(),
        ..default
    });
    
    // Same for LanceDB...
}

Query Benchmark Scenarios:

async fn benchmark_query_operations(&mut self, dataset_size: usize) -> Result<()> {
    let scenarios = vec![
        ("simple_filter", "type = 'task'"),
        ("nested_property", "json_extract(metadata, '$.status') = 'done'"),
        ("complex_filter", "type = 'task' AND json_extract(metadata, '$.priority') > 5"),
        ("full_text_search", "content LIKE '%important%'"),
    ];
    
    for (name, query) in scenarios {
        self.benchmark_single_query(name, query, dataset_size).await?;
    }
}

Report Generation:

pub struct BenchmarkReport {
    summary: String,
    detailed_results: Vec<BenchmarkResult>,
    comparisons: Vec<Comparison>,
}

pub struct Comparison {
    operation: String,
    dataset_size: usize,
    turso_ms: f64,
    lance_ms: f64,
    winner: String,
    speedup_factor: f64,
}

impl BenchmarkReport {
    pub fn to_markdown(&self) -> String {
        let mut md = String::new();
        md.push_str("# LanceDB vs Turso Performance Comparison\n\n");
        
        // Summary table
        md.push_str("## Performance Summary\n\n");
        md.push_str("| Operation | Dataset Size | Turso (ms) | LanceDB (ms) | Winner | Speedup |\n");
        md.push_str("|-----------|--------------|------------|--------------|--------|----------|\n");
        
        for comp in &self.comparisons {
            md.push_str(&format!(
                "| {} | {} | {:.2} | {:.2} | {} | {:.2}x |\n",
                comp.operation, comp.dataset_size, comp.turso_ms,
                comp.lance_ms, comp.winner, comp.speedup_factor
            ));
        }
        
        md
    }
    
    pub fn to_csv(&self) -> Result<String> {
        // CSV export implementation
    }
}

Tauri Command for UI Integration:

#[tauri::command]
async fn run_performance_benchmarks(
    state: State<'_, AppState>,
    dataset_sizes: Vec<usize>,
) -> Result<BenchmarkReport, String> {
    let mut runner = BenchmarkRunner::new(
        state.turso_db.clone(),
        state.lance_db.clone(),
        dataset_sizes,
    );
    
    runner.run_all_benchmarks().await
        .map_err(|e| e.to_string())?;
    
    let report = runner.generate_report();
    Ok(report)
}

Resources & References

Rust Benchmarking:

  • Criterion.rs patterns for micro-benchmarks
  • std::time::Instant for timing measurements

Related Issues:

Definition of Done

  • Benchmark framework implemented and tested
  • All operation categories benchmarked
  • Multiple dataset sizes tested and compared
  • Results exportable to CSV and JSON
  • Markdown reports generated automatically
  • Memory usage tracking functional
  • Framework documented with usage examples
  • Code passes cargo fmt and cargo clippy

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions