Repository navigation
Create performance benchmark framework #456
Copy link
Copy link
Closed
Description
Create performance benchmark framework
Parent Epic: #451
Dependencies
- Implement parallel CRUD with experimental endpoints #453 (blocks this issue)
Overview
Build comprehensive benchmarking framework to compare LanceDB and Turso performance across all operations. Framework enables data-driven decision making by measuring real-world performance differences.
Self-Contained Scope
- Benchmark harness for CRUD operations
- Side-by-side LanceDB vs Turso comparison
- Multiple dataset sizes (100, 1k, 10k, 100k nodes)
- Operation categories: reads, writes, queries, structural operations
- CSV/JSON export of benchmark results
- Zero impact on production Turso operations
Implementation Approach
Phase 1: Benchmark Harness
- Create benchmark runner infrastructure
- Implement operation timing utilities
- Setup dataset generation for various sizes
- Implement result collection and aggregation
Phase 2: Operation Benchmarks
- Benchmark single node CRUD operations
- Benchmark batch operations (structural changes)
- Benchmark query operations (filters, sorting)
- Benchmark version operations (LanceDB time-travel)
Phase 3: Comparative Analysis
- Run identical operations on both backends
- Collect timing data and memory usage
- Generate comparison reports
- Visualize performance differences
Phase 4: Report Generation
- Export results to CSV for analysis
- Generate markdown summary reports
- Create performance visualization charts
- Document performance characteristics
Acceptance Criteria
- Benchmark framework implemented and functional
- All CRUD operations benchmarked
- Dataset sizes tested: 100, 1k, 10k, 100k nodes
- Side-by-side comparison for all operations
- Results exportable to CSV and JSON formats
- Markdown report generation working
- Memory usage tracked alongside timing
- Framework reusable for future benchmarking
Technical Specifications
Benchmark Framework Structure:
pub struct BenchmarkRunner {
turso_db: Arc<TursoConnection>,
lance_db: Arc<LanceDataStore>,
dataset_sizes: Vec<usize>,
results: Vec<BenchmarkResult>,
}
pub struct BenchmarkResult {
operation: String,
backend: String, // "turso" or "lancedb"
dataset_size: usize,
duration_ms: f64,
memory_mb: f64,
throughput_ops_per_sec: f64,
success: bool,
error: Option<String>,
}
impl BenchmarkRunner {
pub async fn run_all_benchmarks(&mut self) -> Result<()> {
for size in &self.dataset_sizes {
self.benchmark_create_operations(*size).await?;
self.benchmark_read_operations(*size).await?;
self.benchmark_update_operations(*size).await?;
self.benchmark_delete_operations(*size).await?;
self.benchmark_query_operations(*size).await?;
self.benchmark_structural_operations(*size).await?;
}
Ok(())
}
}Operation Benchmark Pattern:
async fn benchmark_create_operations(&mut self, dataset_size: usize) -> Result<()> {
let nodes = generate_test_nodes(dataset_size);
// Benchmark Turso
let turso_start = Instant::now();
for node in &nodes {
self.turso_db.create_node(node).await?;
}
let turso_duration = turso_start.elapsed();
// Benchmark LanceDB
let lance_start = Instant::now();
for node in &nodes {
self.lance_db.create_node(node).await?;
}
let lance_duration = lance_start.elapsed();
// Record results
self.results.push(BenchmarkResult {
operation: "create".to_string(),
backend: "turso".to_string(),
dataset_size,
duration_ms: turso_duration.as_secs_f64() * 1000.0,
throughput_ops_per_sec: dataset_size as f64 / turso_duration.as_secs_f64(),
..default
});
// Same for LanceDB...
}Query Benchmark Scenarios:
async fn benchmark_query_operations(&mut self, dataset_size: usize) -> Result<()> {
let scenarios = vec![
("simple_filter", "type = 'task'"),
("nested_property", "json_extract(metadata, '$.status') = 'done'"),
("complex_filter", "type = 'task' AND json_extract(metadata, '$.priority') > 5"),
("full_text_search", "content LIKE '%important%'"),
];
for (name, query) in scenarios {
self.benchmark_single_query(name, query, dataset_size).await?;
}
}Report Generation:
pub struct BenchmarkReport {
summary: String,
detailed_results: Vec<BenchmarkResult>,
comparisons: Vec<Comparison>,
}
pub struct Comparison {
operation: String,
dataset_size: usize,
turso_ms: f64,
lance_ms: f64,
winner: String,
speedup_factor: f64,
}
impl BenchmarkReport {
pub fn to_markdown(&self) -> String {
let mut md = String::new();
md.push_str("# LanceDB vs Turso Performance Comparison\n\n");
// Summary table
md.push_str("## Performance Summary\n\n");
md.push_str("| Operation | Dataset Size | Turso (ms) | LanceDB (ms) | Winner | Speedup |\n");
md.push_str("|-----------|--------------|------------|--------------|--------|----------|\n");
for comp in &self.comparisons {
md.push_str(&format!(
"| {} | {} | {:.2} | {:.2} | {} | {:.2}x |\n",
comp.operation, comp.dataset_size, comp.turso_ms,
comp.lance_ms, comp.winner, comp.speedup_factor
));
}
md
}
pub fn to_csv(&self) -> Result<String> {
// CSV export implementation
}
}Tauri Command for UI Integration:
#[tauri::command]
async fn run_performance_benchmarks(
state: State<'_, AppState>,
dataset_sizes: Vec<usize>,
) -> Result<BenchmarkReport, String> {
let mut runner = BenchmarkRunner::new(
state.turso_db.clone(),
state.lance_db.clone(),
dataset_sizes,
);
runner.run_all_benchmarks().await
.map_err(|e| e.to_string())?;
let report = runner.generate_report();
Ok(report)
}Resources & References
Rust Benchmarking:
- Criterion.rs patterns for micro-benchmarks
std::time::Instantfor timing measurements
Related Issues:
- LanceDB Parallel Implementation for Database Evaluation #451 (parent)
- Implement parallel CRUD with experimental endpoints #453 (blocks this issue)
Definition of Done
- Benchmark framework implemented and tested
- All operation categories benchmarked
- Multiple dataset sizes tested and compared
- Results exportable to CSV and JSON
- Markdown reports generated automatically
- Memory usage tracking functional
- Framework documented with usage examples
- Code passes
cargo fmtandcargo clippy
🤖 Generated with Claude Code
Co-Authored-By: Claude noreply@anthropic.com
Reactions are currently unavailable
Activity
WIP: LanceDB integration and parallel CRUD operations complete (#452, #…
WIP: JSON scalar index validation and performance benchmarking comple…