Skip to content

About

No description, website, or topics provided.

Resources

Stars

22 stars

Watchers

1 watching

Forks

Repository files navigation

SpatialBench

Open source benchmarks for multimodal AI reasoning.

SpatialBench is designed to evaluate the next generation of multimodal AI models on their ability to reason about space, structure, and pathing. We test models not just on what they know, but on how well they can "see" and manipulate abstract concepts in 2D and 3D space.

🎯 Features

  • 2D Path Tracing: Test visual path-following capabilities with complex Bezier curve networks
  • 3D Mental Rotation: Shepard-Metzler style mental rotation tasks for spatial reasoning
  • Interactive Practice: Try the challenges yourself and compare against AI models
  • Live Leaderboard: Track the performance of state-of-the-art multimodal models

🚀 Live Demo

Visit the live benchmark at: https://spicylemonade.github.io/spatialbench/

📊 Current Results

Using the Probability Equivalence Method:

  • GPT-6 Astra (Max Reasoning + Code Interpreter): 91.78% (23/25 2D, 24/25 3D; 47/50 raw)
  • Human Baseline: 80.0%
  • GPT-6 Astra (Max Reasoning, No Tools): 67.07% (24/25 2D, 16/25 3D; 40/50 raw)
  • Qwen3-VL-235B-A22B-Instruct: 13.5%
  • GPT-5.5 (XHigh Reasoning): 13.2%
  • Qwen-2.5-VL-72B-Instruct: 12.9%
  • Claude Opus 4.7: 11.5%
  • Gemini 3.0 Pro Preview: 9.55%
  • GPT-5.4 (XHigh Reasoning): 8.29%
  • GPT-5.1 (High Reasoning): 7.51%
  • Random Guessing: 5.0%

GPT-6 Astra scored 67.07% without tools and 91.78% with Code Interpreter, making the tool-enabled run the first tested AI result on this leaderboard to exceed the 80.0% human baseline under probability-equivalence scoring. Most earlier models remain near the random band. Because the benchmark has only 25 questions per section, the nominal 5.0% random baseline has a wide exact 95% single-run interval of about 0.5% to 12.3%.

🛠️ Technology Stack

  • Frontend: React + Vite
  • 3D Rendering: Three.js
  • 2D Graphics: SVG
  • Styling: Tailwind CSS
  • Deployment: GitHub Pages

🏃 Local Development

# Install dependencies
npm install

# Run development server
npm run dev

# Build for production
npm run build

📝 Methodology

Models are evaluated on 50 total problems:

  • 25 3D mental rotation tasks (Shepard-Metzler style with 4 multiple-choice options)
  • 25 2D path tracing tasks (with 20 possible numeric endpoints)

Probability Equivalence Scoring

Since 3D questions have a 25% random guess rate (1/4) while 2D questions have only 5% (1/20), we normalize the difficulty:

  1. Calculate raw scores for each section independently
  2. Adjust 3D score: Raise to the power of 2.16 to match 2D difficulty
    • Exponent derived from: 0.25^n = 0.05, solving n ≈ 2.16
  3. Final score: Average of raw 2D score and adjusted 3D score

Formula: Final = (S_2D + S_3D^2.16) / 2

This gives a nominal random-guess baseline of 5.0% and prevents "stat-padding" on easier questions. In a finite 50-question run, however, random guessing has non-trivial variance; the exact 95% random interval is about 0.5% to 12.3%.

Methodology developed by Alejandro Zarzuelo

🤝 Contributing

Contributions are welcome! Feel free to:

  • Submit issues for bugs or feature requests
  • Add new spatial reasoning tasks
  • Improve the UI/UX
  • Test additional AI models

📄 License

MIT License

📬 Contact

Follow @spicey_lemonade on Twitter/X


Why spatial reasoning matters: Just as humans use visual tracing and mental rotation for complex tasks like circuit analysis, CAD engineering, and molecular biology, future AI systems must possess these intrinsic capabilities to fully automate physical world reasoning.

About

No description, website, or topics provided.

Resources

Stars

22 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages