CadEval
Code · 2025-04-23
Will Patrick's CadEval text-to-CAD benchmark: a model writes a one-shot OpenSCAD script for each of 25 text-only mechanical-part descriptions (1 to 5 modelling operations each). The rendered STL is ICP-aligned to a Fusion 360 reference and a task passes only if it is watertight, a single component, within 1 mm on bounding box, within 2% on volume, and within 1 mm on Chamfer and 95th-percentile Hausdorff distance. Scores are the percentage of attempts passing; runs use one or two replicates per task. Distinct from CadQueryEval, which reuses the task descriptions with CadQuery code.
Top models (higher is better)
| Model | Score |
|---|---|
| o3 | 74.0 |
| Gemini 2.5 Pro Preview 03-25 | 64.0 |
| O4 Mini | 62.0 |
| o1 (2024-12-17) | 56.0 |
| Claude 3.7 Sonnet | 54.0 |
| o3-mini | 54.0 |
| Claude 3.5 Sonnet (Oct 2024) | 48.0 |
| DeepSeek-R1 | 48.0 |
| GPT-4.1 | 42.0 |
| Gemini 1.5 Pro 002 | 34.0 |
| Haiku 3.5 | 32.0 |
| Qwen3 32B | 32.0 |
| Gemini 2.0 Flash | 30.0 |
| GPT-4o (2024-08-06) | 26.0 |
| GPT-4.1 Mini | 16.0 |
| Qwen3 8B | 16.0 |
| Claude 3 Haiku | 12.0 |