Ruby LLM Benchmarks — Program Fixer
Code · 2025-08-01
Ruby LLM Benchmarks Program Fixer evaluates one-shot repairs of four broken Ruby programs: a calendar, parking garage, school library, and vending machine. Each task score is 90% test-suite success plus 10% RuboCop quality; the headline score is the equal-weight mean of the four task scores, with a missing task contributing zero.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude Opus 5 | 79.8 |
| GPT-5.6 Terra Pro | 78.1 |
| GPT-5.5 | 77.6 |
| Claude Fable 5 | 77.2 |
| Opus 4.7 | 77.1 |
| Opus 4.6 | 76.4 |
| Sonnet 4.6 | 76.3 |
| GLM-5 | 75.6 |
| GPT-5.6 Sol Pro | 75.3 |
| Opus 4.8 | 75.2 |
| GPT-5.6 Sol | 75.2 |
| GPT-5.4 | 74.8 |
| GLM-5.2 | 74.2 |
| Kimi K2.6 | 74.2 |
| GPT-5.6 Terra | 74.0 |
| GPT-5.6 Luna | 73.3 |
| GPT-5.6 Luna Pro | 73.1 |
| Sonnet 4 | 72.6 |
| DeepSeek-V4-Pro | 72.5 |
| Sonnet 4.5 | 72.3 |
| GPT-5.2 Instant | 72.1 |
| Opus 4.5 | 71.6 |
| Opus 4.1 | 71.3 |
| Gemma 4 26B A4B IT | 71.0 |
| Gemma 4 31B IT | 70.9 |
| Opus 4 | 70.8 |
| GPT-4.1 | 70.8 |
| Sonnet 5 | 70.3 |
| GPT-5.3 Instant | 70.2 |
| Grok 4.5 | 70.0 |
| Horizon Beta | 69.8 |
| GPT-5.1 Instant | 69.8 |
| Kimi K3 | 69.6 |
| GPT-5.1-Codex | 69.4 |
| DeepSeek-V3.2-Speciale | 69.3 |
| Kimi K2 Thinking | 69.2 |
| DeepSeek-V4-Flash | 68.8 |
| GPT-5 | 68.8 |
| ChatGPT-4o Latest (source-unspecified snapshot) | 68.7 |
| GPT-5.3-Codex | 68.6 |