← All benchmarks

Concept Synth INDUCTION Challenge64 Holdout v2

General QA · 2026-10-17

Post-selection diagnostic for direct Round-1 Challenge64 formulas against the holdout-complete v2 sidecar (challenge-holdouts-v2), which gives every task exactly five generated holdout worlds: train-correct responses verified correct on all five worlds, divided by all 64 tasks. V2 preserves all v1 worlds and adds seven under a frozen, model-blind completion policy with a variable-degree fallback sampler, so it is not an IID resample of v1. Only the three changed tasks were re-scored from stored responses; responses, selection and train scores are unchanged. Train-incorrect responses receive no credit. Holdout results are never used for prompting or response selection.

Top models (higher is better)