← All benchmarks

Concept Synth INDUCTION Challenge100 Holdout v2

General QA · 2026-10-17

Post-selection diagnostic for direct Round-1 Challenge100 formulas against the holdout-complete v2 sidecar (challenge-holdouts-v2), which gives every task exactly five generated holdout worlds: train-correct responses verified correct on all five worlds, divided by all 100 tasks. V2 preserves all 449 v1 worlds and adds 51 under a frozen, model-blind completion policy with a variable-degree fallback sampler, so it is not an IID resample of v1. Only the 14 changed tasks were re-scored from stored responses; responses, selection and train scores are unchanged. Train-incorrect responses receive no credit. Holdout results are never used for prompting or response selection.

Top models (higher is better)