Methodology — Benchmark Atlas

Methodology

Atlas estimates a latent capability score for each model and reasoning state from heterogeneous public benchmark results. The priorities are to prevent data artifacts from masquerading as capability, to quantify uncertainty, and to validate predictions out of sample rather than relying on fit statistics alone.

Loading the generated methodology contract. The repository version records the dated production study and reproduction commands.

1. What the score estimates

Atlas estimates how well a model setup is likely to perform across benchmarks in general. It looks for the pattern shared across many well-covered benchmark families rather than simply averaging their scores. This matters because benchmarks use different scales, contain different amounts of noise, and vary in how clearly they separate stronger models from weaker ones.

A model is the released AI system. A reasoning state is that model run with a normalized reasoning setting, such as default, low, medium, or high. Atlas pools harness-specific configurations that represent the same model and reasoning setting, then estimates the state's capability as the model's shared ability plus what that reasoning setting is worth to that model. Harness identity and a configuration-specific residual are deliberately not included in the latent score.

The headline score for a model uses its highest-scoring eligible reasoning state that Atlas has observed. Scores are meant for comparison, not as absolute measures of intelligence. To give them a stable, readable scale, Atlas uses two reference reasoning states. These reference points only set the units; they do not change the rankings or uncertainty.

2. Data and inclusion

Benchmark scores come from three separately preserved source classes:

Model releases: launch posts, papers, and model cards published by the model's own lab.
Evaluator releases: scores from independent organizations running their own harnesses.
Benchmark releases: leaderboards maintained by benchmark authors.

Each benchmark score must be associated with a model configuration, an exact benchmark (the specific variant), and an interpretable score direction. Composite indexes are excluded before fitting because their inputs already contain benchmark evidence; including them would count some observations twice.

3. Preprocessing

—retained benchmark scores

↓ orient so higher is better · screen sampling protocols · resolve duplicate reports

—modeled observations — one per reasoning state × family cell

↓ place on the declared range · guessing floor becomes the curve's asymptote · one likelihood vote per family

—fitted battery

Figure 1. The preprocessing pipeline. Counts are read live from the current fit.

Orientation

The capability fit accepts only benchmarks with an explicitly declared higher-is-better or lower-is-better direction. Higher-is-better scores are retained as-is; lower-is-better scores are reflected so higher always means better. Zero-best, target, custom, missing, and unknown directions are excluded rather than folded or inferred.

Duplicate resolution

Multiple sources can report the same configuration × benchmark cell. Before cells are resolved, benchmark scores whose sampling class—consensus-style (majority voting, best-of-N) versus single-attempt—does not match the benchmark's dominant protocol are excluded. Each remaining cell then resolves in four ordered stages:

Consensus. Exact duplicates and reports within the configured tolerance of the benchmark's score span collapse to their median. Repetition earns no extra weight.
Protocol tiebreak. Averaged runs are preferred to conflicting single attempts; mixed shot counts resolve to the benchmark's modal regime.
Majority. With at least three reports, a lone outlier is removed when doing so restores consensus.
Conflict. Remaining irreconcilable cells are excluded and written to an audit queue. Atlas does not average measurements that describe different runs.

Pooling harness configurations

After exact configuration duplicates are resolved, Atlas groups scores that have the same model, normalized reasoning condition, and exact benchmark. If several harness-specific configurations remain in one group, their scores collapse to the median before fitting. The group therefore receives one vote, no matter how many harnesses reported it. A separate audit records the contributing configuration IDs, harnesses, and their score spread; the likelihood does not estimate a harness coefficient.

Putting scores on a common scale

Each retained score is placed on its benchmark's own declared range, then corrected for the floor a model reaches by guessing. A four-way multiple choice bottoms out at 25%, not zero, so that floor is the lower asymptote of the curve rather than a subtraction from the data:

u_i = \frac{s_i - \text{min}_j}{\text{max}_j - \text{min}_j}, \qquad \mathbb{E}[u_i] = c_j + (1 - c_j)\,\sigma\!\left(\beta_j (\varphi_i - b_j)\right)

Here, $i$ indexes a retained model–reasoning-state score on benchmark $j$ ; $u_i$ is that score on its benchmark’s declared range, oriented so higher is better; $c_j$ is the benchmark’s guessing floor where one has been researched; $\varphi_i$ is the reasoning state’s capability; and $\beta_j$ and $b_j$ are the benchmark’s discrimination and difficulty, both estimated jointly with capability.

Scores are used at their own level rather than converted to ranks within each benchmark. That means the distance between two models carries information, not just their order — but it also requires every benchmark to declare a score range and to resolve to a single measurement protocol. Benchmarks that report an unbounded quantity, or that pool variants on different scales, are held out of the fit and say so on their own page.

Where a benchmark is scored as a proportion of a countable number of items, the model works in whole items rather than in the reported percentage, so a score of 95.45% on 198 questions is read as 189 correct. Its predictions then land on the same lattice the data does.

The benchmark parent tree records provenance and UI hierarchy. Every benchmark must separately declare its statistical family and whether it is a primary, component, or diagnostic: an adequately covered primary is used globally for its family; otherwise coherent components are pooled. Diagnostic metrics are excluded. This keeps each matrix column's definition stable across reasoning states.

Keeping a family to one measurement protocol is essential. A family that combined variants with different scales and difficulty would mistake variant difficulty for model capability, and a single curve would describe none of them. Benchmarks whose retained scores never vary carry no information about capability and are discarded.

Aggregation and eligibility

Safety benchmark families are excluded before capability-family selection and remain excluded from every validation split. They measure behavior and policy rather than the monotone capability axis and belong in a separate model.

Selected component benchmark scores are averaged into one observation per model–reasoning state × family. Every family observation receives one likelihood vote; correlated component count is descriptive metadata, not a precision multiplier. Families with fewer than the configured minimum number of models are removed iteratively. A model stays connected when it has at least the configured minimum number of families across all its reasoning states, so a sparse state can borrow its base capability from the model.

4. Statistical model

After exact benchmark scores are pooled by model and reasoning condition, placed on their benchmark’s declared range and—when needed—combined into families, Atlas models the expected score $u_{m,r,j}$ for model $m$ using reasoning setting $r$ on benchmark $j$ as a curve in capability:

\mathbb{E}\!\left[u_{m,r,j}\right] = c_j + (1-c_j)\,\operatorname{logistic}\!\left(\beta_j\left(\varphi_{m,r}-b_j\right)\right)

\beta_j=\exp(\lambda_j)>0, \qquad \varphi_{m,r}=\theta_m+\gamma_{m,r}

Parameter intuition

Benchmark difficulty, $b_j$ . The capability at which a model is expected to score halfway between the benchmark’s guessing floor and a perfect result. It is expressed in the same units as capability, which is what makes the comparison $\varphi - b_j$ meaningful: a benchmark’s difficulty is literally the score a model needs to reach its midpoint.
Benchmark discrimination, $\beta_j$ . How sharply the benchmark separates models. A high value means a small capability difference produces a large score difference; a low value means the benchmark barely distinguishes anyone. The spread is wide in practice, and it is why one observation on a sharp benchmark tells you far more than one on a flat benchmark.
Guessing floor, $c_j$ . The score a model reaches without ability. A four-way multiple choice bottoms out near 25%, so the curve starts there rather than at zero. It is researched by hand per benchmark, not estimated, and is left at zero where no floor applies.
Model capability, $\theta_m$ . The general ability shared by every reasoning state of a model. Evidence from a sparsely measured state therefore informs the same model parameter through the rest of that model’s coverage.
Reasoning effect, $\gamma_{m,r}$ . What a reasoning setting is worth to that model. Each setting has a typical value across models, and each model is allowed its own departure from it — see below. Turning reasoning off costs some model families several times what it costs others; raising effort varies much less.
Dispersion, $\kappa_j$ . How tightly a benchmark’s scores cluster around the fitted curve. It carries the same meaning whether or not the benchmark’s questions can be counted, which is what lets both kinds share one population.

What the likelihood assumes

Benchmark scores are bounded proportions, so their scatter is not constant: a score near 50% can vary far more between runs than one near 95%, where there is little room left to move. A Normal likelihood would treat both as equally noisy and would happily predict values below zero or above one. Atlas instead uses distributions defined on the unit interval, whose spread narrows automatically toward each end.

Where the questions can be counted. Where a benchmark is scored as a proportion of a known number of items, the model works in whole questions rather than in the reported percentage. A score of 95.45% on 198 questions is read as 189 correct, and the fitted distribution places its predictions on the same lattice the data lies on. A dispersion parameter per benchmark absorbs the fact that questions within a benchmark rise and fall together, so a long benchmark is not treated as more precise than its items genuinely are.

Everywhere else. Judge scores, composite indices and averages of partial credit have no denominator to count. They use the continuous form of the same distribution, sharing one dispersion population with the countable benchmarks, because the continuous form is what the counted form becomes as the number of items grows.

Exact zeros. Scores of exactly zero occur far more often than either distribution predicts. Some are genuine, and some are refusals, parse failures, or harness errors that have nothing to do with capability. The model gives them their own probability, tied to the same capability comparison, so that scoring a flat zero becomes more likely the further a model sits below a benchmark rather than being a fixed property of the benchmark.

Figure 2. How two benchmarks respond to capability (illustration). A discriminating benchmark moves sharply with capability and is measured precisely; a weakly discriminating one barely moves and is noisy, so one observation there carries far less information about a reasoning state's capability. Each curve also has its own difficulty, the capability at which it crosses the midpoint between the guessing floor and a perfect score.

How reasoning settings are modeled

The reasoning effect $\gamma_{m,r}$ is a hybrid of shared structure and per-model freedom, split by whether the setting sits on the ordered effort ladder (minimal → low → medium → high → xhigh → max):

On the ladder. Every effort level has a shared across-model effect, and each model additionally gets its own amplitude and curvature over the whole ladder, expressed through fixed orthogonal linear and quadratic contrasts. A model whose scores climb steeply from low to high effort can say so; one that plateaus early can say that instead. The contrasts are fixed rather than estimated, which is what keeps this identifiable.
Off the ladder. Settings with no natural order — reasoning disabled, or an unlabeled “enabled” — get a free per-model deviation around the shared effect, with a per-setting spread learned from the data. Turning reasoning off is the clearest case: it costs some model families far more than others, and a shared effect alone would miss that.

This structure was chosen by measurement, not preference: against a single shared effect per setting it improves held-out state-level prediction by about 2.3% CRPS, and a ladder-only variant was substantially worse exactly on the off-ladder settings that vary most across models.

Priors and identification

Letting benchmark families borrow strength. Every family gets its own difficulty, discrimination, and dispersion, but families with little data should not receive extreme estimates from a handful of scores. Atlas therefore estimates each set of family parameters as a population: the well-covered families establish a typical value and a typical amount of variation, while sparse families are pulled toward that shared pattern. This compromise—called partial pooling—still lets a family depart from the group when its own evidence is strong.

What “hierarchical Normal” and “shared hyperprior” mean. The individual family values are modeled as draws from a common Normal distribution. The center and spread of that distribution are learned from all families rather than fixed in advance. Because those shared center-and-spread parameters also have priors, their priors are called hyperpriors. Difficulties are modeled directly; discriminations and dispersions are modeled on the log scale so their fitted values remain positive.

Regularizing reasoning effects. The shared reasoning-setting effects have a HalfNormal(configured scale) prior on their population scale, and the per-model ladder amplitudes, curvatures, and off-ladder deviations each have their own hierarchical scales. Capability has a one-unit standard deviation, so these settings express the prior judgment that repeatable reasoning-setting differences may be moderate. They are soft constraints, not caps: the data can learn larger scales.

Fixing the ruler. Without reference conventions, the same predictions could be produced by shifting every capability and compensating with the difficulties, or by stretching capability and shrinking every discrimination. Atlas fixes this ambiguity two ways. First, model capability is standardized to mean zero and standard deviation one across models with an observed default reasoning state; this is genuine identification, not a convention — removing it collapses the sampler's effective sample size by roughly sevenfold. Second, exactly one reasoning state per model carries no deviation of its own (the default state where one is observed), because otherwise a model's base capability and its reasoning deviations could shift together without changing any prediction. The display anchors in §5 later convert this latent ruler into Atlas score points.

5. Inference, uncertainty, and displayed scores

Inference uses nutpie's No-U-Turn Sampler with the configured family effects and a target acceptance probability of the configured value. The current fit uses NUTS sampling.

Every reported score, rank probability, and interval is computed from posterior draws after the affine anchor transformation, which is itself applied per draw so the display scale carries its own uncertainty. Displayed intervals are posterior% posterior credible intervals. Wide intervals mean the available benchmark panel does not locate that model precisely; they are not error bars around a fixed leaderboard average.

Reasoning-state scores are the primary likelihood estimand. For each model, Atlas selects the rank-eligible reasoning state with the highest posterior mean capability, then reports that one fixed state's posterior score, interval, and rank probability. It does not take a new maximum in every posterior draw. Because harness-specific records were pooled before fitting, this is a best observed reasoning-state score rather than a score for a particular harness.

The model partially adjusts for differences among observed panels through each benchmark's difficulty and discrimination, conditional on missingness being ignorable. For example, a benchmark evaluated only on strong models can learn a higher difficulty rather than forcing those models toward the center of the global population. The model does not explicitly model how models or reasoning settings are selected for evaluation.

6. Benchmark difficulty

Difficulty is a parameter of the fit, not a quantity recovered afterwards. Because the model reads score levels, each benchmark’s difficulty $b_j$ sits on the same scale as capability, is estimated jointly with it, and carries posterior uncertainty like every other parameter. The curve below is the same one the likelihood uses:

\mathbb{E}\!\left[\frac{\text{score}-\text{lo}}{\text{hi}-\text{lo}}\right] = c_j + (1-c_j)\operatorname{logistic}\!\left(\beta_j(\varphi_{m,r} - b_j)\right), \qquad \beta_j > 0

Difficulty is the capability at which the curve reaches the midpoint between the guessing floor and a perfect score, reported on the display scale. The same curve yields a saturation point (the configured crossing) and the expected frontier score, and its slope is informative on its own: a shallow curve means task difficulty varies widely inside the benchmark.

Figure 5. Two difficulty curves (illustration). Difficulty is where each curve crosses half the score range; the steep benchmark saturates soon after its crossing, while the shallow one holds headroom far past it. When no evaluated reasoning state is near the crossing, the estimate is reported as a bound (≳ / ≲) rather than a point.

When a benchmark’s 50% crossing lies outside the capability range actually evaluated on it—nothing has come close to half-solving it, or everything already scores beyond its midpoint—the estimate is flagged and shown as a bound (≳ / ≲) rather than a point, because the data only brackets it from one side. Estimates appear as the Difficulty column in the benchmark catalog; benchmarks outside the fit explain on their own page why none is available.

7. Diagnostics for the current fit

These panels are read directly from the current fit artifact. They cover sampler convergence and the posterior reasoning-setting effects, including whether each step up the effort ladder actually helps.

Sampler

passes

max R-hat

—

min ESS

—

divergences

—

Checks whether the posterior sample looks numerically trustworthy before interpreting model structure. R-hat and ESS are the worst values across capability, discrimination, and difficulty parameters.

Reasoning effects

This run does not include posterior reasoning effects, so monotonic reasoning checks are unavailable.

Convergence diagnostics are necessary but not sufficient: a sampler can converge to a model that predicts poorly. Atlas therefore treats the holdout studies below as the primary empirical validation.

8. Out-of-sample validation

Each study refits the production specification and compares predictions with observations the fit was not allowed to use.

Entirely unseen benchmark families

Families are divided into the configured number of folds. Each fold is removed entirely, the model is refit without it, and each held-out leaderboard is predicted from reasoning-state capability alone. The baseline is a simple average: rank models by their mean score across the benchmarks that stayed in the fit. Because an unseen family has no learned discrimination or difficulty, rank correlation is the primary metric; RMSE and MAE standardize both vectors within the shared panel before comparison.

held-out family rank prediction	capability model	simple average
mean Spearman ρ	—	—
median Spearman ρ	—	—
panel-size-weighted mean ρ	—	—
mean normalized RMSE	—	—
mean normalized MAE	—	—
families where capability has higher ρ	—	—

Forward-in-time prediction

The temporal study orders models by release date, keeps the earliest generations and withholds cells for the latest models. Benchmark curves learned from earlier generations must then predict the newest ones.

temporal holdout metric	value
held-out cells	—
RMSE (unit scale)	—
CRPS	—
Spearman ρ	—
50% / 90% / 95% coverage	— / — / —

Systematic undercoverage here would be direct evidence of generational drift; calibration close to nominal means the fitted curves transfer forward.

9. Interpretation and scope

What capability explains

Capability summarizes the shared ordering visible across the retained benchmark battery. For each family, the share of score variation the fitted curve accounts for is measured empirically as $R^2$ on the logit scale. The residual is everything capability does not explain: benchmark-specific strengths, measurement noise, and harness variation.

What capability does not explain

No held-out family in the current validation artifact has a negative association with capability.

Because discriminations are constrained positive, the production fit cannot encode an inverse relationship: a family that genuinely runs against capability contributes little and is explained poorly, rather than pulling scores down. Negative held-out estimates on small panels are also unstable across evaluation splits. Either way, capability is not a composite measure of safety, reliability, honesty, product quality, or social value.

10. Limitations

Sparse, non-random coverage. Only a small fraction of the reasoning-state–family matrix is observed. Missingness is treated as ignorable conditional on model and family; the holdout studies probe but do not eliminate selection concerns.
Constrained reasoning structure. On the effort ladder, a model's departure from the shared pattern is limited to an amplitude and a curvature. A model whose response to effort has a more unusual shape than that is partially smoothed toward the shared ladder.
Uncalibrated vendor labels. Identically normalized effort labels share structure across labs. Because vendor labels are not compute-calibrated, one lab's “high” may be another's “medium”; the per-model terms absorb some of this, but the labels themselves remain nominal.
Harness differences are pooled, not estimated. The model assumes records with the same model and normalized reasoning condition measure the same latent state. Taking their median prevents duplicate votes but can hide a real, systematic advantage from a harness. The audit spread reveals disagreement; it does not identify a causal harness effect.
Time-varying evaluations. Conflicting snapshots from evolving leaderboards are excluded rather than modeled dynamically.
Monotone single-dimension summary. Negative relationships are deliberately not incorporated into capability, and family-specific strengths can remain after conditioning on capability.
Relative units. Scores depend on the covered population and two anchors. Differences are meaningful within this system but are not absolute units of intelligence.
Public-data quality. Source metadata can be incomplete or wrong. Ambiguous cells are audited and excluded, but curation cannot guarantee that every surviving attribution is correct.

11. Reproducing the fit

Dependencies are declared in pyproject.toml and resolved with uv. The model is a single file, model/fit.py, with the data pipeline in model/prepare.py.

# production fit: writes model_scores.csv, posterior.npz, diagnostics.json
uv run -m model.fit --out-dir model_output/current

# publish the app's three artifacts from that fit; runs the masked-cell and
# temporal holdout studies and stamps a consistent spec fingerprint
uv run -m model.publish --fit model_output/current

# optionally attach a benchmark-family holdout study to the published contract
uv run -m model.publish --fit model_output/current \
  --family-holdout family_holdout.json

The publish step writes factor_model.json, validation_holdout.json, and methodology.json together from one fit, and the production build fails unless their hashes, timestamps, and specification fingerprint agree. This makes every displayed score traceable to its input data and exact inference settings.

Implementation: methodology and reproduction notes.