← All benchmarks

CyberGym (pass@3)

Agents

CyberGym Level 1 with three independent full runs per model: the share of the 1,507 vulnerability-reproduction tasks for which at least one of three runs produces a proof of concept that crashes the vulnerable build but not the fixed build. Fireworks runs it with Claude Code in isolated E2B sandboxes under a six-hour task cap. Kept separate from the one-trial canonical leaderboard metric because additional attempts materially change the success rate.

Top models (higher is better)