Benchmarks

← All benchmarks

Snorkel SWE-bench CLI

Code

Pass@1 on 200 repository engineering tasks spanning eleven languages, ten technical domains and eight task types. Harbor runs fail-to-pass and pass-to-pass tests in reproducible network-disabled Docker environments; credit requires all required tests and substantive changes spanning at least two files. Native effort and agent scaffold are not disclosed.

Top models (higher is better)