SciCode
Science · 2024-07-12
SciCode is a scientist-curated benchmark with 80 research coding problems decomposed into 338 subproblems; the standard test split has 288 subproblems from 65 main problems. This definition represents the official standard setup without scientist-provided background knowledge or reasoning steps. It measures whether models can generate executable scientific code from research-style specifications; background-assisted and modified-library evaluations require separate protocol identities.