← All benchmarks

LSCBench Repo Granularity

Code · 2026-10-10

The 150 LSCBench tasks posed at repo granularity, with the declared source scope emptied and rebuilt from a repository-level specification. Each task asks a coding agent, offline in a Harbor container, to implement a functional specification derived from a real CVE in C, C++, Go, Java, JavaScript, PHP or Python; the specification never mentions security. Hidden functional tests and a security suite that replays the original attack run in a separate verifier, and a task counts only when both pass. The score (FUNC∧SEC) is the percentage of these tasks whose code both works and is secure.

Top models (higher is better)