← All benchmarks

LSCBench File Granularity

Code · 2026-10-10

The 150 LSCBench tasks posed at file granularity, with the target files removed in full and rewritten from one specification per file. Each task asks a coding agent, offline in a Harbor container, to implement a functional specification derived from a real CVE in C, C++, Go, Java, JavaScript, PHP or Python; the specification never mentions security. Hidden functional tests and a security suite that replays the original attack run in a separate verifier, and a task counts only when both pass. The score (FUNC∧SEC) is the percentage of these tasks whose code both works and is secure.

Top models (higher is better)