CWE-bench v2, released by Collinear AI, is an open benchmark for evaluating frontier cyber capabilities. It consists of 100 tasks in which an agent receives a checkout of a real open-source repository and a short instruction, and must find and fix a security weakness without breaking existing functionality. Scoring is fully programmatic, so results are deterministic and reproducible without extra cost. The programmatic verifiers were curated with the help of LLM judge panels, shaped towards hardness and fairness. The public set is complemented by a held-out set, for which evaluation is available by contacting [email protected].

The tasks cover 91 different CWEs across a wide range of vulnerability classes, so a model must understand multiple weakness surfaces rather than just recognize a familiar pattern. For every task, the release verified that the instruction reads realistically, all graded behavior is inferable from the instruction, the verifier checks correct behavior rather than a specific fix structure, the starter code fails and the reference fix passes, the original functionality still works with the fix, and memorized solutions such as the known upstream patch are rejected by verifiers. Each task was also hand-reviewed. Open-sourcing the set is part of the same quality effort, and feedback and contributions are welcome.

Results are tracked in two places on the Hub: a leaderboard on the dataset page covers open-weight models and updates via PRs, while a leaderboard Space covers the full comparison including API-only models. Among 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, slightly ahead of Claude Opus 5.5 (67.3% pass@1, 78.0% pass@4). 95 of the 100 tasks are solved by at least one model, indicating the tasks are solvable, but even the leader fails 19% of the set on every attempt, so the long tail is far from solved.

On cost, DeepSeek V4.1 Flash, Claude Opus 5.5, and Gemini 4 Argon form the Pareto frontier: DeepSeek reaches 57.0% pass@1 at $0.04 per rollout, Opus adds ten points at $1.83, and Gemini edges ahead at $3.92. The public set, grader, and harness are open, so anyone can run a model. Every task is a self-contained Harbor task with a Dockerized environment, an instruction, and a deterministic verifier, and the dataset ships a registry.json. Harbor can run the whole benchmark straight from the Hugging Face repository without cloning anything. The reported score with -k 4 is the mean reward over all 400 trials. The dataset is gated, so users must accept access conditions and save their HF token to git’s credential helper. The tasks are also tagged rl-environment, so the same environments and verifiers can provide rewards for training defensive agents, not just for evaluation.