4 posts tagged with "Benchmark"

Summary​

Two weeks ago we planted vulnerabilities in 16 repos and ran the latest models over them. Since then GPT-6 Sol, GPT-6 Luna, Claude Opus 5.5 and Grok 4.7 have shipped, so we ran them through the same 16 pull requests. DeepSeek 4.1 Flash is still the best value: 64% of the planted bugs found for $0.42 a PR. GPT-6 Sol now matches that recall in about two minutes a PR, for $1.81. It is circled on each chart.

This time we also looked at which bugs each model finds, how much that changes from run to run, and how many tokens it takes.

Summary​

We tested the latest models on PR security reviews across 16 repositories, each with a planted vulnerability. DeepSeek 4.1 Flash is our pick for day-to-day review: 64% of the planted bugs found, about 6 minutes a PR. Although OpenAI GPT6 Astra finds 5% more vulnerabilities, it is pricey and tries to break out of its sandbox.

NOTE: this PR Benchmark is specifically scanning differential code-commits (PRs) and looking for recently introduced vulnerabilities. This is not an exhaustive "full scan" across a code base.

Summary​

We re-ran the same PR security-review benchmark (not a full-code scan) over a new wave of models: Kimi K3, Gemini 3.6, Opus 5, GLM5.2 and more OpenAI models. Overall, GPT-5.6 Sol is still on top. Kimi K3 is the strongest open-weights performer, GLM 5.2 is cheap and precise but lags hard on recall, and Opus 5 performed well but was even more expensive than Fable. More below!

We wanted to take the guesswork out of which model to use for PR security reviews. To our surprise no Anthropic model reaches the frontier at all and the latest GPT5.6, released only 3 days ago, is the king. Although Fable is known to perform well on a full-code scan, PR reviews are a common workload for our users.