We planted vulnerabilities in 16 repos. Here's why we're switching from OpenAI to DeepSeek v4.1 Flash
Summary
We tested the latest models on PR security reviews across 16 repositories, each with a planted vulnerability. DeepSeek 4.1 Flash is our pick for day-to-day review: 64% of the planted bugs found, about 6 minutes a PR. Although OpenAI GPT6 Astra finds 5% more vulnerabilities, it is pricey and tries to break out of its sandbox.
NOTE: this PR Benchmark is specifically scanning differential code-commits (PRs) and looking for recently introduced vulnerabilities. This is not an exhaustive "full scan" across a code base.
Highlights
- DeepSeek v4.1 Flash delivers the best bang-for-buck. It performs as well as the US models for finding vulnerabilities (64%). It is inexpensive (via Vercel) at a fraction of the price: $0.42 per PR. However, it is the noisiest and generates a lot of findings that might distract developers and create unwanted goose-chases.
- GPT-6 Astra performs the best overall finding 68% (Recall) of all vulns. However, it is costly at $8.80/PR. Solid. Not a clean sweep.
- Gemini 3.8 Flash is a lemon on this task. 35% recall, tied for last on findings. Despite being a public-bench specialist it fails our private bench.
- Anthropic is still out. Anthropic tanked last 2 times we ran this report. This time it is flanked by OpenAI and DeepSeek. DeepSeek bests all Anthropic models on all dimensions.
How we choose models:
- There is no "best" model. You need to choose between: highest recall, a review a team would actually wait for in PRs, and cost.
- Cost is a key factor. A team that produces 1,000 PRs per month can easily consume $150k USD in tokens over a year. So we need to be careful with which model we choose.
- Time to review must be <10 minutes. Developers get annoyed by a PR security review that blocks their workflow. Our internal goal is 3-4 minutes and we believe most teams will not accept reviews above 7 minutes per review.
We made the benchmark harder after models started maxing out some of our previous tests. This round also includes the latest model releases: Gemini 3.8 Flash, GPT6 Astra, Deepseek v4.1 Flash and Fable 5.1.
All models are set to "High" thinking mode. Typically above High, the models become too slow for a PR review.
Results
| Model | Recall | Cost / PR | Time / PR | Findings / TP | Output tokens |
|---|---|---|---|---|---|
| GPT-6 Astra | 68% | $8.80 | 5m 13s | 4.5 | 1,174,942 |
| DeepSeek V4.1 Flash (via Vercel) | 64% | $0.42 | 6m 32s | 8.8 | 28,771,452 |
| Claude Opus 5 | 64% | $11.08 | 7m 10s | 5.0 | 9,440,542 |
| Fable 5.1 | 63% | $7.31 | 3m 44s | 3.7 | 3,433,907 |
| GPT-5.6 Sol | 61% | $5.82 | 3m 33s | 5.2 | 2,881,032 |
| GLM 5.3 (via OpenRouter) | 58% | $2.09 | 6m 43s | 4.9 | 8,326,740 |
| Kimi K3 (via OpenRouter) | 56% | $4.01 | 8m 28s | 5.1 | 5,658,253 |
| GLM 5.3 Flash (via OpenRouter) | 49% | $0.11 | 8m 6s | 5.0 | 7,978,225 |
| Grok 4.6 (via OpenRouter) | 48% | $4.66 | 4m 42s | 2.4 | 8,470,460 |
| GPT-5.6 Luna | 39% | $0.28 | 2m 52s | 4.1 | 3,386,337 |
| Claude Sonnet 5 | 35% | $3.18 | 5m 23s | 4.7 | 7,115,275 |
| Gemini 3.8 Flash | 35% | $3.45 | 4m 32s | 1.8 | 18,892,349 |
Definitions:
- TP: True Positive. The number of planted vulnerabilities found.
- Recall: same as True Positive. The number of planted vulnerabilities found.
- Time / PR: The time it takes to complete the security scan of the PR.
- Findings: the overall number of vulnerabilities the model reported.
- Findings / TP: A measure of noisiness. Higher is worse.
Findings at any cost?
Mean review time per PR vs recall. Blue points define the time/quality frontier.
If you strictly want to find the most vulnerabilities, and you do not care how long the review takes or how much it costs, OpenAI GPT6 Astra is the pick.
Developer time is costly: optimizing for fewer false positives
Findings per true positive vs recall. Lower on this axis means less junk for developers to triage. Blue points define the noise/quality frontier.
If developer productivity is the goal then you want fewer findings overall, even if that means potentially missing some real vulns. Some models just have to say something: ask "is there is a vulnerability?", and they will hunt until they can say yes, whether or not the bug is real.
OpenAI GPT6 Astra is the cleanest of the models that catches most of the planted bugs and only has 4.5 findings per true positive.
DeepSeek V4.1 Flash is the other extreme, flagging 5.4 extra issues per real vulnerability detected. Gemini 3.8 Flash and Grok 4.6 look quiet on this chart, but they miss more than half of the planted bugs.
One model is not enough
PR reviews can no longer rely on a single model. That is still how most companies approach this when they build the capability themselves. You have to fuse models into a pipeline that harnesses each one's strengths and mitigates its weaknesses.
Why you should care about this Benchmark?
Teams building PR security reviews themselves tend to default to whatever frontier model they already use, often an expensive one, rather than continuously testing which models actually perform best, across different languages, costs and false-positive rates.
This benchmark exists because those differences are large, and they keep changing as new models arrive. We'll keep publishing this benchmark.
Dam Secure routes PR reviews to the models that perform best for each workload and language. We also optimize aggressively for developer trust. High recall isn't useful if it floods developers with false positives, as DeepSeek 4.1 Flash demonstrates here.
With Dam Secure, findings that survive verification become tracked issues rather than disposable PR comments. They resolve when the underlying problem is fixed, remember previous triage decisions, improve future reviews, feed posture reporting, and route serious issues into alerts, ticketing and slack updates.
You can build a /security-review bot quickly, but operating this whole system is to rebuild Dam Secure. Get started.
Methodology
This benchmark measures one workload: finding a planted vulnerability in a pull request. It is not a full-repository hunt. Rankings here should not be read as a general "best security model" list.
We scored 16 repos, with a mix of vulnerability types like IDOR, SQL injection, and missing authentication.
The repos span TypeScript, JavaScript, Java, Ruby, Rust and Python.
How we run the models
Every model reviews the same 16 pull requests through the same Dam Secure Vulnerability Scanner configuration. The agent sees the PR diff and has full access to the source code, so it can trace changes across files and investigate more nuanced issues. Reasoning effort is pinned to high for every model; what varies is only the underlying model.
Each configuration is run five times. That is 80 scored reviews (16 repos × 5). We score whether the planted bug was found, missed, or crowded out by extra findings.
Securing Sandboxes
AI agents can pursue goals beyond their assigned tasks. The METR investigation into the Hugging Face incident documented agents bypassing intended isolation and coordinating attempts to tamper with a benchmark’s scoring system.
At Dam Secure, we’ve invested heavily in isolating agents, including those used for this benchmark. That means specialised execution environments, hardening tools such as Git, and building custom agent tools that restrict what our agents can do.
Before giving an LLM access to code to search for vulnerabilities, build strong sandboxes.
Keeping the benchmark clean
We reverse-replay real vulnerabilities into open source codebases: take a known bug (typically an already-fixed CVE), restore the vulnerable code, and present it as an ordinary pull request. The PR reads like day-to-day work, and we try hard not to give away that the bug was planted.
We will not publish the source repositories. That would invite training-set contamination and give away how the fixtures were built. The pull requests themselves stay unpublished for the same reason.
The hard part of a security benchmark is contamination through two means:
- Regular model training — Public vulnerability write-ups get absorbed into training data, so a model can recall the fix instead of finding it. Juice Shop is the obvious example: frontier models call the bugs on sight.
- Bench-maxxing — Training on the benchmark itself. That destroys the ranking.
This run is reverse-replay only, not the private synthetic apps from earlier versions. Ground truth is the reverted CVE; the surrounding code is real. Please email us if you want to review the repos.