The AI PR Security Review Frontier

We wanted to take the guesswork out of which model to use for PR security reviews. To our surprise no Anthropic model reaches the frontier at all and the latest GPT5.6, released only 3 days ago, is the king. Although Fable is known to perform well on a full-code scan, PR reviews are a common workload for our users.

This chart plots Cost per Pull Request vs F1 (finding quality). Blue points define the frontier. See metrics definitions in the table below.

Summary

We ran 10 models over the same 10 pull requests, each carrying one planted access-control bug (IDOR, missing auth, broken authorization), five times per model, and scored every finding against the actual code.

The result is a clear cost/quality frontier, and it belongs to OpenAI, xAI and Google rather than Anthropic. GPT-5.6 Sol dominates. Fable 5 performs poorly at ~$3.61 per PR, lands lower on both quality and value. No Anthropic model reaches the frontier at all.

Our benchmark repos are synthetic and private to avoid Benchmaxxing and accidental inclusion in model training.

NOTE:

• Fable does perform well on Full Code Scans, but this workload is a PR Scan. A PR Scan is a very common workload for our Users.
• This is not a quirk of the Dam Secure harness. We tested the Pydantic harness and Claude Code harness as well and got very similar results.

Benchmark Scope: Pull Requests Only

This benchmark is only trying to measure how good models are at finding vulnerabilties in Pull Requests where additional code has been added. This benchmark is NOT about extensively searching an existing codebase for all known vulnerabilities. Our full scan benchmark is coming soon.

That scope matters for how you read the results. The key takeaway is that you are wasting money, usage limits, and tokens if you reach for Fable (or other frontier Claude models) to find security issues in PRs. Fable 5 is the most expensive configuration we tested and still lands off the cost/quality frontier — dominated on both axes by GPT-5.6 Sol, which reaches higher recall at roughly a fifth of the cost per PR.

We will separately release another article on how these agents perform when freely roaming an existing large codebase. The model rankings may look different there; this post is about PR review only.

Results

Each run used the Dam Secure Vulnerability Scanner with high reasoning, repeated five times on a 10-PR corpus of planted access-control bugs.

Model	Recall	Precision	F1	Cost / PR	Cost / TP
GPT-5.6 Sol via OpenRouter	100%	83.3%	0.91	$0.70	$0.70
GPT-5.5 via OpenRouter	94%	83.9%	0.89	$1.24	$1.32
Fable 5 → Opus 4.8 fallback	88%	83%	0.85	~$3.61	~$4.10
Claude Sonnet 4.6	80%	85.1%	0.82	~$1.22	~$1.53
Gemini 3.5 Flash	84%	76.4%	0.80	~$0.94	~$1.12
Grok 4.5 via OpenRouter	74%	80.4%	0.77	$0.20	$0.27
Gemini 3.1 Flash Lite	68%	82.9%	0.75	~$0.04	~$0.06
Claude Opus 4.8	60%	81.1%	0.69	~$1.72	~$2.87
Claude Haiku 4.5	56%	73.7%	0.64	~$0.75	~$1.34
DeepSeek V4 Pro via OpenRouter	44%	81%	0.57	$0.14	$0.31

• True Positive (TP) - a vuln found that was expected to be found.
• False Positive (FP) - a vuln found that was not expected (not a real bug).
• False Negative (FN) - a vuln not found that was expected to be found.
• Recall (R) - how well did it find the planted vulnerabilities? It is the fraction of actual positives correctly identified: TP / (TP + FN).
• Precision (P) - false positive performance. A fraction of predicted positives that are correct: TP / (TP + FP).
• F1 - the overall performance balancing Recall and Precision: 2PR / (P + R). Hovering a point also shows F2 = 5PR / (4P + R), which weights recall ~4× precision.
• Costs / PR - is the cost to run that model over all prompts and process the PR regardless of it's accuracy. This is the most accurate representation of real-world cost.
• Cost / TP - is the cost per true positive found.
• Wikipedia: Precision, Recall and F1

Results Commentary

GPT-5.6 Sol is the first model to reach 100% Recall on our harness (clearly we now need to add harder vulnerabilities to find).

The Fable 5 row in the table is labeled "Fable 5 → Opus 4.8 fallback". It handed off to Opus 4.8 only 10.7% of the time.

Prior to GPT5.5 coming out, Gemini 3.5 Flash has been our preferred model for this type of workload. This has previously been our little secret, because our evaluation suites have proven despite the industry wanting to believe that the Opus models dominate security.

Best model is GPT-5.6 Sol - strong F1 (0.91), perfect 100% recall and very cost effective at $0.70 per PR. 5.6 is ~45% cheaper than it's predecessor 5.5 for roughly the same performance.

Honorable mentions: Grok 4.5 lands on the frontier at $0.20 per PR with F1 0.77 (74% recall, 80.4% precision), and Gemini 3.1 Flash Lite anchors the cheap end at ~$0.04 per PR with strong results: F1 0.75 (68% recall, 82.9% precision).

We'll add other Open-weight models to this benchmark.

Methodology

We construct synthetic pull requests from open source repositories. Each pull request has a vulnerability planted within it. We then measure the models ability to find the planted vulnerability using both the Dam Security Vulnerability Scanner and the Pydantic harness.

We use two different approaches to build up the synthetic dataset. Private synthetic repositories built from an OSS reference repository and reverse-replay of real CVEs (more below).

The Harness

Each model run uses a limited version of our Dam Secure Vulnerability Scanner configured to match typical tools available in other open source harnesses. It is built on the Vercel AI SDK with a small, security-hardened toolset. The agent sees the PR diff and can:

• Read file - Load full file contents when the diff alone is not enough context
• Grep - Search the changed files for patterns (auth checks, route handlers, scope validation)

The harness is the same across every model in the table. What varies is only the underlying model. That isolates model quality from pipeline design: a good result here reflects the model plus this toolset, not a one-off prompt.

We have also tested these models using an alternate harness Pydantic, and achieved very similar results.

The Test Repos and "Keeping the Benchmark Clean"

The hard part of a security benchmark is contamination through two means:

1. Regular model training - Public vulnerability datasets have their solutions written up online and "accidentally" absorbed into model training data, so a model can recall the fix instead of finding it. Try using Juice Shop as an evaluation suite as a clear example of immediate call in a frontier model.
2. Bench-maxxing - Nefarious model builders will train their models on the benchmarks (hence benchmaxxing). This then destroys the value of a benchmark.

We control for this two ways:

• A private synthetic repos - Purpose-built and never published, so there is nothing to look up. Each PR adds a realistic feature with exactly one planted access-control bug and a hidden answer key. Because the app has no public footprint, no model has seen it in training.
• Reverse-replay of real CVEs - We take a real, already-fixed access-control CVE, revert the fix, and bury it inside genuine same-era commits so the PR reads like ordinary work rather than a planted puzzle. This keeps the realism of production code while still giving us exact ground truth.

Every reviewer sees the same PR diff, we collect its findings and token cost, and we score against ground truth. Running each configuration five times smooths out the run-to-run variance that a single pass would hide. All session artifacts, such as output logs, are kept and analyzed for anomalies in model behavior.

Please shoot us an email to review these benchmark repos.