Skip to main content
View all authors

PR Security Review Benchmark Update: New Model Showdown

· 4 min read

Summary

We re-ran the same PR security-review benchmark (not a full-code scan) over a new wave of models: Kimi K3, Gemini 3.6, Opus 5, GLM5.2 and more OpenAI models. Overall, GPT-5.6 Sol is still on top. Kimi K3 is the strongest open-weights performer, GLM 5.2 is cheap and precise but lags hard on recall, and Opus 5 performed well but was even more expensive than Fable. More below!

Guardrails vs Guidelines

· 6 min read

In the rush to integrate generative AI into the software development lifecycle, engineering teams frequently confuse two distinct operational concepts: guardrails and guidelines. To put it simply, a guideline advises: it's soft natural-language context, like an AGENTS.md or Skills file, that the model may or may not act on. A guardrail gates: it's a firm, automated constraint that forces an explicit, binary pass/fail and blocks on failure. Relying on soft guidelines for critical validation is why automated code reviews are risky. They're non-deterministic and quietly ignored.