A security scanner has two ways to fail. It can miss real problems, and it can cry wolf. The second failure is the quiet one, because a scanner that floods you with false alarms does not look broken, it looks thorough, right up until you start ignoring it. So we measured the thing that actually decides whether a scanner survives in your workflow: of the high-severity issues it reports, how many are real?
How we ran it
We took 17 real AI-built web apps, sampled from our study corpus with a fixed random seed before either tool touched them, so neither tool got a favorable set. We cloned each repo once and ran both tools on the exact same checkout: Prbl, and Semgrep with its default ruleset. We compared each tool at its own top severity, the findings a developer is told to act on now: Prbl's “high” and Semgrep's “ERROR.” Then we read every finding by hand, with the surrounding code, and marked each one a real issue or a false positive under the same standard for both tools.
Want to see exactly what Prbl flags? Watch it scan a demo app, no repo or account needed.
See a live scan →The result
| Tool | High-severity findings | Real | False positives | Precision |
|---|---|---|---|---|
| Prbl | 8 | 8 | 0 | 100% |
| Semgrep (default ruleset) | 24 | 1 | 23 | ~4% |
Same repos, same checkouts, same reviewer. Semgrep's default ruleset surfaced 24 high-severity findings and one of them was a real, exploitable issue. Prbl surfaced 8 and all 8 were real. This is not a story about Semgrep missing things, and that is the part worth slowing down on.
Both tools found the one real thing Semgrep found
The single genuine vulnerability in Semgrep's list, a shell command built with shell=True in a way that can pass untrusted input to a shell, was also caught by Prbl. So Prbl did not win by finding less. It found the same real issue, plus seven more real issues Semgrep's default ruleset reported nothing about, with zero noise attached.
Those seven were not subtle. In one app, the highest-severity findings were several admin passwords hardcoded directly into source, alongside a database connection with certificate verification turned off and a one-time login code generated with a non-cryptographic random source. On that repo, Semgrep's default ruleset reported zero high-severity findings. Prbl reported six, all real. A scanner can be noisy and still miss the leaked admin password; those are not opposite failures, they are the same failure of aim.
What Semgrep's 23 false positives actually were
None of them were nonsense. They were defensible hardening advice firing at the top severity level, where a real vulnerability should live. That is what makes them expensive: they are individually reasonable and collectively drown the signal.
| Pattern | Count | Why it is not an actionable high-severity finding |
|---|---|---|
| XML parse without defusedxml | 15 | Fires on every use of the standard library XML parser, regardless of whether the input is attacker-controlled. All instances here parsed local files the app itself produced. A hardening suggestion, not a shipped vulnerability. |
| Dockerfile missing USER | 5 | Flags any Dockerfile that does not set a non-root USER. Container-hardening advice, and not even application code. Not an exploitable flaw in the app. |
| Raw SQL query | 3 | Flagged three queries that were actually parameterized (values passed as bound parameters) or built from constants. Misreads a safe pattern as SQL injection. |
A developer who runs Semgrep out of the box on their vibe-coded app gets 24 red alerts, spends an evening chasing container-hardening lints and safe SQL queries, and has no way to know that exactly one of them is the finding that matters. That is not a rule-quality problem. It is a targeting problem, and it is the specific problem Prbl exists to solve.
Why Prbl is built the other way
Prbl is deliberately narrow. It asks one question, did an AI tool generate code with the vulnerability classes those tools actually introduce, and it reserves high severity for findings where the risk is real in context. That narrowness is the whole product. It is why the list is short, why the precision on this sample was 100%, and why it stays in a developer's workflow instead of getting muted after the third false alarm. We have measured the same shape before: on create-t3-app, a clean starter template, Semgrep's default ruleset produced 54 findings and Prbl produced 0.
Where Snyk fits, and why it is not in this table
People often group Snyk into this comparison, so it is worth being precise. Snyk's core strength is scanning your dependencies, containers, and infrastructure, the supply chain around your code. Prbl and Semgrep scan the first-party code you wrote (or your AI wrote). A false-positive-rate test only makes sense between tools aimed at the same target, which is why this study is Prbl against Semgrep. Most teams run a dependency scanner and a first-party scanner, because they cover different halves of the app.
Frequently asked questions
Does Prbl replace Semgrep?
No. Semgrep is a powerful, general-purpose engine that shines once you invest in tuning rulesets for your stack. Prbl is narrow on purpose: it ships tuned for the specific vulnerability classes AI coding tools introduce, so a developer with no time to tune gets a short, high-precision list. Many teams run both.
Isn't this unfair to Semgrep?
We used Semgrep's default ruleset at its ERROR (highest) severity, which is what a developer gets by running it with no configuration. That is the honest baseline for someone who just wants to scan their repo. Semgrep's false positives here were not wrong rules, they were hardening lints firing at high severity, which buries the one real finding. Tuning improves this, and tuning is exactly the work Prbl removes.
Why do false positives matter so much?
Because a scanner a developer stops trusting is a scanner they stop running. When 23 of 24 high-severity alerts are noise, the real one gets ignored along with them. Precision is what keeps the tool in the workflow.
Methodology and limits
17 AI-built web apps that both tools scanned successfully, sampled by a fixed seed from our study corpus before scanning, cloned once and run through Prbl and Semgrep's default ruleset on the identical checkout. Each tool compared at its own highest severity tier. Every finding from both tools was manually reviewed against the surrounding code under one consistent standard, and the same reviewer triaged both tools' output.
The honest limits: this is a small sample, and precision on a larger corpus would carry a wider margin than these round numbers suggest. We measured precision, not recall, so this study does not claim Prbl finds everything, only that what it reports as high severity was real here. We used Semgrep's default configuration; a security team that invests in curated rulesets will see far better precision than the out-of-the-box baseline, and that investment is exactly the work Prbl is designed to remove. No repository, owner, file, or credential is identified anywhere in this post; every pattern above is described generically.
The takeaway is not that Semgrep is a bad tool. It is a great engine that rewards tuning. The takeaway is that out of the box, on the kind of AI-generated app a solo builder actually ships, its high-severity tier is mostly noise, and the one real finding is buried in it. Prbl hands you the short list of things that are actually wrong. That is the entire job.