Most of the engineering effort on Prbl hasn’t gone into finding more vulnerability classes. It’s gone into not crying wolf. A rule that fires on clean code trains you to ignore the tool, which defeats the purpose of having it.
The pipeline every rule has to pass
- A synthetic test suite: minimum 10 cases per rule, covering both true positives and known false-positive shapes
- A batch stress test against at least 20 real, public codebases
- A confirmed false-positive rate under 10% on human-written code before merge
- Validation against at least one codebase with 1,000+ files
Rules that skip this pipeline get reverted. One of our own rules, an SSRF detector, was removed after producing a 100% false-positive rate on the first production codebase it touched. It had been added without running any of the four steps above. That’s the failure mode this pipeline exists to catch.
Wondering if your code has any of this? Scan a public repo free, no account needed.
Scan a repo →Where the testing has gone so far
15 rule classes, validated across 113 repositories in four separate batches: established enterprise open-source projects, Next.js/Vercel-style SaaS codebases, solo-developer tutorial projects, and the batch most relevant to Prbl’s actual thesis, output from AI app builders (Lovable, Bolt, v0). 303 regression tests currently pass with zero failures.
In the AI-builder batch specifically: 4 out of 20 generated repos contained a hardcoded fallback secret, process.env.SECRET || 'hardcoded-value', baked directly into scaffolded auth or session config. That pattern doesn’t show up in the other three batches anywhere near as often. It’s a generation-time default, not a developer mistake, and general-purpose scanners we compared against didn’t flag it.
A direct precision comparison
We ran Semgrep’s default ruleset (p/default) and Prbl against create-t3-app, a clean, widely-used Next.js scaffolding tool with no known path-traversal vulnerabilities. Semgrep’s default ruleset produced 54 findings on it. 53 of them were path-traversal false positives from pattern-matching on file-path handling that was already safe. Prbl produced zero findings on the same repo.
This isn’t a claim that Semgrep is bad. It’s a much broader tool solving a much broader problem. It’s a demonstration of what a narrower, more targeted rule set buys you: on a codebase with nothing wrong, a precision-tuned scanner should say nothing, not generate 54 things to triage.
What we don’t publish
Stress-testing against real, popular open-source repositories occasionally turns up genuine, exploitable findings in projects we don’t maintain. When that happens, we report them privately to the maintainers and give them time to fix the issue before any of it becomes a public case study. That’s the same standard disclosure window any security researcher should follow. A couple of those reports are open right now. We’ll write them up once they’re resolved, not before.
Why this matters more than rule count
Anyone can write a regex that matches password\s*=. The hard part is making it not fire on a test fixture, a Swagger example, or a TypeScript enum value that happens to contain the word “secret.” That’s where most of the actual engineering time goes, and it’s the part that doesn’t show up in a feature list.