This is a companion to our scan of 119 Claude Fable 5 web apps, which found that 12.6% had a high-severity security flaw. The obvious next question is whether that number is good or bad, and the only way to answer it is to compare against something. So we ran the same scan over the two model generations that came before Fable 5, and put the three side by side.
How we built the cohorts
When you commit code with Claude Code, it writes a co-author trailer into the commit naming the exact model. That trailer lets us attribute a repository's code to a specific model generation with commit-level precision, without guessing. We searched GitHub for repositories majority-authored by each of the three models, filtered to the ones that are genuinely web applications (that filter was fixed before we looked at any results), and scanned every one with Prbl. Same scanner, same rules, same reviewer, run over all three sets.
Wondering if your code has any of this? Scan a public repo free, no account needed.
Scan a repo →The three numbers
| Model | Web apps | With a high-severity flaw | 95% CI |
|---|---|---|---|
| Claude Fable 5 (newest) | 118 | 12.7% (15) | ±6.0 pts |
| Claude Opus 4.8 | 144 | 15.3% (22) | ±5.9 pts |
| Claude Opus 4.5 (oldest) | 102 | 16.7% (17) | ±7.2 pts |
At a glance there is a gentle slope: the newest model looks a little cleaner than the oldest. That slope is a mirage, and it is worth being precise about why, because it is exactly the kind of number that gets turned into a headline it can't support.
The trend is not real, and we checked
The widest gap in the table is Fable 5 (12.7%) against Opus 4.5 (16.7%). We ran a two-proportion significance test on it. The result is p = 0.41, which is nowhere near the p < 0.05 threshold that would let you call a difference real. The 95% confidence intervals, shown in the last column, overlap almost completely: each rate could plausibly sit anywhere in a twelve-point band. With roughly 15 to 22 flagged apps per group, that is all the resolution the sample can give.
In plain terms: these are the same number. Three model generations, a genuine leap in coding capability from oldest to newest, and no measurable change in how often the resulting app ships a serious security flaw.
We want to be honest that an earlier, rougher version of this analysis showed Opus 4.5 at 18.6%, which would have made the slope look steeper and more tempting to write up. When we reviewed those findings by hand, two of the flagged apps were false positives, translation files and config templates our scanner misread. Fixing that (and re-running all three cohorts through the corrected scanner) pulled the number down and flattened the curve. The trend that looked publishable did not survive contact with manual review. That is the whole reason we do the manual review.
Why capability doesn't move the number
The findings are dominated, in every cohort, by the same short list: hardcoded credentials, weak randomness in a security-sensitive spot, disabled TLS verification, and hardcoded session secrets. These are not reasoning failures. A smarter model is better at the parts of programming that reward reasoning: structuring the app, handling edge cases, writing correct logic. None of that is what puts a Supabase key in a committed file. A human does that, to make something work right now, meaning to fix it later. The model was never the one deciding to hardcode the secret, so making the model smarter doesn't stop it.
That is why the number is flat, and it is why we would expect the next generation after Fable 5 to land in the same band too, unless something changes about the workflow rather than the model.
What this means if you ship with any of them
The practical takeaway does not depend on which model you use, which is the point. The security of an AI-assisted app is set by the human review the code still needs before it ships, and that need is constant across model versions. Before a repo goes public: pull every secret out of source and into an environment variable, rotate anything that was ever committed (it is in your git history even after you delete it), use a cryptographic random source for anything security-sensitive, and never ship with certificate verification turned off.
Methodology and limits
118 Fable 5, 144 Opus 4.8, and 102 Opus 4.5 web apps, sourced from GitHub via each model's Claude Code commit trailer, filtered to web applications by a rule fixed before any results were seen, and scanned with Prbl. Every high-severity finding was manually reviewed, and the false-positive patterns that review surfaced were fixed in the scanner and the cohorts re-scanned before these numbers were computed. The anonymized aggregate data is available as a CSV.
The limits, stated plainly: this is observational, not a controlled experiment. The three cohorts are different apps by different people, the sample sizes are modest, and Claude Code users skew more technical than the general population, so these rates describe apps built by developers who reach for a coding agent, not all software. No specific repository, owner, or file is identified anywhere in this post; every pattern described is a generalization across many apps, not a quote from one.
The models are getting better at writing code, quickly. The rate at which that code ships a secret it shouldn't is holding still. The gap between those two facts is the review step, and it is still yours to do.