We scanned 119 web apps built with Claude Fable 5. The model got smarter. The secrets still leaked.

Every model release comes with a benchmark jump. Fewer bugs, better reasoning, cleaner code. A fair question for anyone shipping software with these tools is whether that curve bends toward security, or just toward code that runs. So we measured it, on real code, with our own scanner.

How we found the apps

When you commit code with Claude Code, it stamps the commit with a co-author trailer naming the exact model. That trailer is a structural artifact of the tool, not something we keyword-matched, so it lets us attribute a repository's code to a specific model generation with commit-level precision. We searched GitHub for repositories whose commits are majority-authored by Claude Fable 5, filtered to the ones that are actually web applications (a React or Vue frontend, an Express or FastAPI or Django backend, a real database), and scanned every one. We pulled a parallel set built with the previous generation, Claude Opus 4.8, the same way, to have something to compare against.

The web-app filter matters and we fixed it before looking at any results: we only scan applications, not libraries, CLIs, or developer tooling. Those have different security profiles and are not what most people mean when they ask “is my app safe.” That left 119 Fable 5 web apps and 145 Opus 4.8 web apps.

The result

12.6% of the Fable 5 web apps had at least one high-severity security finding. Fifteen of 119. When we ran the same scanner over the Opus 4.8 set, 15.2% did. At these sample sizes those two numbers are the same number: the newer, more capable model did not measurably change the rate.

That is the interesting part, so it is worth sitting with. Fable 5 is a genuine step up in coding ability. If the security issues in AI-assisted apps were reasoning failures, subtle logic the model gets wrong, you would expect a smarter model to make fewer of them. It didn't. Which tells you something about what these findings actually are.

What we found

Almost everything was a secret that should not have been in the repository, or a transport-security setting that should not have been off. Here is the Fable 5 breakdown by finding class, counted as affected repositories.

The specific patterns, described without identifying any repository:

• Hardcoded backend keys. The most common finding by far. A Supabase project URL and its key written directly into committed source, sometimes copied across a handful of debug scripts in the same repo. This is the same pattern we found dominating our study of app-builder platforms, now showing up in hand-written code too.
• Hardcoded session secrets. A framework's signing key set to a fixed string in the source, the value that signs every user's session, sitting in git.
• Weak randomness where it counts. Math.random() used to generate an OAuth state parameter (the value that protects the login flow from cross-site request forgery) and a password-reset token. Math.random() is predictable; these values need a cryptographic source.
• TLS verification turned off. A database or mail connection configured to skip certificate checks, which removes the protection against a machine-in-the-middle intercepting the connection.

None of these are things a model reasons its way into. They are things a developer pastes in to make something work locally, means to swap for an environment variable later, and commits before later arrives. A smarter model writes the surrounding application more competently. It does not stop a human from hardcoding a key, because it was never the one deciding to.

Where Opus 4.8 fits

We include the previous generation only as a reference point, and the point it makes is the headline: two model generations apart, the security finding rate is flat. Opus 4.8 came in slightly higher (15.2% versus 12.6%), but the gap is within the noise for these sample sizes, and the finding mix is identical, the same secrets and the same TLS settings, in the same proportions. This is not a story about one model being safer than another. It is a story about a class of problem that model capability doesn't touch.

Why our number is believable

Developer-written web apps are a harder scanning target than template-generated ones, because real developers write migration scripts, dynamic query helpers, and CLI utilities whose shape can superficially resemble a vulnerability. Getting an honest number meant making our scanner tell the difference. Before publishing, we reviewed every single high-severity finding by hand and traced each false positive back to a specific gap in the scanner, then fixed it.

The largest of those fixes was adding data-flow (taint) analysis to our injection and traversal rules: instead of asking “does this line look like a dangerous query,” the scanner now asks “does user input actually reach it.” A query built from a hardcoded column-name constant looks identical to one built from a request parameter until you follow where the value came from. High severity is now reserved for findings where user-controlled input provably reaches the sink; anything the scanner cannot trace is reported at a lower severity with a note, rather than asserted as high. That single change is why the credential and transport-security findings in this post are the ones that survived: they are the findings that don't depend on data flow to be real.

What to actually do about it

The fix is boring, which is the whole point, because boring is what gets skipped. Before a repo goes public:

• Search your own code for supabase.co, apiKey, SECRET, and long random-looking strings. If a key is in a committed file, it is in your git history even after you delete it, so rotate it, don't just remove it.
• Move every secret to an environment variable and confirm the .env file is in .gitignore before the first push, not after.
• For anything security-sensitive (tokens, session IDs, OAuth state, reset codes), use a cryptographic random source, not Math.random().
• Never ship with certificate verification disabled. If a local cert is the problem, fix the cert, don't turn off the check and forget.

Methodology and limits

119 Fable 5 web apps and 145 Opus 4.8 web apps, sourced from GitHub via each model's Claude Code commit trailer, filtered to web applications by a scope rule fixed before any results were seen, and scanned with Prbl. Every high-severity finding was manually reviewed, and the false-positive patterns that review surfaced were fixed in the scanner across several rounds before these numbers were computed. The anonymized aggregate data is available as a CSV.

The honest limits: this is observational, not a controlled experiment. The two cohorts are different apps by different people, and Claude Code users skew more technical than the general population, so these rates describe apps built by developers who reach for a coding agent, not all software. Our taint analysis works within a single file, so a value that becomes user-controlled through a call in another file can be under-counted. And no specific repository, owner, or file is identified anywhere in this post; every code pattern described above is a generalization across multiple apps, not a quote from one.

The takeaway is not that Claude Fable 5 writes insecure code. It writes good code. The takeaway is that the security of an AI-assisted app is decided by the human review the code still needs, and that need doesn't shrink as the model improves. The model got smarter. The part that leaks secrets is still us.