We scanned 83 apps built with OpenAI's Codex. They weren't the apps we expected.

We recently ran a head-to-head study of Codex against three Claude models and found no meaningful difference in the high-severity flaw rate. That post was about the numbers. This one is about something we noticed while collecting them that the numbers alone don't show: the Codex apps were not the same kind of apps.

What most AI-built apps look like

Across our other studies, the app-builder scans and the Claude web-app scans, the median AI-generated project is a consumer web app. A frontend framework, a hosted database, user accounts, a dashboard. The dominant security problem fits that shape: a backend key hardcoded into the client, a missing authorization check on a generated route, a session secret left as a default. It is the security profile of someone building a product.

What Codex was building instead

The Codex sample skewed somewhere else entirely. Filtering to apps that expose a web interface, the repositories were dominated by machine learning and AI infrastructure: model-serving endpoints, fine-tuning and dataset-preparation scripts wrapped in a small API, local inference servers, and agent and evaluation frameworks. These are not products with a login screen. They are tools that developers build for themselves and other developers.

That is a meaningful signal about who reaches for Codex and what for. In our sample it looked less like a way to ship a consumer app and more like a power tool for people already deep in the ML and developer ecosystem. We would want more data before treating that as a hard fact about the whole user base, but it was striking enough in the scan to call out.

How the security profile shifts with the code

Different software has different failure modes, and the shift showed up in the findings. The classic consumer-app problem, a database key hardcoded into a frontend with no row-level security behind it, was rare here, because these apps mostly don't have that shape. What took its place were the mistakes native to service-and-tooling code:

• TLS verification switched off on connections between internal services, the kind of “just make it connect” shortcut that is easy to leave in when the two ends are your own.
• Weak randomness used for identifiers and tokens, Math.random() standing in where a cryptographic source belongs.
• Unsafe dynamic operations, dynamic SQL assembled for a schema migration, object keys assigned from external input, the patterns that turn into injection or prototype pollution when the input is attacker-controlled.

A subtle consequence: this kind of code is genuinely harder to scan well. ML and tooling repositories are full of things that look like secrets but are not, placeholder API keys like "not-used-by-llama-server", local test keys, status labels like "(configured)". A scanner tuned on consumer web apps will flag those as leaked credentials and be wrong. Getting an honest read on Codex-built code meant teaching our scanner the difference, which we did before publishing any number.

The part that did not change

Here is what is striking. The software is different, the failure modes shifted, and yet the root cause is identical to every other cohort we have measured. Every real high-severity finding in the Codex apps was a value a human hardcoded or a check a human turned off to make something work. Not a flaw the model reasoned its way into. A more capable model, or a different vendor's model, does not change that, which is exactly why the head-to-head rate came out flat. The tool changes what you build. It does not change the fact that the security-relevant shortcuts are yours.

If you build with Codex, check these

The checklist shifts slightly for tooling and service code:

• Grep for rejectUnauthorized: false, verify=False, and CERT_NONE. Internal service connections are where TLS gets disabled and forgotten.
• Check every random value that identifies a session, request, or job. If guessing it grants access, it needs crypto, not Math.random().
• Look at any query or object key built from a variable. In a migration script it is probably fine; if that variable can come from a request, it is an injection.
• And still: pull real keys into environment variables. The rate is lower in this kind of code, but a leaked model-provider or cloud key is just as costly.

Methodology and limits

83 web-interface apps sourced from GitHub via the OpenAI Codex commit trailer (Co-authored-by: Codex <[email protected]>), verified as majority Codex-authored, filtered to web applications, and scanned with Prbl. Every high-severity finding was manually reviewed and the false-positive patterns that review surfaced were fixed before the numbers were finalized. This is a smaller sample than our Claude cohorts, and the observation about what Codex builds is exactly that, an observation from this sample, not a proven claim about all Codex users. No specific repository, owner, or file is identified anywhere in this post. The full cross-model comparison, including the significance tests, is in the companion study.