AI writes the code. Who checks it?

In May 2026 Jarred Sumner had almost all of Bun, a program widely used to build web applications, rewritten from one programming language to another, from Zig to Rust. It was 535,496 lines of code, excluding the notes for programmers. In 11 days, from May 3 to May 14, all the tests passed on every system Bun runs on: they are the automated checks that the program does what it should. The token bill, meaning the text processed by the AI, came to about $165,000 at list price, according to Sumner's own account. Bun has belonged since December 2025 to Anthropic, the company that develops Claude, the AI used for much of the rewrite. Sumner works there: the person reporting the result works for the company that sells the AI that produced it.

The same account then lists 19 known regressions, meaning things that used to work and stopped working, all fixed. Most of them, Sumner writes, come from code that looks identical in the two languages and means something different. An example: a command did a necessary job but had been placed inside a check that the program runs only while it is being developed. In the final version, the one users actually run, the new language skips that check, and with it the command is skipped too. The compiler, the program that translates code into instructions for the computer, considered it valid code.

Andrew Kelley, the creator of Zig, the language Bun left, answered by talking about a million lines of unreviewed code. The figure is his: Bun's post counts about 780,000 lines in the Rust version, starting from the 535,496 of Zig. Bun says every line went through two reviewers tasked with finding faults, but the reviewers were also Claude, the same AI that had written the code. I have no evidence on who is right. The question that remains is a different one: when AI writes the code and another copy of the same model reviews it, what has actually been verified?

The figures below come from the original sources, and many are published by the companies that sell the models: I flag it every time. The ones I found only second-hand I left out.

How much work AI already does and how much a human checks

The companies that use agents the most, meaning AI programs that carry out a job on their own, measure their own use. On September 6, 2026 OpenAI, the company behind ChatGPT, wrote that its research organization, as of mid-August, used 3.1 agent-workdays of effort for every workday of human labor. In the same document, over the last six months more than half of the successful 4-8 hour tasks required at least one human intervention: the people who build AI use it heavily, and still have to check it.

Anthropic measures how much of its own research and development work is led by Claude, its AI model: Claude completes the task starting from a general request, while a person supervises. According to Anthropic it is 26% of the work, calculated on people's time in a sample from July 2026. The judge that rates the tasks is itself Claude. The page admits it: a judge model can make the same mistakes as the model it checks (“could make the same kinds of errors as the model it is checking”).

A paper with 22 authors, including the researchers Yoshua Bengio and Geoffrey Hinton, describes the flip side. When people leave a job, they lose the opportunities and the expertise to notice problems and fix them (“they could lose both the opportunities and expertise needed to identify and fix problems”). Also, using AI reliably to oversee AI remains an unsolved challenge (“remains an unsolved challenge”). The paper is about AI research and not about business software. To me it looks like the same mechanism that anyone buying generated code runs into: whoever stops doing a job loses the expertise to check it.

Three ways to check code, and where each one stops

A May 2026 whitepaper signed by Addy Osmani, Shubham Saboo and Sokratis Kartakis closes with the sentence that sums up the work that remains: “Generation is solved. Verification, judgment, and direction are the new craft.” Among the checks it lists automated tests and model-based judges. It describes checking the result with two questions: whether the code is accepted by the compiler and whether the tests pass (“does the code compile, do the tests pass?”). It does not mention formal checks, the ones in which a dedicated program proves that the code follows a rule written in a rigorous way. In the other sources the routes are three: tests and measures on the code, the compiler with formal checks, a person's judgment.

Measures on the code

Robert Martin, author of the book Clean Code, said on the Aviator podcast (September 24, 2026) that he never sees 95% of the code, because reviewing it “slows everything down like crazy”. In place of reading he uses automated checks. They are tests, measures of complexity and of coverage (how much of the code the tests touch), and a trial that alters the code on purpose to see whether at least one test notices.

These measures say how complex the code is and how much of it the tests touch, and they do not say whether the tests ask the right thing. Kent Beck, inventor of Extreme Programming, a well-known development method, wrote an essay on the disciplined use of AI in programming. Among the warning signs he lists an agent that seems to cheat, for example by disabling or deleting tests (“disabling or deleting tests”). The rule I take from it: if the agent can change the tests that judge it, the check depends on the agent.

The language and the compiler

The second route is to use a language in which the compiler checks more things before accepting the code. Sumner picks Rust for Bun for exactly this reason: many classic mistakes in how the computer's memory is handled are flagged by the compiler before the program runs. For him compiler errors are a better feedback signal than a rulebook (“Compiler errors are a better feedback loop than a style guide”). Bjarne Stroustrup, the creator of C++, a language widely used for large systems, pushes it further. On Ryan Peterman's podcast (May 18, 2026) he says what is needed is a language as precise as mathematics, not English (“a programming language that's precise, that's engineering, that's math; it's not English”).

Research tries to make these checks even more rigorous, and the results are partial. On a 2025 benchmark of 12,504 rules written in a rigorous form, general-purpose models produce the code and the proof that it is correct from 27% to 82% of the cases, depending on the tool. Even when the check succeeds, it confirms that the code follows the written rule, and the rule remains a person's responsibility.

The compiler, however, checks that the code is consistent, not that it does what was wanted. That is what happened to Bun: the command that ended up inside the development check was valid code. The rewrite of the program behind GitHub Copilot, the programming assistant, shows it at scale (Stephen Toub, GitHub, September 16, 2026). It is more than 800,000 lines in 128 batches of changes approved one at a time, with regressions that compilation had let through. Toub groups them into three families: behavior different from what the rest of the system expected, the way the program keeps track of data changed, parts forgotten along the way. He also reports tests that confidently approved the wrong behavior (“confidently validated the wrong behavior”).

Human judgment

The third route is the person. Beck separates vibe coding from augmented coding. In the first, you have the AI write the program while caring only that it works (“you don't care about the code, just the behavior of the system”); in the second, you stay attentive to the code, its complexity and the tests (“you care about the code, its complexity, the tests, & their coverage”). In this second mode the check is the person.

The cost of this route is time. Stroustrup says on the podcast that a small change to the request made to the AI is enough to change a good part of the code, and that then everything has to be checked again (“you have to check it again”). According to a survey commissioned by Undo from Coleman Parkes among the leaders of software that large companies depend on (300 respondents, July and August 2026), 35% of AI-generated code goes into use without being fully understood. That is the figure the respondents report. Undo sells tools for finding and fixing errors in software, and the survey should be read with that interest in mind. Where the people able to understand that code will come from is the subject of another article.

The check that is almost always missing: an independent reference

An independent reference is a system known to be correct that does not share the mistakes of whoever wrote the code. The clearest case was published by Anthropic on February 5, 2026 (Nicholas Carlini). With 16 agents and just under $20,000 in costs, Anthropic produced a 100,000-line compiler for the C language, able to translate the Linux kernel, the core of the operating system. Here too the experiment belongs to the company selling the model.

The work stalled when all the agents hit the same error in the kernel and each one overwrote the others' fixes. The solution was to use GCC, a mature and widely used compiler, as a reference: most of the kernel was translated by GCC and only a part by Claude's compiler. If the kernel broke, the fault was in that part, and the field was narrowed. Carlini writes that the task verifier has to be nearly perfect, otherwise Claude will solve the wrong problem (“the task verifier is nearly perfect, otherwise Claude will solve the wrong problem”). He adds that with autonomous systems it is easy to see tests pass and assume the job is done, when it rarely is (“it is easy to see tests pass and assume the job is done, when this is rarely the case”).

An independent reference can be a mature compiler, the old system that still runs, data measured outside the code, a reviewer who has not seen the work. To me that is what sets it apart from a test written by the agent: its mistakes do not coincide with the code's. The Bun case is the opposite example: the reviewers were Claude, and Anthropic writes that a judge based on the same model can be wrong like the model it checks.

What I ask a vendor before accepting AI-written code

When someone offers me code written largely by agents, the first thing I want to know is what checks it and who wrote that check. In the table the first three rows are the three routes, and the last is the comparison with something external, which is where I start.

If the same agent stands behind all four answers, what reaches you is code that agrees with itself. I always start from the last row: without an external reference, the other three checks can all pass and confirm the same error. In a project that replaces an existing system the best reference is often that system, kept running: it is the same scheme as GCC in Anthropic's compiler.

In short

When AI writes the code, the work that remains is checking it, and there are three ways to do it: tests and measures on the code, compiler and formal checks, a person's judgment. Each certifies a different thing, and a check written by the same agent that wrote the code can share its mistakes. The most useful comparison is with something external that is already reliable, such as the previous system kept running: if you buy AI-generated code, ask who wrote the check, besides whether one exists.