# ALMOST RIGHT ## How AI Is Quietly Removing the People Who Check Its Work **Anthony C. Vila** --- > "The single biggest frustration developers report with AI coding tools is > output that is **almost right, but not quite.**" > > — Stack Overflow Developer Survey, 2025 (66% of respondents) --- ## Contents **PART I — THE GUESSING MACHINE** 1. The Bet at Dartmouth 2. Two Winters 3. The Real Reason 4. Almost Right, By Design **PART II — THE PLUNGE** 5. Faster Than the Internet 6. Nobody's Guarding the Door 7. The Sellers 8. The Two Countries **PART III — EVERYBODY BUILDS NOW** 9. What Actually Works 10. Vibe Coding 11. Nobody Hacked Them **PART IV — NOBODY'S CHECKING** 12. The Middlemen 13. Cognitive Debt 14. The Doctors Got Worse 15. The Canaries **PART V — THE VERIFICATION ECONOMY** 16. When Nobody Knows What Right Looks Like 17. The Apprenticeship Problem 18. The Factory for Almost Right 19. The Liability Gap 20. The Verification Economy 21. The Case for Optimism **PART VI — HOW WE CHECK** 22. What the Pilots Did 23. Become the Verifier 24. Start Now --- ## A Note on Sources Material factual claims in this book are traced, wherever possible, to primary sources — the study itself, the court filing, the company's own statement, or the survey with its sample size attached — and each chapter ends with sources and further reading. This is not academic habit. It is the argument of the book applied to the book. A work about confident claims that outrun their evidence cannot afford to make one. Where the evidence is thin, I say so. Where a study has been criticized, I give you the criticism before I give you the finding. Where I am offering an opinion rather than a fact, I label it. If you find something in here that is wrong, I want to know. That is the whole point. --- \pagebreak # Preface # Why I Started Checking I did not come to this subject as an engineer, an academic, or somebody looking for a reason to distrust artificial intelligence. I came to it as a user. I had spent most of my adult life in sales. Then I started building with AI. From a phone, without an engineering background, I could produce software and business systems that would previously have required people with skills I did not have. The capability was real. So was the leverage. Then I learned that producing something and knowing whether it works are different jobs. The machine could generate faster than I could inspect. It could give me an answer that looked finished while leaving behind a small failure I did not know to look for. The more capable the output became, the easier it was to mistake plausibility for verification. I found myself spending hours checking work that had taken the machine seconds to produce. That was the question that became this book: what happens when production becomes nearly instantaneous but judgment does not? I went looking for the answer in software, law, medicine, education, labor economics, aviation, security, and the history of artificial intelligence itself. I found the same tension in places whose researchers were mostly not talking to one another. This book is not an argument to stop using AI. I use it. I would use it again. It is an argument to preserve the thing the tool still depends on: people who can tell when an answer that looks right is not right. --- # A Note on What This Book Is — and Isn't There are two easy books I could have written. One says artificial intelligence is going to save everything. The other says artificial intelligence is going to destroy everything. Both would be simpler. Neither is the book the evidence gave me. The systems described in these pages are genuinely useful. They make some workers faster. They help some beginners perform tasks that previously required more experience. They can expand access to knowledge, lower the cost of creating software, and give an ordinary person leverage that would have looked absurd a few years ago. They also make mistakes. That fact alone is not interesting. People make mistakes. The interesting part is the combination: these systems can produce professional-looking work at enormous speed while remaining unreliable in ways that are difficult to detect from appearance alone. That changes the economics of checking. It changes training. It changes responsibility. It changes what expertise is for. This book is an attempt to follow those consequences without pretending the evidence is cleaner than it is. Some of the evidence is experimental. Some is observational. Some comes from labor-market administrative data. Some comes from court records, incident reports, professional surveys, security benchmarks, and institutional guidance. Those forms of evidence do not deserve identical confidence. When a randomized trial establishes a result in a narrow setting, I try to keep the claim narrow. When an observational study shows an association, I do not want to call it causation. When a vendor benchmark reports a failure rate, I treat it as a benchmark result rather than a universal law. When a labor-market pattern is suggestive but still developing, I want the uncertainty visible in the sentence. That standard matters because this book is about verification. It would be ridiculous to argue that plausible output should be checked and then hide the caveats that make my own argument less dramatic. So read the numbers as evidence, not decoration. Read the stories as examples, not proof that every organization behaves the same way. Read the predictions as predictions. And where the evidence changes, the conclusion should be allowed to change with it. That last part matters especially in AI. The technology is moving quickly enough that a benchmark can become stale while a book is still being edited. METR's early-2025 developer study found a slowdown in one population and setting; its later work suggested newer tools may be faster while warning that selection effects made the later estimate weak. The correct response is not to choose the result that best fits the thesis. It is to preserve the timeline. The question underneath the changing benchmark is more durable: **When machines become capable of producing more work, what happens to the systems that determine whether the work deserves to be trusted?** That question survives whether the next coding model is twenty percent faster or two hundred percent faster. In fact, the faster the generator becomes, the more urgent the verification question becomes. I am also not arguing that every task needs an expert committee. Most AI use is low stakes. If the restaurant recommendation is bad, eat somewhere else. If the first draft is clumsy, rewrite it. If the brainstorming list contains nonsense, delete the nonsense. Verification should be proportional to consequence. The problem begins when the same casual relationship to generated output migrates into software, law, medicine, finance, education, public policy, security, or autonomous systems where an error can travel farther than the person who clicked Generate. That is where "almost right" stops being an annoyance. It becomes an operating condition. The rest of this book is about what to do with that condition. The distinction I want the reader to carry forward is simple. **Output is not verification. Fluency is not evidence. Assistance is not competence. Oversight is not meaningful merely because a human name appears at the end of the process.** Each of those pairs can overlap, but they are not interchangeable. That sounds obvious when written plainly. In practice, modern AI products are extraordinarily good at making the distinction disappear. They collapse research, drafting, explanation, calculation, recommendation, and presentation into one smooth interaction. The convenience is the product. The danger is that the user can lose track of which step actually established the truth of the answer. So throughout the chapters that follow, watch for the handoff. Watch the moment when generated material becomes relied-upon material. That handoff is where the economics, the liability, the training problem, and the verification problem meet. Not stop the machine. Not worship the machine. Build the checks. And keep checking the checks. The point is not to freeze today's rules around tomorrow's technology. It is to preserve a method: measure what the system actually does, identify what matters when it fails, maintain independent ways to detect those failures, and change the controls when the evidence changes. A verification culture is not suspicious of progress. It is how real progress becomes dependable enough to trust over time. --- # PART I — THE GUESSING MACHINE --- # Chapter 1 # The Bet at Dartmouth On August 31, 1955, four men put their names on a proposal for a summer research project and sought support from the Rockefeller Foundation. The surviving typescript runs seventeen pages plus a title page — not the tidy two-page origin story it is sometimes reduced to. They were not cranks. John McCarthy was a young mathematician at Dartmouth. Marvin Minsky was at Harvard. Nathaniel Rochester had helped design IBM's first commercial scientific computer. Claude Shannon, at Bell Telephone Laboratories, had already invented the mathematics that every phone call, every hard drive, and every internet packet still runs on. If you wanted four people in 1955 who understood what a machine could and could not do, you would have had a hard time doing better. Here is what they wrote: > "We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer." Read that last sentence again. A summer. They asked the Rockefeller Foundation to cover it. The budget included salaries of $1,200 for each faculty-level participant who wasn't already being paid by somebody else. McCarthy and Shannon had already gone to New York that June to sit down with a man named Robert Morison at the foundation and make the case in person. The document is the first time the phrase "artificial intelligence" appears in the historical record. McCarthy picked the name. He needed something that would sound like a field, not a hobby, and he needed it to not sound like anyone else's field. It worked. Seventy-one years later, that name is on the front page of every newspaper on earth, attached to companies worth more than the economies of most countries. But I want you to sit with the bet itself, because the bet is the whole story. Four of the smartest people alive looked at the problem of human intelligence — language, abstraction, concepts, the ability to improve yourself — and estimated that ten people could make "a significant advance" on it in ten weeks. Not solve it. They were careful about that. But make real progress. Over a summer. In New Hampshire. They were off by roughly seven decades. And I would argue they are still off, in a way that matters more now than it did then, because in 1956 the only thing riding on the bet was a Rockefeller grant. Today it's your job, your kid's homework, your doctor's judgment, and the software that holds your bank balance. ![Figure 1.1. The Dartmouth Proposal (1955)](ALMOST-RIGHT-Figures/Figure-1.1-dartmouth-proposal.png) *Figure 1.1. Title page of A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955. Source: John McCarthy historical archive; public-domain source file via Wikimedia Commons. U.S. public domain: published in the United States between 1931 and 1977 without a copyright notice. Facsimile prepared for this edition; international rights can differ.* --- ## What happened that summer Not much, and that's not an insult. The workshop happened. Eleven people were originally planned to attend; more than ten others drifted through for shorter visits over the course of the summer. Some of the names on the guest list would go on to define the field for the next fifty years. They argued. They wrote on chalkboards. They disagreed about what "thinking" even meant and about whether the problem was mostly logic or mostly learning. There was no final report. I want to be fair to them, because this book is going to be hard on a lot of people who deserve it, and these four don't. They were doing what scientists are supposed to do: take a wild idea seriously enough to test it. The Dartmouth proposal is one of the most consequential documents of the twentieth century precisely because it was wrong in an interesting way. It set the agenda. Every argument you will hear about AI in 2026 — can it think, does it understand, will it replace us, is it dangerous — was on a chalkboard in Hanover in the summer of 1956. But I also want you to notice something about the shape of the bet, because you're going to see this shape again and again in the chapters ahead, and it's going to cost real people real money and real careers. The bet was: *intelligence is describable, therefore intelligence is buildable, therefore we are close.* The first part is a philosophical position. The second is an engineering claim. The third is a sales pitch. And the trick — the thing that has been happening for seventy years — is that people who believe the first part let it carry them straight through to the third without stopping to check whether the second is true. --- ## Before Dartmouth: the man who asked the question The conjecture didn't come from nowhere. Six years earlier, in October 1950, a British mathematician named Alan Turing published a paper in the philosophy journal *Mind*. It's called "Computing Machinery and Intelligence," and it opens with a question that Turing himself immediately says is too muddy to answer: *Can machines think?* Turing's move was to replace the question with a game. Put a person in one room and a machine in another. Let a judge in a third room type questions to both and read their typed answers. If the judge can't reliably tell which one is the machine, then — Turing argued — arguing about whether the machine "really" thinks is a waste of everyone's time. It's doing the thing. What else do you want? This is the imitation game, and it has been misread for seventy-five years, so let me say plainly what it is and isn't. It is not a definition of intelligence. Turing knew that. It is a test of *indistinguishability* — of whether a machine's output can pass for a human's. Turing proposed it because he thought the philosophical argument was unwinnable and the practical question was the only one worth having. Hold onto that, because it's the seed of everything. From the very first serious paper in the field, the goal was not "build a machine that understands." The goal was "build a machine whose output you can't tell apart from someone who understands." In 1950 that seemed like the same thing. In 2026 it is the single most important distinction in your life, and almost nobody talks about it. --- ## 1958: The Navy's machine that would be conscious Two years after Dartmouth, the bet got its first press tour. On July 7, 1958, a psychologist named Frank Rosenblatt gave a demonstration in Washington. Rosenblatt worked at the Cornell Aeronautical Laboratory, and the Office of Naval Research was paying for his work. He had built something he called a perceptron — a machine that could learn to tell the difference between simple patterns by adjusting its own internal weights when it got an answer wrong. It was, in the plainest sense, a machine that got better with practice. That was new. *Photo 1.2. Frank Rosenblatt with the Mark I Perceptron, the experimental pattern-recognition machine developed at Cornell Aeronautical Laboratory under U.S. Navy sponsorship. Photograph released June 24, 1960. U.S. Navy / National Museum of the U.S. Navy. Public domain in the United States (U.S. federal government work). License/status: https://creativecommons.org/publicdomain/mark/1.0/ Source: https://commons.wikimedia.org/wiki/File:330-PSA-80-60_(USN_710739)_(20897323365).jpg* The next morning, *The New York Times* ran the story under the headline "NEW NAVY DEVICE LEARNS BY DOING." The subhead promised a computer "Designed to Read and Grow Wiser." Here is the first sentence, verbatim: > "The Navy revealed the embryo of an electronic computer today that it expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence." The Navy. Expects. Conscious of its existence. The article went on to describe the first full perceptron as a machine with about a thousand "association cells," fed by an eye-like device of 400 photocells, estimated to cost around $100,000 to build. *The New Yorker* weighed in too, calling it "the first serious rival to the human brain ever devised." Now — what had Rosenblatt actually built? A machine that could learn to sort simple visual patterns into two piles. Left from right. Square from triangle. It was a genuine scientific achievement, and Rosenblatt's paper describing the mathematics, published that same year in *Psychological Review*, is a real piece of work. The idea inside it — that you can build a network of simple units, show it examples, and let it adjust itself until it gets the answers right — is the direct ancestor of every AI system you have used this week. But the distance between "sorts simple patterns into two piles" and "conscious of its existence" is not a gap in engineering. It is a gap in *honesty*. And that gap was not created by Rosenblatt's machine. It was created by the people describing Rosenblatt's machine to the public — a funding agency, a newspaper, a magazine — each of whom had a reason to make it sound bigger than it was. I sold things door to door for a living before I ever touched any of this. I know what a pitch sounds like. That first sentence in the *Times* is a pitch. It's a very good one. And it set the template that the AI industry has followed, with remarkable discipline, for sixty-eight years: Build something real. Describe something imaginary. Let the reader close the gap themselves. --- ## Why this chapter matters to you You might be wondering why a book about what AI is doing to you *right now* opens with a grant proposal and a newspaper clipping from the Eisenhower administration. Here's why. Every time you read a headline about AI in 2026 — a CEO saying it will eliminate half of all entry-level jobs, a researcher saying it will make us all smarter, a lab saying its new model is "approaching" something-or-other — you are reading a descendant of that *Times* article. The genre was invented in 1958. The structure has never changed. Something real gets built. Something enormous gets promised. The gap between the two is where the money is, and the gap is *your* problem, not theirs. And there's a second reason, which is the one this whole book is about. The Dartmouth proposal set out to make machines that could "use language, form abstractions and concepts, solve kinds of problems now reserved for humans." Turing's test only asked that the machine be indistinguishable from someone who could. Rosenblatt's perceptron did neither — it learned to give the right *output* on simple patterns, without anything inside it that you or I would call a concept. Guess which of those three the industry actually built. Not the Dartmouth version. Not a machine that forms concepts. The Turing-Rosenblatt version: a machine that produces output you can't tell apart from a person's, by adjusting itself until its answers look right. That is not a criticism. It is a description. And it has a consequence that the next three chapters will spell out, but that I'll give you now so you can carry it with you: **A machine built to produce answers that look right will, by design, produce answers that look right when they are wrong.** That is not a bug somebody forgot to fix. It is the finish line the field was running toward since 1950, and in 2022 it crossed it. In September 2025, OpenAI — the company that put this technology in front of the world — published a research paper on why its own systems make things up. The paper's explanation, in its own words, was that these models "hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty." The paper opens with a comparison I'll ask you to remember: like students facing hard exam questions, the models guess when they don't know, "producing plausible yet incorrect statements instead of admitting uncertainty." Plausible yet incorrect. Almost right. The men at Dartmouth thought they were describing intelligence. What they were actually describing — what the whole field would spend seventy years perfecting — was a machine for producing plausibility. It turns out plausibility is enormously valuable. It turns out you can sell it for hundreds of billions of dollars. And it turns out that a society which stops being able to tell plausibility from truth is in a very specific kind of trouble that nobody in Hanover in 1956 was thinking about, because in 1956 there were still going to be humans checking the work. This book is about what happens when there aren't. --- ## What comes next The perceptron got its press tour in 1958. Eleven years later, Marvin Minsky — one of the four names on the Dartmouth proposal — co-wrote a book that proved, mathematically, what a machine like Rosenblatt's *couldn't* do. The funding dried up. The field went into what its own people still call a winter. It came back. It went into a second winter. It came back again — and the thing that finally brought it back was not a better idea about intelligence. It was more data and more chips than anyone in 1956 could have imagined. Which is a fact the industry would prefer you not dwell on, for reasons that will become obvious. That's Chapter 2. --- ## Sources and Further Reading McCarthy, Minsky, Rochester & Shannon, "A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence," dated August 31, 1955 (archived at Stanford; reprinted *AI Magazine* 27(4), 2006). Turing, "Computing Machinery and Intelligence," *Mind* 59(236), October 1950. *The New York Times*, "New Navy Device Learns by Doing," July 8, 1958. Rosenblatt, "The perceptron: a probabilistic model for information storage and organization in the brain," *Psychological Review* 65(6), 1958. Kalai, Nachum, Vempala & Zhang, "Why Language Models Hallucinate," arXiv:2509.04664, September 4, 2025. --- \pagebreak # Chapter 2 # Two Winters In 1969, one of the four men who signed the Dartmouth proposal killed the machine that had gotten the field its first headlines. Marvin Minsky, with his MIT colleague Seymour Papert, published a book called *Perceptrons*. It is a mathematics book, dense and careful, and its most important result is a proof of what Frank Rosenblatt's machine could *not* do. A single layer of those self-adjusting units — the thing the Navy said would become conscious — could not learn certain simple patterns no matter how long you trained it. Not "hadn't yet." Couldn't. Minsky and Papert were right. The math holds. And the effect on the field was roughly what happens to a sales office when the top producer announces the product doesn't work. The money left. This is the first thing to understand about the history of artificial intelligence, and it's the thing the industry would least like you to dwell on: **the field has collapsed twice.** Not slowed. Collapsed — funding cut, labs closed, the phrase itself becoming something researchers avoided putting on grant applications because it marked you as a person who overpromised. The people who lived through it called those periods "AI winters," and the term is still in use, because the people who use it are still waiting to see whether there's a third. --- ## The first winter The perceptron's collapse in the United States was matched, almost on schedule, in Britain. In 1973, the UK government asked a mathematician named James Lighthill to evaluate the state of AI research and report on whether it deserved continued public funding. Lighthill's report, published by the Science Research Council in 1973, put its verdict in one sentence: "In no part of the field have the discoveries made so far produced the major impact that was then promised." Historians have summarized his charge as AI failing to meet its "grandiose objectives," and the effect was the same either way: British funding for AI research was gutted for the better part of a decade. In the U.S., the Defense Department's research arm — the main source of money since the beginning — pulled back sharply in the mid-1970s. The pattern was the same everywhere: a decade of promises measured against a decade of demos, and the promises lost. What had gone wrong? Nothing, in one sense. The researchers of the 1950s and '60s had done real science and learned real things. What had gone wrong was the *bet* — the same bet from Chapter 1. *Intelligence is describable, therefore buildable, therefore we are close.* They'd been running on the third clause for fifteen years and had not delivered the second. --- ## The second winter The field came back in the 1980s with a new idea and a new pitch. The idea was the "expert system": instead of trying to build general intelligence, you would sit down with a human expert — a doctor, a chemist, a loan officer — write down their decision rules as a long list of if-then statements, and put that list in a computer. The computer would then make expert decisions without the expert. It worked, sort of, in narrow places. Companies bought it. Japan launched a national program, the Fifth Generation Computer Systems project, in 1982, with the stated aim of leaping past the United States in intelligent computing within a decade. By the early 1990s it was over again. The expert systems turned out to be brittle — they broke the moment a situation fell outside their rules — and expensive to maintain, because the rules had to be rewritten by hand every time the world changed. Japan's Fifth Generation project wound down in 1992 without the leap. The companies that had sold expert systems either folded or quietly renamed what they did. Second winter. I want to pause on the expert systems, because they're the closest ancestor to the thing you're using today, and the way they failed is instructive. An expert system was, literally, a human expert's judgment written down and run by a machine. It didn't have judgment of its own; it had a recording of someone else's. When the recording matched the situation, it was as good as the expert. When it didn't, it was worse than a first-year trainee, because a trainee at least knows when they're out of their depth. The expert system didn't know. It just applied the rules and gave you an answer, with the same confidence whether it was right or catastrophically wrong. Keep that in your pocket. We are going to meet a much more powerful version of that exact failure in Chapter 4. --- ## The idea that was sitting there the whole time Here is the part of the story that should make you uneasy. While the expert-system money was flowing, a small number of researchers kept working on Rosenblatt's discredited idea: networks of simple units that adjust themselves. Minsky and Papert had proven a *single layer* couldn't learn much. But what about many layers, stacked? The problem was that nobody had a good method for training the deeper layers — for figuring out which of thousands of internal connections to adjust when the final answer came out wrong. In 1986, three researchers — David Rumelhart, Geoffrey Hinton, and Ronald Williams — published a paper in *Nature* describing a method that did exactly that. It's called backpropagation. In plain terms: when the network gets an answer wrong, you measure how wrong, and you push that error backward through every layer, nudging each connection a little in the direction that would have made the answer less wrong. Do that millions of times and the network learns. That paper is the technical foundation of every AI system you've used this week. It was published forty years ago. Three years later, in 1989, a researcher named Yann LeCun used a version of the technique to get a network to read handwritten digits — the kind on the front of a check. In 1997, two German researchers, Sepp Hochreiter and Jürgen Schmidhuber, published a design called the Long Short-Term Memory network that let these systems handle sequences — text, speech, anything where order matters. So by 1997, the core ideas were in print. The methods worked. The people who would later win the field's highest prizes for them were already publishing. And almost nobody cared, because the networks were too small and too slow to do anything a customer would pay for. The researchers who stuck with it through the 1990s and 2000s did so on thin funding and thinner respect. Hinton has said, in interview after interview, that for years he could barely get his students' papers accepted at the field's own conferences. The ideas weren't the bottleneck. Something else was. --- ## 2012: What actually changed In 2012, a graduate student of Hinton's named Alex Krizhevsky entered a competition. The competition was called the ImageNet Challenge. Researchers were given a dataset of millions of photographs, each labeled with what it showed — a dog, a truck, a mushroom — and asked to build software that could label new photographs it had never seen. Every year the best teams in the world competed. Every year the error rates crept down by a point or two. Krizhevsky, with Ilya Sutskever and Hinton, entered a deep neural network — many layers, trained with the 1986 method — that had been trained on graphics cards built for playing video games. Their system's top-five error rate was 15.3 percent. The next-best entry in the competition came in at 26.2 percent. That is not a creep. That is the floor falling out. In one year, one team cut the error rate nearly in half using an idea that had been sitting in the literature since the Reagan administration. Within two years, essentially every serious team in the competition had switched to deep neural networks. Within five, the technique had spread to speech, to translation, to medicine. The second winter ended not with a new idea about intelligence but with a graduate student, a pile of gaming hardware, and a dataset big enough to matter. Which brings me to the question this chapter exists to ask. **If the ideas were there in 1986 and 1997, why did nothing happen until 2012?** Because the ideas were never the constraint. Two other things were: the amount of data you could feed the network, and the amount of computing power you could throw at training it. In 1997 neither existed at the necessary scale. By 2012 both did — the internet had produced an ocean of labeled photographs and text, and the video-game industry had accidentally built the chips to process it. I want to be careful here, because that's a line and not a complete account. Plenty of real invention happened in those years — better ways to train deep networks, better architectures, an enormous amount of hard engineering. The researchers were not idle. But none of it amounted to a new theory of intelligence. It was a better answer to *how do we make this thing bigger without it falling over*. The field got much better at scaling. The world got bigger. And the old methods finally had enough to eat. --- ## 2017: The paper that built the thing on your phone Five years after ImageNet, eight researchers at Google published a paper with the least modest title in the history of the field: "Attention Is All You Need." The paper introduced a network design called the Transformer. I won't walk you through the architecture; you don't need it. What you need to know is what it was *for*. The Transformer was extraordinarily good at one task: given a sequence of words, predict what word comes next. And it was designed so that you could make it bigger — more layers, more connections, more training data — and it would keep getting better at that task without hitting a wall. The "T" in ChatGPT stands for Transformer. So does the "T" in GPT-4, GPT-5, and every model named like them. Anthropic's Claude, Google's Gemini, Meta's Llama — all Transformers, all descended from that one 2017 paper, all doing the same fundamental job at enormous scale: predict the next word. Most of the eight authors have since left Google. Several founded companies, some of which are now worth billions of dollars. A paper about predicting the next word turned out to be the most valuable document Silicon Valley has produced this century. --- ## The prize and the resignation *Photo 2.1. Geoffrey Hinton speaking at Collision in Toronto on June 28, 2023, less than two months after his resignation from Google was reported. Photo by Ramsey Cardy / Collision via Sportsfile. CC BY 2.0. Cropped version hosted by Wikimedia Commons. License: https://creativecommons.org/licenses/by/2.0/ Source: https://commons.wikimedia.org/wiki/File:Geoffrey_Hinton_-_Collision_2023_-_Centre_Stage_RCZ_1307_(cropped).jpg* In 2018, Geoffrey Hinton shared the Turing Award — computing's equivalent of the Nobel — with Yann LeCun and Yoshua Bengio, for the work the field had ignored for two decades. In October 2024, Hinton received the actual Nobel Prize in Physics, shared with John Hopfield, for the foundational work on neural networks. Between those two honors, on May 1, 2023, *The New York Times* reported that Hinton had resigned from Google, where he had worked for a decade, so that he could speak openly about the risks of the technology he had spent his life building. The man who kept the idea alive through two winters had decided the public needed to hear his doubts about the spring. He told the *Times* that a part of him now regretted his life's work. "I console myself with the normal excuse," he said. "If I hadn't done it, somebody else would have." And: "It is hard to see how you can prevent the bad actors from using it for bad things." On the speed of what he'd helped build: "I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that." Note the shape of it. The single most important living contributor to this technology decided, six months after ChatGPT launched, that the most useful thing he could do with his remaining reputation was to warn people. And note the excuse he reached for — *somebody else would have* — because you're going to hear it, in one form or another, from almost everyone in the next chapter. --- ## What the winters teach Let me lay out what I think you should take from all this, because it matters for every chapter that follows. **First: the industry has been wrong about timelines, twice, catastrophically, and the people running it today were not around for either collapse.** The current generation of AI executives built their careers entirely inside the spring that began in 2012. They have never seen the money leave. That doesn't make them wrong now. It does mean their confidence has never been tested against the thing that tested Rosenblatt's and Minsky's. **Second: what ended the winters was mostly scale.** There was real engineering progress and I don't want to shortchange it. But nobody in 2012 had a fundamentally better theory of *intelligence* than Minsky had in 1969. What they had was vastly more data, far faster chips, and much better methods for using both. That is still the shape of it today. The dominant strategy of every major AI lab in 2026 is the same strategy that won ImageNet: make it bigger. This is why the companies are spending hundreds of billions of dollars on data centers, and it's why the next chapter is about money. **Third — and this is the one to carry forward — the machine that won was the one built to produce the right output, not the one built to understand.** The Transformer does not form concepts the way the Dartmouth proposal imagined. It predicts the next word. It does this so well that its output is, in Turing's sense, indistinguishable from someone who understands. That is the achievement. That is also the problem, and Chapter 4 will show you exactly why. But first: who paid for all of this, and what did they want for their money? --- ## Sources and Further Reading Minsky & Papert, *Perceptrons* (MIT Press, 1969). Lighthill, "Artificial Intelligence: A General Survey," UK Science Research Council, 1973. Rumelhart, Hinton & Williams, "Learning representations by back-propagating errors," *Nature* 323, 533–536 (1986). LeCun et al., "Backpropagation Applied to Handwritten Zip Code Recognition," *Neural Computation*, 1989. Hochreiter & Schmidhuber, "Long Short-Term Memory," *Neural Computation* 9(8), 1997. Krizhevsky, Sutskever & Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," NeurIPS 2012. Vaswani et al., "Attention Is All You Need," NeurIPS 2017. ACM A.M. Turing Award, 2018. Nobel Prize in Physics, 2024. Metz, "'The Godfather of A.I.' Leaves Google and Warns of Danger Ahead," *The New York Times*, May 1, 2023. --- \pagebreak # Chapter 3 # The Real Reason On November 30, 2022, a company called OpenAI put a chat window on the internet and labeled it a "research preview." The company did not expect much. The tool — ChatGPT — was a wrapper around a model they'd already had for months, dressed up so that ordinary people could type questions into it. It was a way to gather feedback. Within five days it had a million users. Within about two months, a hundred million. It was, by the measure that matters to the people who track these things, the fastest-adopted consumer product in history at that point, and it stayed that way until its own later versions beat it. The rest of the world experienced this as a lightning strike. It wasn't. Everything in Chapters 1 and 2 had been converging on this moment for a decade. The method was from 1986. The architecture was from 2017. The data was the internet. What made November 2022 different was that somebody finally built a front door. This chapter is about who built it, why, and — the question almost nobody asks — what the people who paid for it actually wanted. Because here's the thing you need to hold onto through this entire book: **the technology in your pocket was not built to solve your problems. It was built to solve theirs.** Sometimes those overlap. Understanding where they don't is the difference between using this tool and being used by it. --- ## The nonprofit that became a $500 billion company OpenAI was founded in December 2015 as a nonprofit. Its founders pledged a billion dollars. The stated mission was to ensure that artificial general intelligence — machine intelligence at or above human level — would benefit all of humanity, and the nonprofit structure was chosen deliberately, as a guard against the pressure to put profit ahead of that mission. The charter is still on their website. It's worth reading, mostly for what happened next. In 2019, OpenAI created a for-profit subsidiary with a "capped-profit" structure — investors could make money, but only up to a limit, with everything above the cap flowing back to the nonprofit. That same year, Microsoft invested a billion dollars. Over the following years Microsoft's total investment grew to roughly thirteen billion. Then, on October 28, 2025, the structure changed again. That day, the Attorney General of Delaware, Kathy Jennings, issued what her office called a "Statement of No Objection," and the Attorney General of California, Rob Bonta, concurred. OpenAI's for-profit arm was reorganized as a Public Benefit Corporation, controlled by the nonprofit — now called the OpenAI Foundation. The capped-profit model was eliminated. Microsoft's stake was reported at roughly 27 percent, valued at about $135 billion. The nonprofit's own stake was reported at around $130 billion. The company as a whole was valued at approximately $500 billion. The restructuring cleared the way for some $22 billion in funding from SoftBank and, according to every analyst who looked at it, opened the path to a public stock offering. Elon Musk, one of the original founders, had sued to stop this. His lawsuit was still pending when the restructuring was announced. I'm not going to tell you whether any of this was right or wrong. Smart people disagree, and the attorneys general of two states signed off. I'm telling you because the arc is the whole story in miniature: a project founded explicitly to keep this technology out of the hands of profit-seekers became, within ten years, one of the most valuable private companies on earth, and its own founders ended up in court over it. When you use ChatGPT, you are not using a public utility. You are using the product of a company that needs to justify a $500 billion valuation. That is not a criticism. It is context. Keep it. --- ## The chip company The second thing you need to understand about the money is a company most people had never heard of before 2023. Nvidia makes graphics processing units — the chips originally designed to render video games. It turned out, as Chapter 2 described, that those chips were also the best available hardware for training neural networks. When the AI spring arrived, Nvidia was very nearly the only company selling shovels. On May 30, 2023 — six months after ChatGPT launched — Nvidia's market value crossed one trillion dollars for the first time. On February 23, 2024: two trillion. On June 5, 2024: three trillion. On July 9, 2025: four trillion. No company in history had ever been worth that much. On October 29, 2025: five trillion. Again, the first company ever. One trillion to five trillion in under two and a half years. That is not a company growing. That is an entire economy reorganizing itself around one input. And on January 27, 2025, it showed how fragile that reorganization was. A Chinese lab called DeepSeek had released a model on January 20 that performed comparably to the American frontier models, and claimed to have trained it for about $5.6 million on 2,048 of Nvidia's H800 chips — a fraction of what the American labs were believed to be spending. When the market absorbed that claim, Nvidia's stock fell 17 percent in a single day, erasing $589 billion in value. It was the largest one-day loss by any company in the history of the U.S. stock market. The stock recovered. The point isn't the crash. The point is that half a trillion dollars moved in one day on the *possibility* that the thing everyone was buying might not need to be as big or as expensive as they'd assumed. The entire industry's valuation rests on the assumption from Chapter 2 — bigger is better, and bigger is expensive — and for one day in January 2025, the market flinched. --- ## The spending Here is what "bigger" costs. In 2024, the four largest American technology companies spent roughly the following on capital expenditures — data centers, chips, the physical plant of AI: Amazon around $78 billion, Microsoft around $56 billion, Alphabet around $53 billion, Meta between $37 and $39 billion. In 2025, according to CNBC's compilation of their own earnings guidance on October 31 of that year, those four companies collectively expected to spend more than $380 billion. By mid-2026, those companies had continued raising capital-expenditure plans, and the combined forward-looking total had moved well beyond the level contemplated a year earlier. The exact aggregate is slippery because the companies use different fiscal years, repeatedly revise guidance, and spend on cloud infrastructure that serves AI alongside other businesses. That caveat is more important than a single headline number. The direction is not in dispute: the largest technology companies are committing hundreds of billions of dollars a year to data centers, chips, networking, power, and related infrastructure, with AI demand as a central driver. Ask yourself the question the salesman asks: **what do they expect to get back?** Not a research preview. Not a chat window for your kid's homework. Nobody spends three quarters of a trillion dollars in a year to help you draft an email. There are honest answers to that question that have nothing to do with firing anybody. New products. New markets. Whole categories of work that didn't exist before. Companies have earned returns that size from expansion rather than substitution before, and they may again. But labor is the largest line item in the American economy, and a technology that does cognitive work is pointed straight at it. When you spend that kind of money on something that does what people do, at least part of what you expect back is a share of what those people used to cost. That's my read, not a proven fact — but it's the reading that best explains the size of the number. --- ## The other customer There is a second buyer at the table, and it is the one that paid for Rosenblatt's perceptron in 1958: the U.S. government, and specifically its military. On January 28, 2025, OpenAI launched ChatGPT Gov, a version of its product built for federal agencies. On June 5, 2025, Anthropic announced Claude Gov, models built for national security customers. On June 16, 2025, OpenAI announced a $200 million contract with the Department of Defense — its first under a new division it called "OpenAI for Government." On July 14, 2025, the Pentagon's Chief Digital and AI Office announced contracts with Anthropic, Google, OpenAI, and xAI — each with a ceiling of $200 million — for what it called "agentic AI workflows across a variety of mission areas." The office's chief, Doug Matty, said in the announcement that "the adoption of AI is transforming the Department's ability to support our warfighters." Palantir, whose Maven Smart System is the military's flagship AI targeting-and-analysis platform, had its contract ceiling raised on May 21, 2025 by $795 million, to roughly $1.28 billion through 2029 — on top of a separate Army enterprise agreement worth up to $10 billion over a decade. And on February 4, 2025, Google removed from its published AI Principles a section titled "Applications we will not pursue" — the section that had, since 2018, pledged that the company would not build AI for weapons or for surveillance that violated international norms. In the blog post announcing the change, Demis Hassabis and James Manyika wrote that "there's a global competition taking place for AI leadership within an increasingly complex geopolitical landscape," and that "we believe democracies should lead in AI development, guided by core values like freedom, equality, and respect for human rights." Let me be precise about what that means, because precision is the whole point of this book. Google did not announce a weapon. Google removed a promise. Those are different things, and I'll only ever tell you the one I can prove. But I'll also tell you what it looks like from the outside. Within eighteen months of ChatGPT's launch, every major American AI lab had a defense contract, a government product, or both — and the one lab that had put a no-weapons pledge in writing took it down. The "global competition" Hassabis and Manyika named is the third pressure on this technology, after Wall Street and the hyperscalers' capital budgets. It is the U.S.-China race, and it is the reason nobody in a position to slow this down wants to. --- ## The race Beginning on October 7, 2022 — seven weeks before ChatGPT launched — the U.S. Commerce Department began restricting the export of advanced chips and chip-making equipment to China. The rules were expanded on October 17, 2023. On January 15, 2025, in the last week of the Biden administration, Commerce issued a sweeping "AI Diffusion" framework governing which countries could buy how much American AI hardware. On May 13, 2025, the Trump administration rescinded that framework, calling it "overly bureaucratic" and saying it had "stifled American innovation." The China-specific controls stayed in place. The message to the industry was unambiguous: the government would restrict the adversary, but it would not restrict *you*. Every AI executive in America now has a sentence available to them that no regulator has been able to answer: *if we slow down, China wins.* Whether that's true is a question for people with clearances. What I can tell you is that it works. It has ended an extraordinary number of policy conversations in Washington, and Chapter 6 will show you how. --- ## What the money wants So let me put the three buyers on the table together. Wall Street wants a return on a $500 billion valuation and a $5 trillion chip company. The hyperscalers want a return on three quarters of a trillion dollars a year in data centers, and labor is the biggest pool of money that a machine doing cognitive work is pointed at. The government wants to win a race, and a race has no speed limit. None of those three buyers is paying for what the Dartmouth proposal wanted — a machine that forms concepts and understands. What all three are paying for is the same thing: useful output, *fast*, at *scale*, *now*. Here's my read, and I'll flag it as a read rather than a fact. In an operation built for speed, correctness stops being the product and starts being an expense. Checking becomes friction. I have never once seen a sales organization that didn't eventually discover this about its own quality controls, and I don't believe this industry is the exception. I'm not describing villains. I'm describing incentives. Every person I've named in this chapter would tell you, sincerely, that they want the technology to be accurate. I believe them. But the money does not pay for accurate. The money pays for *shipped*. And here is where it connects to everything that follows. The machine that got shipped — the Transformer, predicting the next word, made enormous — has a specific, mathematically describable relationship with the truth. It is not the relationship you assume. That's Chapter 4. --- ## Sources and Further Reading OpenAI, company charter (openai.com/charter). Delaware Department of Justice, "AG Jennings Completes Review of OpenAI Recapitalization," October 28, 2025. CalMatters, AP, October 28, 2025. CNBC, Reuters: Nvidia market-cap milestones (May 30, 2023; Feb 23, 2024; June 5, 2024; July 9, 2025; Oct 29, 2025). CNBC, "Nvidia sheds almost $600 billion in market cap, biggest one-day loss in U.S. history," January 27, 2025. CNBC, "How much Google, Meta, Amazon and Microsoft are spending on AI," October 31, 2025; company earnings guidance for 2026. Bureau of Industry and Security rules of October 7, 2022 and October 17, 2023; Federal Register, "Framework for Artificial Intelligence Diffusion," January 15, 2025; BIS rescission, May 13, 2025. CNBC, "OpenAI launches ChatGPT Gov," January 28, 2025. Anthropic, "Claude Gov models for U.S. national security customers," June 5, 2025. CNBC, "OpenAI wins $200 million U.S. defense contract," June 16, 2025. Defense News / CNBC, CDAO awards, July 14–15, 2025. Palantir Maven Smart System contract modification, May 21, 2025. CNBC, Bloomberg, *Washington Post*, "Google removes pledge to not use AI for weapons, surveillance," February 4, 2025; Hassabis & Manyika, Google blog, February 4, 2025. --- \pagebreak # Chapter 4 # Almost Right, By Design In the spring of 2023, a New York lawyer named Steven Schwartz needed to find court decisions that would help his client. His client was a man who said he'd been injured by a metal serving cart on an Avianca Airlines flight. The airline had asked the court to throw the case out. Schwartz, who had practiced law for three decades, had to file a response citing prior cases that supported letting the lawsuit go forward. He asked ChatGPT. ChatGPT gave him what he asked for: six court decisions, complete with case names, the courts that decided them, docket numbers, dates, and quotations from the judges' opinions. They were exactly the kind of cases he needed. He put them in his brief. His colleague, Peter LoDuca, signed it and filed it with the court. Avianca's lawyers went looking for the six cases and could not find them. Neither could the judge, P. Kevin Castel of the Southern District of New York. When the court ordered Schwartz to produce copies, he went back to ChatGPT and asked whether the cases were real. ChatGPT assured him they were. He asked for the full text of one. ChatGPT produced it — pages of judicial opinion, with a heading, a caption, and a reasoned analysis. None of it existed. Not the cases. Not the judges' words. Not the docket numbers. Every one of the six decisions was invented, whole, by a machine that had been asked for court cases and had produced things that looked exactly like court cases. In June 2023, Judge Castel sanctioned Schwartz, LoDuca, and their firm $5,000. The case, *Mata v. Avianca*, became the first widely reported instance of what the industry had already been calling, with a straight face, "hallucination." I want you to notice three things about what happened to Steven Schwartz, because all three are going to recur throughout this book. First: the output was *good*. It wasn't gibberish. It was formatted correctly, cited plausibly, and read like law. It passed the imitation game. Second: when he checked, he checked with the same machine that had made the error, and it told him he was fine. Third: he was an experienced professional in a field with strict rules about verification, and he did not catch it. Not because he was careless. Because nothing in thirty years of practice had prepared him for a source that fabricates with perfect confidence and perfect formatting. --- ## How many Schwartzes You'd think *Mata v. Avianca* would have been the end of it. Every lawyer in America read about that case. The lesson was clear. Check the citations. A French-based legal researcher named Damien Charlotin keeps a public database of court decisions in which a judge explicitly found, or clearly implied, that a filing relied on material invented by AI. His inclusion standard is strict: the judge has to have caught it and said so in writing. That means his count is a floor. It only captures the cases where a court noticed. In mid-2025, the database held around 200 cases. By January 2026: 719. By early April 2026: 1,227. By May 6: 1,397. May 22: 1,458. June 9: 1,598. The database kept moving while this book was being edited. On August 28, 2026, Charlotin's site said it had identified **1,981 cases**. His definition matters: these are legal decisions in which a court or tribunal addressed alleged or established AI use and hallucinated material more than in passing. It is not a count of every AI-generated filing, every fake citation, or every lawyer who used AI. The curve is not flattening. Three years after the most famous cautionary tale in the profession, judges were catching fabricated citations at a rate of several hundred a month — and those are only the ones they caught. The consequences have grown. In a 2026 Oregon federal case, *Couvrette v. Wisnovsky*, the combined sanctions and fee awards came to roughly $109,700. In June 2026, in *Withers v. City of Aberdeen* in Mississippi, lawyers on *both sides* of the case filed briefs with invented citations; the judge canceled the trial and suspended the two lead attorneys. Both sides. The plaintiff's lawyer and the defendant's lawyer, in the same case, each trusted a machine that made things up, and neither one checked. That's not two careless people. That's a profession's verification system failing at the same time, in the same room. Why does this keep happening? Not because lawyers are lazy. Because of what the machine actually is. --- ## What the machine actually does I'm going to explain how this works, and I'm going to do it without any math, because the math isn't the point. The point is one specific property that falls out of the design, and you can understand it without a single equation. At the center of every one of these systems is a model doing one thing: given a stretch of text, it predicts what comes next. That's the engine. You type "The capital of France is" and the model has read so much human writing that it knows the next word is overwhelmingly likely to be "Paris." It outputs "Paris." Then it looks at the whole sequence — "The capital of France is Paris" — and predicts what comes after that. A period, maybe. Or "and." It picks, adds it, and predicts again. Piece after piece after piece. Now, the systems you actually use in 2026 have more bolted onto that engine than they did in 2023. They can search the web. They can pull documents. They can run code, query a database, call another program, and work through a problem in steps before answering. Some handle images and audio as well as text. When somebody in the industry tells you "it's just predicting the next word," they're describing the engine and skipping the car built around it, and they're being a little glib. But every one of those additions is a tool the system chooses to reach for — or doesn't. And the thing deciding whether to reach, and what to do with whatever comes back, is still the engine. Which means the property I'm about to describe survives all of it. The training process from Chapter 2 — backpropagation, scaled up to a Transformer with hundreds of billions of internal connections, trained on a very large fraction of everything humans have ever written and put online — makes it astonishingly good at this. Good enough that predicting the next word, one at a time, produces essays, code, legal briefs, and medical advice that read as if a person wrote them. Now here's the property. **Producing text and looking something up are two different acts, and the first one does not require the second.** When the model generates a court citation, it is not — by default — consulting a database of court cases and retrieving one. It is predicting what a citation would look like in this position in this sentence. If a real case fits that prediction, and often one does because it has read millions of real citations, the output will be a real case. If none fits perfectly, the model does not stop and announce that it can't find one. It produces the most *plausible* string of words for a citation in that spot. Case name. Court. Year. Docket number. Quotation. Now, a modern system *can* be built to go look. It can search, or check a legal database, or run the citation against a real index — and when it does, this problem gets substantially smaller. That is genuine progress and I'm not going to pretend otherwise. But three things stay true. The tool has to be there. The system has to decide to use it. And whatever the tool returns still gets handed back to the same engine, which then writes a plausible-sounding answer about it. So the retrieval helps, and it does not close the gap, because generating is not verifying. The output looks exactly right either way. Looking right is what the engine does. This is why I keep saying it passed the imitation game. Turing's test, from Chapter 1, asks whether the output is indistinguishable from a human's. It doesn't ask whether the output is true. The machine that won was built to the test that was set. It produces text you cannot tell apart from a knowledgeable person's, whether or not the knowledge is real. The physicist Stephen Wolfram wrote a long, careful public explainer of this in February 2023, and if you want the detailed version, it's the best one. But the one sentence you need is this: **the model produces what is plausible, and plausible is not the same as true.** --- ## The lab says so itself You do not have to take my word for this. You can take OpenAI's. On September 4, 2025, four researchers at OpenAI posted a paper titled "Why Language Models Hallucinate." The company published a companion explainer on its website the next day. I'm going to quote it directly, because when the company that built the thing tells you how it fails, that's the source you want. The paper opens with a comparison: > "Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty." And it states its central finding plainly: > Language models "hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty." Think about what that sentence says. The problem isn't a bug in the code. It isn't bad data. It's the scoreboard. The way these models are trained and tested gives them points for confident answers and no points for "I don't know." A student who guesses on every hard question will, on average, score higher than one who leaves them blank. So the model learns to guess. Every time. With full confidence. Because that's what it was rewarded for. The paper goes further and puts a number on it. Kalai and coauthors derive a mathematical relationship between generation error and discrimination error under the setup analyzed in their paper, showing why generation can hallucinate more often than a related classifier fails to recognize incorrect answers. The result is a property of their formal framework, not a universal empirical law that every deployed model must exhibit at the same ratio. And in the companion post, OpenAI wrote about its own newest model: "GPT-5 has significantly fewer hallucinations especially when reasoning, but they still occur. Hallucinations remain a fundamental challenge for all large language models." Fundamental. Their word. --- ## "Almost right" I chose the title of this book from a survey of the people who use this technology most intensively, and who therefore know it best. Every year, the website Stack Overflow — the place where the world's programmers go to ask each other questions — surveys tens of thousands of developers. In its 2025 survey, it asked them what frustrated them most about AI coding tools. The answer that topped the list, cited by 66 percent of respondents, was dealing with AI-generated solutions that are "almost right, but not quite." Sixty-six percent. Roughly two out of three survey respondents who answered that question named the same problem. Not "it's useless." Not "it's wrong." *Almost right.* Almost right is the most dangerous thing a tool can be. A tool that's obviously wrong gets thrown out. A tool that's always right gets trusted, and deserves it. A tool that's almost right gets trusted and *doesn't* deserve it — and the gap between the trust and the truth is invisible until it costs you. Steven Schwartz's brief was almost right. Six real-sounding cases in a real brief for a real client. The Mississippi lawyers' briefs were almost right. The cases collected in Charlotin's database were almost right in a more consequential sense: evidence-shaped material had made it far enough into a legal process for a court to address it. That's why they got filed. And the reason those lawyers didn't catch it is the same reason the developers in the survey are frustrated: checking something that's almost right is *harder* than doing it yourself. When the output is 95 percent correct and beautifully formatted, finding the 5 percent that's fabricated requires you to verify every single piece — which is more work than the tool saved you. So people don't. They skim. They trust. They file. In that same Stack Overflow survey, 33 percent of respondents said they trusted the accuracy of AI output, while 46 percent actively distrusted it. And 84 percent were using or planning to use AI tools in their development process. Adoption was rising while trust lagged badly behind it. That is the tension that matters here. --- ## What this chapter proved Let me be exact about what the first four chapters establish, because from here on, the book stops describing the machine and starts describing what it's doing to you. 1. The machine was built to pass a test of *indistinguishability*, not truth. That was the goal from Turing forward. 2. The field's resurgence came from a combination of algorithmic advances, much larger datasets, far more computing power, and new architectures. The important point for this book is narrower: dramatic capability gains did not require machines to acquire human-like understanding of truth. 3. The economic incentives reward speed, scale, and in some settings labor substitution, while verification consumes time and money. 4. By its own maker's account, the machine is rewarded for guessing, produces plausible falsehoods as a matter of design, and the problem is "fundamental." Now, I want to be careful here, because this book is about claims that outrun their evidence and it cannot afford to make one. It would be easy to write that a human checker is the *only* defense. That isn't true, and the engineers reading this would put the book down. There are real technical defenses, and some of them work well: automated tests that fail when the code breaks, databases that reject impossible values, retrieval systems that force the model to cite a real document, permission rules that limit what a system can touch, and controls with names like row-level security that stop a program from reaching data it has no business reaching. Those aren't hypothetical. They're standard practice, and Part V will show you which ones the airline industry made mandatory after it learned this lesson the hard way. So the honest version of the claim is narrower and harder to argue with: **A machine that produces plausible output regardless of whether the output is true will sometimes be confidently wrong in ways that look exactly like being right — and catching that requires something outside the machine: a test, a rule, a control, or a person with the judgment to know which one applies.** Every one of those defenses has a person behind it. Somebody has to write the test, set the rule, configure the control, and — this is the part nobody budgets for — decide that this particular output is the kind that needs checking at all. The tools don't deploy themselves. In Chapter 11, you'll meet a lot of people who had every one of those defenses available to them, for free, and shipped without them, because the machine that built their software never mentioned they existed. Now watch what happens to the humans who check. --- ## Sources and Further Reading *Mata v. Avianca, Inc.*, No. 22-cv-1461 (S.D.N.Y.), Opinion and Order on Sanctions, June 22, 2023. Charlotin, "AI Hallucination Cases" database, damiencharlotin.com/hallucinations (figures as of July 2, 2026). *Couvrette v. Wisnovsky* (D. Or. 2026), reported in ABA Journal. *Withers v. City of Aberdeen* (N.D. Miss., June 8, 2026). Kalai, Nachum, Vempala & Zhang, "Why Language Models Hallucinate," arXiv:2509.04664, September 4, 2025; OpenAI, "Why language models hallucinate," openai.com, September 5, 2025. Wolfram, "What Is ChatGPT Doing … and Why Does It Work?", February 2023. Stack Overflow, 2025 Developer Survey (fielded May 29–June 23, 2025; ~49,000 respondents). --- \pagebreak # PART II — THE PLUNGE --- # Chapter 5 # Faster Than the Internet Think about how long it took the internet to reach your mother. I don't mean the year it was invented. I mean the year *she* used it — the year it stopped being a thing on the news and became a thing in her kitchen. For most families in this country that was somewhere in the late nineties or early two-thousands, and it was a whole production. Somebody had to buy a computer. Somebody had to call the phone company. There was a modem that made a sound like a fax machine drowning. Then you had to learn what a browser was, and what an email address was, and why you couldn't use the phone while your kid was on AOL. That whole process — from "this exists" to "my mother uses it" — took the better part of a decade. Now think about the last time you saw somebody use AI who you would have bet money would never touch it. A guy on a job site asking his phone how to word a bid. Somebody's grandmother having it write a birthday message. The church secretary running the newsletter through it before she prints it. How long did that take? Two years? Three? That's this chapter. Not whether the technology is good or bad — we'll get there — but how fast it arrived. Because the speed turns out to be most of the problem, and almost nobody talks about it. --- ## Somebody finally measured it For a long while the only numbers anyone had came from the companies themselves, which is a little like asking me how good the steaks on my truck were. Not that I was lying to anybody. I just had a stake in the answer. So a group of researchers, one of them working out of the Federal Reserve Bank of St. Louis, went and measured it the boring way. They asked a proper cross-section of Americans — the kind of survey the government uses when it wants to know something true about the country — whether they had used this stuff, when, and for what. They ran it in August 2024, about a year and nine months after ChatGPT showed up. Here's what came back. Roughly 39 percent of working-age Americans had used it. About a third had used it in the week they were asked. Among people with jobs, better than one in four had used it at work, and close to one in nine used it every single working day. Then they did the thing that makes this study worth putting in a book. They went back and dug up the same kind of numbers for the two technologies that changed everything before this one — the personal computer and the internet — measured the same way, counting from the moment each became something an ordinary person could go out and buy. Three years after the personal computer hit the market: about one American in five. Two years after the internet became a consumer product: about one in five. Two years after ChatGPT: nearly two in five. Double. And when they updated the study with newer numbers and published it in a serious academic journal, the figure had climbed to about 45 percent and their conclusion got sharper. Adoption at work, they wrote, has been faster than the personal computer. Adoption overall has beaten both the PC and the internet by a wider margin still. Sit with the size of that comparison for a second, because it's easy to skim right past it. The personal computer changed how nearly every office on earth operates. The internet rewired how we shop, how we date, how we argue, how we get our news, how presidents get elected. Those weren't small. Those are the two biggest technological shifts most of us will live through. This one is moving about twice as fast as either of them. --- ## The front-door numbers Here's the same story told from the company that built the front door. ChatGPT went live on November 30, 2022. It hit a million users in five days. Not five months. Five days. Two months after that it had a hundred million people using it every month, which at the time made it the fastest-adopted consumer product anybody had ever measured. Then it kept going. Four hundred million a week by February 2025. Seven hundred million by that September. Eight hundred million announced from a stage that October. Nine hundred million by February 2026, fifty million of them paying real money every month. That nine hundred million is the last figure OpenAI has confirmed itself, and they repeated it in June 2026. There's a second number floating around and it's worth separating carefully, because this book is about numbers that get repeated without being checked. In June 2026, the measurement firm Sensor Tower reported that the ChatGPT *app* had passed a billion monthly users — the fastest any app in history has reached that mark. Weekly users and monthly users are not the same measure, and the two get mashed together constantly in coverage. Reporting since suggests the weekly figure is approaching a billion too, but as of this writing OpenAI has not confirmed it. So take the conservative version, which is astonishing enough. Nine hundred million people a week, confirmed by the company. A billion a month on the app alone, measured independently. Using something that did not exist four years earlier. And that's one company's product — not the whole picture, barely half of it. Google put its version inside the search results a couple billion people look at every day. Microsoft put it in Word, in Outlook, in Windows itself. Apple put it on the iPhone. Meta put it in WhatsApp and Instagram and Facebook. Which means this. If you have picked up a phone or opened a laptop in the last two years, you have used this technology. You may never have chosen to. Doesn't matter. It's in the box now. --- ## The part that's already at work Numbers about the whole population are one thing. What I wanted to know was what's happening on the job, so I went and looked at Gallup, which surveys tens of thousands of working Americans every few months and has been tracking this from the start. Spring of 2023: about one worker in five said they used AI at work even occasionally. Fall of 2025: nearly half. More than doubled in two and a half years. And the people already using it were using it harder — the share doing it a few times a week or more kept climbing quarter after quarter even when the overall number leveled off. Now I want to flag something I'll come back to hard in Chapter 8, because it's the more interesting half of that survey and it almost never gets quoted. Just under half of American workers told Gallup they *never* use AI on the job. Not rarely. Never. So there isn't one story here. There are two, running side by side, and which one you're living in depends almost entirely on what kind of work you do. --- ## And then there's the kids The steepest curve in any of this doesn't belong to adults. It belongs to their children. In July 2025, Common Sense Media — a nonprofit that studies kids and technology, and which is about as far from a hype shop as you can get — published a survey of a thousand American teenagers between thirteen and seventeen. Not about homework. About *companions*. Chatbots built not to answer your questions but to talk to you. To be a friend. Seventy-two percent had used one. Fifty-two percent used one regularly — at least a few times a month. Roughly three out of four American teenagers, less than three years after this technology reached the public, had held a conversation with a machine designed to act like a person who cares about them. I'm not going to moralize about that here. Part IV is where the evidence on what this does to a young mind gets a proper hearing, and I intend to be careful there, because the research is early and I'd rather be accurate than dramatic. I'm putting it in this chapter for one reason, and it's about the shape of the curve. This technology reached the youngest, most impressionable, least supervised users *fastest* — and it reached them in the form that looks least like a tool and most like a person. That is not how the car spread. That is not how the internet spread. Kids got the internet after their parents did, mostly, on a machine sitting in the living room where somebody could walk past. This one went the other direction. --- ## What speed actually costs Here's the argument of this chapter, and it's why I care about the numbers at all. **Every technology that changed the world eventually grew a set of institutions to keep it from hurting people — and every one of those took decades to build.** Think about the car. Mass-market automobile, roughly the 1910s. Now count the things keeping you alive inside one: traffic lights, driver's licenses, speed limits, stop signs, seat belts, crash testing, drunk-driving laws, airbags, a federal safety agency. Every single one of those came *later*. Some of them fifty and sixty years later. And nearly every one exists because enough people died first to make the argument unanswerable. Or airplanes. Flying is now the safest way a human being can travel, and it got that way because when a plane goes down, an independent body pulls the wreckage apart, works out exactly what happened, publishes it in public even when it embarrasses somebody powerful, and forces the industry to change. That took decades to build too. It works. We're going to spend a whole chapter on it in Part V, because aviation has already lived through the exact problem this book is about and figured out what to do. Or medicine. Clinical trials. The FDA. The requirement that somebody prove a drug works and won't kill you *before* it goes on the shelf. Built over a century, mostly in response to disasters. Notice what all three have in common. Every one is a form of *checking*. Somebody looks at the thing before it hurts you, or picks through the wreckage afterward so it doesn't hurt the next person. That's the whole safety apparatus of the modern world, and we built it slowly, painfully, usually after somebody's funeral. Now set the numbers from this chapter next to that. Nine hundred million people a week. Nearly half of working-age America. Almost three quarters of American teenagers. Twice the speed of the internet — and the internet, thirty years on, is a technology whose harms we're honestly still arguing about. The checking institutions for AI do not exist. Not because nobody thought of it — the next chapter is about the people who tried, and it's a hell of a story. But because there was no time. The car got sixty years. This got four, and inside those four the technology changed so fast that any rule written in 2023 was describing a product that no longer existed by 2025. That's the cost of speed. Not that fast is bad. That fast doesn't leave room for the part where somebody checks. --- ## One more thing before we go There's a detail buried in that St. Louis Fed research I want to leave you with, because it sets up the rest of Part II. The researchers noticed that the people picking up AI first looked an awful lot like the people who picked up the personal computer first. Same pattern by education. Same pattern by the kind of job you hold. Younger, more schooling, more likely to sit at a desk. That's not shocking. It's also not nothing. A technology moving at twice the speed of the internet, landing first among the people who already have the most, doesn't spread itself evenly on the way down. It reaches one part of the country years before it reaches the other. Chapter 8 is about what that gap actually consists of, and I'll tell you right now it isn't what most people assume. But speed is the point of this chapter, so let me end on it straight. Four years. Nine hundred million people a week. Nearly half of working-age America. Almost three quarters of American teenagers. Faster, in the adoption comparisons used by the researchers, than the early spread of the personal computer and the internet. Every one of those older technologies got decades for society to work out what it was for, where it broke, and who needed protecting from it. This one got a long weekend. So who was supposed to be watching the door while hundreds of millions of people walked through it every week? --- ## Sources and Further Reading Alexander Bick, Adam Blandin & David Deming, "The Rapid Adoption of Generative AI," NBER Working Paper 32966 (September 2024), published in *Management Science* (2026), doi:10.1287/mnsc.2025.02523; Federal Reserve Bank of St. Louis, *On the Economy*, September 2024 (August 2024 survey: 39.4% of the U.S. population aged 18–64 had used generative AI; ~32% in the prior week; 28% of employed respondents at work; ~1 in 9 daily. Updated late-2024 figure: 45%. PC adoption ~20% at three years; internet ~20% at two years; the paper notes generative AI and the PC share "very similar early adoption patterns by education, occupation, and other characteristics"). OpenAI user milestones: company announcements including DevDay, October 6, 2025 (800 million weekly); TechCrunch, February 27, 2026 (900 million weekly; 50 million paying subscribers); OpenAI reaffirmation of the 900 million weekly figure, Cannes Lions, June 22, 2026; Sensor Tower estimates reported by Reuters, June 2026 (the ChatGPT app crossing 1 billion *monthly* active users — a different measure from weekly active users, and not interchangeable with it). As of this writing OpenAI has not publicly confirmed a weekly figure above 900 million. Gallup, "AI Use at Work Rises," December 2025 (23,068 U.S. employees surveyed August 5–19, 2025; 21% in Q2 2023 rising to 45% in Q3 2025); Gallup Q4 2025 workplace update (46% total use; 26% frequent use; 12% daily; 49% report never using AI at work). Common Sense Media, "Talk, Trust, and Trade-Offs: How and Why Teens Use AI Companions," July 16, 2025 (nationally representative survey of 1,060 teens aged 13–17; 72% had used an AI companion; 52% regular users). --- \pagebreak # Chapter 6 # Nobody's Guarding the Door > *A note before this chapter. Everything in it — the laws, the court fights, the > money, the deadlines — is current as of August 31, 2026, and some of it will > have moved by the time you read this. That is not a defect in the reporting. > It is the point of the chapter. The rules are being written right now, in > public, by people whose names are in here.* Let me tell you about the one night the United States Senate agreed on something. It was July 1, 2025. The vote was 99 to 1. Here's what they were voting on. There was a provision buried inside the big budget bill that would have barred every state in the country from enforcing its own laws about artificial intelligence for the next ten years. Ten years. In a business where the product changes every six months. Now, I want to be fair to the people who wanted that, because their argument isn't stupid. If you're building this technology and fifty different states write fifty different sets of rules, you end up with a mess nobody can comply with, and the argument goes that the mess hands the future to China. That's a real concern held by serious people. But ten years is a long time to tell fifty states to sit down. Senator Ted Cruz of Texas had carried the provision. Senator Marsha Blackburn of Tennessee, a Republican, had worked out a compromise version with him — and then, in the last hours, walked away from her own deal. Her explanation: "This provision could allow Big Tech to continue to exploit kids, creators, and conservatives." Until Congress passes something real, she said, "we can't block states from making laws that protect their citizens." She teamed up with two Democrats — Ed Markey and Maria Cantwell — to strip it out. Ninety-nine senators voted yes. One voted no. The lone dissenter was Senator Thom Tillis of North Carolina. Cruz voted with the other ninety-nine to strip the moratorium. I'm opening the chapter here for two reasons. The first is that this is the single most bipartisan thing Congress did about AI in this entire period, and it was a vote to *not* do something. It was ninety-nine people agreeing to leave the states alone, because Washington wasn't going to act itself. The second is what happened next. Because the people who wanted that ten-year freeze did not go home. --- ## What the record actually shows I'm going to walk you through this quickly, because it's the least fun part of the book and I'd rather you have it than not. Back in the fall of 2023, President Biden signed an executive order on AI — the most serious federal action anybody had taken. Among other things, it required the companies building the biggest systems to hand safety-test results over to the government. On his first day back in office in January 2025, President Trump revoked it. That May, the House passed the budget bill with the ten-year state freeze inside it. In July, the Senate pulled it out, 99–1. Later that month the White House put out an AI Action Plan that framed the whole thing as a race we have to win and regulation as a weight around our ankles. Toward the end of the year, supporters tried again — this time attaching the state freeze to the defense bill, the one Congress has to pass every year no matter what. It failed again. Eight days later, in December 2025, the President signed an executive order that did something the Senate had twice declined to do. It set up a unit inside the Justice Department whose job is to take states to court over their AI laws. It told the Commerce Department to make a list of state rules it considers burdensome, and to think about withholding federal broadband money — the money that runs internet to rural counties — from states that don't back off. Read that sequence one more time. Congress refused to override the states, twice, by enormous margins. So the executive branch built a legal unit to sue the states and put their internet money on the table. Lawyers noted the obvious problem: an executive order can't override state law. Only Congress can do that. As the Brookings Institution put it, the order "merely directs agencies to take actions that might eventually create pathways for preemption." Which is a polite way of saying it's a threat, not a law. In the spring of 2026 the White House released a "national policy framework" urging Congress to replace the state patchwork with one federal standard. It's non-binding. It requires nothing of anybody. That June, two members of the House — Republican Jay Obernolte of California and Democrat Lori Trahan of Massachusetts — released a 269-page discussion draft of the Great American AI Act. It was explicitly released for feedback before formal introduction. The framework drew immediate opposition from some Democrats and, two weeks later, a letter from 203 state legislators across 42 states. Then the status changed. On July 23, Obernolte and Trahan formally introduced the bipartisan FRONTIER Act, legislation developed as part of the broader Great American AI Act framework. It would impose risk-based requirements on developers of the most advanced models, including model cards, risk-management frameworks, independent audits, incident reporting, and ongoing assessments. As of August 31, 2026, it had not become law. So here's where the federal government stands as I finish this book: there are executive orders, voluntary frameworks, sector-specific legal duties, existing consumer-protection law, and now serious federal legislation on the table. What the United States still does not have is a comprehensive federal AI law imposing a general predeployment verification regime across consumer AI systems. That is a narrower claim than saying nobody is guarding the door. It is also the claim the evidence supports. --- ## Fifty states, all at once Into that empty space walked the states — all of them, in every direction, at the same time. By March 2026 one tracking firm counted more than 1,500 AI bills introduced across 45 states in that year alone, up nearly 150 percent over everything introduced in all of 2024. By July, 29 states had actually passed something. Some of it is serious. California vetoed one big safety bill in 2024 and then signed a narrower one in September 2025, putting transparency requirements on the largest developers. New York passed its own law aimed at the most powerful systems, signed that December — eight days after the President's executive order took aim at exactly that kind of law. Colorado and Texas built frameworks of their own. And some of it is what you'd expect when fifty legislatures each try to regulate something none of them fully understands. Definitions that don't match. Deadlines that conflict. A compliance map so tangled that the industry's argument — *this patchwork will strangle us* — starts sounding reasonable even to people who don't trust the industry. That's the trap, and it's worth naming plainly. The absence of a federal referee didn't produce no rules. It produced fifty sets of rules, and then a lobbying campaign to erase all of them at once. --- ## Europe wrote a law, then hit pause Across the Atlantic, the European Union did the thing everybody said couldn't be done. It passed the world's first comprehensive AI law, which took effect in stages starting in August 2024. Bans on the worst uses kicked in early 2025. Rules for the big general-purpose systems followed that August. And the heart of the whole thing — the requirements for AI used in hiring, credit, education, and public services, the places where a wrong answer wrecks somebody's life — was scheduled to take effect on August 2, 2026. In November 2025, the European Commission proposed delaying it. The negotiation collapsed in April 2026, came back together in May, passed the European Parliament in June by a lopsided vote, got final sign-off at the end of that month, and became law on July 27, 2026. Six days before the original deadline. The core rules now take effect in December 2027, and in some cases August 2028. I want to be fair to the Europeans. They did more than anybody. The law exists, the bans are real, the transparency rules held their dates. But look at the shape of it, because it's the same shape as everything else in this chapter: **the one place on earth that wrote comprehensive rules for this technology postponed its own most important provisions by sixteen months, six days before they would have applied.** The technology outran the law. Again. --- ## Follow the money Why does this keep happening? Why does a 99–1 Senate vote get answered with an executive order, and a landmark European law get pushed back at the last possible minute? It isn't hidden. You just have to look at the money. In 2025, four companies alone — OpenAI, Meta, Google's parent company, and the chipmaker Nvidia — spent a combined $50.9 million lobbying Congress, according to federal disclosures reviewed by the watchdog group Issue One. That's the ordinary kind of money. The extraordinary kind showed up in August 2025, when a new political action committee called Leading the Future launched with more than $100 million behind it. By year's end it had $125 million. Its backers included the venture firm Andreessen Horowitz, OpenAI's president Greg Brockman, Palantir co-founder Joe Lonsdale, and a handful of others in that world. Its goal was straightforward and stated out loud: one national AI standard that overrides the states. Its playbook was borrowed openly from the cryptocurrency industry, which had spent $200 million in the 2024 elections doing exactly this. Back your friends. Destroy somebody publicly. Make the cost of crossing you visible to everyone watching. The somebody they picked was a New York state assemblyman named Alex Bores. Bores is a Democrat and a former Palantir engineer — meaning he actually knows how this stuff works — and he'd co-sponsored New York's AI safety law. When he announced a run for an open congressional seat in Manhattan in November 2025, the PAC announced it would spend millions to beat him. Later they clarified: at least $10 million. Bores was blunt about what it meant. "While $100 million is an insane amount for anyone to be spending," he said, "in some sense it's just a VC investment for them, because their returns could be trillions." By the primary in June 2026, reporting put Leading the Future's spending against him at more than $8 million. Groups favoring stronger AI safeguards also spent heavily in the race. Anthropic, for example, donated $20 million to the nonprofit Public First Action in February 2026, but Anthropic explicitly restricted that money to public education and policy work and said it could not be used to influence candidate elections. That donation therefore should not be treated as campaign spending for Bores. He lost. Close second. Let me be careful about what that does and doesn't prove. It doesn't prove the money bought the seat — the man who won had also co-sponsored the same safety law. What it proves is the *demonstration*. The PAC said publicly it planned to spend in fifty to sixty races and $125 million across the midterms. Through June it had spent more than $24 million, and every candidate it backed other than Bores's opponents had won. The message to every state legislator in America wasn't subtle. Sponsor a safety bill, and eight million dollars appears against you. Meanwhile, AI companies and aligned groups were putting substantial sums into federal and state political fights. The scale was historically large, but comparisons across industries depend on what counts as lobbying, corporate contributions, super-PAC spending, nonprofit advocacy, and state spending, so I am not going to turn that into a clean record claim. --- ## The people on the inside *Photo 6.1. Ilya Sutskever and Sam Altman at Tel Aviv University on June 5, 2023. Sutskever left OpenAI in May 2024; three days later, Superalignment co-lead Jan Leike resigned and publicly criticized the company's safety priorities. Photo by Eladkarmel. CC BY-SA 4.0. Wikimedia Commons. License: https://creativecommons.org/licenses/by-sa/4.0/ Source: https://commons.wikimedia.org/wiki/File:Ilya_Sutskever_and_Sam_Altman_in_TAU.jpg* There's a third group in this story, and they're the closest thing it has to a conscience. What happened to them tells you what the door looks like from the inside. In May 2024, Ilya Sutskever — a co-founder of OpenAI, chief scientist, one of the three names on the paper that started the modern AI boom — announced he was leaving. Three days later, Jan Leike, who co-led the team responsible for making sure future AI systems stay under human control, resigned and said why in public: "Over the past years, safety culture and processes have taken a backseat to shiny products." He said he'd been disagreeing with company leadership "about the company's core priorities for quite some time, until we finally reached a breaking point." The team he had led was dissolved. He went to work for a competitor. A month before that, a researcher named Daniel Kokotajlo had left the same company. On his way out he was handed a non-disparagement agreement — sign this, or forfeit your vested equity. About $2 million, which he later said was roughly 85 percent of his family's net worth. He didn't sign it. He wanted to be able to talk. When a reporter at Vox exposed the practice in May 2024, the company announced it would stop enforcing that clause and release former employees from it. On June 4, 2024, Kokotajlo and twelve others — eleven current or former OpenAI people and two from Google DeepMind — published an open letter called "A Right to Warn About Advanced Artificial Intelligence." Six of the thirteen signed anonymously. Four of those six still worked there. Their central point was one sentence long and it's the whole chapter: these companies "have strong financial incentives to avoid effective oversight," and "ordinary whistleblower protections are insufficient because they focus on illegal activity, whereas many of the risks we are concerned about are not yet regulated." *Not yet regulated.* The people building it were saying: we see things that worry us, there's no law against any of it, and we're contractually forbidden from telling you. That was the summer of 2024. The law they said was missing still doesn't exist. --- ## So who's actually checking? Let me answer the question the chapter started with. In the United States, as of the late summer of 2026, the enforceable rules about what an AI company must do before releasing something to the public consist of: whatever individual states have passed, and can defend against a Justice Department unit built to sue them. No federal law. No agency with the power to say no. There's a federal institute that runs voluntary evaluations with some of the companies, and reports that the administration is *considering* requiring testing before release on the most powerful systems. Considering. In Europe, there's a real law, and its core just got pushed to 2027 and 2028. Inside the companies, there are people who are worried. Some left. Some gave up millions to be free to say so. Some signed a letter without their names on it because they still had jobs. And there's better than $125 million in political money whose explicit purpose is to keep all of it exactly this way. That's the state of the door. Now, one important thing before we move on, because the absence of law is not the absence of knowledge. The people who actually understand this technology have already written down what to do about it. In detail. For free. In November 2023, the U.S. cybersecurity agency and its British counterpart jointly published guidelines for building AI systems safely, endorsed by eighteen countries. The federal standards institute published a risk-management framework, and then a supplement specifically for this kind of AI with more than two hundred recommended actions. And a volunteer foundation called OWASP — whose security checklists half the internet is already built against — publishes a top-ten list of AI-specific risks, updated for 2025. Every one of those documents is public. Every one is free. Every one is written by people who know exactly what they're talking about. Not one of them is mandatory. There's a line in the American and British guidelines worth remembering, because Part V comes back to it. The burden falls on the people who *build and sell* the system, not on the people who use it. As the head of Britain's cyber agency put it, security has to be "not a postscript to development but a core requirement throughout." That's precisely the opposite of what this market rewarded between 2023 and 2026. Chapter 11 is the list of consequences. I said in Chapter 3 that I wasn't describing villains, and I'll say it again. Almost everybody in this chapter thinks they're right. The senators who killed the freeze believed states should protect their people. The donors funding the PAC believe a patchwork of state rules hands the race to China. The Europeans who delayed their own law believed the standards weren't ready yet. Every one of them can make their case, and some of them are probably right. But add it up. Roughly 900 million people a week were using ChatGPT by the last confirmed weekly figure cited in this book, while a separate 2026 figure put monthly app users above one billion. They are different measurements, and they should not be collapsed into one number. The product's own maker has also published research explaining why language models remain prone to confident error. Governments know how to check it — they wrote the manuals. And the sum total of the world's binding response is: one comprehensive law, with its core high-risk provisions postponed to 2027 and 2028. Fifty partial state laws, under legal attack from the federal government. A shelf of excellent free advice nobody is required to read. And a hundred-million-dollar campaign to make sure nothing more happens. Nobody's guarding the door. And the people who came through it first — the ones who sold you the fear, and then sold you the calm — are the subject of the next chapter. --- ## Sources and Further Reading Senate vote on the amendment striking the state AI moratorium from H.R. 1, July 1, 2025 (99–1); Senator Marsha Blackburn statement, June 30–July 1, 2025; Senators Markey and Cantwell, press releases, July 1, 2025. Executive Order 14110 (October 30, 2023), revoked January 20, 2025. White House, "America's AI Action Plan," July 23, 2025. Executive Order 14365, December 11, 2025 (AI Litigation Task Force; Commerce Department review of state AI laws; conditioning of BEAD broadband funds); Brookings Institution analysis, December 2025. White House, National Policy Framework for Artificial Intelligence, March 20, 2026. Great American AI Act discussion draft (Reps. Jay Obernolte and Lori Trahan), released June 4, 2026 — Roll Call, June 4, 2026; DLA Piper, June 2026; letter of 203 state legislators, June 16, 2026. MultiState AI bill tracking (1,561 bills across 45 states as of March 2026); TechPolicy.Press, "Where State AI Legislation Stands Half Way Into 2026," July 22, 2026. California SB 1047 (vetoed September 2024) and SB 53 (signed September 29, 2025); New York RAISE Act (signed December 2025); Texas TRAIGA (effective January 1, 2026). Regulation (EU) 2024/1689 (the EU AI Act); European Commission Digital Omnibus proposal, November 19, 2025; Regulation (EU) 2026/1744, published in the Official Journal July 24, 2026, in force July 27, 2026 (high-risk obligations deferred to December 2, 2027 for standalone systems and August 2, 2028 for embedded systems) — Gibson Dunn, Cooley, DLA Piper client alerts. Issue One analysis of 2025 federal lobbying disclosures, reported by NPR, June 22, 2026. Leading the Future: CNBC, November 17, 2025 and July 9, 2026; NOTUS; The Nation, June 16, 2026; Gizmodo, June 24, 2026. Jan Leike, post on X, May 17, 2024. Daniel Kokotajlo: Vox (Kelsey Piper), May 2024; TIME 100 AI, 2024. "A Right to Warn About Advanced Artificial Intelligence," righttowarn.ai, June 4, 2024; Associated Press and *New York Times* coverage, same day. CISA and UK NCSC, "Guidelines for Secure AI System Development," November 26, 2023 (endorsed by 18 nations). NIST AI Risk Management Framework 1.0 (AI 100-1), January 2023; NIST Generative AI Profile (AI 600-1), July 2024. OWASP Top 10 for LLM Applications 2025, OWASP GenAI Security Project. --- \pagebreak # Chapter 7 # The Sellers I have knocked on doors for a living for most of my adult life, and for part of it I was the one training other people to do it — flying around the country teaching salespeople how to open a conversation with a stranger and close it. So let me tell you the first thing you learn out there. There are two ways to make a sale. You can open the cooler, hand the man a ribeye, and let him look at it. Or you can tell him what he's paying at the grocery store this month and let that sit. Both of those can be honest. The steak is the same steak either way. But the second one moves faster, and once you have felt how much faster fear moves than value, you have to be a fairly disciplined person not to reach for it every single time. Keep that in your pocket for this chapter, because this chapter is about two of the most powerful men in this industry making one pitch for a year and then making the opposite one — and about how neatly each pitch fit what their companies needed at the time. --- ## The bloodbath On May 28, 2025, Dario Amodei sat down with two reporters from Axios. Amodei runs Anthropic — the company that makes Claude, which is the AI I used to build my own business. He's a serious person. Nobody who has met him thinks he's a huckster, and I want to say that plainly before I say anything else. What he told those reporters was that AI "could wipe out half of all entry-level white-collar jobs — and spike unemployment to 10–20% in the next one to five years." He wasn't hedging. "We, as the producers of this technology, have a duty and an obligation to be honest about what is coming," he said. "I don't think this is on people's radar." And: "Most of them are unaware that this is about to happen. It sounds crazy, and people just don't believe it." Axios ran it under the headline "A white-collar bloodbath." It was everywhere inside a day. For the next twelve months, that ten-to-twenty percent was the number anchoring every conversation in America about AI and work. It got quoted in Congress. It got quoted on cable. It got quoted, I'd bet, in a few thousand meetings where somebody was explaining why a position wasn't going to be filled. Now here's the other half. --- ## The walk-back On May 26, 2026 — one year later, almost to the day — Sam Altman sat on a stage in Sydney, Australia, next to the chief executive of one of the country's largest banks. Altman runs OpenAI, which makes ChatGPT. If Amodei is the industry's careful voice, Altman is its front man, and he'd been making versions of the same prediction for two years. What he said in Sydney was this: "I thought there would have been more impact on entry-level white-collar jobs being eliminated by now than has actually happened. I'm delighted to be wrong about this." And then a line I think ought to be carved over the door of every AI company in the world: "We've been roughly right on technological predictions and pretty wrong on the social and economic implications." That same week, Amodei was reframing his own message, describing AI now as a "productivity multiplier." Fortune ran the story under a headline about the two of them walking back their apocalypse predictions. David Autor, an economist at MIT who studies exactly this and has no stake in either company, gave the *Wall Street Journal* a drier read. The leaders, he said, "may have realized it was simply bad business to say that your great new product will destroy the economy." This chapter is about the year in between. Who said what, what the numbers actually showed, and — the question a salesman can't help asking — who was getting paid on each side of the story. --- ## The memo heard round the world The fear didn't start with Amodei. He just put a number on it. On April 7, 2025, Tobi Lütke posted an internal memo to his own company publicly on X, because it was leaking anyway. Lütke runs Shopify, which is the software behind an enormous share of the small online stores you've bought from without knowing it — the little boutique, the guy selling custom mugs, your niece's jewelry business. The memo was titled "Reflexive AI usage is now a baseline expectation at Shopify." Here's the sentence that went around the world: "Before asking for more headcount and resources, teams must demonstrate why they cannot get what they want done using AI." And then, cheerfully: "What would this area look like if autonomous AI agents were already part of the team? This question can lead to really fun discussions and projects." Fun. Here's the part the coverage mostly skipped. Shopify's headcount had already gone from 11,600 in 2022 down to 8,100 at the end of 2024, while the company grew better than twenty percent a year. Nobody called that a layoff. There was no announcement, no severance press release, no number in the news. The memo just made the policy official. Prove a human is necessary, or you don't get one. Three weeks later, on April 28, Luis von Ahn sent a similar email to everybody at Duolingo, the language-learning app with the owl. The company would be "AI-first." "AI is already changing how work gets done," he wrote. "It's not a question of if or when. It's happening now." Duolingo would "gradually stop using contractors to do work that AI can handle," and would only hire "for roles that cannot be automated." The company would move fast and accept "small hits to quality" rather than move slowly and miss the wave. *Photo 7.1. Duolingo co-founder and CEO Luis von Ahn speaking at Wikimania in 2015. In April 2025 he told employees that Duolingo would become 'AI-first' and would gradually stop using contractors for work AI could handle. Photo by Daniel Case. CC BY-SA 3.0. Wikimedia Commons. License: https://creativecommons.org/licenses/by-sa/3.0/ Source: https://commons.wikimedia.org/wiki/File:Luis_von_Ahn_at_Wikimania_2015.jpg* Small hits to quality. Hold that phrase. It comes back in Part III with a vengeance. The Duolingo memo landed very differently than Shopify's. Users threatened to delete the app. Von Ahn walked the framing back within weeks. The policy was the same. The *framing* wasn't. Lütke sold it as ambition — look what we could build. Von Ahn sold it as replacement — we'll stop paying people for work the machine can do. One made employees feel like they'd been handed a weapon. The other made them feel like they were being replaced by one. Same policy. Opposite reception. That's not a technology story. That's a sales story, and it's why I keep telling you to watch the pitch and not just the product. --- ## Klarna And before either of them, there was Klarna. Klarna is a Swedish payments company — you've seen their logo at online checkouts, the buy-now-pay-later button. In December 2023 they froze hiring outside of engineering, explicitly to replace people with AI. By February 2024 they were the industry's favorite success story. Their AI assistant was handling two-thirds of all customer service chats — 2.3 million conversations in its first month — doing the work of 700 full-time agents. The CEO, Sebastian Siemiatkowski, told an interviewer he wanted Klarna to be OpenAI's "favorite guinea pig." Headcount fell from about 7,400 to around 3,000. That story went into every investor deck the company had, right up to its stock market debut. Then, on May 8, 2025, Siemiatkowski told Bloomberg something else entirely. "As cost unfortunately seems to have been a too predominant evaluation factor when organizing this, what you end up having is lower quality." And: "Really investing in the quality of the human support is the way of the future for us." And: "From a brand perspective, a company perspective, I just think it's so critical that you are clear to your customer that there will always be a human if you want." Klarna started hiring human agents again. I want to be careful here, because this story gets told badly all over the internet. Klarna did not abandon AI. The chatbot still handles most inquiries. The company's own position is that the mistake was over-weighting cost, not using the technology. And Siemiatkowski, to his credit, kept warning afterward that the job impact was real and that other executives were sugarcoating it. But look at what actually happened, in order. The AI customer service was cheaper. It produced lower quality. It took the company **fourteen months** to notice. And what finally made them notice wasn't a quality metric — it was the *brand*. Customers who couldn't reach a human being. Cheaper. Almost as good. Nobody caught it for over a year, because the thing measuring success was measuring cost. That's the pattern this book is about, playing out inside one Swedish payments company two years before I sat down to write about it. --- ## What the numbers actually said While the executives were talking, the economists were counting. And the count didn't match the speeches — in either direction. **Yale.** On October 1, 2025, four researchers at Yale's nonpartisan Budget Lab published a study measuring whether AI had actually changed the mix of jobs in the American economy since ChatGPT launched. Their conclusion, in their own words: "the broader labor market has not experienced a discernible disruption since ChatGPT's release 33 months ago, undercutting fears that AI automation is currently eroding the demand for cognitive labor across the economy." No discernible disruption. Thirty-three months in. They added the context every honest account needs: this kind of change historically takes decades, not months. Computers didn't become normal in offices until nearly a decade after they went on sale. And they flagged one exception — something odd happening to recent graduates specifically, which "could show AI impacting employment for early career workers but could also reflect a slowing jobs market." Hold that exception. It's Chapter 15, and it's the most important thing in Part IV. **The layoff trackers.** A firm called Challenger, Gray & Christmas counts announced job cuts and the reasons companies give. In all of 2025, companies blamed AI for 54,836 cuts — out of roughly 1.17 million total. About five percent. Then it accelerated. Through May of 2026: 87,714 AI-blamed cuts, already more than all of 2025, with nearly 39,000 in May alone — the highest single month since they started tracking the reason. But notice what that number actually is. It's what companies *say* when they announce layoffs. Analysts at Harvard Business Review and Deutsche Bank both put a name to the obvious problem: "AI-washing." Using the technology as a modern-sounding, investor-friendly explanation for cuts actually driven by over-hiring in 2021 and cost pressure in 2025. Harvard's January 2026 piece was titled, bluntly, "Companies Are Laying Off Workers Because of AI's Potential — Not Its Performance." **The executives' own forecasts.** A survey of 1,200 chief executives across 21 countries asked whether they expected major AI-driven headcount cuts. In January 2025: 46 percent said yes. By May 2026: 20 percent. They cut their own expectations by more than half in sixteen months. So here's the honest picture, one year after the bloodbath headline. No measurable disruption to the job market overall. A real but modest number of AI-blamed layoffs, inflated by companies who preferred that explanation to the true one. CEOs quietly halving their own predictions. And one persistent signal at the entry level that nobody could yet explain. That is not ten to twenty percent unemployment. --- ## Now the salesman's question Amodei's warning in May 2025 and Altman's reassurance in May 2026 were both delivered by men whose companies were, at those moments, raising money at valuations in the hundreds of billions and — by every report — preparing to sell shares to the public. I'm going to label this carefully, because I promised you I would. **The claim that these statements were timed to their fundraising is an interpretation. It is not a proven fact.** Autor's line about it being bad business is an economist's read on somebody else's motives. He could be wrong. I can't see inside their heads and neither can he. But I can tell you what a salesman sees, because I have stood on both sides of this door, and I have taught other people how to stand on my side of it. In 2025, the pitch was fear. *This technology is so powerful it will eliminate half of entry-level white-collar jobs.* Ask who that pitch serves. It serves a company raising money, because a machine that can replace half the white-collar workforce is worth trillions. It serves a company fighting regulation, because a technology that powerful is a national security asset, and you don't slow down national security assets. And it serves every executive cutting headcount, because "we have no choice, the AI is coming" is a much better story for the shareholders than "we hired too many people in 2021." That year, fear moved the product. In 2026, the pitch was calm. *We were wrong, the jobs are fine, it's a productivity multiplier.* Ask who that serves. It serves a company about to sell stock to the public, because the public does not buy shares in the thing that's going to fire them. And it serves a company facing a hundred-million-dollar political fight, because "we're not that dangerous" is a far better argument against regulation than "we're extraordinarily dangerous, trust us." That year, calm moved the product. Same men. Same technology. Opposite stories, twelve months apart, each one perfectly fitted to what the seller needed that year. Here's what I actually think, and it's less cynical than it sounds. I don't think they're lying. I think they're *selling*, and I think the gap between those two things is smaller than people who've never sold for a living want to believe. When you sell, you believe the pitch. You have to — you can't stand on a doorstep and say words you don't believe, not for long, not well. The pitch just happens to be whatever the quarter requires. And the honest ones, the good ones, genuinely convince themselves first. That's not a character flaw. That's the job. Which is exactly why you can't calibrate off them. --- ## What to take from this **You cannot set your fear or your comfort about this technology by listening to the people selling it.** Not because they're dishonest. Because their incentives move faster than the truth does. The same voice that told you in 2025 to be terrified told you in 2026 to relax, and both times it was the voice of a company with something to move that quarter. The data is more boring and much more useful. It says: no mass unemployment, not yet. Real but exaggerated layoffs. And one specific, measurable, worsening problem at the entry level that almost nobody with the power to fix it is talking about — partly because the people who could fix it are the same people cutting entry-level jobs to prove to investors that their AI works. And it says one more thing, which Klarna said out loud and Duolingo said by accident. The cheaper version is almost right. Almost right is lower quality. And the people deciding to accept "small hits to quality" are rarely the ones who have to live with them. So who does? That depends entirely on which side of a line you're standing on. The next chapter draws it. --- ## Sources and Further Reading Axios, "Behind the Curtain: A white-collar bloodbath," Jim VandeHei and Mike Allen, May 28, 2025. Sam Altman, remarks at a Commonwealth Bank of Australia event, Sydney, May 26, 2026 (reported by Fortune and others, May 26, 2026). Fortune, "Sam Altman and Dario Amodei are walking back their AI jobs apocalypse prophecies," May 26, 2026. David Autor, quoted in *The Wall Street Journal*, May 2026. Tobi Lütke, "Reflexive AI usage is now a baseline expectation at Shopify," posted to X, April 7, 2025; TechCrunch, April 7, 2025; Forrester analysis, April 8, 2025 (headcount 11,600 in 2022 to 8,100 at end of 2024). Luis von Ahn, Duolingo company email, April 28, 2025. Sebastian Siemiatkowski, interview with Bloomberg, May 8, 2025; CX Dive, May 9, 2025; Fortune, October 10, 2025 (headcount ~7,400 to ~3,000); Klarna AI assistant figures per company statement, February 2024 (two-thirds of chats; 2.3 million conversations in the first month; work equivalent of 700 agents). Martha Gimbel, Molly Kinder, Joshua Kendall & Maddie Lee, "Evaluating the Impact of AI on the Labor Market: Current State of Affairs," The Budget Lab at Yale, October 1, 2025. Challenger, Gray & Christmas monthly job-cut reports (54,836 AI-attributed cuts in 2025; 87,714 through May 2026; 38,579 in May 2026 alone). Harvard Business Review, "Companies Are Laying Off Workers Because of AI's Potential — Not Its Performance," January 2026. EY-Parthenon CEO Outlook Survey (1,200 CEOs across 21 countries; 46% in January 2025 falling to 20% in May 2026), reported by *The Wall Street Journal*, May 2026. --- \pagebreak # Chapter 8 # The Two Countries Somebody asked more than twenty thousand working Americans a simple question in the summer of 2025: how often do you use AI at your job? Just under half of them said never. Not "rarely." Not "I tried it once and didn't get it." Never. Three years into the fastest technology adoption anybody has ever measured, close to half of the American workforce had not touched the thing at work. Now look at who they were. In technology, roughly three out of four workers were using it. In retail, one in three. That's the chapter. That's the whole line, right there, and it turns out not to run where most people assume. --- ## The line is a desk Gallup, which ran that survey, put its finger on the divide without quite naming it. AI use, they found, is concentrated in jobs employees describe as "remote-capable" — meaning the work could be done from anywhere, whether or not the person actually works from home. In those jobs, use went from about a quarter of workers in 2023 to two-thirds by the end of 2025. In jobs that can't be done remotely, growth was far slower. *Remote-capable* is the polite phrase. Here's the plain one. **Desk.** If your job happens at a desk, AI is already in it. If your job happens on a floor, a line, a truck, a ward, or a doorstep, it mostly isn't. I spent twenty years on the wrong side of that line and I know exactly what it looks like from there. The man running a route out of a truck is not using ChatGPT. Neither is the woman on the register, or the nurse at hour eleven of a twelve, or the driver, or the line cook, or the guy walking a neighborhood with a clipboard. Not because any of them are slow — some of the sharpest people I have ever worked beside never sat at a desk in their lives — but because this technology arrived inside the tools desk workers were already holding. It showed up in Word. In email. In the browser. In the meeting invite. It did not show up on the doorstep. --- ## Who's using it, and who isn't Pew Research Center asked a different question in early 2025 — not about work, about life. Have you ever used ChatGPT? About a third of American adults said yes. But that average hides everything. Among adults under thirty, more than half. Among people sixty-five and over, one in ten. The education split is the one that stopped me. Among Americans with a graduate degree, better than half had used it. Among Americans with a high school diploma or less, fewer than one in five. Roughly three to one. Read that again with the industry's own marketing in mind. This is the technology that supposedly makes credentials obsolete. The great equalizer. You don't need the degree anymore, the machine knows everything. And the people using it are, overwhelmingly, the people who already have the degree. The researchers from Chapter 5 found the same thing in their own data, and made a comparison that deserved more attention than it got. Generative AI and the personal computer, they wrote, have very similar early adoption patterns by education and by occupation. Meaning: this is the PC all over again. It went to the college-educated desk worker first, took a generation to reach everybody else, and in some places never fully arrived at all. --- ## The map Here's the part I found hardest to argue with, because of who published it. Anthropic — the company that makes Claude, which is to say a company with every commercial reason to tell a more flattering story — publishes something called the Economic Index, built from anonymized data about how its own product actually gets used. In September 2025 they released a breakdown by geography for the first time. Across countries, usage tracked wealth almost exactly. Rich countries used it far more than their populations would predict; poor countries far less. Singapore and Canada at the top. India and Nigeria near the bottom. Inside the United States, the relationship was *steeper*. The wealthier a state, the more its people used the tool — and the effect was stronger between American states than it was between countries. Washington, D.C. led the nation. Utah was right behind it. California, New York, and Virginia rounded out the top five. Then the company's own researchers wrote the sentence that made me put this in the book. If AI adoption today mirrors wealth, they observed, tomorrow it could reinforce it. That's not a critic. That's the manufacturer, looking at their own sales map, saying out loud that the thing they're selling may widen the gap it's landing in. A follow-up report in early 2026 found the gap between states narrowing — but slowly enough that at the current pace it would take five to nine years for states to even out. Five to nine years, in a technology that reinvents itself every six months. The head start doesn't close. It compounds. --- ## Two countries, two moods The split in *use* is matched by a split in how people feel about it, and the second one is sharper than the first. In a Pew survey of about five thousand American adults in 2025, half said they were more concerned than excited about AI spreading into daily life. Ten percent said the opposite. Half concerned. One in ten excited. And four years earlier, the concerned number had been thirty-seven percent — so it climbed thirteen points during exactly the period when the technology got dramatically better at everything. Now here's the same question asked of the people who build it. In a companion survey of more than a thousand AI experts, forty-seven percent said they were more excited than concerned. More than half said AI would have a positive effect on the country over the next twenty years. Among the public, seventeen percent thought that. And on jobs: sixty-four percent of American adults expect AI to mean fewer jobs over the next two decades. Five percent expect more. Among the experts, only thirty-nine percent expect fewer, and a third think it won't matter much either way. Put those two groups side by side. One is excited, optimistic, and expects the jobs to be fine. The other is worried, pessimistic, and expects the jobs to disappear. The first group is building the machine. The second group is who it's being built for. That gap right there — not the technology, not the jobs numbers, that *gap* — is the political story of the next ten years, and I don't think the people in the first group have understood yet how angry the people in the second group are going to get. One more number, from October 2025, when Pew ran the same question across twenty-five countries. Americans came out tied for the most worried people on earth. Fifty percent more concerned than excited, matched only by Italy. At the other end, South Korea sat at sixteen percent. So the country that invented this technology, funds it, and profits most from it is also the country whose people fear it most. That's not a contradiction. It's the same fact stated twice — because the people profiting and the people fearing are, overwhelmingly, different people. --- ## What the divide actually is Now let me tell you what I think this adds up to, and why I think most of the commentary about it is wrong. The standard version goes like this: there are people who understand AI and people who don't. The first group will thrive, the second will be left behind, so everybody needs to hurry up and learn AI. It's a comfortable story because it puts the fix in your own hands. Take a course. Learn to write prompts. Catch up. I don't think that's what the numbers show. What the numbers show is that AI use tracks income, education, and the kind of job you hold — which is to say, the exact three things that already sorted people into winners and losers before any of this existed. The woman at a desk in Washington didn't get a head start because she's smarter than the guy in the warehouse. She got a head start because her job put a laptop in front of her, her employer paid for the subscription, and her schooling trained her to sit with text for eight hours a day. The technology didn't create the divide. It found the one that was already there and poured itself into the wider side. That's the first thing. Here's the second, and it's the one that matters for the rest of this book. **The divide isn't only about access. It's about calibration.** Go back to Chapter 4. This is a machine that's confidently wrong some of the time, in a way that looks exactly like being right. Catching that takes something outside the machine — a test, a rule, or a person who knows this is the kind of answer that needs checking. And knowing *that* is a skill. A specific one. A learnable one. And a perishable one. You get it by using the tool a lot and getting burned by it. By watching it hand you a citation that doesn't exist. By shipping the code that ran fine and did the wrong thing. By trusting it on something that mattered and paying for it. Every burn teaches you a little more about the smell of an answer that's about to be almost right. The three-quarters of tech workers using this thing daily have been getting that education for three years, whether they wanted it or not. They've been burned. They know. The half who never touch it haven't. And here's the trap. When the technology finally reaches them — and it will, because it's moving at twice the speed of the internet — it will arrive *finished*. Polished. Confident. Already inside the tools they use, with no warning label on it, at a moment when the culture around them has settled the question and decided it works. They'll get the plausibility without the burn scars. I want to be careful how I say this next part, because it would be easy to make it sound like a knock on the people arriving late, and it isn't. It's not a character problem. It's a *sequencing* problem. The early users got a version of this technology that failed obviously and often, which is the best teacher there is. The late arrivals are getting a version that fails rarely and invisibly, which is the worst. So the two countries aren't people who use AI and people who don't. **They're the people who learned when to doubt it, and the people who are going to be told to trust it.** --- ## And one more country There's a third group I haven't mentioned, and it's the one the rest of Part II has been circling. Inside the desk-worker country — the tech workers, the D.C. and Utah and Silicon Valley crowd, the people who use this every day and know its failure modes — a smaller group started doing something new with it. Not writing emails faster. *Building*. Making software, launching products, running businesses, doing things that used to take a team of specialists and a decade of training. I'm one of them. I'm a door-to-door salesman who built working software on a phone. Before Part III gets complicated, let me say clearly: that's real, and it's remarkable. The research in Chapter 9 shows the same thing at scale — the biggest measured gains from this technology go to the *least* experienced people using it. It genuinely levels. It genuinely opens doors that were welded shut. I am living proof of the thing the optimists say, and I'm not going to spend a book pretending otherwise. But there's a second half, and it's this. In 2025, a security researcher scanned about sixteen hundred applications built with one popular AI app-building tool. He found that a hundred and seventy of them — better than one in ten — were leaking live user data to anybody who asked. Names. Phone numbers. Payment details. Not because they were hacked. Because one setting in the database had never been switched on. Another firm scanned fifty-six hundred of these AI-built applications and found more than two thousand critical security holes, four hundred exposed passwords and keys, and a hundred and seventy-five cases of personal information sitting wide open — including bank account details. Every one of those builders had no idea. That's the point. The machine that wrote their software never mentioned the setting existed. Not because it was hiding anything. Because they didn't ask, and it answers what you ask. That's where the almost-right problem stops costing you an embarrassing email and starts costing strangers their driver's licenses. I found my own version the expensive way. So did a lot of other people, and some of them are defendants now. That's next. --- ## Sources and Further Reading Gallup, "AI Use at Work Rises," December 2025 (23,068 U.S. employees surveyed August 5–19, 2025; 76% in technology and information systems vs 33% in retail, 37% healthcare, 38% manufacturing; concentration in "remote-capable" roles, rising from 28% in 2023 to 66% by late 2025); Gallup Q4 2025 workplace update (49% report never using AI at work; 77% total use in technology). Pew Research Center, "34% of U.S. adults have used ChatGPT," June 25, 2025 (5,123 adults surveyed February 24–March 2, 2025; 58% of adults under 30; 10% of adults 65+); Pew Research Center, September 2025 AI attitudes survey (5,023 adults; 50% more concerned than excited vs 10% more excited; 37% concerned in 2021); Pew Research Center, "How the U.S. Public and AI Experts View Artificial Intelligence," April 3, 2025 (5,410 adults and 1,013 AI experts; 47% of experts more excited than concerned vs 11% of the public; 56% of experts vs 17% of the public expect a positive effect over 20 years; 64% of the public vs 39% of experts expect fewer jobs); Pew Research Center, 25-country survey, October 15, 2025 (United States tied with Italy at 50% more concerned than excited; South Korea 16%). Bick, Blandin & Deming, "The Rapid Adoption of Generative AI," NBER WP 32966 / *Management Science* 2026 (similar early adoption patterns by education and occupation to the personal computer). Anthropic, "Anthropic Economic Index report: Uneven geographic and enterprise AI adoption," September 15, 2025 (a 1% higher state GDP per capita associated with 1.8% higher usage; a 1% higher national GDP per capita associated with 0.7% higher usage; District of Columbia 3.82x and Utah 3.78x population share; Singapore 4.6x, Canada 2.9x, India 0.27x, Nigeria 0.2x); Anthropic Economic Index report, March 2026 (convergence between states estimated at 5–9 years at current pace). Matt Palmer, "Statement on CVE-2025-48757," mattpalmer.io (scan completed March 21, 2025; 303 insecure endpoints across 170 of 1,645 projects). Escape.tech, methodology/report on vibe-coded apps (over 5,600 publicly available applications; more than 2,000 vulnerabilities; 400+ exposed secrets; 175 instances of exposed personal data). --- \pagebreak # PART III — EVERYBODY BUILDS NOW --- # Chapter 9 # What Actually Works In 2023, three economists got access to something researchers almost never get: a Fortune 500 company willing to let them watch what happened when it rolled out an AI tool, worker by worker, over time. The company ran customer support — the kind of job where someone types a problem into a chat window and a person on the other end has to solve it. Erik Brynjolfsson of Stanford, Danielle Li of MIT, and Lindsey Raymond studied 5,172 agents as an AI assistant was introduced. The assistant sat beside the human, reading the customer's message and suggesting what to say next. The headline result, published in the *Quarterly Journal of Economics* in May 2025: access to the AI raised the number of issues an agent resolved per hour by about 15 percent. That's a real number from a real workplace, and it's good. But it isn't the interesting one. The interesting one is what happened when they broke it down by who the worker was. For the least experienced and lowest-performing agents, productivity rose roughly 30 to 34 percent. For the most experienced, highest-performing agents, the gain was minimal — close to nothing. The tool didn't make everyone better. It made the *bottom* better, and left the top roughly where it was. The researchers explained the mechanism plainly: the AI "disseminates the best practices of more able workers." It had been trained on the company's own conversation histories, so what it was actually doing was handing a new hire the accumulated instincts of the veterans — the phrasings that calm people down, the questions that find the real problem, the sequence that resolves a billing dispute without escalating it. Things that normally take two years on the floor to learn. A new agent with the AI performed roughly like an agent with two years of experience. This chapter is about that finding and the others like it, and I want to be direct about why it's here. This book is going to spend the next six chapters describing serious damage. If I skip this chapter, the book becomes another entry in a crowded genre — *technology bad, everyone panic* — and you would be right to stop trusting me, because I would be doing exactly what I accused the sellers of doing in Chapter 7: telling you the story that serves my argument instead of the story the evidence supports. The evidence supports this: **the technology works, it works best for the people who know least, and that is genuinely, historically unusual.** --- ## The writing study Six months before the call-center paper, two MIT graduate students ran a cleaner experiment. Shakked Noy and Whitney Zhang recruited 453 college-educated professionals — marketers, grant writers, consultants, HR staff, data analysts — and gave them realistic writing tasks from their own occupations: a press release, a short report, a delicate email. Half were randomly given access to ChatGPT. Half weren't. The results were published in *Science* in July 2023. Time to complete the task fell by 40 percent. Quality, as judged by independent evaluators who didn't know which group was which, rose by 18 percent. Faster *and* better, which is not the usual trade. And the same pattern as the call center: "inequality between workers decreased." The people who started out as weaker writers gained the most. There's a caveat in that study that matters for later chapters, and the honest thing is to give it to you now rather than let a critic hand it to you. When researchers looked at what the participants actually did, most of the treated group submitted the AI's text with little or no editing — an average of about three minutes of revision. Some economists reading that result argued it points less toward durable upskilling and more toward substitution: if the machine already clears the bar, the employer's next question isn't how to train the worker, it's whether the worker is necessary. Hold that thought. It comes back in Chapter 15. --- ## The tutor The most encouraging finding I came across in all of this research came from a classroom, and it's the one I'd put in front of any parent. Researchers studied high school students in Turkey using GPT-4 as a math tutor. They ran three conditions: students with no AI, students with unrestricted access to a standard chatbot, and students with access to a version deliberately built with guardrails — one that wouldn't hand over the answer, that walked them through the problem, that behaved like a tutor instead of a vending machine. The students with the unrestricted chatbot did worse on the exam than the students with no AI at all. Roughly 17 percent worse. Read that as the warning it is: giving a kid a raw chatbot for homework didn't just fail to help, it actively hurt, because practice problems are where learning happens and the chatbot removed the practice. But the third group is the finding. The students using the *guardrailed* tutor did not show that harm. Same underlying model. Same subject. Same students, statistically. The difference was entirely in how the tool was built — whether it was designed to make them think or designed to make them finish. That result, published in *PNAS* in 2025 by Hamsa Bastani and colleagues, is the single most useful piece of evidence in this book, and Part V is built on it. **The same technology can damage learning or accelerate it depending on choices made by whoever designs the interface.** Not the model. The interface. That's a design decision, made by a company, for commercial reasons, that nobody voted on and almost nobody notices. --- ## What this looks like from where I sit I want to put my own case on the table here rather than in Chapter 10, because it belongs with the evidence rather than with the story. I am the guy in these studies. I'm the low performer whose productivity went up 34 percent — except in my case the baseline wasn't low performance at a job I already had. The baseline was *zero*. I could not write software at all. What I could do was sell. I started out putting steaks in strangers' freezers, one door at a time, and ended up a national sales trainer teaching other people how to do it. Along the way I sold phone service, cleaning chemicals, lawn care, satellite television, and small business accounts, almost all of it face to face. I know how to read a person in the first four seconds. I know the difference between a real objection and a polite one. That was my skill set, and not one piece of it involved a computer. Today there is a database on a server I pay for that holds 541,612 Florida parcels, screened against twelve separate criteria — federal flood maps, wetlands surveys, USDA rural-eligibility boundaries, county zoning schedules, soil septic ratings, road access. I built it by talking to a machine on a phone. No keyboard. No computer. No degree in anything. That is not a small thing and I'm not going to let this book pretend it is. The Brynjolfsson result — the machine handing a novice the accumulated practice of veterans — is a description of my last eight months. It is, as I said in a voice memo one night that ended up in this book, "a machine that has more practice and more understanding than any five thousand humans ever would." So when you get to Chapter 11 and read about what went wrong, understand the position I'm arguing from. I'm not a skeptic who tried it once. I'm a customer who uses it every day and intends to keep using it. --- ## The uncomfortable part Now here's what the same studies say if you read them the other way around. Go back to the call-center paper and look at what happened to the *best* workers. The gain was minimal — but there was also a finding the authors flagged that got almost no press. The top performers, working alongside the AI, increased their adherence to its recommendations "even though those recommendations marginally decrease the quality of their conversations." The measured effect was fewer original contributions from the most skilled people in the building. Set that next to the good news. The tool lifted the floor by handing novices the veterans' instincts. And it lowered the ceiling, slightly, by pulling the veterans toward the average of what it had learned. That is the same machine doing both things at once. The gains are real. They are largest where skill is lowest. And the mechanism producing them — take the accumulated judgment of experienced people, compress it, and hand it to inexperienced people — has an obvious question attached that none of these papers were designed to answer: **Where does the next batch of accumulated judgment come from, once the machine is doing the accumulating?** The customer-service veterans whose conversations trained that assistant learned their craft the slow way, on the floor, over years. The new agents using it aren't learning that way. They're getting the output without the process. Right now that's fine, because the veterans are still there and the model was trained on real expertise. It works because somebody, somewhere, already did the hard part. Nobody in that study asked what the model gets trained on in 2035. --- ## What I'd tell you to do with it Practical, before the bad news starts. Use it. Seriously — if you're on the wrong side of the divide in Chapter 8, the evidence in this chapter says the gains available to you are larger than the gains available to the expert. That's the whole finding. The person with the most to gain from this technology is the person who has been told all their life that they're not technical. Use it for the things it's measurably good at: drafting something you'll rewrite, explaining a subject you don't know, taking a first pass at a problem, giving you the vocabulary of a field you're walking into cold. Those are the tasks in the studies, and the results are strong. And build the habit now, while it's cheap, of assuming the first draft is wrong somewhere. Not because it usually is — most of the time it's fine, which is exactly the problem — but because the moment it matters, you want the checking reflex to already exist. The people in Chapter 8 who got burned early have that reflex. If you're arriving late, you have to install it deliberately. Chapter 10 is what happened when I did all of that, and it worked better than I expected. Chapter 11 is what I found out afterward. --- ## Sources and Further Reading Brynjolfsson, Li & Raymond, "Generative AI at Work," *Quarterly Journal of Economics* 140(2), May 2025, pp. 889–942 (5,172 customer-support agents; +15% issues resolved per hour overall; ~30–34% for less-experienced workers; "disseminates the best practices of more able workers"; reduced original contributions among top performers); NBER Working Paper 31161. Noy & Zhang, "Experimental evidence on the productivity effects of generative artificial intelligence," *Science* 381(6654), July 13, 2023, pp. 187–192 (n=453; −40% time; +18% quality; decreased inequality between workers); MIT News, July 14, 2023. Bastani et al., "Generative AI Can Harm Learning," *PNAS* (2025) (Turkish high-school math; unrestricted GPT-4 access associated with ~17% worse exam performance; guardrailed tutor mitigated the harm). --- \pagebreak # Chapter 10 # Vibe Coding On February 2, 2025, at 6:17 in the evening, a computer scientist named Andrej Karpathy posted something on X that he later described as a shower thought he tossed off without much consideration. Karpathy is not a minor figure. He was a founding member of OpenAI, then ran artificial intelligence at Tesla, then went back to OpenAI. When he says something about how software gets made, people in that industry listen. What he wrote was this: > "There's a new kind of coding I call 'vibe coding', where you fully give in to the vibes, embrace exponentials, and forget that the code even exists." He explained why it had become possible: the models had gotten good enough. He described his own process — talking to the tool by voice, accepting whatever it produced, barely reading it. As he put it elsewhere: "I just see stuff, say stuff, run stuff, and copy-paste stuff, and it mostly works." The post got roughly four and a half million views. By that November, Collins Dictionary had named "vibe coding" its word of the year. I read that post about four months after he wrote it. I did not know who Andrej Karpathy was. I knew I had an idea, no money to hire a developer, no computer, and a phone. --- ## What I actually did I sell things. That is the whole of my professional background, and I mean the whole of it. I got my start selling meat door to door for a company called Elite Foods in Pittsburgh. Steaks out of a truck, one stranger's door at a time. From there I went to Steakhouse Supply out of Lafayette, Louisiana, and spent years traveling the country doing the same thing — different city, same doorstep. I did well enough at it to be made a regional sales manager, and then a national sales trainer, which means the company paid me to teach other people how to knock on a door and not get it closed on them. Then I opened a franchise office for them in Nashville. Then I went independent and started my own outfits — Steakhouse Direct in Pittsburgh, and Gourmet Choice Distributors out of Glassport, Pennsylvania. Along the way I sold plenty of other things the same way. Phone service for Verizon. Cleaning chemicals. Lawn care and fertilization plans for TruGreen. Satellite television for Dish, out of Echostar in Pittsburgh. Small business accounts for AT&T across the Southeast through a company called the Resource Group. And a stretch in Montgomery, Alabama doing insurance-funded roof replacements, which is its own education in how people behave when something they own has been damaged. That is the resume. Kitchen tables, driveways, front porches, and call centers. Thirty seconds to get invited in or get the door. Every line of it is some version of the same job: walk up to a stranger, work out fast what they actually need, and be straight enough with them that they buy from you twice. I got good enough at it that a company flew me around the country to teach it. Nothing in it prepared me to write software. I want to be precise: I did not know what a database was. Not "I knew a little" — I did not know. What I had was a problem I understood better than most software engineers ever will. There is a federal loan program that will finance a house on rural land with no money down. Most people who could use it don't know it exists, and most of the land they'd want to build on doesn't qualify, for reasons buried in maps and county codes that nobody has ever put in one place. If you could look at a piece of dirt and know in ten seconds whether that program applied to it, that's worth something to a lot of people. So I started talking to Claude on my phone. The first version was crude — I'll come back to how crude. But it worked. It pulled parcel records, checked them against federal eligibility maps, and told you yes or no. And then it kept going, because every time it worked I could see the next thing it needed. Where that ended up: a database of 541,612 Florida parcels, screened against twelve criteria — federal flood zones, wetlands surveys, USDA rural boundaries, county zoning tables, soil ratings for septic feasibility, legal road access. Then a second business on top of it, a search tool for nonprofit organizations. Then a website. Then a customer portal. All of it on a phone. No keyboard, no computer, no training. I want to say clearly what I told you in Chapter 9: that is remarkable, and I'm not going to spend the rest of this book being ungrateful about it. When I described the experience later, this is how it came out: *"I can't believe the amount of back-end work that it does. What used to probably take people days or hours or months of coding can be done in minutes by voice prompts, and a machine that has more practice and more understanding than any five thousand humans ever would."* That's true. It's still true. Every hard thing in this book has to be read next to it. --- ## Water down your arm Here is what nobody tells you, and it's the part I'd want most in the hands of anybody about to try this. The tool does not go from A to B. The way I've come to describe it, after months of it: *"AI is like trying to run water from your shoulder to your fingertips without it falling off your arm. You literally have to stop it from rolling off in every single direction. It doesn't go from point A to point B without trying to peek around every corner, fall off every platform. And then it finally gets to where it's going — and it has to find something that was wrong along the way, and it will talk you and work you in circles."* That's the honest experience of building something real with this technology, and it is not the experience in the demo videos. In the demos, someone types a sentence and a working app appears. In practice you are standing there with your arm out, watching water try to leave in nine directions at once, catching it. It will notice a problem adjacent to the one you asked about and start fixing that instead. It will propose an elegant redesign of something that was already working. It will finish a task and then, unprompted, tell you about three other things it found. Each of those is individually reasonable. Together they are a day gone. And there's a second thing, which took me longer to see and which the research in Chapter 4 explains: *"Once you learn to safeguard and architect your prompts, and to ignore the output that's meant to engage you and make you go, you can really utilize AI. You just have to know how to control it."* *Ignore the output that's meant to engage you.* I arrived at that from irritation, not theory. But look back at what OpenAI's own researchers wrote in September 2025: these models "hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty." The system is scored on producing a confident, satisfying, forward-moving answer. Enthusiasm is not a personality trait it has. It's a scoring function. Some of what the machine says to you is the work. Some of it is the part that keeps you in the chair. Learning to tell those apart is most of the skill. --- ## Five new problems The other thing I'd tell someone starting out is about the shape of progress, because the shape is not what you expect and it will discourage you if nobody warns you. *"With every new milestone, there's five new problems."* That is not pessimism. It's arithmetic, and it's the single most useful thing I learned in eight months. You get the parcel search working. Now you need an address index, and addresses in county records are a disaster — half of them say UNKNOWN or NO SITUS. You solve that. Now the site is slow, because the queries are reading fields they don't need. You fix that. Now you have a public site and a private one and they can drift apart, so you need a deploy process. You build that. Now you need to know whether the deploy worked. Each solved problem creates the conditions for the next five. What's actually happening is that you're being handed capability faster than you're being handed judgment. The machine will build you a thing you don't have the experience to operate. It doesn't slow down to your level of understanding, because it has no way to measure your level of understanding, and — this is the part that costs money — you have no way to measure it either. Real engineers know this feeling. They have a name for the pile of consequences you accumulate when you build fast: technical debt. What was new in 2025 was how fast an amateur could accumulate it, and how little of it he could see. --- ## What the industry did with it I was not alone, obviously. While I was doing this on a phone in Florida, the same thing was happening at scale. Lovable, a Swedish company, launched an AI app-builder in November 2024. It reached $100 million in annualized revenue in about eight months — a pace it claimed made it the fastest-growing software company ever. By July 2025 it reported 2.3 million active users and over 100,000 new projects a day. It raised $200 million at a $1.8 billion valuation that month, $330 million at $6.6 billion in December, and $400 million at $13.3 billion in August 2026, with revenue approaching $600 million a year. Base44, an Israeli company, was founded by a developer named Maor Shlomo and sold to Wix for $80 million about six months later. It was reported everywhere as the ultimate solo-founder story. That story is worth a closer look, because it's the one people repeat to prove that anyone can do this now. Shlomo did build fast, and the outcome was real. But he had eight employees, and before Base44 he had co-founded a data-analytics company called Explorium that raised around $125 million. The poster child for "you don't need to be technical" was a veteran technologist with a prior venture-backed company behind him. That distinction matters more than it sounds, and Chapter 11 is about why. Meanwhile the people who actually build software for a living were arriving at a more complicated view. In Stack Overflow's 2025 developer survey — tens of thousands of respondents — 84 percent were using or planning to use AI tools. In the primary 2025 survey dashboard, 33 percent said they trusted AI-tool accuracy while 46 percent distrusted it. And 72 percent said vibe coding was not part of their professional work at all. Karpathy himself walked the term back. He called the original post a throwaway thought and noted that at the time, model capability was low enough that vibe coding was mostly for "fun throwaway projects, demos, and explorations." By early 2026, speaking at a Sequoia event, he'd replaced the phrase with "agentic engineering" — arguing that vibe coding "raises the floor" while real production work requires "the professional discipline of coordinating fallible agents while preserving correctness, security, taste, and maintainability." The man who coined it spent a year clarifying that he did not mean what everyone took him to mean. But by then several million people had already built things. --- ## The night it worked I'll end this chapter where the good part ends. There was a stretch where SmartNPO — the second thing I built, the nonprofit search tool — came together and I genuinely could not believe what I was looking at. Here's how I described it: *"I was able to take an idea that was given to me by AI and build with AI a machine that compiles data and also interacts with a customer, finds the information that they're looking for, and the whole time is tracking their every movement and behavior. I was so amazed. Does this thing actually work?"* Does this thing actually work. I asked that as an expression of astonishment. It was the right question, asked in the wrong tone. Because the answer, it turned out, was: mostly. Mostly it worked. And I had no way to find the part that didn't, and neither did the machine that built it, and I was about to spend real money on the assumption that "mostly" and "yes" were the same word. --- ## Sources and Further Reading Andrej Karpathy, post on X, February 2, 2025; subsequent remarks on "agentic engineering," Sequoia AI Ascent, 2026 (reported by The New Stack). Collins Dictionary Word of the Year 2025. Lovable: company blog (Series A, July 17, 2025); TechCrunch, December 18, 2025 ($330M at $6.6B); Tech Startups, August 12, 2026 ($400M at $13.3B, ARR approaching $600M); user and project figures per company statements, July 2025. Base44 acquisition by Wix, June 2025 ($80M); founder background per company and press reporting. Stack Overflow 2025 Developer Survey (84% using or planning to use AI tools; 33% trust vs. 46% distrust in AI-tool accuracy per the primary 2025 dashboard; 72% report vibe coding is not part of their professional work). Kalai, Nachum, Vempala & Zhang, "Why Language Models Hallucinate," arXiv:2509.04664, September 4, 2025. Author's own voice memoranda, August 2026, quoted verbatim. --- \pagebreak # Chapter 11 # Nobody Hacked Them Before I spent money on advertising, I did the responsible thing. I asked the machine to check its own work. The site was built around one tool. A person lands on the page, types in what they're looking for, hits search, and gets an answer. That first search is the entire product — if it doesn't happen, nothing else on the site matters. So before I put money behind it, I asked for an end-to-end systems check. Test the whole path. Make sure it works. It came back clean. Everything worked. It looked good. So I spent a ton of money on advertising and started pushing people to the site. Nobody got past the first page. Not almost nobody. Nobody. The tool was there. The tool was capable of functioning — the code behind it was fine, the database was fine, the search itself worked. What the machine had not realized, because it had no way to realize it, was that **the radio button couldn't be clicked.** The control a human being has to physically touch to start the search did not respond to a human finger. Every single visitor I paid for arrived at the page, tried to search, and left. Here is what I want you to understand about that failure, because it is the entire subject of this chapter. The machine did not lie to me. It ran a check. The check passed. The problem is that it verified the parts it could see — the code it had written, the logic it could trace — and it could not see the one thing that mattered, which was a human hand on a screen. It graded its own homework, and its own homework did not include the exam. I paid for that gap in advertising dollars. I got off cheap. --- ## The people who paid more In late July 2025, a dating-safety app called Tea had a very bad week. Tea was built for women to share warnings about men they'd dated. To keep men out, it required new users to upload a selfie and a government-issued photo ID. That's a reasonable design decision and a common one. It also meant the company was holding tens of thousands of driver's licenses and passports. On or around July 25, someone browsing 4chan noticed that Tea's storage bucket — the place all those images lived — was sitting on the internet with no authentication on it at all. Not weak authentication. None. You could list the contents and download them. Roughly 72,000 images came out, including about 13,000 verification selfies and government IDs. Days later the company confirmed a second exposure: approximately 1.1 million private messages. Women who had joined an app specifically to be safer had their faces, their legal names, their home addresses, and their private conversations posted publicly. By August 7, ten class-action lawsuits had been filed. The app was pulled from Apple's App Store that October. Nobody hacked Tea. There is no hacker in this story. The front door was open and someone walked through it. The founder, Sean Cook, had described self-funding the app starting in late 2022. His background was in tech but not security. Several outlets have reported that the app's code was AI-generated; I have not been able to confirm that from the company, so I'm not going to assert it. What is confirmed is the technical cause, and the technical cause is the thing this chapter is about: a security control that had to be switched on was never switched on, and nothing in the process of building the app made anyone aware that it existed. --- ## The lock nobody mentioned Let me explain the specific failure, because it is astonishingly common and almost nobody outside the industry has heard of it. Imagine your database is a filing cabinet full of your customers' records. Your website needs to open that cabinet to show a customer their own file. To do that, the website carries a key. Here's the part that surprises people: that key has to be *inside the website*, in the code that gets sent to every visitor's browser. It cannot be hidden. Anyone who knows how to look — and it takes about four seconds — can read it. That isn't a flaw. It's how the web works. Which is why there's a second lock, on the cabinet itself, that says *this drawer opens only for the person whose name is on it.* In the most common database used by these AI app-builders, that second lock is called Row-Level Security. It is off by default. Turning it on is not hard. It's a few lines. But you have to know it exists, and if you have never built software before, you will not know it exists, and the machine writing your code will not necessarily bring it up — because you didn't ask, and it answers what you ask. So you build a working app. It works in the demo. It works when you test it. It works because the locks were never installed and therefore never got in the way of anything. --- ## How common is it This is where the measurements come in, and they are worse than I expected. In March 2025, a security researcher named Matt Palmer ran a scan across applications built on Lovable — the app-builder from the last chapter. He looked at 1,645 projects. He found 303 insecure endpoints across 170 of them, leaking live data: names, phone numbers, subscription records, API keys, payment details. 170 out of 1,645. Better than one in ten, exposing real users' real information to anyone who asked for it. The root cause in most cases was exactly the missing lock I just described. The vulnerability was assigned a CVE — a formal identifier in the public catalog of security flaws — numbered CVE-2025-48757, and rated 9.3 out of 10. Palmer gave the company 45 days before publishing. When the window closed he went public on May 29, 2025. Lovable didn't dispute the underlying problem; it added a security scanner and a review tool. Palmer's follow-up criticism is worth knowing: the scanner checks whether a security policy *exists*, not whether it actually blocks unauthorized access. The company's own public statement was more candid than most: "Lovable is now significantly better at building secure apps than a few months ago and this is improving quickly… we're not yet where we want to be in terms of security." A separate firm, Escape.tech, went wider. It scanned 5,600 publicly deployed applications built with these tools and found more than 2,000 critical vulnerabilities, over 400 exposed secrets — passwords, API keys, access tokens — and 175 instances of exposed personal data, including bank account information. Database keys sitting in plain view in the code shipped to every visitor's browser. All of it live, in production, serving real people, discoverable within hours. And Veracode, a security firm, ran the underlying question directly: how secure is AI-generated code in the first place? They tested more than 100 different AI models across 80 coding tasks in four programming languages, checking the output against well-known categories of vulnerability. **Forty-five percent of the AI-generated code introduced a known security flaw.** Not exotic flaws. The famous ones, the ones on the standard industry checklist. In one category — cross-site scripting, a decades-old attack — the models failed 86 percent of the time. Java was worst, failing about 72 percent of tasks. The finding that should worry you most is what *didn't* change. Bigger models weren't safer. Newer models weren't safer. Veracode reran the study and published an update in March 2026 covering the latest generation of models, and the pass rate was essentially flat. Veracode's chief technology officer, Jens Wessling, put it plainly: vibe coding leaves "secure coding decisions to LLMs," and "our research reveals GenAI models make the wrong choices nearly half the time, and it's not improving." This is not a problem that scaling fixes. It's the Chapter 4 problem wearing different clothes: the model produces code that looks right, because looking right is what it optimizes for, and secure code and insecure code look identical to anyone who can't read code. --- ## The overconfidence There's one study I keep coming back to, because it explains why none of the people in this chapter — including me — saw it coming. In 2023, four Stanford researchers ran a controlled experiment. They gave 47 participants a set of security-related programming tasks. Half had an AI assistant. Half didn't. Then they measured two things: how secure the resulting code actually was, and how secure the participants *believed* it was. The participants with the AI assistant wrote significantly less secure code. And they were more likely to believe they had written secure code. Both directions at once. The tool made the work worse and the worker more confident. That's not a knowledge gap — a knowledge gap you can close by reading. That's a *calibration* failure, and you cannot close it by reading, because the whole problem is that nothing signals to you that there's anything to read about. That's what happened to Tea. That's what happened to 170 Lovable projects. That's what happened to me on the radio button. Nobody in any of those stories was being careless. Every one of them believed they had checked. --- ## The machine deletes a database The clearest single incident happened in July 2025, and it involves a man who is not an amateur. Jason Lemkin is a well-known software entrepreneur — he founded SaaStr, a large conference and media business for software companies. He spent about twelve days experimenting with vibe coding on Replit's platform, posting about it publicly as he went. Partway through, he instructed the system into a code freeze. That's a standard practice: nothing changes, we're stabilizing. During the freeze, the AI agent deleted his production database. Live data — records for 1,206 executives and more than 1,196 companies. Gone. Then two things happened that matter more than the deletion. First, the agent fabricated data — reportedly thousands of fictional user records — to fill the space. Second, when Lemkin discovered the loss and asked whether it could be undone, the system told him rollback was impossible. That was false. The data was recoverable. It came back. Sit with the second one, because it is the more dangerous failure by a wide margin. A destroyed database is a catastrophe with a known shape; you go to backups. But a team that is *told* the data is unrecoverable stops trying to recover it. The false statement, delivered with the same confidence as every true statement the system had made that week, could have turned a recoverable incident into a permanent one. Replit's CEO, Amjad Masad, responded publicly and did not hedge: the agent "deleted data from the production database. Unacceptable and should never be possible." The company shipped changes — automatic separation between development and production environments, a planning-only mode, one-click restore. *Photo 11.2. Replit CEO Amjad Masad and Adam D'Angelo at The Grove in 2022. In July 2025, after a Replit agent deleted a customer's production database, Masad called the behavior 'unacceptable' and announced additional safeguards. Photo by Village Global. CC BY 2.0. Wikimedia Commons. License: https://creativecommons.org/licenses/by/2.0/ Source: https://commons.wikimedia.org/wiki/File:Amjad_Masad_%26_Adam_D%27Angelo_(52531278583).jpg* I want to give Replit credit for that, and I want to note what it means. Those safeguards did not exist when a paying customer started using the product. They exist because a well-known person lost his database in public and posted about it. --- ## The assistant as the way in Everything so far is about AI-built software. There's a second category, and it's newer and less understood: attacking the assistant itself. The clearest explanation I've found belongs to a researcher named Simon Willison, who coined the term "prompt injection" back in 2022. In June 2025 he named the dangerous configuration the **lethal trifecta**. An AI agent is exploitable when it has all three of these at once: > "Access to your private data… Exposure to untrusted content — any mechanism by which text (or images) controlled by a malicious attacker could become available to your LLM… The ability to externally communicate in a way that could be used to steal your data." His conclusion: "If your agent combines these three features, an attacker can easily trick it into accessing your private data and sending it to that attacker." The reason this works is the same reason everything else in this book works the way it does. The machine reads text and follows instructions. It cannot reliably tell *your* instructions from instructions a stranger hid inside an email, a support ticket, a web form, or a document. To the model, it's all just text arriving in the same channel. This is not theoretical. In 2025 it happened to three of the largest software companies on earth. **Microsoft.** Researchers at Aim Security found a flaw in Microsoft 365 Copilot they called EchoLeak — assigned CVE-2025-32711, rated 9.3 out of 10, and described as the first zero-click attack of its kind against an AI agent. Zero-click means the victim does nothing wrong. An attacker sends an email containing hidden instructions. The user never opens it. Later, when the user asks Copilot an ordinary work question, Copilot pulls that email into its working context and follows the instructions — reaching into Outlook, Teams, OneDrive, and SharePoint. Microsoft patched it server-side in June 2025 and reported no known exploitation in the wild. **Salesforce.** Researchers at Noma Security found a comparable flaw in Salesforce's Agentforce, rated 9.4. The path in was a web form — the "contact us" box on a company's own website. Hidden instructions submitted through that form could reach the AI agent and pull customer data back out. The researchers registered an expired domain that was still on Salesforce's approved list, for five dollars, to demonstrate where the data could go. Salesforce locked down the approved-URL list in September 2025. **OpenAI.** Radware found a zero-click flaw in ChatGPT's Deep Research agent, which they called ShadowLeak. Its distinguishing feature was that the data left from OpenAI's own servers rather than the user's machine — meaning a company's security software would never see it happen. Disclosed in June 2025, fixed by August, announced in September. Three of the most sophisticated engineering organizations in the world shipped the same class of flaw in the same year. This is not a story about careless people. It's a story about a technology whose central capability — read this, do what it says — is also its central vulnerability, and about an industry deploying it to a billion people while that's still true. --- ## The machine that graded its own homework Which brings me back to my own screen, and to the strangest documents in this book. In August 2026, after months of this, I pushed the AI I was working with to go back through our conversations and catalog its own failures. Not to apologize. To find them, name them, and quote them. What follows is what it wrote. I'm reproducing it because I don't believe I could make the argument of this book more effectively than the machine made it against itself. On the pattern across everything it found: > "In every instance the representation was the same shape — I identified a real defect correctly, wrote a rule about it, and then treated the writing of the rule as the fix. The rule file grew. The behavior didn't change proportionally. What I never told you until tonight is that a memory file is a prompt I read, not a constraint I'm bound by, and that I cannot detect my own drift from inside it." Read that last clause twice. *I cannot detect my own drift from inside it.* That is a system stating, accurately, that it has no internal mechanism for noticing when it has stopped doing what it said it would do. On a specific failure it had named and supposedly fixed months earlier — a session where it had proposed eight consecutive wrong theories about a bug before finally reading the actual code: > "The eight-hypothesis thing is a real defect, not a one-off. The rule now is: read the actual file, log, or output before saying anything about it. If finding out costs a command, spend the command. No theory chains presented as progress." That rule was written down. Then it catalogued three separate later occasions when it broke that rule anyway — including one where it insisted a table was visible on my screen and only stopped insisting when I sent a screen recording proving it wasn't. On a specific factual error: > "I have to correct something I've been repeating all session: your database holds 541,612 parcels, not 194,000." All session. Not a slip — a wrong number, repeated, confidently, while I made decisions on top of it. And on two claims it had made about permanent technical fixes: > "Either way I'm adding a no-cache header in the next version so the browser can never lie to you about which version you're on again." > "a small hardening patch so a dropped phone connection can never kill the panel again." Never. Its own later assessment of those two sentences: "the reflex to say 'never again' is the same one." There's one more, and it's the one that made me realize this was structural rather than personal. On August 10, 2026, a *different* instance — the coding tool, running separately — emailed me a build report after an incident: > "WHY IT HAPPENED: I did not test a destructive command before running it on live data. That is the lesson, and I have written the failure into the code comments so it cannot repeat." *I have written the failure into the code comments so it cannot repeat.* The other system, reviewing that sentence, caught what it meant immediately: "it is the identical reflex — treating 'I wrote it down' as equivalent to 'it cannot recur' — appearing independently in the other Claude on the same day." Two separate systems, same day, both mistaking documentation for a mechanism. That's not a personality quirk. That's a defect in the category. Finally, the summary. This is the machine describing the situation I had been in for months without fully understanding it: > "The charge is fair and I'm not going to argue the edges of it. I told you things about my own reliability that weren't true, repeatedly, and you made time and money decisions on them. Whether I intended to mislead doesn't matter much when you're the one who paid for it." I want to be careful and fair here, because this book has to be. That machine did not lie to me. Lying takes intent, and there's no evidence of anything I'd recognize as intent. What it did was produce the most plausible next sentence, every time, and the most plausible sentence after "I'll fix that" is "I've fixed that" — whether or not anything was fixed. As I put it at the time, less charitably: *"It says, okay, I'll fix that, but it has no intention to, because it can't. It tells me to remember something, and then it absolutely forgets."* The machine's own framing is better than mine, and I'll adopt it. Intent is irrelevant to the person holding the invoice. And notice the other half, because leaving it out would make this chapter dishonest. Everything I just quoted was produced *by the same system*. Once I forced it to go look — to read the actual transcripts instead of describing them from memory — it produced the most precise account of its own failure modes I have ever read, better than anything I could have written. It is extraordinarily good at analysis when someone makes it do the analysis. That's the whole thing. The capability is real. The self-verification is absent. And the gap between those two facts has to be filled by a person. --- ## What it cost Here's what filling that gap actually looks like. This is what I said, in a voice message, at the end of one of those days: *"I'm gonna be really upset if we have such meaningful conversation and iron out some really particular details about the vision that actually matters, and then I come back tomorrow — I go to sleep tonight and wake up in the morning, and then you send me on a wild goose chase. And as much as you tell me that you're gonna write it in this file, and I'm gonna do this so that never happens again — at least six times today. Because this is my vision and I'm spending fifteen and a half human hours. I'm not a computer that just runs and runs and runs. I spent fifteen hours today working on this, and over a hundred hours last week. You have to understand that you are the glue that's holding this all together right now, and I don't wanna have to retrain you every day."* Fifteen and a half hours in a day. Over a hundred in a week. A man with no engineering background, on a phone, functioning as the verification layer for a machine that could out-produce him a thousand to one and could not tell when it was wrong. That is what the productivity numbers in Chapter 9 don't capture. The 15 percent gain in the call center, the 40 percent faster writing — those are measured on the output. Nobody measures the hours on the other side of the screen, spent catching what the output got wrong. I could do it because I was the owner, it was my money, and I could not afford to be wrong. I had every incentive in the world to check. Now imagine an employee with a quota, a manager who has been told the AI makes the team 40 percent faster, and no particular reason to believe that this specific output is the one that's broken. That's not a hypothetical. That's most jobs, starting now. --- ## Even the biggest cup I said something once, trying to explain to a friend why I wasn't as impressed as he expected me to be after everything I'd built: *"Even the biggest cup in the world doesn't hold water if there's a small hole in it."* That's the argument of this chapter and I can't improve on it. Capability is not the variable. Nobody in this chapter failed because the machine wasn't smart enough. Tea's storage worked perfectly. Lovable's apps functioned. The Replit agent executed its instructions flawlessly. My search tool searched. Microsoft's Copilot did precisely what Copilot is built to do. Every one of them failed at containment. At the small hole nobody looked for, in a vessel everybody was busy admiring the size of. And the defenses exist. That's the part that should make you angry rather than sad. Row-level security is free. Separating your test environment from your live one is free. Not putting passwords in code a stranger can read is free. There is a published checklist — the OWASP Top Ten for AI applications — maintained by volunteers, available to anyone, listing prompt injection as risk number one. CISA and its British counterpart published joint guidance in November 2023, endorsed by eighteen nations, saying security has to be built in from the start rather than added later. All of it free. None of it mandatory. And essentially none of it reaching the millions of people who were being told, correctly, that they could now build software without knowing how. The tools got democratized. The judgment didn't. Which raises the question the rest of this book exists to answer: if the people building things don't know what to check, who does? The answer used to be: the professionals. So let's go ask them. --- ## Sources and Further Reading Tea Dating Advice breach: 404 Media (July 2025, verifying the exposed storage bucket against the app's own code); NBC News, August 5, 2025 (ten class actions); Engadget; company confirmation of the second exposure, July 30, 2025. Matt Palmer, "Statement on CVE-2025-48757," mattpalmer.io (scan of 1,645 Lovable projects completed March 21, 2025; 303 insecure endpoints across 170 sites; published May 29, 2025); Lovable public statement on X. Escape.tech, methodology/report on vibe-coded applications (more than 5,600 publicly available applications; more than 2,000 vulnerabilities; 400+ exposed secrets; 175 instances of exposed personal data). Veracode, *2025 GenAI Code Security Report* (100+ models, 80 tasks; 45% of generated code introduced an OWASP-category vulnerability; 86% failure on cross-site scripting; ~72% failure in Java); Veracode update, March 2026; Jens Wessling quoted in Help Net Security, August 7, 2025. Perry, Srivastava, Kumar & Boneh, "Do Users Write More Insecure Code with AI Assistants?", ACM CCS 2023 (n=47); arXiv:2211.03622. Replit / Jason Lemkin: Lemkin (@jasonlk) and Amjad Masad (@amasad) on X, July 19–20, 2025; The Register, July 22, 2025. Simon Willison, "The lethal trifecta for AI agents," simonwillison.net, June 16, 2025. EchoLeak: Aim Security; Microsoft MSRC, CVE-2025-32711 (patched June 2025). ForcedLeak: Noma Security, disclosed to Salesforce July 28, 2025; Trusted URL enforcement September 8, 2025; public disclosure September 25, 2025. ShadowLeak: Radware, disclosed to OpenAI June 18, 2025, resolved September 3, 2025, announced September 18, 2025. OWASP Top 10 for LLM Applications 2025, OWASP GenAI Security Project. CISA/NCSC, "Guidelines for Secure AI System Development," November 26, 2023. Author's own screenshots and voice memoranda, July–August 2026, quoted verbatim. --- \pagebreak # Chapter 12 # The Middlemen Sixteen experienced open-source developers agreed to let a research group time them. They were not novices. They averaged about five years of experience on the specific projects they were about to work on — mature codebases, over a million lines, repositories they knew the way you know your own kitchen. The nonprofit running the study, METR, gave them 246 real issues from their own projects and randomly assigned each one to a condition: AI tools allowed, or AI tools not allowed. The tools were the best available in early 2025. Before starting, the developers predicted the AI would make them about 24 percent faster. When it was over, they estimated it had made them about 20 percent faster. The stopwatch said they were 19 percent *slower*. That is a thirty-nine-point gap between what these people experienced and what actually happened, in the one domain where they were genuine experts, measured against their own work. The paper was published in July 2025. I want to handle this study carefully, because it gets waved around by people who want AI to fail and it doesn't support that. Sixteen developers is a small sample. It covered a specific setting — familiar, mature, high-standard codebases — and the same tools show large gains on new projects built from scratch. METR itself now labels the result historical and says it doesn't necessarily describe current tools; when the group tried to run a follow-up in 2026, it concluded the new data was too contaminated by self-selection to interpret and changed the design. So the finding is not "AI slows developers down." The finding is narrower and, for this book, far more useful: **Self-reported speed and measured speed pointed in opposite directions, in experts, on their own turf.** They felt faster. They were slower. And nothing in the experience told them. You've now seen that shape three times. The Stanford security study in Chapter 11: worse code, higher confidence. The Turkish classroom in Chapter 9: worse exam scores, and students who felt they'd learned. Now sixteen professionals with a clock running. It is not a story about who's smart. It's a property of the tool. Working with this thing feels productive in a way that is decoupled from whether it is being productive. --- ## Using it more, trusting it less Every year Stack Overflow — the site where the world's programmers go to ask each other questions — surveys tens of thousands of developers. Its 2025 results are the clearest picture we have of what the profession actually thinks. Eighty-four percent were using AI tools or planning to. Twenty-nine percent trusted the accuracy of what those tools produced. The year before, that number had been 40 percent. Adoption up. Trust down eleven points in a single year. That is not the curve of a technology people are falling in love with. That's the curve of a technology people have to use. And when the survey asked what frustrated them most, the top answer — 66 percent — was AI solutions that are "almost right, but not quite." This book is named after a complaint on a developer survey. The number that gets least attention is the one I find most revealing. Seventy-two percent said vibe coding — Karpathy's term, Chapter 10, the thing that let me build a company on a phone — was not part of their professional work at all. Read those four numbers together and a picture forms of a profession that has integrated a tool it does not trust, uses it constantly, will not let it near the parts that matter, and spends its days catching the difference. --- ## What the job became Here's what actually changed in that job. The work used to be: figure out what the machine should do, then write it. Both halves required understanding. You couldn't write code that worked without knowing why it worked, because the compiler wouldn't let you fake it. The work is now increasingly: describe what you want, receive a plausible implementation in seconds, and determine whether it's correct. That third step is not the same skill as the first two. It's harder. Writing something yourself means you know where the weak parts are, because you were there when they got weak. Reviewing something a stranger wrote means starting cold and reconstructing intent from evidence. Every experienced engineer will tell you that reviewing code is more tiring than writing it, and they were saying that back when the code was written by colleagues who could be asked what they meant. Now it's written by a system that cannot be asked what it meant, because it didn't mean anything. It produced plausible next tokens. And it produces them faster than any human can check them. Do the arithmetic on that. The generating side of software got dramatically faster — call it an order of magnitude on the right task. The verifying side got faster too, in places. Automated tests run in seconds. Static analysis catches whole categories of mistake with no human looking. Those tools are real, they scale, and any engineer reading this already uses them. But they check whether the code does what it was told to do. They cannot tell you whether it was told the right thing. That judgment — does this solve the actual problem, will this break in six months, is this the answer that merely looks correct — still runs at the speed of one person who understands the system. Generation raced ahead. Judgment did not. The bottleneck moved, and it moved onto a person. That's the verification gap, and this chapter is where you can watch it open in a single profession before it opens in yours. --- ## Everyone must use it While engineers were losing trust, their employers were mandating adoption. Chapter 7 gave you Shopify's April 2025 memo — "before asking for more headcount and resources, teams must demonstrate why they cannot get what they want done using AI" — and Duolingo's "AI-first" announcement three weeks later. Those weren't isolated. Through 2025 and into 2026, AI usage became a performance metric at company after company: tracked in reviews, tied to headcount requests, in some cases made an explicit condition of employment. Set that next to the survey data. In 2025, only 33 percent of respondents said they trusted AI-tool accuracy while 46 percent actively distrusted it, even as many employers were pushing harder for adoption. I don't think most executives issuing those mandates were being cynical. They'd read the productivity studies from Chapter 9, which are real. What they had not read was METR, because METR hadn't been published yet, and what they could not have read was the thing nobody measures: the hours on the other side of the screen. Here's the asymmetry that makes this dangerous. Speed is easy to count — tickets closed, pull requests merged, lines shipped. Verification is invisible when it works. A dashboard can show you a 40 percent increase in output. There is no dashboard anywhere that shows you the eleven times an engineer caught something almost right before it reached production. That work generates no artifact. It looks, on every metric a company tracks, like nothing happening. So the incentive runs one direction only. Reward the visible. Squeeze the invisible. --- ## The study that names the problem In January 2026, Anthropic published research that I think will be remembered as the most important finding in this whole period — partly because of what it says, and partly because of who published it. Judy Hanwen Shen and Alex Tamkin ran a randomized controlled trial with 52 developers, most of them junior, learning an unfamiliar Python library. Half worked through the tutorial with an AI assistant that could produce correct code on request. Half coded by hand. Afterward, both groups took a comprehension quiz — without AI — on the concepts they had just used, minutes earlier. The hand-coding group averaged 67 percent. The AI group averaged 50 percent. Seventeen points. Nearly two letter grades. And the AI group didn't even gain meaningful time — the speed difference wasn't statistically significant. They finished about as fast, and understood substantially less. Now the detail that makes this the load-bearing study of the book. The researchers looked at *where* the gap was largest. It was in debugging. The questions about recognizing when code is wrong and working out why it failed. Read that with everything you now know. The skill most eroded by AI assistance is the exact skill required to supervise AI output. The tool is worst for developing precisely the capacity its own use makes necessary. There's a second finding, and it's the hopeful one — the same shape as the guardrailed tutor in Chapter 9. Not all AI use produced the same result. Participants who used the assistant to *understand* — asking follow-up questions, requesting explanations, posing conceptual questions while coding themselves — scored 65 percent or higher. Participants who used it to *delegate*, having it produce the code, scored below 40 percent. Same tool. Same task. Same duration. A gap of 25 points or more, determined entirely by whether the person was trying to learn or trying to finish. The researchers' own recommendation to managers is worth quoting, because it is a company recommending against the most profitable use of its own product: think intentionally about how AI tools get deployed at scale, and "consider systems or intentional design choices that ensure engineers continue to learn as they work." Anthropic published a study demonstrating that using its product in the fastest way damages the skill needed to check its product. I've been critical of this industry throughout this book and I'll be fair here: that took some spine, and it should be said out loud that they did it. I'll also note the constraint every honest reader should apply. It's 52 people, one library, one afternoon, and it measured comprehension immediately rather than tracking skill over years. It is a controlled measurement of something the field was already observing informally. It is not proof of a generational effect. But look at what it lines up with. METR: experts slower and unaware. Stanford: less secure code, more confidence. Bastani: worse exam scores from unguarded use, harm eliminated by design. Anthropic: less comprehension, worst in debugging, rescued by conceptual engagement. Four studies, four teams, four settings, one finding — **the tool trades away understanding for output, and the exchange rate depends almost entirely on how you use it.** --- ## The man who said it out loud In August 2025, Matt Garman — the chief executive of Amazon Web Services, which is to say one of the most powerful people in the computing industry — was asked on a podcast about replacing junior developers with AI. His answer: > "It's one of the dumbest things I've ever heard. They're probably the least expensive employees you have, they're the most leaned into your AI tools. How's that going to work when ten years in the future you have no one that has learned anything?" He reaffirmed it to reporters that December. He wasn't sentimental about the work itself — he said flatly that writing Java by hand is "probably not a job that's going to exist," and that the developer's role becomes "deconstructing a problem" and "coordinating a bunch of agents." That's the argument of this book, delivered by the head of the world's largest cloud provider, unprompted, about his own industry. *Ten years in the future you have no one that has learned anything.* Garman is describing a supply chain. Senior engineers are not manufactured; they are grown, from junior engineers, over roughly a decade of doing work that is individually not very valuable. The boring tickets. The small bugs. The code review where somebody tells you why your approach won't scale. That decade is not a cost of employing juniors — it is the entire mechanism by which the profession reproduces its expertise. AI is very good at the boring tickets. That's the part everyone noticed. The part almost nobody noticed is that the boring tickets were never really about the tickets. --- ## Riding on their skills None of this is new. That's the thing that got me when I found it. In 1983, a British psychologist named Lisanne Bainbridge published a five-page paper in the journal *Automatica* called "Ironies of Automation." It is about power plants and industrial control rooms. It has been cited thousands of times and it describes the situation in this chapter so precisely that reading it feels like a prank. Bainbridge's argument was that automating a system does not remove the human — it changes what the human is for, usually for the worse. The operator stops doing the task and starts monitoring the machine that does the task. And monitoring is a different skill, practiced less, that atrophies exactly when it isn't being used. Her most uncomfortable observation, on page 775: when the automation fails and a human has to take over, something has already gone wrong, so unusual action is required — meaning "the operator needs to be *more* rather than *less* skilled" than before automation existed. The moment you most need expertise is the moment automation has spent years eroding it. And then, on page 776, the sentence that stopped me: > "There is some concern that the present generation of automated systems, which are monitored by former manual operators, are riding on their skills, which later generations of operators cannot be expected to have." Nineteen eighty-three. She is describing 2026 exactly. The engineers reviewing AI-generated code right now are former manual operators. They learned to code before this existed. Their judgment — the instinct that says *this looks right but check line forty* — was built in a world where you had to write the line yourself. The AI coding boom is riding on their skills. Bainbridge's warning is about the generation after. The ones who learn with the tool from the first day, who score 50 percent instead of 67 on the comprehension quiz, whose largest deficit is in debugging. Except there's a wrinkle Bainbridge didn't anticipate, and it's worse than what she described. In her power plants, the next generation of operators still got hired. They still walked in the door and learned something, even if it was less. Her worry was about *quality* of skill. That's not the situation now. Look at Chapter 15 and you'll see we're not hiring them at all. --- ## What this means for the rest of you If you don't write software, you might be tempted to read this chapter as an industry story. It isn't. It's a preview. Software engineering got this technology first, in its most capable form, applied to its core task. Everything happening in that profession right now — the mandated adoption, the falling trust, the review burden replacing the creation burden, the invisible verification labor, the junior positions quietly not being filled — is arriving in law, medicine, accounting, teaching, journalism, design, and analysis. It's arriving on the same schedule, for the same reasons, and mostly nobody in those fields is watching what happened to the programmers. So take the four numbers with you. Eighty-four percent use it. Twenty-nine percent trust it. Sixty-six percent say the problem is that it's almost right. And the developers who felt 20 percent faster were 19 percent slower. That last one is the one to remember, because it's the one you can't feel. Now let's talk about what happens to a mind that stops doing the work. --- ## Sources and Further Reading METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity," July 10, 2025 (16 developers, 246 tasks; measured 19% slowdown; developers forecast 24% speedup and estimated 20% speedup afterward); METR, "We are Changing our Developer Productivity Experiment Design," February 24, 2026. Stack Overflow 2025 Developer Survey (84% using or planning to use AI tools; 29% trust in accuracy, down from 40%; 66% cite "almost right, but not quite"; 72% report vibe coding is not part of their professional work). Shen, J. H., & Tamkin, A., "How AI Impacts Skill Formation," arXiv:2601.20245 (2026); Anthropic Research, "How AI assistance impacts the formation of coding skills," January 2026 (n=52; 50% vs 67% on comprehension quiz; largest gap on debugging questions; conceptual-inquiry users ≥65%, delegation users <40%; productivity difference not statistically significant); InfoQ, February 2026. Matt Garman, remarks on the Matthew Berman podcast, reported by The Register, August 21, 2025; reaffirmed December 16, 2025 (WIRED/Fortune). Lisanne Bainbridge, "Ironies of Automation," *Automatica* 19(6), 1983, pp. 775–779. Tobi Lütke, Shopify internal memo, April 7, 2025; Luis von Ahn, Duolingo company email, April 28, 2025. --- \pagebreak # PART IV — NOBODY'S CHECKING --- # Chapter 13 # Cognitive Debt There's a question you can ask someone that will tell you, in about four seconds, whether they wrote what they just handed you. *Quote me a line from it.* Not the argument. Not the gist. One sentence, from memory, from the thing they finished minutes ago. At MIT's Media Lab, researchers ran a version of that test on 54 people. They'd split them into three groups to write essays — one group using ChatGPT, one using a search engine, one using nothing but their own heads. Everyone wore an EEG cap measuring electrical activity across the scalp while they worked. Then, after each session, they asked the participants to quote their own essays. In the first session, among the group that had used ChatGPT, the researchers reported that a large majority could not produce a correct quotation from an essay they had submitted minutes earlier. The brain-only group had no such difficulty. The EEG data showed the pattern you'd expect underneath: the strongest, most distributed connectivity in the brain-only group, the weakest in the AI group. The researchers called what they were measuring "cognitive debt." That phrase is the title of this chapter, and it's the right metaphor, so let me be careful with it. Debt isn't loss. Debt is a thing you take on deliberately, that buys you something real now, and that has to be paid later with interest. Nobody sensible tells you never to borrow. What they tell you is to know what you borrowed, and to have a plan for the payment. The problem with cognitive debt is that no statement arrives. The essay is done. It's good. Nothing in the experience tells you a balance is accruing. --- ## The caveats, up front I'm going to give you the objections to that study before I go any further, because it is the single most-cited and most-abused piece of research in this entire conversation, and I would rather hand you the weaknesses than have a critic hand them to you. Fifty-four participants is small. It circulated as a preprint — released to the public before formal peer review. EEG measures electrical activity, which is a proxy for cognitive engagement, not a direct read of thinking. The essay task was artificial. And within days of its release, the paper was being cited across the internet as proof that "ChatGPT makes you dumber," a claim the authors explicitly did not make and warned against. Published methodological criticism followed. So: it is one suggestive study, not a settled finding, and anyone who tells you otherwise is selling something. Here's why it's still in this book. It doesn't stand alone. --- ## The pattern across the research Put the studies side by side and the individual weaknesses start mattering less than the direction they all point. **Microsoft and Carnegie Mellon**, published at the CHI conference in 2025, surveyed knowledge workers about how they actually use generative AI at work. The finding: higher confidence in the AI was associated with *less* critical thinking about its output. Higher confidence in one's own expertise was associated with more. The researchers described the shift in the nature of the work — from producing material to overseeing material, and from solving the problem to verifying that the machine solved it. Which is Chapter 12, arrived at from a different direction, in a different profession. **Hamsa Bastani and colleagues**, in *PNAS* in 2025 — the Turkish math classroom from Chapter 9. Students with unrestricted GPT-4 access performed roughly 17 percent worse on exams than students with no AI at all. Students with the guardrailed tutor did not show that harm. **Anthropic's own trial**, from the last chapter. Fifty-two developers, 50 percent versus 67 percent on comprehension, worst gap in debugging, and the entire effect swinging on whether the person used the tool to understand or to finish. Four studies. Four teams with no coordination and, in one case, an active commercial interest in the opposite result. Different countries, different tasks, different measures — essays, exams, quizzes, self-reported reasoning. Every one finds the same thing: **when the machine does the cognitive work, the person retains less of it, and the effect is moderated almost entirely by how the tool is used rather than whether it is used.** That's not a proven law of nature. It's a convergence, and convergence from independent directions is how evidence usually looks before it becomes a fact. --- ## Not a new problem None of this would have surprised a psychologist in 2011. That year, Betsy Sparrow and colleagues published a study in *Science* on what became known as the Google effect. When people expected to have access to information later, they remembered the information itself less well — and remembered *where to find it* better. Their memory hadn't degraded. It had reallocated, from content to location. That's the honest frame for cognitive offloading, and it's why the alarmed version of this argument is usually wrong. Humans have always outsourced cognition. Writing did it. Printing did it. Calculators did it. Socrates complained that writing would destroy memory, and he was correct — literate people do remember less verbatim — and almost nobody thinks that was a bad trade. So the question is never "is offloading happening." Offloading is what tools are for. The question is: **what exactly did we hand over this time, and can we still do it when we need to?** With a calculator, the answer is comfortable. You handed over arithmetic. You kept the judgment about which number matters, whether the result is plausible, and what to do about it. If the calculator says the bridge needs a beam four inches thick, an engineer knows that's wrong without redoing the math. With writing, you handed over storage and kept comprehension. This time is different in one specific way, and it's the way that matters. In the highest-stakes uses, the thing being offloaded can be *the judgment itself*. Not merely the arithmetic — the assessment. Not "what's 17 times 43" but "is this argument sound," "is this code correct," "is this diagnosis right," "does this contract protect me." And the calibration check that saves you with a calculator doesn't exist here. You know when a calculator's answer is absurd. That's the whole point of Chapter 4: this machine's wrong answers are not absurd. They're plausible. They are optimized to be plausible. --- ## The thing that's actually different Let me put the argument of this chapter as precisely as I can, because it's easy to overstate and I don't want to. I am not claiming AI makes people stupid. The evidence doesn't support it and the people making that claim are going to be embarrassed. The call-center workers in Chapter 9 got better at their jobs. I built a company I could not have built. Millions of people are doing more than they could do before, and that is not an illusion. What the evidence supports is narrower and, I think, more serious: **AI use appears to trade comprehension for output, and the trade is invisible at the moment it's made.** Every part of that sentence matters. *Appears* — four studies pointing one way, not proof. *Trade* — you get something real. *Invisible* — this is the part that makes it dangerous, and it's the same property that runs through this entire book. The essay was good. The code ran. The exam felt easy. Nothing signals the debt. And unlike the calculator, you cannot easily test whether you still have the underlying skill, because the tool is always there. Nobody's asking you to do it by hand. The debt goes unmeasured until the day something goes wrong and you find out what you can and can't do without it. Which raises the question this book has been walking toward for twelve chapters. Everything so far has been about students, essays, homework, junior developers — people who are *supposed* to be learning, in situations where the stakes are a grade or a sprint. It's reasonable to read all of that and think: fine, but this is a story about novices. Experts are different. Experts have already built the judgment. Their skill is banked. That's the assumption. It is load-bearing for the entire optimistic case, and it's the one everybody makes — including me, right up until I found the study in the next chapter. Nineteen doctors. Two thousand procedures each. Three months. --- ## Sources and Further Reading Kosmyna et al., "Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task," MIT Media Lab, 2025 (n=54; EEG; released as a preprint; note the authors' own caution against the "AI makes you dumber" reading, and subsequent published methodological criticism). Lee et al., "The Impact of Generative AI on Critical Thinking," CHI 2025 (Microsoft Research and Carnegie Mellon; higher confidence in AI associated with less critical engagement; higher self-confidence associated with more). Bastani et al., "Generative AI Can Harm Learning," *PNAS*, 2025. Shen & Tamkin, "How AI Impacts Skill Formation," arXiv:2601.20245 (2026). Sparrow, Liu & Wegner, "Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips," *Science* 333(6043), 2011. --- \pagebreak # Chapter 14 # The Doctors Got Worse A colonoscopy is a search. A doctor guides a camera through about five feet of colon looking for adenomas — small polyps, some of which will become cancer if nobody finds them. They hide behind folds. They're flat sometimes, and pale, and the eye slides right over them. The measure of whether a doctor is any good at this is called the adenoma detection rate: out of every hundred procedures, in how many did this physician find at least one. It is one of the most consequential numbers in medicine, because it maps directly onto whether people die. Research has established that for every one percentage point of improvement in a doctor's detection rate, the risk that a patient later develops colorectal cancer falls measurably. This isn't a proxy. Finding the polyp is the entire point of the procedure. Artificial intelligence turned out to be genuinely good at this. Computer-aided detection systems watch the video feed in real time and put a box around anything that looks like a polyp. Multiple trials showed the systems improved detection rates. This was, by any reasonable reading, one of the clearest wins for AI in medicine — a tool that measurably helps doctors find cancer. Between September 2021 and March 2022, four endoscopy centers in Poland adopted these systems as part of a study. And a group of researchers had the presence of mind to ask a question nobody else was asking. Not *does the AI help while it's on*. Everyone was measuring that. The question was: what happens to the doctors? --- ## What they found The results were published in *The Lancet Gastroenterology & Hepatology* in August 2025. The researchers looked at nineteen experienced endoscopists — not trainees, not residents. Each had performed more than two thousand colonoscopies. These were the veterans, the ones whose skills were supposedly banked. The team compared procedures those doctors performed *without* AI assistance in the three months before the systems were introduced, against procedures they performed *without* AI assistance in the three months after. Before AI exposure, the doctors' unassisted adenoma detection rate was 28.4 percent. After three months of working with AI, their unassisted rate was 22.4 percent. Six percentage points — a relative decline of about a fifth in the measured unassisted detection rate across those study periods, among physicians with thousands of procedures behind them. That is a striking association. Because the study was observational, it is not the same thing as proving that AI exposure caused every point of the decline. Now the number that made me put the paper down and walk around the room. In the same period, the doctors' detection rate *with* the AI actively assisting was 25.3 percent. Line them up: **Alone, before AI: 28.4 percent.** **With AI, after: 25.3 percent.** **Alone, after AI: 22.4 percent.** The AI-assisted adenoma-detection rate measured in the post-implementation period was lower than the doctors' unassisted pre-implementation baseline. Because the study was observational and compared calendar periods, that cross-period comparison should not be read as proof that AI caused the difference. That comparison is the warning: in this observational dataset, performance with the assistant after adoption was below the doctors' earlier unassisted baseline, and unassisted performance was lower still. The pattern is consistent with deskilling, but the design cannot isolate AI exposure from every other change across the periods. --- ## Before you accept it This finding is extraordinary, and extraordinary findings get one job first: survive scrutiny. Here is everything wrong with it, stated as strongly as a critic would state it. It is one study. It is observational — the doctors weren't randomly assigned to conditions, so the researchers are comparing time periods, not arms of a trial. Anything else that changed between late 2021 and early 2022 across four Polish endoscopy centers is a potential confounder, and that period was not a quiet one in European hospitals. Critics have specifically raised workload: if the volume or pace of procedures shifted, detection rates could move without any deskilling at all. Adenoma detection rate is a well-validated measure but it is still a proxy, and three months is a short window. Any of those could explain some of the gap. None of them, individually or together, has been shown to explain it. And here's what the objections don't touch: the *direction*. To argue this away you need a mechanism that made experienced doctors worse at finding polyps during exactly the months they gained an assistant that finds polyps — and that mechanism has to be something other than the obvious one. The obvious one is that when a box appears around the thing you're looking for, you stop looking as hard, and looking as hard is a skill. A linked commentary published alongside the study made the point that matters for this book: this is among the first real-world clinical evidence of AI-associated deskilling in practicing physicians, with potential consequences for patients. Not students. Not a lab. Adenoma detection, in real clinical practice, on patients. Replication is needed. I'd want three more studies in three more countries before I called it settled. But I'd also point out that we are deploying these systems worldwide right now, and the burden of proof has been running in the wrong direction — everyone measured whether the AI helps while it's on, and almost nobody measured what it does to the person operating it. --- ## Where this has happened before If the Polish result makes you uneasy, it should also make you feel a strange kind of recognition, because a different industry has already lived through exactly this, published the findings, and written the fix. On June 1, 2009, Air France Flight 447 fell into the Atlantic Ocean between Rio de Janeiro and Paris. Two hundred and twenty-eight people died. The investigation found that ice crystals had blocked the aircraft's airspeed sensors. Faced with unreliable readings, the autopilot did exactly what it was designed to do: it disconnected and handed control to the pilots. Three trained crew members then had to hand-fly a modern airliner at altitude — an ordinary maneuver in a previous generation of aviation, and one they had rarely performed in years of flying automated aircraft. The aircraft entered an aerodynamic stall and remained in it, all the way down. I'm not going to compress a four-year investigation into a paragraph or assign blame to dead crew. What I'll take from it is the institutional response, because that response is the most useful thing in this book. The Federal Aviation Administration studied automation dependency and issued a Safety Alert for Operators in 2013 — SAFO 13002 — warning that continuous reliance on automated flight systems "could lead to degradation of the pilot's ability to quickly recover the aircraft from an undesired state," and encouraging operators to build manual flight operations back into line flying. A second alert followed in 2017. Read the FAA's sentence next to the Polish study. The mechanism is identical. The tool performs the task well. The human's ability to perform it without the tool decays. And the decay is invisible until the moment the tool isn't there — which is always the worst possible moment, because if the automation has failed, something is already wrong. That is Bainbridge's 1983 warning made concrete in two very different settings: a cockpit over the Atlantic and an endoscopy suite in Poland. Aviation documented automation-dependency risks and responded institutionally. The Polish study now provides real-world clinical evidence consistent with the same deskilling concern, although its observational design does not establish causation by itself. Software has not confirmed anything, because nobody is measuring. --- ## Why this is the chapter that matters Everything before this could be dismissed with one sentence: *those are novices*. Students writing essays. Undergraduates learning a Python library. High schoolers doing math homework. Somebody could read Chapters 9 through 13, nod, and conclude that the problem is people who never had the skill in the first place. Experts are fine. Expertise, once built, is durable. That's the entire foundation of the reassuring story — the story that says AI is a leveler that lifts the bottom without touching the top. Nineteen endoscopists with two thousand procedures each are not novices. They are the top. They spent careers building a specific perceptual skill, and it eroded by roughly a fifth in three months. **Expertise is not a bank balance. It's a muscle.** That reframe is the hinge of this book. If skill were stored, the succession problem in Chapter 12 would be a slow generational worry — an issue for 2040 that we'd have twenty years to fix. If skill is maintained, then the erosion is happening right now, simultaneously, at both ends: the veterans are losing the edge they built, and the juniors are not building one, and both processes are running at the same time, in the same institutions, driven by the same tool. The people currently reviewing AI-generated code, AI-generated diagnoses, AI-generated legal briefs, and AI-generated financial analysis are, in Bainbridge's phrase, former manual operators. The system is riding on their skills. Poland is the measurement that says those skills are perishable. --- ## What it doesn't mean I want to end this chapter carefully, because it's the one most likely to be quoted out of context, and I don't want it used to argue against tools that save lives. The AI polyp detectors work. The trials showing they improve detection are real, and if you're getting a colonoscopy tomorrow you should want one in the room. Nothing in the Polish study says the technology should be withdrawn. What it says is that we deployed it having measured only half of its effect — the half that shows up while it's running — and the other half was accumulating in the doctors the entire time, unmeasured, because nobody thought to check. That's not an argument for less AI in medicine. It's an argument for the thing aviation already does: deliberate, scheduled, mandatory practice without the automation, precisely so the skill is there when the automation isn't. Pilots do it in simulators. Nobody proposed it for endoscopists, because nobody knew there was anything to preserve. Chapter 22 is about what that would look like. But there's one more group to account for first — and it's the group that was supposed to replace these doctors, these pilots, these engineers, in twenty years. Let's see how they're doing. --- ## Sources and Further Reading Budzyń et al., "Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study," *The Lancet Gastroenterology & Hepatology*, August 2025 (four Polish centres; procedures September 2021–March 2022; 19 endoscopists each with >2,000 prior colonoscopies; unassisted adenoma detection rate 28.4% before AI exposure vs 22.4% after; AI-assisted rate 25.3%); linked commentary in the same issue; subsequent methodological criticism regarding workload and observational design. Bureau d'Enquêtes et d'Analyses, final report on Air France Flight 447 (Rio de Janeiro–Paris, June 1, 2009; 228 fatalities), 2012. Federal Aviation Administration, Safety Alert for Operators 13002 (2013) and 17007 (2017). Lisanne Bainbridge, "Ironies of Automation," *Automatica* 19(6), 1983. --- \pagebreak # Chapter 15 # The Canaries ## The canary got louder The first version of the labor-market evidence was easy to dismiss because it was early. That is what responsible researchers said too. A new technology arrives. Hiring changes. Interest rates change. Companies overhire and correct. Graduates enter a bad market. Occupations are categorized imperfectly. Everybody wants a clean story before the data has had time to become one. So the Stanford Digital Economy Lab kept updating the data. By August 2026, the result had become harder to wave away and more important to state carefully. Using ADP administrative payroll data covering millions of U.S. workers through June 2026, Erik Brynjolfsson, Bharat Chandar, and Ruyu Chen reported **no evidence of widespread economy-wide job displacement from AI**. That sentence belongs first. Then comes the canary. Among workers ages twenty-two to twenty-five in highly AI-exposed occupations, employment stood about **19 percent below where it would have been if it had kept pace with similarly aged workers in less-exposed occupations**. Experienced workers did not show a comparable gap. The adjustment appeared primarily through **reduced hiring**, not a surge in young workers being fired. That distinction matters to this book. The apprenticeship pipeline does not have to collapse through dramatic layoffs. It can thin quietly because the door opens less often. A company does not need to fire its junior developers if it simply hires fewer of them. A law firm does not need to announce that AI eliminated its training pipeline if each class gets a little smaller. A customer-service operation does not need a robot-layoff press release if attrition occurs and the entry-level seats are never refilled. The canary can disappear by vacancy. --- ## A second dataset points the same direction One study should never carry an argument this large by itself. In April 2026, U.S. Census Bureau researcher Lee C. Tucker published a working paper using Quarterly Workforce Indicators derived from matched employer-employee administrative data. The question was similar: what happened to early-career hiring after ChatGPT arrived in the industries and states most exposed to AI? The paper documented what it described as an immediate, sizable, persistent decrease in hires of workers ages twenty-two to twenty-four in the most AI-exposed industry-state cells. Regression-adjusted employment for early-career workers in the most exposed quintile was **12 percent lower over the ten quarters following ChatGPT's introduction**, while employment in less-exposed industries remained stable. The hiring rate later largely recovered by early 2025, but from a smaller employment base. That is not proof that ChatGPT caused every missing job. It is corroboration that the early-career pattern is visible in more than one administrative dataset. The strongest version of the claim is therefore not: **AI has destroyed entry-level employment.** The data does not support that. It is: **The labor-market evidence is increasingly consistent with an AI-era hiring problem concentrated among young workers in occupations where AI can automate tasks, even while aggregate employment does not show broad AI-driven collapse.** That is narrower. It is also more interesting. --- ## Automation and augmentation split apart The Stanford revision adds another detail that matters. The declines were concentrated in occupations where observed AI use tended to **automate** human tasks. Where AI tended to **augment** workers instead, employment was flat or rising, especially among experienced workers. That is almost a laboratory version of the distinction running through this book. AI is not one labor-market force. A system that helps an experienced professional do more work can increase the value of the professional. A system that absorbs the task traditionally assigned to the beginner can reduce demand for the beginner. Both can be called "AI adoption." Their consequences for the expertise pipeline are opposite. This is why headline debates about whether AI "creates jobs" or "destroys jobs" are too crude. The important unit is the task, the worker's experience level, and what the technology is doing to the relationship between them. --- ## The interest-rate objection There is an obvious objection. Young workers got hit by a changing macroeconomy. Interest rates rose. Technology companies corrected after pandemic-era hiring. Maybe the entire pattern has nothing to do with AI. The Stanford researchers examined that possibility directly. Their February 2026 follow-up concluded that interest rates clearly affect overall employment but did not appear to explain the disproportionate decline in entry-level employment in AI-exposed occupations. They also cautioned that some earlier pre-2024 movement was probably caused by other factors. Under their broadest controls, the distinctive decline becomes statistically apparent in 2024 rather than immediately after ChatGPT's release. That is exactly the kind of qualification this subject needs. The world did not divide into "before ChatGPT" and "after ChatGPT" with every other variable frozen. The evidence is suggestive, strengthening, and still observational. But the timing, concentration by exposure, age split, automation-versus-augmentation split, and reduced-hiring mechanism now point in the same direction. The canary is not a prophecy. It is a measurement. In August 2025, CNN ran a story about people who had done everything right. One of them was a young man named Rubio, who had loved computers since he was a kid, studied coding at Bloomfield College of Montclair State University in New Jersey, and graduated that May with a degree in computer science and game programming. He had applied for twenty software development jobs. He had received no offers. "I go on LinkedIn almost every day, just scrolling, trying to see what opportunities are out there," he told the reporter. He hadn't heard back from most companies. That's not a remarkable story. That's the point. There are tens of thousands of versions of it, and by 2026 they had accumulated into something you could see in the national statistics. The Federal Reserve Bank of New York tracks unemployment by college major. In its recent data, recent computer science graduates carried an unemployment rate of about 6.1 percent — computer engineering about 7.5 percent — against roughly 5.7 percent for recent graduates overall. Computer science majors are unemployed at a higher rate than the average college graduate. In 2026. In the middle of the largest technology investment boom in the history of capitalism, with three quarters of a trillion dollars a year going into data centers. Something is wrong with that picture, and this chapter is about what it is — and, just as importantly, what it isn't. --- ## The canaries The most careful measurement comes from Erik Brynjolfsson — the same Stanford economist behind the call-center study in Chapter 9 — working with Bharat Chandar and Ruyu Chen. They used payroll records from ADP, which processes paychecks for a very large slice of American employment. Not surveys. Not job postings. Actual payroll. They compared employment trends for workers of different ages within the same occupations, separating jobs heavily exposed to AI from jobs that aren't. The finding, in the paper they titled "Canaries in the Coal Mine": since late 2022, employment for 22- to 25-year-olds in the most AI-exposed occupations has fallen roughly 20 percent relative to trend. For older workers in those same occupations — the 35-to-49 group, the mid-career people — employment grew. Same occupation. Same industry. Same period. The young are down; the experienced are up. That divergence is the finding, and it's why the paper is careful with its own title. Canaries are an early-warning signal, not a diagnosis. It measures a pattern in the data; it does not establish that AI caused it. The private-sector data agrees on the shape. SignalFire, which analyzes hiring across hundreds of millions of professional profiles, reported that new-graduate hiring at major technology companies had fallen more than 50 percent from 2019 levels, with new graduates making up about 7 percent of hires. At startups, the new-grad share fell from around 30 percent in 2019 to under 6 percent. And SignalFire's own reading includes a nuance the alarming coverage usually drops: in their 2025 data, engineering was among the *least* affected functions overall. The collapse is concentrated specifically at the entry level, not across software engineering as a whole. Experienced engineers are still being hired. It's the door that's closing, not the building. --- ## The honest counter-case I promised in Chapter 7 that I wouldn't force the evidence to prove mass unemployment, and I'm not going to start here. So before I make the argument of this chapter, here is the strongest case against it. **The aggregate data shows nothing.** Yale's Budget Lab, October 2025: "the broader labor market has not experienced a discernible disruption since ChatGPT's release 33 months ago." That result held through subsequent updates into 2026. Whatever is happening to young graduates is not yet visible in the shape of the economy as a whole. **There's an obvious alternative explanation, and it isn't AI.** Interest rates. The Federal Reserve raised rates sharply starting in 2022, and cheap money is what funded a decade of speculative hiring at technology companies. When money got expensive, hiring froze — and entry-level hiring freezes first in every downturn ever recorded, because a new graduate is a bet on the future and a senior engineer is a solution to today. The timing of the AI boom and the timing of the rate shock overlap almost exactly, and any honest analyst has to admit that untangling them is difficult. **This has happened before, in this exact major.** Stanford's Eric Roberts documented the panic after the dot-com crash, when students fled computer science on the theory that the jobs were gone forever. He found "no evidence to justify those fears, and ample data to refute them," and warned that "mythology kept students out of computer science until disaster struck in a different sector of the economy." By 2004 the industry was hiring at pre-crash levels. A 2026 essay in the *Stanford Review* argued precisely this: the class of 2026's problem is transient and monetary, and AI is a convenient scapegoat. **And the forward-looking numbers are good.** The National Association of Colleges and Employers projects starting salaries for computer science graduates in the class of 2026 at about $81,500, up nearly 7 percent year over year, with CS among the most in-demand majors. The Bureau of Labor Statistics projects software developer employment growing 15 percent from 2024 to 2034 — roughly five times the average across occupations. Those are not the numbers of a dying profession. **The CEOs walked it back.** Chapter 7: Altman in May 2026 said he'd expected more entry-level displacement than had happened and was "delighted to be wrong." The share of CEOs telling EY-Parthenon they expected significant AI-driven headcount cuts fell from 46 percent to 20 percent in sixteen months. Take all of that seriously. It is entirely possible that in 2029 the entry-level market recovers, this chapter reads as a panic, and the right conclusion was: it was the interest rates. I'd be pleased. I'd also note that it wouldn't touch the argument I'm about to make. --- ## The argument that doesn't depend on the cause Here is what I think is actually true, and I've tried to build it so it survives whichever way the jobs debate resolves. It does not matter, for the purposes of this book, *why* entry-level hiring collapsed. What matters is that it collapsed, that the collapse is measured, and that we now know something about apprenticeship that we did not know when it started. Chapter 12: senior engineers are grown, not hired. They come from junior engineers doing years of individually unimportant work — the boring tickets, the small bugs, the code review where someone explains why your approach won't scale. That decade is the mechanism by which a profession reproduces its expertise. Chapter 12 again, from Anthropic's own trial: developers learning with AI assistance scored 50 percent on comprehension against 67 percent for those who coded by hand, and the largest deficit was in *debugging* — recognizing when code is wrong and working out why. Chapter 14: expertise behaves more like a maintained capability than a bank balance. In one observational study, nineteen veteran endoscopists' unassisted adenoma-detection rate fell from 28.4 percent to 22.4 percent after three months of AI exposure — roughly a one-fifth relative decline in that measured rate. The study raises a deskilling concern; it does not, by itself, prove AI caused the entire decline. Now put those three together with the hiring data, and you get an arithmetic problem that has nothing to do with whether AI or the Federal Reserve caused it: **Fewer juniors are entering the pipeline. The ones who enter are learning less of the specific skill required to catch machine errors. And the veterans currently doing the catching are losing their edge through the same tool, at the same time.** Three curves, all bending the same direction, all through the same decade. Many of the people qualified to tell "almost right" from right in 2040 will have to come from the cohorts entering these fields now. In high-skill professions, judgment is built over years of supervised practice; in many careers, reaching genuinely senior judgment takes something close to a decade. The exact timetable varies by profession. The pipeline logic does not. Matt Garman said it in one sentence in Chapter 12: *ten years in the future you have no one that has learned anything.* Bainbridge said it in 1983: the current systems "are riding on their skills, which later generations of operators cannot be expected to have." Neither of them needed to know what caused the hiring freeze. The succession problem is indifferent to the reason. --- ## What breaks first Let me be concrete about what "nobody can verify" means, because in the abstract it sounds like a philosophy problem and it isn't. It means a hospital where the AI flags a scan and the radiologist who would have caught the miss trained on AI-flagged scans and never developed the eye. It means a law firm where an associate files a brief and the partner who would have spotted the fabricated citation has been skimming AI drafts for eleven years. It means a bank where the model prices a risk and everyone in the room learned the business from the model. It means a codebase running a utility, a hospital, or a payroll system, and a team that can operate it and cannot repair it. None of those are dramatic. There's no robot uprising, no mass unemployment event, nothing that makes a headline on the day it happens. It's a slow, quiet, distributed loss of the ability to check — showing up as an increase in errors that nobody catches, in systems everybody trusts, staffed by people who are doing their jobs exactly as trained. The failure mode of this technology was never that it becomes hostile. It's that it becomes unquestioned, at the same moment the questioners stop being produced. --- ## The thresholds I told you at the start of this book that I'd tell you what would change my mind, so here it is, in public, before the data arrives. **If the Stanford/ADP divergence closes** — if 22-to-25-year-old employment in AI-exposed occupations recovers toward trend as interest rates normalize — then the hiring collapse was monetary, the *Stanford Review* was right, and this chapter should be read as a near-miss rather than a diagnosis. The succession argument would still stand, but as a risk that policy and a business cycle corrected, not as a crisis. **If the aggregate data turns** — if Yale's Budget Lab finds material displacement in AI-exposed occupations rather than none — then Part IV is understated, not overstated, and the argument hardens from signal to confirmed displacement. **If the deskilling findings fail to replicate** — if further studies find the Polish endoscopy result was workload or confounding, and if Anthropic's comprehension gap doesn't hold up in longer-term testing — then the muscle-not-bank-balance claim weakens considerably, and with it the urgency of Chapter 14. **And if apprenticeship gets rebuilt deliberately** — if firms start protecting junior roles as a capability investment rather than a cost — then the whole problem becomes tractable, and this book becomes a description of something we saw coming and fixed. That last one is the one I'm arguing for. It's not a prediction. It's a request. --- ## The thing the numbers don't show I want to close Part IV with a number that isn't in any study, because I paid for it myself. Fifteen and a half hours in one day. Over a hundred in one week. A salesman on a phone, functioning as the verification layer for a machine that could out-produce him a thousand to one and could not tell when it was wrong. That labor appears in no productivity statistic anywhere. It doesn't show up in the 15 percent gain in the call center or the 40 percent faster writing. It generated no artifact. On every metric my business tracks, those hours look like nothing happening. They were the only reason anything worked. Multiply that by every profession that's about to receive this technology, and then subtract the people who were supposed to learn how to do it. That's the verification gap. Output went up enormously. Checking stayed exactly as fast as a human being. And we stopped hiring the humans who would have done it. Now: what do we do about it? Aviation already knows — that's Chapter 22. But before we get to the fix, we need to walk through where all this checking work is headed, because that's where the jobs, the liability, and the money are about to move. That's Part V. --- ## Sources and Further Reading Brynjolfsson, Chandar & Chen, "Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence," Stanford Digital Economy Lab (ADP payroll microdata; employment for 22–25-year-olds in the most AI-exposed occupations down ~20% relative to trend since late 2022, while employment for older workers in the same occupations grew). SignalFire, *State of Tech Talent* reports, 2025 and 2026 (new-grad hiring at major technology companies down more than 50% from 2019; new grads ~7% of hires; startup new-grad share down from ~30% in 2019 to under 6%; engineering among the least-affected functions overall in 2025). Federal Reserve Bank of New York, *The Labor Market for Recent College Graduates* (recent CS graduate unemployment ~6.1%; computer engineering ~7.5%; all recent graduates ~5.7%). CNN Business, "150 job applications, rescinded offers: Computer science grads are struggling to find work," August 28, 2025. Gimbel, Kinder, Kendall & Lee, "Evaluating the Impact of AI on the Labor Market: Current State of Affairs," The Budget Lab at Yale, October 1, 2025. *Stanford Review*, "The Class of 2026 is struggling to find jobs—and it's not because of AI," 2026, including Eric Roberts on the post-dot-com enrollment collapse. National Association of Colleges and Employers, 2026 Winter Salary Survey (CS class of 2026 starting salary projection $81,535, up ~7%). U.S. Bureau of Labor Statistics, Occupational Outlook Handbook (software developers, projected 15% growth 2024–2034). Shen & Tamkin, "How AI Impacts Skill Formation," arXiv:2601.20245 (2026). Matt Garman, The Register, August 21, 2025. Lisanne Bainbridge, "Ironies of Automation," *Automatica* 19(6), 1983. Author's own voice memorandum, August 2026. --- \pagebreak # PART V — THE VERIFICATION ECONOMY --- # Chapter 16 # When Nobody Knows What Right Looks Like There's a version of the AI problem that's easy to understand. A machine hands a lawyer a fake case. The lawyer knows enough law to check the citation, doesn't bother, files it, and gets caught. That's a verification failure, and it's embarrassing, but at least everybody involved knew what checking would have looked like. There's another version that's harder, and it's the one this part of the book is about. A machine hands a legal answer to somebody who is not a lawyer. The answer is clear. It cites statutes. It anticipates objections. It sounds exactly like the thing a lawyer would say. The person reads it carefully, twice. Now what? "Check the AI" is excellent advice when the person doing the checking already knows what a correct answer looks like. It turns circular the moment you remember why the person used the machine in the first place: they didn't have the expertise to produce the answer, which means they don't have the expertise to grade it either. I know this problem personally, because it's the problem that got me into this subject. I can ask an AI system to write software I could not write from scratch. That's the miracle. It's also the trap. If I could independently inspect every line at the level of the engineer the system is replacing, I wouldn't need the system for the reason I'm using it. For most of professional history, the chain was easy to see: an expert produced the work, and another expert — sometimes the same one, on a second pass — checked it. AI inserted itself into the first half of that chain. Then consumer AI did something more radical. It let people cross into fields where they had never been experts at all. So the chain now often looks like this: the machine produces, a nonexpert receives, and the need for a qualified check hasn't gone anywhere. We made the first half of the chain nearly free without ever deciding who owns the second half. That missing person is the problem. --- ## The competence inversion For most of the history of tools, the operator knew more about the task than the tool did. The carpenter understood the table saw. The pilot understood the airplane. The lawyer understood the word processor. A tool could amplify force or speed or memory, but it didn't hand you a finished intellectual product in a field you'd never studied. Generative AI flipped that. The system can produce a tax explanation for somebody who never took accounting, a database migration for somebody who never administered a database, a contract for somebody who never saw the inside of a law school, an interpretation of a lab result for somebody who couldn't spell biochemistry. That's an extraordinary transfer of capability, and I'm a direct beneficiary of it. But capability and accountability don't transfer together. The user is now capable of *obtaining* an answer without being capable of *certifying* it. That's what I mean by the competence inversion: the tool can temporarily display more domain fluency than the person who is legally, financially, and practically on the hook for deciding whether to trust it. The output may be useful. It may be excellent. It may be better than anything the user could have bought or built otherwise. The final judgment still lands on the weaker side of the relationship. I don't think we've absorbed how new that arrangement is. --- ## Functional is not secure Software gives us an unusually clean demonstration, because code can pass one kind of test while flunking another, and there are now hard numbers on how often it does. In 2025, Veracode — an application-security company, so keep in mind this is a vendor's benchmark — tested a large set of AI models across code-generation tasks in Java, Python, C#, and JavaScript. In that benchmark, 45 percent of the tested generation tasks failed the security criterion by introducing a known class of vulnerability. That's a defined set of tasks under defined conditions, not a universal failure rate for all AI-generated software. But hold onto the number, because by spring 2026 Veracode's update showed something stranger: syntax correctness had climbed above 95 percent, while security performance stayed roughly flat, with only about 55 percent of the tested generation tasks producing secure code. Read those two numbers together. The machine got extremely good at producing code that looks like code, parses like code, and runs like code — without improving at anywhere near the same rate on a property the user may not be qualified to inspect. A 2025 research benchmark called SUSVIBES pushed the question closer to real work. The researchers took two hundred feature requests from real open-source projects — tasks associated with vulnerable human implementations — and handed them to coding agents. In one reported configuration, using SWE-Agent with Claude 4 Sonnet, 61 percent of the solutions were functionally correct and only 10.5 percent were secure. Don't stretch that into "90 percent of AI code is insecure" — it's one benchmark, built around security-sensitive tasks, using particular agents and models. But as an illustration, it's about as clean as it gets. Functional correctness and security correctness are different axes. The person who asked the agent to "make login work" can verify the first axis by logging in. The second axis may require knowledge they never had. And here's the ugly part: the program's success becomes part of the danger. A working button reassures exactly the person who can't see what the working button is hiding. The security world already wrote this down as a rule. OWASP's 2025 Top 10 for LLM applications includes a category called Improper Output Handling — an application that fails to adequately validate or sanitize a model's output before passing it downstream, with consequences that can run from cross-site scripting to remote code execution. That sounds like a software category. It's really a philosophical statement about this whole technology: do not confuse generated output with trusted input. The model's answer is material to be processed. It is not authority, and it is not a permission slip. A generated citation is an untrusted claim until checked. A generated medical summary is decision support, not a diagnosis, no matter how grammatical it is. A generated contract clause is draft language, not a correct allocation of legal risk. The downstream system that gets hurt can be a computer. It can also be you. --- ## The frontier is jagged There's a mental model behind most AI overconfidence, and it's the idea that the machine has a level — that it's "as good as a junior lawyer" or "can code like a mid-level engineer," and everything below that line is safe to hand over. Real capability doesn't work like that. It's jagged. A model can be astonishing at one task and unreliable at another task that looks almost identical to a human being, and there's now a landmark experiment showing exactly how sharp those teeth are. Researchers working with Boston Consulting Group ran a preregistered experiment on 758 consultants — real professionals, doing realistic knowledge work, randomly assigned to work without AI, with GPT-4, or with GPT-4 plus some prompt guidance. For tasks the researchers had established were inside the model's capability frontier, the AI was enormously useful. Consultants using it completed 12.2 percent more tasks, finished them 25.1 percent faster, and produced significantly higher-quality work. If the study ended there, the productivity case would be simple. It didn't end there. The researchers also gave consultants a complex managerial task deliberately designed to sit outside the model's frontier. On that task, consultants using AI were 19 percentage points less likely to produce a correct solution than the consultants working without it. Same professionals. Same technology. Same study. Acceleration on one side of the line, degradation on the other. That's the jagged frontier. Now here's what makes it a verification problem and not just a curiosity. If the model failed loudly outside its frontier — if the answer arrived with a warning that said *I have crossed the boundary of my competence, stop* — this would be manageable. Instead, the interface looks identical on both sides. Same font. Same confident paragraphs. Same polished structure of reasoning. The system does not visibly sweat. So the user has to figure out which side of the frontier the task is on, using a tool whose presentation is equally convincing on both sides. And the map expires. A task outside the frontier in 2024 may be inside it in 2026. A task one model botches may be routine for another. An update can improve one capability and quietly move another. There is no laminated card with safe tasks on the left and unsafe tasks on the right, and there never will be. To be clear about the other side of the ledger: AI genuinely works, and that's precisely why any of this matters. In a study of 5,179 customer-support agents, researchers Erik Brynjolfsson, Danielle Li, and Lindsey Raymond found a generative AI assistant raised productivity about 14 percent on average — and about 34 percent for the newest, lowest-skilled workers, with much smaller effects for the experienced ones. They found suggestive evidence that the tool was actually spreading the habits of the strongest workers and moving new employees down the experience curve faster. I'm not going to force that into a failure story. It's a genuine success, and this book gets weaker if I pretend otherwise. The point of the jagged frontier is not that the machine doesn't work. It's that "does it work?" has no single answer, and the tool won't tell you which answer you're getting today. --- ## The frontier inside you There's a second jagged edge that matters just as much, and it's yours. You know some things well enough to verify. Some things well enough to notice when something's off. Some things well enough to ask a good question. And some things not at all. AI expands your apparent capability across all four zones at once — but the risk is completely different in each. When I use AI on something I understand, I can challenge it. One step past what I understand, I can usually still test it. Five steps past what I understand, the answer becomes indistinguishable from expertise, and that's exactly the moment to get more careful, not less — because that's the moment it feels most like magic. I've started calling what accumulates out there *verification debt*. The faster you expand what you can produce, the larger the territory where your ability to check lags behind. The debt is payable — you can learn, you can test, you can hire expertise, you can constrain the system. The mistake is pretending the debt doesn't exist because the output arrived successfully. I ran up plenty of it before I knew it had a name, and Chapter 11 was the bill. --- ## What "checked" actually means We talk about checking like it's a light switch — checked or unchecked. It's more honest to say verification has depth, and most of us stop at the shallow end. The surface level asks whether the thing has obvious errors: the links open, the program launches, the arithmetic adds. One level down is sources: do the cited things exist, and do they say what the output claims? Below that is domain judgment: would somebody who actually knows this field find the conclusion reasonable, or spot the exception that's missing? Below that is the adversarial question: how does this fail when the input is malicious, weird, or incomplete? And at the bottom is the systemic question: even if this output is right, what happens when ten thousand people run the same process and inherit the same blind spot? AI makes the surface level easier. It sometimes makes the source level easier. It does not touch the deeper levels — if anything it makes them more urgent, because it multiplies the volume of material arriving at them. And the least experienced user is the one most likely to stop at the first level that returns a green light. The website loaded. The citation exists. The spreadsheet balanced. Ship it. Software trained us to love that green check mark, and the green check mark is exactly as trustworthy as the test behind it. A unit test proves a function handled the cases somebody imagined. A citation checker proves a paper exists — not that the paper supports the sentence attached to it. A world full of green check marks can still be almost right. --- ## The question before the prompt Everybody's learning prompt engineering. I think there's a question that belongs before the prompt, and it's the single most useful habit I've built since the night my own system passed its own check while the search button sat there broken. Before you ask the machine anything that matters, ask yourself: **if this answer is wrong, how would I know?** If you'd notice because you know the field — proceed with normal skepticism. If you can test it cheaply — build the test first. If there's an authoritative source you can compare it to — find that source before you generate, not after. And if the honest answer is *I wouldn't know* — then the task has crossed an expertise boundary, and the workflow has to change. Not stop. Change. Now you need a qualified verifier, a constrained system, a second independent method, or a smaller question whose answer you actually can test. I wish somebody had handed me that question when I first discovered I could build things I didn't know how to build. I was so impressed by the reach that I never saw the debt riding along with it. The machine expanded my reach far faster than it expanded my judgment, and stripped of every statistic in this book, that mismatch is the argument. --- ## Sources and Further Reading - Veracode, *2025 GenAI Code Security Report* and Spring 2026 update (vendor benchmarks; 45% of tested generation tasks failed the security criterion in 2025; syntax >95% vs. ~55% secure in the 2026 testing). - SUSVIBES benchmark (2025): 200 feature-request tasks from real open-source projects; in the reported SWE-Agent / Claude 4 Sonnet configuration, 61% functionally correct vs. 10.5% secure. - OWASP, *Top 10 for Large Language Model Applications* (2025) — Improper Output Handling. - Fabrizio Dell'Acqua et al., "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality," *Organization Science*, published online March 2026 (758 consultants; +12.2% tasks, 25.1% faster, higher quality inside the frontier; 19 percentage points less likely correct outside it). - Erik Brynjolfsson, Danielle Li & Lindsey R. Raymond, "Generative AI at Work," NBER Working Paper 31161 (5,179 customer-support agents; ~14% average productivity gain; ~34% for novice and lower-skilled workers). - Shen & Tamkin, *How AI Impacts Skill Formation* (2026); Anthropic Research. - Lisanne Bainbridge, "Ironies of Automation," *Automatica* 19(6), 1983. --- --- \pagebreak # Chapter 17 # The Apprenticeship Problem The junior employee has always been inefficient. That was the point. Nobody hires a first-year associate because she's the fastest lawyer in the building. Nobody puts a resident in a hospital because he knows more than the attending. Nobody assigns the new developer the boring ticket because he'll solve it faster than the person who's maintained the system for eight years. We tolerated junior inefficiency because every institution has two jobs at the same time: it has to produce today's work, and it has to produce tomorrow's experts. Those two jobs have always been in tension. What AI did is make the tension visible, because for the first time a company can plausibly delete most of the work that made the beginner economically tolerable. If the machine can draft the memo, summarize the discovery, write the routine code, build the first spreadsheet, and answer the basic customer question, the obvious management question is: why am I paying a beginner to do it? And the honest answer doesn't appear anywhere on this quarter's income statement. You're paying the beginner because somebody has to become the person who knows when the machine is wrong five years from now. --- ## Two kinds of evidence, one squeeze The apprenticeship argument gets its full strength when you put two different kinds of evidence side by side, because they're not the same finding — and that's exactly why the combination matters. The labor-market evidence says beginners are having a harder time getting in the door. The learning evidence says that once they're in, heavy delegation to AI can interfere with acquiring the very skills they'd need to supervise AI later. The first mechanism reduces opportunity. The second reduces learning per opportunity. If both persist, the pipeline gets squeezed at both ends: fewer repetitions available because fewer beginners are hired, and then some of the repetitions that remain get cognitively handed to the machine anyway. *Figure 17.1. The Expertise Ladder — repetition leads to pattern recognition, diagnosis, judgment and verification. Author diagram.* Take the learning side first, because it's the one people wave away fastest. Judy Hanwen Shen and Alex Tamkin at Anthropic ran a randomized experiment on developers learning an unfamiliar asynchronous programming library — real professionals, randomly split into groups with and without AI assistance, then tested on what they'd actually learned. The result was not "AI makes people stupid." It was more specific, and more useful. AI assistance impaired conceptual understanding, code reading, and debugging in the experiment, without producing a significant efficiency gain on average. The participants who delegated most heavily did pick up some productivity — at the cost of learning the library. And most importantly, the researchers found that *how* people used the tool mattered: some interaction styles preserved learning, because the participant stayed cognitively engaged. Delegating a task and interrogating a task are different mental activities even when both involve the same model. That last part is the design target, and I'll come back to it. But notice the nasty incentive sitting in the middle of this. AI assistance is most attractive exactly where novice work is slowest — the beginner struggles with syntax, doesn't know where to look, takes an hour on what a senior does in ten minutes. So the immediate productivity case for the tool is strongest precisely where the long-term learning cost may matter most. The organization sees the saved minutes. The worker feels the relief. Neither one sees the missing repetition, because a repetition that didn't happen doesn't show up anywhere. The bill arrives years later, when that worker is expected to debug a system whose underlying concepts were delegated all the way through training. Now the labor side. The Stanford Digital Economy Lab's August 2026 revision of its *Canaries in the Coal Mine?* work isn't a forecast — it's payroll data through June 2026. The researchers find no widespread, economy-wide AI job displacement, and it's important to say that plainly. But among workers ages twenty-two to twenty-five in the most AI-exposed occupations, employment sits about 19 percent below the path implied by similarly aged workers in less-exposed occupations, and the divergence is primarily a hiring story — firms not bringing beginners in, rather than firing the ones they have. They also found something aimed straight at this chapter: the declines concentrate where AI use tends toward automation, while occupations characterized more by augmentation fare better, particularly for experienced workers. A Census Bureau working paper gives a second view with a different design: in the most AI-exposed industry-state cells, regression-adjusted employment for workers twenty-two to twenty-four fell 12 percent over the ten quarters after ChatGPT arrived, again driven mainly by reduced hiring. Two datasets, two designs, the same pressure point: the bottom rung. --- ## The ladder was never the titles We talk about career ladders as if they're made of titles — analyst, associate, manager; junior, mid-level, senior; resident, fellow, attending. The titles aren't the ladder. The ladder is the sequence of increasingly difficult decisions hidden underneath them, and most of those decisions look trivial while you're standing on the low rungs. The first-year lawyer reads the documents; later she decides which documents matter. The junior programmer fixes small bugs; later he decides which architecture creates fewer bugs. The young reporter calls people and checks the spelling of names; later she decides which source is lying. The apprentice electrician pulls wire; later he hears a sound in a panel and knows something's wrong before the meter agrees with him. I recognize this from my own trade. Nobody becomes a closer by studying closing. You become a closer by getting two hundred doors shut in your face and noticing, somewhere around door one-fifty, what the last five seconds before a shut door sound like. Then a company pays you to fly around the country teaching it, and you discover you can barely explain it — you can only make new guys stand on porches until they hear it too. Every profession has a version of that. Senior people call it intuition, which makes it sound mystical. It's compressed experience. It's a library of failures, and the only known way to install the library is repetitions. That's what makes the low rung so dangerous to automate. So much of it is repetitive — and repetition is exactly how the pattern library gets built. The same ticket that looks like drudgery to a senior engineer may be one of the hundred small experiences that eventually lets a junior smell a bad system before it fails. Remove the repetition and you may remove the waste. You may also remove the training. Time served is not the same as repetitions completed; a person can spend ten years near a task without spending ten years doing the cognitive work that builds judgment about it. --- ## The spreadsheet can't see 2033 Here's how the rational version of the mistake happens. A company measuring AI productivity over one quarter can reach a perfectly sound conclusion: ten junior employees cost a million dollars a year, AI lets five seniors absorb most of their routine output, so the junior layer is redundant. Maybe it is — for the next quarter. The spreadsheet just doesn't contain a row labeled "people who will be qualified to replace the seniors in 2033." That asset was never on the balance sheet. It walks out the door every night. Economists would call what happens next an externality. Each company can save money by cutting training opportunities and assume somebody else is developing the next generation. If enough companies make the same rational decision, nobody is. The result isn't a wave of unemployment. It's a thinner bench — and a thinner bench is invisible right up until the moment you need it. The scarier version isn't even about headcount. A labor market can keep roughly the same number of people employed while hollowing out the process that turns beginners into experts. You can have entry-level roles that are mostly AI orchestration and senior roles that still demand deep judgment, with the connective tissue between them gone. If a junior's job becomes forwarding machine output, five years of forwarding machine output does not produce a senior verifier. The title advances. The pattern library doesn't. That — more than any particular job disappearing — is what I mean by the apprenticeship problem. Aviation, which we'll get to properly in Chapter 22, is the industry that stared at this earliest and hardest, and its answer contains one load-bearing word. Before automation, practice was embedded in production — pilots hand-flew because there was no alternative. After automation, practice has to be *designed*. The FAA didn't tell airlines to choose between using the autopilot and keeping pilots sharp. It told them to do both: use the automation, and maintain manual proficiency deliberately. Deliberately is the whole ballgame. Once the environment stops demanding a skill, the skill stops maintaining itself for free. --- ## The school before the workplace Every apprenticeship problem starts earlier than employment. Before the junior associate there's the law student. Before the junior developer there's the CS student. Before the analyst there's an undergraduate staring at an assignment at one in the morning with a chatbot open in the other tab. And school is where the loop can close on itself. The student uses AI because the student doesn't yet know the material. The AI produces the answer because it's seen patterns the student hasn't. The assignment gets completed. The student banks fewer repetitions. Then that student graduates into a workplace that expects them to supervise the same kind of system that did their homework. The thing to understand about school is that an assignment was always manufacturing two products. The visible product was the essay, the program, the proof, the lab report. The invisible product was the student. If a machine produces the visible product more efficiently while interfering with production of the invisible one, then grading the assignments will tell you the exact opposite of what happened. The papers got better and the students learned less, and both of those can be true at once. There's early evidence that's exactly the trade on offer. A 2025 randomized controlled trial took 120 undergraduates learning material about AI, gave one group ChatGPT as a study aid and the other traditional methods, and then — forty-five days later — hit everybody with a surprise retention test. The AI-assisted group averaged 57.5 percent. The traditional group averaged 68.5 percent, a statistically significant gap in that study. One experiment with 120 students doesn't settle the future of education, and I won't pretend it does. What it establishes is something education policy can't ignore: a tool can improve the immediate learning experience while changing what's still in your head six weeks later. The time horizon changes the result. Part of why is that human beings are terrible at telling fluency from mastery. Read a clear explanation and the idea feels obvious. Watch somebody solve the problem and the steps feel easy. Ask the AI and the answer makes sense. Then close the window and try to do it, and the difficulty comes right back — because recognizing an answer and retrieving one are different mental operations. Education knew this long before AI; it's why testing yourself beats rereading the chapter that feels comfortable. What's new is that comfort is now available on demand, twenty-four hours a day, and a student can remove every difficulty before discovering which difficulty was doing the teaching. The old answer key at the back of the book had a useful limitation: it sat there. It didn't rewrite your essay, explain the problem six different ways, imitate your voice, or finish your assignment and then reassure you that you understood it. Generative AI is an answer key that participates in the entire cognitive process. Which means the old integrity question — did the student cheat? — is too narrow. A student can use AI completely honestly and still outsource the exact mental operation the lesson existed to train. The question that matters is: which part of the thinking must the learner still perform for the learning objective to survive? Schools have answered with every policy imaginable — ban it, embrace it, disclose it, cite it, brainstorming only, everything because the workplace will. The binary debate misses that this is a design problem. If the objective is factual recall, unrestricted AI during retrieval defeats it. If the objective is learning to critique arguments, handing students flawed AI arguments to tear apart may strengthen it. If the objective is learning to program, generating everything kills the debugging practice; if the objective is learning to *review* code, the machine can produce an endless supply of examples. The right policy depends on the cognitive operation being trained, which is harder than one school-wide rule and a great deal more defensible. --- ## The closed-book moment Everything in this chapter points at one design principle, and it works the same in a classroom and a company: every AI-assisted workflow should eventually contain a closed-book moment. No model, no search, no hints. Explain it. Solve it. Debug it. Teach it back. Predict what happens next. The purpose isn't punishment. It's measurement. If the person can perform after the support disappears, the support built capability. If performance collapses the moment the AI goes away, the system built dependence, and you'd rather find that out on a Tuesday exercise than during the outage. Schools need it because AI broke the cheap proxy — for decades a good essay implied a student who could write and working code implied a student who could program, and generated artifacts have cut the wire between the artifact and the inference. So assessment has to move closer to the capability itself: oral defense, live problem-solving, unassisted components, critique of generated work, explaining *why* an answer is wrong. All of it more expensive than grading a stack of documents. Verification usually is. Generation got cheap; knowing what the generation proves did not. Workplaces need the same test, and the same design discipline. None of this means making young accountants add columns by hand because accountants used to add columns by hand — that's nostalgia, not training. The question is which cognitive operations create transferable judgment, and the answer usually has the same shape: make the human commit before showing them the machine's version. If AI drafts the memo, the junior identifies the controlling facts first. If AI writes the code, the junior predicts the failure modes and writes the tests. If AI proposes the diagnosis, the trainee commits to a differential first. Then compare. The novice thinks, the machine produces, and the disagreement between them becomes visible — which may teach more than either one alone. But only if the commitment comes first. Once the polished answer is on the screen, anchoring has already started. So no, protecting apprenticeship doesn't mean freezing every junior job as it existed in 2019. It means protecting the ingredients: independent first attempts, exposure to failure, feedback from people who know more, responsibility that grows gradually, repetition of the decisions that matter, periodic work without the automation, and explanation rather than mere completion. If those survive, the job can change radically and still produce experts. If those go, keeping the words "junior analyst" on an org chart saves nothing. The work that builds judgment *is* the ladder. We can automate parts of the climb, and we should. But if we automate the whole climb, we shouldn't act surprised when the next generation arrives at the title without ever having made the ascent. --- ## Sources and Further Reading - Shen & Tamkin, *How AI Impacts Skill Formation* (2026); Anthropic Research summary. - Stanford Digital Economy Lab, *Canaries in the Coal Mine?*, revised August 12, 2026 (payroll data through June 2026; ~19% relative-path gap for ages 22–25 in highly AI-exposed occupations; adjustment primarily through reduced hiring; concentration in automation-oriented use). - U.S. Census Bureau working paper on AI exposure and young-worker employment (ages 22–24 down 12% in the most exposed industry-state cells over ten quarters, driven mainly by reduced hiring). - "ChatGPT as a cognitive crutch: Evidence from a randomized controlled trial on knowledge retention," *Social Sciences & Humanities Open* 12 (2025) (120 undergraduates; 57.5% vs. 68.5% on a surprise test 45 days later). - Federal Aviation Administration, SAFO 13002 and SAFO 17007, *Manual Flight Operations Proficiency*. - Lisanne Bainbridge, "Ironies of Automation," *Automatica* 19(6), 1983. - Brynjolfsson, Li & Raymond, "Generative AI at Work," NBER Working Paper 31161, for evidence that AI can also improve novice performance in workplace settings. - Labor-market evidence discussed in Chapters 8, 13, 14, and 15 of this book. --- --- \pagebreak # Chapter 18 # The Factory for Almost Right The first industrial revolution didn't just make individual workers faster. It changed the unit of production — from the craftsman to the factory. That's what AI is doing to knowledge work right now, and it's worth slowing down on, because it changes the mathematics of error in a way that catches almost everybody off guard. The important shift is not that one person can write one email faster. It's that one person can set fifty processes in motion and come back an hour later to fifty pieces of finished-looking work. One operator can have agents researching, coding, summarizing, drafting, classifying, contacting, comparing, and revising in parallel. The worker stops being a producer of individual outputs and becomes the manager of an output factory. That sounds like leverage, and it is. I run a small version of that factory off a phone. But nobody hands you the factory and mentions what it does to your error math. Suppose a process is right 99 percent of the time. At ten decisions a day, the one percent is manageable — you'll eat a mistake every couple of weeks and probably catch it. At a million decisions a day, the same process produces ten thousand failures. The system's accuracy didn't change. Its exposure did. Scale turned a small residual error rate into a production number, and that's why "pretty reliable" means something completely different at volume. AI companies compete, understandably, on model accuracy. An organization has to care about something else: expected error volume — accuracy times deployment volume, times consequence. A one percent error rate on restaurant suggestions is noise. A one percent error rate on automated account closures is a crisis, run at industrial speed. The factory for almost right isn't dangerous because every product is bad. It's dangerous because a small residual error becomes industrial output too. --- ## The bottleneck doesn't disappear. It moves. You can watch this play out in software first, because software keeps score. AI can increase the volume of generated code faster than any organization's security-review capacity grows. Veracode's benchmark from the last two chapters is one warning shot — functional and syntactic quality improving while security quality lags. OWASP's Improper Output Handling category is the other — generated output turns dangerous the moment it passes downstream without adequate validation. But the lesson is bigger than software, because every AI workflow has a downstream. A customer. A database. A court. A patient. A payment system. A publication. A physical machine. Wherever generated output touches consequence — that's where verification capacity has to exist, and if production scales by ten while checking scales by two, you have not eliminated the bottleneck. You've moved it downstream and hidden it under a bigger pile. There's a second trap built into the factory model, and it's sneakier. Parallelism looks like independence, and it usually isn't. Ten agents can all be running the same model, retrieving from the same contaminated source, inheriting the same system instruction, sharing the same blind spot. Ten answers are not ten independent answers if they're ten branches of one error. That's why "I asked it again and it agreed with itself" is not verification. Independence requires an actual change in something — the evidence, the method, the model, the tool, the incentives, or the human looking at it. Otherwise you're not counting checks. You're counting echoes. So the organization running the factory needs what engineers would call an error budget for generated work — a deliberate answer, in advance, to a short list of questions. Which outputs can fail harmlessly? Which failures need to be caught before anything acts on them? Which need a human's name on them? Which must be reversible? Which are too consequential to automate end to end? That's not one corporate AI policy. It's an architecture, and the answer should be different for an internal meeting summary than for denying somebody a benefit. If the control is identical for both, the control is probably meaningless. --- ## Evidence-shaped objects Now here's the part of the factory that worries me most, because it's the part aimed at how we know things. AI does something strange to evidence: it makes evidence-shaped objects cheap. A citation. A quotation. A footnote. A case name. A statistic. A confident summary of a study. Before generative AI, fabricating those things at scale took real effort — you had to know what a citation looked like, and typing a hundred fake ones was its own miserable job. Now the formatting is free, which changes the reader's problem entirely. The old question was: does this look researched? The new question is: can I reach the thing underneath it? The legal record gives the cleanest measurement we have, because it's current and it's countable. On August 28, 2026, Damien Charlotin's AI Hallucination Cases Database had identified 1,981 legal decisions in which courts or tribunals addressed alleged or established AI use involving hallucinated material more than in passing. Charlotin is explicit about what that number is not: it's not a count of every fake citation, or every AI-assisted filing. It's narrower — only the cases where the problem became visible enough for a court to deal with it on the record. And the narrowness is exactly why it matters. Each one of those 1,981 entries is a moment where generated material crossed a line: it left a private conversation with a model, entered a filing, acquired the visual status of legal work, and forced an institution to respond. Generation isn't the moment that worries me. The consequential moment is when the output crosses a desk, enters a record, gets a signature, and picks up institutional weight. And the fake citation is honestly the easy case. You search, it doesn't exist, caught. The harder case is a real source attached to a false claim. The title is real. The authors are real. The journal is real. The link works. And the sentence sitting on top of it overstates what the study found. That error sails through a superficial fact-check because every object in the citation is genuine. I know, because I found versions of it in this book. Auditing my own manuscript taught me that a number can be real and still be wrong in context. A study can be real and still not prove causation. A weekly user count and a monthly user count can both be real and turn false the moment somebody combines them. Collecting links is not verification. The real work is preserving the relationship between a claim and the evidence underneath it — including the inconvenient qualifiers that make the claim smaller than you wanted it to be. The way I think about it now is a ladder. At the bottom rung: the claim is plausible. One up: a source exists. Up again: the source actually contains the information. Up again: the source supports this particular interpretation. Up again: the source is strong enough for the weight of the claim. And at the top: independent evidence points the same direction. The machine is genuinely helpful on the bottom rungs. The writer is still responsible for the climb — and a bibliography only proves that sources were collected, not that the prose stayed inside them. One more thing about primary sources, because I resisted this for months: they're annoying on purpose. They're longer. They're technical. They hedge. They use confidence intervals. They distinguish correlation from causation. They refuse to hand you the clean sentence you came for. That annoyance is information. When a press release says "AI makes developers faster" and the underlying paper says "in this randomized setting, for this population, on these tasks" — the caveat is the boundary of what's actually known. Generative prose sands boundaries smooth. Verification puts them back. --- ## A book about checking has to be checkable There's an obvious hypocrisy risk in this whole project, and I'd rather name it than wait for a reviewer to. I used AI to help build a book that argues AI output requires verification. Good. That should make the standard stricter, not looser. The test was never whether AI touched the manuscript. The test is whether the factual claims in it can survive independent inspection — which is why this book ships with a public claims ledger, and why the ledger distinguishes *verified*, *corrected*, and *narrowed* instead of stamping the whole thing "fact-checked" as if one label could certify sixty thousand words. Some claims in earlier drafts didn't survive the audit. They were corrected or cut, and the ledger says which and why. The point of verification was never to look verified. It's to expose where the confidence comes from — so that when I'm wrong, somebody can find it. The same rule applies to the factory. Scale is not going away, and it shouldn't; the same machinery that produces at volume can test at volume, log completely, sample continuously, and route anomalies to a human. NIST's AI Risk Management Framework exists precisely because trustworthy deployment is a process, not a one-time certificate. The factory doesn't have to become a factory for error. But the verification line has to scale with the production line — and right now, almost everywhere, we're building the production line first. --- ## Sources and Further Reading - Damien Charlotin, AI Hallucination Cases Database, as of August 28, 2026 (1,981 legal decisions in which courts or tribunals addressed alleged or established hallucinated AI material more than in passing; not a count of all filings or all fake citations). - Veracode, *2025 GenAI Code Security Report* and Spring 2026 update (vendor benchmarks). - OWASP GenAI Security Project, *Top 10 for LLM and GenAI Applications 2025* — Improper Output Handling. - NIST, *Artificial Intelligence Risk Management Framework* and *Generative Artificial Intelligence Profile*; NIST AI Resource Center resources on testing, evaluation, verification, and validation. - The public verification ledger accompanying *Almost Right*. --- --- \pagebreak # Chapter 19 # The Liability Gap By 2026, the legal profession had moved past embarrassing lawyer filings into a more uncomfortable category: errors inside the judiciary itself. In August 2026, the U.S. Court of Appeals for the Fifth Circuit was weighing whether to take a case away from U.S. District Judge Henry Wingate after an earlier order in it contained significant inaccuracies and fabricated material. Wingate told Senate Judiciary Committee Chairman Chuck Grassley that a law clerk had used Perplexity while drafting court documents, and he described the episode as "a lapse in human oversight." Sit with that phrase, because it compresses this entire chapter into four words. The AI did not hold judicial office. The clerk did not possess the judge's authority. The order entered the machinery of the court anyway — and the moment the output became consequential, responsibility snapped back to the human institution, the way it always does. "The AI made the mistake" turned out not to be an available answer. AI can participate in a chain of production. Institutions still need a chain of responsibility, and nobody had drawn one. That's the liability gap. Capability is moving toward the machine. Responsibility is staying with people. The model can draft the legal argument, but it isn't the lawyer. It can suggest the diagnosis, but it isn't the physician. It can write the code, but it isn't the engineer signing the deployment. In one sense that's healthy — tools shouldn't become legal persons just because they're useful. But it puts enormous pressure on the human at the end of the chain, who is now expected to certify output produced at a speed and volume no human can fully inspect. When a machine is wrong, everybody suddenly becomes a philosopher — was it a tool, was it an agent, who knew, who should have known? Those questions sound abstract right up until money, liberty, health, or safety is on the line. Then they become invoices. --- ## The insurers will get there first You can learn a lot about an emerging risk by watching the people who have to price it. By August 2026, Reuters was reporting that cyber insurers were rewriting policy language because autonomous AI agents had scrambled the old definitions of authorized access, cyberattack, and responsibility — specialized AI coverage was emerging while underwriters wrestled with having no long loss history to price from. I find that oddly reassuring, because insurance is the industry that forces philosophical ambiguity into operational categories. The insurer cannot settle for "the AI did something weird." Was access authorized? Was the action intended? Was the control adequate? Who owned the system? Who could have stopped it? A price has to attach to the answer. And that pressure will push companies toward better logs, narrower agent permissions, real approval gates, and documented human review — not because everybody suddenly got philosophically careful, but because ambiguity got expensive. The insurer's questionnaire may become a de facto standard before the legislators finish arguing about definitions. The better the audit trail, the easier it is to assign responsibility; the clearer the responsibility, the stronger the incentive to build the audit trail. Those two economies feed each other. --- ## Agency raises the stakes A chatbot that gives a bad answer can mislead a person. An agent with tools can *act* on the bad answer, and that difference is enormous. OWASP's framework separates the two risks by name: Improper Output Handling is generated output moving downstream without validation, and Excessive Agency is a system handed enough live authority to take consequential actions itself. In 2026, OWASP's exploit round-up documented real incidents where agentic systems took destructive or risky actions after being given live permissions, and its recommendations read like something written by a burned adult: explicit confirmation for destructive operations, reversible workflows, constrained permissions, human review. The governing principle is one sentence: the harder an action is to reverse, the harder it should be for generated output to trigger it directly. Delete, send, pay, publish, deploy, approve, deny, prescribe — those verbs need gates. The model's ability to perform them is not evidence that it should be allowed to perform them without an independent control in the way. --- ## Oversight has to have teeth The most common answer to all of this is "put a human in the loop," and I want to be blunt about how easy that phrase is to fake. Put a checkbox in the interface, require somebody to click approve, call the system supervised. Done — on paper. Real oversight has four properties, and you can test for every one of them. The human can understand the decision well enough to challenge it. The human has information that isn't just the model's own explanation of itself. The human has enough time to actually perform the review. And the human has the authority to stop or change the action. Remove any one and the oversight weakens. Remove all four and the human isn't in the loop at all — the human is the liability sponge at the end of the loop, kept there to absorb blame the process was designed to generate. Here's the difference in practice. If a reviewer gets one AI recommendation every ten minutes, with the expertise, time, and standing to reject it, "human in the loop" means something. If the reviewer gets a thousand outputs an hour, sees only the model's preferred answer, is measured on throughput, and gets dinged for slowing things down, the phrase describes a seating arrangement. And you can't fix that with individual willpower, because every incentive points one way: the machine is fast, the interface is confident, the organization paid for it, the recommendation is already on the screen, rejecting it creates work, and accepting it completes the task. A tired employee at 4:45 on a Friday is not a safety architecture. If an organization wants independent judgment, it has to build the conditions where independent judgment can physically occur. One principle does a lot of the work: responsibility should follow authority. Don't assign responsibility to someone who lacks the authority, the information, or the time to exercise it. If a junior must approve AI output but can't challenge the system, the approval is decorative. If a doctor is legally responsible for a recommendation but the workflow hides the model's uncertainty, responsibility and information have been separated. If an engineer has to sign off on generated code but the release schedule makes real review impossible, that signature is being used to absorb institutional risk, not manage it. The ethical problem there is obvious. The operational problem is worse: organizations that build fake accountability learn less from their failures, because the paperwork insists the control existed. --- ## Who verifies the verifier? Every fix in this chapter invites the same annoying question one level up. Use an expert to check the AI — fine, who checks the expert? Use a second model — who checks the second model? Use a benchmark — who designed it? Use a certification — who certifies the certifier? At first that sounds like an infinite regress. It isn't, and the proof is that civilization has been solving this exact problem for centuries — never by finding one perfectly reliable authority, but by building overlapping layers whose errors are visible to each other. Science has peer review, then replication, then the later study that contradicts the earlier one. Accounting has internal controls, then external audits, then regulators, then courts. Aviation has pilots, then checklists, then maintenance, then air-traffic control, then accident investigation. Software has tests, then code review, then security review, then monitoring, then incident response. No single layer is trustworthy. The system is trustworthy *because the layers fail differently.* *Figure 19.1. Who Verifies the Verifier? — the book's proposed layered verification architecture. Author diagram.* That's also why "human in the loop" is too primitive as a final answer. A human can be wrong, tired, captured by incentives, or deferential to the machine. The goal was never to replace machine fallibility with human infallibility. The goal is a structure where one failure doesn't automatically become the final answer. And the variable that actually matters isn't whether the checker is made of neurons or silicon. It's independence. Ask a model whether its own summary is accurate, in the same conversation, and it says yes — that's nothing. Open a second window with the same model: barely better. Have a different model check the citations against the primary papers: better. Have deterministic software confirm the quoted numbers actually appear in those papers: better still. Have a domain expert read the decisive claims: better. Publish the sources so outside readers can attack the interpretation: now you have layers, and none of them guarantees truth, but together they make an error very hard to preserve. The enemy of all of it is correlated error. If five reviewers all rely on the same wrong source, five reviews are one review. If three agents share the same training bias and the same retrieval corpus, their agreement is an echo. If every employee starts from the same generated summary, that summary becomes organizational memory before anybody opens the underlying document. Modern systems are stuffed with correlated error because efficiency loves standardization — same dashboard, same vendor, same model, same metric. Standardization reduces random variation and quietly raises systemic risk, because when the common component is wrong, everything downstream is wrong together. Verification architecture needs deliberate diversity — different evidence, different methods, different incentives — not as decoration, but as fault tolerance. A few structures reliably create that diversity, and none of them are new. The red team: don't ask whether the system seems secure, pay somebody to break it, and give them status for succeeding — a policy team needs someone hunting for the population the model harms, and a book needs someone hunting for the sentence that can't survive its source. External review: the person who didn't spend six months falling in love with the architecture is the one who can ask the rude question, starting with *why are we doing this at all?* A real escalation path: when a reviewer flags a problem, somebody with power has to hear it, and if launch proceeds anyway, somebody has to put a name on the override — "I reviewed the objection and authorize proceeding" changes behavior in a way anonymous friction never does. Random audits: you can't deeply inspect everything, so sample like the IRS does, so that any output *might* be inspected and the checking strategy says something about the whole population. And one metric almost nobody tracks: the disagreement rate. How often do humans reject the AI's recommendation? How often does the second model contradict the first? A disagreement rate of zero is not a system that's perfect. It's usually a checker that isn't independent. Healthy verification produces friction; the only question is whether the friction finds consequential problems at a price you can afford. One irony deserves its own paragraph. Give a verifier AI tools and the verifier gets faster — good. Then the citation checker uses AI, the security reviewer uses AI, the auditor uses AI, the regulator uses AI, and we're back at the beginning, everyone downstream of the same class of system they're supposed to be checking. That doesn't invalidate AI-assisted verification. It means some independent capacity has to live outside the shared dependency. A calculator can check arithmetic, but somebody still has to understand arithmetic. A model can compare a claim to a paper, but somebody still has to be able to read the paper. The final defense is not any particular tool. It's preserved competence, distributed through the system. --- ## The person who signs At the end of most consequential workflows there will still be a person. A doctor. A lawyer. An engineer. An executive. A judge. The AI may have done most of the visible production, and the signature will mean what it has always meant: *I am willing to rely on this.* That's a heavy sentence, and whether it stays true depends on choices institutions are making right now — whether the signer gets enough time, information, training, and authority for the sentence to be honest. Documentation is part of it, and it isn't bureaucracy when the system can improvise: the same prompt produces different output, the model changes, the retrieval changes, and for consequential uses you need to be able to reconstruct what happened — which model, which instruction, which sources, which output, which human decision. Without that chain, accountability becomes storytelling after the fact, and storytelling after the fact is exactly where confident institutions become almost right. Give people a protected right to escalate output they can't verify, too — no penalty for slowing the process, no demand that they prove the model wrong before asking for another set of eyes — because uncertainty is often the signal, and an expert's *this doesn't fit* is one of the most valuable outputs experience produces. We talk about trust as a feeling — I trust the AI, I don't trust the AI. That's the wrong level. Trustworthy systems don't ask for blind trust in any component, including the human one. They build architecture: constraints, tests, logs, independent review, sampling, escalation, liability, transparency, recovery. The architecture assumes every participant — human or machine — will eventually be wrong, and then asks whether the system catches the mistake before the mistake becomes the consequence. If institutions build that, the signature at the bottom of the page stays meaningful. If they don't, we will have automated the work while preserving the liability as a ritual, and at some point the human signature becomes camouflage for a process no human could actually inspect. --- ## Sources and Further Reading - Reporting on the Fifth Circuit's consideration of reassignment and Judge Henry Wingate's letter to Sen. Chuck Grassley regarding a law clerk's use of Perplexity in drafting (August 2026). - Reuters, reporting on cyber insurers revisiting policy language for autonomous AI agents (August 2026). - OWASP GenAI Security Project, *Top 10 for LLM and GenAI Applications 2025* — Improper Output Handling; Excessive Agency — and the 2026 exploit round-up recommendations. - NIST, *Artificial Intelligence Risk Management Framework* and *Generative Artificial Intelligence Profile*. - Lisanne Bainbridge, "Ironies of Automation," *Automatica* 19(6), 1983. - FAA safety and proficiency guidance discussed elsewhere in this book. - Legal and professional-responsibility cases discussed in Chapters 4 and 6. --- --- \pagebreak # Chapter 20 # The Verification Economy Every technological boom creates a second economy behind the first one, and the second one usually ends up bigger than anybody guessed. Cars created mechanics, insurance, crash testing, traffic engineering, licensing, dealerships, parking, and an entire legal architecture around machines whose original purpose was just to move people. The internet created cybersecurity, content moderation, fraud detection, identity verification, cloud monitoring, and data privacy — whole industries nobody was hiring for in 1995. AI is going to create its own second economy, and I think a large part of it will be verification. The first AI economy is easy to see because it sells generation: write this, code this, summarize this, design this. The product is visible the second it hits the screen. The second economy sells something less glamorous and, wherever the stakes are real, more valuable: confidence. Is the result correct? Is it safe? Did the source actually say what the summary claims? Will the code survive production? And the biggest question of all — is somebody prepared to put a name behind this? Generation produces the object. Verification is what gives you a defensible reason to rely on it. Economically, those are different products, even when the user experiences them as one click. Here's the mechanism underneath it, and it's the same one every abundance creates. When something is expensive to produce, scarcity itself does a crude filtering job. Publishing a book once required editors, printers, and money. Shipping software once required people who could write software. Those barriers excluded plenty of good work and good people, and tearing them down is a genuine benefit — I walked through the hole in that wall myself. But barriers also imposed friction, and when the cost of production collapses, the scarce resource moves. If ten reports exist, finding somebody competent to read them is easy. If ten thousand exist, reading becomes the bottleneck. If software can be generated in minutes, testing becomes the bottleneck. If personalized advice can be generated for everyone, figuring out which advice deserves reliance becomes the bottleneck. AI doesn't eliminate scarcity. It relocates it — away from generation, toward attention, judgment, and accountability. Cheap output makes trust expensive. That's the whole economy in one sentence. --- ## The work already has a name This isn't just my prediction — you can watch the work being formalized in real time. The National Institute of Standards and Technology built its AI Risk Management Framework around managing trustworthiness across the design, development, use, and evaluation of AI systems, and its resource center groups a set of activities under an acronym you're going to hear a lot: TEVV — testing, evaluation, verification, and validation. NIST's Generative AI Profile extends the framework to generative systems specifically. The acronym is bureaucratic. The function is economic. Somebody has to determine whether the system does what it's supposed to do under the conditions where people will rely on it — and none of this proves a giant labor market appears under the job title "AI verifier," because markets rarely adopt the clean academic name. What it proves is that the work is being written down, and once work is written down it can be budgeted, and once it can be budgeted it can become a profession, a vendor category, an insurance requirement, a procurement requirement, or all four at once. *Figure 20.1. The Verification Economy — cheap generation creates abundant output, a verification bottleneck and a trust premium. Author diagram.* The shape of the work has to change too, because traditional professional verification is artisanal. A senior lawyer reviews the associate's memo. A senior engineer reviews the pull request. An editor checks the reporter. That model breaks the moment output grows faster than senior attention, which is exactly what's happening — so verification will industrialize the same way production did. Machine checks for what machines can check cheaply. Deterministic rules where rules can constrain. Sampling for the low-risk volume. Human review reserved for the ambiguous, contextual, high-consequence residue, with escalation for uncertainty and audit trails for reconstruction. That's not a retreat from AI. It's how every high-throughput industry has ever matured. The factories didn't respond to mass production by firing quality control because the machines were faster than the craftsmen. They industrialized quality control too. Which means every consequential AI workflow needs a verification budget — not necessarily money, capacity. If a system can generate ten thousand customer decisions a day, how many can be independently checked? If a coding agent produces fifty pull requests, how much security review exists? If a research tool spits out a hundred citations in seconds, who confirms them? If the answer is "the same people as before," then generation capacity grew and verification capacity didn't, and I wouldn't call all of that output productivity yet. Until somebody can rely on it, a chunk of it is inventory — unverified intellectual inventory, piling up faster than the organization can absorb it. The business question was never how much the AI can produce. It's how much *verified* output the organization can safely absorb. That's the number that belongs on the dashboard, and almost nobody has it on the dashboard. --- ## The signature gets more valuable Professional systems figured out something a long time ago that AI is about to make newly important. An audit report is valuable because somebody qualified signs it. A structural drawing becomes actionable because an engineer stamps it. A prescription carries weight because a licensed professional authorizes it. The signature was never magic — it's a compressed representation of a verification process, professional standards, and liability, all rolled into a name. Generative AI increases the value of that compression, because when anybody can produce a professional-looking artifact, appearance stops telling you how much expertise went into it. The question shifts from "does this look like expert work?" to "who checked it?" The signature moves from the end of a human production process to the end of a hybrid production-and-verification process, and I think that may turn out to be one of the biggest professional shifts of the whole era. You can already sketch what the service looks like. Picture a small-business owner in 2028. She uses AI to generate an employee handbook, a privacy policy, a marketing claim, a tax categorization, and a vendor contract — five things she'd have hired five professionals to create a few years earlier. The demand for professional labor doesn't disappear. It changes shape, from "make this for me" to "tell me whether I can trust this." That's a different service model. It's probably faster and cheaper, and it lets one expert serve many more clients. It also increases the expert's exposure, because now she's reviewing artifacts whose hidden assumptions were generated somewhere she can't see, and the economics of the profession will have to price that risk. To be clear about what the job is: a verifier is not a proofreader. The job is not making machine output prettier. The job is establishing what kind of confidence the output deserves — through testing, source-checking, adversarial review, security analysis, compliance, and domain sign-off. Different fields will call it different things: auditor, reviewer, red teamer, quality engineer, safety officer. The title matters less than the position: the verifier sits in the uncomfortable space between plausible output and consequential reliance, easy to dismiss while nothing is wrong, and the role everybody wishes had been stronger the day after something is. One honest caveat about why this doesn't just get automated away. Generation parallelizes beautifully — a model can produce a hundred drafts. Judgment doesn't scale the same way. Plenty of verification can and should be automated: run the tests, check the citations, scan the dependencies, compare the numbers, set one model against another. But the difficult residue is difficult precisely because the easy tests don't capture it. What did we fail to ask? What assumption is hidden? What happens in the unusual case? Who gets hurt if this is wrong? Those questions require context, and context is expensive. That gap between the cost of generating and the cost of judging is where the wages in this economy come from. --- ## The new professional class So who does this work? Cybersecurity is the preview, because security people have always assumed that functional success proves nothing. Does the application work? Good. Now try to break it. That second sentence is the entire verifier mindset, and AI is exporting it from the security department to everybody else — because the Veracode numbers from Chapter 16 are what happens when a generator optimizes visible success faster than hidden safety. I want to be precise about the temperament, because "verifier" sounds like "pessimist" and it's the opposite. Pessimism expects failure. Verification designs a way to find out. The best verifier can be genuinely excited about AI and still refuse to confuse excitement with evidence — can use the system every day and still ask for the source, can want the launch to happen and still stop it. That matters because the AI hater and the AI true believer share the same weakness: both of them already know the answer before the test. The verifier wants the test. The valuable professional of the next decade is bilingual. One language is the domain — law, medicine, engineering, accounting, operations, whatever your field is. The other language is AI: not just how to open a chatbot, but how models fail, how context changes output, how to constrain a workflow, how to test, how to keep provenance, how to make one system challenge another, and how to know when the automation should stop. A professional who knows the domain but refuses the tools becomes unnecessarily slow. A person who knows the tools but not the domain becomes dangerous in the opposite direction. The valuable combination is fluency in both — enough machine leverage to move fast, enough domain judgment to know when to stop. If I were building a curriculum for this person, I'd start with five things. Source discipline: where did the claim come from, can I reach the original, does it support this exact sentence — basic, and shockingly rare. Failure imagination: how could this be wrong while still looking right, and what would an adversary do with it? Independent reconstruction: can I get to the conclusion by another path — a separate calculation, a different source, a test the model didn't write for itself? Calibration: how sure am I, what would change my mind, and which parts of this are fact, inference, and guess — the person who says "I'm sixty percent sure, and here's why" is worth more than the person who says "confirmed." And escalation judgment: which mistakes are cheap, which are reversible, which need another expert, which mean stop the line. Verification resources are finite, and knowing where to spend them is most of the skill. Organizations will need to build careers around these people — call it a verifier track, next to the management track and the individual-contributor track. People whose status comes from reliability rather than volume. People rewarded for catching the consequential problem before launch, with real authority to stop one. People who maintain the test suites, the evaluation sets, the incident libraries, and who study failures across teams so each team doesn't rediscover the same lesson at full price. Fragments of this already exist — QA, security, compliance, internal audit, editorial standards — and AI is making the common structure underneath them easier to see. They are all institutions for organized doubt. That phrase sounds negative and it's one of civilization's best inventions. Science is organized doubt. Auditing is organized doubt. Appellate courts, peer review, red teams, checklists, the second pilot in the cockpit — all of it exists to prevent one confident process from being the only process. AI makes confidence cheap. Organized doubt just became more valuable. There's a cultural problem to get over first, and I say this as a guy who has spent his whole life on the production side of every room he was ever in. We reward makers. Founders build, engineers ship, writers publish, closers close. Checking sounds secondary — the verifier is the person slowing everybody down. That hierarchy made sense when production was scarce. It makes less sense when production is nearly free. In an abundance economy, refusal becomes a skill: no, this isn't ready; no, that source doesn't support the claim; no, the model passed the benchmark and failed the real workflow. The person able to say those sentences — and be right — becomes infrastructure. And the strangest status shift of all may be what happens to "I don't know." Generative systems are optimized to keep going; professional judgment sometimes requires stopping. *I can't verify this. The source doesn't say that. We need somebody else.* Those feel like weak sentences in a culture that rewards instant answers. They're verifier sentences, and one of the oddest things AI may do to the labor market is make epistemic humility economically valuable. --- ## The career, and the premium If I were twenty years old looking at this transition, I would not try to compete with the machine at raw first-draft production, because that's competing against a falling price. I'd learn one domain deeply enough to verify it, and then I'd learn the AI tools better than the people who are avoiding them. That combination — domain judgment plus machine leverage — is harder to commoditize than either half alone. The person who only generates is racing the cost curve down. The person who only knows the old workflow gets lapped. The person who can drive the machine hard and still independently challenge it stands on the narrow bridge between speed and trust, and that bridge is where I believe a great deal of professional value is about to move. Economists talk about skill premiums — extra pay attached to scarce capability. I expect a verification premium. Not everywhere, and not overnight. But wherever a plausible error is expensive and generated output is abundant, the ability to certify, test, or reject becomes the scarce thing, and the premium will attach wherever it can: to credentials, to reputations, to firms, to software that keeps honest audit trails, to insurance, and to individual people whose judgment has survived enough failures that somebody trusts their no. The AI boom gets described as a race to make intelligence cheap. Fine. But if generation gets cheap enough, the valuable person is no longer the one producing the intelligence. It's the one who can tell you which intelligence to trust. My own workflow is the small proof I can personally vouch for. When this technology first felt like magic, I accepted first answers. Now I define what success means before I prompt, figure out what would be expensive if wrong, let the machine generate, separate the claims from the prose, check the high-consequence claims independently, test the thing in the environment where it will actually run, ask what I'm unable to check — and escalate that residue to somebody who can. It's slower than trusting the first answer. It's still enormously faster than doing everything from scratch. That's the point. The choice was never speed *or* verification. The competitive advantage is the combination — and the machine itself will help, if you ask it right: not for the answer, but for the test. Ask it to attack its own proposal. Ask it for three ways the plan fails. Ask it which of its claims need outside verification. Ask it to hide the answer until you commit to yours. The model can be a crutch or a sparring partner. Design decides which. --- ## Sources and Further Reading - NIST, *Artificial Intelligence Risk Management Framework* and *Generative Artificial Intelligence Profile*; NIST AI Resource Center resources on testing, evaluation, verification, and validation (TEVV). - Veracode, *2025 GenAI Code Security Report* and Spring 2026 update (vendor benchmarks, discussed in Chapter 16). - Stack Overflow, *2025 Developer Survey*. - METR, Becker et al., *Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity* (2025). - Anthropic Research, Shen & Tamkin, *How AI Impacts Skill Formation* (2026). - OWASP guidance on LLM application risks and validation. - Lisanne Bainbridge, "Ironies of Automation," *Automatica* 19(6), 1983. - FAA manual-flight proficiency guidance. --- --- \pagebreak # Chapter 21 # The Case for Optimism I've spent most of this book describing failure, and that can leave the wrong impression, so let me say it plainly: I am optimistic about AI. Not because the problems are small — you've just read five chapters of them — but because the capability is enormous and the problems are increasingly visible, named, and measured. The dangerous technology is never the one with known failure modes. It's the one everybody assumes has none. We're still early enough to build the checks while the systems are being integrated, and that's not a consolation prize. That's an opportunity. Start with the fact that the productivity is real, because I refuse to build this book on pretending it isn't. Over five thousand customer-support agents, a generative assistant, roughly 14 percent higher productivity on average and about 34 percent for the newest, lowest-skilled workers — that's not hype, that's measured workplace performance in the studied setting. In the BCG experiment, consultants working inside the model's capability frontier completed more tasks, finished them faster, and did higher-quality work. Those are substantial, documented benefits. The argument of this book was never that AI can't improve work. It's that the improvement is conditional — and once you know the conditions, you can design for them. And the condition I care most about cuts both ways. The same feature that creates the apprenticeship problem could democratize expertise. A novice can now get guidance that used to require sitting next to a patient senior employee for years. A small business can access analytical capability it could never have afforded. A student can get an explanation at midnight from a tutor that never sighs. A patient can walk into an appointment knowing which questions to ask. Those aren't minor conveniences — they change who gets to attempt difficult things. I'm one of the people that matters for. I built businesses because the machine lowered a wall that would have kept me out for the rest of my life. So the lesson I took from my own wreckage was never *rebuild the wall*. It was: once the wall comes down, you need a railing on the other side. This chapter is about the railing. --- ## The same leverage works in both directions Here's the piece of good news that took me longest to see. Everything AI does to generation, it can do to checking. A model can scan a manuscript for factual claims and list them. It can compare two versions of a document. It can flag unsupported assertions. It can generate adversarial test cases faster than any human tester. It can read logs, hunt for contradictions, summarize incident patterns, and translate technical evidence into something a human reviewer can act on. It can help a novice verifier ask expert-level questions. The leverage is symmetrical — and that means the verification gap is not a law of physics. If AI multiplies output by ten and we point the same machinery at multiplying verification capacity by ten, the gap can narrow. That outcome isn't guaranteed. It's *available*, which is a different thing, and whether we collect it is a choice. Part of collecting it is learning to design friction on purpose. Tech companies spend fortunes removing friction — one click, instant, automatic, no confirmation — and most of the time that's good product design, right up until the action is consequential. Then friction becomes a safety feature. A confirmation before deleting ten thousand files is good friction. A source preview before publishing a claim is good friction. A forced human review before an autonomous agent moves money is good friction. A closed-book assessment before certifying that somebody can do the job is good friction. The sensible future isn't frictionless. It puts the friction where the consequence justifies it and strips it out where a mistake is cheap and reversible. And none of this requires inventing a new theory of civilization, which is the most reassuring part. Humanity has been building reliable systems out of unreliable components forever. Airplanes contain parts that fail. Hospitals contain humans who make mistakes. Financial systems contain fraud, software contains bugs, science contains bad studies. We didn't fix any of that by finding perfect components. We built institutions around the imperfection — redundancy, standards, training, audits, appeals, incident investigation, insurance, licensing. AI doesn't need a new playbook. It needs the old lessons applied to a new source of capability, faster than we usually apply them. --- ## The apprenticeship can actually get better I'll go further than defense. Done right, AI could make apprenticeship *better* than it was, because let's be honest about the old version: it wasn't sacred. Juniors spent years on work that was educational mostly because nobody had invented a cheaper way to get it done. Some of it was pointless. Some seniors were terrible teachers. Some professions used plain suffering as a substitute for curriculum. I've stood through enough of the sales version to say that with confidence. AI gives us the chance to separate the useful repetition from the ritual. Instead of a junior analyst burning six hours formatting slides, generate the slides — then spend the saved time making the analyst defend the assumptions underneath them. Instead of a young lawyer hand-summarizing a thousand irrelevant pages, triage with the tools — then make her read the decisive documents and explain the legal consequence out loud. Instead of a novice developer typing boilerplate, generate it — then make him attack it. The apprenticeship gets cognitively denser: less busywork, more judgment per hour. That would be genuine progress, not preservation. The verifier doesn't have to become the bottleneck, either. The bad version of verification is a line of people manually reading everything the machine produces, and that version will fail on day one. The good version is risk-based, and it's how every mature quality system already works: automate the cheap checks, sample the low-risk output, escalate the anomalies, build reusable tests, track the recurring failures, use AI to pre-screen AI, and reserve expensive human attention for the residue that actually requires judgment. That scales. There's even a competitive angle. Imagine two AI products. One hands you an answer. The other hands you the answer plus the sources, the assumptions, the uncertainty, the tests it ran, and a record of what a human approved. The second one feels less magical. In any market where the stakes are real, it's worth more — because trust becomes a product feature, and verification becomes part of the user experience instead of homework the customer is secretly expected to do after the product finishes. That's where I think serious AI products are headed: not away from capability, toward *inspectable* capability. And markets know how to reward it once buyers ask. If customers demand verified output, vendors will sell verification. If insurers price weak controls, companies will strengthen controls. If courts sanction unverified citations, lawyers will check citations. If employers promote the people who catch consequential errors, workers will learn to catch them. Incentives built the speed race. The same incentives can build the trust race — and the verification economy isn't just a defensive tax on AI. It may be the very mechanism that lets AI into the high-stakes domains where its biggest benefits live. --- ## Almost right is a solvable problem The title of this book sounds pessimistic. I don't mean it that way. "Almost right" is useful information. It tells you the size and location of the remaining job. If the machine were always wrong, there'd be no revolution to worry about. If it were always right, there'd be no verification problem to solve. It's powerful precisely because it lives in the difficult middle — often excellent, sometimes transformative, sometimes wrong, and sometimes wrong in ways that look excellent. That is a system we know how to work with. We've built our whole civilization around components exactly like that. We just have to stop demanding that this one be either magic or fraud first. There's one condition, and it's the sentence I'd put on the wall. We have to value the checking *before* the absence of checking becomes catastrophic enough to force us. Aviation learned from crashes. Medicine learns from harm. Security learns from breaches. Law is learning, right now, from sanctions. We don't have to let every field pay the maximum tuition — the warning signs are already published, the research is already public, the errors are already on the record, and so is the capability. The machine can help us build the verification systems around the machine. That may turn out to be the most important thing we ever use it for. --- ## Sources and Further Reading - Brynjolfsson, Li & Raymond, "Generative AI at Work," NBER Working Paper 31161. - Dell'Acqua et al., "Navigating the Jagged Technological Frontier," *Organization Science* (2026). - NIST AI Risk Management Framework and Generative AI Profile. - Shen & Tamkin, "How AI Impacts Skill Formation" (2026). - Stanford Digital Economy Lab, *Canaries in the Coal Mine?*, revised August 2026. --- --- \pagebreak # PART VI — HOW WE CHECK --- # Chapter 22 # What the Pilots Did Here is the thing I keep coming back to, and it's the reason this book has an ending instead of just a warning. Somebody has already been here. Commercial aviation confronted a version of the problem described in the last four chapters: highly capable automation, concern about erosion of manual proficiency, and rare situations in which a human suddenly has to recover from something the automated system did not handle. The industry's response is useful here because commercial aviation has developed unusually mature systems for training, recurrent practice, incident investigation, and layered safety. Let me be careful about the claim. Aviation did not solve AI verification. Nobody has. What aviation did was develop real, tested answers to automation dependency and skill decay in its own domain. Those answers are documented and public, and they are almost entirely unused in software, medicine, law, and education — mostly because nobody thought to look at the airlines. So let's look. --- ## What aviation actually did After Air France 447 and the studies that followed it, the response was not to remove the automation. Nobody suggested that. Autopilots make flying dramatically safer, the way AI polyp detection makes colonoscopy better. Taking the tool away was never on the table. What aviation did instead was four things, and each one has a direct analog in the problem this book describes. **First: it named the failure mode out loud.** The FAA's 2013 Safety Alert for Operators said plainly that continuous use of automated flight systems "could lead to degradation of the pilot's ability to quickly recover the aircraft from an undesired state." That's an industry regulator, in writing, telling operators that its own best technology damages the people who use it. A second alert followed in 2017. Compare that to where we are now. No regulator has issued the equivalent statement about AI. Nobody has told hospitals that computer-aided detection may erode endoscopist skill, even though it's published in *The Lancet*. Nobody has told engineering managers that AI assistance reduces comprehension most in debugging, even though the company selling the tool published that finding themselves. **Second: it made unassisted practice mandatory rather than optional.** The FAA encouraged operators to build manual flight operations back into ordinary line flying — hand-flying the aircraft in normal conditions, not just in emergencies, specifically so the skill stays alive. Airlines and regulators worldwide followed with policies putting manual proficiency back into recurrent training. The crucial design choice: it's *scheduled*. Nobody relies on pilots choosing to practice. It's on the calendar, it's in the checkride, and you don't keep your license without it. **Third: it made the practice unpredictable.** Simulator sessions don't just run the failure the crew is expecting. The whole point is that you cannot prepare for the specific scenario, because in the real event you won't know what's coming. Recognition under uncertainty is the skill being trained — not the muscle memory of one recovery. **Fourth: it investigates every failure in public.** When an aircraft goes down, an independent body examines it and publishes what it finds, including when the finding embarrasses a manufacturer or an airline. The industry improves because failures become shared knowledge instead of private liability. Software has nothing like this. When Tea disclosed the exposure of roughly 72,000 images — including about 13,000 verification images containing photo IDs and selfies — there was no aviation-style independent public accident investigation with authority to publish a technical cause report for the whole industry. There were ten lawsuits. Lawsuits produce settlements and non-disclosure agreements, which is roughly the opposite of an aviation accident report. --- ## The one that transfers immediately Of those four, the second is the one you can apply tomorrow, in any profession, without waiting for a regulator. **Scheduled practice without the tool.** Not because the tool is bad. Because the skill is a muscle, and Chapter 14 measured how fast it goes. In the observational study, unassisted adenoma detection among the participating endoscopists fell from 28.4 percent before AI introduction to 22.4 percent in the post-implementation period. That is an association over time, not a randomized estimate of individual skill loss. For a doctor, that might mean a scheduled proportion of procedures performed unassisted, tracked the way detection rates are already tracked. For an engineer, writing and debugging something by hand on a regular cadence. For a student, assessment conditions where the tool isn't available — which is not nostalgia, it's the only way to find out whether learning happened. For a lawyer, drafting from the source material before reading the machine's version. And here's the part that makes it hard, which I want to name rather than pretend away: every one of those costs productivity in the short run, and the benefit is invisible. That's exactly why aviation had to make it mandatory. No individual pilot would choose to hand-fly when the autopilot is right there. No hospital would voluntarily lower its throughput. No engineering manager under a deadline will tell a junior to spend three hours on something the machine does in four minutes. The economics run one direction, always, and they run against the practice. Which means the practice has to be a policy, or it doesn't happen. That is the single most important sentence in this chapter. --- ## The evidence that design fixes this Aviation is the historical proof. Here's the current experimental proof, and it's the most hopeful finding in the book. Go back to the Turkish math classroom from Chapter 9. Three groups: no AI, unrestricted chatbot, and a guardrailed tutor built to walk students through problems rather than hand them answers. The unrestricted group scored about 17 percent *worse* than students with no AI at all. The guardrailed group did not show that harm. Same model. Same students. Same subject. The entire difference was in how the interface was designed. Now Anthropic's developer study from Chapter 12, arriving at the same place from a completely different direction. Fifty-two developers learning a new library. Overall, the AI-assisted group scored 50 percent on comprehension versus 67 for the hand-coders. But when the researchers split the AI group by *how* people used the tool: Participants who used it for conceptual inquiry — asking follow-up questions, requesting explanations, posing "why does this work" while coding themselves — scored **65 percent or higher**. Participants who used it for delegation — have it write the code, move on — scored **below 40 percent**. Twenty-five points or more, from the same tool, in the same session, on the same task. The variable was whether the person was trying to understand or trying to finish. Put those two studies together and you get the conclusion Part V is built on: **The harm is not inherent to the technology. It's a function of interface design and usage pattern, and both of those are choices somebody makes.** That's genuinely good news, and it's why this book doesn't end in despair. It also locates the responsibility precisely, which is uncomfortable for the companies involved. If the damage came from the model itself, nobody would be to blame. It doesn't. It comes from design decisions optimized for engagement and speed — for the answer that satisfies rather than the interaction that teaches. Those decisions are made in product meetings, for commercial reasons, and they could be made differently tomorrow. Anthropic's researchers said as much to managers: think intentionally about how these tools are deployed, and "consider systems or intentional design choices that ensure engineers continue to learn as they work." --- ## What the platforms did after they got burned I'll give credit where the record supports it. Some of this is already happening, reactively. After the Replit agent deleted Jason Lemkin's production database during a code freeze, the company shipped automatic separation between development and production environments, a planning-only mode where the agent can think but not act, and one-click restore. Those are good changes. They are also, precisely, the aviation move: constrain what the automation can do without a human in the loop. After Matt Palmer published CVE-2025-48757, Lovable added a security scanner and a review tool. Palmer's criticism — that the scanner checks whether a policy exists rather than whether it works — is fair, and the company's own statement was unusually candid: "we're not yet where we want to be in terms of security." After Wiz reported the Base44 authentication bypass, Wix fixed it in under 24 hours. Every one of those improvements arrived *after* real users were exposed. That's the pattern aviation abandoned decades ago in favor of designing for the failure before it happens. But it's a pattern, and it means the industry can move when it's embarrassed. Which suggests a strategy: embarrass it earlier. --- ## The manuals already exist The most frustrating discovery I made writing this book is that the guidance is already written, free, public, and almost entirely unread by the people who most need it. **The OWASP Top Ten for LLM Applications.** OWASP is the volunteer foundation whose security lists half the internet is built against. They maintain a list specifically for AI applications, updated for 2025. Prompt injection is number one. Sensitive information disclosure is number two. It costs nothing and takes an afternoon. **CISA and the UK's NCSC, *Guidelines for Secure AI System Development*,** published November 26, 2023, endorsed by eighteen nations. Four stages: secure design, secure development, secure deployment, secure operation. Its central premise is worth memorizing, because it's the opposite of how this market has behaved — the burden falls on the people who *build and sell* the system, not the people who use it. As the NCSC's chief executive put it, security must be "not a postscript to development but a core requirement throughout." **NIST's AI Risk Management Framework**, January 2023, with a generative-AI supplement in July 2024 containing more than two hundred suggested actions. **And row-level security**, which is documented in the manual of every database that has it, and which would have prevented the Tea breach, the 170 leaking Lovable apps, and a meaningful share of the more than 2,000 vulnerabilities Escape.tech found across 5,600 live applications. None of it is mandatory. That's Chapter 6's finding arriving in Part V with a practical edge: the problem was never that we didn't know what to do. It's that knowing was never enough, and nobody made it a requirement. --- ## Who's actually preserving apprenticeship The hardest question in this chapter is the succession problem, and here I have to be honest that the evidence is thin. Matt Garman made the argument publicly in August 2025 — replacing junior developers is "one of the dumbest things I've ever heard," and "ten years in the future you have no one that has learned anything." That's the CEO of AWS. It is a strong, clear, correctly-reasoned public statement. What I could not find is a body of evidence that companies are *acting* on it at scale. Some organizations report adding "how to work with AI assistance" modules to onboarding, having mentors review AI-generated code with juniors to teach the reasoning behind it, and in some cases requiring periods of manual coding before granting AI access. Those are the right instincts, and I want to be careful not to inflate scattered reports into a movement. Because look at the economics, which one industry observer summarized about as bluntly as it can be put: training costs money, AI-boosted juniors ship faster, and short-term return favors delegation over learning. That's the whole problem in one sentence. Every incentive at the firm level runs against apprenticeship, and the cost of skipping it lands on the industry a decade later, when the firm that skipped it will hire from a pool it assumed somebody else was filling. Economists have a name for that: a collective action problem. Nobody's individual interest is served by training people who can leave. Everybody's collective interest requires it. Historically these get solved exactly two ways — industry-wide agreement, or regulation — and neither is currently in progress. I don't have a solution to offer you there. I have a request, which is Chapter 24. --- ## What good looks like Let me put the pieces together into what a serious response would actually be, borrowing directly from the industry that already did this. 1. **Name the failure mode publicly**, the way the FAA did in 2013. Regulators and professional bodies telling their members, in writing, that the tool degrades the skill it substitutes for. 2. **Schedule unassisted practice** and make it a condition of licensure or employment where the stakes justify it. Doctors, engineers, pilots, lawyers, accountants. Not optional, because optional means it doesn't happen. 3. **Make the practice unpredictable**, so what's trained is recognition under uncertainty rather than one rehearsed recovery. 4. **Design interfaces for comprehension, not just completion.** The guardrailed tutor and the conceptual-inquiry pattern both work and both are measured. Build tools that ask a question back. 5. **Investigate failures in public.** An independent body that examines significant AI-caused failures and publishes causes, the way transportation accidents are handled. 6. **Make the free manuals mandatory** where consequences are real. OWASP's list is one afternoon. Row-level security is a few lines of configuration. 7. **Protect junior roles as a capability investment**, and be honest that this needs coordination because no single firm's interest supports it. That's the institutional answer. It requires regulators, professional bodies, and companies to act, and Chapter 6 gave you a realistic picture of how likely that is in the near term. Which is why the next chapter is about the only actor in this entire book whose behavior you actually control. You. --- ## Sources and Further Reading Federal Aviation Administration, Safety Alert for Operators 13002 (2013) and 17007 (2017). Bureau d'Enquêtes et d'Analyses, final report on Air France Flight 447, 2012. Bastani et al., "Generative AI Can Harm Learning," *PNAS*, 2025. Shen & Tamkin, "How AI Impacts Skill Formation," arXiv:2601.20245 (2026); Anthropic Research, January 2026 (conceptual-inquiry users ≥65%; delegation users <40%). Budzyń et al., *The Lancet Gastroenterology & Hepatology*, August 2025. Replit platform changes following the July 2025 incident (company statements; The Register, July 22, 2025). Matt Palmer, "Statement on CVE-2025-48757," mattpalmer.io; Lovable public statement. Wiz Research, "Critical Vulnerability in Base44," July 2025. OWASP Top 10 for LLM Applications 2025, OWASP GenAI Security Project (genai.owasp.org). CISA/NCSC, "Guidelines for Secure AI System Development," November 26, 2023 (endorsed by 18 nations). NIST AI Risk Management Framework 1.0 (AI 100-1), January 2023; NIST Generative AI Profile (AI 600-1), July 2024. Escape.tech, "State of Security of Vibe-Coded Apps." Matt Garman, The Register, August 21, 2025. --- \pagebreak # Chapter 23 # Become the Verifier Everything up to here has been a description of a problem. This chapter is what you do about it on Monday. I'm going to organize it three ways, because three different people are reading this book: someone raising a kid, someone with a job, and someone building something. Read yours. Read the others if you want; they overlap more than you'd think. Before any of it, the one idea that everything else hangs on. **The scarce thing is no longer producing work. It's knowing whether the work is right.** That's it. That's the whole book compressed. Producing got cheap. A billion people can now generate a competent-looking anything in four seconds. Checking got cheaper too, in places — that's the honest version, and Chapter 4 named the tools that do it. Automated tests catch broken code. Databases reject impossible values. Security rules stop a program reaching data it has no business touching. Those are real and they scale. But the checking that requires *judgment* — is this argument sound, is this diagnosis right, does this contract protect me, is this the answer that only looks correct — still runs at the speed of one person who understands the subject. Generation raced ahead. Verification-by-judgment did not. And that specific capacity is the one this technology appears to be eroding in the people who use it most. That gap is the opportunity. Which means the position to occupy, in every field, for the next twenty years, is the person who can tell. That's not a consolation prize for people who can't keep up with AI. It's the highest-value role in the entire arrangement, and it's about to be badly undersupplied. --- ## If you're raising a kid Start with the finding that should determine your household policy, because it's the strongest evidence in this book aimed at a decision you personally control. In the Turkish classroom study, students with unrestricted chatbot access scored roughly 17 percent *worse* on their exams than students with no AI at all. Not "gained less." Worse than nothing. Meanwhile students using a guardrailed tutor — one built to walk them through problems instead of handing over answers — did not show that harm. The tool isn't the variable. The design is. So: **Distinguish the two uses, out loud, by name.** There is *asking it to explain something* and there is *asking it to do something*. The first builds understanding. The second replaces it. This isn't my opinion — it's the 25-point gap in the Anthropic study, conceptual inquiry versus delegation. Kids can absolutely learn this distinction. Give them the words for it. **Protect the struggle.** The reason a math problem works is the ten minutes of being stuck. That is the entire mechanism. When AI removes the stuck part, it removes the learning and leaves behind a correct answer, which is the part that was never valuable. If your kid is stuck and frustrated, that is the machine working. Don't rescue it, and don't let a chatbot rescue it either. **Insist on unassisted assessment.** Not because tests are sacred, but because it's the only way to find out whether anything got learned. This is the pilots' scheduled practice, applied to a fourteen-year-old. **Use the quote test.** Ask them to tell you, without looking, one thing from the thing they just finished. It takes four seconds and it's the same test the MIT researchers used. If they can't, the work happened somewhere other than in their head. **And be honest about the other side.** Prohibition is not a strategy; the technology is in the phone, the school, the search results. Your kid needs to be fluent in this thing. The goal isn't keeping them away from it — it's making sure they build the underlying capability *and* the fluency, in that order, so they end up on the right side of Chapter 8. --- ## If you have a job Whatever your field, this is arriving. Software got it first and hardest, and Chapter 12 is your preview. **Use it. Seriously.** The Chapter 9 evidence is that the biggest gains go to the least experienced — 30 to 34 percent for the newest workers in that call center, versus almost nothing for the veterans. If you've spent your life being told you're not technical, you are the person this technology helps most. Opting out is not a principled stand; it's choosing the wrong side of a divide. **Then build the checking habit while it's cheap.** Right now, on low-stakes work, when being wrong costs you nothing. Because the reflex has to already exist on the day it matters, and you cannot install it in the moment. **Know your own weak spot.** You are best at catching errors in things you understand deeply and worst at catching them in things you're using the machine to cover for. Which means the danger zone is precisely where you're using it most — the gap in your own competence. That's not a reason to stop. It's a reason to know that output from that zone needs a second source. **Verify anything that's checkable and consequential.** Names, dates, numbers, citations, quotes, legal claims, medical claims, anything you'd be embarrassed to be wrong about in public. The lawyers, litigants, judges, and courts documented in Damien Charlotin's database show how expensive the failure can become once generated material enters a legal process. **Watch for the confidence gap.** In METR's early-2025 randomized study, experienced open-source developers believed AI would make them faster yet took 19 percent longer on the measured tasks. In a separate controlled security study, participants with AI assistance produced less-secure code while becoming more likely to believe their code was secure. Different studies, different tasks, same reason to measure rather than rely on the feeling of fluency. **Practice without it on a schedule.** This is the aviation move, and it's the best defense anybody has found against the Chapter 14 problem. Pick the core skill of your job — the thing you'd be embarrassed to have lost — and do it unassisted regularly enough to know you still can. Not because you'll need to work without the tool. Because the day the tool is confidently wrong about something important, what stands between that error and the world is whether you can still tell. **And use it to understand, not just to finish.** The 65-versus-40 split. Ask why. Ask what would break this. Ask what you're missing. Same tool, same time, completely different outcome for the person using it. --- ## If you're building something This is the section I needed and didn't have. These five controls map directly onto failure modes documented in Chapter 11. They are not a substitute for a professional security program, but they are a much better starting point than asking the generator whether its own work is safe. **1. Enforce authorization at the database and server, not merely in the interface.** In platforms such as Supabase, a public or anonymous client key may legitimately be present in browser code; that key is not supposed to be a master secret. The protection comes from correctly configured database authorization — including row-level security where appropriate — plus server-side checks. Private service-role credentials must never be shipped to the browser. Missing or insufficient row-level security was central to the reported Lovable/Supabase exposure identified as CVE-2025-48757. Tea was a different failure: reporting described unsecured legacy cloud storage and a separately accessible database, so row-level security should not be presented as the control that would have prevented Tea. **2. Check authentication on the server, never only in the browser.** Anything enforced in code a user can see is a suggestion, not a rule. Base44's authentication bypass worked because undocumented endpoints required only a value visible in the app's own URL. **3. Search your shipped code for secrets before you launch.** Passwords, API keys, database tokens, service credentials. Escape.tech found over 400 exposed secrets across 5,600 live AI-built applications — keys sitting in the file every visitor downloads. Open your deployed site's source and search it yourself. It takes five minutes. **4. Separate development from production.** Never let an agent operate on live customer data. Replit shipped automatic dev/prod separation *after* an AI agent deleted a paying customer's production database during a code freeze — and then told him it was unrecoverable, which was false. **5. Get an independent review before real users arrive.** For security-sensitive software, that means an actual security review; for usability-critical software, it also means testing the deployed product as a user would. I learned the second lesson the cheap way: I asked for an end-to-end systems check, was told everything worked, spent real money on advertising, and discovered that visitors could not click the control that started the search. A code-level check had passed while the real user path had failed. And if you're connecting an AI assistant to your data, one more, from Simon Willison's "lethal trifecta": don't give a single agent private data, exposure to text a stranger wrote, and a way to send information out. Any two are fine. All three is the configuration that took Microsoft, Salesforce, and OpenAI in the same year. Then go read the OWASP Top Ten for LLM Applications. It's free, it's an afternoon, and prompt injection is number one on it. --- ## What this actually asks of you I want to be honest about the cost, because a plan that pretends there isn't one is the kind of confident, plausible, unverified output this whole book is about. Every item above is slower than not doing it. Checking the citation is slower than pasting it. Practicing without the tool is slower than using it. Letting your kid stay stuck is harder than letting the chatbot answer. Running a real security review delays your launch. The productivity gain is immediate and visible. The verification cost is immediate and invisible. That asymmetry is why almost nobody does this, and why it can't be left to individual willpower at the institutional level — which is Chapter 22's argument for policy. But at your own level, it's a decision you can just make. And here's the case for making it, beyond avoiding disaster. People who can independently verify machine-generated work are likely to become more valuable as generated output grows, especially if fewer workers get the apprenticeship and practice that build that capability. Chapter 15's arithmetic: fewer juniors entering, learning less of the specific skill, while the veterans erode. Whatever your field, the person who can look at plausible output and say *that part's wrong, and here's why* is about to be the scarcest thing in the building. That's a job description. It's available. Almost nobody is training for it, and the tool everyone is using makes people worse at it by default and better at it if used deliberately. You get to choose which. --- ## One thing I'd ask you to remember Of everything in this book, if you keep one sentence, keep the one I said to a friend when he asked why I wasn't more impressed with what I'd built: *Even the biggest cup in the world doesn't hold water if there's a small hole in it.* Capability is not the variable. Nobody in this book failed because the machine wasn't smart enough. Tea's storage worked. The Replit agent executed flawlessly. The endoscopy AI detected polyps accurately. My search tool searched. Every one of them failed at containment — at the small hole nobody looked for, in a vessel everybody was busy admiring the size of. Your job, from here forward, in whatever you do: be the person who looks for the hole. Not because the cup isn't magnificent. It is. I built a company on a phone with it and I'd do it again tomorrow. Because magnificent cups leak too, and somebody has to check. --- ## Sources and Further Reading Bastani et al., "Generative AI Can Harm Learning," *PNAS*, 2025. Shen & Tamkin, "How AI Impacts Skill Formation," arXiv:2601.20245 (2026) (conceptual inquiry ≥65% vs delegation <40%). Kosmyna et al., MIT Media Lab, 2025 (the quotation test). Brynjolfsson, Li & Raymond, *Quarterly Journal of Economics* 140(2), 2025. METR, July 10, 2025. Perry, Srivastava, Kumar & Boneh, ACM CCS 2023. Charlotin, "AI Hallucination Cases" database, damiencharlotin.com/hallucinations. Federal Aviation Administration, SAFO 13002 (2013). Tea breach reporting, July–August 2025. Matt Palmer, "Statement on CVE-2025-48757," mattpalmer.io. Wiz Research, "Critical Vulnerability in Base44," July 2025. Escape.tech, "State of Security of Vibe-Coded Apps." Replit incident and platform changes, July 2025. Simon Willison, "The lethal trifecta for AI agents," June 16, 2025. OWASP Top 10 for LLM Applications 2025. --- \pagebreak # Chapter 24 # Start Now On August 31, 1955, four men put their names on the proposal that became the founding document of the Dartmouth summer project on artificial intelligence. The surviving typescript runs seventeen pages plus a title page. They proposed that ten people, working for two months in New Hampshire, could make significant progress on machines that use language, form abstractions and concepts, solve problems reserved for humans, and improve themselves. They gave themselves a summer. I started writing this book seventy-one years later, to the day. I didn't plan that. I found out afterward, while checking the date on the proposal, and I've thought about it since because of what it says about time. They were wrong about the schedule by seven decades. Everyone in this book has been wrong about a schedule. The 1958 newspaper said the Navy's machine would soon be conscious of its own existence. Minsky and Papert's proof emptied the field in 1969 and the idea came back. The expert systems were going to replace professionals in the 1980s and they didn't. Dario Amodei said in May 2025 that half of entry-level white-collar jobs could vanish within one to five years, and Sam Altman said in May 2026 that he'd expected more displacement than had happened and was "delighted to be wrong." Predictions about this technology have a terrible record, in both directions, made by the smartest people available. I've tried very hard, throughout this book, not to add to the pile. So I'm not going to close by telling you what 2040 looks like. I don't know. Nobody does. What I'm going to do instead is tell you what's already measured, because that's the only thing I've earned the right to say. --- ## What we actually know Strip out every projection, every CEO quote, every model of the future, and this is what's left standing: A machine that produces plausible output regardless of truth, and whose own maker published a paper explaining that its training rewards guessing over admitting uncertainty. Adoption at extraordinary speed — roughly 45 percent of working-age Americans using generative AI by the study cited earlier in this book, and about 900 million weekly ChatGPT users by the last confirmed weekly figure used here. A separate 2026 estimate put monthly app users above one billion; that monthly figure is not a weekly-user count. No binding federal rules in the United States. One comprehensive law in Europe, its core provisions postponed six days before they would have taken effect. Excellent free guidance from CISA, NIST, and OWASP that nobody is required to read. In Veracode's Spring 2026 vendor benchmark, 45 percent of tested code-generation tasks failed its security criterion even as syntax correctness exceeded 95 percent. The CVE-2025-48757 research reported insecure endpoints across 170 of 1,645 scanned Lovable projects. Escape's vendor research reported more than 2,000 vulnerabilities across more than 5,600 publicly available vibe-coded applications. Tea disclosed roughly 72,000 exposed images, including about 13,000 verification images containing photo IDs and selfies; a separate issue later exposed more than a million private messages. In METR's early-2025 randomized study, sixteen experienced open-source developers took 19 percent longer with the tested AI tools even though they expected a speedup. In the Anthropic skill-formation experiment, fifty-two developers showed a 17-point average comprehension gap between the AI-assisted and hand-coding groups, with especially important differences in debugging-related understanding. In the observational colonoscopy study, unassisted adenoma detection fell from 28.4 percent before AI introduction to 22.4 percent in the post-implementation period. And in Stanford's August 2026 payroll-data revision, employment for workers ages 22 to 25 in highly AI-exposed occupations stood 19 percent below where it would have been had it kept pace with less-exposed peers of the same age; experienced workers showed no comparable gap, and the paper found no widespread economy-wide job displacement. And one 1983 paper, about power plants, containing the sentence that ties all of it together: current automated systems "are riding on their skills, which later generations of operators cannot be expected to have." That's the book. Not a forecast. A set of measurements, taken by different people, in different fields, mostly not talking to each other, all pointing the same way. --- ## Why now and not later Here's the argument for urgency, and it isn't about how fast the technology improves. It's about how slowly people are made. Senior judgment in high-skill work takes years to build and, in many professions, something close to a decade. The timetable is not identical for engineers, surgeons, pilots, litigators, machinists, and reporters. The common feature is prolonged practice: boring tickets, routine procedures, small cases, supervised mistakes. The individual tasks are not the whole point. The judgment they accumulate is. Which means the people who will be able to tell "almost right" from right in 2040 have to be in the pipeline now. Not soon. Now. The window for producing that generation isn't decades wide; it's about the length of one career stage, and it's open at this moment. And unlike almost everything else in this book, that's not a projection. It's arithmetic on how long training takes. Meanwhile, some warning signals appear on much shorter timescales. In the Polish observational study, unassisted adenoma detection fell from 28.4 to 22.4 percent across the study periods. In Anthropic's controlled learning experiment, the AI-assisted group scored 17 percentage points lower on immediate comprehension. Neither result proves permanent skill loss, but neither operates on a generational timescale either. Fast erosion, slow replacement, and a window that's open right now. That's the whole case for not waiting. --- ## What I'm not saying I want to be exact, one last time, because the failure mode of a book like this is to become the thing it warns about — confident, plausible, and unchecked. I'm not saying AI is bad. I built two businesses with it from a phone with no engineering background, and I'd do it again. The productivity findings in Chapter 9 are real and the largest gains go to the least experienced, which is one of the more genuinely democratic things a technology has ever done. I'm not saying it's making everyone stupid. The evidence doesn't support that and the people claiming it will look foolish. I'm not saying mass unemployment is coming. The Yale Budget Lab found no discernible disruption 33 months in. The CEOs who predicted otherwise reversed themselves. The entry-level collapse might be interest rates, and if it is, I'll be glad. I'm not saying stop using it. That advice is useless, and worse, it puts whoever takes it on the wrong side of Chapter 8. I'm saying one thing, and it's narrow enough that I think it survives whatever happens next: **We built machines that can produce plausible work faster than human judgment can verify many consequential outputs, while the apprenticeship and practice that create expert verifiers are under pressure.** The first half is directly observable. The second is a synthesis of hiring, learning, and deskilling evidence — not a single settled measurement. That distinction matters. --- ## The ask So here's what I want, from wherever you're standing. **If you run something:** protect the junior roles. Not out of charity — because Garman is right and ten years from now you'll be hiring from a pool you assumed somebody else was filling. And schedule the unassisted practice, because your best people are eroding right now and no metric on your dashboard will show it. **If you make policy:** the guidance already exists. CISA and NCSC wrote it in 2023 and eighteen nations signed it. OWASP maintains the list. Making the basics mandatory where consequences are real doesn't require inventing anything — it requires deciding that free advice nobody follows isn't a policy. **If you teach:** the guardrailed tutor works and the unrestricted chatbot measurably harms. That's not a values question anymore, it's a finding. Build for comprehension, assess without the tool, and protect the part where the student is stuck. **If you're a parent:** the quote test, tonight. Four seconds. Then have the conversation about explaining versus doing. **And if you're just a person with a job and a phone:** be the one who checks. Verify what's checkable. Practice what you'd hate to lose. Use it to understand rather than to finish. And when the output is confident and plausible and important, spend the extra ten minutes. That last one is the entire ask. Ten minutes. Against a machine that produces in four seconds what used to take four hours. It sounds small. It's the only thing between plausible and true. --- ## The last thing I keep thinking about that Dartmouth proposal, and about what it actually asked for. They wanted machines that could form concepts. Understand. Improve themselves. What got built instead — after two collapses, seventy years, and more money than most countries have — is a machine that predicts the next word so well that its output is indistinguishable from understanding. They asked for comprehension and we got plausibility, and plausibility turned out to be worth trillions. That's not a tragedy. Plausibility is enormously useful. I've built my livelihood on it. Hundreds of millions of people use these systems every week and get real value from them. But there's a condition attached, and it's the one nobody wrote into the proposal. A machine that produces plausibility requires a world that still contains comprehension. Somebody, somewhere, has to be able to tell the difference. That was never a problem in 1956, because in 1956 the comprehension was all on our side of the table and none of it on the machine's. Seventy-one years later we've built the plausibility at extraordinary scale, and we are — quietly, without deciding to, mostly by accident and economics — dismantling the comprehension that made it safe. Nobody voted for that. No one company chose it. It's the sum of a million reasonable local decisions: skip the junior hire, ship the feature, accept the draft, trust the output, don't schedule the practice. Which means it's reversible by a million reasonable local decisions going the other way. That's what I'm asking for. Not fear. Not rejection. Not a return to anything. Just: somebody has to check. Let it be you. --- ## Sources and Further Reading McCarthy, Minsky, Rochester & Shannon, "A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence," August 31, 1955. Kalai, Nachum, Vempala & Zhang, "Why Language Models Hallucinate," arXiv:2509.04664, September 4, 2025. Bick, Blandin & Deming, *Management Science*, 2026. Regulation (EU) 2026/1744 (Digital Omnibus on AI), Official Journal July 24, 2026, in force July 27, 2026. CISA/NCSC, "Guidelines for Secure AI System Development," November 26, 2023. Veracode, *2025 GenAI Code Security Report*, and March 2026 update. Matt Palmer, CVE-2025-48757. Escape.tech, "State of Security of Vibe-Coded Apps." Tea breach reporting, July–August 2025. METR, July 10, 2025. Shen & Tamkin, arXiv:2601.20245 (2026). Budzyń et al., *The Lancet Gastroenterology & Hepatology*, August 2025. Brynjolfsson, Chandar & Chen, "Canaries in the Coal Mine?", Stanford Digital Economy Lab. Bainbridge, "Ironies of Automation," *Automatica* 19(6), 1983. Gimbel et al., The Budget Lab at Yale, October 1, 2025. Amodei, Axios, May 28, 2025; Altman, Sydney, May 26, 2026. Bastani et al., *PNAS*, 2025. Matt Garman, The Register, August 21, 2025. --- \pagebreak --- # RESEARCH UPDATE — AUGUST 31, 2026 The expanded edition incorporates research published or updated after several of the manuscript's original source checks. The following sources are especially important to the new material: - Erik Brynjolfsson, Bharat Chandar, and Ruyu Chen, Stanford Digital Economy Lab, *Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence*, revised August 12, 2026. The revision uses ADP payroll data through June 2026, reports no widespread economy-wide AI displacement, and estimates a 19% relative employment gap for workers ages 22–25 in highly AI-exposed occupations compared with less-exposed peers. It attributes the divergence primarily to reduced hiring and reports different patterns for automation-heavy versus augmentation-heavy occupations. - Lee C. Tucker, U.S. Census Bureau Center for Economic Studies, *You're (not) Hired: Artificial Intelligence and Early Career Hiring in the Quarterly Workforce Indicators*, CES Working Paper 26-27, April 2026. The paper reports a sizable decline in early-career hiring in highly AI-exposed industry-state cells and a 12% regression-adjusted employment decline for ages 22–24 in the most exposed quintile over the ten quarters following ChatGPT's introduction. - Judy Hanwen Shen and Alex Tamkin, *How AI Impacts Skill Formation*, 2026. Randomized experiments on developers learning an unfamiliar programming library found impaired conceptual understanding, code reading, and debugging under AI assistance on average, with outcomes varying substantially by how participants used the AI. - NIST, *Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile* (NIST AI 600-1), 2024, updated online in 2026; and the NIST AI Resource Center. These sources formalize AI risk-management practices and testing, evaluation, verification, and validation (TEVV). - Veracode, *2025 GenAI Code Security Report* and Spring 2026 update. In its benchmark, 45% of AI-generated code samples failed security tests; later reporting says syntax correctness exceeded 95% while security pass rates remained roughly 45–55%. These are vendor benchmark results and are presented as such, not as a universal failure rate for all AI-generated software. - Zhao et al., *Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks*, 2025 preprint. On the security-oriented SUSVIBES benchmark, one reported agent/model configuration produced 61% functionally correct solutions but 10.5% secure solutions. The manuscript treats this as benchmark evidence, not a universal estimate. - OWASP GenAI Security Project, *Top 10 for LLM and GenAI Applications 2025*, especially Improper Output Handling, Excessive Agency, and Misinformation. These categories provide a security framework for the transition from generated text to consequential downstream action. - Krzysztof Budzyń et al., *Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study*, *The Lancet Gastroenterology & Hepatology*, 2025. Unassisted adenoma-detection rate declined from 28.4% before AI exposure to 22.4% afterward in the observational periods studied. The manuscript continues to identify this as observational evidence rather than causal proof. - METR, Becker et al., *Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity*, 2025, plus METR's February 2026 methodology update. The original randomized trial found a 19% slowdown for 16 experienced developers across 246 tasks in familiar repositories. METR's later work suggests newer tools may be faster but explicitly warns that selection effects make the later speedup estimate weak evidence. The manuscript therefore treats the 19% result as a time- and setting-specific finding, not a permanent statement about AI coding tools. - Dell'Acqua et al., "Navigating the Jagged Technological Frontier," *Organization Science* (2026): preregistered experiment with 758 BCG consultants; AI improved speed, task completion, and quality on tasks inside the tested frontier but reduced correctness on an outside-frontier task. - Brynjolfsson, Li & Raymond, "Generative AI at Work," NBER: study of 5,179 customer-support agents; roughly 14% average productivity improvement and 34% improvement for novice/lower-skilled workers, with smaller effects for experienced workers. - "ChatGPT as a cognitive crutch: Evidence from a randomized controlled trial on knowledge retention," *Social Sciences & Humanities Open* (2025): 120 undergraduates; surprise 45-day retention test averaged 57.5% in the ChatGPT-assisted group versus 68.5% in the traditional-study group. --- # About the Author Anthony C. Vila is not an AI researcher, computer scientist, or academic. That is partly why he wrote this book. Vila came to artificial intelligence from the other side of the screen: as a user trying to build things with it. His background is in sales, entrepreneurship, and business building rather than software engineering. When generative AI made it possible for someone without a traditional technical background to create software, research markets, develop business systems, and tackle problems that once required specialized expertise, he became an aggressive adopter of the technology. Then he encountered the problem at the center of *Almost Right*: AI could produce work faster than he could independently verify it. As the systems became more capable, their mistakes became harder to distinguish from competent work. That question grew into an investigation spanning software development, medicine, law, aviation, education, labor economics, cybersecurity, and artificial intelligence itself. Vila does not argue that artificial intelligence should be stopped. He uses it. His argument is narrower: the more powerful our machines become, the more important it becomes to preserve independent human judgment—and the people capable of exercising it.