Timber wood In sports, as in healthcare and customer service, somebody eventually has to explain what happened. Preparation matters. getty
This week Anthropic disabled access to Fable 5 and Mythos 5 following a government directive citing national security concerns. Predictably, camps developed quickly. Some saw proof that stronger oversight is necessary. Others saw proof that governments cannot be trusted with technologies this important.
I suspect both sides missed something. What hit me was how quickly the conversation centered around control and how little time was spent discussing who was expected to absorb the consequences.
Everyone says they want guardrails. Everybody says they want responsible AI. But very few people seem particularly interested in what happens when those guardrails create friction, limit access or force uncomfortable tradeoffs. The situation reminded me of something much smaller. A few months ago, I noticed some meaningful errors in my medical record.
The existence of the error itself didn’t particularly surprise me. Healthcare is a human system layered on top of technology systems, and both humans and technology occasionally fail. The information was wrong, multiple people could see that it was wrong, and yet months later it still hasn’t been completely corrected.
What I still haven’t figured out is who actually owns the problem now that it exists. I’ve asked my physical therapist, largely because they’re the person I see most often. No real answers there. I have an appointment with the physician who created the original note next week, and we’ll see where that leads. Beyond that, I simply haven’t had enough time between work and traveling with my family to fully investigate how this process is supposed to work. It now feels less like a documentation error and more like an ownership problem. We Keep Returning To Replacement Part of the problem, I think, is that we continue to frame AI primarily through the lens of replacement.
Every conversation eventually seems to arrive there. Will radiologists disappear? Will programmers disappear? Will customer service representatives disappear? During a Senate committee hearing earlier this year, Senator Bernie Moreno walked through examples stretching back nearly a century. Cars were supposed to eliminate jobs. Automation was supposed to eliminate jobs.
The internet was supposed to eliminate jobs. I added that Excel spreadsheets were once forecast to eliminate accountants. The predictions never really played out the way people imagined, although work itself changed dramatically. I don’t say that because I think AI is overhyped. Quite the opposite; some of these capabilities are extraordinary.
But I increasingly wonder whether replacement is simply the wrong frame. Ironically, a surprising amount of the past decade of my career has involved telling people not to use AI. That sounds strange given the organizations I’ve worked with and the roles I’ve held, but many problems don’t actually require probabilistic systems. Sometimes good software is cheaper, easier to govern and easier to explain.
It also produces more consistent outcomes. Yet we’re increasingly approaching every problem as though AI must be involved simply because AI is available. Where Uncertainty Can Exist And Where It Can’t Consider the process of implementing clinical trials at an investigation site.
Protocols change constantly. Inclusion criteria evolve. New exclusions emerge. Amendments arrive. Research teams are asked to identify eligible patients while simultaneously ensuring that the protocol is followed exactly as written.
The instinctive answer is often to continuously feed every patient into AI and ask the model whether they qualify. A more interesting approach, at least to me, is almost boring. When the protocol changes, use AI once.
Let it interpret the update, compare it to the previous version, and translate complicated language into a structured ruleset. Have domain experts review that output and make sure it accurately reflects the protocol.
Then let conventional software do what conventional software has done extraordinarily well for decades and execute those rules consistently until the next update arrives. I increasingly suspect we’re allocating AI to the wrong layer.
Some of these systems are incredible. But if the protocol changes once a month, why would I invoke AI millions of times on millions of patients? Use the model ten times a year. Have somebody who actually understands the trial review the output.
Then push the rules into software and move on until the next update. One approach asks experts to inspect the uncertainty once.
The other spreads it around and quietly asks everybody downstream to absorb it. Clinicians. Patients.
Support teams. Those aren’t the same thing. Maybe that’s less exciting. It certainly makes for a worse keynote slide.
But when you’re the person who has to explain why Patient A got enrolled and Patient B didn’t eighteen months later, boring starts to look pretty attractive. Nobody gets bonus points because the application itself is AI. The Human May Not Be The Expensive Part One thing I wonder about is whether AI ends up partially reversing the offshoring story. For decades, organizations pursued labor arbitrage because it made economic sense. The assumption now seems to be that the next step is obvious: eliminate the person entirely. I’m not sure the math works that way.
The standard conversation focuses on how the AI economics themselves are evolving. Much of software spent decades charging for seats and subscriptions. AI introduces a different model. Usage matters. Tokens matter. Compute matters. AI folks use the term inference to describe the process by which models generate response.
As subsidies fade and utilization costs become more visible, companies are beginning to discover that inference isn’t free. But not every interaction carries the same stakes. Password resets and routine requests are one thing.
When customers are frustrated, frightened, confused or considering leaving, those become high-risk moments. Those are the moments customers remember.
The cost of failure isn’t measured by the first interaction. It’s measured by the second call, the escalation, the customer who quietly disappears or the relationship that never recovers. AI creates an interesting possibility.
Instead of replacing three people with nobody, perhaps one highly skilled person supported by AI can deliver similar economics while staying closer to the customer. Imagine that.
Rather than pursuing ever lower labor costs, maybe we can use these systems to build expertise and amplify trust. And maybe I’m wrong about that. Valuations are myths more than science anyway.
I once heard somebody describe them as a pitch deck with a really good presentation attached to it, and there’s probably some truth there. But the assumptions underneath many of these numbers implicitly depend on replacement. At some point the cost has to go somewhere. This still sounds strange to me every time I say it: in some situations, the human may actually be the most cost-effective inference layer.
Because the cost of intelligence isn’t measured solely in tokens and GPUs. Sometimes it’s the second call. Sometimes it’s the escalation. Sometimes it’s the customer who quietly disappears and never tells you why. Those costs eventually show up somewhere, even if they don’t appear on the cloud bill. Somebody Eventually Gets The Phone Call In reality, very little of this looks like the demos.
Most of it looks like redesigning workflows and trying to answer questions nobody had to answer before. Maybe that’s ultimately what made the Anthropic story so interesting to me. Everyone immediately argued about control. Governments.
Companies. Models. Access.
I found myself thinking about something much less grand. Clinical trials, customer service, healthcare: somebody eventually has to explain what happened. We’re becoming increasingly confident in the technology.
I’m less certain we’ve spent the same amount of time deciding who we’re comfortable asking to absorb the consequences when things don’t work exactly as planned.
Read More