On Monday, Pope Leo XIV did something no pope had done before. He stood in the Vatican’s Synod Hall and personally presented an encyclical to the world. The document, Magnifica Humanitas, runs to 42,000 words and carries the subtitle “On Safeguarding the Human Person in the Time of Artificial Intelligence.” Seated among the cardinals and theologians was a 33-year-old atheist named Chris Olah, co-founder of Anthropic, the company behind Claude. An unlikely pairing, and that is precisely what makes it worth your attention.

I want to sit with the strangeness of this for a moment, because I think the technology press largely missed the point. The headlines fixated on the optics: a tech billionaire sharing a stage with the pontiff, the church “legitimising” Silicon Valley, that sort of thing. Tristan Harris, of The Social Dilemma fame, was in Rome and called the unease about Anthropic’s presence a “valid concern.” Fair enough. But the more interesting thing is what was actually said, and the uncomfortable historical rhyme underneath it all.

What the encyclical actually argues

Leo did not pull his punches, which surprised me. With Anthropic in the room, he described AI as a potential “instrument of domination, exclusion and death.” He spent a good chunk of the text attacking the concentration of power and data in the hands of a very small number of companies and nations. He called for robust regulation, insisted that lethal decisions must never be delegated to machines, and argued that developers should serve the common good rather than profit. He warned about the displacement of human labour and the erosion of human dignity.

This is not a benediction of the AI industry. It is closer to a charge sheet. And the genuinely odd part is that Anthropic helped unveil it, with Olah standing there nodding along.

Olah’s own remarks were more candid than I expected from a frontier lab founder. He opened by admitting that every AI lab, his own included, “operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing.” He named them plainly: commercial pressure, the race to stay at the research frontier, geopolitical pressure, and the older human failings of pride and ambition. His conclusion was that the people building this stuff cannot be trusted to police themselves, and that the world needs “moral voices that the incentives cannot bend.”

That is a remarkable thing to say out loud. It is also, if you read it cynically, exactly what you would say if you wanted the church’s moral authority to provide cover for a commercial enterprise. Both readings can be true at once. I lean towards taking it at face value, because the more I look at the situation Anthropic is actually in, the more I think the man might genuinely mean it.

The part everyone is dancing around

Here is the context that turns this from a feel-good interfaith moment into something with real teeth. Anthropic is currently in open conflict with the Trump administration over military and surveillance use of its models. The Pentagon responded by labelling Anthropic a supply chain risk, the first time that designation has ever been applied to a US company, and blocked its work with government contractors. OpenAI promptly signed a Defense Department contract in the space Anthropic vacated.

Read that again. A company drew a line about how its technology could be used in warfare and surveillance, paid a direct commercial and political price for it, and watched a competitor walk straight through the door it had just closed.

And that is where the Oppenheimer question stops being a lazy comparison and starts being the actual story.

The Oppenheimer rhyme

J. Robert Oppenheimer is the obvious reach for anyone writing about powerful technology and regret, and I was wary of using him precisely because he is overused. But the parallel here is sharper than the usual hand-wringing, and it is worth being precise about why.

Oppenheimer’s tragedy was not that he built the bomb and then felt bad about it. The tragedy was structural. He built it inside a system of incentives, wartime urgency, national competition, institutional momentum, that made the building feel not just justified but inevitable. The horror came afterwards, when he realised that the thing he had made had its own logic, and that logic was an arms race he could not stop. His famous line to Truman, that he felt he had blood on his hands, earned him not sympathy but contempt. Truman called him a crybaby and refused to see him again. The system Oppenheimer helped create then spent the next decade chewing him up, stripping his security clearance, sidelining him for the very moral hesitancy he is now remembered for.

The lesson is not “powerful technology is dangerous.” Everyone knows that. The lesson is that individual conscience is structurally weak against a competitive dynamic. Oppenheimer’s misgivings changed nothing about the trajectory of nuclear weapons, because the moment the technology was demonstrated to be possible, the race was on regardless of what any single physicist felt.

Now look at Olah’s speech again. He is, in effect, describing the same trap in advance. He is saying: the incentives will bend us, no matter how sincere we are, so we need an external force the incentives cannot reach. He is trying to build the institutional brake that Oppenheimer never had. The Pentagon episode is the demonstration that the brake is needed, because it shows what happens when one actor exercises restraint in a competitive field. Restraint becomes a competitive disadvantage that someone else immediately exploits.

This is the genuinely hard problem, and it is one I recognise from a much smaller scale in my own world. Anyone who has worked in platform engineering has felt the pull of the same dynamic. You know the responsible thing is to slow down, to fix the foundations, to refuse the architecture that will bite you in two years. And then a competitor ships the irresponsible version, the market rewards them for it, and your principled caution looks like incompetence on the quarterly review. I have lost that argument more than once. I have also, to my discredit, won it by simply shipping the thing I knew was wrong because the alternative was being the person who blocked delivery. The difference between my version and the AI version is only one of stakes. The structure is identical.

So which is it

The cynical reading says Anthropic is buying indulgences. Stand next to the Pope, absorb some moral legitimacy, and carry on doing whatever the market demands while pointing at the photo op as proof of good faith. There is a real risk of exactly this, and the people raising it are not fools.

But I do not think that is the most likely reading, and here is why. Buying indulgences is cheap and comfortable. What Anthropic actually did was invite the most credible external critic it could find, hand that critic a 42,000-word platform to call AI an instrument of domination, and stand there while it happened. You do not do that if you want cover. You do that if you have concluded, correctly, that your own conscience is not enough, and that you need someone outside the incentive structure to hold the line you cannot reliably hold yourself.

Whether it works is another matter entirely. Oppenheimer’s regret was sincere and it changed nothing. The church’s voice may turn out to be just as structurally weak against the same competitive logic. A moral authority with no enforcement mechanism is a conscience, not a brake, and we have just established that conscience loses.

The honest answer is that we are watching people who can see the trap closing try to wedge something into the gap before it shuts. They may be too late, or too weak, or both. But the alternative, the people building this technology pretending the trap does not exist, is demonstrably worse. At least someone in that room on Monday was willing to say out loud that they cannot be trusted alone. Oppenheimer only worked that out afterwards.