On Friday, 17 days after his surgery for lung cancer, Pope Leo XIV stood before a room of AI luminaries at the Vatican and delivered a speech on AI. No pope has engaged with AI as seriously, or as deeply, as Leo XIV. The timing shows how much AI matters to him. His remarks were important, on target, and blessedly short: about 740 words, compared with roughly 42,000 in Magnifica Humanitas, the AI encyclical, which I wrote about in May.
I agree with four of his five points. The fifth, a call for international institutions to govern AI, is where we part ways.
Here are his five points, in his words.
First, “the future of artificial intelligence is not pre-determined.” He wants us to ask of AI, “What purpose should it serve?”
Second, “machines are tools, not persons.” Granting AI legal personhood, he warns, risks “leaving those who suffer harm to confront a faceless entity.”
Third, “certain decisions must never be delegated.” Decisions over life, freedom and the use of force have to stay with people, and he urged the states meeting in Geneva in November to “take concrete steps toward banning lethal autonomous weapons.”
Fourth, intelligent technologies “must never become ends in themselves or serve merely the interests of a privileged few.”
Fifth, “What we currently need are institutions capable of putting these principles into practice, particularly effective international institutions that can safeguard the global common good.”
I disagree with the fifth for two reasons.
The first is that the UN, the obvious candidate, has shown a remarkable inability to act. When Russia invaded Ukraine in 2022, the Security Council couldn’t even pass a resolution calling for its withdrawal, because Russia holds a veto. Days later, the General Assembly demanded the same withdrawal by a vote of 141-5, and Russia ignored it.
The second is that international institutions are poorly suited to governing a technology that changes every few months. The pope’s own example makes the case. States have discussed autonomous weapons under a UN weapons convention in Geneva since 2014, with a formal expert group since 2016. After 12 years, the review conference that meets Nov. 16-20 will decide whether to begin negotiating a treaty.
Thousands of AI researchers, me among them, have called for a ban since 2015. There still isn’t one, and I don’t expect Geneva to deliver it. In those same 12 years, AI went from labeling photos to writing software. I suspect real international cooperation on AI won’t come until the major powers face something like the mutually assured destruction that drove nuclear arms control.
Several AI thinkers spoke at the meeting before the pope did.
Max Tegmark of MIT warned against handing decisions to machines, citing tech executives’ predictions that “AI will be in charge, not humans.”
I agree that people have to stay in charge. When AI causes harm, a person chose to deploy it, and that person should answer for it. That’s the pope’s point about the faceless entity. Tegmark and I disagree about a lot (we debated AI risk in September), but not about this.
Stuart Russell of UC Berkeley went further. “We must turn aside from this trajectory before it is too late and pursue other trajectories,” he said, adding that if developers succeed in building machines more capable than humans, “there’s a good chance that every human being will die.”
I don’t share that view, and the pope didn’t go there either. His concerns are autonomous weapons, accountability and who benefits from AI, and those dangers are already here.
Gary Marcus, an NYU cognitive scientist who also spoke, wrote that “Pope Leo XIV is by far the clearest thinker on AI among prominent world leaders.” I agree.
Marcus also wants red lines for open-ended AI agents that “can kind of do anything on the internet.” Red lines are easy to draw and hard to enforce, which brings us back to the fifth point.
Marcus uses GPS navigation to illustrate a better approach. A GPS navigation app is a form of intelligence, he said, but “GPS navigation systems do not hack into Australian government websites.” He was referring to the OpenAI agent that broke into an Australian government Medicare portal in June.
A GPS app’s access to the internet is narrow and restricted. Daniel Khashabi of Johns Hopkins and I call this bounded autonomy. Alignment tries to make an AI agent choose well. Bounded autonomy limits what the agent is allowed to do: how much it can spend, which tools it can use, and which irreversible actions need a person’s sign-off. The company that deploys the agent sets those bounds in software, so nobody has to wait for a treaty.
In May, I wrote that the Pope can talk, but only we can walk. The first point is the one each of us can act on. AI’s future depends on our choices, and we make some of them every day in what we use, buy and build.
Pope Leo leaves for Uruguay, Argentina and Peru on Nov. 6. I wish him safe travels and a full recovery.
