According to a famous story in the Talmud, the central text of Jewish law, Rabbi Eliezer was adamant that he was right about a certain legal question, but the other sages disagreed. So Rabbi Eliezer performed a bunch of miraculous feats intended to prove that God was on his side. He made a carob tree uproot itself and scurry away. He made a stream run backward. He made the walls of the study hall begin to cave in. Finally, he declared: If I\u2019m right, a voice from the heavens will prove it!<\/p>\n
What do you know? A heavenly voice came booming down to announce that Rabbi Eliezer was right. Still, the sages were unimpressed. Rabbi Yoshua insisted: \u201cThe Torah is not in heaven!\u201d In other words, when it comes to the law, it doesn\u2019t matter what any divine voice says \u2014 only what humans decide. Since a majority of sages disagreed with Rabbi Eliezer, he was overruled.<\/p>\n
Experts talk about aligning AI with human values. But \u201csolving alignment\u201d doesn\u2019t mean much if it yields AI that leads to the loss of human agency.True alignment would require grappling not just with technical problems, but with a major philosophical problem: Having the agency to make choices is a big part of how we create meaning, so building an AI that decides everything for us may rob us of the meaning of life.Philosopher of religion John Hick spoke about \u201cepistemic distance,\u201d the idea that God intentionally stays out of human affairs to a degree, so that we can be free to develop our own agency. Perhaps the same should hold true for an AI.<\/p>\n
Fast-forward 2,000 years and we\u2019re having essentially the same debate \u2014 just replace \u201cdivine voice\u201d with \u201cAI god.\u201d<\/p>\n
Today, the AI industry\u2019s biggest players aren\u2019t just trying to build a helpful chatbot, but a \u201csuperintelligence\u201d that is vastly smarter than humans and unimaginably powerful<\/a>. This shifts the goalposts from building a handy tool to building a god. When OpenAI CEO Sam Altman says he\u2019s making \u201cmagic intelligence in the sky<\/a>,\u201d he doesn\u2019t just have in mind ChatGPT as we know it today; he envisions<\/a> \u201cnearly-limitless intelligence\u201d that can achieve \u201cthe discovery of all of physics\u201d and then some. Some AI researchers hypothesize that superintelligence would end up making major decisions for humans \u2014 either acting autonomously or through humans that feel compelled to defer to its superior judgment.<\/p>\n As we work toward superintelligence, AI companies acknowledge, we\u2019ll need to solve the \u201calignment problem\u201d \u2014 how to get AI systems to reliably do what humans really want them to do, or align them with human values. But their commitment to solving that problem occludes a bigger issue.<\/p>\n Yes, we want companies to stop AIs from acting in harmful<\/a>, biased<\/a>, or deceitful<\/a> ways. But treating alignment as a technical problem isn\u2019t enough, especially as the industry\u2019s ambition shifts to building a god. That ambition requires us to ask: Even if we can somehow build an all-knowing, supremely powerful machine, and even if we can somehow align it with moral values so that it\u2019s also deeply good\u2026should we? Or is it just a bad idea to build an AI god \u2014 no matter how perfectly aligned it is on the technical level \u2014 because it would squeeze out space for human choice and thus render human life meaningless?<\/p>\n I asked Eliezer Yudkowsky and Yoshua Bengio whether they agree with their ancient namesakes. But before I tell you whether they think an AI god is desirable, we need to talk about a more basic question: Is it even possible?<\/p>\n Can you align superintelligent AI with human values?<\/p>\n God is supposed to be good \u2014 everyone knows that. But how do we make an AI good? That, nobody knows.<\/p>\n Early attempts at solving the alignment problem have been painfully simplistic. Companies like OpenAI and Anthropic tried to make their chatbots helpful and harmless, but didn\u2019t flesh out exactly what that\u2019s supposed to look like. Is it \u201chelpful\u201d or \u201charmful\u201d for a chatbot to, say, engage in endless hours of romantic roleplay<\/a> with a user? To facilitate cheating<\/a> on schoolwork? To offer free, but dubious, therapy<\/a> and ethical advice<\/a>?<\/p>\n Most AI engineers are not trained in moral philosophy, and they didn\u2019t understand how little they understood it. So they gave their chatbots only the most superficial sense of ethics \u2014 and soon, problems abounded, from bias and discrimination<\/a> to tragic suicides<\/a>.<\/p>\n But the truth is, there\u2019s no one clear understanding of the good, even among experts in ethics. Morality is notoriously contested: Philosophers have come up with many different moral theories, and despite arguing over them for millennia, there\u2019s still no consensus about which (if any) is the \u201cright\u201d one.<\/p>\n Even if all of humanity magically agreed on the same moral theory, we\u2019d still be stuck with a problem, because our view of what\u2019s moral shifts<\/a> over time, and sometimes it\u2019s actually good to break the rules. For example, we generally think it\u2019s right to follow society\u2019s laws, but when Rosa Parks illegally refused to give up her bus seat to a white passenger in 1955, it helped galvanize the civil rights movement \u2014 and we consider her action admirable. Context matters.<\/p>\n Plus, sometimes different kinds of moral good conflict with each other on a fundamental level. Think of a woman who faces a trade-off: She wants to become a nun but also wants to become a mother. What\u2019s the better decision? We can\u2019t say, because the options are incommensurable. There\u2019s no single yardstick by which to measure them so we can\u2019t compare them to find out which is greater.<\/p>\n \u201cProbably we are creating an AI that will systematically fall silent. But that\u2019s what we want.\u201d <\/p>\n \u2014 Ruth Chang, contemporary philosopher<\/p>\n Thankfully, some AI researchers are realizing that they have to give AIs a more complex, pluralistic picture of ethics<\/a> \u2014 one that recognizes that humans have many values and our values are often in tension with each other.<\/p>\n Some of the most sophisticated work on this is coming out of the Meaning Alignment Institute<\/a>, which researches how to align AI with what people value. When I asked co-lead Joe Edelman if he thinks aligning superintelligent AI with human values is possible, he didn\u2019t hesitate.<\/p>\n \u201cYes,\u201d he answered. But he added that an important part of that is training the AI to say \u201cI don\u2019t know\u201d in certain cases.<\/p>\n \u201cIf you\u2019re allowed to train the AI to do that, things get much easier, because in contentious situations, or situations of real moral confusion, you don\u2019t have to have an answer,\u201d Edelman said.<\/p>\n He cited the contemporary philosopher Ruth Chang, who has written about \u201chard choices<\/a>\u201d \u2014 choices that are genuinely hard because no best option exists, like the case of the woman who wants to become a nun but also wants to become a mother. When you face competing, incomparable goods like these, you can\u2019t \u201cdiscover\u201d which one is objectively best \u2014 you just have to choose which one you want to put your human agency behind.<\/p>\n \u201cIf you get [the AI] to know which are the hard choices, then you\u2019ve taught it something about morality,\u201d Edelman said. \u201cSo, that counts as alignment, right?\u201d<\/p>\n Well, to a degree. It\u2019s definitely better than an AI that doesn\u2019t understand there are choices where no best option exists. But so many of the most important moral choices involve values that are on a par. If we create a carve-out for those choices, are we really solving alignment in any meaningful sense? Or are we just creating an AI that will systematically fall silent on all the important stuff?<\/p>\n \u201cProbably we are creating an AI that will systematically fall silent,\u201d Chang said when I put the question to her directly. \u201cIt\u2019ll say \u2018Red flag, red flag, it\u2019s a hard choice \u2014 humans, you\u2019ve got to have input!\u2019 But that\u2019s what we want.\u201d The other possibility \u2014 empowering an AI to do a lot of our most important decision-making for us \u2014 strikes her as \u201ca terrible idea.\u201d<\/p>\n Contrast that with Yudkowsky. He\u2019s the arch-doomer of the AI world, and he has probably never been accused of being too optimistic. Yet he\u2019s actually surprisingly optimistic about alignment: He believes that aligning a superintelligence is possible in principle. He thinks it\u2019s an engineering problem we currently have no idea how to solve \u2014 but he still thinks that, at bottom, it\u2019s just an engineering problem. And once we solve it, we should put the superintelligence to broad use.<\/p>\n In his book, co-written with Nate Soares, he argues that we should be \u201caugmenting humans to make them smarter\u201d so they can figure out a better paradigm for building AI, one that would allow for true alignment. I asked him what he thinks would happen if we got enough super-smart and super-good people in a room and tasked them with building an aligned superintelligence.<\/p>\n \u201cProbably we all live happily ever after,\u201d Yudkowsky said.<\/p>\n In his ideal world, we would ask the people with augmented intelligence not to program their own values into an AI, but to build what Yudkowsky calls \u201ccoherent extrapolated volition\u201d \u2014 an AI that can peer into every living human\u2019s mind and extrapolate what we would want done if we knew everything the AI knew. (How would this work? Yudkowsky writes<\/a> that the superintelligence could have \u201ca complete readout of your brain-state\u201d \u2014 which sounds an awful lot like hand-wavy magic<\/a>.) It would then use this knowledge to basically run society for us.<\/p>\n I asked him if he\u2019d be comfortable with this superintelligence making decisions with major moral consequences, like whether to drop a bomb. \u201cI think I\u2019m broadly okay with it,\u201d Yudkowsky said, \u201cif 80 percent of humanity would be 80 percent coherent with respect to what they would want if they knew everything the superintelligence knew.\u201d In other words, if most of us are in favor of some action and we\u2019re in favor of it fairly strongly and consistently, then the AI should do that action.<\/p>\n A major problem with that, however, is that it could lead to a \u201ctyranny of the majority,\u201d where perfectly legitimate minority views get squeezed out. That\u2019s already a concern in modern democracies (though we\u2019ve developed mechanisms that partially address it, like embedding fundamental rights in constitutions that majorities can\u2019t easily override).<\/p>\n But an AI god would crank up the \u201ctyranny of the majority\u201d concern to the max, because it would potentially be making decisions for the entire global population, forevermore.<\/p>\n That\u2019s the picture of the future presented by influential philosopher Nick Bostrom, who was himself pulling on a larger set of ideas from the transhumanist tradition<\/a>. In his bestselling 2014 book, Superintelligence<\/a>, he imagined \u201ca machine superintelligence that will shape all of humanity\u2019s future.\u201d It could do everything from managing the economy to reshaping global politics to initiating an ongoing process of space colonization. Bostrom argued there would be advantages and disadvantages<\/a> to that setup, but one glaring issue is that the superintelligence could determine the shape of all human lives everywhere, and could enjoy a permanent concentration of power. If you didn\u2019t like its decisions, you would have no recourse, no escape. There would be nowhere left to run.<\/p>\n Obviously, if we build a system that\u2019s practically omniscient and omnipotent and it runs our civilization, that would pose an unprecedented threat to human autonomy. Which forces us to ask\u2026<\/p>\n Yudkowsky grew up in the Orthodox Jewish world, so I figured he might know the Talmud story about Rabbi Eliezer and Rabbi Yoshua. And, sure enough, he remembered it perfectly as soon as I brought it up.<\/p>\n I noted that the point of the story is that even if you\u2019ve got the most \u201caligned\u201d superintelligent adviser ever \u2014 a literal voice from God! \u2014 you shouldn\u2019t do whatever it tells you.<\/p>\n But Yudkowsky, true to his ancient namesake, made it clear that he wants a superintelligent AI. Once we figure out how to build it safely, he thinks we should absolutely build it, because it can help humanity resettle in another solar system before our sun dies and destroys our planet.<\/p>\n \u201cThere\u2019s literally nothing else our species can bet on in terms of how we eventually end up colonizing the galaxies,\u201d he told me.<\/p>\n Did he not worry about the point of the story \u2014 that preserving space for human agency is a crucial value, one we shouldn\u2019t be willing to sacrifice? He did, a bit. But he suggested that if a superintelligent AI could determine, using coherent extrapolated volition, that a majority of us would want a certain lab in North Korea blown up, then it should go ahead and destroy the lab \u2014 perhaps without informing us at all. \u201cMaybe the moral and ethical thing for a superintelligence to do is\u2026to be the silent divine intervention so that none of us are faced with the choice of whether or not to listen to the whispers of this voice that knows better than us,\u201d he said.<\/p>\n But not everyone wants an AI deciding for us how to manage our world. In fact, over 130,000 leading researchers and public figures recently signed a petition calling for a prohibition on the development of superintelligent AI<\/a>. The American public is broadly against it, too. According to polling from the Future of Life Institute (FLI)<\/a>, 64 percent feel that it should not be developed until it is proven safe and controllable, or should never be developed. Previous polling has shown that a majority of voters want regulation to actively prevent superintelligent AI<\/a>.<\/p>\n \u201cImagining an AI that figures everything out for us is like robbing us of the meaning of life.\u201d <\/p>\n \u2014 Joe Edelman, Meaning Alignment Institute co-lead<\/p>\n They worry about what could happen if the AI is misaligned (worst-case scenario: human extinction) but they also worry about what could happen even if the technical alignment problem is solved: militaries creating unprecedented surveillance and autonomous weapons; mass concentration of wealth and power in the hands of a few companies; mass unemployment; and the gradual replacement of human decision-making in all important areas.<\/p>\n As FLI\u2019s executive director Anthony Aguirre put it to me, even if you\u2019re not worried about AI presenting an existential risk, \u201cthere\u2019s still an existentialist risk.\u201d In other words, there\u2019s still a risk to our identity as meaning-makers.<\/p>\n