For this week\u2019s Open Questions column, Cal Newport is filling in for Joshua Rothman.<\/p>\n
Much of the euphoria and dread swirling around today\u2019s artificial-intelligence technologies can be traced back to January, 2020, when a team of researchers at OpenAI published a thirty-page report<\/a> titled \u201cScaling Laws for Neural Language Models.\u201d The team was led by the A.I. researcher Jared Kaplan, and included Dario Amodei, who is now the C.E.O. of Anthropic. They investigated a fairly nerdy question: What happens to the performance of language models when you increase their size and the intensity of their training?<\/p>\n Back then, many machine-learning experts thought that, after they had reached a certain size, language models would effectively start memorizing the answers to their training questions, which would make them less useful once deployed. But the OpenAI paper argued that these models would only get better as they grew, and indeed that such improvements might follow a power law\u2014an aggressive curve that resembles a hockey stick. The implication: if you keep building larger language models, and you train them on larger data sets, they\u2019ll start to get shockingly good. A few months after the paper, OpenAI seemed to validate the scaling law by releasing GPT-3, which was ten times larger\u2014and leaps and bounds better\u2014than its predecessor, GPT-2.<\/p>\n Suddenly, the theoretical idea of artificial general intelligence, which performs as well as or better than humans on a wide variety of tasks, seemed tantalizingly close. If the scaling law held, A.I. companies might achieve A.G.I. by pouring more money and computing power into language models. Within a year, Sam Altman<\/a>, the chief executive at OpenAI, published a blog post titled \u201cMoore\u2019s Law for Everything,\u201d which argued that A.I. will take over \u201cmore and more of the work that people now do\u201d and create unimaginable wealth for the owners of capital. \u201cThis technological revolution is unstoppable,\u201d he wrote. \u201cThe world will change so rapidly and drastically that an equally drastic change in policy will be needed to distribute this wealth and enable more people to pursue the life they want.\u201d<\/p>\n It\u2019s hard to overstate how completely the A.I. community came to believe that it would inevitably scale its way to A.G.I. In 2022, Gary Marcus, an A.I. entrepreneur and an emeritus professor of psychology and neural science at N.Y.U., pushed back on Kaplan\u2019s paper, noting that \u201cthe so-called scaling laws aren\u2019t universal laws like gravity but rather mere observations that might not hold forever.\u201d The negative response was fierce and swift. \u201cNo other essay I have ever written has been ridiculed by as many people, or as many famous people, from Sam Altman and Greg Brockton to Yann LeCun and Elon Musk,\u201d Marcus later reflected. He recently told me that his remarks essentially \u201cexcommunicated\u201d him from the world of machine learning. Soon, ChatGPT would reach a hundred million users faster than any digital service in history; in March, 2023, OpenAI\u2019s next release, GPT-4, vaulted so far up the scaling curve that it inspired a Microsoft research paper titled \u201cSparks of Artificial General Intelligence.\u201d Over the following year, venture-capital spending on A.I. jumped by eighty per cent.<\/p>\n After that, however, progress seemed to slow. OpenAI did not unveil a new blockbuster model for more than two years, instead focussing on specialized releases that became hard for the general public to follow. Some voices within the industry began to wonder if the A.I. scaling law was starting to falter. \u201cThe 2010s were the age of scaling, now we\u2019re back in the age of wonder and discovery once again,\u201d Ilya Sutskever, one of the company\u2019s founders, told Reuters in November. \u201cEveryone is looking for the next thing.\u201d A contemporaneous TechCrunch article summarized the general mood: \u201cEveryone now seems to be admitting you can\u2019t just use more compute and more data while pretraining large language models and expect them to turn into some sort of all-knowing digital god.\u201d But such observations were largely drowned out by the headline-generating rhetoric of other A.I. leaders. \u201cA.I. is starting to get better than humans at almost all intellectual tasks,\u201d Amodei recently told Anderson Cooper. In an interview with Axios, he predicted that half of entry-level white-collar jobs might be \u201cwiped out\u201d in the next one to five years. This summer, both Altman and Mark Zuckerberg<\/a>, of Meta, claimed that their companies were close to developing superintelligence.<\/p>\n Then, last week, OpenAI finally released GPT-5, which many had hoped would usher in the next significant leap in A.I. capabilities. Early reviewers found some features to like. When a popular tech YouTuber, Mrwhosetheboss, asked it to create a chess game that used Pok\u00e9mon as pieces, he got a significantly better result than when he used GPT-o4-mini-high, an industry-leading coding model; he also discovered that GPT-5 could write a more effective script for his YouTube channel than GPT-4o. Mrwhosetheboss was particularly enthusiastic that GPT-5 will automatically route queries to a model suited for the task, instead of requiring users to manually pick the model they want to try. Yet he also learned that GPT-4o was clearly more successful at generating a YouTube thumbnail and a birthday-party invitation\u2014and he had no trouble inducing GPT-5 to make up fake facts. Within hours, users began expressing disappointment with the new model on the r\/ChatGPT subreddit. One post called it the \u201cbiggest piece of garbage even as a paid user.\u201d In an Ask Me Anything (A.M.A.) session, Altman and other OpenAI engineers found themselves on the defensive, addressing complaints. Marcus summarized the release as \u201coverdue, overhyped and underwhelming.\u201d<\/p>\n In the aftermath of GPT-5\u2019s launch, it has become more difficult to take bombastic predictions about A.I. at face value, and the views of critics like Marcus seem increasingly moderate. Such voices argue that this technology is important, but not poised to drastically transform our lives. They challenge us to consider a different vision for the near-future\u2014one in which A.I. might not get much better than this.<\/p>\n OpenAI didn\u2019t want to wait nearly two and a half years to release GPT-5. According to The Information, by the spring of 2024, Altman was telling employees that their next major model, code-named Orion, would be significantly better than GPT-4. By the fall, however, it became clear that the results were disappointing. \u201cWhile Orion\u2019s performance ended up exceeding that of prior models,\u201d The Information reported in November, \u201cthe increase in quality was far smaller compared with the jump between GPT-3 and GPT-4.\u201d<\/p>\n Orion\u2019s failure helped cement the creeping fear within the industry that the A.I. scaling law wasn\u2019t a law after all. If building ever-bigger models was yielding diminishing returns, the tech companies would need a new strategy to strengthen their A.I. products. They soon settled on what could be described as \u201cpost-training improvements.\u201d The leading large language models all go through a process called pre-training in which they essentially digest the entire internet to become smart. But it is also possible to refine models later, to help them better make use of the knowledge and abilities they have absorbed. One post-training technique is to apply a machine-learning tool, reinforcement learning, to teach a pre-trained model to behave better on specific types of tasks. Another enables a model to spend more computing time generating responses to demanding queries.<\/p>\n A useful metaphor here is a car. Pre-training can be said to produce the vehicle; post-training soups it up. In the scaling-law paper, Kaplan and his co-authors predicted that as you expand the pre-training process you increase the power of the cars you produce; if GPT-3 was a sedan, GPT-4 was a sports car. Once this progression faltered, however, the industry turned its attention to helping the cars that they\u2019d already built to perform better. Post-training techniques turned engineers into mechanics.<\/p>\n Tech leaders were quick to express a hope that a post-training approach would improve their products as quickly as traditional scaling had. \u201cWe are seeing the emergence of a new scaling law,\u201d Satya Nadella, the C.E.O. of Microsoft, said at a conference last fall. The venture capitalist Anjney Midha similarly spoke of a \u201csecond era of scaling laws.\u201d In December, OpenAI released o1, which used post-training techniques to make the model better at step-by-step reasoning and at writing computer code. Soon the company had unveiled o3-mini, o3-mini-high, o4-mini, o4-mini-high, and o3-pro, each of which was souped up with a bespoke combination of post-training techniques.<\/p>\n