OpenAI chief executive Sam Altman \u2013 perhaps the most prominent face of the artificial intelligence (AI) boom that accelerated with the launch of ChatGPT in 2022 \u2013 loves scaling laws. <\/p>\n
These widely admired rules of thumb linking the size of an AI model with its capabilities inform much of the headlong rush among the AI industry to buy up powerful computer chips, build unimaginably large data centres, and re-open shuttered nuclear plants<\/a>.<\/p>\n As Altman argued in a blog post earlier this year<\/a>, the thinking is that the \u201cintelligence\u201d of an AI model \u201croughly equals the log of the resources used to train and run it\u201d \u2013 meaning you can steadily produce better performance<\/a> by exponentially increasing the scale of data and computing power involved.<\/p>\n First observed in 2020<\/a> and further refined<\/a> in 2022, the scaling laws for large language models (LLMs) come from drawing lines on charts of experimental data. For engineers, they give a simple formula that tells you how big to build the next model and what performance increase to expect.<\/p>\n Will the scaling laws keep on scaling as AI models get bigger and bigger? AI companies are betting hundreds of billions of dollars that they will \u2013 but history suggests it is not always so simple.<\/p>\n Scaling laws aren\u2019t just for AI<\/p>\n Scaling laws can be wonderful. Modern aerodynamics is built on them, for example.<\/p>\n Using an elegant piece of mathematics called the Buckingham \u03c0 theorem<\/a>, engineers discovered how to compare small models in wind tunnels or test basins with full-scale planes and ships by making sure some key numbers<\/a> matched up. <\/p>\n Those scaling ideas inform the design of almost everything that flies or floats, as well as industrial fans and pumps. <\/p>\n Another famous scaling idea underpinned the boom decades of the silicon chip revolution. Moore\u2019s law \u2013 the idea that the number of the tiny switches called transistors on a microchip would double every two years or so \u2013 helped designers create the small, powerful computing technology we have today.<\/p>\n But there\u2019s a catch: not all \u201cscaling laws\u201d are laws of nature. Some are purely mathematical and can hold indefinitely. Others are just lines fitted to data that work beautifully until you stray too far from the circumstances where they were measured or designed.<\/p>\n When scaling laws break down<\/p>\n History is littered with painful reminders of scaling laws that broke. A classic example is the collapse of the Tacoma Narrows Bridge<\/a> in 1940. <\/p>\n The bridge was designed by scaling up what had worked for smaller bridges to something longer and slimmer. Engineers assumed the same scaling arguments would hold: if a certain ratio of stiffness to bridge length worked before, it should work again. <\/p>\n Instead, moderate winds set off an unexpected instability called aeroelastic flutter. The bridge deck tore itself apart, collapsing just four months after opening.<\/p>\n Likewise, even the \u201claws\u201d of microchip manufacturing had an expiry date. For decades, Moore\u2019s law (transistor counts doubling every couple of years) and Dennard scaling (a larger number of smaller transistors running faster while using the same amount of power) were astonishingly reliable guides for chip design and industry roadmaps. <\/p>\n As transistors became small enough to be measured in nanometres, however, those neat scaling rules began to collide<\/a> with hard physical limits.<\/p>\n When transistor gates shrank to just a few atoms thick, they started leaking current and behaving unpredictably. The operating voltages could also no longer be reduced with being lost in background noise. <\/p>\n Eventually, shrinking was no longer the way forward. Chips have still grown more powerful, but now through new designs rather than just scaling down.<\/p>\n Laws of nature or rules of thumb?<\/p>\n The language-model scaling curves that Altman celebrates are real, and so far they\u2019ve been extraordinarily useful. <\/p>\n They told researchers that models would keep getting better if you fed them enough data and computing power. They also showed earlier systems were not fundamentally limited<\/a> \u2013 they just hadn\u2019t had enough resources thrown at them.<\/p>\n But these are undoubtedly curves that have been fit to data. They are less like the derived mathematical scaling laws used in aerodynamics and more like the useful rules of thumb used in microchip design \u2013 and that means they likely won\u2019t work forever.<\/p>\n The language model scaling rules don\u2019t necessarily encode real-world problems such as limits to the availability of high-quality data for training, or the difficulty of getting AI to deal with novel tasks \u2013 let alone safety constraints or the economic difficulties of building data centres and power grids. There is no law of nature or theorem guaranteeing that \u201cintelligence scales\u201d forever.<\/p>\n Investing in the curves<\/p>\n So far, the scaling curves for AI look pretty smooth \u2013 but the financial curves are a different story.<\/p>\n