![\"\"](\"https:\/\/substackcdn.com\/image\/fetch\/$s_!b_cj!,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3782662-bab9-4353-9206-8431fcda4598_1200x732.jpeg\")
<\/a><\/p>\nIn large part, what I prophesied has come true; we now have many GPT-4 level models (more than I foresaw); GPT-5 was disappointing, and didn\u2019t come in 2024; price wars; very little moat; no robust solution to hallucinations; a lot of corporate experimentation but not a ton of permanent inclusion in production. Modest profits was maybe a bit optimistic, with most LLM developers losing money.<\/p>\n
Nonetheless, a very large fraction of our economy is going into the world of price wars and little moat, massive, massive investments into companies that are all basically following the same strategy of investing more and more money into one single idea, viz. scaling ever larger language models, in hopes that something magical will emerge. <\/p>\n
Troublingly, the nation is now doubling down on these bets, still largely to the exclusion of developing other approaches, even as it has become clearer and clearer that problems of hallucinations and unreliability persist, and many are starting to report diminishing returns. <\/p>\n
\u00a7<\/p>\n
My own campaign against the single-minded narrowness of LLMs is not new (see e.g., the Scaling-uber-alles essay that launched this Substack in May 2022<\/a>), but others are now speaking out, as VentureBeat just reported<\/a>:<\/p>\n In a striking act of self-critique, one of the architects of the transformer technology that powers ChatGPT<\/a>, Claude<\/a>, and virtually every major AI system told an audience of industry leaders this week that artificial intelligence research has become dangerously narrow \u2014 and that he\u2019s moving on from his own creation.<\/p>\n Llion Jones<\/a>, who co-authored the seminal 2017 paper \u201cAttention Is All You Need<\/a>\u201c and even coined the name \u201ctransformer,\u201d delivered an unusually candid assessment at the TED AI conference<\/a> in San Francisco on Tuesday: Despite unprecedented investment<\/a> and talent flooding into AI, the field has calcified around a single architectural approach, potentially blinding researchers to the next major breakthrough.<\/p>\n \u201cDespite the fact that there\u2019s never been so much interest and resources and money and talent, this has somehow caused the narrowing of the research that we\u2019re doing,\u201d Jones told the audience. The culprit, he argued, is the \u201cimmense amount of pressure\u201d from investors demanding returns and researchers scrambling to stand out in an overcrowded field.<\/p>\n Yet investments in the same single approach grow every day.<\/p>\n \u00a7<\/p>\n Meanwhile we have entered an era, perhaps unprecedented, of extreme financial interdependence and circular investment:<\/p>\n