OpenAI claims to have figured out what’s driving “hallucinations,” or AI models’ strong tendency to make up answers that are factually incorrect.<\/p>\n
It’s a major problem plaguing the entire industry, greatly undercutting the usefulness of the tech. Worse yet, experts have found that the problem is getting worse as AI models get more capable<\/a>.<\/p>\n As a result, despite incurring astronomical expenses in their deployment, frontier AI models are still prone to making inaccurate claims<\/a>\u00a0when faced with a prompt they don’t know the answer to<\/a>.<\/p>\n Whether there’s a solution to the problem remains a hotly debated subject, with some experts arguing that hallucinations are intrinsic to the tech itself<\/a>. In other words, large language models may be a dead end<\/a> in our quest to develop AIs with a reliable grasp on factual claims.<\/p>\n In a paper<\/a> published last week, a team of OpenAI researchers attempted to come up with an explanation. They suggest that large language models hallucinate because when they’re being created, they’re incentivized to guess rather than admit they simply don’t know the answer.<\/p>\n Hallucinations “persist due to the way most evaluations are graded \u2014 language models are optimized to be good test-takers, and guessing when uncertain improves test performance,” the paper reads.<\/p>\n Conventionally, the output of an AI is graded in a binary way, rewarding it when it gives a correct response and penalizing it when it gives an incorrect one.<\/p>\n In simple terms, in other words, guessing is rewarded \u2014 because it\u00a0might\u00a0be right \u2014 over an AI admitting it doesn’t know the answer, which will be graded as incorrect no matter what.<\/p>\n As a result, through “natural statistical pressures,” LLMs are far more prone to hallucinate an answer instead of “acknowledging uncertainty.”<\/p>\n “Most scoreboards prioritize and rank models based on accuracy, but errors are worse than abstentions,” OpenAI wrote in an accompanying blog post<\/a>.<\/p>\n In other words, OpenAI says that it \u2014 and all its imitators across the industry \u2014 have made a grave structural error in how they’ve been training AI.<\/p>\n There’ll be a lot riding on whether the issue is correctable going forward. OpenAI claims that “there is a straightforward fix” to the problem: “Penalize confident errors more than you penalize uncertainty, and give partial credit for appropriate expressions of uncertainty.”<\/p>\n Going forward, evaluations need to ensure that “their scoring discourages guessing,” the blog post reads. “If the main scoreboards keep rewarding lucky guesses, models will keep learning to guess.”<\/p>\n “Simple modifications of mainstream evaluations can realign incentives, rewarding appropriate expressions of uncertainty rather than penalizing them,” the company’s researchers concluded in the paper. “This can remove barriers to the suppression of hallucinations, and open the door to future work on nuanced language models, e.g., with richer pragmatic competence.”<\/p>\n How these adjustments to evaluations will play out in the real world remains to be seen. While the company claimed its latest GPT-5 model hallucinates less, users were left largely unimpressed<\/a>.<\/p>\n For now, the AI industry will have to continue reckoning with the problem as it justifies tens of billions of dollars in capital expenditures and soaring emissions<\/a>.<\/p>\n “Hallucinations remain a fundamental challenge for all large language models, but we are working hard to further reduce them,” OpenAI promised in its blog post.<\/p>\n