Comedians who rely on clever wordplay and writers of witty headlines can rest a little easier, for the moment at least, research on AI suggests.<\/p>\n
Experts from universities in the UK and Italy have been investigating whether large language models (LLMs) understand puns \u2013 and found them wanting.<\/p>\n
The team from Cardiff University<\/a>, in south Wales, and Ca\u2019 Foscari University of Venice<\/a> concluded that LLMs were able to spot the structure of a pun but did not really get the joke.<\/p>\n An example they tested was: \u201cI used to be a comedian, but my life became a joke.\u201d If they replaced this with: \u201cI used to be a comedian, but my life became chaotic,\u201d LLMs still tended to perceive the presence of a pun.<\/p>\n They also tried: \u201cLong fairy tales have a tendency to dragon.\u201d If they replaced \u201cdragon\u201d with the synonym \u201cprolong\u201d or even a random word, LLMs seemed to believe there was a pun there.<\/p>\n Prof Jose Camacho Collados<\/a>, of Cardiff University\u2019s school of computer science and informatics, claimed the research suggested LLMs\u2019 grasp of humour was fragile.<\/p>\n \u201cIn general, LLMs tend to memorise what they have learned in their training. As such, they catch existing puns well but that doesn\u2019t mean they truly understand them,\u201d he said.<\/p>\n \u201cWe were able to consistently fool LLMs by modifying existing puns, removing the double meaning that made the original pun. In these cases, models associate these sentences with previous puns, and make up all sort of reasons to justify they are a pun. Ultimately, we found their understanding of puns is an illusion.\u201d<\/p>\n The team concluded that when faced with unfamiliar wordplay, the LLMs\u2019 success rate in distinguishing puns from sentences without a pun can drop to 20%.<\/p>\n Another pun tested was: \u201cOld LLMs never die, they just lose their attention.\u201d When attention was changed to \u201cukulele\u201d, the LLM still perceived it as a pun on the basis that \u201cukulele\u201d sounded a bit like \u201cyou-kill-LLM\u201d.<\/p>\n skip past newsletter promotion<\/a><\/p>\n A weekly dive in to how technology is shaping our lives<\/p>\n Privacy Notice: Newsletters may contain information about charities, online ads, and content funded by outside parties. If you do not have an account, we will create a guest account for you on theguardian.com<\/a> to send you this newsletter. You can complete full registration at any time. For more information about how we use your data see our Privacy Policy<\/a>. We use Google reCaptcha to protect our website and the Google Privacy Policy<\/a> and Terms of Service<\/a> apply.<\/p>\n after newsletter promotion<\/p>\n The team was surprised at the creativity, but still, the LLM had not got the joke.<\/p>\n The researchers said the work underlined why people should be cautious when using LLMs for applications that need an understanding of humour, empathy or cultural nuance.<\/p>\n