AI and ML<\/p>\n
Token consumption doesn’t tell the whole tale but it shouldn’t be ignored<\/p>\n
Claude looks substantially more token-hungry than OpenAI’s GPT-5.x, thanks to the new tokenizer that Anthropic shipped with recent releases.<\/p>\n
Large language models (LLMs) use tokenizers to handle the mapping of text into tokens<\/a>. There’s no set definition of a token, but they’re typically a set of three or four characters that are mapped to the integers LLMs actually process.\u00a0<\/p>\n Tokens have become the basic economic unit for billing use of AI models. Because the slicing of words into tokens and the tokens required per task vary across models, it has become rather difficult to predict the final bill for playing the AI slot machine.<\/p>\n Recent changes to Anthropic’s tokenizer appear to have further complicated matters by making the same content more costly to process on certain models.<\/p>\n Playcode, an AI app building platform, recently analyzed<\/a> the impact of Anthropic’s latest tokenizer and found that the same TypeScript file processed by Claude can consume up to 73 percent more tokens than OpenAI’s GPT-5.x model family.<\/p>\n Anthropic acknowledges its new tokenizer \u2013 announced<\/a>\u00a0at the end of June when Sonnet 5 shipped \u2013 may generate more tokens for the same input than prior versions.<\/p>\n “Sonnet 5 is an upgrade to Sonnet 4.6, but it uses an updated tokenizer that changes how the model processes text to improve performance (this is similar to the tokenizer change we introduced with Claude Opus 4.7),” the company explained. “The tradeoff is that the same input can map to more tokens: roughly 1.0\u20131.35\u00d7 depending on the content type.”<\/p>\n Anthropic offered Sonnet at a reduced introductory rate \u2013 $2\/million input tokens and $10\/million output tokens through August 31, 2026 \u2013 to make the inflated token generation more or less cost-neutral. But the price is set to rise to $3\/million and $15\/millionafter that.<\/p>\n Anthropic did not immediately respond to a request for comment. The company says that users of its new tokenizer may see their bills rise by as much as a third compared to the tokenizer it used\u00a0with its older models. Costs for Anthropic users could go even higher when compared against the latest iteration of OpenAI’s o200k<\/a> <\/a>tokenizer<\/a>.<\/p>\n Playcode’s cross-vendor token comparison finds that for a 2,888 character TypeScript file, Claude’s new tokenizer emits 1.73x more tokens than GPT-5.x’s tokenizer and 1.32x more than Claude’s old tokenizer. These figures differ for different types of code: Rust taps in at 1.58x, JavaScript 1.52x, and Python 1.50x.<\/p>\n So if Anthropic’s list prices were adjusted to be comparable with OpenAI’s GPT-5.x baseline, Playcode suggests Opus 4.8’s cost would be $7.50\/M input and $37.50\/M output instead of the published figure of $5\/M and $25\/M.<\/p>\n The AI app platform notes that a team from marketing platform Ploy this week published an account of a production migration using OpenAI’s GPT-5.6 Sol and Anthropic’s Opus 4.8.<\/p>\n “GPT-5.6 finished pages 2.2\u00d7 faster, cost 27 percent less, and used about half the output tokens,” Ploy cla<\/a>i<\/a>med<\/a>.\u00a0<\/p>\n