The same way as using non-English makes it harder to find any data manually in books/forums/etc(except for strictly local data like laws or similar).
This was mostly noticed already in the earliest days of LLM spreading (I think several months since GPT went public), when Russian users noticed it has much worse output in Russian than English. The reason was simply that each letter was treated as a whole distinct token in Russian, leading to more difficulties in outputing words without typos. It is logical that this approach would be more resource-expensive and prone to errors, if the AI has to select each and every letter one by one instead of immediately jumping to selecting words and forming the whole sentence.
English is a fairly analytic language, words barely change, and subtleties in meaning and relationships are expressed with other words. Many other languages encode such information inside the word by various ways of modifying the same base form (inflectional languages). The same way English forms plurals by adding -s, so can other languages add a whole bunch of possible endings to express e.g. that the object is used as a tool, or that it’s the recipient of an action. To analyse and tokenise all that stuff would require decades of manual work. Alternatively I would assume there are some automatised tools that would generally correctly split the words into tokens, at least for the major inflectional languages, such as Russian. But most likely they just split it down to random syllables or even individual letters. Indeed, I just tested Russian, Polish and Croatian in their linked tokenizer, and the results are mostly syllabic or letter-based or just sliced into random sections that don’t correspond to anything linguistically relevant. Not a trace of morphemic analysis (morphemes are meaningful units, e.g. level-head-ed-ness, this style of slicing words down into pieces would probably be the ideal approach here). [Edit: holy shit, it actually seems to treat almost every capital letter as a distinct token.]
Or, to put it most simply, AI bullshits and brute-forces through languages with complicated words. And it’s been known for years. Nice to see additional confirmation, of course, and to learn that they still haven’t improved on it.
When you realize English is such a terrible language that even AI can’t comprehend it.
Sorry, isn’t that the opposite of the point made by this post?
I mean, kind of, yeah. If you just read the headline, then my comment does seem contradictory. ‘Some’ languages are definitely less optimal for LLMs than English.
But this is nothing new, and if you read about it before, you’d know there are many that are more optimal for LLMs, i.e., use fewer tokens for the exact same conversation content. Including languages that have just a fraction of the English language’s native speaker counts.
The article in general isn’t very good, because it provides just one concrete example - English vs Malayam. The rest is pretty much extrapolating that one case to all languages of the world.
It keeps reiterating that ‘Non-English languages’ are at a disadvantage, without any data to back this claim, other than a link to the study it’s about, which is 10x longer than the already lengthy article.
The study itself focuses on the difference between the best and worst case scenarios, which is perfectly fine, but does not exactly present whether LLMs work better with English or not.
Silicon Valley built a universal intelligence that apparently charges extra for speaking most of the world’s languages. And the surcharge doesn’t even buy better answers: it buys slower responses and less context. Truly revolutionary: exporting the English only customer service experience to the entire planet.
Sure, but the Chinese and other models in the source study were no better than the US ones.
