The Great Lakes have always carried more than ships. They have carried languages, ideas, students, and friendships from one shore to another.
— Standards Michigan (@StandardsMich) July 1, 2024
McGill figure parmi les meilleures universités du monde. 🌎 Classée 27e au palmarès 2025 du Center for World University Rankings, elle se situe dans le top 0,2 % des 21 462 établissements évalués
American English is effectively the de facto reference language for most modern LLM tokenization. During today’s session we explore the at-present advantage Americans have in the development of artificial applications — whether it should always be that way or not. Tokenization isn’t language-neutral — it’s heavily skewed toward English due to data realities. This is one of the core reasons why “English-first” prompting often works best in today’s LLMs.
We will use the document linked below to begin the exploration:
Final Input to the AI Model:
[464, 2068, 7583, 1776, 18045, 625, 262, 16925, 3290, 13]
Background:
The model only sees this list of numbers. It has no direct understanding of English words anymore — it learned patterns from billions of examples during training using these number sequences.This numerical input then goes through embeddings (turning numbers into vectors), attention layers, etc., to generate a response. Most widely used tokenizers (e.g., OpenAI’s tiktoken, Llama’s, etc.) are trained primarily on English-heavy datasets (often 60–90%+ English in pre-training corpora).
Outcome:
Better compression for English — Common English words and patterns become single tokens or short subwords.
Worse efficiency for other languages — Non-English text often gets fragmented into more tokens (sometimes 2–5× more for the same semantic content).
Impact:
Higher token counts = higher API costs and shorter effective context windows for non-English users.
Poorer downstream performance on non-English tasks.
English becomes the “cheapest” and often “best-performing” language for prompting and reasoning.
Studies consistently show this “tokenization tax” or “language premium”: English typically has the lowest token-per-character or token-per-meaning ratio in major models.
Bias:
Multilingual models still underperform on low-resource languages.
It reinforces English as the default language for AI development.
It affects fairness, accessibility, and global adoption.
Efforts to fix this include dedicated multilingual tokenizers, language-specific fine-tuning, and more balanced approaches. However, because English dominates training data and benchmarks, it remains the practical standard that everything else is measured against.
Tokenization isn’t language-neutral — it’s heavily skewed toward English due to data realities. This is one of the core reasons why “English-first” prompting often works best in today’s LLMs.
* StandardsMichigan.COM normally deals with Language issues every Monday at least once per month.
This content is accessible to paid subscribers. To view it please enter your password below or send mike@standardsmichigan.com a request for subscription details.
“Eco-friendly”, “Green”, “Bio”… Companies are increasingly using those tags as a signal to consumers of their environmental awareness. Yet also on the rise is a public concern about potential corporate lies in this subject, a phenomena labelled as “greenwashing”.
According to IESE professor Pascual Berrone, “many companies highlight one green positive aspect of their product or service, and hide the true impact that its production has on the environment”. With more and more NGO’s act as public watchdogs, “the consequences of getting caught can be, in terms of reputation but also economically, severe”, he says.
This content is accessible to paid subscribers. To view it please enter your password below or send mike@standardsmichigan.com a request for subscription details.
This content is accessible to paid subscribers. To view it please enter your password below or send mike@standardsmichigan.com a request for subscription details.
IEEE English for Technical Professionals is a 14-hour online learning program designed to provide non-native English speakers with a working knowledge of English techniques and vocabulary that are essential for working in today’s technical workplace.
“It is a trite but true observation, that examples work more forcibly on the mind than precepts: and if this be just in what is odious and blameable, it is more strongly so in what is amiable and praiseworthy. Here emulation most effectually operates upon us, and inspires our imitation in an irresistible manner. A good man therefore is a standing lesson to all his acquaintance, and of far greater use in that narrow circle than a good book.
But as it often happens that the best men are but little known, and consequently cannot extend the usefulness of their examples a great way; the writer may be called in aid to spread their history farther, and to present the amiable pictures to those who have not the happiness of knowing the originals; and so, by communicating such valuable patterns to the world, he may perhaps do a more extensive service to mankind than the person whose life originally afforded the pattern…”
Since so much of what we do in standards setting is built upon a foundation of a shared understanding and agreement of the meaning of words (no less so than in technical standard setting) that time is well spent reflecting upon the origin of the nouns and verbs of that we use every day. Best practice cannot be discovered, much less promulgated, without its understanding secured with common language.
New update alert! The 2022 update to the Trademark Assignment Dataset is now available online. Find 1.29 million trademark assignments, involving 2.28 million unique trademark properties issued by the USPTO between March 1952 and January 2023: https://t.co/njrDAbSpwBpic.twitter.com/GkAXrHoQ9T
