Author Archives: mike@standardsmichigan.com

Standards Michigan: Language*

American English is effectively the de facto reference language for most modern LLM tokenization.   During today’s session we explore the at-present advantage Americans have in the development of artificial applications — whether it should always be that way or not.  Tokenization isn’t language-neutral — it’s heavily skewed toward English due to data realities. This is one of the core reasons why “English-first” prompting often works best in today’s LLMs.

We will use the document linked below to begin the exploration:

NIST AI Consortium

Use the login credentials at the upper right of our home page

Background:

Outcome:

• • Better compression for English — Common English words and patterns become single tokens or short subwords.
  • Worse efficiency for other languages — Non-English text often gets fragmented into more tokens (sometimes 2–5× more for the same semantic content).

Impact:

• • Higher token counts = higher API costs and shorter effective context windows for non-English users.
  • Poorer downstream performance on non-English tasks.
  • English becomes the “cheapest” and often “best-performing” language for prompting and reasoning.

Studies consistently show this “tokenization tax” or “language premium”: English typically has the lowest token-per-character or token-per-meaning ratio in major models.

Bias:

• • Multilingual models still underperform on low-resource languages.
  • It reinforces English as the default language for AI development.
  • It affects fairness, accessibility, and global adoption.

Efforts to fix this include dedicated multilingual tokenizers, language-specific fine-tuning, and more balanced approaches. However, because English dominates training data and benchmarks, it remains the practical standard that everything else is measured against.

Tokenization isn’t language-neutral — it’s heavily skewed toward English due to data realities. This is one of the core reasons why “English-first” prompting often works best in today’s LLMs.

* StandardsMichigan.COM normally deals with Language issues every Monday at least once per month.

How Your LLM Costs 5X More If You Don’t Speak English

Same content, 65% more expensive in Chinese! Cross-model tokenization comparison: Claude users pay the highest ‘Chinese tax’

Starting from the canonicals of any standard suitable for optimizing innovation (and incorporation by reference into public law) — Title, Scope, Purpose and Definitions — we will turn our attention to the Call for Public Comment by NIST which is open until July 28th.

Innovation and Competitiveness in Artificial Intelligence

Artificial intelligence is built upon language, but the meaning of “standard language” becomes complicated when machines learn from billions of human expressions. Unlike electrical, building or safety standards, language standards are rarely governed by a single authority. Dictionaries, style manuals, universities, publishers, governments and professional societies establish conventions, while ordinary speakers continuously modify them. AI systems operate in the middle of this tension between prescription and usage.

Training data. Large language models learn statistical patterns from books, websites, journalism, academic literature, software and other sources. The composition of that material influences what the model recognizes as normal, authoritative or acceptable language. Decisions about inclusion, exclusion and weighting can therefore function as de facto standards, even when no formal standards organization is involved.

 Grammar and intelligibility. AI can reinforce conventional spelling, syntax and technical terminology, making communication across institutions and borders easier. This is particularly valuable in engineering, medicine, law and education, where small differences in terminology can have substantial consequences. Yet excessive normalization may flatten dialect, regional vocabulary and inherited forms of expression.

Meaning. Words change over time and contested words often carry political, cultural or institutional assumptions. When an AI system chooses one definition over another, it may unintentionally appear to settle a dispute that society itself has not settled. Transparency about ambiguity is therefore an important characteristic of trustworthy AI.

Tchnical standardization. AI increasingly depends upon formal vocabularies, ontologies, metadata, machine-readable definitions and interoperability protocols. Standards organizations such as International Organization for Standardization, IEEE and National Institute of Standards and Technology have roles in developing frameworks through which AI systems can be evaluated and governed.

Above all we still face the question: Who gets to set the language standard? Should AI reflect contemporary majority usage, established literary traditions, professional terminology, institutional style or the language of particular communities? Probably some combination is unavoidable.

For Standards Michigan, the deeper question may be this: AI does not merely follow language standards; through widespread daily use, it may increasingly help create them. If millions of students, teachers, engineers and institutions rely upon AI to write and interpret language, the model’s linguistic choices can become conventions themselves. Understanding how those choices are made may therefore become as important as understanding the standards written by traditional standards-setting bodies.