When you send text to a language model, it never sees your letters. Before anything else happens, your words are chopped into tokens — chunks roughly the size of a common word-piece — and each token is swapped for a number the model can do maths on. Tokenization is the quiet first step in every model, and once you understand it, a whole set of otherwise baffling AI behaviours suddenly make sense.

What a token actually is

A token is not a word and not a letter — it sits in between. A tokenizer is trained on a large body of text to find the most efficient set of recurring chunks. Common words usually become a single token. Rarer or longer words get split into pieces: "tokenization" might break into "token" + "ization". Spaces and punctuation are tokens too, and the same word can tokenize differently depending on whether it has a leading space.

A rough rule of thumb for English: one token is about four characters, or roughly three-quarters of a word. So 1,000 tokens is around 750 words. That ratio is the hidden unit behind almost everything you pay for and wait on.

Why this explains the model's "weird" failures

Several of the most-mocked LLM weaknesses are tokenization artefacts, not reasoning failures:

• Counting letters. Ask a model how many "r"s are in "strawberry" and it may stumble — because it never saw the letters. It saw two or three tokens, and the individual characters inside them are not directly visible to it. It is reasoning about chunks, not spelling.
• Reversing or manipulating characters is hard for the same reason: character-level operations cut against the grain of a token-level view.
• Maths on long numbers wobbles partly because digits get grouped into tokens inconsistently, so the model isn't always working with the place-values you'd assume.
• Rare words, code, and other languages can be less reliable and more expensive, because unfamiliar text fragments into many small tokens instead of a few efficient ones.

None of these are signs the model "can't think". They are signs that its sense organs work in word-pieces, not characters.

Why tokens are the unit of cost and context

Two of the most practical facts about using models are denominated in tokens, not words:

• Pricing. API costs are per token, input and output. A verbose prompt or a chatty system message is a standing tax on every call. Trimming tokens is the most direct lever on cost.
• Context windows. When a model advertises a "200,000-token context", that limit covers everything at once — your system prompt, the conversation history, any retrieved documents, and the reply. Run out and the earliest content falls off the back. Knowing the ratio lets you budget: a 50-page document is roughly 25,000 tokens, so it fits comfortably in a large window but would blow past a small one.

This is also why non-English text and code often cost more: they tokenize less efficiently, so the same meaning consumes more of your budget.

Practical takeaways

• Estimate before you build: words × 1.3 ≈ tokens. Use it to size context budgets and forecast cost.
• If a task is genuinely character-level — counting, spelling, exact string surgery — don't trust the model's intuition; give it a tool, or restructure the task so the characters are spelled out.
• Watch your "free" overhead: long system prompts, repeated context, and bloated formatting are paid for on every single request.
• Expect more variance on rare words, unusual formatting, and languages far from the tokenizer's training data.

Tokenization is invisible, but it is the lens every model looks through. Understanding it turns a lot of AI's surprising behaviour from mysterious to obvious — and makes you better at both budgeting for models and knowing which tasks to hand them.