What Is a Token, Anyway?

The pricing page says per million tokens, and the natural assumption is that a token is a word. It is not. A token is a piece of text the model's tokenizer chose to treat as one unit — sometimes a whole word, sometimes part of a word, sometimes a single character, sometimes a punctuation mark with its neighbors. The tokenizer decides, and it decides differently for different languages, different formats, and different models.

The practical consequence: the same sentence can be a different number of tokens on different models, and the number on the invoice is the tokenizer's count, not your word count. The price-list post's advice to read the fine print applies here too — the fine print is the tokenizer.

This post is the mechanics: how tokenization works, why it varies, and how to estimate your own numbers without guessing.

By frequency. Tokenizers are built from large text corpora: the pieces of text that appear often — common words, common word parts, common punctuation patterns — become single tokens, and rarer text gets cut into smaller pieces. "The" is one token. A rare technical term might be several. The tokenizer is optimizing for the average text it was trained on, and your text may not be average.

The method is called byte-pair encoding, and the intuition is compression: the tokenizer is a dictionary of frequent pieces, and every piece of text is expressed as a sequence of dictionary entries. The dictionary is fixed per model — it was built once, before training — which is why the count is stable for a given model and different across models.

And the dictionary is public. Tokenizer tools exist precisely because the count is deterministic: the same text, the same model, the same number, every time. Estimation is arithmetic, not divination.

Because the invoice counts tokens, and the tokenizer decides the count. Three variables move the number. Language: tokenizers trained mostly on English cut English efficiently and other languages less so — a sentence in a language the dictionary covers poorly can cost several times its English equivalent. Format: code, JSON, and tables are token-hungry, because punctuation-heavy text uses many small tokens. And vocabulary: specialized terms the dictionary has never seen get cut into pieces.

The same content, the same model, can vary by a factor of several depending on these three. The long-context post's crossover math and the cost-per-task framework both inherit this variance — the token count is the input to every cost calculation on this blog, and the tokenizer is what makes the input honest.

The discipline is simple: measure, do not assume. Send your actual prompts through the tokenizer before you budget — the pricing calculator and the model library both work in tokens, and the only way to feed them honest numbers is to count your own text. A prompt you have counted once is a fact; a prompt you have estimated is a guess wearing a number.

Count the whole request, not just the question: the system prompt, the examples, the tool schemas, the conversation history — everything that rides along is tokens, and the prompt-caching post covers which parts can be made cheaper. And count both directions: input and output are priced differently on most models, and the split matters to the blended rate.

Then re-count when the prompt changes. A prompt is a living document, and its token count is a metric that should be reviewed with the same seriousness as its quality.

The two taxes worth knowing by name. The language tax: if your product serves users in a language the tokenizer covers poorly, your per-answer cost is structurally higher than an English product's, and the fix is not a cheaper model but a tokenizer-aware choice — some models tokenize multilingual text far better than others, and the difference is visible in the count before you spend a dollar.

The format tax: structured output is expensive to tokenize. JSON with its braces and quotes, code with its punctuation, tables with their spacing — all of it costs more tokens than prose carrying the same information. Getting JSON reliably is its own discipline; the cost side is that reliability has a token price, and it is worth knowing before the invoice explains it.

When the workload is small enough that the difference between an efficient tokenizer and a wasteful one is a rounding error. A hobby project, an internal tool, a prototype — the token count is a curiosity, and optimizing it is premature.

Tokens also stop mattering when a larger cost dwarfs them: the engineering time spent optimizing a prompt that runs a few times a day, or the quality loss from compressing a prompt until it stops working. The cheapest request is the one that can wait, and the cheapest token is sometimes the one you did not spend an afternoon saving.

The honest hierarchy: correctness first, cost per completed task second, token count third. The tokenizer is worth understanding because it is the unit of the bill — not because it is the point of the product.