Methods

Measuring tokenizer efficiency for Chinese text

The same Chinese paragraph can cost very different numbers of tokens. That changes price, context budget, and latency, so measure it directly.

By LLM Model Notes · 4 November 2026 · 8 minute read

Prices, context limits, and generation speeds are all stated in tokens. A token is whatever the model's tokenizer produces, and tokenizers differ in how they split Chinese text. A vocabulary built mostly from English text may split a common Chinese word into several byte-level pieces, while a vocabulary built with more Chinese data may keep it as one token. Comparing two models on cost or context without measuring this difference compares two different units.

Choose the unit you report

Report efficiency against something that does not depend on the tokenizer. Three units are common:

• Characters per token. Easy to read for Chinese, where one character is a natural unit. Count Unicode code points after normalization, not bytes.
• UTF-8 bytes per token. Works for any script and for mixed text, but a Chinese character takes three bytes, so the numbers are not comparable with English figures.
• Tokens per reference document. The most direct for budgeting: how many tokens a fixed set of real documents costs.

Pick one as the headline and keep the others in the result bundle. State which one each table uses.

Build a fixed corpus

The result depends heavily on the text. Formal news prose, chat messages, classical quotations, source code with Chinese comments, and tables of numbers all tokenize differently. Build a corpus that resembles the intended workload and split it into labelled slices:

• Simplified and Traditional characters, kept separate.
• Pure Chinese prose and mixed Chinese and English text.
• Text with numbers, dates, and units.
• Source code or structured data with Chinese strings.

Give each document a stable identifier and record the corpus revision, as the comparison note recommends for evaluation cases. If the text cannot be published, publish the slice definitions and a small sample with permission.

Normalize before counting

Unicode offers several ways to write what looks like the same text. Full-width and half-width forms of Latin letters and digits, different punctuation marks, and compatibility characters all change the token count. Decide on one normalization, such as NFKC or none, apply it to the corpus once, and record it. Do not let each tokenizer apply its own normalization silently, or the comparison will measure the normalizers rather than the vocabularies.

Whitespace also matters. Chinese text has no spaces between words, but mixed text and code do. Keep the original whitespace and count it as part of the input.

Run every tokenizer from its own repository

Load each tokenizer from the exact model repository and commit you plan to serve. Tokenizers change between releases even inside one model family, and a chat template adds its own tokens around every message. Measure two numbers per document:

• Raw tokens for the text alone, without special tokens.
• Tokens for the text wrapped as one user message in the model's chat template.

The first compares vocabularies. The second is what a request actually costs. For short chat messages, the template overhead can be a large share of the total.

Turn counts into decisions

Token counts become useful when you convert them into the units of a decision. Multiply tokens per reference document by the price per token to compare real cost. Divide a context limit by tokens per document to see how many documents fit. Combine output tokens per answer with the measured generation rate from the latency note to estimate response time in seconds rather than in tokens.

Keep the per-document numbers, not only the averages. A tokenizer that is efficient on prose can be inefficient on code or numbers, and the average hides that if one slice dominates the corpus.

Limits of the method

Fewer tokens is not the same as better quality. A larger vocabulary can make each token rarer in training data, and a model can perform well despite a less efficient tokenizer. Treat efficiency as a cost and capacity measurement and evaluate quality separately on the same text.

Primary references

• Neural Machine Translation of Rare Words with Subword Units — byte-pair encoding for subword vocabularies.
• SentencePiece — language-independent subword tokenization without pre-segmentation.
• Unicode Normalization Forms — the NFC and NFKC definitions used before counting.