Glossary

Terms used in the notes

Short definitions with a link to the note that uses each term in context.

Active parameters

The parameters that run for one token in a mixture-of-experts model: the shared layers plus the experts the router selects. Close to the per-token compute cost. Read more

Chat template

The rule that turns a list of messages into one token sequence, with role markers and special tokens. A changed template can change behavior on the same input. Read more

Contamination

Overlap between evaluation items and training data, including near copies and memorized solutions. It can raise a score without a matching gain in capability. Read more

Context length

The maximum number of tokens a model accepts in one request. Cards often give a native length and a longer extended length that needs runtime support. Read more

Dense model

A model in which every parameter takes part in every token. Its total and active parameter counts are the same. Read more

End-to-end latency

Time from sending a request until the complete response arrives. It depends strongly on output length. Read more

Inter-token latency

Time between consecutive generated tokens after generation starts. Its inverse is the per-request generation rate. Read more

Key-value cache

Stored attention keys and values for the tokens a request has processed. It grows with context length and concurrency and often limits capacity. Read more

Mixture of experts

An architecture with many expert blocks per layer and a router that selects a few of them for each token. Memory follows the total size; compute follows the active size. Read more

Model revision

The exact commit of a model repository. A repository name is a moving label; a revision identifies fixed files. Read more

Open weights

Model parameters published for download under a stated license. Open weights do not imply open training data or an open-source license. Read more

Paired comparison

Scoring every candidate on the same cases and analysing per-case differences instead of comparing two unrelated averages. Read more

Prefix caching

Reusing the processed state of a repeated prompt prefix across requests. It reduces prompt work and must be disclosed in a benchmark. Read more

Quantization

Storing weights or activations with fewer bits per value to reduce memory and speed up serving. It can change output quality and needs its own check. Read more

Time to first token

Delay from sending a complete request until the first generated token arrives. It includes queueing and prompt processing. Read more

Tokenizer

The component that splits text into the tokens a model reads. Prices, limits and speeds are stated in its units, which differ between models. Read more