LLM Radar Glossary

How a model serves a request

Prefill / decode: The two phases of generation. Prefill processes the entire prompt in one parallel pass and is limited by raw compute. Decode emits one token at a time, each depending on the last, and is limited by memory bandwidth. They have opposite bottlenecks, which is why every speedup in this issue is quoted for one or the other — a 9.5× prefill gain may do nothing for decode.
KV cache: Each generated token attends to every token before it. Rather than recompute the key and value vectors of all those earlier tokens at every step, they are stored. The cache grows linearly with context length, and at long contexts it routinely exceeds the size of the weights themselves — which is why compressing it (issue 01 §2) beats compressing weights for long-context work.
Prefix caching: When many requests begin identically — the same system prompt, the same loaded document — the KV cache for that shared opening can be computed once and reused instead of re-prefilled per request. llama.cpp's cache cloning (issue 01 §1) is this idea exposed as a server endpoint.
Context window: The maximum number of tokens the model can attend over at once. The headline numbers in issue 01 §3 (262K, 1M) are what the architecture supports, not necessarily what it uses well or what fits in your memory.
Decision models (JEV-style): Models that answer typed questions about a state — pick one of these labels, yes or no, a score — with a calibrated probability per option, in one forward pass and without generating text. Named after TypeSafe's Jev, whose /v1/systemone wire format the open models and llama.cpp adopted. They replace "generate, then parse" for routing, moderation and agent choices; the catch is that a well-formed, confident answer can still be wrong (issue 05 §3).

Attention, and why there are so many kinds

Attention (full, dense, softmax): The core operation: every token compares itself against every other token to decide what to draw on. Exact, and quadratic — double the sequence, quadruple the cost. Everything below is an attempt to avoid paying that.
Flash attention: Not an approximation. The same exact maths, computed in tiles so the full attention matrix is never written to memory. Pure win, which is why it is everywhere.
MHA / GQA / MLA: Multi-head attention runs several attention operations in parallel. Grouped-query has heads share key/value vectors, shrinking the cache. Multi-head latent compresses keys and values into a low-rank latent space, shrinking it further. Mostly these are cache-size reductions, not compute reductions.
Sliding-window attention (SWA): Each token attends only to the last N tokens. The cache stops growing and becomes fixed-size — at the cost of the model genuinely not being able to see further back from that layer.
Attention sink: The first few tokens of a sequence, which attention heads keep attending to heavily no matter what the content is. Keep them around and a sliding window behaves; drop them and quality collapses. Load-bearing this week because SWA with sinks matched post-trained linear-attention retrofits (issue 03 §2).
Sparse attention: Attend to a chosen subset of earlier tokens rather than all of them, with the selection either structural or learned. DeepSeek-V4's Compressed Sparse Attention consolidates the cache into blocks and then picks the top-k blocks, which is how it reaches a million tokens at 27% of the previous generation's compute.
Linear attention: Reformulates attention to cost linearly rather than quadratically, by carrying a fixed-size recurrent state instead of an ever-growing cache. Cheap and constant-memory — but the state is lossy, so it forgets. The whole design question is what it forgets and how fast.
Hybrid attention: Mixing two or more of the above inside one model, usually by interleaving layers. The trade is direct: linear and sliding-window layers are cheap but lossy, full and sparse layers are exact but expensive. Interleaving them captures most of the cost saving while keeping enough exact layers for genuine long-range recall. This is the dominant architecture of 2026 — Kimi K3 pairs KDA with MLA, DeepSeek-V4 interleaves compressed sparse with heavily compressed attention, Qwen3 and Granite interleave sliding-window layers with full ones. When this issue says "hybrid", that is what it means.
KDA: Kimi Delta Attention, K3's linear variant. Its distinguishing feature is per-channel control over how fast each dimension of the recurrent state decays — finer-grained forgetting than a single global decay rate.
NoPE: Training a model with no positional encoding at all, letting the causal mask supply order implicitly. It changes what a cache looks like: without rotation applied to the keys, a query-independent notion of "this token is salient" survives in the stored row, which is why an evictor can work before any query exists — and why the same trick collapses on a RoPE model (issue 04 §2).
RoPE: Rotary position embeddings: how a token's position gets encoded, by rotating its query and key vectors by an angle proportional to position. The standard approach, and the thing that extending a context window usually has to stretch.

Model structure

Dense vs MoE: A dense model runs every parameter for every token. A mixture of experts holds many parallel feed-forward "experts" and a router selects a few per token, so most of the model sits idle on any given token.
Total vs active parameters: Why MoE models are quoted as "2.4T total, 95B active". Total determines how much memory you need to hold the model; active determines how much compute each token costs. This is the entire reason trillion-parameter models are servable, and the reason they are still enormous to host.
Router / shared experts: The router is the small network choosing which experts handle each token — training it to distribute load evenly is a persistent difficulty. Shared experts run for every token regardless, holding general capability so the routed experts can specialise.
Expert offload and caching: Keeping a MoE's experts in system RAM, or on SSD, because the whole model does not fit in VRAM, and bringing in only the ones the router picks. It works because each token uses a few experts; it is slow because whichever few they are has to cross the bus. The design question is what stays resident: an LRU cache of recently used experts on the GPU, or a predictor that knows the next layer's experts in advance (issue 05 §1, §2).
MTP: Multi-token prediction: training the model to predict several tokens ahead rather than one. The extra prediction heads then double as a built-in drafter for speculative decoding, which is why MTP support and speculation support keep appearing together in issue 01 §1.
Residual and hyper-connections: Residual connections are the skip paths that let signal and gradient flow through a deep stack unimpeded. Recent work generalises them — manifold-constrained hyper-connections, attention residuals that let a layer pull representations from any earlier layer — because in deep sparse models a plain skip path is a bottleneck.
Looped (recurrent- depth) models: Models that run the same block of layers several times per token instead of stacking more distinct layers, so depth costs compute but not parameters. They buy reasoning per byte of weights, which is why they interest anyone memory-bound; the open questions are how to serve them — one KV cache per loop is expensive — and what they trade away, with early evidence that it is stored knowledge (issue 05 §2).

Making a model smaller

Quantization: Storing weights, and sometimes activations, at lower numeric precision. The dominant compression technique, and per issue 01 §2 the one that damages model behaviour least.
PTQ vs QAT: Post-training quantization converts a finished model — cheap, minutes to hours, no training. Quantization-aware training simulates the low precision during training so the model adapts to it — expensive, but the only thing that works at genuinely aggressive bit widths.
Calibration: PTQ needs a sample of data to choose its scaling factors. Long treated as a detail; issue 01 §2 is largely the story of that assumption breaking — calibrating on the right data is what made ternary quantization of reasoning models work at all.
Bit formats: FP16 and BF16 are the 16-bit baseline; FP8 and INT8 halve it; 4-bit halves it again. Q4_K and friends are llama.cpp's own block-quantized formats. MXFP4 and NVFP4 are 4-bit microscaling formats where a small block of values shares one scale factor — NVFP4 uses smaller blocks, capturing local dynamic range better, and needs Blackwell-generation hardware to run natively.
Weight-only vs W4A4: Weight-only stores weights at 4 bits but converts back up to compute, so it saves memory but not much bandwidth. W4A4 quantizes the activations too, so the arithmetic itself runs at 4 bits. That is the harder problem and the one that actually unlocks low-precision matrix engines.
STE: Straight-through estimator: the trick that makes QAT trainable at all. Rounding has a gradient of zero everywhere, so the backward pass simply pretends the rounding step was the identity function and passes the gradient through unchanged.
Proactive interference: A retrieval failure mode, borrowed from human working-memory research: when a value is overwritten repeatedly in the context, earlier versions keep intruding and accuracy on the current one decays as the overwrites pile up. It matters here because it is a behaviour ordinary benchmarks do not probe, so compression can measurably worsen it while aggregate scores look untouched.
Pruning: Deleting parts of the model outright. Unstructured pruning zeroes individual weights, producing sparsity that is hard for hardware to exploit. Structured pruning removes whole units — heads, layers, experts, vocabulary entries — which genuinely shrinks the model. Per issue 01 §2, it also damages reasoning more than quantization does.
On-policy distillation (OPD): Distillation where the student is trained on its own generated trajectories, with the teacher scoring each token the student actually produced, rather than on a fixed corpus of teacher outputs. It corrects the mistakes the student really makes, which is why it beats off-policy distillation on far less data. Its limitation is the flip side: the signal is per-token and therefore local, so it can patch a bad step without ever showing a complete correct path — the problem issue 02 §2 is about.
Distillation: Training a small student model to reproduce the behaviour of a large teacher. Distinct from compression proper: you are training a new small model, not shrinking the big one.
Off-policy vs on-policy distillation: Off-policy trains the student on text the teacher produced. On-policy has the student generate, and the teacher grade the student's own attempts — so the states the student trains on are the states it will actually encounter when deployed. That mismatch is what off-policy distillation gets wrong, and closing it is the reason on-policy took over the field this year.
Forward vs reverse KL: The two directions of the divergence a student minimises against its teacher. Forward KL penalises the student for missing anything the teacher considers likely, so it spreads out to cover the teacher. Reverse KL, the default in on-policy distillation, penalises the student for putting mass where the teacher would not, so it settles on the modes it already reaches. That difference is the reason OPD sharpens skills the student has but transfers little new knowledge (issue 05 §2).
LoRA / QLoRA: Fine-tuning by training small low-rank adapter matrices alongside frozen weights, instead of updating the whole model. QLoRA does that on top of an already-quantized base, which is what makes fine-tuning large models on one GPU possible.

Making inference faster

Speculative decoding: Something cheap proposes the next several tokens; the real model then verifies all of them in a single parallel pass and keeps the ones it agrees with. Because verification is parallel and generation is serial, this converts memory-bound decode steps into compute-bound ones. Critically, the output distribution is unchanged — it is a speedup, not an approximation.
Acceptance rate: The share of a draft's proposed tokens the target model actually keeps. It is what converts into speedup, and it is highly content-dependent: repetitive, structured output is easy to guess and open-ended prose is not, so a single average speedup figure for speculative decoding hides most of what you need to know.
Committed length and the ceiling bin: How many drafted tokens the target actually commits per verification round. A block drafter proposes a fixed-size block, so when the target accepts all of it the round is capped by the block size rather than by the drafter running out of skill: the mean hides this, and the histogram shows it as a spike in the top bin — speed-up you paid to train and never collect (issue 04 §2).
Draft model: The small model doing the proposing. It must be resident in memory and roughly aligned with the target model, which is the main cost of the technique. Since DFlash2, the draft is often not a smaller sibling but a purpose-trained head shipped as its own small checkpoint against one specific target, predicting a block of tokens per pass (issue 03 §1).
Suffix decoding: Speculation with no draft model at all: an index over text already seen proposes continuations, which works well because real generation repeats itself — code, names, quoted context. Removes the draft model's memory cost entirely (issue 01 §1).
Disaggregated prefill: Running the prefill phase on separate workers so that one long prompt does not stall token generation for everyone else sharing the server.
KV-cache eviction: Discarding entries from the KV cache so a long context stops costing memory. The selection rule is what papers compete on, but the memory only comes back if storage is physically reorganised per head — a selection kept as a mask frees nothing, and rankings that ignore this measure accounting, not selection (issue 03 §2).

Training stages

Pre- vs post-training: Pre-training is the large unsupervised run that builds the base model. Post-training is everything after — supervised fine-tuning, reinforcement learning, distillation. GLM-5.3 moving twenty points on the same base (issue 01 §4) is why post-training is now treated as a scaling axis of its own.
SFT: Supervised fine-tuning: training on demonstrations of the desired behaviour. The simplest and still the backbone of post-training.
RL / RLVR / GRPO: Reinforcement learning optimises against a reward rather than against fixed targets. RLVR uses rewards that can be checked automatically — the tests pass, the answer is right — which is why it works so well for code and maths. GRPO is the group-relative policy optimisation algorithm that became the standard way to run it.

All issues