Words the field guide keeps using.
Every term the guide leans on, defined in a sentence or two. Four of them have a full page that links to the simulator where you can watch the idea work; the rest are listed A to Z below.
Start here
Four ideas worth a full page.
A to Z
All 63 terms.
- Adam · Adam optimizer · AdamW
- The standard optimizer for LLM training. It keeps two running averages per parameter (momentum and variance), usually in FP32, which is why optimizer state dominates training memory.
- all-gather
- Second half of a ring all-reduce, also used alone by ZeRO-3/FSDP: every GPU’s slice is copied to every other GPU until all hold the full tensor.
- all-reduce · AllReduce · reduce-scatter · all-gather
- A collective that sums a value (e.g. gradients) across all GPUs and hands the result back to every one. Done as a ring (reduce-scatter then all-gather) so each GPU sends about twice the data once, however many GPUs there are.
- all-to-all
- A collective where every GPU sends a different slice of data to every other GPU. Mixture-of-experts layers use two per layer: one to dispatch tokens to their experts, one to bring results back.
- arithmetic intensity
- FLOPs performed per byte moved from memory. Low intensity = memory-bound; high intensity = compute-bound.
- Chinchilla
- DeepMind’s 2022 result showing most LLMs were undertrained: for a fixed compute budget, params and tokens should scale together — roughly 20 tokens per parameter.
- chiplet · chiplets
- A smaller die combined with others in one package instead of building a single giant chip — improves yield and sidesteps the reticle limit.
- chunked prefill · chunked prefills
- Splitting a long prompt into fixed-size slices and processing one slice per step alongside other users’ decodes, so a big prompt never pauses everyone else’s token stream.
- continuous batching
- Serving the batch as a rolling pool of slots — a finished request is instantly replaced and new prefills ride alongside ongoing decodes — instead of waiting for a fixed batch. The biggest single throughput lever.
- CoWoS · Chip-on-Wafer-on-Substrate
- TSMC's Chip-on-Wafer-on-Substrate advanced packaging: the GPU die and its HBM stacks sit on an interposer: plain silicon in CoWoS-S, or silicon bridges in a larger organic one in CoWoS-L (Blackwell, Rubin). Its capacity, not wafers, often gates GPU output.
- CUDA
- NVIDIA's parallel-programming platform and the software moat behind its dominance; AMD's open equivalent is ROCm.
- data parallelism · data-parallel · data parallel
- Replicate the whole model on each GPU, feed each a different slice of the batch, then average the gradients with an all-reduce every step.
- decode
- The inference phase that generates one token at a time — memory-bandwidth-bound, since the full weights and KV cache are streamed for each single token.
- disaggregated serving · disaggregated · prefill/decode disaggregation
- Running prefill and decode on separate GPU pools (Splitwise, DistServe) and streaming the KV cache between them, so the two phases scale independently and stop interfering.
- expert parallelism · expert-parallel · expert parallel
- Place a Mixture-of-Experts layer’s experts on different GPUs; an all-to-all routes each token to its chosen experts and back.
- FlashAttention
- An IO-aware attention algorithm that tiles the computation into SRAM and recomputes intermediates instead of writing them to HBM — far less memory traffic, much faster long-context attention.
- FP8 · FP4 · FP16 · low precision · lower precision
- Low-precision floating-point formats (8-, 4-bit…). Each halving of the bits per number roughly doubles throughput, with care taken to preserve accuracy.
- goodput
- Throughput counting only requests that met their latency targets. A server can post high tokens per second while most users wait too long; goodput catches that.
- GQA · Grouped-Query Attention · grouped-query attention · MQA · Multi-Query Attention
- Grouped-Query Attention — groups of query heads share one set of key/value heads, shrinking the KV cache (the limit on batch size and context) many-fold with little quality loss. MQA is the extreme, one KV head.
- HBM · High-Bandwidth Memory · High Bandwidth Memory
- High-Bandwidth Memory — DRAM dies stacked vertically and bonded next to the GPU, trading a very wide bus for enormous bandwidth at low energy per bit. Full page →
- InfiniBand
- A low-latency RDMA datacenter network long favoured for AI clusters; the scale-out fabric that links nodes, competing with RoCE/Ethernet.
- Infinity Fabric
- AMD's on-package and chip-to-chip interconnect, its answer to NVLink for linking compute dies and GPUs into a shared fabric.
- interposer
- A silicon (or organic) baseplate carrying tens of thousands of ultra-dense traces that wire the GPU die to its HBM stacks — finer than any PCB can route.
- KV cache · KV-cache
- The stored keys and values from earlier tokens that let an LLM avoid recomputing attention each step. It grows with sequence length × batch and must be streamed from HBM every token.
- memory wall
- The widening gap between how fast cores compute and how fast memory can feed them — for LLM inference, bandwidth, not FLOPS, is usually the bottleneck.
- memory-bound · memory bound · compute-bound · compute bound · bandwidth-bound
- A kernel is memory-bound when it waits on data movement (bandwidth), and compute-bound when it waits on the arithmetic units. LLM decode is memory-bound; prefill is compute-bound.
- micro-batch · micro-batches · microbatch · microbatches
- A slice of the training batch sent through the pipeline on its own, so different stages can work on different slices at the same time.
- microscaling · MXFP · MXFP8 · MXFP4
- Low-precision formats that attach a shared scale to each small block (e.g. 32 values), taming outliers so FP8/FP4 stay accurate. Standardised by the Open Compute Project (OCP); NVIDIA’s NVFP4 is a cousin with 16-value blocks and an FP8 scale.
- MIG · Multi-Instance GPU
- Multi-Instance GPU — NVIDIA partitioning that splits one physical GPU into several isolated instances, each with its own memory and compute slice.
- Mixture-of-Experts · Mixture of Experts · MoE
- A model with many expert sub-networks and a router that fires only a few per token — so total parameters balloon while compute per token stays roughly flat.
- Model FLOPs Utilization · MFU
- The fraction of a cluster’s peak FLOPs a real training run actually sustains after communication, pipeline bubbles and memory stalls. 40–50% is considered good.
- multi-head latent attention · MLA
- DeepSeek’s attention design that caches one small compressed vector per token instead of full keys and values, shrinking the KV cache far below GQA.
- NCCL
- NVIDIA Collective Communications Library — the GPU-to-GPU routines (all-reduce, all-gather, broadcast…) that make multi-GPU training go, over NVLink and InfiniBand/RoCE.
- NVLink
- NVIDIA's high-speed GPU-to-GPU interconnect — 1.8 TB/s per GPU on Blackwell, 3 TB/s on Rubin, far faster than PCIe — letting a rack of GPUs behave like one big accelerator.
- NVSwitch
- NVIDIA’s switch chip for NVLink — it wires many GPUs into one all-to-all fabric, turning a rack (e.g. 72 GPUs in GB200 NVL72) into a single memory domain.
- PagedAttention · vLLM
- vLLM’s scheme that stores the KV cache in fixed-size blocks (like OS virtual-memory pages) that can sit anywhere and be shared — cutting KV waste from 60–80% to under 4% and lifting throughput 2–4×.
- pipeline bubble · bubble
- Idle time in pipeline-parallel training while the pipeline fills at the start of a step and drains at the end. Roughly (stages − 1) ÷ (micro-batches + stages − 1) of each step.
- pipeline parallelism · pipeline-parallel · pipeline parallel
- Assign different layers to different GPUs like an assembly line; micro-batches flow stage to stage. The idle start/end gap is the pipeline “bubble”.
- prefill
- The inference phase that processes the whole prompt in parallel and builds the KV cache — compute-bound, since many tokens are crunched at once.
- prefix caching · prompt caching · RadixAttention
- Keeping the KV cache of prompts already processed, so requests that start with the same tokens (a system prompt, a document, earlier chat turns) skip recomputing them.
- quantization · quantized · AWQ · GPTQ
- Storing weights (and often the KV cache) in fewer bits — FP8 or 4-bit (AWQ, GPTQ). Cuts memory ~2–4× and speeds the bandwidth-bound decode, with little accuracy loss when done carefully.
- reduce-scatter
- First half of a ring all-reduce: each GPU ends up owning one slice of the data, summed across every GPU.
- reticle limit · reticle-limited · reticle
- The maximum area (26 × 33 mm, about 858 mm²; half that on High-NA EUV) a lithography scanner can pattern in one shot — the hard ceiling on single-die size that pushes designs to chiplets and dual-die.
- ridge point · the ridge · ridge
- Where a roofline bends: peak FLOPs divided by memory bandwidth. Kernels doing fewer FLOPs per byte than this wait on memory; kernels doing more are limited by compute.
- RoCE
- RDMA over Converged Ethernet — running remote-direct-memory-access networking on Ethernet, the open alternative to InfiniBand for GPU clusters.
- ROCm
- AMD's open GPU compute software stack — its alternative to CUDA for running AI workloads on Instinct accelerators.
- roofline
- A performance model that plots attainable FLOPs against arithmetic intensity: below a threshold you’re capped by memory bandwidth, above it by raw compute.
- scaling laws · scaling law
- Empirical power-law relationships predicting how model loss falls as parameters, data and compute grow — the basis for sizing frontier training runs.
- SerDes
- Serializer/Deserializer — the circuitry that turns parallel data into fast serial links (and back), the basis of every high-speed interconnect.
- SIMT · Single Instruction, Multiple Threads
- Single Instruction, Multiple Threads — the GPU execution model where one instruction drives a whole group of threads (a warp/wavefront) in lockstep.
- speculative decoding
- A small draft model proposes several tokens; the large target model verifies them in one parallel pass and keeps those it agrees with — 2–3× faster decode with identical output.
- Streaming Multiprocessor · Streaming Multiprocessors · SM
- NVIDIA's core building block: a cluster of CUDA cores, Tensor Cores, schedulers, registers and on-chip SRAM. AMD's equivalent is a Compute Unit (CU). Full page →
- structured decoding · constrained decoding · structured output · structured outputs
- Masking out, at every step, the tokens that would break a required format such as a JSON schema, so the output is valid by construction.
- TDP
- Thermal Design Power — the sustained heat (in watts) a chip’s cooling must remove; a proxy for how much power it draws under load.
- Tensor Core · Tensor Cores · matrix cores · Matrix Cores
- A dedicated unit that computes a small matrix multiply-accumulate every clock — the engine behind the headline "AI TFLOPS". Full page →
- tensor parallelism · tensor-parallel · tensor parallel
- Split each layer’s weight matrices across several GPUs so one layer’s matmul runs in parallel. Chatty — needs the fast NVLink links, so it stays inside a node.
- tile · tiles · tiling
- A block of the output matrix computed by one group of threads. Loading inputs once into on-chip memory and reusing them across the tile is what lets a kernel do many FLOPs per byte read from HBM. Full page →
- TPOT · time per output token
- Time Per Output Token — the gap between streamed tokens after the first, set by memory-bound decode.
- Transformer Engine
- NVIDIA hardware+software that dynamically picks numeric precision (e.g. FP8) per layer to keep Tensor Cores saturated without hurting model quality.
- TSV · through-silicon vias · through-silicon via · TSVs
- Through-Silicon Via — a vertical copper channel etched through a chip so stacked dies (e.g. HBM layers) can talk to each other directly.
- TTFT · time to first token
- Time To First Token — how long before generation starts, set by the compute-bound prefill of the whole prompt.
- warp · wavefront
- The smallest scheduled bundle of threads that execute together — 32 on NVIDIA (a "warp"), 64 on AMD (a "wavefront").
- ZeRO
- Zero Redundancy Optimizer (DeepSpeed) — shards optimizer state, gradients and parameters across data-parallel GPUs to remove the memory wasted by replicating them.