Tiling
A block of the output matrix computed by one group of threads. Loading inputs once into on-chip memory and reusing them across the tile is what lets a kernel do many FLOPs per byte read from HBM.
Nearly all of a transformer’s work is matrix multiplication. Every output value is a dot product of one row and one column, and none depends on another — so a GPU can cut the output into tiles and compute them in parallel.
Loading inputs once into on-chip memory and reusing them across the tile is what lets a kernel do many FLOPs per byte read from HBM. Bigger tiles mean more reuse; the overview demo lets you change tile size and watch the traffic trade-off.
TensorLight glossary · Streaming Multiprocessor · Tensor Core · tile · HBM · also called tiles, tiling