An Illustrated Guide to GPU Kernels for ML

There is a particular kind of frustration in reading GPU kernel code. Every line is simple — an index computation, a load, a barrier — and yet the whole is opaque, because the reasons live in hardware behaviour that nobody writes down next to the code. Why 128×128 tiles? Why is the shared-memory array indexed with an XOR? Why does the fast attention kernel have two warpgroups doing different things?

This series answers those questions in order, from the bottom. It is built the way the MMA swizzle-layout post that inspired it is built: numbered sections, a diagram for every mechanism, and a refusal to hand-wave past the confusing part.

Every diagram in the series is drawn from one small palette, and colour always means the same thing. There are only two hues. Blue is the execution side of the machine — lanes, warps, warpgroups, blocks. Amber is the storage side — registers, shared memory, L2, HBM. Within each, darker means further out: a darker blue is a wider scope, a darker amber is further from the ALU and slower to reach.

That leaves pink for the one thing that is neither, the tensor core, and grey for anything idle, masked or wasted. Green and red are held back for verdicts — conflict-free versus serialised, above the ridge versus below it — and never used to name a component.

• interactive in a caption means the figure has controls. Drag them; the readout under the chart updates. Nothing is a video — every figure is live SVG.
• Code appears in tabs. CUDA is the ground truth, Triton is what most people write, and PyTorch is the reference semantics you are trying to match. Not every block has all three.
• Key numbers are sourced inline. Where NVIDIA quotes a figure "with sparsity", the dense number is shown for an apples-to-apples roofline. The higher sparse peak is real when operands satisfy NVIDIA's supported 2:4 structured-sparsity pattern; an ordinary dense LLM kernel does not receive it automatically.
• Solid fills are measurements — a bar's length or a cell's value. Pale washes with a coloured outline are containers — a box in a floorplan or a dataflow. If a shape is pale, its size means nothing.
• The dark button in the top bar switches themes; the diagrams recolour with it.

Almost every design decision in the series is downstream of one of these. If you only take one thing from Parts 1–3, take this table.

* NVIDIA does not publish B200's SM count; 148 is a third-party measurement. NVIDIA's Blackwell Tuning Guide publishes the 126 MB L2 capacity. Tensor figures here are dense; NVIDIA's 2:4 sparse figures are twice the dense peak and require compatible structured operands. Ridge point = dense peak ÷ HBM bandwidth: the arithmetic intensity above which a kernel becomes compute-bound.

You need to be able to read C and Python. You do not need prior CUDA. If you have written a Triton kernel and want to know what it compiles into, or you have read the FlashAttention paper and want to know what "warp specialization" physically means, this is aimed at you.

The series leans on primary documentation throughout — the CUDA Programming Guide, PTX ISA, CUTLASS/CuTe source, and NVIDIA's architecture tuning guides — plus the following, which are worth reading in full:

• Yifan Yang, Tensor Core MMA Swizzle Layout — the post that prompted this series.
• Simon Boehm, How to Optimize a CUDA Matmul Kernel — the canonical optimization ladder.
• Horace He, Making Deep Learning Go Brrrr From First Principles — the compute / bandwidth / overhead taxonomy.
• Vasily Volkov, Better Performance at Lower Occupancy (GTC 2010) — still the correct framing of occupancy.
• Dao et al., FlashAttention-2, FlashAttention-3, and FlashAttention-4.
• Colfax Research's CUTLASS tutorials — the best writing on Hopper-era kernel structure.

@misc{mapika2026gpukernels,
  author = {Marosi, Mark},
  title = {An Illustrated Guide to GPU Kernels for ML},
  year = {2026},
  url = {https://gpu-kernels.pages.dev/}
}