Most training jobs leave half the hardware idle — learn to measure where the time and the money actually go, then take them back.
Your training run finishes eventually, costs more than you budgeted, and you have no real idea which part of it is slow. This course teaches you to find out. You start by learning to time a step honestly and price a model's memory, so you can say whether the graphics processing unit (GPU) you're paying for is limited by math, by memory traffic, or by waiting around. Then you learn how a GPU actually executes code, and how to read real profiler traces from PyTorch and NVIDIA's Nsight tools. From there you write your own Triton kernels — plus just enough CUDA, NVIDIA's low-level GPU programming language, to read one — apply the levers that turn an out-of-memory crash into a run that fits, and scale out: several GPUs on one machine, sharded training for models too big for any single card, then multi-node clusters with real torchrun and SLURM launch scripts. The final modules cover serving a trained model to users at low latency, and forecasting a run's hours and dollars before you commit to them. Everything is runnable Python, real kernels, and commands you can lift straight into your own project.
Built by Lakshya Kumar
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyFinished the tasks? Take the prompt to your AI and get tested on it. We copy the prompt and open the app — just paste it in.
The hardware model behind every later trick: why identical math runs ten times slower when the data is laid out wrong.
Stop optimizing on a hunch — capture a real trace, find the biggest block of wasted time, and prove your fix actually moved it.
When the built-in ops leave speed on the table, write the kernel yourself — and learn to tell when it isn't worth the effort.
The levers that turn a crashed run into one that fits on the card you have, and what each one costs you in step time.
Extra cards buy less than you'd hope; measure the shortfall, find the communication hiding inside it, and close what you can.
When the model itself won't fit on a single card, shard it across many — and predict per-GPU memory before you burn cluster hours.
Week-long jobs on many machines will break; launch them properly, checkpoint so a crash costs minutes, and debug a hang that never errors.
Serving is not training — the numbers users feel are first-token wait and tokens per second, and each needs a different fix.
Forecast the loss, the hours, and the dollars from small cheap runs — and decide up front when to stop paying for more.
Complete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Take a real training or inference workload you can run, profile it end-to-end using Module 3's workflow, identify the dominant bottleneck, and apply techniques from this course to speed it up by a target factor — at least 2x tokens/sec for a training workload, or 2x MFU/throughput for an inference workload. Submit the before/after trace, the specific fix or fixes applied, and a written diagnosis tracing the speedup to a specific bottleneck category (compute, memory, latency, or communication).
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
I'm taking a "GPU Performance: Train Big Models Faster and Cheaper" course. It covers, in order: the roofline model, arithmetic intensity, and a profiling-first mindset; GPU fundamentals (SIMT execution, the memory hierarchy, tensor cores, occupancy); real profilers (torch.profiler, Nsight Systems, Nsight Compute) and reading a trace; writing Triton kernels (elementwise fusion, softmax, matmul) plus just enough CUDA to read one, and the flash-attention online-softmax algorithm; memory optimization (mixed precision, activation checkpointing, gradient accumulation, CPU/NVMe offload); data parallelism and DDP's all-reduce/bucketing/overlap; sharded training (ZeRO stages, FSDP, tensor and pipeline parallelism, Megatron-style 3D parallelism); multi-node training (interconnects, NCCL collectives, torchrun/SLURM, fault tolerance, checkpointing); inference optimization (the KV cache, continuous batching, PagedAttention, quantization, speculative decoding); and scaling laws, MFU, and training-cost planning. My context: 1. My workload is: [training a model from scratch / fine-tuning / serving an LLM for inference / not sure yet] 2. My hardware access: [free Colab/Kaggle GPU / a single cloud GPU / multiple GPUs on one node / a multi-node cluster] 3. My current bottleneck (if I know it): [slow training steps / OOM errors / poor multi-GPU scaling / slow/expensive inference / I haven't profiled yet] 4. My framework: [PyTorch version, and whether I already use DeepSpeed, FSDP, vLLM, or similar] Given that, answer: - Which module should I start with, and why, given my current bottleneck? - What's the single fastest diagnostic (from Module 1 or 3) I should run first to confirm what's actually limiting me? - Name one technique from this course that's very likely to give me the biggest win for my specific workload and hardware. - If I only have a free-tier GPU, which hands-on tasks in this course will I need to adapt or simulate rather than run for real? - Name one thing this course will NOT solve for me (e.g. it won't pick my model architecture or dataset), so I have the right expectations.
Pick a real fusion opportunity in an actual model — not a toy example — implement it as a Triton kernel, verify its numerical correctness against the eager PyTorch reference, and benchmark it against both the eager baseline and torch.compile at production-realistic shapes. Submit the kernel, the verification results, and the benchmark with a defensible ship-or-don't-ship verdict per Module 4's decision rule.
Take a real training job and scale it from 1 GPU to at least 4 (or a believable, clearly-labeled simulated equivalent), measuring throughput and scaling efficiency at each step. Profile the multi-GPU run to attribute any inefficiency to a specific cause — communication, bucketing, or per-GPU batch size — and apply the correct learning-rate scaling rule for the resulting effective batch size. Submit the throughput table, the profiler evidence, and the scaling-efficiency diagnosis.
The FLOPs/bytes/KV-cache arithmetic behind Modules 1 and 9.