A demo that answers instantly for one user can crawl — and bill a fortune — once real traffic arrives. Fix that before it ships.
Your prototype answers in a second. Then real traffic arrives, the same endpoint takes twenty, and the bill for the GPUs (graphics processing units) running it climbs past what the product earns. This course is about closing that gap. Working in Python on real models, you'll trace one request end to end: how a large language model (LLM) reads your whole prompt in one pass, then writes the answer one token at a time, and why that second phase is starved by memory bandwidth rather than raw arithmetic. You'll calculate how much memory the key-value cache (KV cache — the per-request state that keeps generation fast) eats, and see why it, not the model weights, caps how many users one card can serve. From there you'll run vLLM and measure what actually moves the numbers: PagedAttention, continuous batching, prefix caching, chunked prefill, speculative decoding, and 8-bit quantization. You'll benchmark competing engines head to head, then finish by sizing GPUs for a stated latency target, computing cost per million tokens, and making a build-versus-buy call you can defend.
Built by Lakshya Kumar
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyFinished the tasks? Take the prompt to your AI and get tested on it. We copy the prompt and open the app — just paste it in.
Every token you generate leaves memory behind. Learn how much, why it caps concurrent users, and how to share it across requests.
Your GPU sits idle most of the time, waiting on memory. Prove which limit you're hitting before you optimize the wrong thing.
Naive allocation throws most of your GPU memory away to fragmentation. Paging reclaims it and multiplies the users you can serve.
One request at a time barely touches the hardware you're paying for. Continuous batching packs the GPU without wrecking latency.
Average latency hides the users who wait longest. Measure the tail, find whether the delay is first token or every token, then fix it.
Config flags stay guesswork until you can watch the engine batch, queue, and preempt requests step by step — then tune on evidence.
Every framework claims to be fastest. Learn which one fits your model, hardware, and workload — and how to switch without a rewrite.
Shrinking weights and swapping kernels can double throughput or quietly degrade answers. Measure both, then stack only the wins that hold.
Turn a latency target and a traffic forecast into a GPU count, an autoscaling plan, and a cost per million tokens you can defend.
Complete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Stand up an OpenAI-compatible vLLM endpoint for a model of your choice, instrument TTFT/TPOT/throughput with percentiles, run a load test, and produce a capacity-and-cost report that meets a stated SLO. The report must justify GPU count, per-replica concurrency, autoscaling thresholds, and projected cost-per-million-tokens with measured numbers.
I'm taking a course on LLM inference and serving internals. Help me tailor it to my situation. My model: [e.g. Llama 3.1 8B / Qwen2.5 7B / a 70B model] My hardware: [e.g. one 24GB consumer GPU / an A100 80GB / 2 nodes x 4 H100 / CPU only] My workload: [e.g. interactive chat / batch summarization / an agent with long shared system prompts] My SLO: [e.g. p99 TTFT < 1s, p99 TPOT < 50ms / "throughput, latency doesn't matter"] My priority: [throughput | latency | cost | data control] For each lesson, ground the concept in my setup: compute my KV-cache size and max concurrency for my model+GPU, tell me whether I'm prefill- or decode-bound for my workload, recommend a serving framework and quantization stack for my priority, and walk me through sizing and cost-per-million-tokens for my SLO. When a choice depends on details I haven't given, ask me before assuming.
Build a benchmarking harness that drives any OpenAI-compatible endpoint with a configurable workload (prompt/output mix, concurrency sweep) and reports req/s, prefill/decode tok/s, and TTFT/TPOT percentiles, plus a throughput-vs-latency frontier plot. It must produce apples-to-apples comparisons across configs or engines.
Take one model and serve it under fp16 plus a stack of optimizations (weight quantization, KV-cache quantization, attention backend, chunked prefill), measuring decode tok/s, memory, and a task-relevant quality metric at each step. Deliver a Pareto report and a recommended stack that holds quality within a stated budget.
Benchmark at least two serving frameworks (e.g. vLLM and TGI, optionally SGLang or llama.cpp) on the same model and load test via the OpenAI-compatible API, comparing throughput, latency percentiles, and operational ergonomics. Deliver a recommendation for a stated workload with the runner-up and decision-flip conditions.
Build a capacity-planning model that turns an SLO and a traffic forecast into GPU count, per-replica concurrency, autoscaling thresholds, capacity mix, and projected cost-per-million-tokens, validated by a load test and including a build-vs-buy comparison against a hosted API at the forecast volume.
The PagedAttention paper: the KV-cache memory management that underpins modern serving.