One Python process per model works fine until you have thirty models, shared GPUs, and a latency promise you have to keep.
You launched one model behind a single Python process and it worked fine. Then came the second model, then the tenth — different frameworks, different hardware appetites, all competing for the same graphics processing units (GPUs), all carrying a latency promise. Restarting to swap a version drops live traffic, and when something gets slow nobody can say which model did it. This course builds the layer that fixes that. You'll stand up NVIDIA Triton Inference Server, write model configs that batch arriving requests automatically, and compile TensorRT-LLM engines with FP8 and INT4 quantization to cut cost per token. You'll move whole pipelines — tokenize, guard, generate, rerank — inside the server using ensembles and Business Logic Scripting, so one client call does the work of five. You'll pack many models onto shared GPUs with instance groups and Multi-Instance GPU (MIG) partitioning, try NVIDIA Inference Microservices (NIM) as the buy-instead-of-build path, wire Prometheus and Data Center GPU Manager (DCGM) metrics into a dashboard that names the slow model, and ship a version change through a canary rollout with a rollback you have actually tested.
Built by Lakshya Kumar
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyFinished the tasks? Take the prompt to your AI and get tested on it. We copy the prompt and open the app — just paste it in.
Stop writing a bespoke web server for every model — Triton loads PyTorch, ONNX, TensorRT, and vLLM behind one standard API.
A few lines of config.pbtxt can multiply your throughput or wreck your p99 latency — learn which knob does which, and why.
Build an engine, quantize it to FP8 or INT4, split it across GPUs, and serve it through Triton — same model, far less hardware.
TensorRT, ONNX Runtime, and torch.compile rewrite your graph for the hardware — and sometimes make it worse. Learn to tell.
Five network hops between tokenizer, guard, and model collapse into one call — with real if-statements running inside the server.
Load, unload, route, and isolate models so the batch job hogging a GPU can't break the endpoint a paying customer is waiting on.
A prepackaged container with an OpenAI-compatible endpoint saves you weeks — if you can live with the licensing and the registry.
When p99 latency doubles at 3am, queue time, compute time, and GPU utilization tell you which of your thirty models to blame.
Package the repository, send a new model version to 5% of traffic, watch the metrics, and get back to the old one in seconds.
Complete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Build and ship a two-model pipeline — embed-then-rerank or guard-then-LLM — as a Triton ensemble/BLS deployment (or a NIM plus custom logic) with in-server tokenization, dynamic batching tuned to an SLA, and full Prometheus + DCGM metrics. Submit the running endpoint, the config/repository, and a dashboard screenshot proving the SLA holds under load.
I'm taking an "Enterprise Serving Stacks" course covering NVIDIA Triton, TensorRT-LLM, NIM, graph compilation (TensorRT/ONNX Runtime/torch.compile), ensembles + Business Logic Scripting, multi-model GPU sharing, observability (Prometheus/DCGM/tracing), and production deployment (canary/blue-green/Kubernetes/autoscaling). It's hands-on with Python clients, config.pbtxt, bash for tritonserver/docker, and Kubernetes YAML. Here's my context: 1. My models to serve: [list models, frameworks, sizes] 2. My GPUs: [type, count, memory, interconnect] 3. My SLAs: [TTFT/TPOT/p99 latency targets per model] 4. My current serving setup: [nothing yet / a single vLLM process / something else] 5. My deployment target: [single host / Kubernetes / managed] Given that, answer: - Which module should I prioritize first, and why, for my situation? - For my model fleet and GPUs, should I lean toward self-built Triton, TensorRT-LLM engines, or NIM — and why? - Name 3 concrete wins this course would unlock for my deployment. - Name 1 thing the course won't help me with so I set expectations correctly. - If I only had one weekend, which single technique (batching tuning, quantization, GPU sharing, or observability) would give me the biggest lift, and how would I measure that it worked?
Build FP16 and quantized (FP8 or INT4 AWQ) TensorRT-LLM engines for an open model, serve them through Triton, and benchmark throughput, TTFT, and TPOT against each other and a hosted baseline, with a small quality eval on the quantized variant. Submit the build commands, the benchmark table, and the eval results.
Deploy at least five models sharing GPUs with instance groups, priorities/rate limits (or MIG isolation), and a router (by tenant/difficulty with failover). Load-test to prove the latency-critical model holds its SLA while a bulk model saturates the same hardware. Submit the fleet plan and the load-test results.
Build an in-server Business Logic Scripting pipeline with real control flow — e.g. a cheap guard/router that conditionally invokes an expensive model, or a refinement loop gated by a quality-check model. Prove the expensive model is skipped when it should be and the whole thing is a single client call. Submit the BLS code, the model repository, and a trace.
Instrument a serving deployment end to end (Triton + DCGM metrics, tracing, Grafana dashboard, SLO alerts) and ship a model-version change through a canary (or blue/green) rollout with a tested rollback. Induce a regression and show your observability caught it and the rollout aborted. Submit the dashboard, alert rules, and a deploy/rollback log.
Module 4's home base: engine builds, in-flight batching, quantization, and multi-GPU.