Three Forces That Make Serving Hard (SLAs, Frameworks, GPUs)

Why this matters

Three forces make enterprise serving hard at once, and they pull against each other. SLAs say a chat endpoint must answer first token under 200ms while an offline embedding job can wait seconds. Mixed frameworks mean a TensorRT-LLM engine and a PyTorch reranker must coexist on the same machine. GPU sharing means you can't afford one GPU per model, so several models must pack onto each device without one starving another. A serving platform has to satisfy latency targets, hide framework differences, and arbitrate a scarce shared GPU simultaneously. Seeing how these three constraints interact is what tells you when you've outgrown a single process and need real orchestration.

Demo

The demo packs several models onto a fixed GPU and checks both the memory fit and whether the latency-sensitive model still gets enough of the device — the core tension of co-location.

Try it yourself

1. Increase the chat model to 60GB and re-run; identify which models must move to another GPU once memory no longer fits.
2. Mark the embedding job as a bulk workload and reason about why it should be rate-limited or de-prioritized so it doesn't blow the chat SLA.
3. Compute leftover GPU memory and decide how many more copies (instances) of the small guard model you could add.
4. Describe what signal you'd watch to know that co-location is hurting the latency-critical model's p99.

References

• Triton — optimization & concurrent model execution
• NVIDIA — Multi-Instance GPU (MIG) overview

Chat about this lesson