Zero errors, normal latency, confidently wrong answer. The instrumentation that makes an LLM system debuggable rather than merely monitored.
Your agent returned 200 in 1.9 seconds and told the customer something that is not true. Nothing in your monitoring will catch that, because every assumption ordinary tracing makes — same input, same call graph; bounded cost; success is a status code — is false for an agent. This course is the instrumentation that closes it, built on the OpenTelemetry GenAI semantic conventions rather than any vendor's SDK, because the instrumentation outlives the tool. You start by writing down the questions you get asked after a bad answer, then make each answerable: the gen_ai.* vocabulary, including the request-versus-response model split that catches a provider repointing an alias underneath you; step spans that make the agent loop measurable and a context-growth curve that predicts a runaway before the bill does; a tool span recording whether the tool returned nothing, the strongest predictor of a hallucinated answer; retrieval provenance that routes a RAG complaint to retrieval, the prompt, or the context budget; and a prompt-capture policy you could defend to an auditor. It ends with quality as a sampled, judged SLI. Python throughout, verifiable on localhost with no vendor account.
Built by Lakshya Kumar
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyFinished the tasks? Take the prompt to your AI and get tested on it. We copy the prompt and open the app — just paste it in.
One agreed vocabulary for models, tokens, tools and agents — experimental, moving, and still far better than inventing your own.
One wrapper around your provider client, emitting conventional spans and metrics that any OTLP backend understands.
Steps, retries, sub-agents and fan-out each need a shape. Get the tree right and the critical path reads itself.
Tools are where an agent touches the world. Instrument the argument, the result, the failure kind and the authorisation.
Which chunks came back, which survived reranking, and which actually reached the prompt. Provenance, per span.
The span must outlive the first byte. Instrument the stream itself and time-to-first-token becomes your real SLI.
You cannot debug what you cannot see, and you cannot store what you must not keep. Redact at the Collector, sample by tenant.
Derive cost from span attributes rather than maintaining a parallel counter, so the trace and the invoice cannot disagree.
Every handoff is a lossy summary. Instrument what was passed, what was dropped, and whether the intent survived.
Your error rate can be zero during a total quality outage. Sample a quality signal, then burn-rate alert on it.
Langfuse, Phoenix and Arize all speak OTLP. Fan out from one Collector, keep what each is good at, change your mind later.
I'm taking "See Inside Your Agents: Telemetry for AI Systems" — an agent instrumentation course in the Agentic and Applied AI track. Twelve modules: why LLM systems break normal tracing, the gen_ai semantic conventions, vendor-neutral instrumentation, span topology for agent loops, tool and MCP telemetry, RAG span provenance, streaming, prompt capture without leaking, cost as a derived metric, multi-agent handoffs, quality SLOs, and wiring up a backend. My context: 1. My stack and framework: [describe — LangChain / LlamaIndex / bare SDK / other] 2. What telemetry exists today: [nothing / a vendor LLM tool / partial OTel / a full APM] 3. Whether my agent uses tools, retrieval, sub-agents, streaming: [describe] 4. My monthly model spend, if I know it: [number] 5. The last wrong answer I could not explain: [describe] 6. Whether I can store prompts at all, and under what policy: [describe] Given that, answer: - Which module should I start with, given what already exists? Skipping ahead is fine if the earlier ground is covered. - Which parts of this course do NOT apply to my situation, and why? - For my last unexplained wrong answer, which specific attribute would have explained it? - What is the single cheapest change that would most improve my ability to diagnose the next one?
The specification pages matter more than the language guides once you are past setup.