Shipping on vibes means every prompt tweak is a coin flip — build the datasets, scorers, and CI gates that catch damage before users do.
You changed a prompt, the demo looked fine, and three days later support tickets say the answers got worse. Without a way to measure quality, every change to a large language model (LLM) feature is a guess and every rollback is an argument. This course builds the measurement layer. You start by hand-labeling a small golden set and running error analysis on real failures, then pick scorers that fit the job — exact match and structural checks, embedding similarity, precision and recall — and learn when a number is real versus noise from a 30-example sample. You then build an LLM-as-judge scorer and calibrate it against human labels so you can trust it. From there you wire a repeatable harness into continuous integration (CI) as a release gate, and specialize it for retrieval-augmented generation (RAG) pipelines, tool-using agents, and adversarial attacks. The last modules move to production: tracing, sampling live traffic, drift detection, guardrails, and A/B tests. You finish with an eval pack and CI gate that provably blocks a planted regression. Python-first, all runnable.
Built by Lakshya Kumar
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyFinished the tasks? Take the prompt to your AI and get tested on it. We copy the prompt and open the app — just paste it in.
Your scores are only as honest as your examples — curate, label, version, and check for leaked test data before you measure anything.
A single average hides the failures that matter — combine cheap checks with semantic scores, slice by segment, and test if a gain is real.
A model can score what no regex can — write the rubric, kill position and verbosity bias, and prove the judge agrees with humans.
Turn a score into a decision — a repeatable runner, cost and latency budgets, and a GitHub Actions gate that fails the pull request.
Bad answer, or bad retrieval? Score context relevance, faithfulness, and citations separately so you fix the stage that actually failed.
Agents can reach the right answer the wrong way — verify task success, check every tool call, and catch loops before they burn budget.
Someone will try to break your app — find the prompt injections and jailbreaks first, then freeze each breach into a permanent test.
Offline scores go stale the day traffic shifts — trace real calls, sample them for review, detect drift, and get paged when quality drops.
Gates alone don't improve anything — decide what blocks a release, triage regressions to root cause, and scale the practice across a team.
Complete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Build a complete eval pack and CI gate for a real (or realistic) LLM app. It must include a versioned golden set, a validated metric suite (with at least one calibrated LLM judge), cost/latency budgets, and a tiered release gate wired into CI. Prove it works: plant a genuine regression in a branch (a prompt or model change that degrades a real behavior), open a PR, and show the gate blocks the merge and the report localizes the failing cases. Submit the repo, the CI run showing the block, and a short writeup of the regression and how the gate caught it.
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
I'm taking an "AI Evaluation & Reliability Engineering" course. It covers, in order: the eval-first mindset and small-sample statistics; datasets and ground truth (golden sets, sampling, labeling, synthetic data, leakage); metrics (deterministic, semantic, precision/recall/F1, calibration, significance); LLM-as-judge (rubrics, pairwise, bias, judge-human agreement); harnesses and CI release gates (promptfoo/Braintrust patterns); RAG and retrieval evals (the triad, faithfulness, citations); agent and tool-use evals (trajectories, sandboxes); red-teaming and adversarial evals; online eval and observability (tracing, sampling, drift, guardrails, A/B); and eval-driven development (gating, budgets, dashboards, the full program). My context: 1. The LLM app I want to make reliable is: [describe — chatbot / RAG / agent / classifier] 2. My current evaluation situation: [none / I eyeball outputs / some offline tests / I have a CI gate] 3. My biggest quality problem right now: [hallucinations / inconsistency / cost / latency / safety / I don't even know] 4. My stack: [language, model provider, any eval tools already in use] Given that, answer: - Which module should I start with, and why? - Name the first three eval cases I should hand-label for my app. - Name the cheapest scorer that would catch my biggest quality problem (deterministic, semantic, or LLM-judge) and why. - If I only had one afternoon, what single eval-driven habit would give me the biggest reliability lift? - Name one thing this course will NOT solve for me, so I have the right expectations.
Build a standalone RAG evaluation suite: a golden set with deliberate hard negatives, the full triad (context relevance, faithfulness, answer correctness), retrieval metrics (recall@k, MRR), citation-accuracy checks, and an end-to-end diagnostic that localizes each failure to a pipeline stage. Run it against a real or open RAG system and submit the report plus a ranked list of the highest-leverage fixes.
Build an agent eval harness in a deterministic sandbox (task-success verifiers, tool-call correctness, trajectory scoring, operational health) plus an adversarial regression suite (injection/jailbreak, safety two-sided, robustness, fairness) fed by an automated attacker loop. Wire both as a combined gate where a safety violation is disqualifying, and submit the harness, the attacker's harvested breaches, and per-category resist rates.
Error analysis, human review, and the data flywheel. Modules 1, 2, and 9.