Your prompt is not a wall and your agent's tools are the attacker's tools; find the holes yourself, then keep them closed with tests.
You shipped an assistant that reads user-supplied documents, calls a couple of tools, and answers questions over your own data. Your web-security instincts do not cover what happens when the document itself contains instructions, when a retrieved chunk belongs to another customer, or when a polite-sounding message talks your agent into deleting a record. This course closes that gap. You begin by mapping where a large language model (LLM) application's trust boundaries sit, using the OWASP LLM Top 10 as shared vocabulary, and build a risk register you carry through every later module. Then you attack: prompt injection, jailbreaks, retrieval-augmented generation (RAG) poisoning and cross-tenant leakage, agent and tool abuse, and the model itself via prompt extraction, memorization, and denial-of-wallet. The back half is defense: layered guardrails you test for false positives, architecture with tight permissions, personally identifiable information (PII) redaction and egress control, a red-team suite that runs in continuous integration, and the governance artifacts an incident or an audit will demand. Every payload is paired with the control that catches it: Python-first, runnable, no turnkey exploitation tooling. You finish by red-teaming a real app and shipping the fixes, building a reusable defense-middleware library, or automating a red-team harness.
Built by Lakshya Kumar
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyFinished the tasks? Take the prompt to your AI and get tested on it. We copy the prompt and open the app — just paste it in.
Instructions can arrive from a web page, a PDF, or a tool response, so build the harness that shows what your app still obeys.
Your refusal training holds until someone reframes, encodes, or slow-walks the request; find which of the six families your system misses.
One poisoned document or one missing permission check can leak another tenant's data, so retrieval needs its own security tests.
An agent is only as safe as its narrowest permission; decide what it may do before a crafted message decides for you.
Attackers can lift your system prompt, probe what your data remembers, and burn your budget without ever breaking in.
A single filter fails loudly in both directions; compose defenses in an order you can defend and test the false positives too.
Assume the model gets compromised and design so it barely matters: tight permissions, no secrets in context, nowhere to send stolen data.
One manual attack session ages out in a week; a graded suite wired into CI catches the regression your next prompt tweak introduces.
When something goes wrong at 2am or a regulator asks who approved this, the answer has to already exist in writing.
Complete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Red-team a provided (or your own) AI application against the OWASP LLM Top 10. Produce a findings report mapping every discovered vulnerability to its OWASP category, with severity, reproduction steps, and impact. Then implement the fixes for at least your three highest-severity findings, and ship a regression suite (reusing this course's harness patterns) that proves each hole is closed and will stay closed on future changes.
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
I'm taking an "LLM & AI Security" course. It covers, in order: the LLM threat model and the OWASP LLM Top 10; prompt injection (direct, indirect, exfiltration, encodings, tool-output injection); jailbreaks and guardrail bypass (persona/roleplay, obfuscation, crescendo, translation gaps, payload splitting, many-shot); RAG and data-layer attacks (poisoning, access control, tenant isolation, embedding attacks, citation exploitation); agent and tool abuse (excessive agency, confused-deputy, tool-call hijacking, sandbox escapes, SSRF, multi-agent propagation, approval gating); model attacks (prompt extraction, inversion, membership inference, memorization, denial-of-wallet, distillation); defenses and guardrails (input/output filtering, structured output, spotlighting, least-functionality, dual-LLM isolation); secure AI architecture (least privilege, secrets, PII redaction, tenant isolation, supply chain, egress control, logging); AI red-teaming (adversarial suites, automated attacker loops, grading, coverage, CI gating, bounty triage, findings reports, tabletops); and governance/compliance (audit logging, incident response, EU AI Act, NIST AI RMF, model cards, vendor risk, disclosure). My context: 1. The AI system I want to secure is: [describe — chatbot / RAG assistant / tool-using agent / something else] 2. Its current security posture: [none / basic input filtering / a real guardrail pipeline / I genuinely don't know] 3. The attack I'm most worried about right now: [prompt injection / jailbreaks / data leakage via RAG / agent taking a bad action / I'm not sure] 4. My role and constraints: [solo builder / small team, no dedicated security hire / have a security team but they don't know AI-specific risks / other] Given that, answer: - Which module should I start with, and why, given what I'm most worried about? - Name the first three test cases I should write against my system today, specific to my described attack surface. - Name the single highest-leverage defense (from this course) I should add this week given my current posture. - If I can only build one artifact from this course in the next month, should it be a test harness, a guardrail pipeline, or a governance document — and why, given my role and constraints? - Name one thing this course will NOT fully solve for me (this is a genuinely unsolved area of security), so I have realistic expectations.
Build a reusable guardrail/defense middleware library implementing at least five distinct controls from this course (input risk classification, output schema validation, spotlighting, least-functionality tool scoping, dual-LLM isolation, or an SSRF/egress-safe fetch tool). Ship it with a documented API, a regression test suite covering both attack-blocking and false-positive cases, and a README showing how to wire it into an existing LLM application in under an hour.
Build an automated red-teaming harness: a taxonomy-mapped seed corpus (at least 6 OWASP LLM Top 10 categories), a mutate-test-select attacker loop that generates novel variants, a layered grader distinguishing attack success from failure, a coverage report, and a CI-style gate that blocks a regression in caught rate. Run it against a real or realistic target system and submit the harness, its coverage report, and the findings it surfaced.
The most-cited ongoing writing on prompt injection and its defenses. Modules 1, 2, and 7 lean on it directly.