Open this lesson in your favourite AI. It'll walk you through the why, explain the demo, and quiz you on the try-it list.
Agent-driven browsing is fundamentally different from Googlebot: it is stateful, goal-directed, and often JavaScript-aware, which means the content that surfaces to an agent is determined by interaction patterns, not just link graphs.
Use these three in order. Each builds on the one before.
In one paragraph, explain the difference between how a traditional search crawler indexes a webpage and how an AI agent browses it.
Walk me through the step-by-step process an AI agent uses when it calls a web browsing tool — from URL fetch to text extraction to passing content back to the LLM.
Given a product page that loads pricing data via a client-side API call after initial render, how would a traditional crawler versus an agent-powered browser each handle it, and what does that mean for what gets indexed?