AI Agents Browse, Search Engines Crawl

easy

Learn with your AI

Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.

Or teach me the whole lesson:

Why this matters

Agent-driven browsing is fundamentally different from Googlebot: it is stateful, goal-directed, and often JavaScript-aware, which means the content that surfaces to an agent is determined by interaction patterns, not just link graphs.

Try it yourself

Compare the HTTP headers sent by a known agent browser tool (e.g., Bing's AI user-agent string) against a traditional crawler user-agent — identify at least one behavioral difference implied by the difference.
Load a JavaScript-heavy product page and inspect the DOM before and after JS execution — note which pricing or feature text would be invisible to a non-JS crawler but visible to a headless-browser agent.
Check your product's robots.txt and identify which rules would block or allow the user-agents used by OpenAI's GPTBot and Anthropic's ClaudeBot.

References

https://platform.openai.com/docs/plugins/bot