robots.txt

By · July 11, 2026

Contents
  1. What it does, and does not, do
  2. robots.txt and AI crawlers
  3. Sources
  4. Related terms
robots.txt explained, a BeeRanked diagram

robots.txt is a plain-text file at the root of a domain that tells automated crawlers which parts of the site they may or may not request. It follows the Robots Exclusion Protocol (Google Search Central).

What it does, and does not, do

  • It controls crawling, not indexing. A page blocked in robots.txt can still appear in search if other pages link to it; to keep something out of the index, use a noindex tag instead.

  • Well-behaved bots obey it, but it is a request, not a lock, and not a security measure.

robots.txt and AI crawlers

The same file is how you manage AI crawlers. Each has its own user-agent token you can allow or disallow:

  • GPTBot, OpenAI's crawler (OpenAI).

  • Google-Extended, controls whether your content trains Google's Gemini and AI features, separately from normal Search (Google).

  • ClaudeBot, PerplexityBot, CCBot and others, each documented by its owner.

Blocking them protects content from AI training, but can also remove you from AI answers that cite live sources. That trade-off is a real decision, not an automatic yes.

Sources

Crawl budget · llms.txt · Answer Engine Optimization

← Back to Wiki