Don't block AI crawlers with your robots.txt

A robots.txt written years ago to keep scrapers out often turns away the AI crawlers too. A crawler that is refused never reads the page, so ChatGPT, Gemini and Perplexity have nothing of yours to quote, however good the content is. This 27-second video shows where that block usually sits.

You do not have to read the file yourself. The free AI crawler checker fetches your robots.txt and lists, bot by bot, which AI crawlers get in. If it finds a block you did not intend, Controlling what AI crawlers see says what each crawler is used for, and the wiki entry on robots.txt covers the syntax.