Crawl budget

Crawl budget is the number of pages a search engine will crawl on your site in a given period. It is a function of how many pages Google is willing to request (crawl demand) and how many your server can serve without slowing down (crawl capacity) (Google Search Central).

Who needs to care

Google is explicit: crawl budget is only a concern for large sites (roughly 10,000+ pages) or sites that generate many URLs automatically. A small blog will be crawled fully without any tuning, so this is not something most sites should worry about.

How to spend it well

• Fix or block low-value URLs (faceted-navigation duplicates, infinite calendars, session parameters) so crawlers spend time on pages that matter.

• Keep the site fast; a slow server lowers crawl capacity.

• Return correct status codes and keep your sitemap accurate.

Sources

• Google Search Central: managing crawl budget for large sites

Related terms

robots.txt · Canonical URL