Crawl budget

What is Crawl budget?

Crawl budget is the number of pages a crawler is willing to request from a website within a given period. It isn't unlimited: it depends on how fast the server responds and how much interest the search engine has in that site. On small sites it's rarely a problem. It starts to matter on large sites with thousands of similar URLs, filters, and parameters, where the crawler spends its turn on pages that don't matter and never gets to the ones that do.

Related terms

robots.txt

The file that tells crawlers which parts of your site they can access.

Canonical

The tag that marks which address is the correct one for a page.

GPTBot

OpenAI's crawler. Not the only name the company declares.

And what about your brand?

We’ll tell you how many times your brand shows up in ChatGPT, Gemini, Perplexity, and Claude when someone asks about your category, and who shows up in your place. Free, no card required.

I want my free diagnosis Check my site’s access