robots.txt
What is robots.txt?
robots.txt is a text file at the root of a website that tells automated crawlers which parts of the site they can access and which they can't. It works in groups: one block per agent or wildcard, with allow and disallow rules for specific paths. It isn't a lock, it's a sign that well-behaved crawlers follow by convention. Today it decides something bigger than it used to, because it's where AI crawlers get allowed or blocked, often without the site owner knowing.
Related terms
And what about your brand?
We’ll tell you how many times your brand shows up in ChatGPT, Gemini, Perplexity, and Claude when someone asks about your category, and who shows up in your place. Free, no card required.