llms.txt and AI Crawlers: What to Allow, What to Block, and What Actually Works

Two files control how AI systems see your website, and most advice online confuses them. One, llms.txt, is a proposal that is cheap to add and currently changes nothing about whether ChatGPT names you. The other, robots.txt, is thirty years old and decides, crawler by crawler, whether you exist to an engine at all. This article covers both, with what each company has actually said.

What llms.txt is

llms.txt is a proposed convention: a Markdown file served at /llms.txt that gives a language model a short summary of your site and a curated list of links to the pages that matter, so a model with a limited context window does not have to crawl everything to understand what you do. It was proposed by Jeremy Howard, co-founder of Answer.AI, in a post published on September 3, 2024 at llmstxt.org.

The format is deliberately simple. The proposal specifies, in order: an H1 with the name of the project or site, a blockquote with a short summary, zero or more Markdown sections with details, and zero or more sections delimited by H2 headers that contain lists of links with a short description each. An H2 called "Optional" is reserved by convention for links an agent can skip when it needs a shorter context. The file can sit at the site root or at any path, covering the pages under it.

Some sites also publish a companion file, often called llms-full.txt, that contains the full text of the documentation in one Markdown file. Anthropic's own developer documentation does this: docs.claude.com serves an llms.txt index. The pattern comes from developer documentation, where it solves a real problem: a coding assistant reading a library's docs benefits from a clean Markdown version instead of a rendered HTML page with navigation and scripts.

What the engines have actually said about it

As of September 2026, no major search engine or AI company has said that it uses llms.txt to decide what to show or cite, and Google has said the opposite. That is the whole answer, and it is worth being precise about who said what.

• Google, April 2025. John Mueller, on Reddit: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag."
• Google, June 17, 2025. Mueller again, on Bluesky: "FWIW no AI system currently uses llms.txt." He added that the consumer chatbots fetch your pages for training and grounding, but none of them fetch the llms.txt file.
• Google Search Central, guide updated July 2026. "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." The guide adds that such files will neither harm nor help visibility in Google Search.
• Chrome Lighthouse, 2026. Lighthouse added an llms.txt audit under its Agentic Browsing category. Read it carefully: it flags a page only if the server returns an error when fetching /llms.txt. If the file simply does not exist (404), the audit is marked Not Applicable, because, in Chrome's words, "providing the file is optional at the moment." It is a check for broken files, not a requirement.
• OpenAI, Anthropic, Perplexity. None of the three lists llms.txt in its crawler documentation, and we found no statement from any of them that their search products read it. Anthropic publishes one for its own docs, which is a signal that the format is useful for agents reading documentation, not that Claude's search uses it to rank businesses.

So the honest position is: llms.txt is harmless, takes twenty minutes, and is a reasonable courtesy to coding agents and documentation tools that do read it. It is not a visibility tactic. If your site is a software product with documentation, add one. If your site is a plumbing company, spend the twenty minutes on robots.txt instead.

The AI crawlers that matter

Ten user-agents decide whether the four main engines and Google's and Apple's AI features can see your site, and they come in three kinds: training crawlers, search crawlers, and user-triggered fetchers. The table uses each company's official documentation as read on September 23, 2026.

Three things to notice. First, every company separates training from search, and OpenAI says explicitly that each setting is independent of the others. Second, the user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) act on behalf of a person who asked a question, and two of the three companies say robots.txt may not stop them. Third, Google-Extended and Applebot-Extended are not crawlers at all: Googlebot and Applebot do the crawling, and the -Extended tokens only tell the company what it may do with the content afterward.

Blocking training vs blocking search

You can block your content from being used to train models without disappearing from the AI engines' search, because the companies use different user-agents for each and honor them separately. The mistake sites make is blocking by pattern ("anything with GPT or Claude in the name") or with a one-click "block AI bots" setting, which removes them from search along with training.

Decide the two questions separately. Do you want your content used to train future models? That is a policy and licensing question; there is a reasonable case either way, and blocking has no effect on whether an engine names you. Do you want to be named and cited when someone asks an engine about your category? If you sell anything, the answer is yes, and it requires the search crawlers to be allowed.

A robots.txt that blocks training and keeps search open

The block below denies the three training user-agents and says nothing about the search ones, which means they inherit your default rules (usually allow). Adjust the paths if you only want to protect part of the site.

• Block OpenAI training: User-agent: GPTBot, then Disallow: /
• Block Anthropic training: User-agent: ClaudeBot, then Disallow: /
• Block Gemini training and grounding: User-agent: Google-Extended, then Disallow: /
• Block Apple model training: User-agent: Applebot-Extended, then Disallow: /
• Everyone else, including the search crawlers: User-agent: *, then Allow: / (plus your usual disallows for admin paths)

Note what is absent: no line for OAI-SearchBot, Claude-SearchBot or PerplexityBot. If you want to be explicit, add "User-agent: OAI-SearchBot" followed by "Allow: /" and the same for the other two; it does no harm and it documents the intent for the next person who edits the file.

A robots.txt that allows everything

If you have no objection to training, the simplest correct file is the one you probably already have: a wildcard user-agent with your normal disallows for admin paths and search result pages. Do not add AI-specific lines you do not need. Every extra rule is a chance to block the wrong bot.

How to check what you are blocking today

Most sites that are invisible to an engine were blocked by a setting someone else turned on, so check three places before writing anything: robots.txt, your CDN, and your security plugin.

1. Open yourdomain.com/robots.txt and search for GPT, OAI, Claude, Perplexity, Google-Extended and Applebot. Any Disallow under a search user-agent is a problem.
2. Check your CDN's bot settings. Cloudflare announced on July 1, 2025 that it would block AI crawlers by default. If your site sits behind Cloudflare or a similar service, look for an AI bot or AI crawler control and confirm the search crawlers are allowed.
3. Check WordPress security and SEO plugins. Several offer a "block AI bots" toggle that writes rules for every AI user-agent it knows, search crawlers included.
4. Read your server logs for the user-agents in the table above. If OAI-SearchBot and PerplexityBot have never visited, you are either blocked or not linked from anywhere they crawl. Both are fixable; they need different fixes.
5. Confirm the page itself renders as text. A fetcher that is allowed in but receives an empty shell that fills in with JavaScript sees nothing. Test the URL with JavaScript disabled.

If you write an llms.txt anyway

If you decide to add one, write it for a reader with two minutes and no browser: one H1 with your name, one blockquote that says what you do and for whom in two sentences, then a short list of your most useful pages with a one-line description each. Keep it under a screen. Do not paste your home page copy into it, and do not treat it as a place to put claims you would not put on the page itself; a model that reads both will notice the difference.

Then forget about it and do the work that moves the needle: allow the search crawlers, publish pages that answer buying questions with real numbers, and get named on the third-party sites the engines cite. That is the subject of How to get recommended by ChatGPT, and the reasoning behind it is in What is generative engine optimization.