robots.txt and llms.txt for AI crawlers

Two small text files at the domain root now show up in many “are we ready for AI search?” checklists: robots.txt and llms.txt. One is a decades-old crawler-directive standard. The other is a newer, unofficial map. Neither is a security boundary, and mixing them up wastes a sprint.

robots.txt controls compliant crawlers

robots.txt tells crawlers which paths they may request. Search engines have honored the convention for a long time. AI vendors added their own user agents on top of the usual Googlebot.

The names that matter depend on the product behavior you want to allow:

• OAI-SearchBot for ChatGPT search discovery and GPTBot for potential model-training use. ChatGPT-User handles user-initiated visits and is not an automatic crawler; OpenAI says robots.txt rules may not apply to those requests. (OpenAI crawler documentation)
• Google-Extended, a control token for Gemini model training and grounding. It does not affect Google Search inclusion or ranking and does not have a separate HTTP user agent. (Google crawler documentation)
• ClaudeBot for potential model-training use, Claude-SearchBot for search, and Claude-User for user-directed retrieval. (Anthropic crawler documentation)
• PerplexityBot for search discovery and Perplexity-User for user-initiated retrieval. Perplexity says the latter generally ignores robots.txt. (Perplexity crawler documentation)
• Applebot-Extended
• CCBot (Common Crawl, which many models have used as source material)

A Disallow: / asks a named, compliant crawler not to request the site. It does not authenticate visitors, remove content already indexed, or stop a user-directed fetch that the provider treats differently. Sites sometimes copy a “block AI” snippet from a privacy post and later wonder why search products cannot discover their service pages. Decide separately whether you want to allow training, automatic search indexing, and user-directed retrieval.

Googlebot and Google-Extended are not the same allow-list. You can stay in organic search and still opt out of extended use. Read the group you actually edited.

Also check the boring misses: a wildcard Disallow: / meant for staging that shipped to production, or a sitemap URL in robots.txt that 404s.

llms.txt is a map, not a ranking lever

llms.txt (and sometimes llms-full.txt) is a Markdown file at the site root. The emerging convention is a short, human-and-model-readable table of contents: what the organization is, which pages matter, how to describe products or service areas.

It may be useful the way a clean sitemap and honest about page are useful, especially for agents navigating documentation. It is not a switch that inserts a business into ChatGPT, and the published specification still describes itself as a proposal. Treat it as an optional experiment, not a technical SEO defect. (llms.txt proposal)

If you add one, keep it true:

• Use the same name, address, and phone you use everywhere else.
• Link to canonical service and location pages, not a junk drawer of parameters.
• Do not stuff keywords. A model that reads the file can also read that you stuffed it.

An empty or auto-generated llms.txt that disagrees with the homepage is worse than none.

What to check on Monday

1. Fetch https://yourdomain/robots.txt. Search for the AI user agents above. Confirm you did not disallow / unless that is the policy.
2. Fetch https://yourdomain/llms.txt. 404 is common. That is a gap, not an emergency.
3. Confirm sitemap.xml is live and lists the URLs you actually want retrieved.
4. Look at schema on the homepage and key service pages (LocalBusiness or Organization where accurate). FAQPage must match visible FAQ content, and Google now limits FAQ rich results mainly to well-known government and health sites. (Google's FAQ guidance)

Crawler access is one eligibility input for products that use that crawler. Citation is a different problem — content quality, listings, third-party corroboration, indexing, and each provider's retrieval and ranking systems. Check access first so you are not debugging visibility while the relevant search crawler is turned away.