AI visibility
llms.txt, robots.txt and AI crawlers: what to allow and what to block
Ilyes Massoudi6 min readThe short answer
llms.txt is a proposed text file placed at a site's root to summarise its content for AI models, and to date no major engine says it reads it or builds anything on it; Google has said plainly it does not use it. What decides your presence in AI answers is robots.txt: whether it lets search crawlers such as Googlebot, OAI-SearchBot and PerplexityBot in, or blocks them inside a blanket rule. Separate search crawlers, which bring visits, from training crawlers such as GPTBot, which collect data for models; always allow the first, and decide on the second by policy. And make sure your text is in the page source, because most of these crawlers do not execute JavaScript.
llms.txt: what it is, and what it is not
In 2024 a developer proposed a simple standard: a text file named llms.txt at the site root, summarising what the site offers and pointing to its key pages, so AI models could read it instead of crawling everything. The idea is elegant, and it spread quickly through SEO tools that sell its creation.
The problem is that none of the major engines uses it. Google said clearly in 2025 that its systems do not read the file and do not plan to, and one of its engineers compared it to the old keywords meta tag. Neither OpenAI, Anthropic nor Perplexity has announced building on it. Studies that examined logs of hundreds of millions of AI-crawler visits found requests for the file close to nonexistent, most of them from SEO tools themselves.
The practical conclusion: add it if you like, it does no harm, do not pay anyone for it, and do not call it AI-visibility work. What I wrote in the ChatGPT visibility article still stands: llms.txt is not the solution being sold to you.
What actually decides your presence: robots.txt
The file every crawler reads before touching your site is robots.txt. And here sits the mistake I see often on Gulf sites: a generic security rule that blocks “all bots” except Google, or a firewall setting that rejects every unknown user agent. The result is that answer-engine crawlers are silently refused, and the site is absent from ChatGPT and Perplexity without its owner knowing.
The crawlers that matter fall into two kinds, and the difference between them is the whole decision:
| Crawler | Operator | What it does | My usual decision |
|---|---|---|---|
| Googlebot | Search, AI Overviews and AI Mode | Always allow | |
| Google-Extended | Gemini training and use in other products; no effect on search | By policy | |
| OAI-SearchBot | OpenAI | ChatGPT search and its linked answers | Allow |
| ChatGPT-User | OpenAI | Fetches a page when a user asks for it in a chat | Allow |
| GPTBot | OpenAI | Collects data for model training | By policy |
| PerplexityBot | Perplexity | Perplexity search and answers | Allow |
| ClaudeBot | Anthropic | Training and fetching | By policy |
| Bingbot | Microsoft | Bing search and Copilot | Always allow |
The simple rule: search crawlers bring you visits and mentions, so allow them. Training crawlers take your content to build models and return nothing directly, so decide on them according to what suits your business. A commercial site that wants to appear in answers allows the first and can block the second at no cost.
What a correct robots.txt looks like
For a commercial site that wants to appear in search and answers while blocking training crawlers:
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /checkout/
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Sitemap: https://example.com/sitemap-index.xmlThree notes: the general rule allows everything except private pages, blocking is named specifically for training crawlers only, and the sitemap is declared in the file. Remember that robots.txt is a polite request, not a security barrier; a malicious bot ignores it, and a page you genuinely want hidden needs password protection.
The check that comes before all of this
Most AI crawlers do not execute JavaScript. They read the HTML as the server sends it and leave. If your page text is drawn after load, allowing them in robots.txt hands them an empty page.
Open your page, view the source, and search for a sentence from your service description. This check precedes any file configuration. Details in technical SEO for Arabic websites.
A five-step checklist
- Open
/robots.txton your site and read it. Is there a generic rule blocking what you did not intend to block? - Review the firewall or Cloudflare settings: is there a mode that rejects unknown bots? Exempt search crawlers by name.
- Confirm the text is in the page source.
- Decide on training crawlers by policy, and write the decision into the file by name.
- Check the server logs after a month to see who actually visits.
What the audit checks here
In the Digital Presence Audit, AI crawler access is a documented item: which crawlers are allowed and which blocked, whether the text is readable from the source, and whether the answers actually mention you. You will not find a recommendation to add llms.txt in it, because the report recommends what changes the result, not what sells.