What is llms.txt — and does your site need one?

The proposal comes from Jeremy Howard of Answer.AI, published September 3, 2024 at llmstxt.org. The idea: HTML pages are heavy and cluttered for language models, so a site should offer a curated markdown map at /llms.txt — one H1 with the site's name, a blockquote summary with "key information necessary for understanding the rest of the file," then sections of links, each one a markdown hyperlink with an optional one-line note. An Optional section marks links an agent can skip when a shorter context is needed. The spec also recommends clean markdown twins of your pages at the same URL with .md appended.

Two things llms.txt is not. It does not control access — it grants and denies nothing; that job belongs to robots.txt, and we've mapped every AI crawler's switches in our robots.txt guide. And llms-full.txt — the single file carrying the full text of your site — is not in the spec at all: it's an ecosystem convention, popularized in November 2024, when Mintlify started auto-generating both files for every documentation site it hosts.

This is the question most guides tiptoe around — do AI crawlers actually use llms.txt? — and the measured answer is blunt: almost nobody.

No AI vendor has committed to consuming it. Not OpenAI, not Anthropic, not Google, not Perplexity — no official documentation from any of them says their systems fetch or parse llms.txt. Google has been openly dismissive: John Mueller wrote in April 2025 that "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag," and Gary Illyes confirmed in July 2025 that Google doesn't support it and doesn't plan to.

The server logs agree. Ahrefs checked 137,000 domains in June 2026: 97% of published llms.txt files received zero requests in the prior month, and even for the 3% that got any traffic, only about 1% of it came from AI-retrieval bots. Evil Martians combed two months of logs on their own site: of roughly 770 fetches of the file, 37 came from named AI agents. Otterly watched 62,100 AI-bot visits over 90 days: 84 touched /llms.txt — a third of what an average content page gets. And SE Ranking tested 300,000 domains for a relationship between having the file and being cited in AI answers: none found.

Meanwhile adoption keeps climbing — from about 4,100 published files in June 2025 to 36,100 a year later, an almost nine-fold rise in files almost nothing reads. The publisher list is impressive and ironic in equal measure: Anthropic, Cloudflare, Zapier, Stripe and Vercel all serve one — mostly on their docs domains — and Perplexity publishes its own llms.txt while never claiming to read anyone else's.

One genuine signal hides in the same logs, and it points at a different part of the proposal: Evil Martians found Claude Code requesting markdown via content negotiation in 76% of its fetches. Machines do prefer clean markdown over HTML — it's the index file they ignore, not the idea.

Our site serves /llms.txt and per-locale llms-full files, and we claim no effect from them: we have no evidence any assistant found our content through the map rather than through crawling and search indexes, and we assume our file fares no better than the measured average. So why keep it?

Because in our architecture it costs nothing and cannot rot. The file is generated from the same content registry that builds the pages, the sitemap and the guide hub — when a guide ships or changes, llms.txt updates in the same commit. A map that maintains itself is a free option on a future where some agent does read it; a map you have to maintain by hand is a liability that will quietly drift out of sync with the site.

Running one in production also surfaces real engineering constraints the think-pieces skip. Our llms-full export has outgrown its size cap twice since the Hindi locale landed — Devanagari runs about three bytes per character in UTF-8, and even with Hindi's shorter texts the same guides weigh roughly twice what they do in English — so the cap now sits at 300 kilobytes, triple the original. If your llms-full.txt is bigger than a model's context budget, you've recreated the problem the file was meant to solve.

That's the honest shape of the decision: we publish it as a zero-cost bet and describe it as exactly that — part of the technical layer documented in our guide on how we do GEO on our own site, not a ranking lever. What has a measurable effect on citations is covered there: crawler access, extractable pages, entity work.

If llms.txt is free in your stack — a plugin, a static-site hook, a CMS template — here is the llms.txt format, straight from the spec:

A minimal llms.txt example: a single H1 with your site's name, then a blockquote that summarizes what the site is in two or three sentences an agent could quote. H2 sections — pages, guides, docs — each a list of markdown links with a one-line note per link. An Optional section for what an agent can skip. That's the whole standard.

The three mistakes that make the file worse than not having one: