What is llms.txt — and does your site need one?

The proposal comes from Jeremy Howard of Answer.AI, published September 3, 2024 at llmstxt.org. The idea: HTML pages are heavy and cluttered for language models, so a site should offer a curated markdown map at /llms.txt — one H1 with the site's name, a blockquote summary with "key information necessary for understanding the rest of the file," then sections of links, each one a markdown hyperlink with an optional one-line note. An Optional section marks links an agent can skip when a shorter context is needed. The spec also recommends clean markdown twins of your pages at the same URL with .md appended.

Two things llms.txt is not. It does not control access — it grants and denies nothing; that job belongs to robots.txt, and we've mapped every AI crawler's switches in our robots.txt guide. And llms-full.txt — the single file carrying the full text of your site — is not in the spec at all: it's an ecosystem convention, popularized in November 2024, when Mintlify started auto-generating both files for every documentation site it hosts.

Published measurements show very little use of llms.txt by AI crawlers.

No AI vendor has committed to consuming it. Not OpenAI, not Anthropic, not Google, not Perplexity — no official documentation from any of them says their systems fetch or parse llms.txt. Google has been openly dismissive: John Mueller wrote in April 2025 that "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag," and Gary Illyes confirmed in July 2025 that Google doesn't support it and doesn't plan to.

The server logs agree. Ahrefs checked 137,000 domains in June 2026: 97% of published llms.txt files received zero requests in the prior month, and even for the 3% that got any traffic, only about 1% of it came from AI-retrieval bots. Evil Martians combed two months of logs on their own site: of roughly 770 fetches of the file, 37 came from named AI agents. Otterly watched 62,100 AI-bot visits over 90 days: 84 touched /llms.txt — a third of what an average content page gets. And SE Ranking tested 300,000 domains for a relationship between having the file and being cited in AI answers: none found.

Meanwhile adoption keeps climbing — from about 4,100 published files in June 2025 to 36,100 a year later, an almost nine-fold rise in files almost nothing reads. The publisher list is impressive and ironic in equal measure: Anthropic, Cloudflare, Zapier, Stripe and Vercel all serve one — mostly on their docs domains — and Perplexity publishes its own llms.txt while never claiming to read anyone else's.

One genuine signal hides in the same logs, and it points at a different part of the proposal: Evil Martians found Claude Code requesting markdown via content negotiation in 76% of its fetches. Machines do prefer clean markdown over HTML — it's the index file they ignore, not the idea.

Our site serves /llms.txt and per-locale llms-full files, and we claim no effect from them: we have no evidence any assistant found our content through the map rather than through crawling and search indexes, and we assume our file fares no better than the measured average. So why keep it?

The file is generated from the same content registry as our pages, sitemap and guide index. When a guide is published or changed, llms.txt updates with it. That makes it inexpensive to keep in case more tools adopt it. A manually maintained file would add work and risk falling out of sync.

Running one in production also surfaces real engineering constraints the think-pieces skip. Our llms-full export has outgrown its size cap several times since the Hindi locale landed — Devanagari runs about three bytes per character in UTF-8, and even with Hindi's shorter texts the same guides weigh roughly twice what they do in English — so the cap has gone from 100 kilobytes in August 2026 to 2 megabytes today. If your llms-full.txt is bigger than a model's context budget, you've recreated the problem the file was meant to solve.

We publish the file as an experiment that costs us nothing to maintain. It is part of the technical setup documented in our guide on how we do GEO on our own site. That guide also covers crawler access, extractable pages and consistent company information, which have a measurable role in citation work.

If llms.txt is free in your stack — a plugin, a static-site hook, a CMS template — here is the llms.txt format, straight from the spec:

A minimal llms.txt example: a single H1 with your site's name, then a blockquote that summarizes what the site is in two or three sentences an agent could quote. H2 sections — pages, guides, docs — each a list of markdown links with a one-line note per link. An Optional section for what an agent can skip. That's the whole standard.

The three mistakes that make the file worse than not having one: