What is llms.txt in plain English?

It is a Markdown file at the root of your website that gives AI models a curated list of your most important pages, so they can find and read your best content without parsing messy HTML. Proposed by Jeremy Howard in September 2024, it is a community convention rather than an official web standard.

What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers where they cannot go (exclusion). llms.txt tells AI models which pages are most worth reading (curation). robots.txt is a near-universal standard with broad support; llms.txt is a newer proposal that only some AI tools currently read. They do different jobs and can coexist.

Do AI crawlers use llms.txt?

Some do, many do not. Anthropic (Claude), Perplexity, and developer tools like Cursor have said they read it in retrieval or coding workflows. Google Search ignores it entirely, and OpenAI's main crawlers request it rarely. Real-world crawler hits on llms.txt files are still very low.

Is llms.txt an official standard?

No. It is a proposed community convention with no backing from the W3C, IETF, or any recognized standards body, and no enforcement mechanism. AI providers support it on their own terms, if at all. That is the core reason to treat it as optional rather than required.

Does llms.txt help my Google rankings or AI Overviews?

No. Google has confirmed its search systems, including AI Overviews, do not read or act on llms.txt. It has zero effect on your classic rankings. If you want to show up in AI answers, the work that matters is entity clarity, structured data, citable content, and off-site authority, not this file. See our take on whether SEO is still worth it in AI search for where the real effort goes.

Is llms.txt worth adding in 2026?

If you are a developer-tools or docs-heavy company, yes, because the tools that read it are the ones your audience uses. For most other businesses it is harmless but low-impact, so add it after your real AEO fundamentals are in place, not before. It is housekeeping, not a growth lever.

What is llms-full.txt?

It is the optional companion to llms.txt. While /llms.txt is a curated index of links, /llms-full.txt bundles the full clean text of your core pages into a single file, so a model can ingest all of it in one request instead of fetching each page individually.

What Is llms.txt? Definition for AI Search

What is llms.txt? It is a proposed file you place at your domain root (yoursite.com/llms.txt) that hands AI models a clean, curated map of your most important content in Markdown. Think of it as a reading list for assistants like ChatGPT, Claude, and Perplexity. It is a community convention, not an official web standard, and most major AI crawlers do not yet rely on it. For most businesses it is a low-priority nice-to-have, not a starting point.

What is llms.txt?

llms.txt was proposed by Jeremy Howard, co-founder of Answer.AI, on September 3, 2024. The idea is simple: a large language model works with a limited context window, and your raw HTML is full of navigation, scripts, cookie banners, and markup that wastes that budget. By the time a model has parsed your header, footer, and three layers of div soup, it has burned tokens on content it will never cite. llms.txt is a plain-Markdown file at /llms.txt that points the model straight to the pages that matter, organized by topic, so it can find and read your best content without wading through the cruft.

It usually comes in two flavors:

/llms.txt is the index. A short, curated list of links to your key pages (docs, pricing, core guides), grouped under headings. It tells an AI agent where to go.
/llms-full.txt is the bundle. A single file containing the full clean text of your core content, so a model can ingest everything in one request instead of crawling page by page.

That is the whole concept: a curated, machine-friendly table of contents for the assistants that are increasingly answering your customers before they ever land on your site. It is the same instinct behind a good sitemap or clean technical SEO, pointed at language models instead of search crawlers.

llms.txt vs robots.txt: not the same job

People conflate these constantly. They solve different problems.

File	What it does	Tone
robots.txt	Tells crawlers where they are not allowed to go. Exclusion.	"Stay out of here."
sitemap.xml	Lists every URL that exists, for discovery.	"Here is everything."
llms.txt	Points AI models to the content that matters most, in clean Markdown.	"Read these first."

robots.txt is a gate. sitemap.xml is an inventory. llms.txt is a recommendation. One blocks, one catalogs, one curates. You can run all three at once and they do not conflict, because none of them is doing the other's job. A useful gut check: robots.txt controls access, sitemap.xml controls discovery, and llms.txt tries to control attention. Only the first two are enforced.

That last point is the real difference. robots.txt has near-universal crawler support and decades of precedent. llms.txt does not. It has no W3C or IETF backing, no enforcement, and no guarantee any given model will request it. Which brings us to the part most "add it today" articles skip.

Who supports llms.txt (and who ignores it)

Here is the honest 2026 read, because this is where the hype and the reality split.

Reads it, at least sometimes:

Anthropic (Claude) has said it respects llms.txt in retrieval workflows.
Perplexity has confirmed it fetches llms.txt to help prioritize which pages to pull.
Developer tools like Cursor and GitHub Copilot, and many retrieval-augmented generation frameworks, read it when present. This is where llms.txt earns its keep, because the file slots neatly into a RAG pipeline as a pre-built source list.

Largely ignores it:

Google Search does not read or act on llms.txt. Google has been explicit about this. It changes nothing about your blue-link rankings or your presence in AI Overviews.
OpenAI's main crawlers request it rarely. GPTBot pokes at it occasionally, not as a core part of how it sees your site.

Adoption studies put real-world usage somewhere in the 5 to 15 percent range of sites, and crawler-traffic monitoring shows AI bots requesting /llms.txt at negligible volumes relative to their total crawling. There is also a tell hiding in the data: big names have published one (Anthropic, Perplexity, Stripe, Cloudflare, Hugging Face, Zapier, among others), which makes the file look more adopted on the publishing side than it is on the consuming side. Plenty of sites have one. Few crawlers are reading them. A file nobody fetches is not a ranking signal; it is good housekeeping.

Does llms.txt help you get cited in AI answers?

Short version: not on its own, and not for most sites. Worth unpacking, because this is the question people mean when they ask about llms.txt.

The assistants that drive most consumer AI answers (ChatGPT and Google's AI Overviews) are not leaning on the file to decide what to cite. They cite content the way they always have: by crawling the open web, weighing authority, and pulling text that is clear, structured, and corroborated elsewhere. llms.txt does not override that. At best it makes your best pages slightly cheaper for a model to read once it has already decided to look at you. It does not get you onto the shortlist.

For the developer-tools crowd it is different, because the consumers and the readers overlap. The same coding assistants and RAG tools that fetch llms.txt are the surfaces where dev audiences ask questions. There, a clean index can genuinely shorten the path from "model reaches your docs" to "model quotes the right page." That is a real, narrow win, not a general one.

If your goal is broad AI visibility, the levers live elsewhere, and we have written about most of them: how to get cited in Google AI Overviews and the harder question of why your brand isn't being cited by ChatGPT. The short answer is that generative engine optimization, clean structured data, entity clarity, and third-party corroboration of your brand move citations. A Markdown index file, today, mostly does not.

Do you need an llms.txt file?

For most businesses right now: no. Not as a priority, anyway.

It is cheap to add and it will not hurt you, so "no downside" is fair. But "no downside" is not the same as "makes a difference." Adding llms.txt is table-stakes housekeeping, not a growth lever. If you are spending real effort here while your entity signals, structured data, and citable content are thin, you are polishing the doorknob before you have built the house.

llms.txt is worth adding if:

You are a developer-tools or documentation-heavy company( docs, APIs, dev platforms). This is the one segment where it genuinely earns citations, because the assistants and coding tools that read llms.txt are exactly the ones your audience uses. If that is you, it pairs naturally with the rest of aSEO program.
You already have your AEO fundamentals handled and want to cover the long tail of possible AI-readability signals.

It is not where to start if:

You are a service business, local brand, or content site hoping llms.txt is the trick that gets you cited in ChatGPT. It is not. The models that drive most consumer AI answers are not leaning on it.
Your structured data, topical authority, and off-site citations are not in place yet. That work is what gets you into AI answers. llms.txt is a footnote next to it.

Not sure where you stand today? Our AI visibility checker gives you a free read on how AI engines see your brand right now, which is a far more useful starting point than a file most crawlers ignore.

How to add one (if you decide to)

If you are in the segment where it makes sense, it takes about an hour:

Create a Markdown file named llms.txt.
Open with an H1 of your brand name and a one-line description of what you do.
Add a short summary blockquote, then group your most important links under H2 headings (Docs, Products, Guides, About).
Keep it curated. The point is signal, not a dump of every URL. That is what sitemap.xml is for, and a bloated llms.txt just reintroduces the noise the format was meant to strip out.
Upload it to your root so it resolves at yoursite.com/llms.txt. Optionally build llms-full.txt with the full text of your core pages.

A few things to get right while you are in there. Link to canonical, clean URLs, not redirects or tracking-laden ones. Keep the descriptions next to each link short and literal (what the page is, not marketing copy). And treat it like a living file: when your core pages change, the index should too, or you are handing models a stale map. There is no schema to satisfy, no validation gauntlet, and no approval body. That convenience is also a tell about how settled the standard is.

The honest bottom line

llms.txt is real, it is cheap, and someday it might matter a lot. Today it is a nice-to-have for most sites and a genuine asset for a narrow slice of developer-tools companies. If an agency is pitching llms.txt as the thing that gets you cited in ChatGPT, that is a tell. The actual work is bigger and harder, and it is the work we do every day.

Want to know where your brand stands in AI search, beyond a Markdown file? That is what our AEO and GEO services are built for. We will show you what moves citations, what does not, and exactly what it costs (our pricing is public, because hiding it is its own kind of tell). No quote-form games, no pressure, just real talk. Email us at admin@moonsauceagency.com or book 30 minutes.

Keep reading: What is generative engine optimization? · Retrieval-augmented generation (RAG), explained · Crawl budget · Back to the glossary

What is llms.txt? A no-hype explainer