llms.txt: What It Is and How to Create One

llms.txt is a plain Markdown file you put at yoursite.com/llms.txt that hands AI models a short, curated map of your most important content. Think of it as a friendly table of contents written for a language model instead of a person: a title, a one-line summary, and a tidy list of links to the pages that actually matter. It was proposed by Jeremy Howard of Answer.AI in September 2024, and (being honest up front) it is an emerging convention, not a confirmed ranking factor.

That last sentence is the one most posts about llms.txt conveniently skip. So let me not skip it. This is a how-to, but it comes with a reality check bolted on, because I would rather you spend twenty minutes on this with clear eyes than treat it like a magic citation button. It is not one. Yet.

Here is the plan. I'll define the file, show you exactly how to write one, compare it to robots.txt and sitemap.xml so you stop confusing them, and then we'll do the uncomfortable part where I show you the data on whether anything actually reads it. Spoiler: mostly not, for now.

Key takeaways

• llms.txt is a curated Markdown map for AI models. It lives at yoursite.com/llms.txt, and it is one H1 title, a blockquote summary, and sectioned links to your best pages. Jeremy Howard of Answer.AI proposed it on September 3, 2024.
• Adoption is real, usage is not proven. According to Ahrefs, about 28% of 137,000 domains (roughly 38,000 sites) had an llms.txt in May 2026, but 97% of those files got zero requests that month. Lots of files, almost nobody reading them.
• No major provider has confirmed using it. Per Search Engine Land and Mintlify, Google, OpenAI, and Anthropic have not confirmed crawler use, though Anthropic, Cloudflare, Vercel, and Mintlify-hosted docs publish one.
• It is cheap, good hygiene, and not a citation hack. The file takes minutes, it cannot hurt you, and it might pay off later. But it will not move your AI citations today, and anyone selling it as a shortcut is selling.
• robots.txt controls access, llms.txt curates content. Different jobs. One says what bots may fetch, the other says what is worth reading. You can and should have both.

What llms.txt actually is

llms.txt is a single Markdown file, served from the root of your domain, that gives a large language model a concise, opinionated map of your site. Not every page. The pages you would point a smart intern to on day one.

The format is deliberately tiny. You start with one H1 (your site or project name). Then an optional blockquote with a single sentence that says what the site is. Then a few H2 sections, each holding a short list of Markdown links with a few words of description. That's it. No XML, no schema, no special syntax to memorize. If you can write a README, you can write this.

The reason it's Markdown and not some machine format is the whole point. Models read Markdown natively and cheaply. A full HTML page is full of nav bars, cookie banners, footer junk, and JavaScript that a model has to wade through to find the actual content. llms.txt skips all of that and just says: here is what we are, here is where the good stuff lives, go.

Who proposed it, and why

The convention came from Jeremy Howard, co-founder of Answer.AI, on September 3, 2024. Howard is not a random guy with a blog. He co-created the fastai library and ULMFiT, one of the techniques that helped kick off the modern transfer-learning era in NLP. So when he floated a proposal about feeding content to language models, people listened.

His argument was practical. Context windows are limited and expensive. Crawling a sprawling HTML site to find the three pages that matter is wasteful, and the model often gets it wrong. A curated Markdown index fixes both problems: it tells the model what's important and serves it in the format the model likes best. Clean idea. Whether the ecosystem actually wired it up is a separate question, which we'll get to.

llms.txt vs robots.txt vs sitemap.xml

This is where most people get tangled, so let's untangle it with a table. These three files all sit at your root and all talk to crawlers, but they do completely different jobs.

The short mental model: robots.txt is the bouncer (who gets in), sitemap.xml is the full guest list (every URL, no opinions), and llms.txt is the concierge (here are the five things actually worth your time). A sitemap is exhaustive and neutral. llms.txt is curated and opinionated. That difference is the entire value.

And no, llms.txt does not replace any of them. You keep robots.txt for access control, you keep sitemap.xml for traditional indexing, and you add llms.txt as a curation layer on top. If you want the deeper version of the access side of this, I wrote a whole piece on AI crawlers and robots.txt that pairs with this one.

Why not just use the sitemap?

Fair question. A sitemap already lists your URLs, so why bother? Because a sitemap is a phone book and llms.txt is a recommendation. The sitemap dumps every URL with no context and no ranking, which is great for an indexer that wants completeness and useless for a model that wants the gist fast. llms.txt says "start here, this is what we do, these five pages explain it." One is for machines that crawl everything. The other is for machines that want the summary.

The honest part: is anything actually reading it?

Here's the thing nobody wants to print in their how-to. Right now, the evidence says crawlers are mostly ignoring llms.txt. I'm not going to dress that up.

According to Ahrefs, in May 2026 about 28% of 137,000 domains they looked at (roughly 38,000 sites) had published an llms.txt file. That sounds like a movement. But here's the gut-punch from the same study: 97% of those files received zero requests for the file in May 2026. Zero. As in, the file existed, sat there politely, and almost nothing fetched it.

So adoption is high and usage is, charitably, unproven. Those are two very different things, and conflating them is how the hype gets made.

Nobody big has confirmed using it

It gets more sobering. No major AI provider has formally adopted llms.txt as a standard. Per Search Engine Land and Mintlify, neither Google nor OpenAI nor Anthropic has confirmed that their crawlers read or use the file. Google's John Mueller has publicly compared its current relevance to the keywords meta tag, which, if you remember that era, is not a flattering comparison.

There's a fun irony in the adopter list, though. Anthropic, Cloudflare, Vercel, and a pile of Mintlify-hosted docs sites all publish an llms.txt. So Anthropic ships the file but hasn't confirmed its own crawler consumes it. Which tells you the file is currently more "good citizen gesture and future bet" than "live, load-bearing infrastructure." That is fine. Just know which one you're signing up for.

Why publish a file that gets zero traffic?

Because cheap future-proofing is a perfectly rational bet when the downside is twenty minutes. Standards have a chicken-and-egg problem: crawlers won't read a file nobody publishes, and nobody publishes a file no crawler reads. Early adopters break the loop. If llms.txt becomes a thing models genuinely use in 2027, the sites that already have a clean, accurate one are ready on day one. If it never takes off, you spent twenty minutes and learned your own site structure better. That's a good risk-reward shape. Just don't tell your boss it's driving citations this quarter, because the data says it isn't.

How to write an llms.txt file

Okay, enough caveats. Let's build one. The structure is simple enough that you'll have it memorized after one read.

The required and optional pieces, in order:

1. An H1 with the name of your site or project. Required. There's exactly one.
2. A blockquote summary (a line starting with >) giving a short description of the site. Optional but you should include it, because it's the single most useful line in the file.
3. Optional prose paragraphs with any extra context a model needs. Keep it short.
4. H2 sections, each grouping related links. Common ones: Docs, Guides, Products, About. Under each, a bullet list of Markdown links, each followed by a colon and a short description.
5. An optional "Optional" section (yes, literally named that) for links a model can skip if it's short on context. This is your "nice to have, not essential" pile.

Here's a complete, working example for an agency site so you can see the shape.

# Outline Technologies

> Outline Technologies is an SEO, AEO, and GEO agency that gets
> brands cited by AI engines like ChatGPT, Perplexity, and Gemini.

Outline helps brands earn visibility in both Google Search and AI
answers. The pages below cover services, proof, and how we work.

## Services

- [SEO](https://outline.ad/services/seo): Rank on Google and in AI Overviews.
- [AEO & GEO](https://outline.ad/services/geo): Get cited by ChatGPT, Perplexity, and Gemini.
- [AI automation](https://outline.ad/services/ai-automation): Automate content and reporting workflows.

## Proof

- [Case studies](https://outline.ad/case-studies): Real growth, built with zero ad spend.
- [Results](https://outline.ad/results): Traffic and citation outcomes by client.

## About

- [How we work](https://outline.ad/how-we-work): Our process, start to finish.
- [Pricing](https://outline.ad/pricing): Engagement models and what each includes.

## Optional

- [Blog](https://outline.ad/blog): Long-form posts on SEO, AEO, and GEO.
- [Glossary](https://outline.ad/glossary): Plain-English definitions of AI search terms.

Notice what's doing the work here. Every link has a description, so the model knows what's behind it without fetching. The sections are obvious. The summary is one sentence a model could lift verbatim. And the whole thing fits on a screen. If your llms.txt scrolls for ten pages, you've missed the point: this is curation, not a sitemap with extra steps.

Where to put it and how to serve it

Save the file as llms.txt (lowercase, exactly that) and serve it from your root so it resolves at https://yoursite.com/llms.txt. Serve it as text/plain or text/markdown. If you're on a static host or a framework like Next.js, drop it in your public directory and you're done. On a CMS, you may need a route or a redirect, but it's a flat file, so nothing fancy.

One small gotcha worth flagging. Use absolute URLs in your links, not relative ones. A model that reads the file out of context won't know what /docs/quickstart is relative to, so spell out the full https://yoursite.com/docs/quickstart. Small thing, easy to miss, annoying to debug later.

The optional llms-full.txt

There's a companion file in the proposal called llms-full.txt. Instead of linking to your pages, it inlines the entire content of those pages as one long Markdown document. The pitch: a model gets everything in a single fetch, zero crawling required, no guessing.

For a docs site this can be genuinely useful, because someone asking an AI about your product gets your real docs handed over in one gulp. But two honest warnings. First, it gets big fast, and a giant file is its own problem. Second, it goes stale the second your docs change, and a confidently wrong stale doc is worse than no file at all. If you ship llms-full.txt, automate its generation from your real content so it can't drift. Hand-maintaining it is a trap.

Who llms.txt actually helps today

Let me be specific instead of hand-wavy, because "it depends" is a useless answer.

Documentation sites get the most out of it right now. If you run developer docs, especially on a platform like Mintlify that auto-generates the file, llms.txt and llms-full.txt let an AI coding assistant or a user's chatbot pull accurate, current docs in the format models prefer. That's a real, today use case, not a someday one. Docs are structured, factual, and frequently asked about, which is exactly the content this file was built for.

Your own internal tooling benefits immediately. This one's underrated. If you build a RAG system, a support bot, or an internal assistant over your own site, an llms.txt is a clean, curated entry point you control completely. You don't have to wait for OpenAI to read it, because your pipeline reads it. I've used a site's own llms.txt as the seed list for an internal retrieval index, and it beat crawling the whole site and guessing what mattered. You wrote the map, so use the map.

Everyone else is making a future-proofing bet. For a typical marketing site, the honest answer is that llms.txt does nothing measurable today. You publish it because it's cheap insurance, it forces you to think about your most important pages, and it's ready if the standard catches on. That's a fine reason. It's just not the same reason as "this will get me cited," and you should know which box you're checking.

What it does not do

It does not get you cited by ChatGPT this week. It is not a ranking factor. It will not rescue a site that AI engines can't crawl or don't trust. If your actual problem is that your brand isn't showing up in ChatGPT, llms.txt is not the fix, and I'd be doing you a disservice to imply otherwise. The fix is upstream: crawlable pages, answer-first content, real authority, and the structural work in the generative engine optimization guide. llms.txt is a nice tidy bow on top of that work, not a substitute for it.

Should you bother? The balanced verdict

Yes, probably, with calibrated expectations. Here's the cost-benefit laid out plainly.

So the verdict is: do it, because the downside is basically nothing and the upside is a free option. But file it under hygiene, not growth. If you've got an afternoon and a choice between writing an llms.txt and fixing the fact that AI engines can't actually find your pricing page, fix the pricing page first. Every time.

The trap to avoid is treating llms.txt as a finish line. It's a five-minute task that some folks turn into a whole content strategy, which is backwards. The strategy is being genuinely useful, well-structured, and crawlable. The file is a footnote to that. A nice footnote. Still a footnote.

How llms.txt fits the bigger picture

Zoom out and llms.txt is one small tile in the crawl-access mosaic. Before a model can cite you, a chain of things has to go right, and the file only touches one link in that chain.

The chain looks like this. First, AI crawlers have to be allowed to fetch your pages, and bots have to be able to render what they find, which is the access layer underneath all of AI search. Then your content has to be structured so a model can parse and trust it, which is where schema markup for AI search and answer-first writing earn their keep. Then you have to actually get retrieved and chosen, which is the deep mechanics behind how AI engines choose sources. llms.txt sits near the top of that funnel as a curation hint. Helpful, maybe. Sufficient, no.

If you're building a real program here, llms.txt belongs in the same checklist as your AI SEO fundamentals and your plan for getting cited by ChatGPT. It's one line item, not the whole list. And if you like crisp definitions, our LLM SEO glossary entry frames where this work sits in the broader picture.

Measure, don't assume

Here's the part the engineer in me cares about most. Whatever you do, including publishing an llms.txt, don't assume it worked. Measure it. AI answers wobble run to run, so a single check tells you nothing reliable. You want repeated sampling across engines, citation rate over time, and share of voice against competitors, with confidence intervals so you know the number is signal and not noise. That's exactly the job AI Citation Monitor does: it watches whether ChatGPT, Perplexity, Gemini, and Google AI Overviews actually cite you, across real buyer prompts, so you can tell the difference between a tactic that moved the needle and one that just felt productive.

Ship the llms.txt. Then check whether anything changed. If your citation rate moves, great, you learned something. If it doesn't, you've still got a clean content map and you didn't waste a Saturday. Either way you made a decision with data instead of vibes, and if you want the broader tooling landscape, we compared the best AI visibility tools so you can see how measurement fits.

FAQ

What is llms.txt?

llms.txt is a plain Markdown file you put at yoursite.com/llms.txt that gives AI models a short, curated map of your most important content. It is a single H1 title, a one-line blockquote summary, and a few sections of links with descriptions. It was proposed by Jeremy Howard of Answer.AI in September 2024, and it is an emerging convention, not a confirmed ranking factor.

Is llms.txt an official standard?

No. It is a community proposal, not a ratified web standard, and no major AI provider has formally committed to reading it. Google, OpenAI, and Anthropic have not confirmed that their crawlers use the file. It is adopted by a lot of sites, but adoption is not the same as usage.

Does llms.txt help SEO or AI citations?

There is no proof yet that it moves citations or rankings. Ahrefs found that 97% of sites with an llms.txt got zero requests for the file in May 2026, which means crawlers are mostly ignoring it for now. It is cheap, harmless hygiene that may pay off later, but do not expect citation magic from it today.

How do I create an llms.txt file?

Write a Markdown file with one H1 (your site or project name), a blockquote summary sentence, and then H2 sections (like Docs or Products) holding bullet links to your key pages with short descriptions. Save it as llms.txt and serve it from your root, at yoursite.com/llms.txt. Keep it short, curated, and accurate, because a stale map is worse than none.

What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers what they are allowed to fetch, written for bots in a rules format. llms.txt tells models what is worth reading and where it lives, written in human-readable Markdown. One is about access permissions, the other is about content curation. They solve different problems and do not replace each other.

What is llms-full.txt?

llms-full.txt is an optional companion file that inlines the actual full content of your key pages as one long Markdown document, instead of just linking to them. The idea is to hand a model everything in a single fetch with no crawling required. It is useful for docs sites, but it gets large fast and can go stale, so keep it scoped.