robots.txt Told Crawlers Where to Stop. llms.txt Tells Agents Where to Go

What the file does, how to write one, and why retrieval is the part that actually matters.

robots.txt Told Crawlers Where to Stop. llms.txt Tells Agents Where to Go

What the file does, how to write one, and why retrieval is the part that actually matters.

American Visonary Art Museum Baltimore, MD

When an AI agent visits your website looking for information, it doesn’t read the way a person does. A person scans. They skip the nav, ignore the footer, find the paragraph they need, and move on. An agent fetches the raw HTML, and that HTML includes everything: the cookie consent banner, the “you might also like” sidebar, the breadcrumb trail, three levels of nested menu, and finally, somewhere in the middle, the content the agent actually came for.

robots.txt taught crawlers where they weren’t welcome. llms.txt is an attempt to teach agents where they should go.

The proposal came from Jeremy Howard of fast.ai in September 2024. The idea is simple: drop a plain text file at the root of your domain — /llms.txt, and use it to tell any AI agent what your site is about, which pages are most important, and where the signal lives. Think of it as the onboarding packet you hand a new employee instead of pointing them at the intranet and saying, "figure it out."

What the file actually looks like

llms.txt uses Markdown. No new syntax to learn. There’s a title, an optional description block, and one or more sections of links with short descriptions. Each link points to a page on your site along with a one-sentence explanation of what’s on it.

Here’s a minimal but real example for a software documentation site:

# Acme API Docs
> Acme provides a REST API for managing customer orders, inventory, and fulfillment.
> Authentication uses OAuth 2.0. All endpoints return JSON.
## Getting Started
- [Quickstart](https://docs.acme.com/quickstart): Create your first order in five minutes.
- [Authentication](https://docs.acme.com/auth): How to obtain and refresh access tokens.
- [Rate Limits](https://docs.acme.com/rate-limits): Default limits and how to request an increase.
## Core Concepts
- [Orders](https://docs.acme.com/orders): The order lifecycle, from draft to fulfilled.
- [Inventory](https://docs.acme.com/inventory): How stock levels are tracked and reserved.
- [Webhooks](https://docs.acme.com/webhooks): Subscribing to fulfillment events.
## Reference
- [API Reference](https://docs.acme.com/api): Full endpoint reference with request and response schemas.
- [Error Codes](https://docs.acme.com/errors): Every error code, what it means, and how to recover.
- [Changelog](https://docs.acme.com/changelog): What changed in each release.

That’s the whole file. No schema to validate, no configuration to deploy. It lives athttps://docs.acme.com/llms.txt, and any agent that knows to look for it can find it.

There’s also a companion file: /llms-full.txt. Where llms.txt is an index, llms-full.txt is the content itself — the same information, but with the full text of each linked page concatenated in. An agent that wants everything in one request fetches llms-full.txt instead of following links. The tradeoff is size: a full site can produce a very large file, which is why the index format exists.

Training vs. retrieval — where the file actually helps

To understand what llms.txt does and doesn’t do, you have to separate two things that get conflated: training and retrieval.

When a model is trained, it processes enormous amounts of web content and compresses what it learned into weights. That training has a cutoff date and it’s done once. llms.txt does nothing for a model trained last year. The file wasn’t there when the training data was collected, or if it was, the pipeline had no special reason to prioritize it. The base model’s knowledge of your product is whatever it absorbed from your HTML and any third-party writing about you. llms.txt doesn’t change that.

Where llms.txt matters is at inference time, when an agent is doing live retrieval. A user asks an AI assistant a question. The assistant, rather than answering from training, fetches current information from the web, reads it, and builds its answer from what it found. Perplexity works this way. Claude with web search works this way. An internal support bot built on RAG works this way.

When that agent visits your site, it faces a choice: fetch every page and parse the HTML, or look first for a guide that tells it which pages matter. llms.txt is that guide.

Here’s what an agent pulls from a typical docs page without one:

Orders - Acme API Docs ... We use cookies to improve your experience... Accept

In this section

Orders

An order represents a customer's intent to purchase...

...actual content here... © 2026 Acme Inc. | Privacy Policy | Terms of Service

A capable agent can extract the main content from this. It’ll strip the nav, the sidebar, and the footer. But that extraction costs tokens, and on a site with hundreds of pages, the agent still has no signal about which pages to visit first. It’s wandering.

With llms.txt available, an agent that checks for it first gets a map: eight pages, each described in a sentence, organized by purpose. It can fetch exactly the pages relevant to the question and skip everything else. The token cost drops. The answer quality goes up. The agent isn't wandering; it was handed an org chart.

What this looks like in practice

Say someone asks an AI assistant: “How do I subscribe to fulfillment events in the Acme API?”

Without llms.txt, the agent searches for “Acme API documentation,” lands on the homepage, finds a docs link, crawls the docs root, parses the navigation, and eventually reaches the webhooks page; three or four fetches in, with accumulated noise at each step.

With llms.txt, the agent fetches https://docs.acme.com/llms.txt, reads the Webhooks entry ("Subscribing to fulfillment events"), fetches that single page, and answers with clean content. One fetch. No wasted context.

The same logic applies to RAG systems, where your documentation is the knowledge base. A retrieval pipeline that indexes llms.txt first builds a cleaner document store: structured entries, meaningful descriptions, no navigation boilerplate. When a user asks a question, retrieval hits the right chunks instead of pulling fragments of cookie banners and sidebar links alongside the actual answer.

Writing one that actually helps

The file is easy to write badly. A few things that matter:

The description does the work, not the link. A link with no description forces the agent to guess what’s on the page from the URL. “Getting Started” is a guess. “Create your first order in five minutes” is an instruction. Write the description as the one sentence you’d say to a new engineer on their first day.

Sections are a priority signal, not a sitemap. The structure isn’t for human readers. An agent that runs out of context budget will stop reading, so the pages that answer the most common questions belong at the top, not wherever they fall in your information architecture.

A stale llms.txt is worse than none. If it points to a page that no longer exists, the agent fetches a 404, and the map is broken. Treat it like a CHANGELOG and update it when something significant changes.

**llms-full.txt** is worth building for documentation sites. If your docs are the primary thing agents need, no sensitive pricing, no account-specific data, then generating a concatenated full-text file at build time costs almost nothing and means an agent can get everything in one round trip. Most static site generators can do this with a template.

Who’s using it

Adoption is still thin. The sites most likely to have llms.txt today are developer tools companies and open-source projects that think carefully about how agents consume their documentation. Anthropic’s own docs have one. Stripe has experimented with it. Fast.ai, where the proposal originated, implemented it immediately.

The pattern makes sense. Developer-facing products with rich documentation, built for an audience that already uses AI assistants heavily. A developer asking Claude or Perplexity about an API wants a current, accurate answer, and the company building that API would rather the answer come from their docs than a six-month-old Stack Overflow thread.

Consumer sites have been slower. A news organization, a restaurant, a retail store, the case is less obvious when the content isn’t technical documentation. That’ll probably change as agents become a more common entry point for information, but for now, this is mostly a developer ecosystem story.

What it doesn’t fix

llms.txt is a convention, not a standard. There’s no enforcement mechanism. An agent doesn’t have to check for it, and most don’t yet. The file is only useful if the agent doing the retrieval was built to look for it, the same way a browser is built to check robots.txt. That behavior is still inconsistent across tools. Some retrieval pipelines check for it. Many don't.

It also doesn’t control what gets indexed for training. If a scraper is collecting data for a future model and you don’t want your content included, llms.txt does nothing for you. robots.txt with the appropriate user-agent restrictions is the right tool for that, and even then enforcement depends on good-faith compliance.

And it doesn’t fix the base model’s knowledge of you. If a model was trained before your llms.txt existed, its understanding of your product is whatever it absorbed from the web at that point. The file helps agents retrieve accurate current information at inference time. It doesn’t rewrite history.

By Joshua McDonald on June 4, 2026.

Canonical link

Exported from Medium on August 26, 2026.