Skip to content
Flowpane
Level 3 / Method

llms.txt

llms.txt gives language models a curated Markdown map of your best pages. It is only useful while every link is current and reachable by AI crawlers.

Updated 29 September 2026·4 min read·Weight 7 of 100·Markdown

llms.txt is a Markdown file at the root of a site that gives language models, and the assistants and agents built on them, a short curated map of what matters: a name, a one-paragraph summary and grouped links to the best sources. It comes from the community proposal at llmstxt.org, not an RFC, and adoption is optional. Its value depends entirely on whether the links still point somewhere true.

What it is for

A model answering a question about your organisation has a limited context budget. llms.txt tells it which pages to read first, in what order of importance, and what each one covers. The readers are AI assistants and developer tools that fetch context at the moment they answer, plus the people who configure them.

It is not an access control. llms.txt does not grant or refuse crawler access, does not permit or refuse AI training and does not affect indexing. Access belongs in robots.txt and usage preferences in ai.txt. llms.txt is editorial: it points, it does not permit.

The structure

The proposal defines a small, predictable shape so that tools can parse it without guessing:

  1. One H1 with the name of the site or project. This is the only element the proposal treats as essential.
  2. A blockquote with a short summary that a reader could quote as it stands.
  3. Optional prose with context that does not fit a link.
  4. H2 sections, each holding a list of Markdown links in the form - [Name](url): description.
  5. An optional section headed ## Optional for secondary resources that a reader can skip when context is short.

Why curation beats a dump

A sitemap is the complete inventory. llms.txt is the editor's cut. Generating it from every URL in the sitemap produces a file that repeats the sitemap, spends the reader's context budget on tag archives and pagination, and buries the ten pages that actually answer questions. The Optional section exists to separate priority from completeness, and it only works if someone makes that call.

Flowpane recommends a curation review once a file passes 200 links and notes lists above 100 as information. Neither figure is a protocol limit. They are the point at which a list usually stops being a selection.

How it goes stale

llms.txt is almost always written by hand, once, and then left alone while the site changes around it.

  • Moved pages. A redesign or migration changes the URL structure. The old links now redirect or return 404, and nothing prompts anyone to edit a text file nobody browses.
  • Newly blocked pages. Someone adds a Disallow for AI crawlers on /docs/. llms.txt still sends models to /docs/, so the two files now tell robots-respecting crawlers different stories.
  • An outdated summary. The blockquote still describes last year's product or positioning, and it is the sentence most likely to be quoted.
  • An HTML fallback. A catch-all route serves the homepage template with status 200 at /llms.txt. It looks present to a quick check; it is not a Markdown file.
  • Broken syntax. Hand edits leave bare URLs, unclosed brackets or a second H1.

What true looks like

# Example Ltd

> Example Ltd makes payroll software for small businesses. This file lists the documentation and policies most useful for answering questions about the product.

## Documentation

- [Getting started](https://www.example.com/docs/getting-started.md): Account setup and first payroll run
- [API reference](https://www.example.com/docs/api.md): Endpoints, authentication and rate limits

## Policies

- [Pricing](https://www.example.com/pricing): Current plans and what each includes
- [Security](https://www.example.com/security): How customer data is protected

## Optional

- [Changelog](https://www.example.com/changelog): Release history
  • One H1, then a summary you would be content to see quoted.
  • Descriptive Markdown links, never bare URLs, and each link resolves without a redirect chain.
  • No link points at a path that robots.txt blocks for the AI crawlers you want to read it.
  • Review the file in the same release that moves or retires URLs.

How Flowpane checks it

Flowpane fetches /llms.txt over public HTTPS and assesses it against the proposal's structure. The file carries a weight of 7 in the site score.

  • Missing. Scores 0 and is labelled a recommended file: optional on the open web, but recommended and scored by Flowpane.
  • Empty, or HTML instead of Markdown. Scores 0. An HTML response needs a routing fix, so Flowpane does not offer a text patch for it.
  • Structure. A missing H1, several H1s, malformed Markdown links and bare URLs are Issues. A missing summary or no H2 sections are Recommendations.
  • Curation. Above 200 links is a Recommendation to review; 101 to 200 links, no Optional section, or mostly HTML targets are Information and never deduct.
  • Links. Paid plans can verify the listed URLs. Separately, the coherence engine reports llms.txt links that robots.txt blocks for named AI crawlers.

For a missing file, Flowpane drafts a starter with an H1, a placeholder summary and one link to the observed homepage. It never invents descriptions or URLs. A draft does not change the score: you publish the file through your own host or CMS, and the next public check scores it.

Previousads.txtNext ai.txt