What to put in llms.txt (and what to leave out)

A file written for a context window

The llms.txt proposal, published by Jeremy Howard in September 2024 at llmstxt.org, starts from a constraint: a model's context window cannot hold a website. Converting HTML pages with navigation and scripts into clean text is imprecise, so the proposal asks site owners to publish a small Markdown file at /llms.txt that says what the site is and links to the pages worth reading, ideally as Markdown.

That framing decides what goes in. The file is not for discovery of everything (that is the sitemap) and not for access rules (that is robots.txt). It is the page you would hand a capable stranger who has five minutes and one question about your site.

The format

The specification is short. An H1 with the site or project name is the only required element. A blockquote under it carries a summary with the key information needed to understand the rest. Zero or more H2 sections follow, each a list of Markdown links, one per line, with an optional note after a colon. A section titled Optional marks links that can be dropped when a shorter context is needed. Plain paragraphs between headings are allowed for guidance.

# Brand

> Brand sells made-to-measure blinds online in Switzerland. Orders ship in
> 10 working days. Prices include Swiss VAT. Support in DE, FR and EN.

## Start here
- [How ordering works](https://www.brand.ch/how-it-works.md): measuring, ordering, delivery
- [Pricing](https://www.brand.ch/pricing.md): per square metre, by fabric group
- [Returns and warranty](https://www.brand.ch/help/returns.md): 5-year warranty, remake policy

## Catalogue
- [Product feed](https://www.brand.ch/feed/products.xml): all products, prices, stock
- [Fabric collections](https://www.brand.ch/collections.md)

## Machine access
- [Automated-access policy](https://www.brand.ch/ai-policy.md)
- [OpenAPI](https://www.brand.ch/openapi.json): stock and price lookup
- [Contact](https://www.brand.ch/.well-known/security.txt)

## Optional
- [Company history](https://www.brand.ch/about.md)
- [Press](https://www.brand.ch/press.md)

The .md links follow the second half of the proposal: pages worth reading should have a clean Markdown version at the same URL with .md appended. Where you cannot do that, link the HTML page; a link is better than no link. Sites behind a CDN that offers Markdown negotiation can skip the .md files entirely and serve Markdown on request, as the Markdown negotiation article explains.

What to leave out

• Every URL. A 400-line llms.txt is a sitemap with worse syntax. If an agent wants everything it reads the sitemap. Pick the 10 to 30 pages that answer the questions people actually ask.
• Marketing copy. The summary is for orientation: what you sell, where, to whom, on what terms. Adjectives waste the tokens you are trying to save.
• Access rules. Allow and Disallow belong in robots.txt. A policy statement belongs on a policy page you link from here.
• Stale links. Regenerate the file from the same source as your sitemap or navigation. A file that 404s on the third link teaches the reader to stop.

If you need a long version, the convention is a second file, /llms-full.txt, with the expanded content. Qomvia publishes both: a short /llms.txt that points at the leaderboard, methodology and API, and a /llms-full.txt that lists every scored site on one line each. The llms.txt guidance file check in the methodology gives 4 points for the short file and 5 when both exist.

Does anyone read it?

Adoption among AI operators is uneven and mostly undocumented; Cloudflare's readiness scanner, for instance, does not check for it by default. That is a reason to keep the file cheap, not a reason to skip it. It costs one screen of Markdown, it is the only place a model can read a plain-language statement of what your site is for, and it is where you point at the machine resources (feed, API, policy, contact) that agents otherwise have to guess at.

Questions

Where does the file go?

At the root of the host: https://www.example.com/llms.txt. Subdomains get their own. Serve it as text/plain or text/markdown.

Can I generate it automatically?

Yes, and you should, from the same data as your navigation or sitemap, so links stay valid. Curate which sections are included; do not include everything just because it is available.

What is llms-full.txt?

A convention for the expanded version: the same structure with the linked content inlined or a much longer list. Keep llms.txt short and put the bulk there.