llms.txt

What it is

`/llms.txt` is a proposal to standardise a file that a website publishes to help AI agents use it ^[1]. The file has a fixed name, so a tool can look for it at a known place instead of being told in advance where to go, and what it contains is information about the site written for a machine reader rather than for a person browsing ^[1]. The proposal was published by Jeremy Howard on 3 September 2024 ^[1].

Why it exists

The argument starts with how web pages are made. HTML pages are built for people: the useful text is wrapped in navigation, advertising and JavaScript, and converting that mixture back into clean text is hard ^[1]. An agent that fetches such a page therefore spends much of its effort on material it does not need, and has to guess which parts matter.

The second half of the argument is about size. Context windows — the amount of text a model can hold in view at once — have grown, but they are still too small for most websites in their entirety ^[1]. And every wasted token costs time and money ^[1]. A token is the small chunk of text a model reads and is charged for, so filling the window with menus and scripts is pure overhead: it slows the work down and adds to the bill without adding anything the agent can use.

Put together, the two points describe a mismatch. Sites are written for human readers and their browsers; agents are given a limited budget of text and need the substance quickly. A short, purpose-written file at a known location is meant to close that gap.

How it works

The format is Markdown-based ^[1]. Markdown is plain text with a few light conventions for headings, lists and links, which makes it readable both to a person opening the file and to a model consuming it, without the surrounding machinery of a rendered page. A site can use the file to describe what it offers and to point at the parts that matter, rather than leaving an agent to infer structure from markup.

The proposal document itself is organised into a proposal section, a format section, an example and a list of directories ^[1]. That structure reflects what a convention of this kind needs in order to be adopted: a statement of the problem and the idea, a precise description of what a valid file looks like, something concrete to copy, and a list of places where such files can be found.

The second version

Version 2 of the proposal was modified on 10 August 2026 ^[1]. It was not a rewrite from first principles. The revision was made on the evidence of two years of adoption, during which thousands of sites published an llms.txt file ^[1]. Two years of real files, written by many different publishers for many different kinds of site, is exactly the material needed to see which parts of an early convention hold up and which need adjusting.

What it does and does not settle

It remains a proposal ^[1]. No one is required to publish an llms.txt file, and a site that does so is making a voluntary choice about how it presents itself to automated readers; by the time version 2 was written, thousands of sites had made that choice ^[1]. The case for the file rests on the two pressures described above — pages that are awkward to convert into clean text, and context windows that are still too small to swallow a whole site — and it will keep resting on them for as long as both remain true.