llms.txt Explained: Do You Need One, and How to Create It

llms.txt gets pitched as everything from "robots.txt for AI" to a ranking shortcut. It is neither. This guide covers what the file actually is, what publishing one does and does not do, how to decide whether your site needs one at all, and three practical ways to create it.

What llms.txt actually is

llms.txt is a convention proposed in 2024: a Markdown file served at your domain root (https://example.com/llms.txt) that gives language models and AI agents a curated, readable map of your site's important content. Where robots.txt tells crawlers what they may not do, llms.txt is the opposite gesture — a short, human-written index of what your site is and where its substantive pages live, in a format models parse well.

The conventional structure is simple: one H1 with the site or project name, a short blockquote summary, then a few H2 sections containing link lists with one-line descriptions. A companion convention, llms-full.txt, inlines the actual page content instead of linking to it, for consumers that want everything in one fetch.

# Project Name

> One-paragraph summary of what this site is and who it serves.

## Products

- [Product page](https://example.com/product/): what it is, price, what it includes
- [Free tool](https://example.com/tool/): what it does, what it requires

## Docs

- [Getting started](https://example.com/docs/start/): setup in five steps

A live example: this site's own llms.txt lists every product with its price, checkout path, and honest scope notes, so an AI assistant summarizing the site has current facts to work from instead of a guess.

What publishing one does not do

Honesty first, because this is where most llms.txt content oversells:

• It is not a standard. It is a community convention. No major model provider has committed to fetching it on a schedule, and publishing the file does not force any search engine, model, or agent to use it.
• It is not a ranking or citation guarantee. There is no verified mechanism by which llms.txt improves search rankings or makes AI assistants cite you. Anyone claiming otherwise is selling something.
• It is not access control. It cannot stop a crawler from reading anything. If you want to restrict AI training crawlers, that conversation lives in robots.txt user-agent rules, not here.

What it does do is narrower and still useful: when a model or agent does fetch it — some do today, ad hoc, and agent frameworks increasingly check for it — it finds your curated summary with current facts instead of reconstructing your site from whatever pages happen to be in context. For a site that sells things, that means correct prices and honest scope statements rather than hallucinated ones.

Decide whether you need one — and document the decision

A useful decision rule: llms.txt earns its keep when your site has facts that go stale (prices, availability, versions, terms) or structure a model would misread (many similar pages, mixed free/paid content, multiple products under one domain). A five-page portfolio with static content gains little; a storefront, documentation site, or product catalog gains more.

Either answer is legitimate. What matters operationally is that the decision is documented — "we publish llms.txt and regenerate it when the catalog changes" or "we reviewed the convention and decided it is not applicable because X." A recorded not-applicable decision is a complete, defensible outcome; an undocumented shrug is not. (This is exactly how the free evidence-led scorecard treats it: the llms.txt check awards full credit for a documented decision either way, and zero for an unresolved one.)

Three ways to create the file

1. By hand. For a small site this is a 30-minute job and produces the best file, because you curate it. Write the H1, the summary blockquote, and sections linking your genuinely important pages with one honest line each. Skip boilerplate pages; a curated 30-line file beats a 3,000-line sitemap dump.

2. Generate it from a crawl. For larger sites, generating a first draft from the site itself is faster and catches pages you forgot. This is the job of my llms.txt Generator, a paid Apify Actor (from $5 per 1,000 results): point it at a domain, cap the page count, and it discovers same-domain pages, extracts main content, and writes llms.txt, an optional llms-full.txt, and a per-page dataset exportable as JSON or CSV. Start with a 10-page crawl on a site you control and inspect every output before publishing — generated drafts are a starting point for curation, not a replacement for it.

3. Keep it fresh. A stale llms.txt actively misinforms — an old price in your llms.txt is worse than no file, because it presents wrong facts with confidence. Regenerate or re-edit the file whenever listed facts change, and put a reminder or change-monitoring step behind it. If the file will not be maintained, prefer the documented not-applicable decision over publishing something that will rot.

Publishing checklist

• Serve the file at the domain root as plain text/Markdown: /llms.txt (and /llms-full.txt if used).
• Read the whole file once as a stranger: is every fact current, every price right, every claim one you can defend?
• Keep prices and terms consistent with the pages you link — the file is a promise that those pages agree with it.
• Record the maintenance rule: what event triggers a regeneration, and who does it.
• Re-check the file whenever products, prices, or scope statements change.