What is llms.txt, and should your website have one? It is a proposed plain-text file, written in markdown and placed at yourdomain.com/llms.txt, that gives AI tools a short summary of your site and a curated list of links to your most useful content. The idea is simple: instead of making a language model dig through menus, scripts and cookie banners, you hand it a clean map.
Founders ask us about it because it keeps appearing in AI and SEO checklists. The honest answer is that llms.txt is cheap to add and can help some AI tools, but it is not a ranking factor, not a requirement, and not a replacement for the basics that actually decide whether assistants use your content.
What is llms.txt, in plain terms
The llms.txt proposal, published by Jeremy Howard in September 2024, suggests adding a markdown file at the path /llms.txt that provides brief background information, guidance and links to more detailed content. Markdown was chosen because language models read it easily, while the fixed structure still lets ordinary programs parse the file.
The problem it addresses is real. Web pages are built for people: navigation, ads, scripts and layout wrap around the actual information. Converting that back into clean text is messy, and a language model working with a limited context window cannot read an entire site. A short, curated file lets it find the right page quickly.
The same proposal also suggests offering clean markdown versions of important pages at the same URL with .md added, so a tool following a link from llms.txt gets text rather than a full HTML page.
The llms.txt format
The format is deliberately small. A file following the proposal contains these parts, in order:
- An H1 with the name of the project or site. This is the only required section.
- A blockquote with a short summary containing the key information needed to understand the rest of the file.
- Optional detail in paragraphs or lists (no headings) explaining how to interpret the site or the linked files.
- H2 sections of links, each a list of markdown links with an optional note after a colon.
- An "Optional" section by convention, for secondary links a tool can skip when it needs a shorter context.
Here is how those parts might look for a small software company. Each line below is one element of the file, from top to bottom:
| Part of the file | Example content |
|---|---|
| H1 (required) | # Example Co |
| Blockquote summary | > Example Co builds scheduling software for dental clinics in the US and UK. |
| Plain detail | Pricing is quoted per clinic after a demo. All product docs are in English. |
| H2 link section | ## Product, followed by a list linking to the features and integrations pages, each with a one-line note |
| Second H2 section | ## Support, linking to the help center and setup guides |
| Optional section | ## Optional, linking to lower-priority pages such as company history |
llms.txt vs robots.txt vs sitemap.xml
The three files are often confused because they all sit at the root of a site and all talk to machines. They do very different jobs.
| File | What it does | Who reads it | Status |
|---|---|---|---|
| robots.txt | Tells crawlers which URLs they may or may not request | Search and AI crawlers that honor it | Long-established, widely supported |
| sitemap.xml | Lists the URLs you want discovered, with details such as last update | Search engines | Long-established, widely supported |
| llms.txt | Summarizes the site and points language models to the most useful content | AI tools and agents that choose to look for it | Proposal; support varies by tool |
Google describes robots.txt as a way to manage crawler traffic, and notes it is not a mechanism for keeping a page out of Google; noindex or password protection does that. Sitemaps have hard limits, 50MB uncompressed or 50,000 URLs per file, because they are meant to be complete lists. llms.txt is the opposite: a short, hand-picked guide.
The most important difference: llms.txt does not grant or block access. If your robots.txt or firewall blocks an assistant's crawler, an llms.txt file will not change that. OpenAI, for example, documents access for its search crawler in terms of robots.txt rules for OAI-SearchBot, and says sites opted out of it will not be shown in ChatGPT search answers.
Common myths about llms.txt
Search for what is llms.txt and you will find confident claims that go well beyond the proposal itself. The ones we hear most often:
- "It is required for AI search." It is not. Google says no AI text files are needed for its AI features, and plenty of sites are cited in AI answers without one.
- "It blocks AI training." It does not. Access rules live in robots.txt and your server or CDN settings.
- "It replaces a sitemap." It does not. Search engines still rely on sitemaps and links to discover URLs.
- "More links are better." The opposite is closer to the truth. A long, unfiltered list defeats the point of a short, curated guide.
Does llms.txt help SEO or AI visibility?
For Google, no. Its guidance on AI features is explicit that you don't need to create new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode, and that there is no special structured data to add. Standard SEO practice is what counts.
Where llms.txt does earn its place is with tools that actively look for it: coding assistants reading developer documentation, AI agents asked to research a product, and chat assistants that fetch a page when a user shares a link. For a software product with an API or extensive docs, a good llms.txt can make those tools more accurate about how your product works.
Should your site have one?
Knowing what is llms.txt is the easy part. Whether it deserves a place on your site depends on who, or what, reads your content:
- Likely worth it: SaaS products, developer tools, APIs, documentation-heavy sites, and businesses whose buyers use AI agents for research.
- Low priority: small brochure sites with a handful of pages, where the pages themselves are already short and clear.
- Not a fix for: blocked crawlers, pages that only render with JavaScript, thin content or inconsistent brand facts.
If your key pages depend on client-side JavaScript to show their text, fix that first. Server-rendered HTML is easier for every crawler; our comparison of Next.js vs React explains why we default to it. And for the bigger picture on how assistants choose sources, see our guide to LLM SEO.
How to create and check an llms.txt file
- List the 10 to 30 pages that matter most: product, pricing explanation, key services, docs, policies.
- Write the H1 and a two or three sentence summary that states plainly what you do, for whom and where.
- Group links under H2 headings with a short note on what each page answers.
- Publish it at the root so it loads at yourdomain.com/llms.txt as plain text.
- Check it: open the URL in a browser, confirm it loads without a redirect loop or login, and click every link.
- Keep it in sync. Generate it from your site's content where possible so it updates when pages change.
Keep the facts identical to your website. A file that describes old services or contradicts your pricing page teaches tools the wrong thing. If your site runs on a CMS or a framework such as Next.js, it is usually straightforward to build the file from the same data that renders your pages, so a new service page appears in llms.txt the day it goes live.
Where llms.txt fits in an AEO plan
Once you know what is llms.txt and what it is not, it drops into its proper place: a small, useful extra at the end of an AI visibility checklist, after crawler access, indexable pages, answer-first content and consistent brand information.
Our SEO and AEO practice sets up llms.txt as part of that wider work, generated from the site's own content so it never drifts. If you want to know whether AI assistants can find and describe your business correctly, tell us what you're building and we will reply within one business day.
Frequently asked questions
Are llms.txt files worth it?
For documentation-heavy sites, SaaS products and APIs, usually yes: the file takes little time and helps AI tools that look for it find the right pages. For small brochure sites the benefit is modest. It should never take priority over crawler access and clear, indexable pages.
Is llms.txt mandatory?
No. It is a community proposal, not an official standard, and Google says you don't need AI text files to appear in AI Overviews or AI Mode. Sites can be crawled, indexed and cited in AI answers without one.
What does "LLM text" mean?
It usually refers to llms.txt, where LLM stands for large language model, the technology behind assistants such as ChatGPT, Claude and Gemini. The .txt file is plain text written in markdown so these models can read a summary of a site and follow links to its key content.
How to check llms.txt file?
Open yourdomain.com/llms.txt in a browser and confirm it loads as plain text without a login or redirect loop. Check that it starts with an H1 and a short summary, that every link works, and that the facts match your current website.
What is llms.txt used for in practice?
It gives AI tools a curated starting point: a summary of what the site is and links to the pages that answer the most common questions. It is most used by coding assistants and AI agents reading product documentation, where accurate, current detail matters most.
