llms.txt Explained: What It Is and Does Your Site Need One?
llms.txt is a proposed plain-text Markdown file placed at the root of a website that gives AI systems a short summary of the site and links to its most important pages. It’s easy to add and does no harm, but it is not an official standard, support is limited, and it is no substitute for crawl access, structured data and clear content.
Where did llms.txt come from?
The format was proposed in 2024 by Jeremy Howard of Answer.AI. The idea is simple: websites are built for humans and browsers, full of navigation, scripts and ads. A large language model reading a site would benefit from a clean summary that says “this is what we are, and these are the pages that matter”.
How is llms.txt different from robots.txt and sitemap.xml?
| File | Purpose | Audience |
|---|---|---|
| robots.txt | Allows or blocks crawling of paths | Crawlers |
| sitemap.xml | Lists every URL you want indexed | Search engines |
| llms.txt | Summarises the site and highlights key pages, in Markdown | AI systems (and humans) |
The key difference is that llms.txt is curated. A sitemap lists everything; llms.txt lists what matters, with a sentence of context for each link.
Do AI tools actually use llms.txt?
This is where honesty matters. llms.txt is a proposal, not a standard. Some AI tools and developer platforms read it, but there’s no evidence that the major AI assistants rely on it to decide what to cite, and Google has publicly said it doesn’t use it.
So treat it as a low-cost extra, not a ranking factor.
Why add one anyway?
- It takes ten minutes. The cost is close to zero.
- It forces clarity. Writing a one-paragraph summary of your site is a useful exercise in its own right.
- It may help some tools. Coding assistants and AI agents that browse the web can use it to find documentation and key pages quickly.
- It’s future-proof. If support grows, you’re already set up.
What should an llms.txt file contain?
The proposed format is:
- An H1 with the site or brand name.
- A blockquote with a one-line summary.
- A short paragraph of context.
- H2 sections with lists of links, each followed by a short description.
A minimal example:
# Example Agency
> Example Agency builds fast websites and SEO systems for e-commerce brands.
We are a web and AI agency based in Lahore, working with UK and US retailers.
## Key pages
- [Services](https://example.com/services/): what we offer
- [Case studies](https://example.com/case-studies/): results with numbers
- [Contact](https://example.com/contact/): start a projectYou can see a live version for this site at /llms.txt. It’s generated automatically from the same data as the rest of the site, so it never goes out of date.
What matters more than llms.txt for AI search?
If you only have time for a few things, prioritise these first:
- Crawl access. Make sure robots.txt and your CDN or firewall don’t block GPTBot, ClaudeBot, PerplexityBot or Google-Extended.
- Content that renders without JavaScript. Many AI crawlers don’t run scripts.
- Structured data. Organization or Person schema with consistent
sameAslinks. - Answer-first writing. Open key pages with a direct two to three sentence answer.
- Authority. Mentions and links from sources AI systems already trust.
Then add llms.txt as the finishing touch.
Frequently asked questions
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may access. llms.txt is a human-readable summary of what your site is and which pages matter. It does not grant or block access.
Will llms.txt get me cited by ChatGPT?
Not on its own. AI assistants mainly rely on crawling, search indexes and authority signals. llms.txt is at most a small helper.
Where should the file go?
At the root of your domain, so it's reachable at https://yourdomain.com/llms.txt, served as plain text.
Should I block AI crawlers instead?
That's a business decision. If you want to be cited in AI answers, blocking crawlers such as GPTBot, ClaudeBot or PerplexityBot works against that goal.