- Intro: why llms.txt exists and who it helps
- The conclusions up front
- What llms.txt is and which problem it solves
- File structure: required and optional fields
- How llms.txt differs from robots.txt and sitemap.xml
- How to create llms.txt: a step-by-step guide
- Common mistakes and how to avoid them
- What llms-full.txt is and whether you need it
- Automation and CI/CD: keeping the file current
- Is it worth implementing llms.txt now
- Conclusion and a call to action
Ihor Nikolenko here. When AI turns into your new reader, a question comes up: how do you present content so that the machine can tell what matters and what does not?
That is exactly what llms.txt is for: a plain text file that introduces language models to your web page. In this article we will work out what this file is, why you need it, how to create it and which details are worth keeping in mind. It will be a very detailed piece, so that you are left with no questions.
Intro: why llms.txt exists and who it helps
Language models like ChatGPT or Claude increasingly pull content from the web to answer user queries. But they have two problems. First, model context windows are limited: they can only “read” a certain number of characters, so long section pages have to be cut short. Second, most modern sites have a complex structure with HTML markup, tables, scripts and ads, which makes it hard for a machine to tell what actually matters. That is where the idea came from: a separate file that hands language models concentrated information about your site.
The author of the proposal, Jeremy Howard, published the first description of the standard on llmstxt.org in September 2024. He noted that llms.txt sits in the site root and holds a short description of the project, the important files and structured links. Instead of processing HTML pages, an agent or an LLM can read this file and immediately understand which parts of the site are key. The idea is to make answer generation simpler and to avoid burning resources on information nobody needs. Experts point out that llms.txt does more than duplicate the sitemap: it gives a curated list of important content.
In other words, llms.txt sits alongside search optimisation and robots.txt and improves how language models in particular interact with your site. In this piece you will learn which fields the file should contain, how to structure it properly, and how to avoid the usual mistakes.
The conclusions up front
If you do not have time to read the whole thing, here is the short version:
- llms.txt is a text file in Markdown that helps language models understand the structure of your site faster and pick out the key sections.
- The file should contain a heading (the project name) in H1, a short description as a blockquote, and structured lists of links with brief descriptions. That keeps the information readable for LLMs and for people.
- llms.txt complements robots.txt and the sitemap rather than replacing them: robots.txt restricts access, the sitemap lists every page, and llms.txt highlights the valuable content.
- A text editor and basic Markdown are all you need to create the file. What matters is putting it in the right place on the server, usually the site root (yourdomain.com/llms.txt).
- Support for the standard is still limited: the large AI providers have not officially confirmed that they read llms.txt. That said, some companies (Anthropic, for example) already publish such files, and interest is growing.
Next we will go through all of it in more detail, from how the idea came about to step-by-step advice on creating and automating the file.
What llms.txt is and which problem it solves
Unlike search crawlers, which scan HTML and build an index from keywords and links, AI agents process text inside a limited context. They cannot tell straight away which pages matter more than others. That is exactly what llms.txt sets out to fix.
The main goal of the format is to make AI agents interact with your site more efficiently. An article on the Parallel.ai portal explains that the file acts as a “map” for language models: it holds the site name, a short description and a list of priority pages with an explanation of why they matter. That way an agent can quickly work out where to look for what it needs, without spending compute on secondary pages.
The standard also helps solve the “clutter” problem of modern web resources. According to the Semrush blog, AI crawlers usually hit two challenges: 1) Modern sites are hard to read because of complex markup and dynamic content.2) Most sites carry far too much information, and the agent does not know what to focus on. llms.txt offers a clear, compact structure that works around both. On top of that, the file can cut compute costs: an LLM will not waste time on content it does not need.
Where does the name come from? LLMs.txt is a literal mash-up of “Large Language Models” and .txt. The authors are nodding at robots.txt and sitemap.xml. The key difference, though, is that llms.txt is meant for language models specifically and, unlike robots.txt, does not restrict access; it recommends what to read. It does not duplicate sitemap.xml either, which lists every page; llms.txt highlights the most important ones instead.
File structure: required and optional fields
For agents to read the file correctly, its structure has to be standardised. The official description on llmstxt.org recommends this order:
- Heading (H1). This is the name of your project or site. The heading is required and has to be the first line of the file.
- A short description in a blockquote. After the heading comes a short description of the project: literally 1–3 sentences that convey what the resource is. It helps the model grasp the subject quickly.
- Details and notes. Next you can add a few paragraphs or lists with important clarifications. For example, which services you support, what makes your product different, whether there are limitations. It is the equivalent of the “Important notes” section in the FastHTML example.
- Sections with files (H2). After the notes come the sections listing files or pages. Section names are H2 headings; underneath them are bulleted lists of links. Every item should contain:
- The resource name in square brackets;
- The URL in parentheses;
- a short description after a colon. The comment explains why this page matters.
- The optional “Optional” section. If some materials are not critical, you can group them into an “Optional” section. That is a hint to the agent that these resources can be skipped when needed.
The Parallel.ai article also lists the “required elements” of the format: the site name, the description and a structured list of URLs. Optionally you can add contact information, a link to your API documentation, a version number or an update date. That helps agents work out how to interact with your service and check how current the data is.
An important advantage is that the format is based on Markdown, so it is easy to write and read. The standard recommends the basic elements: headings (#, ##), blockquote (>), bulleted lists (-) and links in the [text](url) form. If your resource is very large you can also use H3 headings, tables or even code blocks; the main thing is to keep it readable.
Even if your site has hundreds of pages, you do not need to put them all into llms.txt. Experts reckon the file should hold a relatively short list of the most important pages, usually 10–15 entries. If you have a large section aimed at LLMs, it is better to create a separate llms-full.txt file (more on that below).
How llms.txt differs from robots.txt and sitemap.xml
At first glance it may look as though llms.txt duplicates what robots.txt and sitemap.xml do. According to the people behind the standard, though, each of these files plays a different role:
- robots.txt tells search crawlers what they may or may not index on the site. It is a server file that governs access. llms.txt sets no restrictions, it only recommends what is worth reading.
- sitemap.xml lists every page available for indexing. It carries no descriptions and no order of importance, and the list itself can get very long.
- llms.txt adds a different kind of value: it provides a curated list of the most important materials, with descriptions and split into sections. It helps AI agents understand the structure of the site and set priorities instead of simply listing everything in a row.
The Parallel.ai article stresses that llms.txt is a tool of “inclusion”: it invites the agent to read particular content instead of restricting access. Bots that respect robots.txt may skip the parts of a site that are off limits. llms.txt, by contrast, points out what matters and helps AI services work more efficiently.
How to create llms.txt: a step-by-step guide
Any site owner or developer can create llms.txt. Here is a detailed walkthrough based on the recommendations in the Semrush and Parallel.ai articles:
- Decide which pages to include. The first step is to go through your site and pick the pages you want to recommend to AI agents. Semrush suggests focusing on products or services, current blog articles, the pricing page, the “About us” information and “Contacts”. Ask yourself: “If an agent could read only ten pages, which ones would give the best picture of my business?”.
Create the file in Markdown. Open any text editor (Notepad, VS Code, Sublime). Name the file llms.txt and save it. Start with a heading carrying the project name, then a short description in a blockquote. After that add the list of sections with links. The structure looks something like this:
# MySite > A short description of your project in two sentences Important notes: - Item 1 - Item 2 - Item 3 ## Products - [Product A](https://example.com/a): A short description of the main benefits - [Product B](https://example.com/b): A short description ## Blog - [Article 1](https://example.com/blog-1): What it covers and why it is useful - [Article 2](https://example.com/blog-2): … ## Company - [About us](https://example.com/about): Information about the team - [Pricing](https://example.com/pricing): An overview of the plans - [Contacts](https://example.com/contact): How to get in touchA format like that is clear to people and to machines: use bulleted lists and short descriptions to explain what each page is for.
- Put the file in the site root. The saved file has to be uploaded to the server. If it describes the whole site, put it in the root directory (public_html), and it will be available at https://yourdomain.com/llms.txt.
If you are creating the file only for documentation, upload it into the matching subsection (https://docs.yourdomain.com/llms.txt, for example). If you use a CMS or a static site generator (Docusaurus, VitePress), put the file in the folder that is emitted to the root at build time.
- Validate the structure. Before you publish the file, check that it is correct with an online validator. The llmtext.com service, for example, scans the file, checks the formatting and the availability of the links. That helps you avoid mistakes (broken links, missing descriptions) and make sure the file reads well for a human and for a machine alike.
- Update it regularly. llms.txt is not a static document. When you publish new articles or sections, or when the structure of the site changes, update the file. Experts advise syncing updates with your editorial calendar or your product releases so the file stays current. Use version control (Git) to track changes and keep a history of edits.
Note that creating the file may take just a couple of hours, yet it can pay off later with a better representation of your content in AI answers. Even though the format is not official today, it may well become a standard in the next few years.
If your site runs on NEO, llms.txt comes with the platform's standard SEO setup, along with Schema.org and clean markup. Your part then comes down to the main job: picking the key pages and writing clear descriptions for them.
Common mistakes and how to avoid them
According to the Parallel.ai article, many companies building llms.txt for the first time make the same mistakes.
Here are the three main ones:
- Incomplete or generic descriptions. Writing just “Documentation” or “Blog” is not enough. Every item has to explain why the page matters, which questions it answers or what value it gives. For example: “API Reference: a full list of endpoints with request and response examples” instead of a bare “API”.
- Wrong or unreachable links. URLs that lead to a 404 or require authorisation make the file useless. Check every link before publishing: does it open without errors, does it hold the content you promised.
- Poor organisation. If the list of links is very long and has no logical grouping, the model will struggle to tell which section is which. Use H2 headings to group by topic (Documentation, Products, Support, Examples and so on).
To avoid these mistakes, stick to the principles of clarity, structure and freshness. Remember that the file is written not only for AI but for people too: your partners or your users may open it as well, to find useful information quickly.
What llms-full.txt is and whether you need it
If llms.txt is a short guide for language models, then llms-full.txt is the “full reference”. The Parallel.ai article notes that the main file holds only the priority pages, while llms-full.txt can include the entire documentation or a large volume of content. It is normally used in big projects where agents need as much information as possible. If you want to, you can create both: the main one for a quick overview and the full one for the details.
The Mintlify documentation describes llms-full.txt as combining the whole documentation of a site, generated automatically for projects that use the service. In most cases a basic llms.txt is enough. If your site consists of several enormous sections (API references, numerous guides) and your users or partners need full access to all of it, you can add llms-full.txt.
Automation and CI/CD: keeping the file current
To stop the file going stale, it is important to automate updates. The Parallel.ai article advises wiring llms.txt generation into your CI/CD pipelines. If you use a static generator or a content management system with frontmatter, for example, you can write a script that:
- Scans the structure of the site and collects page names and their descriptions from the frontmatter;
- Generates the structure with the heading, the description and the sections;
- Places the generated file in the root folder on every deploy;
- Creates llms-full.txt with all the materials when needed.
That approach means you never have to think about updating it by hand: the file always reflects the current state of your content. It is especially useful for large technical projects and documentation sites.
For small projects where content changes rarely, automation is not mandatory. It is enough to set a reminder or fold the file update into your publishing routine.
Is it worth implementing llms.txt now
“Why do I need this?” is a fair question. Articles on Semrush and elsewhere note that the standard is not official yet and is not supported by the big AI companies. On top of that, search engines do not use llms.txt for ranking, so it has no direct effect on SEO. Still, there are a few arguments in favour of the experiment:
- A low barrier to entry. Creating the file is not hard. You only need a few hours to gather the links and write the descriptions. Even if the format never goes mainstream, you will have spent a minimum of resources.
- Potential upside later. If llms.txt does become a standard, early adopters get a competitive edge: their content will be understood better by agents, which may improve visibility in AI answers.
- Transparency for partners and clients. The file can be useful even to people: suppliers, partners or journalists can quickly understand the structure of your site and find the resources they need.
- More efficient indexing. Even though the big LLMs do not support llms.txt yet, some tools and platforms (Parallel, Anthropic) already work with it. If you use those services, the file may improve your results.
At the same time, if your resources are limited and you have plenty of other work, rolling out llms.txt can wait. Remember: this file will not necessarily change your traffic or your sales, but it does show a proactive approach to future standards. The decision depends on your strategy, your goals and your audience.
Conclusion and a call to action
llms.txt is a new initiative meant to make your site clearer to language models. It is a simple text file with an understandable structure: a heading, a short description, a list of important pages and, where needed, extra notes. Its job is to highlight the most valuable information and save the AI system’s resources. The format complements robots.txt and sitemap.xml rather than replacing them: instead of restricting access, it marks priorities.
We have established that llms.txt can be useful to AI agents and to people who want to find their way around a site quickly. Official support from the big AI companies is not there yet, though, so adoption stays voluntary. If you enjoy experimenting and being among the first to adapt to new standards, try creating llms.txt on your own site and share what you learn in the comments or on social media.
💡 Your action item: put together a list of the 10 key pages on your site, write short descriptions and create your own llms.txt. Then check that the file is available at yourdomain.com/llms.txt and test it in an online validator. Write in the comments what results you got. Your experience may help someone else!
And if you want to go deeper into digital marketing and automation, subscribe to my Telegram blog @nikolenkoclub and through the form in the site footer, and follow the new articles. Together we will learn to stay one step ahead! 🚀