Generate an llms.txt for your site

What llms.txt is

The llms.txt proposal describes one file at the root of a site: "We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content." Its shape is fixed. "An H1 with the name of the project or site. This is the only required section", then "A blockquote with a short summary of the project", then sections "delimited by H2 headers" that hold file lists, where each entry is "a required markdown hyperlink [name](url), then optionally a : and notes about the file".

The proposal is also clear about what the file is for: "While sitemaps list all pages for search engines, llms.txt offers a curated overview for LLMs." The information "is instead used on demand, when an agent needs information about a topic while assisting a user." So it is a reading list for agents, and the sitemap stays the full list.

What this generator writes

The name comes from your home page's Organization markup or site name, the summary from its description, and the pages from the same discovery BeeRanked's importer uses: sitemaps, feeds and a bounded crawl. Pages are grouped into one section per part of the site (the first segment of each address: blog, docs, guides), newest first, each as a link with the page's own meta description as the note. Nothing is invented: a page without a description is listed without a note.

Read it before you publish it. The summary line was written for a search result, so rewrite it for someone asking what the site is; drop the pages that do not help that person; and add a sentence under the summary about who the site is for. The file is yours, and the specification expects it to be curated.

What it cannot know

• The free preview reads 30 pages. On a bigger site the list is a start, and the sections show you which parts of the site the crawl reached.
• Pages your robots.txt closes to crawlers are not read, and pages that are an empty shell filled in by JavaScript come back without text.
• A page with no meta description gets no note. The specification allows that, and an agent reads the page anyway; a note helps it choose.

Does it help with search?

Google says no. Its AI features documentation says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." The file is for agents that fetch pages on demand, in the proposal's own words, and those agents prefer Markdown for a measurable reason: Cloudflare found that one of its own posts "takes 16,180 tokens in HTML and 3,150 tokens when converted to markdown". An llms.txt tells an agent that already came where the useful pages are.

Questions people ask

• Where does the file go? At the root, or at a subpath for one section: the specification allows a file "at the root path /llms.txt of a website or at any subpath (e.g. /docs/llms.txt)", and "A file covers the URLs under its path".
• Should I list every page? No. A sitemap does that. List the pages that answer questions about you, and keep the notes short.
• Does Google read it? Google states no such file is needed to appear in its AI features. Other agents may fetch it on demand; the AI crawler checker shows whether yours is published and which agents your robots.txt lets in.
• Is it free? Yes. No sign-up, no email, and nothing is stored.

Read more

• llms.txt in our wiki: what the file is and what goes in it.
• llms.txt: what it is, and whether your site needs one.
• HTML to Markdown: the page-level version of the same idea.