How to Generate llms.txt Files for AI Visibility

Most sites hand AI crawlers pages that are mostly structural noise. An llms.txt file is a Markdown index at your domain root that gives LLMs a curated map of what's worth reading instead. In this guide you'll generate a correctly formatted file, host it where crawlers can find it, and get an honest read on whether it improves AI search visibility.

First, what does llms.txt actually do?

What is llms.txt and why does it matter for AI search?

An llms.txt file is a plain-Markdown document at your site's root that tells AI models and agents which pages carry your most important content, in a format they can parse without wading through HTML. Jeremy Howard of Answer.AI proposed the convention in September 2024, and the specification lives at llmstxt.org. The pitch is simple. Give language models a curated table of contents instead of forcing them to reverse-engineer your site structure from bloated markup.

The strategic case ties directly to how buyers discover software now. They ask ChatGPT which tool to use and Perplexity who the category leaders are. Sometimes they ask Claude to compare options. If your content is legible to those systems, you have a shot at being cited. If structural noise buries it, you don't. That's the discoverability argument behind llms.txt, and it slots into the broader work of Answer Engine Optimization (AEO), the practice of positioning content so AI platforms cite, recommend, or mention you when users search for answers.

Adoption by the AI engines themselves is thin, and the evidence for citation lift is speculative (more on that below). But the file is cheap to produce and correct to publish, so treat it as a low-cost hedge. We publish one on growthx.ai for exactly that reason.

Why AI crawlers struggle with standard HTML

Raw HTML burns through an LLM's context window on content the model doesn't need. One analysis of 14,000 Common Crawl pages found raw HTML averaging around 45,000 tokens per page versus under 7,000 once simplification strips the structural noise, an 84% reduction. Even when a page fits, CSS and JavaScript bloat degrade retrieval through the "lost in the middle" problem, where models lose track of relevant content buried in long, noisy inputs.

With an llms.txt file, the model skips the reconstruction step and reads a clean Markdown index of your high-value URLs, tuned for the way agents read.

How to generate your llms.txt file: step by step

The fastest path is an llms.txt generator. You enter your domain, the tool crawls your site and drafts a structured file, you review and edit the sections, then you place the result at your root. The whole loop takes minutes for a small site. If you want programmatic control, skip the UI and run a CLI tool instead.

Step 1: Enter your site URL

Start by pasting your domain into a generator that crawls the live site. The Python llmstxt-generator package runs this from the command line, where llmstxt-gen stripe.com reads a set of pages and composes a draft file. URL-based tools work the same way, discovering pages and extracting titles and descriptions before grouping them into sections.

Most tools cap how many pages they read by default. llmstxt-generator reads 12 pages unless you raise --max-pages, so for a large site, decide upfront which subtree matters most, because the crawler won't know your priorities.

Step 2: Preview and edit the output

Review the generated sections and link lists before you trust them, because auto-generated drafts over-index on whatever the crawler found first. Check the project title and summary first. The H1 should match your project, and the blockquote should state what your site does. Then make sure the H2 sections group links the way a reader would expect (docs, product pages, blog posts).

Cut low-value links, and move anything skippable into an ## Optional section so context-limited tools know they can drop it. This is the step where editorial judgment earns its keep. A generator can list your pages, but it can't rank them by strategic value.

Step 3: Download or copy the file

Export the reviewed content as llms.txt and place it at your domain root. Many tools also produce an llms-full.txt that inlines the full text of every linked page instead of the links alone. If your generator only outputs the index file, that's the spec-required one, and it's enough to ship.

Generate via CLI or API

For pipelines, several current CLI tools take a domain argument and an API key set as an environment variable. The Python llmstxt-generator is the most feature-complete option, with multi-provider support across OpenAI and Anthropic, plus local Ollama, among others:

export OPENAI_API_KEY=sk-...
llmstxt-gen stripe.com             # print to stdout
llmstxt-gen stripe.com -o llms.txt # write to file
llmstxt-gen stripe.com --verbose   # watch discovery trace

Switch providers with --provider anthropic and the matching ANTHROPIC_API_KEY. For a no-API-key path, the npm llmstxt package turns a sitemap into a file directly with npx -y llmstxt gen https://vercel.com/sitemap.xml.

One migration note. Firecrawl deprecated its dedicated /v1/llmstxt endpoint in 2025 with no replacement planned, so if a guide points you at /v1/llmstxt, it's stale. The maintained path is the generate-llmstxt npm wrapper (npx generate-llmstxt --api-key YOUR_FIRECRAWL_API_KEY) or Firecrawl's create-llmstxt-py script.

llms.txt file format and Markdown structure

Then, get the format right. The spec defines a strict order, and only one part is mandatory, an H1 with your project name. Everything else is optional but conventional. The full structure runs H1 title, then a blockquote summary carrying the key context, then any non-heading Markdown, then zero or more H2 sections that each hold a list of links.

Here is the canonical shape from the specification:

# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)

Each link item is a Markdown list entry with a required [name](url) hyperlink, followed optionally by a colon and a short note. A tool can skip URLs in the ## Optional section when it needs a shorter context. Markdown is the deliberate choice over XML because language models and agents read it more easily than a rigid schema.

llms.txt vs. llms-full.txt: which do you need?

Publish llms.txt first, and add llms-full.txt for ingestion pipelines. The distinction is content depth. The core-spec llms.txt is a curated index of links plus short descriptions, while llms-full.txt inlines the entire text of every linked page into one document, including anything in your Optional section.

Use case determines the file. The index file suits real-time AI assistants like ChatGPT and Claude, where context window size is a live constraint and a compact map beats a full dump. The full file suits ingestion pipelines and IDE or RAG (retrieval-augmented generation) indexing systems that want the complete text in one fetch. Mintlify auto-generates both at the root of every docs project, and it describes llms-full.txt as the entire documentation site combined into one file.

The formal spec actually names the derived artifacts llms-ctx.txt and llms-ctx-full.txt, but Mintlify and Firecrawl popularized llms-full.txt for the same idea, and that's the filename you'll see in practice.

Where to host your llms.txt file

Place the file at your domain root so it resolves at yourdomain.com/llms.txt. The spec allows a subpath, but root is the discoverable default and what tools check first. Keep it current too. Update it whenever your high-value pages change, or generate it dynamically so it stays in sync with your CMS, and remember that each hostname needs its own file (yourdomain.com's doesn't cover yourdomain.de). How you get it there depends on your platform.

Shopify

Shopify now natively serves /llms.txt, /llms-full.txt, and /agents.md, so you customize content with Liquid template files. Add templates/llms.txt.liquid (or a single templates/agents.md.liquid, which acts as a fallback for all three paths) under Online Store > Themes > Edit code. Templates must use the .liquid extension, and JSON templates aren't supported for these paths.

Next.js (13+ App Router)

Two approaches work. Drop a static file in the public/ directory and Next.js serves it at /llms.txt automatically. Or, for CMS-driven content, use a route handler at app/llms.txt/route.ts and return text/plain:

export async function GET() {
  const content = `# My Site

> A description of my site for AI agents.

## Docs

- [Getting Started](/docs/getting-started): How to get started.
`;

  return new Response(content, {
    status: 200,
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
    },
  });
}

Vercel recommends route handlers for production, and that's how we serve ours. App Router routes win over public/ at the same path, so if you switch from a static file to a route handler, delete public/llms.txt to avoid a conflict.

WordPress

Upload the file to your site's root directory over FTP or your host's file manager so it resolves at yourdomain.com/llms.txt.

Generic cPanel or FTP

Upload llms.txt into the public_html directory (or your document root) so it serves from the domain root.

llms.txt vs. robots.txt vs. sitemap.xml

These three files answer different questions. robots.txt tells crawlers what they may access, sitemap.xml tells search engines what exists, and llms.txt tells AI assistants what's worth reading.

Here is how they differ where it matters for implementation:

The spec is explicit that a sitemap doesn't substitute for llms.txt. Sitemaps often lack LLM-readable versions of pages, exclude helpful external URLs, and typically cover more documents than fit in a context window. And robots.txt operates at crawl time to gate access, while agents consult llms.txt on demand when a user asks about a topic. Keep all three, since this is additive work on top of your existing SEO fundamentals, not a replacement for them.

Which AI crawlers and search engines read llms.txt?

No major AI provider has officially confirmed that its crawlers consume third-party llms.txt files. ChatGPT, Perplexity, Claude, and Gemini are the intended beneficiaries in theory, but the evidence for functional use is thin, and one of them has ruled it out.

• OpenAI (GPTBot / ChatGPT): No documented support. OpenAI's crawler docs direct site owners to robots.txt and don't mention llms.txt.
• Anthropic (ClaudeBot): No confirmed support. ClaudeBot respects robots.txt, and the llms.txt requests attributed to it in crawl logs amount to a rounding error.
• Perplexity (PerplexityBot): Not mentioned in crawler documentation, which covers robots.txt for search inclusion only.
• Google (Gemini / Google-Extended): Explicitly not supported. Gary Illyes has said Google doesn't support llms.txt and isn't planning to, and Search ignores these files entirely.

llms.txt is a community-managed convention without W3C or IETF ratification, and the adoption-vs-traffic gap is the figure that matters. Published llms.txt files grew nearly ninefold across twelve months, yet 97% of them received zero AI requests in May 2026. So publish the file (it's cheap and correct), but don't build your AI search strategy on it.

Before you invest much in a file no engine confirms it reads, find out how your brand actually appears in AI answers today. We operate CheckThat, which benchmarks brand visibility across 172 categories and 5,800+ brands using 2.6M+ AI responses, and it's a free way to see where you stand. If AI answers are already shaping how buyers find you and you want a system behind your visibility rather than a hedge file, book a demo and we'll walk you through how GrowthOS builds it. Engagements start from $6,000/mo.