How to Build an llms.txt File for Your Website (And Why It Matters)

llms.txt is a proposed standard for a plain Markdown file placed at the root of a domain — yourdomain.com/llms.txt — that gives AI models and LLM-powered tools a clean, curated index of a site's most important content. The idea, first proposed by Jeremy Howard and the team at Answer.AI in late 2024, borrows the same logic as two files most sites already have: robots.txt tells crawlers what they're allowed to access, and sitemap.xml gives search engines a structured map of every page. llms.txt does something similar, but for language models specifically — a short, hand-curated summary written in plain Markdown instead of HTML, designed to be read directly by an LLM without wading through navigation menus, ads, and JavaScript.

It's worth being precise about what it is not: it's not a ranking signal, it's not enforced by any search engine, and no major AI lab has officially committed to crawling it as part of how ChatGPT, Claude, or Perplexity retrieve information. It's a voluntary convention — closer to an early, informal standard than an official protocol. That matters for how much time you should spend on it, and we'll come back to that.

The honest reason to care about llms.txt today isn't that it's already a major ranking factor — it isn't. It's that almost nobody has one. Run a quick check against ten competitor websites in your industry and you'll likely find zero llms.txt files among them. That's a familiar pattern: sitemap.xml looked similarly niche in its first couple of years before search engines started treating it as a baseline expectation. Early, low-effort adoption of a format like this costs you almost nothing and gives you a structural head start if — and it's still an if — AI platforms start leaning on it more heavily as a discovery mechanism.

There's a second, more immediate benefit that doesn't depend on any AI company adopting the standard at all: writing an llms.txt file forces you to articulate, in plain language, exactly what your site offers and which pages matter most. That exercise alone often surfaces gaps in how a site is structured for AI readability — the same gaps that show up in a proper AEO audit. Even if llms.txt itself never becomes load-bearing, the thinking behind it is the same thinking behind getting cited by ChatGPT and Google's AI Overviews in the first place: structure facts about yourself clearly enough that a machine doesn't have to guess.

The spec is deliberately simple: an H1 with your project or company name, an optional blockquote with a one- or two-sentence summary, optional free-form context in plain paragraphs, and then one or more H2 sections containing a list of links with short descriptions. A final "Optional" section can hold secondary links that a tool can skip if it needs a shorter context window.

markdown
# Project Name

> A one or two sentence summary of what this site or project is.

Optional free-form context paragraph, in plain markdown, without headings.

## Docs

- [Quickstart](https://example.com/docs/quickstart): Get started in five minutes
- [API Reference](https://example.com/docs/api): Full endpoint reference

## Optional

- [Blog](https://example.com/blog): Company news and announcements

That's the entire format. No nested JSON, no required schema validation, no nesting beyond one level of nesting under each H2. The constraint is intentional — it's meant to be something a developer writes by hand in ten minutes, not something that requires tooling to generate.

Start by listing the pages you'd want an AI assistant to know about if it could only see five to ten links from your entire site. For most businesses that's the homepage, your core service or product pages, pricing if it's public, documentation if you have it, and maybe one or two cornerstone content pieces. Resist the urge to list everything — the value of llms.txt comes from curation, not completeness. A 200-link llms.txt file defeats the purpose; it's just a worse sitemap.xml.

Write the H1 and summary first. Be literal and factual — this isn't marketing copy, it's a machine-readable description of what you are. Then group your chosen links into two or three H2 sections that make sense for your site ("Services," "Resources," "Docs" — whatever reflects your actual structure), and add an "Optional" section at the end for anything secondary.

Here's a real, working example based on our own site structure:

markdown
# Webry Technologies

> Webry is a Cutting-Edge AI Web Agency based in Dhaka, Bangladesh, building AI-first SaaS products, AI SEO/AEO audits, and native mobile apps.

Webry Technologies has engineered AI-first digital products since 2015, with 50+ shipped projects including KaloxAI, Urooai.com, and UniVoiceAI.

## Services

- [AI SEO & AEO Audit](https://webry.tech/services/ai-seo-aeo-audit): Get cited by ChatGPT, Perplexity, and Google AI Overviews
- [Custom AI Development](https://webry.tech/services/ai-first-solutions): AI agents and LLM integrations
- [SaaS Development](https://webry.tech/services/saas-development): MVP to scale in weeks
- [Native Mobile Apps](https://webry.tech/services/native-mobile-apps): iOS and Android development

## Resources

- [AEO Optimization Hub](https://webry.tech/aeo-optimization-hub): Central resource for Answer Engine Optimization
- [AI SEO Audit Guide](https://webry.tech/ai-seo-audit-guide): Step-by-step AI readiness audit methodology

## Optional

- [About](https://webry.tech/about): Company history, founders, and mission
- [Contact](https://webry.tech/contact): Get in touch with the Webry team

Once it's written, save it as plain text named llms.txt and upload it to the root of your domain — the same directory as robots.txt, not inside a subfolder. You can sanity-check placement with a simple request:

bash
curl https://yourdomain.com/llms.txt

If that returns your file as raw text, you're done. No build step, no server configuration beyond serving a static file, no submission process to any search engine or AI company.

The most common mistake is treating llms.txt as a dumping ground — pasting in entire page contents or every URL on the site instead of a curated index. That defeats the format's purpose and produces a file no more useful than a generic sitemap.

The second is writing it once and never updating it. If you launch a new product line or retire a service, your llms.txt should reflect that the same way your main navigation would — an outdated file describing services you no longer offer is worse than having no file at all.

The third is expecting it to move rankings on its own. It won't. Treat it as a small, low-cost piece of a much larger AEO effort — structured data, entity consistency, and direct-answer content formatting still carry far more weight today than llms.txt does. It's a credibility signal and a discipline exercise, not a shortcut.

llms.txt is exactly the kind of low-effort, high-optionality move worth making early: it costs a developer an afternoon, it costs nothing to host, and almost no one in most industries has done it yet. Whether or not it becomes a standard every AI platform actually reads, the act of writing one is a useful forcing function — it makes you state, in plain terms, what your site is and which pages matter, which is the same discipline good AEO requires everywhere else.

We're rolling this out across our own properties as part of our broader AEO methodology, and it's one of the smaller checks we now run as part of every AI SEO & AEO audit we do for clients.