What is llms.txt and does your website need one?

llms.txt was proposed in September 2024 by Jeremy Howard of Answer.AI. The idea was straightforward: robots.txt tells search crawlers which pages they can and can't access, llms.txt would do something similar but for AI tools, giving them a curated, plain-language summary of your site rather than forcing them to crawl and interpret every page themselves.

It's written in Markdown, which is a simple text format that's easy for both humans and machines to read. A basic llms.txt file might include your business name, a short description of what you do, links to your most important pages, and a note on how your content can be used. That's it. No code required, no technical setup just a file that sits at yourdomain.com/llms.txt and waits to be read.

The concept spread quickly. By October 2025, over 844,000 websites had published an llms.txt file. Anthropic, the company behind Claude, officially endorsed the standard. The discussion in the SEO and AI visibility space shifted from "what is this" to "should I add one" — which is exactly where we are now.

This is where we need to be straight with you, because there's a lot of noise in this space and most of it overestimates where things currently stand.

As of mid-2026, the major training crawlers GPTBot (OpenAI) and ClaudeBot (Anthropic) are not reliably requesting or reading llms.txt files. A 30-day audit of over 1,000 websites found essentially zero visits to llms.txt from those bots. ChatGPT and Claude do have real-time web retrieval built in, but that reads your actual page content — your headings, body text, structured data, and metadata. It is not specifically reading your llms.txt file. The honest position right now is: the file exists, hundreds of thousands of sites have published one, but the major AI tools are not yet using it in a meaningful way.

What influences AI recommendations right now is your actual page content — the same content that determined which Queenstown tourism operator appeared in that ChatGPT recommendation and which didn't. If your content is vague about what you do, buried in JavaScript, or spread across pages that don't clearly connect, that is the more immediate problem to fix. llms.txt is not a shortcut around that. It is a separate layer on top of a site that is already well structured, and one that will matter more as adoption grows.

The more useful question isn't who reads it today — it's who is most likely to in the next 12 to 18 months, and why the trajectory points clearly in one direction even if the exact timing doesn't.

Perplexity is already a web-native AI search engine built around live crawling and citations — a structured context file like llms.txt would make that process faster and more accurate, and adoption there is credible in the near term. Anthropic has publicly endorsed the llms.txt standard, which strongly suggests ClaudeBot will move toward reading it as the format matures. OpenAI's agent and operator tools are being designed for exactly the kind of structured, machine-readable context llms.txt provides. None of them have fully committed to reading the file consistently yet. But the direction is clear.

The framing that makes most sense here is Business-to-Agent (B2A) communication. Right now, your marketing is written for humans. llms.txt is the first standardised format for communicating directly with AI agents — the tools that are increasingly doing research, comparing options, and making recommendations on behalf of real people. The training crawlers aren't reading it consistently yet. But structured content is what's determining AI recommendations right now, and llms.txt is the next layer of that picture. You want to be ready when adoption arrives, not scrambling to catch up.

There's no official standard, but the convention that's emerged follows a consistent pattern. At minimum, a well-structured llms.txt includes: your business name and a clear one-paragraph description of what you do; a list of your most important pages with short descriptions of what's on each one; notes on how your content may or may not be used by AI tools; and any key facts you want AI tools to get right — your location, your specialisations, your differentiators. This is how we have structured ours.

The goal is to give an AI tool enough context to describe your business accurately without having to infer everything from your homepage copy. Think of it as the briefing document you'd give a journalist before an interview — the facts you want them to have before they start asking questions.

What it shouldn't be is a wall of marketing language. AI tools are better at parsing plain, specific statements than they are at interpreting brand positioning. "Helicopter tour operator based in Queenstown offering scenic flights over Fiordland, the Southern Alps, and Milford Sound, with private charter options for groups of up to six." is more useful to an AI tool than "Queenstown's most trusted helicopter tour operator"

The argument for doing it now isn't that it'll move the needle tomorrow. It won't — the crawlers aren't reliably reading it yet. The argument is that it takes about an hour to create, costs nothing to host, and positions you ahead of the curve when adoption does happen. Every major shift in how the web works has had an adoption window where early movers got disproportionate benefit. robots.txt was like this. Structured data was like this. llms.txt is at that same early stage.

There's also a secondary benefit: the process of writing your llms.txt forces you to clearly articulate what your business does, who it's for, and what makes it different — in plain language, without marketing gloss. That clarity tends to improve the rest of your content too.

The priority order is: fix your page content first, add structured data, then add llms.txt. If your website already does those things well, there's no good reason to wait.