An llms.txt file can provide a short, maintained introduction to a website's content. Whether a particular agent discovers it, follows its links and handles a task better needs to be tested in that specific workflow. The file's existence alone does not answer those questions.
Google Search now documents its position explicitly. Google's guide to optimizing for generative search features, checked on October 21, 2026, says Google Search does not use llms.txt and similar special AI text files as such. You can still provide a file for other services. That does not establish an effect on Google rankings or visibility.
The useful decision is therefore whether you have a particular agent, a concrete task and a reasonable maintenance effort for this additional entry point. If so, a bounded pilot is possible. That does not make the file a mandatory component of every SEO strategy.
What the proposal actually describes
The original llms.txt description identifies the format as a proposal. Its published version 2 was revised in August 2026. It describes an agent-readable overview with context and links to more detailed material.
One relevant current detail is that the file can live at a subpath. For example, the proposal uses /docs/llms.txt for material under /docs/. The common shorthand “put a file at the site root” does not describe every supported arrangement.
The format uses Markdown. It begins with an H1 containing the project or website name. A brief summary and topic-based link lists can add context. The proposal also describes Markdown versions of individual pages and links through which clients can discover them.
This defines a convention. It does not establish that every agent implements it or automatically fetches each available file. Support in the client you intend to use belongs in the initial decision, not in a footnote added after rollout.
A small example, rather than a second website
Suppose a company maintains public documentation. A short entry point might look like this schematic example:
markdown# Example documentation
> Public guidance for setup and day-to-day operation.
## Setup
- [Getting started](https://example.com/docs/start.md): Requirements and procedure.
## Operation
- [Diagnose a failure](https://example.com/docs/diagnosis.md): Observations and next checks.
These addresses are invented. A real file should link to appropriate versions that actually exist. Appending .md to an address does not create a Markdown page. If the server returns ordinary navigation or an error page, the promised representation is missing.
Select material for the task. An agent explaining setup needs prerequisites and a clear procedure. A list of every blog title may be less useful. A small curated set of links is also easier to maintain than an automatically appended export of the entire site.
The overview should agree with its original sources. When a guide changes, its summary and link need to remain accurate. An additional entry point with outdated information can make orientation worse precisely where it is intended to help.
Content guidance does not replace access controls
An llms.txt file can describe where useful information lives. It cannot remove a login requirement or a firewall block. If the linked page is inaccessible, it remains unavailable to that request.
robots.txt also has a different role. It expresses retrieval rules for crawlers that follow them. Check the relevant service to establish which rules apply to a search bot, training bot or user-triggered fetch. Our guide to allowing and blocking AI crawlers explains the distinction by purpose.
A helpful statement in llms.txt does not create technical access control. The file is also public when served publicly. Do not put internal information, credentials or private preview links in it merely because the destination address is difficult to guess.
Retrieval therefore comes before answer quality in your checks. A 200 response is insufficient on its own. You need to inspect whether the file and linked sources actually deliver their intended content.
What your own request establishes
An ordinary GET can save both headers and content:
bashcurl -sS -D llms-headers.txt -o llms-body.txt "https://example.com/docs/llms.txt"
On Windows PowerShell, use curl.exe where necessary. Replace the example address with a file you are authorized to inspect. Check the status, any redirect and the body. An HTML error page returned with 200 is not a correctly loaded Markdown overview.
Then inspect the listed sources. Do their language, product version and contents match their descriptions? Does direct retrieval work without an existing session? Retain the version checked and its date so later changes can be understood.
This establishes technical availability to your test client. A server log showing a later file request establishes another retrieval. That crawling does not establish that a language model incorporated the file into an answer. A user-agent string is initially a client-supplied statement too.
Four cases for a bounded pilot
The following is an experiment plan, not a report of results already obtained. Choose an agent you can control and a few tasks with correct answers that can be checked against your sources. Where possible, begin with test documentation whose versions you can retain clearly.
| Case | Starting point | What you can observe |
|---|---|---|
| Baseline | Agent receives the ordinary public starting page | Which sources it discovers without a file hint and which questions it answers |
| File available | Same task, file published but not mentioned in the instruction | Whether this client discovers and actually requests it independently |
| File supplied | Instruction explicitly names llms.txt as the starting point | Whether the agent reads it, follows appropriate links and uses correct evidence |
| Controlled failure | Isolated test version contains a known stale or broken reference | Whether the agent identifies the gap or adds unsupported information |
The final case belongs in an appropriate test environment. Do not break production documentation to manufacture a failure. The objective is to understand missing-source handling, not to distribute an incorrect public version.
Record the model or client version, permitted tools, task and source state. A fresh session helps avoid carrying information over from the previous run. If a service provides no visibility into retrieval, record that limit. A plausible answer does not replace missing request evidence.
Retain the necessary evidence promptly. If logs cover only a limited period, a missing entry found later cannot establish that the file was never used. Record the time window and delivery path your observations cover, including any gaps in the available evidence.
Separate discovery from usefulness
An agent may use an explicitly supplied file well without discovering it independently. That can be a useful result for a directed documentation workflow. It does not establish automatic use by public search or answer services.
A link being fetched is not a correct answer either. Check whether the response contains the statements needed for the task, respects the source version and expresses uncertainty where information is absent. Save the references actually used alongside the answer.
Understandable evaluation categories are enough: correct answer with appropriate evidence, correct answer without traceable evidence, incomplete answer and incorrect or unsupported statement. Define the expected statements beforehand. Otherwise, evaluation can easily drift towards the result you wanted from the new format.
One successful run is not a reliable impact study. Repeat comparable tasks and record variations. If the client, sources and tool access are uncontrolled, a changed answer can have many causes. Introducing a file alongside new articles, revised internal links and a model change does not allow clean attribution.
Also consider whether the apparent file benefit comes from better editorial selection. If only llms.txt contains a clear sequence and useful descriptions, you are comparing two entry points with different information quality. A narrower format comparison needs a control version offering the same information and links. A better overview can be useful without the filename itself causing the improvement.
Define what counts as an error too. A freely worded answer need not repeat the exact words of a reference answer. Missing an essential prerequisite or presenting an outdated procedure as current is a substantive failure, though. Evaluate those cases against documented statements before summarizing request counts or completion times. With a very small sample, individual case descriptions are often more informative than an apparently precise percentage with little supporting evidence.
Measure Google and other services separately
For AI Overviews and other generative search features, Google continues to describe the importance of the search index, technical access and useful content. Its documentation on AI features and websites does not require an additional AI file. The more explicit current position on llms.txt appears in the optimization guide linked above.
Generative engine optimization groups together very different activities. Name the specific system and intended workflow in your pilot. Support for a file convention in a documentation agent cannot be generalized to every search feature.
If the destination itself is not available for indexing, investigate that prerequisite first. Our guide to “Crawled, currently not indexed” addresses a different stage from whether an llms.txt file is readable.
For Google, the current optimization guide points to the Generative AI performance report in Search Console. Those performance observations are separate from a file request. Higher visibility after publication does not prove that llms.txt caused it. Other services need their own available observations and an appropriate comparison baseline.
Maintenance and the decision after the pilot
Assign responsibility for selection and freshness. Recheck the file after changes to important sources, language paths or documentation versions. A generator can produce links automatically. Whether its selection serves the task remains an editorial question.
For teams organizing research and content together, the Crawl Foundry platform overview provides a starting point. For the file pilot itself, also retain the task, source version and observed retrievals. That connects editorial responsibility with a test someone else can inspect.
Set a limit on maintenance effort in advance. If no relevant client uses the file, or its links provide no observable benefit for your tasks, it need not become a permanent project. If a particular agent demonstrably reaches the right sources more reliably with it, document that specific and limited benefit.
A useful conclusion identifies the system, tasks tested and steps observed. It may also say: technically available, independent discovery not yet observed, usefulness with an explicitly supplied entry point still to be tested. That supports the next decision better than a universal check mark for supposed AI visibility.