Google’s John Mueller responded to a question about LLMs.txt—a proposed way to show website content to AI bots—and said it’s not very useful. He compared it to the outdated keywords meta tag, agreeing with others who’ve had the same experience.
LLMs.txt
LLMs.txt is often mistakenly described as a Robots.txt for large language models, but that’s not accurate. Robots.txt controls how bots crawl a site, while LLMs.txt isn’t meant for that.
Instead, LLMs.txt is designed to present a page’s main content—without ads or navigation—to language models using markdown format. It helps AI systems focus on core content, not the layout or extras. That’s the key purpose behind it.
What LLMs.txt Is:
- LLMs.txt is not a tool to control AI bots.
- LLMs.txt is a proposal to show the main content of a page to AI bots.
- LLMs.txt is not yet a widely accepted or implemented standard.
This point is significant because of what Google’s John Mueller mentioned:
LLMs.txt Is Comparable To Keywords Meta Tag
A discussion about LLMs.txt began on Reddit when someone shared their experience of AI bots not checking their LLMs.txt files. They wrote:
“I’ve submitted to my blog’s root an LLM.txt file earlier this month, but I can’t see any impact yet on my crawl logs. Just curious to know if anyone had a tracking system in place,e or just if you picked up on anything going on following the implementation.
If you haven’t implemented it yet, I am curious to hear your thoughts on that.”
One commenter, who manages over 20,000 domains, mentioned that no AI bots were downloading the LLMs.txt files. Only niche bots, like one from BuiltWith, were accessing them.
The commenter wrote:
“Currently host about 20k domains. Can confirm that no bots are really grabbing these apart from some niche user agents…”
John Mueller answered:
“AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it). To me, it’s comparable to the keywords meta tag – this is what a site-owner claims their site is about … (Is the site really like that? well, you can check it. At that point, why not just check the site directly?)”
John Mueller’s point is valid: none of the major AI services, including Anthropic, OpenAI, and Google, have announced support for the LLMs.txt standard. So, if these services aren’t using it, what’s the actual benefit of having the file?
He also highlights that the LLMs.txt file seems redundant. Why would AI bots use it when they can already access the original content and structured data? These bots would still need to check the rest of the content to verify that it’s not spam, which defeats the purpose of having an additional file.
Finally, the risk of misuse is significant. Publishers or SEOs could potentially show one version of the content in LLMs.txt for AI bots and another for users or search engines. This could easily lead to spammy practices, essentially allowing cloaking for LLMs.
This is similar to the keywords meta tag, which search engines no longer rely on. It’s too risky to trust the claims made by the tag, and modern search engines are far more advanced in understanding content without relying on such self-reported claims.
Follow-Up Post on LinkedIn
Simone De Palma, who originally started the Reddit thread about LLMs.txt, later shared a post on LinkedIn to continue the discussion. In his post, he talked about his personal experience with LLMs.txt and shared his thoughts on why this file might not be helpful. He pointed out that it could even hurt the user experience, depending on how it’s used.
He wrote:
“LLMs.txt files seem to be ignored by hashtag#AI services and offer little to no real benefit to website owners.
…Moreover, someone argues LLM.txt files can lead to poor user experiences, as they don’t link back to original URLs. Any citations gained by your website may direct users to an unbelievable wall of text instead of proper web pages – so again what’s the point?”
Others in the discussion agreed with De Palma. One person mentioned that very few bots were actually visiting the LLMs.txt file, and suggested that it might be a better use of time to focus on more effective strategies instead.
He shared:
“Agree. From the tests I’m conducting, there are few visits and no advantage so far (my idea is that it could become useful if exploited differently because in this way you can also risk confusing the various crawlers; I left the test active “only” on my site to have other data to think about). At the moment, it is certainly more productive to focus your efforts on structured data done properly, robots.txt and the various sitemaps.”
Read the Reddit discussion here:
