Docs

Should I serve Markdown versions of my pages to AI agents (llms.txt, Markdown for Agents)?

Optional, and never instead of good HTML. Google says you don't need Markdown or AI text files to appear in Google Search, and its AI features read the same HTML pages. A Markdown version helps agents that ask for it, such as Claude Code, read your page with fewer tokens. If you add one, it must say exactly what the HTML page says.

Facts

Who actually reads the Markdown version?

Mostly coding agents and AI tools that fetch a page on request, not the search crawlers that decide what gets cited.

ReaderWhat it readsDoes Markdown help?
Google Search, AI Overviews, AI ModeYour HTML pagesNo: Google says it does not need Markdown
Coding agents such as Claude CodeMarkdown when the site offers it on Accept: text/markdownYes: fewer tokens, cleaner text
Developer docs readersA .md copy of each pageYes: OpenAI's own developer docs offer one by adding .md to the URL
llms.txtA short index fileNot for Google Search, which ignores it

Is serving Markdown to bots cloaking?

Not if the content is the same. Google defines cloaking as showing different content to users and search engines with the intent to manipulate rankings and mislead users.

A Markdown copy that carries the same text, prices and claims as the HTML page is a different format, not different content. A Markdown page with extra claims written for AI is the risk.

Should my business site do this, or only docs sites?

Docs and developer tools gain the most, because their readers are often coding agents. A business site that sells to people gets more from complete, server-rendered HTML pages that answer the buyer's questions.

If your site already sits behind Cloudflare on a paid plan, turning the feature on is a toggle and costs nothing extra. Building and maintaining a second copy of every page by hand is rarely worth it.

How do I check what an agent gets?

Request the page with the Markdown Accept header and read the headers that come back. The response must say content-type: text/markdown and vary: accept, so caches keep the two versions apart.

# What an agent that asks for Markdown receives
curl -s -H "Accept: text/markdown" https://www.example.com/pricing | head -20

# The headers that must come back with it
curl -sI -H "Accept: text/markdown" https://www.example.com/pricing | grep -i -E "content-type|vary"
# expected: content-type: text/markdown ... and vary: accept

How does easyIndex treat Markdown?

easyIndex audits each page when your coding agent submits it, then checks the published HTML page on the live site, because that is what search and AI crawlers read. The page is written once, for indexing, so there is no second version to drift.

Step by step

  1. Fix the HTML first. Make sure each public page has its full text in the server-rendered HTML and is in the sitemap.
  2. Decide who needs Markdown. Add a Markdown version only if coding agents or developers read your pages, as with docs or APIs.
  3. Pick one method. Turn on Cloudflare's Markdown for Agents, or have your coding agent serve the same content as Markdown on Accept: text/markdown.
  4. Keep the content identical. Generate the Markdown from the same source as the HTML so prices and claims never differ.
  5. Test with curl. Request a page with Accept: text/markdown and confirm content-type and vary: accept come back.

Common mistakes

  • Writing Markdown pages with claims that are not on the HTML page, which is what Google calls cloaking.
  • Building Markdown copies while the HTML pages still render their text only in the browser.
  • Serving Markdown on the same URL without vary: accept, so a cache hands Markdown to browsers.
  • Expecting an llms.txt file or Markdown copies to raise rankings in Google, which says it ignores them.

How easyIndex solves it

Your coding agent writes and publishes each page in your repository, guided over MCP with the current rules; easyIndex audits it when submitted, checks the live HTML page itself, and then shows which pages AI crawlers read and how many AI-assistant visitors each one brings. US$ 30 per website a month, with a 7-day free trial.

Frequently asked questions

Is Markdown for Agents the same as llms.txt?

No. llms.txt is one index file at the site root; Markdown for Agents returns a Markdown version of any page when the request asks for text/markdown.

Does Markdown for Agents work on Cloudflare's free plan?

No. Cloudflare's docs list Pro, Business and Enterprise plans and SSL for SaaS customers, at no extra cost.

Will ChatGPT cite my site more if I serve Markdown?

Nobody has shown that. OpenAI's crawler documentation says nothing about Markdown, and citations depend on the page being crawlable and answering the question.

Can my coding agent build this without Cloudflare?

Yes. A route or middleware can return the same content as Markdown when the Accept header asks for it, with vary: accept on the response.

Continue reading

Last verified on .

Serve Markdown to AI agents? llms.txt vs Markdown for Agents