Markdown content negotiation: serve agents the page, not the markup

HTML is expensive to read

A typical page is a few thousand words of content inside tens of thousands of tokens of markup, scripts, inline styles and navigation. A model pays for all of it, and its context window is finite, so long pages get truncated before the useful part. Markdown keeps the structure that matters (headings, lists, links, emphasis) and drops everything else.

Content negotiation is the HTTP mechanism for this, and it is older than the web's AI phase. A client says what formats it accepts in the Accept header; the server picks one and says which in Content-Type. Cloudflare's Markdown for Agents documentation shows what a Markdown-negotiating response looks like today.

What the exchange looks like

curl https://www.example.com/pricing -H "Accept: text/markdown"

HTTP/2 200
content-type: text/markdown; charset=utf-8
vary: accept
x-markdown-tokens: 725
x-original-tokens: 12345
content-signal: ai-train=no, search=yes, ai-input=yes

---
title: Pricing
---

## Plans

- **Starter**: CHF 29 per month ...

• Content-Type: text/markdown is the signal a client checks. A 200 with HTML in the body is a miss.
• Vary: Accept tells caches to store the HTML and Markdown variants separately. Without it a CDN can serve Markdown to a browser.
• The token-count headers are Cloudflare's addition and optional; they let an agent estimate savings and plan chunking.
• Security and caching headers from the HTML response should survive; body-describing headers (Content-Length, ETag, Last-Modified) must be recalculated or dropped.

Three ways to implement it

1. At the CDN. If you are on Cloudflare, Markdown for Agents is a zone setting that converts HTML on the fly for requests that prefer text/markdown. Nothing changes at the origin.
2. At the origin, from the source. Static-site generators and headless CMSs already hold the content as Markdown or structured text. Add a route that checks Accept and returns the source rendered as Markdown instead of HTML. This gives the cleanest output because there is no HTML to reverse.
3. At the origin, by conversion. For server-rendered apps, run the rendered HTML through an HTML-to-Markdown library when Accept prefers text/markdown. Cache the result per URL. Quality depends on the converter and on your landmarks: with a <main> element you convert the content; without it you convert the navigation too.

import { NextResponse, type NextRequest } from "next/server";

export function middleware(request: NextRequest) {
  const accept = request.headers.get("accept") ?? "";
  const wantsMarkdown =
    accept.includes("text/markdown") && !accept.includes("text/html;q=1");
  if (!wantsMarkdown) return NextResponse.next();

  const url = request.nextUrl.clone();
  url.pathname = "/api/markdown" + url.pathname; // route that renders the page as Markdown
  return NextResponse.rewrite(url);
}

Respect the client's preference order. Accept: text/markdown, text/html;q=0.5 prefers Markdown; Accept: text/html, text/markdown;q=0.1 prefers HTML. A browser never asks for Markdown, so a correct implementation is invisible to people.

What Qomvia checks

The Markdown content negotiation check under Agent protocols fetches your homepage with Accept: text/markdown, text/html;q=0.5 and passes when the response is below 400 and its Content-Type is text/markdown. Anything else fails with the content type you returned. It is worth 2 points, the largest single check in the protocols dimension, because it changes what every agent receives on every page rather than adding one discovery file.

Cloudflare measured 3.9 percent of top domains passing this in April 2026. That is early enough that implementing it is a visible difference in a scan, not a hygiene item.

Questions

Is this the same as publishing .md versions of pages?

Same goal, different mechanism. The llms.txt proposal suggests a parallel URL with .md appended. Content negotiation serves Markdown at the original URL, so an agent does not need to know the convention.

Will this hurt SEO or show Markdown to visitors?

No. Browsers do not request text/markdown, and with Vary: Accept caches keep the variants apart. Search crawlers keep receiving HTML.

Which agents actually send Accept: text/markdown today?

Cloudflare's documentation shows its own tooling and readiness scanner doing so, and Qomvia's crawler does. Operators do not publish client lists. It is a standard HTTP convention, so support accrues on the client side without you changing anything.