Web Bot Auth: verifying an agent by signature instead of by IP list

The identity problem in one paragraph

Today a site decides whether a request comes from a legitimate agent by reading the user-agent header and checking the source IP against a list the operator publishes. The header is trivial to fake. The lists change, differ per operator and do not exist for smaller agents. So sites either let everything through or block anything without a browser fingerprint, and the bot protection article shows what the second choice costs.

Web Bot Auth replaces both with cryptography. The agent operator publishes a public key. Each request the agent sends carries an HTTP message signature made with the matching private key. A site, or the CDN in front of it, fetches the key and verifies the signature. Forged headers fail, and no IP list is needed. It is specified in two IETF drafts, one for the key directory and one for the signing protocol, and Cloudflare documents its implementation as a verification method for verified bots.

The three headers

Signature-Agent: "https://signature-agent.test"
Signature-Input: sig2=("@authority" "signature-agent")
  ;created=1735689600
  ;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"
  ;alg="ed25519"
  ;expires=1735693200
  ;nonce="e8N7S2MFd/qrd6T2R3tdfAuuANngKI7LFtKYI/vowzk4lAZYadIX6wW25MwG7DCT9RUKAJ0qVkU0mEeLElW1qg=="
  ;tag="web-bot-auth"
Signature: sig2=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQd…:

• `Signature-Agent` is a quoted https:// URL. The verifier appends /.well-known/http-message-signatures-directory to it and fetches a JSON Web Key Set.
• `Signature-Input` names the components that were signed (at least the authority and the Signature-Agent header itself), the key thumbprint, a creation and an expiry timestamp, and the tag web-bot-auth. Cloudflare recommends expiries of about a minute against replay.
• `Signature` is the Ed25519 signature over those components. Cloudflare supports Ed25519 only.

What a site can do today

Most sites are verifiers, not signers. There are three practical levels of support.

1. Let your CDN verify. On Cloudflare, agents that sign requests and have registered their key directory are treated as verified bots, and you can allow the verified-bot category in your WAF rules while keeping challenges for unsigned traffic. That is the whole point: allow by proof rather than by guess.
2. Say that you accept signatures. A verifier can advertise support with an Accept-Signature response header, which tells signing agents which components to sign for this site. It is a signal that you are ready for signed traffic, and it is one of the two things Qomvia's check reads.
3. Publish your own key directory if you operate agents. A company whose crawler or assistant fetches other sites should host /.well-known/http-message-signatures-directory with its public keys and sign outgoing requests, so the sites it visits can verify it in turn.

{
  "keys": [
    {
      "kty": "OKP",
      "crv": "Ed25519",
      "kid": "<JWK thumbprint of this key>",
      "x": "<base64url Ed25519 public key>"
    }
  ]
}

Serve the directory over HTTPS with the media type the directory draft specifies, and follow the draft's requirement that the directory response is itself signed, so a verifier can check it has not been tampered with. Generate the thumbprint and JWK from the key rather than by hand; Cloudflare's page walks through the exact commands.

Where it stands, and what Qomvia checks

Both drafts are pre-standard and the header formats have changed between draft versions; Cloudflare's page lists the exact forms it accepts and rejects. Adoption on the signing side is limited to operators that have registered with Cloudflare's verified-bots programme and a handful of agent vendors. That makes this a forward-looking check: cheap to be ready for, not yet decisive for traffic.

Web Bot Auth key directory under Agent protocols passes when your domain serves /.well-known/http-message-signatures-directory with a 200, or when your homepage response carries an Accept-Signature or Signature-Agent header. It is 1 point. Cloudflare's readiness scanner lists Web Bot Auth under Bot Access Control for the same reason: it is the mechanism that lets you stop choosing between blocking every bot and trusting every header.

Questions

Do I need to publish a key directory if I do not run any bots?

No. The directory belongs to whoever signs requests. As a site you verify, either through your CDN or by advertising Accept-Signature and verifying at the origin.

Which AI agents sign their requests today?

Cloudflare's verified-bots directory is the public record; operators must register there to be verified on Cloudflare zones. The list is short and growing, and the operators do not all document it themselves.

Is this the same as mTLS or API keys?

No. It is a signature over the HTTP message itself (RFC 9421), verifiable by anyone with the public key, with no shared secret and no TLS client certificate.