Why blocking AI crawlers quietly costs you the answer

The block is rarely a decision

Most AI-crawler blocks we measure were not deliberated. They arrive with a CDN preset, a security plugin's default rule set, or a copy-pasted robots.txt written when the concern was price scrapers. The result is the same either way: the assistant that a customer asks for a recommendation cannot read the page that answers them.

What a block does and does not do

A robots rule does not stop the content from being summarised: a model that learned the domain in training, or a user who pastes a link, still surfaces it. What the rule reliably removes is the current, correct, citable version: the live price, the live stock state, the live policy.

So the cost of blocking is asymmetric. The protection is partial and historical; the loss is immediate and specific.

A sharper rule than allow-everything

The useful position is not blanket permission. It is an explicit, published policy: which automated clients may read what, at what rate, with what attribution expectation, and which paths (checkout, account, search) are off limits for everyone. State it, link it from robots.txt and llms.txt, and cautious agent platforms have something to act on instead of a guess.