What is the crawl-to-refer ratio and how is it calculated?

The crawl-to-refer ratio is the ratio between the pages an artificial intelligence operator takes from a site and the visits that same operator sends back to it. The numerator is the requests made by its crawlers, read off the web server logs and filtered by user agent; the denominator is the requests that arrived from that operator, identified by the Referer header the browser carries with it. The value is normalized to a single visit: a thousand to one means a thousand pages taken for every visit received. Cloudflare, which publishes the same indicator at network scale, counts only HTML responses, on both sides of the ratio, so that images and stylesheets do not inflate the totals. It is the most direct form of a question anyone who runs a site asks: this automated traffic, which consumes bandwidth and compute, does it bring readers back or not.

See also: AI crawler access control · Deciding which AI crawlers may read your site, distinguishing training crawls from real-time answer retrieval.

Why it is computed per agent, not per vendor

A vendor does not run a single crawler. OpenAI documents three, with different stated purposes: GPTBot exists to make the generative foundation models more useful and safe, meaning training; OAI-SearchBot is for search and surfaces sites in ChatGPT's search results; ChatGPT-User acts on certain user-initiated actions inside ChatGPT and custom GPTs. Anthropic follows the same three-way split with ClaudeBot, Claude-SearchBot and Claude-User, and each name is a separate token in the robots file. The difference is not terminological: only two of those three families can generate a return visit, because they feed an answer a person can then click through. Adding them into one number per vendor mixes a pure cost with an acquisition channel, and produces an average that supports no decision. Cloudflare aggregates by platform because it measures the whole network; anyone reading their own site's logs has no such constraint and should not copy it. One technical warning: the user agent is a text field anyone can write, and both OpenAI and Anthropic publish the IP ranges of their crawlers for exactly that reason. A serious numerator comes from intersecting the two.

An enterprise example

A manufacturing group with a public technical documentation portal measures the ratio for each user agent over a quarter, starting from the server logs. It finds that one vendor's training agent accounts for the largest share of requests and no return visits, while the agent that fetches pages in real time makes a fraction of them and brings the only requests attributable to that brand. With that separation the decision becomes writable in one line of the robots file: deny training on the catalog, leave retrieval open. Without the separation the aggregate number would only have said that this vendor takes a lot and gives back little, and the instinctive reaction would have been to close everything, which also means disappearing from the answers the company wanted to appear in. As an order of magnitude: in January 2025 Cloudflare measured 3.8 to 1 for Google and 286,930 to 1 for Anthropic, which by July had fallen to 38,065 to 1.

Why it matters for decision makers

The ratio turns an impression into a budget line, and it supports three concrete decisions: which bots to admit, a matter of AI crawler access control; how much bandwidth and compute the company is giving away; and whether asking for a crawl toll, where the network in front of the site allows it, beats the open-or-closed choice. Two limits should be stated before using it. Visits arriving from native apps often carry no Referer header, so the denominator is understated and the ratio overstates the imbalance: Cloudflare says so itself. And a crawl is not a citation: Microsoft Clarity's documentation, which exposes these counts in its own Bot Activity dashboard, warns that bot activity does not show that the content was retrieved, cited or surfaced in an answer. That part needs share of model and the practices of GEO and AEO: the crawl-to-refer ratio measures the exchange, not the visibility.