Every few weeks somebody publishes a scary chart showing AI crawlers taking content and sending nothing back. We wanted to know if that was true for us specifically, so we stopped guessing and started logging.
For a month we recorded the User-Agent of every bot that hit Topickz at the edge, tallied the AI crawlers, and then pulled the other half of the equation from analytics: how many actual humans arrived on the site from an AI tool. One side is the taking. The other side is the giving back. The distance between them is the story.
What we measured
This is not a survey and it is not a projection. Topickz runs a small logger at the edge that watches the User-Agent on every request and counts the known AI, search and SEO crawlers into daily counters. Humans never touch it. Analytics does the opposite job: it can only see real people running a browser, so it captures the sessions that arrived with an AI tool as the referrer (chatgpt.com, perplexity.ai, and so on).
Put the two together and you get a clean ratio for each engine: how many times it crawled us, divided by how many humans it sent. We ran the window from June 5 to July 4, 2026.
Our methodology covers how the review desk verifies its numbers.
Over those 30 days there were 18,062 crawler hits in total. AI bots were 61% of that (11,063 hits). Classic search engines were 21%, and SEO tools like Ahrefs and Semrush made up the rest. So the majority of the non-human traffic pounding the site is now AI, not Google.
The gap, one number at a time
Here is the whole thing in one table. Crawls are our first-party count. Referrals are the humans those engines actually sent. The Cloudflare column is the public benchmark, its network-wide crawl-to-refer ratios from July 2025, so you can see where we sit against the wider web.
| Engine | AI crawls (30d) | Human referrals | Topickz crawl-to-referral | Cloudflare network (Jul 2025) |
|---|---|---|---|---|
| OpenAI | 6,564 | 21 | 313 : 1 | 1,091 : 1 |
| Perplexity | 1,827 | 1 | 1,827 : 1 | 195 : 1 |
| Anthropic | 831 | 0 | no referrals | 38,066 : 1 |
| All AI | 11,063 | 22 | 503 : 1 | n/a |
The all-in number is grim on its face. 503 crawls for one visitor. If you only read that line you would conclude what everyone concludes, that AI is a parasite on publishers. But the average hides three completely different behaviors, and the differences are where the useful part is.
OpenAI is the one that pays rent
OpenAI crawled us 6,564 times and sent 21 humans. That is 313 crawls per referral, and by the standards of this business that is good. Cloudflare’s network-wide OpenAI figure for July 2025 was 1,091 to 1. We are converting OpenAI crawls into clicks roughly three and a half times better than the wider web.
The reason is buried in which OpenAI bot is doing the crawling. Of those 6,564 hits, 5,443 were ChatGPT-User. That is not the training crawler. ChatGPT-User is the fetch that fires when a person is mid-conversation and ChatGPT goes to read a page to answer them right now. Only 368 hits were GPTBot, the bulk training crawler, and 753 were OAI-SearchBot.
So most of OpenAI’s interest in Topickz is live, human-attached demand, not warehouse scraping. When ChatGPT pulls one of our listicles to answer “what’s the best CRM for a small sales team,” a real slice of those people click the source. That is the AI channel actually working the way the optimists promised it would.
Perplexity reads us and ghosts us
Now the uncomfortable one. Perplexity crawled Topickz 1,827 times in the month and sent exactly one human. That is 1,827 to 1, and here is the part that stings: Perplexity’s network-wide ratio in the Cloudflare data was 195 to 1, one of the healthier numbers among AI engines.
So Perplexity is not a stingy referrer in general. It refers plenty of traffic across the web. It just does not refer it to us. That flips the usual complaint on its head. This is not “Perplexity takes and never gives,” it is “Perplexity has decided our pages are worth reading 1,827 times and worth citing to a human roughly never.” That is a Topickz problem to solve, not a Perplexity conspiracy to write about.
The honest read is that we are getting crawled for coverage and not surfaced as the answer. Being in the index is not the same as being the citation.
Anthropic reads, and refers nobody
Anthropic’s ClaudeBot crawled 831 pages and referred zero humans in the entire window. Not a low number. Zero.
That is not us being singled out. Anthropic’s network-wide crawl-to-refer ratio in the Cloudflare data was 38,066 to 1, the worst of any major operator by a wide margin. Claude, as a product, mostly does not put clickable source links in front of users the way ChatGPT and Perplexity do, so the referral simply never happens. If your traffic model depends on Claude sending readers, the current answer is that it does not, for anyone.
We are not going to pretend one month of zero is a permanent law. Claude’s browsing and citation behavior is changing quickly. But as of this window, ClaudeBot is pure extraction from a referral standpoint.
What this means if you run a content site
The lazy takeaway is “AI steals traffic, block the bots.” Our data says something more specific and more useful.
The engines are not interchangeable. Treating “AI traffic” as one bucket averages a channel that works (OpenAI, because of ChatGPT-User) with two that currently do not (Perplexity for us, Anthropic for everyone). If you are deciding where to spend effort, the live-fetch engines are the ones paying you back today.
Getting crawled is table stakes and getting cited is the whole game. Perplexity crawling us 1,827 times proves the content is discoverable and relevant. The single referral proves discoverability is not conversion. The work is being the source the model quotes with a link, not just one of the pages it read on the way to an answer.
And the raw volume matters for infrastructure, separate from referrals. AI is 61% of our crawler load now. Even the engines that send you nothing are consuming real bandwidth and compute to read you. That is a cost whether or not it ever becomes a visitor.
Why we are publishing ratios, not raw counts
You will notice we lead with ratios and per-engine behavior rather than a single “AI sent us X visitors” number. That is deliberate. Topickz is a young site, our absolute human numbers are small, and a raw count would tell you more about our age than about AI referral behavior. A ratio normalizes that out. 313 to 1 versus 1,827 to 1 is a real, comparable signal no matter how big the site is.
The crawl-versus-referral gap itself is not our discovery. Cloudflare has been publishing network-wide versions of this for over a year, and their numbers are the benchmark we lean on here. What is new is the first-party view: our own logs, our own engines, measured the same way, showing that the averages hide a three-way split worth acting on.
Methodology
Crawl data: every request to Topickz passes through a Netlify edge function that reads the User-Agent, matches it against a curated list of AI, search and SEO crawlers, and increments sharded daily counters. It is fail-open and never blocks a page. We summed the counters for June 5 to July 4, 2026. Bots were grouped by operator (OpenAI = GPTBot + OAI-SearchBot + ChatGPT-User; Anthropic = ClaudeBot + Claude-User; Perplexity = PerplexityBot + Perplexity-User).
Referral data: human sessions whose source was an AI tool (chatgpt.com, perplexity.ai, and the other AI referrers) over the same 30 days, from our analytics. Bots do not run JavaScript, so they never appear in this count. That is exactly why the two data sets can be divided cleanly, one sees only bots, the other sees only humans.
Benchmark: Cloudflare’s published crawl-to-refer ratios for July 2025, from their engineering blog. Their figures were OpenAI 1,091:1, Perplexity 195:1, Anthropic 38,066:1, and Google 5.4:1 for that period. These ratios move month to month; treat them as the shape of the web, not a fixed constant.
Limits: one month, one site, one category (B2B SaaS reviews). Referral counts at this scale carry sampling noise, which is another reason we report ratios and directional behavior rather than precise conversion rates. We will refresh this quarterly as the window grows.
Cite this report
Free to reference with a link back to this page. Suggested credit: “Topickz AI Crawl-to-Referral Gap 2026 (topickz.com/research/the-ai-crawl-referral-gap-2026/).”
To embed the chart:
<a href="https://topickz.com/research/the-ai-crawl-referral-gap-2026/">
<img src="https://topickz.com/images/research/ai-crawl-to-referral-gap-2026.svg"
alt="Topickz AI Crawl-to-Referral Gap 2026" width="760">
</a>
<p>Source: <a href="https://topickz.com/research/the-ai-crawl-referral-gap-2026/">Topickz AI Crawl-to-Referral Gap 2026</a></p>
Related reading: how to get your brand recommended by AI , the AI recommendation scorecard , and our review methodology .