Measured, not estimated
AI Crawler Index
Which AI systems are actually reading the web — measured from verified user agents in the live server logs of the hashtag.org network. Bots spoofing AI user agents are detected and excluded, so these are real machines, really fetching.
Window: 7 days ending 2026-09-11. Updates weekly.
The crawlers, ranked by appetite
Index crawlers stock AI training sets and answer-engine indexes. Verified fetches on our network this window:
| Crawler | Fetches | Share |
|---|---|---|
| meta-externalagent | 8,026,660 | 91.2% |
| GPTBot | 328,931 | 3.7% |
| Amazonbot | 315,502 | 3.6% |
| ClaudeBot | 72,055 | 0.8% |
| PerplexityBot | 41,394 | 0.5% |
| OAI-SearchBot | 6,844 | 0.1% |
| Bytespider | 2,802 | 0.0% |
| CCBot | 2,287 | 0.0% |
| Google-Extended | 425 | 0.0% |
| Claude-SearchBot | 336 | 0.0% |
| MistralAI | 68 | 0.0% |
Live AI reads — the citation signal
ChatGPT-User, Claude-User, and Perplexity-User fire only while a real person is mid-chat: each hit is an AI reading a page to answer someone right now. This is the traffic AEO earns.
History
| Report date | Crawler fetches | Live AI reads | Probes excluded |
|---|---|---|---|
| 2026-09-11 | 8,797,304 | 1,473 | 1,278 |
| 2026-09-10 | 8,490,867 | 1,323 | 1,061 |
| 2026-09-09 | 8,842,521 | 1,292 | 976 |
| 2026-09-08 | 8,984,772 | 1,135 | 737 |
| 2026-09-07 | 8,952,696 | 1,021 | 576 |
| 2026-09-06 | 10,399,851 | 813 | 590 |
| 2026-09-05 | 9,372,295 | 1,026 | 947 |
| 2026-09-04 | 7,660,123 | 920 | 883 |
| 2026-09-03 | 8,238,117 | 975 | 1,047 |
| 2026-09-02 | 8,896,060 | 982 | 1,009 |
| 2026-09-01 | 8,831,270 | 964 | 1,004 |
| 2026-08-31 | 9,006,614 | 909 | 960 |
Methodology
Source: nginx access logs across the hashtag.org network, parsed weekly. A fetch counts toward a crawler only on an exact verified user-agent match. Requests that present an AI user agent while probing for secrets and config files (a common scanner disguise) are classified as spoofed probes and excluded from every count above. Live-read agents are counted separately from index crawlers because they represent a human conversation in progress. No sampling, no extrapolation — these are raw verified counts from our own infrastructure.
Want AI engines reading — and citing — your pages like this?