Findings · 2026-08-13
Which AI crawlers actually visit a small business site
512
visits recorded so far across the sites we run. This number is read from the dataset each time this page loads — it is not a figure from the day the article was written.
Almost everything written about AI crawlers is a probe: someone sends a request wearing a crawler’s name and reports what came back. Useful, but it is not the same question as who actually turns up.
We started recording that on a handful of sites we operate — counts only, no address stored, no cookie set. The first day produced two things worth saying.
Bytespider arrived most, and went for a PDF
ByteDance’s crawler made more visits than any other named agent, every one of them verified against the ranges its operator publishes, and the path it requested most was not the homepage. It was a PDF sitting in an uploads folder.
Most operators we talk to have never considered Bytespider at all. It is not in the usual list, and it is not what people mean when they say AI crawler.
The same name arrives from two different places
One agent name showed up both verified and unverified on the same day. The verified visits came from the operator’s published ranges. The unverified ones came from a cloud network — in that instance, our own testing, which appears in the data as unverified because it is.
That is the point of separating the two columns. Unverified is not proof of an impostor: plenty of legitimate traffic arrives from ranges nobody publishes. But the name in a user-agent string is a claim, and it is checkable.
Why your access log will not tell you this
A log records the user-agent string. It does not record whether that string was earned. Verification means matching the request against the address ranges the operator publishes — a column no access log carries, and the difference between GPTBot visited you and something called itself GPTBot.
The counts here are a sample from a few sites, not a census of the web, and they are published as such.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.