Native verification for Amazonbot, Meta AI and Bytespider — and fixed sitemap syncing
Amazonbot now verifies against Amazon's 524 published crawler IPs instead of coarse cloud ranges, Meta AI and Bytespider verify against their operators' announced networks, and a broken scheduled sitemap sync is fixed fleet-wide.
The final batch from our agency feedback, focused on data quality:
- Amazonbot verifies properly. We previously checked Amazonbot claims
against coarse AWS address blocks, which missed most of Amazon's real crawler fleet — on one engagement, 796k genuine Amazonbot requests showed as "unverified". We now pull Amazon's published crawler IP list (both the main crawler and search variant, ~524 exact addresses, refreshed daily). Re-testing that engagement's data: 99.7% of the previously-unverified requests now match.
- Meta AI and Bytespider verify natively. Meta documents verification
by network ownership (AS32934) rather than a range file, so we verify against Meta's announced prefixes; likewise ByteDance's for Bytespider.
- A false-verification hazard removed. An IP range that belongs to a
different vendor had crept into our Anthropic entry — traffic from that range claiming to be ClaudeBot would have verified. Gone.
- Verification labels, precisely: <em>verified</em> means the IP
matched the operator's official ranges; <em>unverified</em> means it was checked and did not match (possible spoofing); <em>not verifiable</em> means no official ranges exist for that bot. These have always been distinct — worth knowing when you read the numbers.
- Scheduled sitemap syncing is fixed. A permissions regression had
silently broken the nightly sitemap fan-out for every site — sitemap data only refreshed when fetched on demand. Fixed, verified fleet-wide, and the Index Coverage URL view now says explicitly when no sitemap has been synced instead of showing an empty list that reads as "all fine".
New verification applies to traffic from today onward — historical rows keep the verdict recorded when they were ingested.