AI Crawler Impact Report
Analyse your server logs to discover which AI crawlers are accessing your site, how much bandwidth they consume, and how to control them with robots.txt.
What this tool does
AI crawlers like GPTBot (OpenAI), ClaudeBot (Anthropic), Bytespider (ByteDance), and others are visiting your website regularly to train large language models and power AI search products. Most site owners have no visibility into this activity.
The AI Crawler Impact Report analyses your server log files and gives you a complete picture of AI crawler activity on your site:
- Identify every AI crawler visiting your site, including GPTBot, ClaudeBot, Bytespider, CCBot, Google-Extended, Applebot-Extended, and more
- Calculate bandwidth consumption by each crawler — see exactly how many gigabytes each AI company is downloading from your server
- See request frequency and patterns — which pages AI crawlers visit most and how often they return
- Generate a recommended robots.txt based on your preferences — block all AI crawlers, allow specific ones, or fine-tune access per crawler
- Compare your AI crawler traffic to industry averages, so you can see whether your level of AI crawling is typical or unusual
How it works
Upload logs
Upload a server log file or paste log entries directly. Supports Apache, Nginx, and CloudFront formats.
Instant analysis
Your logs are parsed entirely in the browser. Nothing is sent to a server. Results appear in seconds.
Get your report
See a detailed breakdown by crawler with bandwidth, request counts, and a generated robots.txt.
AI crawlers we detect
| Crawler | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data for GPT models and ChatGPT search |
| ClaudeBot | Anthropic | Training data for Claude models |
| Bytespider | ByteDance | Training data for TikTok AI features |
| CCBot | Common Crawl | Open web dataset used by many AI companies |
| Google-Extended | Training data for Gemini and Bard | |
| Applebot-Extended | Apple | Apple Intelligence training data |
| Meta-ExternalAgent | Meta | Training data for Llama models |
| PerplexityBot | Perplexity | AI search engine indexing |
Example output
Your data stays private. All log analysis happens in your browser using JavaScript. Your log files are never uploaded to any server. Once you close the page, your data is gone.
Need real-time AI crawler monitoring?
This free tool analyses a single log file. For continuous, real-time tracking of every AI crawler visiting your site, with automatic identity verification and historical trends, use LogLens.