Site Checks: standing safety checks for crawler access and security hygiene
A new Site Checks page continuously watches your logs for the problems that silently hurt search visibility — Googlebot being blocked or rate-limited, unreliable robots.txt serving, sensitive files exposed — and alerts you the moment a check starts failing.
Some of the most damaging site problems never show up as errors you'd notice. A rate limiter that catches Googlebot. A robots.txt that intermittently returns errors, causing Google to back off crawling your entire site. These issues can cost months of organic traffic before anyone connects the symptoms to the cause — we've seen a real case where exactly this took a team months of external investigation to pin down.
LogLens now checks for these automatically. The new Site Checks page (under Analytics) evaluates your last 7 days of traffic every night against a set of standing safety checks:
Crawler access
- Googlebot isn't being blocked or rate-limited
- Search crawlers aren't receiving 429 responses
- robots.txt is served reliably to crawlers (no 4xx/5xx)
- robots.txt fetch cadence looks healthy (excessive refetching suggests it's being served uncacheably)
- Your sitemap is being fetched and returning 200s
- AI crawler access — which AI bots are allowed, partially blocked, or fully blocked (with no judgement: if blocking is intentional, that's fine)
Security hygiene
- No sensitive files (
.env,.git, backups, config) returning 200 to probes - Bot impersonation levels — how much claimed-Googlebot traffic fails IP verification
- Unidentified bot share — how much of your bot traffic can't be classified
Every verdict comes with the evidence behind it: expand any check to see the actual numbers from your logs. Checks that lack enough traffic to judge are marked N/A rather than guessed at.
When a check starts failing — Googlebot suddenly blocked, robots.txt suddenly erroring — an alert goes out through your email streams immediately. Recoveries are announced too. Steady state stays quiet: the page shows standing status, alerts fire only on change.