Site Checks phase 2: active probes and serving-quality checks
Site Checks now includes active probes — LogLens fetches your site directly to verify what logs can't show, like whether robots.txt is served statically, crawlers are kept out of A/B tests, and HTTPS canonicalisation is clean.
Some serving problems never appear in your logs, because they live in the responses your site sends — cookies set on robots.txt, missing cache headers, crawlers being bucketed into A/B experiments. Site Checks now covers these with active probes: LogLens fetches your site directly (a handful of requests nightly, clearly identified as SalienceBot/1.0) and inspects what comes back.
New probe checks
- Site reachable by our checker — if your firewall or bot manager blocks our probe, we tell you plainly rather than guessing. Blocked probes never masquerade as site problems: the dependent checks show as not evaluated, with instructions to allowlist the
SalienceBotuser agent. - robots.txt served statically and cacheably — 200,
text/plain, no cookies, cacheable, stable ETag/Last-Modified. A dynamically-served robots.txt makes crawlers refetch it constantly and trust it less. - Sitemap served properly — correct XML content type, no cookies, cacheable. We follow the
Sitemap:directive in your robots.txt to find it. - Crawlers kept out of A/B tests — we fetch your homepage twice with a crawler user agent; if experiment cookies or variant headers differ between fetches, search engines are seeing your experiments instead of your pages.
- HTTPS and host canonicalisation — http→https and www/apex should each be a single permanent redirect.
- robots.txt responds quickly — time to first byte on your most-fetched utility file.
New serving-quality checks from your logs
- Crawlers are not hitting server errors (5xx rate to verified crawlers)
- Crawler response times are healthy (where your traffic source carries timings)
- Soft 404 level and redirect hygiene for crawlers
Probes are on by default and documented; you can switch them off per site in Website Settings → Site Check Probes. The probe also settles an old ambiguity: when vulnerability scanners hit paths like /.env and get a 200, we now verify whether your site simply serves a catch-all page for every unknown path — so you're only alerted about sensitive-file exposure when it's likely to be real.
On your dashboard, with history
Your site dashboard now shows a Site Checks summary — how many checks are passing, with any failing checks called out in detail and a link to the full page. And every check now keeps a verdict history: a day-by-day strip on the Site Checks page, plus a "Recent changes" list so you can see exactly when a problem started or cleared.