SalienceBot

The crawler and site checker operated by Salience (LogLens), on behalf of the site's own team.

If you have seen SalienceBot in your logs, the owner of that website uses Salience for log analysis and has asked us either to check the site (a handful of requests a day) or to crawl it (a polite crawl of its HTML pages). We never crawl a site that is not registered with us by its owner.

The crawler

Builds the site's own link map — every page, how many clicks from the homepage it sits, how many internal links point at it — so the owner can cross it with their server logs: which pages Google ignores, which pages it fetches although nothing links to them, and how crawl attention falls away with click depth.

• Rate: about 3 requests per second per site, on 4 connections, with a global pause whenever the site answers 429 or 503. A Crawl-delay in robots.txt is honoured.
• Scope: HTML pages on the site's own host only. Images, scripts, stylesheets, fonts, media and documents are never requested. Query strings are dropped unless the owner lists specific keys.
• Size: capped per site by its owner (1,000 to 50,000 pages), weekly or on demand. Typically well under an hour.
• Rendering: plain HTML by default. When the owner switches on JavaScript rendering, a small number of pages (the homepage, hub pages and pages whose HTML exposes no links) are rendered in a headless browser; that pass loads no images, fonts or media.
• Discovery: links, canonical and hreflang alternates, the site's sitemap, and URLs already seen in its logs.
• Headers: every request carries the user agent above and From: crawler@salience.com.

Verifying it is us

Salience crawler, JavaScript rendering, robots.txt, sitemap, site-check and platform-detection requests use the fixed IPv4 address 18.132.26.88. Allow-list 18.132.26.88/32 for these requests. Check the source IP as well as the identifying user agent; a user agent alone is not proof of identity.

The site checker

Fetches a handful of URLs per site, once per day: the homepage, robots.txt, the sitemap, and a couple of redirect checks — typically about 8 requests. It checks things request logs cannot show: whether robots.txt and sitemaps are served statically and cacheably, whether crawlers get bucketed into A/B tests, HTTPS and host canonicalisation, and utility-file response times. Requests time out after a few seconds and are never retried aggressively.

Controlling it

• robots.txt: SalienceBot obeys User-agent: SalienceBot rules (and *), including Disallow and Crawl-delay.
• Meta and headers: noindex in a robots meta tag or X-Robots-Tag marks a page as not indexable; rel="nofollow" links are not followed.
• In the dashboard: the site owner can pause crawls and probes at any time under SEO → Crawl Join and Website Settings → Site Check Probes.
• Firewall: block the user agent or the address above if you prefer; the bot identifies itself honestly on every request, and the owner is told when it is blocked.

Contact

Questions or concerns: support@salience.com — include your domain and a log line, and we will respond quickly.

Salience — real-time log analysis and crawler analytics. Formerly LogLens.