Crawler reliability guide
How temporary 5xx responses distort technical SEO audits.
A crawler can observe a real temporary failure or manufacture one by overwhelming the origin. Reliable auditing controls request pressure and preserves enough evidence to tell recovered incidents from confirmed broken pages.
Control the crawler before judging the server
Use a global request budget across pages and assets rather than independent pools that accidentally multiply concurrency. Record the crawl profile with the run so later comparisons remain meaningful.
Verify with bounded attempts
- Retry server errors with explicit backoff
- Run a post-crawl verification pass for unresolved failures
- Preserve timestamp, latency, cache state, fingerprint, and attempt outcome
- Report recovered and confirmed responses as separate evidence classes
Do not overstate the cause
A crawl can prove that a response failed and later recovered. Without server logs it cannot prove an out-of-memory event, application exception, timeout source, or infrastructure kill.
Connect a site and run the workflow yourself.
Get the full technical SEO toolkit on every plan. Choose the number of websites you need and use the same crawling, investigation, and verification workflow.
Get access →