Website Crawl Status

Each website has a crawl status that indicates its current state.

Status Types

Your website is fully crawled and ready for analysis.

Initial or weekly crawl is in progress.

Some pages couldn't be crawled. Check for:

Pages blocked by robots.txt

Pages requiring authentication

Server errors or timeouts

The crawl failed completely. Common causes:

Website is offline

Domain has changed

Firewall blocking our crawler

Verify your website is accessible

Check robots.txt isn't blocking us

Ensure pages load within 10 seconds

Contact support if issues persist