About our crawler

This page describes the bot that visits your site when someone uses our free analysis tools. It only reads the address the visitor typed and a few files that page needs; it does not fill in forms, log in or keep crawling your site.

Which User-Agent does it use?

robots.txt, llms.txt, sitemap and internal link checks use this identity:

ExDevBot/1.0 (+https://exextramer.com/bot/)

To measure how the page looks to a visitor, the main page request uses a standard browser identity. The AI readiness test also sends one request each as GPTBot, ClaudeBot, ChatGPT-User, PerplexityBot, Googlebot and Bingbot to see whether your site treats them differently; these requests do not pass real IP verification.

How many requests does it send?

A scan sends at most about 15 requests. A domain can only be scanned a few times per minute and results for the same address are cached for 10 minutes. Private networks and local servers are never scanned.

How do I block your crawler?

If you do not want your domain scanned, write to us and we will add it to our blocklist. We report robots.txt rules in our results, but a single-page analysis explicitly requested by a visitor is not a crawl, so robots.txt does not stop it.