How to use the Robots.txt Tester
- Paste the robots.txt rules and the URLs to test, or try the example.
- Choose the crawler, such as Googlebot, Bingbot or GPTBot.
- Select Test URLs and review the result and deciding rule for each URL.
Supported files and limits
- Follows Google's rules (RFC 9309): the crawler's own group applies instead of *, the longest matching rule wins and Allow wins a tie. * and $ are supported.
- Mistakes are reported, such as rules before any User-agent line, and lines Google ignores, like Crawl-delay and Noindex.
- Paste the robots.txt contents; the tool does not fetch it from your website. Test URLs as full addresses or paths starting with /.
- Up to 5,000 URLs and 500 KB of rules.
Common questions
Is this Google's robots.txt Tester?
No. Google retired the robots.txt Tester in Search Console in December 2023. Its robots.txt report only shows the file Google last fetched. This tool tests your own rules against any URL, following the matching rules Google documents, so you can check a change before you publish it.
How do I test robots.txt for Googlebot?
Paste your robots.txt and the URLs to check, choose Googlebot as the crawler and select Test URLs. Each URL is shown as allowed or blocked, with the line that decides it.
Does blocking a page in robots.txt remove it from Google?
No. robots.txt stops crawling, not indexing. A blocked page can still appear in results if other pages link to it. Use a noindex tag, and do not block that page, to remove it.
Why does Googlebot ignore my "User-agent: *" rules?
When a group names Googlebot, Google follows only that group and ignores the * group. Repeat the rules in the Googlebot group if they should apply to both.
How do I block AI crawlers like GPTBot?
Add a group with "User-agent: GPTBot" and "Disallow: /". Then test it here by choosing GPTBot as the crawler.
Is my robots.txt sent anywhere?
No. The rules and URLs are checked in your browser and are never sent to our server or anyone else.