Bug Circuit

Free tool · no signup

Robots.txt tester and generator

Paste your robots.txt, pick a crawler and test any URL to see whether it is allowed and which rule decided. Or build a new file from presets, including a block list for AI crawlers.

By the Bug Circuit security team · Last updated

Your robots.txt

Result for Googlebot

  • /Allowed

    No rule in the "*" group matches this URL, so it is allowed.

  • /private/notes.htmlBlocked

    Disallow: /private/ (line 2) is the longest matching rule in the "*" group.

  • /private/press-kit.htmlAllowed

    Allow: /private/press-kit.html (line 3) is the longest matching rule in the "*" group.

  • /files/report.pdfBlocked

    Disallow: /*.pdf$ (line 4) is the longest matching rule in the "*" group.

  • robots.txt advertises sensitive paths

    Anyone can read this file, and attackers do. Listing /private/ tells them those paths exist. Protect them with a login or a server rule instead, and leave them out of robots.txt.

    Medium
What is robots.txt? A text file at the root of your site that asks crawlers to skip certain paths. Paste yours, choose a crawler and a URL, and the tester shows whether it is allowed and which rule won. It is a request to polite crawlers, not a lock: never rely on it to protect private pages.

How it works

How a crawler reads the file

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines naming a crawler (or * for everyone) and is followed by Allow and Disallow rules. A crawler uses the group that names it. If none does, it uses the * group. A group that names a crawler replaces the * group for that crawler, the two are not combined.

Inside the group, the most specific rule wins, and specific means the longest path. With Disallow: /private/ and Allow: /private/press-kit.html, the press kit page is allowed because its rule is longer. If an Allow and a Disallow match with the same length, Allow wins. A * matches any run of characters, and a $ at the end means the URL must end there, so Disallow: /*.pdf$ blocks /a/report.pdf but not /a/report.pdf?download=1. This tester follows those rules, which are written down in RFC 9309.

What robots.txt does not do

It is not security. The file is public, and anyone can read it, including attackers looking for an admin folder or a backup directory. Listing /admin/ or /backup/ in robots.txt tells them those paths exist. Protect private areas with a login or a server rule, and leave them out of the file. The tester flags paths that look sensitive.

It does not remove pages from Google. A blocked URL can still appear in results, without a description, if other sites link to it. To keep a page out of search, let it be crawled and add a noindex meta tag or an X-Robots-Tag header. Google ignores a Noindex line inside robots.txt, and so does this tester.

Blocking AI crawlers

Several AI companies publish crawler names you can block, such as GPTBot, ClaudeBot, PerplexityBot and CCBot. Blocking them is a request, and honouring it is voluntary. Some names, like Google-Extended and Applebot-Extended, are controls rather than crawlers: they tell the company not to use your content for AI training, and do not change how Google Search or Applebot index you. Blocking an AI search crawler can also keep your pages out of that product's answers, so decide per crawler.

Worked examples

A WordPress site

Block the admin area but allow the one admin file front-end features need, and publish the sitemap.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

The longer Allow rule beats the shorter Disallow, so admin-ajax.php stays reachable. Without it, some themes and plugins break for crawlers that render pages.

Longest rule wins

Rules: Disallow: /private/ and Allow: /private/press-kit.html. Test /private/notes.html and /private/press-kit.html.

/private/notes.html      Blocked  (Disallow: /private/)
/private/press-kit.html  Allowed  (Allow: /private/press-kit.html)

The Allow rule is 23 characters long and the Disallow rule is 9, so the Allow rule decides for the press kit.

Keep AI training crawlers out, stay in search

Block three AI crawlers, allow everyone else.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

User-agent: *
Disallow:

Googlebot and Bingbot use the * group, which blocks nothing, so search visibility is unchanged.

Frequently asked questions

Where does robots.txt go?
In the root of the host it controls, so https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own file.
What is the difference between robots.txt and noindex?
robots.txt controls crawling: whether a crawler may fetch a URL. noindex controls indexing: whether a fetched page may appear in search results. To keep a page out of results, use noindex and let it be crawled, because a crawler that cannot fetch the page never sees the tag.
Does Disallow remove a page from Google?
No. It stops Google reading the page, but the URL can still be listed if other sites link to it. Use a noindex tag, a password or a 404 or 410 status to remove a page.
Does Google follow Crawl-delay?
No. Bing and some other crawlers honour it, but Google ignores it. Google adjusts its own crawl rate, and you can report a problem in Search Console.
How do the * and $ wildcards work?
* stands for any run of characters, and $ at the end of a pattern means the URL must end there. Disallow: /*.pdf$ blocks every URL that ends in .pdf. Disallow: /search* is the same as Disallow: /search because patterns already match any URL that starts with them.
What happens if my robots.txt is missing or broken?
A 404 means there are no restrictions and everything may be crawled. A server error (5xx) makes Google pause crawling the site until the file loads again, so keep the file reliably available.
Should I block AI crawlers?
It depends on what you want. Blocking training crawlers keeps your pages out of future models but does not remove what was already collected. Blocking AI search crawlers can keep you out of that product's answers. Compliance is voluntary either way.

Keep reading

Security Headers CheckerGrade the headers your site sends.Website Technology CheckerSee what your site reveals to anyone.How to scan a websiteFree tools in the right order.Is my website hackable?A self-check of your real attack surface.

See what attackers see, for free

Run the free passive check on your website. No login, no impact on your site, results in seconds.

Ready for the full manual audit? See pricing

Robots.txt Tester and Generator (Free) · Bug Circuit