Robots.txt Tester
Enter a site's URL to fetch its robots.txt and see the parsed rules for each user agent.
Overview
robots.txt tells crawlers which parts of a site they're allowed to visit, on a per-user-agent basis. This tool fetches a site's robots.txt from its root and parses the plain-text format into structured rules — which user agents are addressed, what paths each disallows or explicitly allows, and any Sitemap directives listed — instead of reading the raw file by eye.
Examples
A typical robots.txt
https://example.com
User-agent: * Disallow: /admin/
How It Works
- Enter any URL on the site you want to check — the tool fetches robots.txt from its root automatically.
- Complete the verification check if shown.
- Click Check to fetch and parse robots.txt.
- Review the rules per user agent and any linked sitemaps.
Use Cases
Confirming a path is actually blocked
Check that a Disallow rule intended for a specific section is parsed the way you expect.
Finding a site's sitemap location
See every Sitemap directive listed in robots.txt, which is often the canonical place a sitemap URL is published.
Tips
- A missing robots.txt (404) means crawlers are allowed to access everything by default — it's a normal, valid state, not an error.
- Rules are matched by the crawler against the most specific applicable User-agent block, falling back to the * (all crawlers) block if there's no exact match — this tool shows every block as parsed, but doesn't simulate that matching logic for a specific crawler.
FAQ
It means the site hasn't published one, which by convention allows crawlers to access everything — not that anything is broken.
This tool makes an outbound request to a URL you choose, so a quick bot check helps prevent the tool being used to flood arbitrary sites with automated traffic.