A plain-text file at the root of your site that tells search engine crawlers which paths they can (Allow) and cannot (Disallow) fetch. It is a suggestion, not a security control.

Does Disallow prevent indexing?

Not necessarily — it only stops crawling. Pages already indexed or linked from elsewhere may still appear in search results without a snippet. Use a <meta name="robots" content="noindex"> tag on the page for true de-indexing.

Most-specific rule wins — what does that mean?

When multiple Allow/Disallow rules match a URL, Google follows the one with the longest matching path. A more specific Allow overrides a broader Disallow.

Robots.txt Tester — Free Online Robots.txt Validator

Free Online Tool

Robots.txt Tester

Paste your robots.txt, enter a URL and user-agent — see exactly which rule allows or blocks the crawler. Validates syntax + spots common mistakes.

Robots.txt content

Test a URL

Path or URL to test User-agent

Robots.txt cheat sheet

User-agent: * — applies to all crawlers
Disallow: /admin/ — blocks /admin/ and everything under it
Allow: /admin/public/ — carve-out inside a blocked folder
Disallow: /*.pdf$ — pattern (Google supports * wildcard and $ end-of-URL)
Sitemap: https://example.com/sitemap.xml — points crawlers to your sitemap
Most-specific rule wins. Longer Allow overrides shorter Disallow.

Vikram Chouhan · Udaipur Web Designer®

Need a site that actually ranks?

Stop guessing — hand your SEO to a studio that's ranked 1000+ sites.

Start a Project WhatsApp Us +91-9602841237

Since 2013 · 11+ Years 4.9 / 5 · 87 Google Reviews 1000+ Projects Delivered Serving India + Global

Robots.txt content

Test a URL

Robots.txt cheat sheet

Need a site that actually ranks?

Web Design & Development

SEO & Growth

Software Development