Does robots.txt block pages from Google?

Disallowing a page in robots.txt prevents Google from crawling it, but does not guarantee it won't appear in search results — Google may still index it if other sites link to it. To fully block a page from search results, use a noindex meta tag or HTTP header instead.

What is the User-agent wildcard (*)?

User-agent: * applies the rules to all crawlers — Googlebot, Bingbot, and every other bot. You can also create rules for specific bots by naming them (e.g. User-agent: Googlebot).

Robots.txt Generator

Build a valid robots.txt file visually. Add rules, set crawl delay, include your sitemap — then copy or download.

Sitemap URL (optional)

Rules

Output

Tips

Use * as the user-agent to target all bots
An empty Disallow allows the bot to crawl everything
Disallow: / blocks the entire site
Test your file in Google Search Console after uploading

What is a robots.txt File?

A robots.txt file is a plain text file placed at the root of your website (e.g. https://example.com/robots.txt) that instructs search engine crawlers which pages they should and shouldn't crawl. It is part of the Robots Exclusion Protocol, a standard followed by all major search engines including Google, Bing, and Yahoo.

While robots.txt does not guarantee a page won't appear in search results, it is essential for preventing crawlers from wasting crawl budget on low-value pages like admin panels, staging directories, or duplicate content.

How to Use This Generator

1Optionally enter your sitemap URL. This will add a Sitemap: directive to the output.
2Click Add Rule to create a new rule block. Set the user-agent (use * for all bots).
3Add Disallow and Allow paths as needed. Add multiple rules for different bots.
4Copy the output or click Download to get the robots.txt file, then upload it to your site root.

Use Cases

Block Admin & Staging Pages

Prevent crawlers from accessing /admin/, /staging/, and /dev/ paths to conserve crawl budget for important pages.

Protect Search & Filter Pages

Block /?s= and faceted navigation URLs like /shop/?color= to prevent duplicate content issues.

Target Specific Crawlers

Create separate rules for Googlebot, Bingbot, or AI scrapers. For example, you can allow Googlebot but block GPTBot from crawling your content.

Declare Your Sitemap

Add your sitemap URL so search engines can discover all your pages faster. This is recommended for every website with more than a handful of pages.

Frequently Asked Questions

A robots.txt file tells search engine crawlers which pages or sections of your website they are allowed or not allowed to crawl. It is placed in the root of your domain and is read by bots like Googlebot before they crawl your site.

Disallowing a page prevents Google from crawling it, but does not guarantee it won't appear in search results. To fully block a page, use a noindex meta tag instead.

User-agent: * applies the rules to all crawlers. You can also target specific bots by naming them, e.g. User-agent: Googlebot.

Yes. Adding a Sitemap: directive pointing to your XML sitemap helps search engines discover all your pages faster.

Upload robots.txt to the root directory of your website so it is accessible at yourdomain.com/robots.txt. Test it in Google Search Console's robots.txt tester.

Related SEO Tools

Meta Tag Generator

Generate SEO, OG, and Twitter Card meta tags.

XML Sitemap Generator

Build and download a valid sitemap.xml file.

UTM Link Builder

Build UTM-tagged campaign URLs instantly.