SEO Tools

Robots.txt Generator

Generate a clean robots.txt file with user-agent rules, allow/disallow paths, sitemap URL, and crawl delay.

Browser-based

Use it instantly with no installation.

Local processing

Your input stays in your browser.

No sign-up

Open the tool and get results.

View all SEO Tools

User-agent

Crawl Delay

Allow Paths

Disallow Paths

Sitemap URL

robots.txt

Continue Workflow

Related tools to try next

All tools

Meta Tag Generator

Generate SEO-friendly HTML meta tags, Open Graph tags, and Twitter Card tags for your website.

SERP Snippet Preview

Preview Google-style search snippets and tune page titles, URLs, and meta descriptions.

Open Graph Preview

Preview social sharing cards and generate Open Graph meta tags for pages and posts.

Browser Local No sign-up

Example

Public site robots.txt

Robots.txt gives crawler instructions, but it is not a security control.

Input


                        User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml

Output


                        User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml

Practical Notes

Do not use robots.txt to protect private data. Blocked URLs can still be visible if linked elsewhere.
Place robots.txt at the root of your domain, such as https://example.com/robots.txt.
Use noindex on pages when you need indexing control; robots.txt only controls crawling behavior.

How to Use

1. Choose a user-agent or keep the default wildcard.
2. Add paths you want to allow or disallow, one per line.
3. Optionally add a sitemap URL and crawl delay.
4. Generate and copy the robots.txt content.

Features

Creates robots.txt rules for all bots or a custom user-agent
Supports allow and disallow path lists
Adds sitemap URL and optional crawl delay
Includes common presets for public sites and staging blocks
Outputs copy-ready robots.txt content
Runs locally in your browser

Use Cases

Creating a starter robots.txt file for a new website
Blocking staging or internal paths from crawling
Adding sitemap discovery to a static website
Preparing SEO handoff files for deployment

Frequently Asked Questions

What is robots.txt?

Robots.txt is a plain text file that gives crawler instructions about which paths should or should not be crawled.

Can robots.txt hide private pages?

No. Robots.txt is public and cooperative. It should not be used as an access control or privacy mechanism.

Where should I upload robots.txt?

Upload it to the root of your website so it is available at /robots.txt.