robots.txt is a file placed in your website root directory that tells search engine crawlers which pages they can and cannot crawl. It is part of the Robots Exclusion Protocol — a standard that most major search engines respect.

Does robots.txt prevent pages from being indexed?

No — robots.txt only blocks crawling. If other sites link to a disallowed page, Google can still index it without crawling the content. To prevent indexing, use a noindex meta tag or X-Robots-Tag header instead.

Should I disallow my admin directory?

Yes, it is good practice to disallow /admin/, /wp-admin/, and any other backend paths from crawlers. This reduces unnecessary crawl budget usage and keeps sensitive URLs out of search engine caches.

Where do I put the robots.txt file?

It must be placed in the root of your domain at exactly https://yourdomain.com/robots.txt. In Next.js, put it in /public/robots.txt. In static sites, place it in the root folder alongside index.html.

🤖

Robots.txt Generator

Build robots.txt visually. Block bots, disallow directories, set crawl delay, add sitemap reference. Free online tool.

About this tool

Build a valid robots.txt file visually without memorizing the syntax. Add rules for specific bots or all crawlers, disallow directories and files, set crawl delays, and include your sitemap URL.

User Agent #1

User-agent

Allow (one per line)

Disallow (one per line)

Sitemap URL

Crawl Delay (optional)

Generated robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/


Sitemap: https://example.com/sitemap.xml

How we calculate this

All calculations run entirely in your browser — no data is sent to any server. Here is exactly how this tool works:

Choose which bots to target (all, or specific like Googlebot)
Add Allow and Disallow rules for paths
Set an optional crawl delay
Add your sitemap URL and copy the generated robots.txt

Frequently Asked Questions

Related Tools

Meta Tag Generator

Generate complete SEO meta tags with live Google preview. Title, descrip…

XML Sitemap Generator

Generate and download an XML sitemap for any website. Add URLs, set prio…

URL Slug Generator

Convert any text into a clean URL slug. Handles special characters, spac…

🔒100% Private — runs in your browser

⚡No uploads — instant results

🆓Always free — no sign-up

📱Works on mobile & desktop