Robots.txt Generator
Create robots.txt files for search engines
Crawler rules
Choose a starting template, then adjust crawler access and path exceptions.
Rule groups
Group 1
Disallow · blocked paths
Allow · exceptions
Your robots.txt
User-agent: * Disallow: /admin/ Disallow: /private/ Allow: /
Preview updated. Review your rules before publishing.
Install on your website
- Copy or download the generated content.
- Save it in a UTF-8 text file named robots.txt.
- Upload it to the root of the matching host, protocol and port.
- Visit yoursite.com/robots.txt and confirm its contents.
- Review the robots.txt report in Search Console after publication.
Before publishing
- robots.txt controls crawling. A blocked URL may still appear in search results.
- To use noindex, allow the crawler to fetch the page and read its meta tag or response header.
- Protect private and staging pages with authentication; robots.txt is publicly readable.
- Keep CSS and JavaScript required to render your pages crawlable.
- Check each template against your own URL structure.
- Use specific user-agents when different crawlers need different rules.
- Paths are case-sensitive; * is a wildcard and $ marks the end of the URL.
Disallow · blocked paths
Allow · exceptions
About Robots.txt Generator
The Robots.txt Generator is a free online tool that helps you create properly formatted robots.txt files for your website. The robots.txt file is a fundamental component of technical SEO that tells search engine crawlers which pages or directories they can or cannot access. Crawler rules can reduce requests to selected paths. Blocked URLs can still appear in search results, and private areas need authentication rather than robots.txt.
A website can use a robots.txt file at the root of its host when crawler rules are needed. Sites can also allow crawling without this file. This generator provides common templates, multiple user-agent groups and optional sitemap and delay fields; review them against your site’s own paths.
For a blog, an online store with filter URLs to keep out of crawls, or a SaaS app with private dashboards, the robots.txt creator lets you configure crawler rules visually, preview the generated file in real time, and copy it with a single click. No coding is required to edit the rules. Directive support varies by crawler: Google supports User-agent, Allow, Disallow and Sitemap, while optional delay and legacy Host fields may be ignored.
Key Features
- Quick templates for basic websites, restrictive setups, WordPress blogs, and e-commerce stores
- Multiple crawler rules with support for different user-agents (Googlebot, Bingbot, Yandex, and more)
- Allow and Disallow path configuration with unlimited entries per rule
- Sitemap URL reference field for improved search engine discovery
- Crawl-delay setting for bots that honor it (Bing and Yandex do; Google ignores it)
- Optional legacy Host line for older Yandex setups (no longer used by Yandex since 2018; use a 301 redirect for domain canonicalization)
- Real-time preview of the generated robots.txt file
- One-click copy to clipboard for quick deployment
- Step-by-step installation instructions included
- Compatible with Google, Bing, Yandex, Baidu, and DuckDuckGo crawlers
How to Use
- Choose a template: Select a quick template (Basic, Restrictive, WordPress Blog, or E-commerce) to pre-fill common rules, or start from scratch.
- Configure crawler rules: Set the user-agent, add Disallow paths for directories you want to block, and add Allow paths for exceptions.
- Set global options: Enter your sitemap URL and, if needed, a crawl delay in the Global Settings section.
- Preview the output: Review the generated robots.txt file in the real-time preview panel on the right.
- Copy and deploy: Click the Copy button to copy the file contents, then save it as "robots.txt" in your website's root directory.
- Test and validate: After uploading, check the robots.txt report in Google Search Console (Settings) to confirm Google fetched and parsed your file.
Use Cases
- Crawl budget optimization: Block low-value pages like admin panels, search results, and tag archives to focus crawler attention on important content.
- Duplicate content prevention: Disallow access to paginated pages, filtered views, and parameter-heavy URLs that create duplicate content.
- Server load management: Set a crawl delay to slow down bots that honor it, such as Bingbot and YandexBot (Googlebot ignores Crawl-delay).
- Staging site protection: Use authentication to restrict access to development or staging environments. robots.txt alone does not prevent their URLs from being indexed.
- WordPress optimization: Block wp-admin and internal search results while keeping admin-ajax.php, themes, and uploads crawlable so pages render and images can be indexed.
- E-commerce tuning: Prevent crawling of cart, checkout, account, and API endpoints while allowing product and category pages.
Frequently Asked Questions
Where should I place the robots.txt file?
The robots.txt file must be placed in the root directory of your website. For example, it should be accessible at https://yourdomain.com/robots.txt for search engines to find it.
Can robots.txt block pages from being indexed?
A blocked URL may still appear in search results. To prevent indexing with noindex, let the crawler fetch the page and read the meta tag or X-Robots-Tag response header. Authentication can protect private pages.
Should I block CSS and JavaScript files?
No. Google needs access to CSS and JavaScript files to render your pages correctly. Blocking them can harm your SEO because Google cannot evaluate your page layout and content properly.
Is robots.txt a security measure?
No. The robots.txt file is publicly accessible and only serves as a suggestion to well-behaved crawlers. Never rely on it to protect sensitive information. Use server-side authentication and access controls instead.
Tips & Best Practices
- Always include your sitemap: Adding a Sitemap directive helps search engines discover all your important pages faster.
- Test after deploying: Use the robots.txt report in Google Search Console and the URL Inspection tool to confirm important pages are not accidentally blocked.
- Be specific with paths: Use precise directory paths rather than broad patterns to avoid accidentally blocking valuable content.
- Use targeted user-agents: If you need different rules for Google vs. Bing, create separate rule blocks with specific user-agent names.
- Review regularly: Update your robots.txt whenever you add new sections, change URL structures, or migrate your website.
- Avoid wildcards carelessly: The * wildcard and $ end-of-URL marker are powerful but can unintentionally block important pages if used incorrectly.