Robots.txt Generator
Create a robots.txt with WordPress-friendly defaults, custom rules, sitemap and AI crawler options.
Rank Math uses /sitemap_index.xml. WordPress core uses /wp-sitemap.xml.
robots.txt
"Block everything" stops search engines crawling your whole site. Never use it on a live website.
How to use the Robots.txt Generator
- Start with the WordPress defaults, which block the admin area but keep admin-ajax.php available.
- Add any extra paths you want to block, one per line, and your sitemap URL.
- Choose whether to block AI crawlers such as GPTBot, CCBot and Google-Extended.
- Copy or download the file, then add it to your site root or paste it into your SEO plugin's robots.txt editor.
Why use the Robots.txt Generator?
The robots.txt file tells crawlers which parts of your site they may visit. A small mistake — such as a stray "Disallow: /" left over from a staging site — can stop search engines crawling your whole website. This generator produces a clean, commented file with sensible WordPress defaults and lets you decide separately whether AI companies' crawlers may collect your content. Remember that robots.txt controls crawling, not indexing: to keep a page out of search results, use a noindex tag instead.
Frequently asked questions
Where do I edit robots.txt in WordPress?
WordPress serves a virtual robots.txt by default. In Rank Math, go to Rank Math → General Settings → Edit robots.txt. If a physical robots.txt file exists in your site root, it takes priority.
Does blocking Google-Extended affect my Google rankings?
No. Google says Google-Extended only controls whether your content is used for its Gemini AI models, and does not affect inclusion or ranking in Google Search.
Do all bots obey robots.txt?
Reputable crawlers such as Googlebot, Bingbot and GPTBot follow it, but it is a request, not a lock. Badly behaved bots can ignore it, so never rely on it to hide private content.
Should I block /wp-content/ or /wp-includes/?
No. Search engines need access to your CSS, JavaScript and images to render pages properly.