Robots.txt Generator

Generate a clean robots.txt file with a sitemap reference.

Your site URL

Result

Was this tool helpful?

Thanks for the feedback!

Robots.txt Generator creates a starter robots.txt file that lets search engines crawl your site, keeps them out of admin and login pages, and tells them where your XML sitemap lives. Enter your site's address, press Generate, and you'll get a file to upload to the root of your domain.

The file it produces

For https://shop.example.in/ the result is:

User-agent: *
Allow: /
Disallow: /admin
Disallow: /login

Sitemap: https://shop.example.in/sitemap.xml

A trailing slash on your address is removed before /sitemap.xml is added. If the field is left blank, the sitemap line uses https://example.com, which you must change. Line by line:

User-agent: * applies the rules below it to every crawler.
Allow: / permits crawling of the whole site.
Disallow: /admin and Disallow: /login block paths that begin with those words. Where rules overlap, Google follows the most specific one, the longest matching path, so the Disallow lines win over Allow: / for those URLs.
Sitemap: gives the full address of your sitemap. It must be an absolute URL.

Adjust it to your platform before uploading

The two Disallow lines are generic, and matching works on prefixes, so check them against your actual URLs:

Disallow: /admin also blocks /admin-guide and /administration-fees. If you have public pages like that, change the rule to Disallow: /admin/.
WordPress keeps its dashboard at /wp-admin/, so its usual rules are Disallow: /wp-admin/ plus Allow: /wp-admin/admin-ajax.php.
The sitemap isn't always at /sitemap.xml. WordPress's built-in sitemap is /wp-sitemap.xml, while Yoast and Rank Math use /sitemap_index.xml. Open the address in a browser to confirm it loads.

Some site owners also add rules for specific crawlers, for example a User-agent: GPTBot group with Disallow: / to opt out of OpenAI's crawler.

Uploading and testing

Save the text as robots.txt, in plain UTF-8, all lowercase.
Upload it to the top-level folder so it opens at https://yourdomain/robots.txt. A file inside a subfolder is ignored.
Each subdomain needs its own file; blog.example.in doesn't follow the rules at example.in.
Check it in Search Console's robots.txt report, which shows whether Google fetched it and flags errors.

What robots.txt can't do

It isn't security. The file is public, and listing /admin tells anyone where to look. Protect private areas with passwords.
It doesn't remove pages from Google. A blocked URL can still be indexed without its content if other sites link to it. To keep a page out of results, use a noindex tag and leave the page crawlable so Google can see it.
It relies on cooperation. Google and Bing follow it, as described in the Robots Exclusion Protocol standard, RFC 9309, but badly behaved bots may ignore it.

If you don't have a sitemap yet, the XML Sitemap Generator builds one from a list of URLs. To control indexing of individual pages, use the robots tag from the Meta Tag Generator.

Frequently asked questions

Where do I upload the robots.txt file?

In the root folder of your domain, so it opens at yourdomain.com/robots.txt. Crawlers ignore robots.txt files placed in subfolders.

Will Disallow remove a page from Google search results?

No. It only stops crawling, and a blocked page can still be indexed from links. Use a noindex tag on a crawlable page to keep it out of results.

Why does Disallow: /admin also block other pages?

Robots.txt rules match URL prefixes, so any path starting with /admin is blocked. Add a trailing slash, as in /admin/, to limit the rule to that folder.

What is the sitemap address for WordPress?

WordPress's built-in sitemap is at /wp-sitemap.xml, while Yoast and Rank Math use /sitemap_index.xml. Update the Sitemap line to match your site.

Do I need a robots.txt file at all?

It isn't required. Without one, crawlers assume they may crawl everything, but the file is a simple way to point them to your sitemap and block areas like admin pages.