Free robots.txt generator & tester
Your robots.txt file tells search engines and crawlers which parts of your site they may access. This free tool — built by a Singapore SEO agency — generates a clean, valid robots.txt with your disallow rules, sitemap and optional bot-blocking, and lets you test any path against the rules to confirm it does what you think. Get one line wrong and you can accidentally deindex your whole site — so test before you deploy. Everything runs in your browser.
What robots.txt does (and the big caveat)
robots.txt controls crawling, not indexing. It asks well-behaved crawlers not to fetch certain paths — useful for admin areas, carts, faceted-search URLs and duplicate content. But two things trip people up: first, a Disallowd page can still appear in search results if it’s linked elsewhere (to keep a page out of the index, use a noindex tag, not robots.txt). Second, blocking a page also stops Google reading its noindex tag — so never disallow a page you’re trying to deindex. The tester above helps you avoid these mistakes.
Should you block AI crawlers?
This is the modern robots.txt question. Blocking crawlers like GPTBot, Google-Extended and PerplexityBot keeps your content out of AI training and answers — but it also means ChatGPT, Google’s AI Overviews and Perplexity can’t cite you, cutting off a fast-growing traffic source. For most businesses that want visibility, the answer is to allow the AI citation crawlers while blocking data-harvesting SEO scrapers (AhrefsBot, SemrushBot and friends) that only feed competitor tools. This tool separates those two toggles so you can make that call deliberately. To go deeper, use our AI crawler & llms.txt generator and check the result with our AI visibility checker.
How to deploy your robots.txt
Save the output as robots.txt and upload it to your site root so it lives at yourdomain.sg/robots.txt. On WordPress with Rank Math or Yoast, you can edit robots.txt directly in the plugin’s settings instead of uploading a file. After deploying, confirm it in Google Search Console’s robots.txt report, and re-test key paths here to be sure nothing important is blocked.
Frequently asked questions
Where does robots.txt go?
Always at the root of your domain: https://yourdomain.sg/robots.txt. It only applies to the host it’s served from, and crawlers won’t find it anywhere else.
Does Disallow remove a page from Google?
No — it stops crawling, not indexing. A disallowed page can still be listed (often with no description) if other pages link to it. To remove a page from search, allow crawling and add a noindex meta tag, or use the removals tool in Search Console.
What’s the difference between Disallow and noindex?
Disallow in robots.txt blocks crawling; noindex in a meta tag blocks indexing. They’re often confused. If you want a page gone from search, use noindex — and don’t also disallow it, or Google can’t see the noindex.
Will blocking AI bots hurt my SEO?
It won’t affect classic Google rankings (those use Googlebot, which you should always allow), but blocking AI crawlers removes you from AI-generated answers and citations — an increasingly important discovery channel. Block data scrapers if you like, but think twice before blocking the AI citation bots.
