Choose which AI crawlers to allow (so they can cite you) or block (so they can’t use your content). Our recommendation favours allowing search/citation bots and lets you block pure training bots.
Add these lines to the robots.txt file at the root of your domain. Existing rules for Googlebot etc. are unaffected.
Save this as llms.txt at the root of your domain (/llms.txt). It’s an emerging, optional standard that gives AI assistants a curated map of your site.
Free AI crawler control & llms.txt generator
This free tool — built by a Singapore AEO/GEO agency — does two jobs. First, it generates the robots.txt directives to allow or block the AI crawlers behind ChatGPT, Claude, Perplexity, Gemini and more — with an expert recommendation for each. Second, it generates an llms.txt file, an emerging standard that gives AI assistants a curated map of your most important pages. Everything runs in your browser.
Should you allow or block AI crawlers?
It depends on your goal, and it’s worth separating two different kinds of bot:
- Citation / search bots (OAI-SearchBot, PerplexityBot, ChatGPT-User, Google-Extended) fetch your content so an AI can answer with it and cite you. If you want AI-search visibility — and most brands should — allow these. Blocking them removes you from AI answers entirely.
- Training bots (GPTBot, CCBot, Bytespider) crawl to train models, often without attribution. Blocking these is a reasonable choice if you don’t want your content reused for training — and it does not stop the citation bots above from citing you.
Our default “recommended” preset reflects this: allow the citation bots, leave the pure-training bots to your discretion. Note that Google-Extended only controls Gemini and AI-feature use — blocking it does not affect your normal Google Search rankings.
What is llms.txt?
llms.txt is a plain-text file placed at your site root (/llms.txt) that gives AI assistants a clean, human-readable summary of what your site is and which pages matter most. It’s inspired by robots.txt and sitemap.xml, but where robots.txt controls access, llms.txt provides context. It’s an emerging, community-driven convention — not yet an official standard — but it’s low-effort, low-risk, and a signal that you’re building for AI search.
Matching your crawler policy to your business model
There is no universally “correct” setting for AI crawlers — the right choice depends on how your business makes money from content. A publisher that sells subscriptions or ad impressions has a legitimate reason to block training crawlers, because an AI answer that reproduces the substance of an article removes the need to visit. A service business like a Singapore agency, law firm or clinic almost always wants the opposite: being quoted inside ChatGPT, Perplexity or Google’s AI Overviews is free brand exposure to exactly the buyers you’re chasing, and the “lost” pageview was never going to convert on its own anyway.
It also pays to distinguish between the different jobs these bots do. Some crawlers gather data to train future models; others fetch a page live at the moment a user asks a question, to ground a real-time answer with a citation. You can allow the second while blocking the first. Blocking GPTBot stops OpenAI using your pages for training, but if you also block OAI-SearchBot you remove yourself from ChatGPT’s live search answers — usually the opposite of what a growth-focused brand wants. If you’re optimising specifically to be cited by AI assistants, our AEO team can map the full bot list to a policy that fits your goals.
Where these files live and how to maintain them
Both files belong at the root of your domain: robots.txt at /robots.txt and llms.txt at /llms.txt. On WordPress, a physical file in the web root overrides any virtual robots.txt the CMS generates, so check which one is actually being served before you assume your rules are live. After deploying, fetch each URL directly in a browser to confirm the raw text is correct — a caching layer or security plugin can silently serve a stale copy.
Treat these as living documents rather than set-and-forget files. New AI crawlers appear regularly, and their user-agent strings change, so a policy written a year ago may no longer name the bots that matter. Review your rules each quarter, and when you publish a new llms.txt, keep it concise: a curated list of your most useful, canonical pages with short descriptions works better than an exhaustive dump of every URL, which only buries the pages you actually want cited.
Frequently asked questions
Will blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI’s crawler and has nothing to do with Googlebot or your Google Search rankings. Similarly, Google-Extended only governs Gemini and AI features — not classic Google indexing.
If I want to be cited by ChatGPT, which bots must I allow?
Allow OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot and Perplexity-User for Perplexity, and Google-Extended to stay eligible for Google’s AI Overviews. Our “recommended” preset does this for you.
Is llms.txt actually used by AI companies yet?
Adoption is early and voluntary — no major AI company has formally committed to it. But it costs nothing to add, can’t hurt, and positions your site as AI-ready. We treat it as a sensible part of a broader GEO strategy, not a magic bullet.
Can I allow AI crawlers on some pages but not others?
Yes. robots.txt supports per-path rules, so you can Allow a bot to crawl your blog and services while adding a Disallow for gated resources, checkout flows or client-only areas. Order your directives from most specific to least, and test the result, as different crawlers interpret conflicting rules slightly differently.
Does blocking a crawler in robots.txt actually stop it?
It stops well-behaved bots that honour the standard — which includes the major AI companies’ official crawlers. It does not physically prevent access, so a bad actor scraping your site will ignore it. For genuine enforcement you need server-level or firewall blocking; robots.txt and llms.txt are instructions of intent, respected by the reputable services you most want to influence.
