Choose which AI crawlers to allow (so they can cite you) or block (so they can’t use your content). Our recommendation favours allowing search/citation bots and lets you block pure training bots.
Add these lines to the robots.txt file at the root of your domain. Existing rules for Googlebot etc. are unaffected.
Save this as llms.txt at the root of your domain (/llms.txt). It’s an emerging, optional standard that gives AI assistants a curated map of your site.
Free AI crawler control & llms.txt generator
This free tool — built by a Singapore AEO/GEO agency — does two jobs. First, it generates the robots.txt directives to allow or block the AI crawlers behind ChatGPT, Claude, Perplexity, Gemini and more — with an expert recommendation for each. Second, it generates an llms.txt file, an emerging standard that gives AI assistants a curated map of your most important pages. Everything runs in your browser.
Should you allow or block AI crawlers?
It depends on your goal, and it’s worth separating two different kinds of bot:
- Citation / search bots (OAI-SearchBot, PerplexityBot, ChatGPT-User, Google-Extended) fetch your content so an AI can answer with it and cite you. If you want AI-search visibility — and most brands should — allow these. Blocking them removes you from AI answers entirely.
- Training bots (GPTBot, CCBot, Bytespider) crawl to train models, often without attribution. Blocking these is a reasonable choice if you don’t want your content reused for training — and it does not stop the citation bots above from citing you.
Our default “recommended” preset reflects this: allow the citation bots, leave the pure-training bots to your discretion. Note that Google-Extended only controls Gemini and AI-feature use — blocking it does not affect your normal Google Search rankings.
What is llms.txt?
llms.txt is a plain-text file placed at your site root (/llms.txt) that gives AI assistants a clean, human-readable summary of what your site is and which pages matter most. It’s inspired by robots.txt and sitemap.xml, but where robots.txt controls access, llms.txt provides context. It’s an emerging, community-driven convention — not yet an official standard — but it’s low-effort, low-risk, and a signal that you’re building for AI search.
Frequently asked questions
Will blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI’s crawler and has nothing to do with Googlebot or your Google Search rankings. Similarly, Google-Extended only governs Gemini and AI features — not classic Google indexing.
If I want to be cited by ChatGPT, which bots must I allow?
Allow OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot and Perplexity-User for Perplexity, and Google-Extended to stay eligible for Google’s AI Overviews. Our “recommended” preset does this for you.
Is llms.txt actually used by AI companies yet?
Adoption is early and voluntary — no major AI company has formally committed to it. But it costs nothing to add, can’t hurt, and positions your site as AI-ready. We treat it as a sensible part of a broader GEO strategy, not a magic bullet.
