Free tools › AI Robots.txt Generator
AI Robots.txt Generator
Decide which AI crawlers can read your website, then get a ready-to-upload robots.txt file. Allow AI search tools like ChatGPT and Perplexity to find you, opt out of AI training, or block everything, in under a minute.
Free · No sign-up · Nothing you type leaves your browser
Ticked means allowed. Search and assistant crawlers decide whether you can appear in AI answers. Training crawlers collect pages to train future AI models.
Upload this to the root of your site so it opens at yourwebsite.com/robots.txt. On WordPress, Yoast SEO and Rank Math both let you edit robots.txt from the dashboard. robots.txt is a request, not a lock: the major AI companies say their crawlers respect it, but it doesn't enforce anything.
Runs in your browser. Nothing you type is sent to us.
What is the AI Robots.txt Generator?
The EduTechToday AI Robots.txt Generator is a free browser tool that writes a robots.txt file controlling which AI crawlers can read your website. It covers 15 crawlers from OpenAI, Anthropic, Google, Perplexity, Apple, Meta and others, and keeps your existing rules for Google and Bing intact.
How to use the AI Robots.txt Generator
- Pick a preset. Most businesses should start with Allow all AI.
- Untick any crawler you want to block. Each one is labelled as Search, Assistant or Training.
- Paste your current robots.txt if you have one, so your existing rules for Google and Bing are kept.
- Click Generate, then copy or download the file and upload it to the root of your website.
Search crawlers vs training crawlers
AI companies run different crawlers for different jobs. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot find pages to show and link inside AI answers. Assistant crawlers such as ChatGPT-User open a page when someone asks about it in a chat. Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages to train future models.
Blocking a search or assistant crawler can stop that tool from recommending or linking to you. Blocking a training crawler only opts you out of training data. That’s why the middle preset, show up in AI answers, opt out of training, is popular with publishers.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls whether your pages are used to train and ground Gemini models. Google’s AI Overviews are part of Google Search and use the normal Googlebot crawler. This generator never blocks Googlebot or Bingbot, because doing so would remove you from regular search results too.
Should a local business block AI crawlers?
For most local businesses, service firms and B2B companies, no. AI assistants can only recommend businesses they have information about, and your website is one of the main sources they trust. Blocking mostly makes sense for publishers and creators who sell their content.
Why I built this
I built this because most robots.txt guides still only mention GPTBot, while the list of AI crawlers keeps growing. My advice for almost every local business and B2B company is simple: allow them. AI assistants can’t recommend a business they can’t read. Blocking only makes sense if your content is the product you sell.
— Atharva Kashikar, founder of EduTechToday
Allowing crawlers is step one. Getting recommended is the rest.
The AI Visibility Toolkit gives you a 30-day plan, a schema generator, a service page builder and a 40+ directory checklist, so AI tools have something worth citing once they can read your site.
Prefer a done-for-you diagnosis? Book a free content audit, or read the plain-English playbook: Get Your Business Recommended by ChatGPT.
Frequently asked questions
What is an AI robots.txt?
It’s a normal robots.txt file with extra rules for AI crawlers such as GPTBot, ClaudeBot and PerplexityBot. Each rule tells one crawler whether it may read your site.
Do AI crawlers actually follow robots.txt?
OpenAI, Anthropic, Google, Perplexity and Apple all say their named crawlers respect robots.txt. It’s a request, not a lock, so it won’t stop scrapers that ignore the rules.
Where do I upload robots.txt?
At the root of your domain, so it opens at yourwebsite.com/robots.txt. On WordPress, Yoast SEO and Rank Math both include a robots.txt editor.
Will this affect my Google rankings?
No. The generated file keeps Googlebot and Bingbot allowed unless your own pasted rules say otherwise.
How often should I update it?
Check it every few months. AI companies add new crawlers from time to time, so re-run this generator when you hear of a new one.
More free tools
- llms.txt Generator: create an llms.txt file in the official format
- ChatGPT Prompt Finder: find the questions customers ask AI about businesses like yours
- Google Review Link Generator: create a direct Google review link and QR code
- Answer Block Checker: score whether your content is ready for snippets and AI answers
- LocalBusiness & FAQ Schema Generator: generate LocalBusiness and FAQ JSON-LD
- Google Business Profile Category Finder: pick the best primary and additional GBP categories
- All free AI visibility tools
Who built these tools?
Atharva Kashikar, SEO, AEO and GEO consultant with 11+ years in search and B2B marketing, and author of Get Your Business Recommended by ChatGPT. Last updated: 31 October 2026.