How to Configure robots.txt for AI Crawlers (GPTBot, ClaudeBot, PerplexityBot)

Your robots.txt file is the first thing AI crawlers check before accessing your site. If it blocks GPTBot, ClaudeBot, or PerplexityBot, your content will not appear in ChatGPT, Claude, or Perplexity search results. This guide shows you how to configure robots.txt for AI search visibility.

The major AI crawlers, by purpose

Each AI engine operates several crawlers with distinct user-agents — and the purpose matters. Search-class agents decide whether you can appear in answers at all; user-browsing agents fetch pages when a user asks; training agents decide whether you are learned from.

• •Search-class: OAI-SearchBot (ChatGPT Search), Claude-SearchBot (Anthropic), PerplexityBot
• •User-browsing: ChatGPT-User, Claude-User, Perplexity-User (fetch pages a user explicitly asks about)
• •Training-class: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot

A practical stance: always allow search-class and user-browsing agents (they drive visibility), and decide training-class agents according to your content policy.

How to check your current robots.txt

Visit yourdomain.com/robots.txt in a browser. Look for User-agent lines that mention the agents above. If you see "Disallow: /" under any of them, that crawler is blocked.

You can also use GEO Probe — our robots.txt check parses the file per RFC 9309 and reports which of 14 AI crawler agents are allowed or blocked, with the wildcard-Disallow edge case handled.

Recommended robots.txt for AI visibility

The simplest approach: allow all AI crawlers by not mentioning them in robots.txt. If you want explicit control, add Allow rules per agent. If you have private areas, disallow those paths specifically while keeping content areas open.

Common mistakes

Many sites unintentionally block AI crawlers. The most common mistakes: using a blanket "Disallow: /" for all user-agents, blocking search-class agents along with training-class ones, or assuming robots.txt is the only gate — a CDN/WAF rule can block AI crawlers at the edge regardless of robots.txt. Our audit checks both layers independently.