Bot Identifying Googlebot, Bingbot, and AI Crawlers
How to verify a real Googlebot or Bingbot request, the current AI crawler user agents (GPTBot, ClaudeBot, CCBot), and how to block or allow them.
The User-Agent string alone doesn't prove anything
Anyone can send a request with User-Agent: Googlebot in the header — plenty of scrapers do exactly that, hoping a site treats "Googlebot" traffic more permissively (skipping rate limits, bypassing a paywall check, etc.). The User-Agent string is a claim, not proof. Verifying a crawler is genuinely Google, Bing, or another named bot requires checking where the request actually came from.
Verifying Googlebot
Google publishes the IP ranges Googlebot crawls from and documents two ways to confirm a request is genuine:
- Reverse DNS + forward DNS match — do a reverse DNS lookup on the request's source IP; a real Googlebot resolves to a hostname ending in
.googlebot.comor.google.com. Then do a forward DNS lookup on that hostname and confirm it resolves back to the same IP. Both directions must match. - Published IP ranges — Google publishes a JSON list of Googlebot's current IP ranges at developers.google.com/static/search/apis/ipranges/googlebot.json, which can be checked directly instead of doing DNS lookups per-request.
Bing documents an equivalent reverse-DNS verification process for Bingbot, resolving to a hostname ending in .search.msn.com.
AI crawler user agents
Several AI companies crawl the web to train models or answer live queries, and — following pressure over unlicensed scraping — now identify themselves with a distinct, documented User-Agent so site owners can allow or block them deliberately via robots.txt.
| User-Agent token | Operator | Purpose |
|---|---|---|
GPTBot | OpenAI | Crawls pages to train future models |
ChatGPT-User | OpenAI | Fetches a page live when a user asks ChatGPT to browse it |
OAI-SearchBot | OpenAI | Crawls for ChatGPT's search feature results |
ClaudeBot | Anthropic | Crawls pages to train future models |
Claude-User | Anthropic | Fetches a page live during a Claude conversation |
PerplexityBot | Perplexity | Crawls for its answer engine's index |
CCBot | Common Crawl | Public web archive many AI labs train on indirectly |
Google-Extended | Controls use of already-crawled content for Gemini/AI training, separate from Googlebot's search indexing | |
Applebot-Extended | Apple | Controls use of already-crawled content for Apple Intelligence, separate from Applebot's Siri/Spotlight indexing |
Allowing or blocking crawlers in robots.txt
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /private/
User-agent: Googlebot
Allow: /robots.txt is an opt-in convention — reputable crawlers (including all the ones listed above) check it and respect Disallow rules, but the file itself enforces nothing technically. A crawler that ignores it entirely won't be stopped by robots.txt alone; that requires blocking by IP range or User-Agent at the server or firewall level instead.
Frequently asked questions
If a request's User-Agent says Googlebot, is it safe to assume it's really Google?
No. Spoofing the User-Agent header costs nothing, so plenty of scrapers claim to be Googlebot specifically because sites sometimes treat it more permissively. Confirming it requires a reverse-DNS lookup on the source IP (or checking it against Google's published IP ranges) — the header text by itself proves nothing.
Does blocking GPTBot in robots.txt stop OpenAI from ever using my content?
It stops OpenAI's documented crawler from fetching new pages for training going forward, for OpenAI specifically. It has no effect on content already collected before the block, on other companies' crawlers, or on content reached indirectly through a source like Common Crawl that isn't operated by OpenAI.
Is there one robots.txt rule that blocks all AI crawlers at once?
No single documented token covers every AI crawler — each company operates its own bot under its own name, and new ones appear over time. Blocking comprehensively means listing each known User-Agent token individually (or blocking by IP range, which is more maintenance but doesn't rely on the crawler's own self-reported name).