Node&LinkDarwin Santos

Resource · Directory

The AI bots
visiting your site.

Who is asking for your pages, what they feed, how many people stand behind them, and how to check that it really is them. Sourced from each operator’s own documentation.

25 entries9 operatorsChecked JSONCSV

Read this first.

A user-agent string is a claim. Anyone can send GPTBot or Googlebot. Before you allow, block or count a bot, check the request came from the operator, using the method each entry lists.

Purpose matters more than the company. One operator can run a search crawler you want, a training crawler you may not, and a fetcher that acts for a person in real time. They often have different names, different IP files and different robots.txt behavior.

Each record was checked against the operator's own documentation on 5 November 2026. IP-range files were requested on that date and returned HTTP 200 unless a record says otherwise. Operators change these pages and files; check the linked source before you rely on a value.

Showing 25 of 25 entries.

All entries at a glance

Ranked by estimated people reached. Select a column heading to sort; select it again to reverse. How the figures work.

AI crawlers, fetchers and agents ranked by estimated people reached. Column headers are buttons that sort the table.
1facebookexternalhitOther purpose-specific fetchersMetaLink previews on Facebook, Instagram and Messenger3.64Bdaily users · Meta apps · Oct 2026 [1]ASN routesMeta says it might bypass robots.txt for security or integrity checks.
2ApplebotSearch and answer indexingAppleSiri, Spotlight, Safari2.5Bactive devices · Apple devices · Jan 2026 [2]Reverse DNS, IP listRespects robots.txt rules aimed at Applebot.
2GooglebotSearch and answer indexingGoogleGoogle Search, AI Overviews, AI Mode, Discover, Google Images, Google News2.5Bmonthly users · Google AI Overviews · May 2026 [3]Reverse DNS, IP listGoogle's common crawlers always obey robots.txt when crawling automatically.
4bingbotSearch and answer indexingMicrosoft BingBing1Bmonthly users · Bing · Apr 2026 [4]Reverse DNS, IP listUNVERIFIED here: Bing's current robots.txt help page renders only with JavaScript and could not be read for this check.
4Google-ExtendedControl tokens without their own crawlerGoogleGemini model training, Gemini Apps grounding, Vertex AI grounding1Bmonthly users · Gemini app · Aug 2026 [5]Not applicableA standalone robots.txt product token with no separate HTTP user-agent string.
4Meta-ExternalAgentModel training and open datasetsMetaMeta model training, Meta product indexing1Bmonthly users · Meta AI · May 2025 [6]ASN routesMeta documents disallowing it by name in robots.txt and says changes can take up to 24 hours.
4Meta-ExternalFetcherLive user requests and agentsMetaMeta AI agentic features1Bmonthly users · Meta AI · May 2025 [6]ASN routesMeta says it may bypass robots.txt because the fetches were requested by the user.
4Meta-WebIndexerSearch and answer indexingMetaMeta AI1Bmonthly users · Meta AI · May 2025 [6]ASN routesMeta says allowing it in robots.txt helps Meta cite and link your content in Meta AI responses.
9ChatGPT agentLive user requests and agentsOpenAIChatGPT agent900Mweekly users · ChatGPT · Mar 2026 [7]Web Bot AuthNot stated on the allowlisting page.
9ChatGPT-UserLive user requests and agentsOpenAIChatGPT, Custom GPTs and GPT Actions900Mweekly users · ChatGPT · Mar 2026 [7]IP listBecause the visits are user-initiated, OpenAI says robots.txt rules may not apply.
9GPTBotModel training and open datasetsOpenAIOpenAI model training900Mweekly users · ChatGPT · Mar 2026 [7]IP listRespects robots.txt.
9OAI-SearchBotSearch and answer indexingOpenAIChatGPT search900Mweekly users · ChatGPT · Mar 2026 [7]IP listRespects robots.txt.
13Amzn-SearchBotSearch and answer indexingAmazonAlexa400Myearly customers · Alexa · Oct 2026 [8]IP listRespects the Robots Exclusion Protocol.
13Amzn-UserLive user requests and agentsAmazonAlexa400Myearly customers · Alexa · Oct 2026 [8]IP listAmazon says it may not follow all robots.txt directives because user actions can initiate it.
15Claude-SearchBotSearch and answer indexingAnthropicClaude search answers≈11.3Mdaily users · Claude · Mar 2026Estimate [9]IP listAnthropic says its bots honor industry-standard robots.txt directives.
15Claude-UserLive user requests and agentsAnthropicClaude≈11.3Mdaily users · Claude · Mar 2026Estimate [9]IP listAnthropic says its bots honor robots.txt.
15ClaudeBotModel training and open datasetsAnthropicAnthropic model training≈11.3Mdaily users · Claude · Mar 2026Estimate [9]IP listHonors robots.txt, per subdomain.
–AmazonbotModel training and open datasetsAmazonAmazon products and services, Amazon AI modelsn/aIP listRespects the Robots Exclusion Protocol and robots meta tags.
–Applebot-ExtendedControl tokens without their own crawlerAppleApple foundation model trainingn/aNot applicableApple says Applebot-Extended does not crawl webpages; it is read from your robots.txt rules.
–CCBotModel training and open datasetsCommon CrawlCommon Crawl open datasetn/aReverse DNS, IP listCommon Crawl documents blocking it with User-agent: CCBot and Disallow: /.
–Google user-triggered fetchersLive user requests and agentsGoogleGemini Notebook, Google Read Aloud, Google Messages previews, Feedfetcher, Chrome Web Storen/aReverse DNS, IP listGoogle says these fetchers generally ignore robots.txt because a user requested the fetch.
–Google-AgentLive user requests and agentsGoogleGoogle-hosted agents acting for usersn/aIP list, Web Bot AuthGoogle says its user-triggered fetchers generally ignore robots.txt because a user requested the fetch.
–OAI-AdsBotOther purpose-specific fetchersOpenAIChatGPT ads (landing-page checks)n/aIP listNot addressed on OpenAI's bots page.
–Perplexity-UserLive user requests and agentsPerplexityPerplexity answersn/aIP listPerplexity says that because a user requested the fetch, it generally ignores robots.txt rules.
–PerplexityBotSearch and answer indexingPerplexityPerplexity searchn/aIP listPerplexity recommends allowing it in robots.txt to appear in results and says changes can take up to 24 hours to apply.

Model training and open datasets

Crawlers whose collection can be used to train models, or that build open web datasets.

OpenAI

GPTBot

Crawls content that may be used to train OpenAI's generative AI foundation models.

Feeds into
OpenAI model training
People reached
900 million weekly users · ChatGPT · source · rank 9
robots.txt
Respects robots.txt. Disallowing it indicates your content should not be used for training.
Verify with
  • IP list

Anthropic

ClaudeBot

Collects web content that could contribute to Anthropic's model training.

Feeds into
Anthropic model training
People reached
≈11.3 million daily users (estimate) · Claude · source · rank 15
robots.txt
Honors robots.txt, per subdomain. Supports the non-standard Crawl-delay extension. Rules that name ClaudeBot take priority over a general AI-training preference signal for that path.
Verify with
  • IP list

Meta

Meta-ExternalAgent

Crawls for training foundation AI models or for improving products by indexing content directly.

Feeds into
Meta model training, Meta product indexing
People reached
1 billion monthly users · Meta AI · source · rank 4
robots.txt
Meta documents disallowing it by name in robots.txt and says changes can take up to 24 hours.
Verify with
  • ASN routes
IP ranges
Routes announced by AS32934.

Amazon

Amazonbot

Used to improve Amazon products and services. Amazon says it may also be used to train Amazon AI models.

Feeds into
Amazon products and services, Amazon AI models
People reached
n/aAmazon does not name the products it improves.
robots.txt
Respects the Robots Exclusion Protocol and robots meta tags. No crawl-delay support. Changes can take about 24 hours.
Verify with
  • IP list
IP ranges
developer.amazon.com/amazonbot/ip-addresses/Published as a web page, not a JSON file.

Common Crawl

CCBot

Builds Common Crawl's open web dataset. Common Crawl's page describes the data as open for research; it does not list who uses it.

Feeds into
Common Crawl open dataset
People reached
n/aAn open dataset used by many organizations; no meaningful audience figure.
robots.txt
Common Crawl documents blocking it with User-agent: CCBot and Disallow: /.
Verify with
  • Reverse DNS
  • IP list
IP ranges
index.commoncrawl.org/ccbot.jsonLinked from Common Crawl's CCBot page. The file did not load during our check on 5 November 2026.
Reverse DNS
*.crawl.commoncrawl.org (IPv4 only, per Common Crawl)

Live user requests and agents

Fetchers and agents that visit a page because a person asked for something. Several say robots.txt may not apply to them.

OpenAI

ChatGPT-User

Visits pages for user-initiated actions in ChatGPT and custom GPTs, including GPT Actions. OpenAI says it is not used for automatic web crawling.

Feeds into
ChatGPT, Custom GPTs and GPT Actions
People reached
900 million weekly users · ChatGPT · source · rank 9
robots.txt
Because the visits are user-initiated, OpenAI says robots.txt rules may not apply. Use OAI-SearchBot for search eligibility.
Verify with
  • IP list

OpenAI

ChatGPT agent

ChatGPT's agent signs its outbound HTTP requests so sites can recognise it.

Feeds into
ChatGPT agent
People reached
900 million weekly users · ChatGPT · source · rank 9
robots.txt
Not stated on the allowlisting page.
Verify with
  • Web Bot Auth
IP ranges
OpenAI's allowlisting page relies on signatures. No user-agent string or IP list is given there.
Signature-Agent
"https://chatgpt.com"
Signed
Every request, per OpenAI. The key directory returned an Ed25519 key set on 5 November 2026.

Anthropic

Claude-User

Fetches pages when people ask Claude questions.

Feeds into
Claude
People reached
≈11.3 million daily users (estimate) · Claude · source · rank 15
robots.txt
Anthropic says its bots honor robots.txt. Blocking it stops retrieval of your content in response to user queries.
Verify with
  • IP list

Perplexity

Perplexity-User

Fetches a page when a user asks Perplexity a question so it can answer and link to the page. Not used for crawling or model training, per Perplexity.

Feeds into
Perplexity answers
People reached
n/aNo official figure, and no reputable third-party estimate was found.
robots.txt
Perplexity says that because a user requested the fetch, it generally ignores robots.txt rules.
Verify with
  • IP list

Google

Google-Agent

Used by agents hosted on Google infrastructure to browse the web and take actions when a user asks.

Feeds into
Google-hosted agents acting for users
People reached
n/aGoogle does not say which products use it.
robots.txt
Google says its user-triggered fetchers generally ignore robots.txt because a user requested the fetch.
Verify with
  • IP list
  • Web Bot Auth
Signature-Agent
g="https://agent.bot.goog"
Signed
A subset of requests. Google calls its implementation experimental and says to keep IP, reverse DNS and user-agent checks as a fallback.

Google

Google user-triggered fetchers

Product fetchers that request a URL because a person asked, such as Google-GeminiNotebook (sources added to Gemini Notebook), Google-Read-Aloud and GoogleMessages link previews.

Feeds into
Gemini Notebook, Google Read Aloud, Google Messages previews, Feedfetcher, Chrome Web Store
People reached
n/aSeveral small products; no combined figure.
robots.txt
Google says these fetchers generally ignore robots.txt because a user requested the fetch.
Verify with
  • Reverse DNS
  • IP list
IP ranges
developers.google.com/static/crawling/ipranges/user-triggered-fetchers.jsonTwo files: user-triggered-fetchers.json (user-owned) and user-triggered-fetchers-google.json (Google-owned).
Reverse DNS
*.gae.googleusercontent.com or google-proxy-*.google.com

Meta

Meta-ExternalFetcher

Fetches individual links at a user's request, including for agentic AI features.

Feeds into
Meta AI agentic features
People reached
1 billion monthly users · Meta AI · source · rank 4
robots.txt
Meta says it may bypass robots.txt because the fetches were requested by the user.
Verify with
  • ASN routes
IP ranges
Routes announced by AS32934.

Amazon

Amzn-User

Handles user-initiated actions, such as fetching live information to answer Alexa queries. Amazon says it does not crawl for generative AI training.

Feeds into
Alexa
People reached
400 million yearly customers · Alexa (Alexa for Shopping) · source · rank 13
robots.txt
Amazon says it may not follow all robots.txt directives because user actions can initiate it.
Verify with
  • IP list
IP ranges
developer.amazon.com/amazonbot/live-ip-addresses/Published as a web page, not a JSON file.

Control tokens without their own crawler

robots.txt product tokens. They change how already-crawled content may be used. No request ever arrives with these names.

Google

Google-Extended

Controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search.

Feeds into
Gemini model training, Gemini Apps grounding, Vertex AI grounding
People reached
1 billion monthly users · Gemini app · source · rank 4
robots.txt
A standalone robots.txt product token with no separate HTTP user-agent string. Google also says it intends to support the shared AI-training preference signal in early 2027.
Verify with
  • Not applicable
IP ranges
Not applicable: no request carries this name.

Apple

Applebot-Extended

Controls whether content Applebot crawls can be used to train Apple's foundation models. Disallowed pages can still appear in search, and Apple says these rules are not used for ranking.

Feeds into
Apple foundation model training
People reached
n/aApple's foundation models run on a subset of Apple devices; no figure for that subset.
robots.txt
Apple says Applebot-Extended does not crawl webpages; it is read from your robots.txt rules.
Verify with
  • Not applicable
IP ranges
Not applicable: no request carries this name.

Other purpose-specific fetchers

Narrow fetchers that matter for logs and allowlists, such as ad landing-page checks and link previews.

OpenAI

OAI-AdsBot

Checks the safety of landing pages submitted as ads in ChatGPT and only visits those pages. OpenAI says its data is not used to train foundation models.

Feeds into
ChatGPT ads (landing-page checks)
People reached
n/aAd landing-page review only.
robots.txt
Not addressed on OpenAI's bots page.
Verify with
  • IP list

Meta

facebookexternalhit

Fetches a title, description and thumbnail when content is shared on Facebook, Instagram or Messenger.

Feeds into
Link previews on Facebook, Instagram and Messenger
People reached
3.64 billion daily users · Meta apps (Facebook, Instagram, Messenger, WhatsApp) · source · rank 1
robots.txt
Meta says it might bypass robots.txt for security or integrity checks.
Verify with
  • ASN routes
IP ranges
Routes announced by AS32934.

How to verify

Four checks cover every entry here. Use the one the entry lists; when an operator offers more than one, the stronger evidence wins.

1. Reverse DNS, then forward confirm

Look up the hostname for the IP, check its domain against the entry, then resolve that hostname and confirm it returns the same IP. This example is from Google’s verification guide.

Terminal
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

2. Match the published IP list

Most operators publish a JSON file with a prefixes array of ipv4Prefix and ipv6Prefix entries. Fetch it fresh, then test the address. Swap in the file for the token you are checking.

Terminal · Python 3
IP=203.0.113.7   # the address from your logs
curl -s https://openai.com/gptbot.json | python3 -c '
import sys, json, ipaddress
ip = ipaddress.ip_address(sys.argv[1])
prefixes = json.load(sys.stdin)["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
print("match" if any(ip in n for n in nets) else "no match")
' "$IP"

3. Match the operator’s network

Meta publishes no crawler IP file. Its crawler documentation gives this command for the routes its AS32934 announces, and warns that they change often.

Terminal
whois -h whois.radb.net -- '-i origin AS32934' | grep ^route

4. Verify the request signature (Web Bot Auth)

Signed agents send Signature-Agent, Signature-Input and Signature headers under RFC 9421, HTTP Message Signatures. The keys live at /.well-known/http-message-signatures-directory on the Signature-Agent’s host. OpenAI says ChatGPT agent signs its requests; Google signs a subset of Google-Agent requests and calls this experimental. Cloudflare documents the drafts it relies on.

Terminal
# 1. Look for all three headers on the request in your logs or edge:
#    Signature-Agent, Signature-Input, Signature
# 2. Fetch the public keys named by Signature-Agent:
curl -s https://chatgpt.com/.well-known/http-message-signatures-directory
curl -s https://agent.bot.goog/.well-known/http-message-signatures-directory
# 3. Verify the Signature over the components listed in Signature-Input
#    with an RFC 9421 library or a provider that supports Web Bot Auth.
#    A header that is present but not verified proves nothing.

Start from your logs

A first pass over a combined-format access log, where the client IP is the first field. It counts claimed bot requests by IP; every IP still needs one of the checks above.

Terminal
grep -E "Googlebot|bingbot|OAI-SearchBot|GPTBot|ChatGPT-User|OAI-AdsBot|Claude-SearchBot|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|Google-Agent|Applebot|Amazonbot|Amzn-SearchBot|Amzn-User|meta-externalagent|meta-externalfetcher|meta-webindexer|CCBot" access.log \
  | awk '{print $1}' | sort | uniq -c | sort -rn | head -50

Separate search from training in robots.txt

Each operator documents its own tokens; see the entries above. Some user-triggered fetchers say robots.txt may not apply to them.

robots.txt
# An example only: keep search and user fetches, opt out of training.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Who is behind them

Estimated people reached is the newest dated figure for the largest product a bot feeds, using the operator's own number where one exists. Audiences overlap, so the figures are not added together. Metrics differ (daily, weekly or monthly users, devices, or yearly customers), so ranks are a rough guide. Third-party estimates are labeled as estimates. Entries without a meaningful figure show n/a and sort last.

Audience figures used for estimated people reached, with sources
NoteProductFigureTypeAs ofSourceUsed for
1Meta apps (Facebook, Instagram, Messenger, WhatsApp)Meta3.64 billionDaily active people across Meta's appsOfficialMeta: 2026 third-quarter results (Japanese newsroom)Family daily active people for September 2026.facebookexternalhit
3Google AI OverviewsGoogle2.5 billionMonthly active usersOfficialGoogle: Sundar Pichai's I/O 2026 opening keynoteOver 2.5 billion. The same keynote gives AI Mode more than 1 billion monthly active users.Googlebot
2Apple devices (Siri, Spotlight, Safari)Apple2.5 billionActive devices (devices, not people)OfficialApple: first-quarter fiscal 2026 resultsMore than 2.5 billion active devices in Apple's installed base. One person can own several.Applebot
4BingMicrosoft1 billionMonthly active usersOfficialMicrosoft: FY26 Q3 earnings call"Bing monthly active users reached 1 billion for the first time."bingbot
5Gemini appGoogle1 billionMonthly usersOfficialGoogle: The Gemini app passes 1 billion monthly usersMore than 1 billion monthly users.Google-Extended
6Meta AIMeta1 billionMonthly active usersOfficialCNBC, via NBC San Diego: Mark Zuckerberg says Meta AI has 1 billion monthly active usersCompany statement at Meta's 2025 shareholder meeting, as reported by CNBC. No newer official figure was found.Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher
7ChatGPTOpenAI900 millionWeekly active usersOfficialOpenAI: OpenAI raises $122 billion to accelerate the next phase of AIMore than 900 million. A July 2026 report by The Information put it near 1 billion; OpenAI has not published that figure.OAI-SearchBot, GPTBot, ChatGPT-User, ChatGPT agent
8Alexa (Alexa for Shopping)Amazon400 millionCustomers who used Alexa for Shopping in the past 12 monthsOfficialAmazon: third-quarter 2026 results400 million+ customers in the last 12 months. Amazon's crawler page names Alexa; it does not name Alexa for Shopping specifically.Amzn-SearchBot, Amzn-User
9ClaudeAnthropic≈11.3 millionDaily active users of the iOS and Android apps (estimate)EstimateTechCrunch, citing Similarweb (6 March 2026)Third-party estimate of mobile-app daily users only; web and API use are not included. Anthropic has not published a user count.Claude-SearchBot, ClaudeBot, Claude-User

Shown as n/a: PerplexityBot, Amazonbot, CCBot, Perplexity-User, Google-Agent, Google user-triggered fetchers, Applebot-Extended, OAI-AdsBot. Each entry’s card gives the reason.

For your AI agent

Give this page to your AI agent.

Paste this into the agent or assistant that can read your logs. Fill in the bracketed parts. The data file has the same entries as this page.

Instruction
Read the AI bot directory at https://nodeandlink.com/ai-bots/ and its data file https://nodeandlink.com/ai-bots/data/ai-bots.json.

Using my server access logs for [date range] (attached or at [path]), and my robots.txt at [https://example.com/robots.txt]:

1. Find every request whose user agent matches a token in the directory.
2. For each token, count requests and list the source IPs.
3. Verify each IP with the method the directory gives for that token: reverse DNS plus a forward lookup, the operator's published IP-range file, the ASN routes, or the Web Bot Auth key directory. Download IP files fresh; do not rely on copies.
4. Report verified, unverified and spoofed traffic separately, per token.
5. Compare what you found with my robots.txt rules, and note any token that the operator says may not follow robots.txt.

Treat the operator documentation linked in the directory as the authority. The directory was checked on its stated date, so re-check any value that may have changed, and say clearly when something cannot be verified rather than guessing.

Download the directory.

The same 25 entries as this page, with every source URL. Checked .

Download JSON Download CSV