Categories
Every agent is sorted into one category so you can filter signal from noise.| Category | What it is |
|---|---|
| AI bot | LLM vendor crawlers (OpenAI, Anthropic, Google, Perplexity, Meta, and more). |
| Agentic commerce | Browser-automation agents running real shopping tasks on your store. |
| Shopping bot | Catalog crawlers that index products for AI shopping. |
| Search engine | Traditional search crawlers (Googlebot, Bingbot, and others). |
| SEO crawler | Competitive-intelligence crawlers (Ahrefs, Semrush, and others). |
| Ads bot | Ad-platform crawlers that check landing pages. |
| Ads verification | Brand-safety and ad-quality crawlers (DoubleVerify, IAS, and others). |
| Social preview | Link-preview fetchers for social platforms (Slack, Discord, X, and others). |
| Feed reader | RSS and feed readers. |
| Uptime monitor | Synthetic monitors (UptimeRobot, Pingdom). Mute these as noise. |
| Security scan | Security scanners (Qualys, Censys, and others). |
| CDN infrastructure | A CDN’s own internal health and cache bots. |
| Scraper | Generic scraper libraries and HTTP clients (curl, Scrapy, Puppeteer). |
| Unknown bot | Matched a bot pattern but couldn’t be classified further. |
Purpose (for AI bots)
Each AI-bot request is tagged with one purpose:Training
Collecting content to train or improve a model. Often high volume.
Search index
Building an index that powers AI answers and citations.
Live fetch
Fetching a page in real time to answer a user’s question. Highest intent.
Vendors
The best-known agents, by vendor. This is a sample; the full set is in the Agent Directory.| Vendor | Example agents |
|---|---|
| OpenAI | GPTBot, ChatGPT-User, OAI-SearchBot, OAI-Operator, ChatGPT-Agent |
| Anthropic | ClaudeBot, Claude-User, Claude-SearchBot, Claude-Code, Anthropic-Agent |
Google-Extended, Googlebot, Storebot-Google, Gemini-Deep-Research | |
| Perplexity | PerplexityBot, Perplexity-User |
| Microsoft | Bingbot, BingPreview |
| Amazon | Amazonbot, Amazon-Rufus, AmazonBuyForMe |
| Meta | Meta-ExternalAgent, Meta-ExternalFetcher |
| Apple | Applebot, Applebot-Extended |
| ByteDance | Bytespider, TikTok-Shop |
| xAI | GrokBot, xAI-Bot |
| DuckDuckGo | DuckAssistBot, DuckDuckBot |
| Mistral | MistralAI-User |
| Cohere | cohere-ai |
| DeepSeek | DeepSeekBot |
| Common Crawl | CCBot |
| Shopify | Shop-Agent, ShopifyAI, Shopify-Commerce-Agent |
Agentic commerce
A fast-growing class of agents that browse, add to cart, and check out for a shopper. Rankly recognizes these so you can see AI-driven shopping on your store:| Agent | Behind it |
|---|---|
Amazon-Rufus, AmazonBuyForMe | Amazon shopping agents |
ChatGPT-Commerce, OAI-Operator | OpenAI checkout and operator agents |
Claude-Commerce | Anthropic shopping agent |
Shop-Agent, ShopifyAI | Shopify shopping agents |
Klarna-Agent, CapitalOne-Shopping | Payment and shopping agents |
Browserbase, MultiOn, Skyvern | General browser-automation agents |
How agents are identified
Rankly identifies an agent from its user-agent string, matched against a tiered set of patterns: hand-curated vendor patterns first, then a large catalog of known agents, then scraper libraries, then synced public lists, and finally a coherence check that catches forged browser user-agents (see The humanity check). The match yields the agent’s name, vendor, category, and purpose.Rankly’s Agent Directory
The full, searchable catalog of every agent Rankly tracks (thousands, across AI crawlers, scrapers, ads bots, and agentic commerce) lives in the Agent Directory. It’s the largest public directory of web agents, updated as new ones appear.Browse the Agent Directory
Search every agent by name, vendor, category, and purpose, with what each one
does and how to identify it.
Don’t see an agent you’re hitting? It may already be in the directory under a
different name, or it’s brand new. Book a call
and we’ll add it.
LLM referrals
Rankly also tracks humans who arrive by clicking a citation in an AI answer. These come from domains likechatgpt.com, perplexity.ai, claude.ai,
gemini.google.com, copilot.microsoft.com, you.com, grok.com, and
mistral.ai. They appear under LLM referrals, separate from crawls.