Browse and search all 784 known bot patterns in the classifier database.
| Name | Category | Organization | Pattern |
|---|---|---|---|
| Ai Search BotLLM | ai | — | AISearchBot |
| Ai2 CrawlerLLM | ai | Amazon | AI2Bot\s |
| Ai2 Crawler DolmaLLM | ai | Amazon | Ai2Bot-Dolma |
| AI2BotLLM | ai | Ai2 | AI2Bot |
| AI2Bot-DeepResearchEvalLLM | ai | Amazon | AI2Bot-DeepResearchEval |
| Aihit CrawlerLLM | ai | — | aiHitBot |
| Amazon CrawlerLLM | ai | Amazon | Amazonbot |
| amazon-kendraLLM | ai | Amazon | amazon-kendra |
| AmazonBuyForMeLLM | ai | Amazon | AmazonBuyForMe |
| Amzn-SearchBotLLM | ai | Amazon | Amzn-SearchBot |
| Amzn-UserLLM | ai | Amazon | Amzn-User |
| AndibotLLM | ai | Andi | Andibot |
| AnomuraLLM | ai | Direqt | Anomura |
| Anthropic CrawlerLLM | ai | Anthropic | [cC]laude(?:[bB]ot|-[Ww]eb) |
| Anthropic Crawler SearchLLM | ai | Anthropic | Claude-SearchBot |
| Anthropic Crawler UserLLM | ai | Anthropic | Claude-User |
| anthropic-aiLLM | ai | Anthropic | anthropic-ai |
| ApifyBotLLM | ai | Apify | ApifyBot |
| ApifyWebsiteContentCrawlerLLM | ai | Apify | ApifyWebsiteContentCrawler |
| Applebot-ExtendedLLM | ai | Apple | Applebot-Extended |
| Aranet-SearchBotLLM | ai | — | Aranet-SearchBot |
| atlassian-botLLM | ai | Atlassian | atlassian-bot |
| AwarioLLM | ai | Awario | Awario |
| AzureAI-SearchBotLLM | ai | Microsoft | AzureAI-SearchBot |
| bedrockbotLLM | ai | Amazon | bedrockbot |
| bigsur.aiLLM | ai | Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for businesses | bigsur\.ai |
| Botify CrawlerLLM | ai | Botify | botify |
| BravebotLLM | ai | https://safe.search.brave.com/help/brave-search-crawler | Bravebot |
| Brightbot 1.0LLM | ai | https://brightdata.com/brightbot | Brightbot 1\.0 |
| BuddyBotLLM | ai | BuddyBotLearning | BuddyBot |
| Bytedance CrawlerLLM | ai | ByteDance | Bytespider |
| Ccbot CrawlerLLM | ai | Common Crawl | CCBot\/ |
| Channel3BotLLM | ai | — | Channel3Bot |
| ChatGLM-SpiderLLM | ai | — | ChatGLM-Spider |
| ChatGPT AgentLLM | ai | OpenAI | ChatGPT Agent |
| Claude-WebLLM | ai | Anthropic | Claude-Web |
| ClaudeBotLLM | ai | Anthropic | ClaudeBot |
| Cloudflare-AutoRAGLLM | ai | Cloudflare | Cloudflare-AutoRAG |
| CloudVertexBotLLM | ai | — | CloudVertexBot |
| Cohere CrawlerLLM | ai | Cohere | cohere-ai |
| cohere-training-data-crawlerLLM | ai | Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products | cohere-training-data-crawler |
| Commoncrawl CrawlerLLM | ai | Common Crawl | CCBot |
| CotoyogiLLM | ai | ROIS | Cotoyogi |
| Crawl4AILLM | ai | — | Crawl4AI |
| CrawlspaceLLM | ai | Crawlspace | Crawlspace |
| Datenbank CrawlerLLM | ai | Datenbank | Datenbank Crawler |
| DeepSeekBotLLM | ai | DeepSeek | DeepSeekBot |
| DevinLLM | ai | Devin AI | Devin |
| DiffbotLLM | ai | Diffbot | Diffbot |
| Diffbot CrawlerLLM | ai | Diffbot | Diffbot\/ |
| DuckAssistBotLLM | ai | DuckDuckGo | DuckAssistBot |
| Echobot BotLLM | ai | Echobox | Echobot Bot |
| EchoboxBotLLM | ai | Echobox | EchoboxBot |
| ExaBotLLM | ai | Exa | ExaBot |
| Facebook CrawlerLLM | ai | Meta | Face(book){0,1}[Bb]ot |
| Facebook Share CrawlerLLM | ai | Meta | facebookexternalhit |
| FacebookBotLLM | ai | Meta/Facebook | FacebookBot |
| Factset_spyderbotLLM | ai | Factset | Factset_spyderbot |
| FirecrawlAgentLLM | ai | Firecrawl | FirecrawlAgent |
| FriendlycrawlerLLM | ai | — | FriendlyCrawler |
| Gemini-Deep-ResearchLLM | ai | Gemini-Deep-Research | |
| Glutenfreepleasure CrawlerLLM | ai | — | Gluten Free Crawler\/ |
| Google Crawler CloudvertexLLM | ai | Google-CloudVertexBot | |
| Google-AgentLLM | ai | Google-Agent | |
| Google-ExtendedLLM | ai | Google-Extended | |
| Google-FirebaseLLM | ai | Google-Firebase | |
| Google-NotebookLMLLM | ai | Google-NotebookLM | |
| GoogleAgent-MarinerLLM | ai | GoogleAgent-Mariner | |
| GoogleOther-ImageLLM | ai | GoogleOther-Image | |
| GoogleOther-VideoLLM | ai | GoogleOther-Video | |
| Iask CrawlerLLM | ai | — | iaskspider |
| iAskBotLLM | ai | — | iAskBot |
| iaskspider/2.0LLM | ai | iAsk | iaskspider/2\.0 |
| IbouBotLLM | ai | Ibou | IbouBot |
| Imagesift CrawlerLLM | ai | — | ImagesiftBot |
| imageSpiderLLM | ai | — | imageSpider |
| Img2datasetLLM | ai | — | img2dataset |
| Infegy CrawlerLLM | ai | — | collection@infegy\.com |
| Integralads CrawlerLLM | ai | — | IAS crawler |
| ISSCyberRiskCrawlerLLM | ai | ISS-Corporate | ISSCyberRiskCrawler |
| kagi-fetcherLLM | ai | Kagi | kagi-fetcher |
| Kangaroo BotLLM | ai | — | Kangaroo Bot |
| KlaviyoAIBotLLM | ai | Klaviyo | KlaviyoAIBot |
| KunatoCrawlerLLM | ai | — | KunatoCrawler |
| laion-huggingface-processorLLM | ai | LAION | laion-huggingface-processor |
| LAIONDownloaderLLM | ai | Large-scale Artificial Intelligence Open Network | LAIONDownloader |
| LCCLLM | ai | — | LCC |
| Leadcrunch CrawlerLLM | ai | — | CrunchBot |
| LinerBotLLM | ai | — | LinerBot |
| LinkupBotLLM | ai | — | LinkupBot |
| Manus-UserLLM | ai | — | Manus-User |
| Mediatoolkit CrawlerLLM | ai | — | Mediatoolkitbot |
| Meta CrawlerLLM | ai | Meta | [Mm]eta-[Ee]xternal[Aa]gent |
| Meta Crawler UserLLM | ai | Meta | meta-externalfetcher |
| meta-externalagentLLM | ai | Meta | meta-externalagent |
| Meta-ExternalAgentLLM | ai | Meta | Meta-ExternalAgent |
| meta-externalagent/LLM | ai | Meta | meta-externalagent\/ |
| Meta-ExternalFetcherLLM | ai | Meta | Meta-ExternalFetcher |
| meta-webindexerLLM | ai | Meta | meta-webindexer |
| MistralAI-UserLLM | ai | Mistral | MistralAI-User |
| MistralAI-User/1.0LLM | ai | Mistral AI | MistralAI-User/1\.0 |
| MyCentralAIScraperBotLLM | ai | — | MyCentralAIScraperBot |
| NagetBotLLM | ai | Naget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts) | NagetBot |
| netEstate Imprint CrawlerLLM | ai | netEstate | netEstate Imprint Crawler |
| newsaiLLM | ai | — | newsai |
| NotebookLMLLM | ai | NotebookLM | |
| NovaActLLM | ai | — | NovaAct |
| Ntent CrawlerLLM | ai | Ntent | NTENTbot |
| Omgili CrawlerLLM | ai | Omgili | omgili |
| omgilibotLLM | ai | Webz.io | omgilibot |
| OpenAILLM | ai | OpenAI | OpenAI |
| Openai CrawlerLLM | ai | OpenAI | GPTBot |
| Openai Crawler SearchLLM | ai | OpenAI | OAI-SearchBot |
| Openai Crawler UserLLM | ai | OpenAI | ChatGPT-User |
| OperatorLLM | ai | — | Operator |
| PanguBotLLM | ai | the Chinese company Huawei | PanguBot |
| PanscientLLM | ai | Panscient | Panscient |
| panscient.comLLM | ai | Panscient | panscient\.com |
| Perplexity CrawlerLLM | ai | Perplexity | PerplexityBot\/ |
| Perplexity UserLLM | ai | Perplexity | Perplexity-User\/ |
| Perplexity-UserLLM | ai | Perplexity | Perplexity-User |
| PerplexityBotLLM | ai | Perplexity | PerplexityBot |
| Petalsearch CrawlerLLM | ai | Petalsearch | PetalBot |
| PhindBotLLM | ai | phind | PhindBot |
| Poggio-CitationsLLM | ai | — | Poggio-Citations |
| Poseidon Research CrawlerLLM | ai | Poseidon Research | Poseidon Research Crawler |
| Primal CrawlerLLM | ai | — | Primalbot |
| Python ScrapyLLM | ai | — | Scrapy |
| QualifiedBotLLM | ai | Qualified | QualifiedBot |
| QuillBotLLM | ai | Quillbot | QuillBot |
| quillbot.comLLM | ai | Quillbot | quillbot\.com |
| Safedns CrawlerLLM | ai | SafeDNS | SafeDNSBot |
| Sbintuitions BotLLM | ai | — | SBIntuitionsBot\/ |
| SBIntuitionsBotLLM | ai | SB Intuitions | SBIntuitionsBot |
| Searchatlas CrawlerLLM | ai | — | SearchAtlas |
| Semanticscholar CrawlerLLM | ai | Semantic Scholar | SemanticScholarBot |
| SemrushBot-OCOBLLM | ai | Semrush | SemrushBot-OCOB |
| SemrushBot-SWALLM | ai | Semrush | SemrushBot-SWA |
| Sentione CrawlerLLM | ai | — | SentiBot |
| ShapBotLLM | ai | Parallel | ShapBot |
| Sidetrade indexer botLLM | ai | Sidetrade | Sidetrade indexer bot |
| SpiderLLM | ai | — | Spider |
| Storygize CrawlerLLM | ai | Storygize | StorygizeBot |
| TavilyBotLLM | ai | Tavily | TavilyBot |
| TerraCottaLLM | ai | Ceramic AI | TerraCotta |
| ThinkbotLLM | ai | Thinkbot | Thinkbot |
| Timpi CrawlerLLM | ai | — | Timpibot |
| TwinAgentLLM | ai | — | TwinAgent |
| Velen CrawlerLLM | ai | — | VelenPublicWebCrawler |
| WARDBotLLM | ai | WEBSPARK | WARDBot |
| Webzio Crawler AiLLM | ai | Webzio | webzio-extended |
| Webzio-ExtendedLLM | ai | Webzio | Webzio-Extended |
| wpbotLLM | ai | QuantumCloud | wpbot |
| WRTNBotLLM | ai | — | WRTNBot |
| YaKLLM | ai | Meltwater | YaK |
| YandexAdditionalLLM | ai | Yandex | YandexAdditional |
| YandexAdditionalBotLLM | ai | Yandex | YandexAdditionalBot |
| You CrawlerLLM | ai | — | YouBot |
| ZanistaBotLLM | ai | Zanista | ZanistaBot |
| A6corp Crawler | seo | A6corp | A6-Indexer |
| aAcunetix | security | Acunetix | [aA]cunetix |
| Aboundex Crawler | seo | — | Aboundex |
| Academicbot Rtu | archiver | RTU | AcademicBotRTU |
| Acapbot | seo | — | acapbot |
| Acoon Crawler | seo | — | acoonbot |
| Adagio Crawler | advertising | — | Adagiobot\/ |
| Adbeat Crawler | advertising | Adbeat | adbeat_bot |
| Addsearch Crawler | search_engine | — | AddSearchBot |
| Addthis Crawler | content_fetcher | AddThis | AddThis |
| Admantx Crawler | advertising | AdmantX | ADmantX |
| Adscanner Crawler | advertising | — | adscanner |
| Adstxtcrawler | advertising | — | AdsTxtCrawler |
| Advbot Crawler | advertising | — | AdvBot |
| Adyen Webhook | webhooks | Adyen | Adyen HttpClient |
| Ahrefs Crawler | search_engine | Ahrefs | AhrefsBot\/ |
| Ahrefs Site Audit | advertising | Ahrefs | AhrefsSiteAudit\/ |
| AhrefsBotSiteAudit | seo | Ahrefs | Ahrefs(Bot|SiteAudit) |
| airai/scanning | ai | — | air\.ai\/scanning |
| Alexandria Crawler | search_engine | Alexandria | AlexandriaOrgBot |
| Algolia Crawler | search_engine | Algolia | Algolia Crawler\/ |
| ALittle Client | tool | — | ALittle Client |
| AliyunSecBot | security | Alibaba | AliyunSecBot |
| Alphaseobot Crawler | seo | — | AlphaBot |
| Amazon Adbot | content_fetcher | Amazon | AmazonAdBot\/ |
| Amazon Alexa Crawler | content_fetcher | Dassault Systèmes | exabot |
| Amazon Cloudfront | content_fetcher | Amazon | Amazon CloudFront |
| Amazon Route53 Health Check | content_fetcher | Amazon | Amazon Route 53 Health Check Service |
| AmazonProductDiscovery | search_engine | Amazon | AmazonProductDiscovery |
| AmazonSellerInitiatedListing | search_engine | Amazon | AmazonSellerInitiatedListing |
| Anderspink Crawler | seo | Anderspink | AndersPinkBot |
| Antibot | security | — | antibot |
| Apercite Crawler | seo | Apercite | Apercite |
| Apple Crawler | search_engine | Apple | Applebot |
| Apple Feedfetcher | feed_fetcher | Apple | i[Tt][Mm][Ss] |
| Ara Crawler | search_engine | — | arabot |
| Araturka Crawler | seo | — | Bot\.AraTurka\.com |
| Archiveorg Archiver | archiver | Internet Archive | archive\.org_bot |
| Arocom Crawler | content_fetcher | — | drupact |
| Ask Crawler | search_engine | Ask.com | Teoma |
| asnriskscorer | monitoring | — | asnriskscorer |
| Aspiegel Crawler | seo | — | AspiegelBot |
| Audisto Crawler | seo | Audisto | Audisto Crawler |
| Avira Crawler | search_engine | Avira | SafeSearch microdata crawler |
| Awario Crawler | seo | Awario | AwarioBot |
| Awario Crawler Rss | feed_fetcher | Awario | AwarioRssBot |
| Awario Crawler Smart | seo | Awario | AwarioSmartBot |
| Awesomecrawler | scraping_framework | — | awesomecrawler |
| Azure App Insights | search_engine | Microsoft | AppInsights |
| B2bbot | seo | — | B2B Bot |
| Backlinktest Crawler | seo | — | backlinkcrawler |
| Baidu Cloud Watch | monitoring | Baidu | Baidu-YunGuanCe |
| Baidu Crawler | search_engine | Baidu | Baiduspider |
| Bazqux Feedfetcher | feed_fetcher | Bazqux | BazQux |
| Betabot | scraping_framework | — | betaBot |
| Betterstack Monitor | monitoring | BetterStack | Better Stack |
| Betteruptime Monitor | monitoring | BetterUptime | Better Uptime Bot |
| Bidswitch Crawler | advertising | BidSwitch | bidswitchbot |
| Bigdatacorp Crawler | content_fetcher | — | BDCbot |
| Biglotron | scraping_framework | — | BIGLOTRON |
| Bing Ads | search_engine | Microsoft | Adidxbot |
| Bing Crawler | search_engine | Microsoft | bingbot |
| Bing Office Store | search_engine | Microsoft | officestorebot |
| Bing Preview | search_engine | Microsoft | BingPreview\/ |
| Binlar | seo | Binlar | binlar |
| Bitly Crawler | page_preview | Bitly | bitlybot |
| Bitsight Crawler | security | Bitsight | BitSightBot\/ |
| Blackboard Crawler | archiver | Blackboard | Blackboard |
| Blogmura Crawler | content_fetcher | Blogmura | blogmuraBot |
| Blogtraffic Feedfetcher | feed_fetcher | Blogtraffic | BlogTraffic\/\d\.\d+ Feed-Fetcher |
| Blp Bbot | seo | BLP | BLP_bbot |
| Bnf Crawler | seo | bnf.fr | bnf\.fr_bot |
| Bombora Crawler | advertising | Bombora | BomboraBot |
| Boxcar Crawler | social_media | Boxcar | BoxcarBot |
| Brainobot | ai | — | brainobot |
| Brandonmedia Crawler | seo | Brandonmedia | BrandONbot |
| Brandverity Crawler | monitoring | Brandverity | BrandVerity |
| Brandwatch Crawler | seo | Brandwatch | magpie-crawler |
| Brightedge Crawler | seo | Brightedge | BrightEdge Crawler |
| Bublup Crawler | seo | Bublup | BublupBot |
| Builtwith Crawler | seo | BuiltWith | ^BW\/ |
| Buzzstream Crawler | seo | Buzzstream | buzzbot |
| Capsulink Crawler | seo | Capsulink | CapsuleChecker |
| Careerx Crawler | content_fetcher | CareerX | careerbot |
| cClaudebBot | seo | — | [cC]laude[bB]ot |
| Censys Inspect | monitoring | Censys | CensysInspect\/ |
| Centurybot | scraping_framework | — | centurybot |
| Changedetection Crawler | monitoring | Changedetection | changedetection |
| Checkly | monitoring | Checkly | Checkly |
| Checkly Monitor | monitoring | Checkly | Checkly\/ |
| Checkmarknetwork Crawler | seo | CheckMarkNetwork | CheckMarkNetwork\/ |
| Chlooe Crawler | content_fetcher | — | bot-pge\.chlooe\.com |
| Cincrawdata Crawler | content_fetcher | — | Cincraw |
| Cispa Crawler | archiver | — | CISPA Webcrawler |
| Citeseerx Crawler | archiver | — | citeseerxbot |
| Clickagy Crawler | seo | — | Clickagy |
| Cliqz Crawler | seo | — | Cliqzbot\/ |
| Cloudflare Archiver | archiver | Cloudflare | CloudFlare-AlwaysOnline |
| Cloudflare Healthchecks | monitoring | Cloudflare | Cloudflare-Healthchecks |
| Cloudflare Prefetch | seo | Cloudflare | CloudFlare-Prefetch |
| Cloudflare Security Center | monitoring | Cloudflare | https:\/\/developers\.cloudflare\.com\/security-center\/ |
| Cloudflare Ssl Detector | monitoring | Cloudflare | Cloudflare-SSLDetector |
| Cloudflare Traffic Manager | monitoring | Cloudflare | Cloudflare-Traffic-Manager |
| Cloudsystemnetworks Crawler | content_fetcher | — | Nimbostratus-Bot |
| CMSChecker | seo | — | CMSChecker |
| Coccoc Crawler | search_engine | Coccoc | coccoc |
| Cocolyze Crawler | seo | — | Cocolyzebot |
| Coda Server Fetcher | http_library | Coda | Coda-Server-Fetcher |
| Codewise Crawler | advertising | — | VoluumDSP-content-bot |
| Cognitiveseo Crawler | seo | — | James BOT |
| colly | scraping_framework | — | colly |
| Companybook Crawler | content_fetcher | — | Companybook-Crawler |
| Conductor Crawler | seo | — | Caliperbot |
| Content Crawler Spider | scraping_framework | — | content crawler spider |
| Contextad Crawler | advertising | — | ContextAd Bot |
| Contxbot | seo | — | contxbot |
| Convera Crawler | seo | — | convera |
| Cookiebot Crawler | security | Cookiebot | Cookiebot |
| Cookiehub Scan | tool | — | CookieHubScan |
| Creativecommons Crawler | archiver | — | CC Metadata Scaper |
| Criteo Crawler | advertising | Criteo | CriteoBot |
| Crystalsemantics Crawler | seo | — | CrystalSemanticsBot |
| Curebot Crawler | seo | — | [Cc]urebot |
| Curl | tool | — | ^curl |
| CustomAsyncHttpClient | http_library | — | CustomAsyncHttpClient |
| Cutbot Crawler | seo | — | Cutbot |
| Cxense Crawler | seo | Cxense | cXensebot |
| Cyberpatrol Crawler | security | — | CyberPatrol |
| Dareboost Crawler | seo | Dareboost | DareBoost |
| Datadog Monitor Synthetics | monitoring | Datadog | Datadog\/{0,1}Synthetics |
| DatadogSynthetics | monitoring | Datadog | DatadogSynthetics |
| Datafeedwatch Crawler | content_fetcher | Datafeedwatch | Datafeedwatch |
| Dataforseo Crawler | seo | DataForSEO | DataForSeoBot |
| Datagnion Crawler | seo | Datagnion | datagnionbot |
| Datanyze Crawler | seo | Datanyze | Datanyze |
| Dataprovider Crawler | content_fetcher | — | Dataprovider\.com |
| Datenbutler Crawler | seo | — | mindUpBot |
| Daum Crawler | search_engine | Daum | Daum\/ |
| Dcrawl | tool | — | dcrawl |
| dDirBbuster | scraping_framework | — | [dD]ir[Bb]uster |
| Deadlinkchecker | monitoring | — | deadlinkchecker |
| Deepnoc Crawler | seo | — | deepnoc |
| Deusu Crawler | seo | — | DeuSu\/ |
| Digg Crawler | social_media | Digg | Digg Deeper |
| Digincore Crawler | seo | — | Digincore bot |
| Digitaldragon Crawler | scraping_framework | — | CrawlyProjectCrawler |
| Discord Crawler | page_preview | Discord | Discordbot |
| Discoveryengine Crawler | seo | discobot | |
| Disqus Crawler | monitoring | Disqus | Disqus |
| Dnyz Crawler | seo | — | DnyzBot |
| Domaincrawler Crawler | seo | DomainaCrawler | domaincrawler |
| Domainreanimator Crawler | seo | Domaina | Domain Re-Animator Bot |
| Domainsbot Crawler | seo | Pandalytics | Pandalytics |
| Domainsproject Crawler | archiver | — | Domains Project\/ |
| Domainstats Crawler | seo | DomainStats | DomainStatsBot |
| Domaintools Crawler | seo | Domaintools | SurveyBot |
| Dotnetdotcom Crawler | content_fetcher | ezooms | ezooms |
| Dragonmetrics Crawler | seo | Dragonmetrics | Dragonbot |
| Driftnet Crawler | monitoring | — | InternetMeasurement |
| Dubbot Crawler | monitoring | Dubbot | Dubbotbot |
| Duckduckgo Crawler | search_engine | DuckDuckGo | DuckDuckBot |
| Duckduckgo Crawler Favicons | page_preview | DuckDuckGo | DuckDuckGo-Favicons-Bot |
| Duedil Crawler | seo | Duedil | electricmonk |
| Dynatrace Monitor | monitoring | Dynatrace | RuxitSynthetic |
| Ec2linkfinder | link_checker | — | ec2linkfinder |
| Edister Crawler | seo | Edister | edisterbot |
| Elisabot | scraping_framework | — | elisabot |
| Embedly Crawler | page_preview | Embedly | Embedly |
| Entireweb Crawler | search_engine | EntireWeb | speedy |
| Epfl Crawler | archiver | EPFL | A Patent Crawler |
| Epictions Crawler | seo | Epictions | epicbot |
| Eright Crawler | seo | eRight | eright |
| Europarchive Crawler | archiver | EuroArchive | europarchive\.org |
| Eventures Crawler | seo | Eventures | e\.ventures Investment Crawler |
| Eventures Crawler Batch | seo | Eventures | evc-batch |
| Everyonesocial Crawler | social_media | EveryoneSocial | EveryoneSocialBot |
| Exensa Crawler | seo | Exensa | Bark[rR]owler |
| Experibot Crawler | seo | Experibot | Experibot |
| Extlinks Crawler | seo | ExtLinks | ExtLinksBot |
| Eyeota Crawler | seo | Eyeota | Eyeotabot |
| Ezid Crawler | archiver | EZID | EZID |
| Facebook Catalog | social_media | Meta | facebookcatalog\/ |
| Fast Crawler | search_engine | Fast Search & Transfer | FAST-WebCrawler |
| Fast Crawler Enterprise | search_engine | Fast Search & Transfer | FAST Enterprise Crawler |
| Fedoraplanet Crawler | content_fetcher | Fedora Project | fedoraplanet |
| Feedafever Crawler | feed_fetcher | Feedafever | Fever |
| Feedbin Crawler | feed_fetcher | Feedbin | Feedbin |
| Feedly Feedfetcher | feed_fetcher | Feedly | Feedly |
| Feedspot Feedfetcher | feed_fetcher | Feedspot | Feedspot |
| Femtosearch Crawler | search_engine | Femtosearch | FemtosearchBot |
| Findthatfile Crawler | content_fetcher | FindThatFile | findthatfile |
| Flamingosearch Crawler | search_engine | Flamingo Search | Flamingo_SearchEngine |
| Flipboard Proxy | monitoring | FlipboardProxy | |
| Fluffy | scraping_framework | — | fluffy |
| Fr Crawler | seo | — | fr-crawler |
| Freewebmonitoring Monitor | monitoring | FreeWebMonitoring | FreeWebMonitoring SiteChecker |
| Freshrss Feedfetcher | feed_fetcher | FreshRSS | FreshRSS |
| Freshworks Monitor | monitoring | Freshworks | FreshpingBot |
| Fuelbot | advertising | — | fuelbot |
| Fyrebot | ai | — | Fyrebot |
| G00g1e Crawler | search_engine | — | g00g1e\.net |
| G2reader Crawler | feed_fetcher | G2reader | g2reader-bot |
| G2webservices Crawler | content_fetcher | G2 Web Services | G2 Web Services |
| Garlik Crawler | seo | Garlik | GarlikCrawler |
| Geedo Crawler | search_engine | Geedo | GeedoBot |
| Geedo Crawler Products | search_engine | Geedo | GeedoProductSearch |
| Genieo Crawler | advertising | Genieo | Genieo |
| Gigablast Crawler | search_engine | Gigablast | Gigabot |
| Gigablast Crawler Oss | tool | Gigablast | Gigablast |
| Ginger Crawler | seo | Ginger | GingerCrawler |
| Gnam Gnam Spider | scraping_framework | — | gnam gnam spider |
| Gnowit Crawler | content_fetcher | — | GnowitNewsbot |
| Go Http | http_library | — | Go-http-client |
| Goo Crawler | search_engine | — | ichiro |
| Google Ads Conversions | search_engine | Google-Ads-Conversions | |
| Google Adsbot | search_engine | AdsBot-Google([^-]|$) | |
| Google Adsbot Mobile | search_engine | AdsBot-Google-Mobile | |
| Google Adsense | search_engine | Mediapartners-Google | |
| Google Adsense Googlebot | search_engine | Mediapartners \(Googlebot\) | |
| Google Adwords | advertising | Google-Adwords-Instant | |
| Google Appengine | search_engine | AppEngine-Google | |
| Google Certificates Bridge | search_engine | Google-Certificates-Bridge | |
| Google Crawler | search_engine | Googlebot\/ | |
| Google Crawler Image | search_engine | Googlebot-Image | |
| Google Crawler Mobile | search_engine | Googlebot-Mobile | |
| Google Crawler News | search_engine | Googlebot-News | |
| Google Crawler Other | search_engine | GoogleOther | |
| Google Crawler Safety | search_engine | Google-Safety | |
| Google Crawler Store | search_engine | Storebot-Google | |
| Google Crawler Video | search_engine | Googlebot-Video | |
| Google Favicon | search_engine | Google Favicon | |
| Google Feedfetcher | search_engine | Feedfetcher-Google | |
| Google Inspection Tool | search_engine | Google-InspectionTool | |
| Google Lighthouse | search_engine | Chrome-Lighthouse | |
| Google Physical Web | search_engine | Google-PhysicalWeb | |
| Google Preview | search_engine | Google Web Preview | |
| Google Push Notifications | search_engine | APIs-Google | |
| Google Read Aloud | search_engine | Google-Read-Aloud | |
| Google Site Verification | search_engine | Google-Site-Verification | |
| Google Structured Data Testing Tool | search_engine | Google-Structured-Data-Testing-Tool | |
| Google Web Snippet | search_engine | developers\.google\.com\/\+\/web\/snippet | |
| Google Xrawler | search_engine | google-xrawler | |
| Got | tool | — | sindresorhus\/got |
| Gowiki Crawler | http_library | — | Gowikibot |
| Grapeshot Crawler | seo | — | GrapeshotCrawler |
| Grob Crawler | seo | — | Grobbot |
| Grouphigh Crawler | seo | — | GroupHigh\/ |
| Groupme Crawler | social_media | GroupMe | GroupMeBot |
| Grub | search_engine | — | grub\.org |
| Gslfbot | seo | — | gslfbot |
| Gwene Crawler | content_fetcher | — | Gwene |
| Haosou Crawler | search_engine | — | 360Spider |
| Hatena Crawler | social_media | Hatena | Hatena |
| Headless Chrome | tool | — | HeadlessChrome |
| Headline Crawler | content_fetcher | — | ev-crawler\/ |
| Hoyer Crawler | social_media | Friendica | Friendica |
| Http Get | http_library | — | http_get |
| HTTPie/ | tool | — | ^HTTPie\/ |
| Httrack | archiver | — | HTTrack |
| Hubspot Crawler | monitoring | HubSpot | HubSpot |
| Hydrozen Monitor | monitoring | — | Hydrozen\.io\/ |
| Hydrozenio | monitoring | — | Hydrozen\.io |
| Hypefactors Crawler | seo | — | Buck\/ |
| Hypestat Crawler | seo | — | hypestat |
| Hyscore Crawler | monitoring | — | hyscore\.io |
| Ia Archiver | archiver | Internet Archive | ia_archiver |
| Ideasandcode Crawler | seo | — | ICBot\/ |
| Iframely Preview | page_preview | Iframely | Iframely |
| Imessage Preview | search_engine | — | Facebot |
| Indeed Crawler | content_fetcher | Indeed | IndeedBot |
| Inetdex Crawler | seo | — | INETDEX-BOT |
| Infoo Crawler | seo | — | infoobot |
| Inoreader Aggregator | feed_fetcher | — | inoreader |
| Integromedb Crawler | seo | — | integromedb |
| Intelium Crawler | seo | — | intelium_bot |
| Internetarchive Crawler Oss | archiver | InternetArchive | heritrix |
| Ionos Crawler | content_fetcher | Ionos | IonCrawl |
| Ip Web Crawler | scraping_framework | — | ip-web-crawler\.com |
| Ipip Crawler | tool | — | HTTP Banner Detection |
| Irc Archivebot | archiver | — | ArchiveBot |
| Iskanie Crawler | seo | — | iskanie |
| Iss Crawler | seo | — | filterdb\.iss\.net\/crawler |
| It2media Crawler | seo | — | it2media-domain-crawler |
| Itinfluentials Crawler | seo | — | FindITAnswersbot |
| Jamiembrown Crawler | scraping_framework | — | Jamie's Spider |
| Java Apache Httpclient | http_library | Apache | ^Apache-HttpClient |
| Java Asynchttpclient | http_library | AsyncHttpClient | AHC\/ |
| Java Crawler4j | http_library | — | crawler4j |
| Java Httpunit | http_library | — | httpunit |
| Java Jersey | http_library | Eclipse | HttpUrlConnection |
| Java Jetty | http_library | Eclipse | Jetty |
| Java Okhttp | http_library | Square | okhttp |
| Java Snacktory | http_library | — | Snacktory |
| Javascript Axios | http_library | — | axios |
| Javascript Node Fetch | http_library | — | node-fetch |
| Javascript Phantom | http_library | PhantomJS | PhantomJS |
| Jetslide Crawler | seo | — | Jetslide |
| Jobboerse Crawler | seo | — | JobboerseBot |
| Jooble Crawler | seo | Jooble | Jooblebot |
| Jusprog Crawler | seo | — | Jugendschutzprogramm-Crawler |
| Jyxo Crawler | seo | — | jyxobot |
| K7computing Crawler | seo | K7 Computing | K7MLWCBot |
| Kemvi Crawler | seo | — | Kemvibot |
| Keybase Bot | page_preview | Keybase | KeybaseBot |
| Komodia Crawler | seo | — | KomodiaBot |
| Kosmio Crawler | seo | — | KosmioBot |
| Kuma Monitor | monitoring | — | Uptime-Kuma |
| L9explore | tool | — | l9explore |
| l9scan | security | — | l9scan |
| Landaumedia Crawler | seo | — | Landau-Media-Spider |
| Laserlike Crawler | seo | — | Laserlikebot |
| Law Unimi Crawler | archiver | University of Milan | BUbiNG |
| Lb Spider | scraping_framework | — | lb-spider |
| Leiki Crawler | seo | — | Leikibot |
| Leipzig Findlinks | archiver | — | findlink |
| Leipzig Lcc | archiver | — | ^LCC |
| Lemmy Crawler | social_media | Lemmy | Lemmy |
| Lightspeedsystems Crawler | seo | Lightspeed Systems | LightspeedSystemsCrawler |
| Line Crawler | seo | — | Linespider |
| Linguee Crawler | search_engine | Linguee | Linguee Bot |
| Linkapedia Crawler | seo | — | linkapediabot |
| Linkarchiver | archiver | — | LinkArchiver |
| Linkdex Crawler | seo | — | linkdex |
| Linkedin Crawler | social_media | LinkedInBot | |
| Linkfluence Crawler | seo | — | YaK\/ |
| Linkis Crawler | seo | — | LinkisBot |
| Lipperhey Crawler | seo | Lipperhey | lipperhey |
| Livelap Crawler | seo | — | Livelap[bB]ot |
| Logly Crawler | seo | — | zenback bot |
| Loop Crawler | seo | Netresearch | netresearchserver |
| Lssbot | seo | — | lssbot |
| Lssbot Rocket | seo | — | lssrocketcrawler |
| Ltx71 Crawler | seo | — | ltx71 |
| Luminator Crawler | seo | Luminator | Luminator-robots |
| Macocu Crawler | archiver | — | MaCoCu |
| Mailru Crawler | search_engine | Mail.ru | Mail\.RU_Bot |
| Majestic Crawler | search_engine | Majestic | MJ12bot |
| Mappydata Crawler | seo | — | mappydata |
| Marginalia Crawler | search_engine | — | search\.marginalia\.nu |
| masscan | security | — | masscan |
| Mastodon Crawler | page_preview | Mastodon | Mastodon |
| Mauibot | search_engine | — | MauiBot |
| Megaindex Crawler | search_engine | MegaIndex | MegaIndex |
| Meltwater Crawler | seo | Meltwater | MeltwaterNews |
| Meta Externalads | advertising | Meta | meta-externalads\/ |
| meta-externalfetcher/ | content_fetcher | Meta | meta-externalfetcher\/ |
| Metadatalabs Crawler | seo | — | mlbot |
| Metajob Crawler | seo | — | MetaJobBot |
| Metauri Crawler | seo | — | MetaURI |
| Metricstools Crawler | seo | — | MTRobot |
| Microsoft Preview | search_engine | Microsoft | MicrosoftPreview\/ |
| Microsoft Research Crawler | search_engine | Microsoft | msrbot |
| Mignify Crawler | seo | — | memorybot |
| Mignify Imrbot | seo | — | imrbot |
| Miniflux Feedfetcher | feed_fetcher | Miniflux | Miniflux\/ |
| Mixnode Cache | content_fetcher | — | MixnodeCache\/ |
| mMechanize | scraping_framework | — | [mM]echanize |
| Moat Crawler | advertising | Moat | moatbot |
| Mojeek Crawler | search_engine | Mojeek | MojeekBot\/ |
| Monitorbacklinks Crawler | monitoring | — | MBCrawler |
| Monsido Crawler | seo | Monsido | Monsidobot |
| Moodle Crawler | content_fetcher | Moodle | MoodleBot |
| Moreover Crawler | content_fetcher | Moreover | Moreover |
| Moz Crawler | seo | Moz | dotbot |
| Moz Site Audit | seo | Moz | rogerbot |
| Msn Crawler | search_engine | Microsoft | msnbot |
| Muckrack Crawler | seo | Muck Rack | MuckRack |
| Multiviewbot | seo | Multiview | Multiviewbot |
| Nagios Check Http | tool | Nagios | check_http |
| Naver Crawler | search_engine | Naver | Yeti |
| Naver Crawler Rss | feed_fetcher | Naver | NAVER Blog Rssbot |
| Neeva Crawler | search_engine | Neeva | Neevabot |
| Nerdbynature Crawler | content_fetcher | NerdByNature | NerdByNature\.Bot |
| Nerdybot Crawler | scraping_framework | — | nerdybot |
| Nessus | security | Tenable | Nessus |
| Netcraft Crawler | monitoring | Netcraft | NetcraftSurveyAgent |
| Netestate Crawler | search_engine | NetEstate | netEstate NE Crawler |
| Neticle Crawler | social_media | Neticle | Neticle Crawler |
| Netsystemsresearch Crawler | seo | NetSystemsResearch | NetSystemsResearch |
| Netvibes Crawler | feed_fetcher | Netvibes | Netvibes |
| Newrelic Monitor | monitoring | New Relic | NewRelicbot\/ |
| newsai/ | ai | — | newsai\/ |
| Newsblur Aggregator | feed_fetcher | NewsBlur | NewsBlur |
| Newsharecounts Crawler | social_media | — | newsharecounts |
| Newspaper | scraping_framework | News | newspaper\/ |
| Nextcloud Crawler | content_fetcher | Nextcloud | NextCloud |
| Nicecrawler Archive | archiver | Nicecrawler | Nicecrawler |
| Nict Crawler | archiver | National Institute of Information and Communications Technology | ICC-Crawler |
| Niki Bot | social_media | Niki | niki-bot |
| Nikto | security | SecurityFocus | Nikto |
| Ning Crawler | content_fetcher | Ning | NING\/ |
| Ninjabot | scraping_framework | — | NINJA bot |
| Nixstats Crawler | monitoring | NixStats | NIXStatsbot |
| Nmap | tool | — | Nmap Scripting Engine |
| Nutch | tool | Apache | Nutch |
| Nuzzel Crawler | feed_fetcher | Nuzzel | Nuzzel |
| ObservePoint | monitoring | ObservePoint | ObservePoint |
| Ocarinabot | scraping_framework | — | Ocarinabot |
| OICrawler | search_engine | — | OICrawler |
| Okru Crawler | social_media | Odkl | OdklBot\/ |
| Oncrawl | seo | OnCrawl | OnCrawl\/ |
| Opengraphcheck Crawler | page_preview | — | OpenGraphCheck |
| Openhose Crawler | content_fetcher | — | OpenHoseBot |
| Openindex Crawler | search_engine | — | openindexspider |
| Orange Crawler | seo | Orange | OrangeBot\/ |
| Orange Ftgroup Crawler | content_fetcher | Orange | voilabot |
| Outbrain Link Checker | monitoring | Outbrain | outbrain |
| Outclicks Crawler | advertising | Outclicks | OutclicksBot |
| Page To Rss | feed_fetcher | — | page2rss |
| Pagepeeker Crawler | monitoring | PagePeeker | PagePeeker\/ |
| Pagething Crawler | seo | PageThing | PageThing |
| Paloaltonetworks Crawler | security | Palo Alto Networks | Expanse |
| Panscient Crawler | seo | PanScientific | panscient |
| Paperli Crawler | seo | PaperLi | PaperLiBot |
| Perl Libwww | http_library | — | libwww-perl |
| Perl Pcore | http_library | — | Pcore-HTTP |
| PerplexityUser | search_engine | Perplexity | PerplexityUser |
| Php Curlclass | http_library | — | ^PHP-Curl-Class |
| Php Phpcrawl | http_library | — | phpcrawl |
| Php Simple Scraper | http_library | — | SimpleScraper |
| Php Simplepie | http_library | — | SimplePie |
| Phxbot | scraping_framework | — | PhxBot |
| Picsearch Crawler | search_engine | Picsearch | psbot |
| Pingdom Crawler | monitoring | SolarWinds | [pP]ingdom |
| Pinterest Crawler | social_media | Pinterest(bot)?\/ | |
| Pintrest Crawler | social_media | pinterest\.com\/bot | |
| Pipl Crawler | search_engine | Pipl | PiplBot |
| Playwright | http_library | — | Playwright |
| Pocket Crawler | feed_fetcher | PocketParser | |
| Postman | tool | Postman | PostmanRuntime\/ |
| Postrank Crawler | content_fetcher | Postrank | postrank |
| Prcy Crawler | seo | — | PR-CY\.RU |
| Privacore Crawler | seo | Privacore | findxbot |
| Privacyaware Crawler | seo | — | PrivacyAwareBot |
| Profound Crawler | seo | — | urlappendbot |
| Proximic Crawler | content_fetcher | Proximic | proximic |
| Pulsepoint Crawler | advertising | Pulsepoint | Pulsepoint |
| Puppeteer | http_library | — | Puppeteer |
| Pure Crawler | seo | — | purebot |
| Python Aiohttp | http_library | — | aiohttp |
| Python Bitbot | http_library | — | BitBot |
| Python Httpx | http_library | — | httpx |
| Python Opengraph | http_library | — | python-opengraph |
| Python Requests | http_library | — | python-requests |
| Python Urllib | http_library | — | Python-urllib |
| Quantcast Crawler | advertising | Quantcast | Quantcastbot\/ |
| Qwant Crawler | search_engine | Qwant | Qwantify |
| Rankactive Crawler | seo | Rankactive | RankActiveLinkBot |
| Reddit Crawler | social_media | redditbot | |
| Refind Crawler | feed_fetcher | Refind | Refindbot |
| Retrevo Page Analyzer | seo | Retrevo | RetrevoPageAnalyzer |
| Ridder Crawler | seo | Ridder | RidderBot |
| Rivva Crawler | seo | Rivva | Rivva |
| Rssbot Feedfetcher | feed_fetcher | — | rssbot\/ |
| Rssing Crawler | feed_fetcher | — | RSSingBot |
| Rssmicro Feedfetcher | feed_fetcher | — | Atom Feed Robot |
| Ruby Metainspector | http_library | — | MetaInspector |
| Ryte Crawler | seo | Ryte | RyteBot |
| Scan Interfax Crawler | seo | Interfax | InterfaxScanBot |
| Schmorp Crawler | tool | — | AnyEvent |
| Scoutjet Crawler | seo | ScoutJet | ScoutJet |
| Screamingfrog Crawler | seo | Screaming Frog | Screaming Frog SEO Spider |
| Scribd Crawler | content_fetcher | Scribd | scribdbot |
| Scritch Crawler | content_fetcher | — | Fetch\/ |
| Seekbot Crawler | seo | — | seekbot |
| Seekport Crawler | search_engine | Seekport | Seekport |
| Seewithkids Crawler | content_fetcher | — | seewithkids |
| Selenium | http_library | — | Selenium |
| Semanticaudience Crawler | seo | — | semanticbot |
| Sempitech Crawler | seo | — | sempi\.tech |
| Semrush Crawler | seo | Semrush | S[eE][mM]rushBot |
| Sentry Crawler | monitoring | Sentry | (^| )sentry\/ |
| Sentry Uptime Monitor | monitoring | Sentry | SentryUptimeBot\/1\.0 |
| Senuto Crawler | seo | Senuto | SenutoBot |
| Seobility Crawler | seo | Seobility | SeobilityBot |
| Seokicks Crawler | seo | — | SEOkicks |
| Seolizer Crawler | seo | — | SEOlizer |
| Seoprofiler Crawler | seo | Seoprofiler | spbot |
| Seoscanners Crawler | seo | — | seoscanners |
| Seostar Crawler | seo | — | seostar\.co |
| Seozoom Crawler | seo | — | ZoomBot |
| SERankingBacklinksBot | seo | SERanking | SERankingBacklinksBot |
| Serendeputy Crawler | feed_fetcher | Serendeputy | SerendeputyBot |
| Serpstatbot Crawler | seo | Serpstat | serpstatbot\/ |
| Seznam Crawler | search_engine | Seznam | SeznamBot |
| Similartech Crawler | seo | Similartech | smtbot |
| Simple Crawler | scraping_framework | — | SimpleCrawler |
| Sistrix 007ac9 Crawler | seo | Sistrix | 007ac9 Crawler |
| Sistrix Crawler | seo | Sistrix | (sistrix|SISTRIX) [cC]rawler |
| Sitebot Crawler | seo | — | sitebot |
| Sitebulb | seo | Sitebulb | sitebulb |
| Sitechecker Crawler | seo | Sitechecker | SiteCheckerBotCrawler |
| Siteexplorer Crawler | seo | — | siteexplorer\.info |
| Siteimprove Crawler | seo | Siteimprove | Siteimprove\.com |
| Skype Preview | search_engine | Microsoft | SkypeUriPreview |
| SlaccaleBot | seo | Slaccale | SlaccaleBot |
| Slack Crawler | page_preview | Slack | Slackbot |
| Slack Image Proxy | page_preview | Slack | Slack-ImgProxy |
| Snap Preview | page_preview | Snapchat | Snap URL Preview Service |
| Socialrank Crawler | seo | Socialrank | SocialRankIOBot |
| Softbytelabs Crawler | seo | Softbytelabs | SBL-BOT |
| Sogou Crawler | search_engine | Sogou | Sogou |
| sqlmap | security | — | sqlmap |
| Startme Crawler | feed_fetcher | Startme | startmebot\/ |
| Statuscake Monitor | monitoring | Statuscake | StatusCake |
| Steam Preview | page_preview | Valve | Valve\/Steam |
| Stract Crawler | search_engine | Stract | StractBot |
| Stripe Crawler | http_library | Stripe | Stripebot\/ |
| Stripe Webhook | webhooks | Stripe | Stripe\/ |
| Stuttgart Crawler | seo | Stuttgart | RegionStuttgartBot |
| Summaly Crawler | tool | Summaly | SummalyBot |
| Summify Crawler | social_media | Summify | summify |
| Superfeedr Crawler | feed_fetcher | Superfeedr | Superfeedr |
| Surly Crawler | monitoring | Surly | SurdotlyBot |
| Swimgbot | seo | — | SWIMGBot |
| Synapse Crawler | page_preview | — | Synapse |
| Sysomos Crawler | social_media | Sysomos | Sysomos |
| T3versions Crawler | seo | — | t3versionsBot |
| Taboola Crawler | advertising | Taboola | Taboolabot |
| Tagoo Crawler | seo | — | tagoobot |
| Tangiblee Crawler | seo | Tangiblee | TangibleeBot |
| Telegram Crawler | page_preview | Telegram | TelegramBot |
| Testomato Crawler | monitoring | — | minicrawler |
| Theoldreader Crawler | feed_fetcher | The Old Reader | theoldreader\.com |
| Thinklab Crawler | seo | — | Thinklab |
| Tiger Crawler | seo | — | tigerbot |
| Tiktok Crawler | social_media | ByteDance | TikTokSpider |
| Tineye Crawler | seo | TinEye | TinEye |
| Tiscali Crawler | seo | — | IstellaBot |
| Tombascraper Crawler | scraping_framework | Tomba | TombaPublicWebCrawler |
| Toplist Crawler | seo | — | toplistbot |
| Torus Crawler | scraping_framework | — | ReverseEngineeringBot |
| Toutiao Crawler | search_engine | Toutiao | ToutiaoSpider |
| Traackr Crawler | seo | Traackr | Traackr\.com |
| Tracemyfile Crawler | seo | — | tracemyfile |
| Trendiction Crawler | seo | — | trendictionbot |
| Trendsmap Crawler | archiver | Trendsmap | TrendsmapResolver |
| Trove Crawler | archiver | Trove | Trove |
| Trovit Crawler | seo | Trovit | trovitBot |
| Ttrss Feedfetcher | feed_fetcher | Tiny Tiny RSS | Tiny Tiny RSS |
| Turnitin Crawler | archiver | Turnitin | Turnitin |
| Tweetedtimes Crawler | social_media | — | tweetedtimes |
| Tweetmemebot | social_media | — | TweetmemeBot |
| Twenga Crawler | seo | — | twengabot |
| Twingly Crawler | seo | Twingly | Twingly |
| Twitter Crawler | page_preview | Twitterbot | |
| Twoip Crawler | seo | — | 2ip bot |
| Twoip Crawler Cms | seo | — | 2ip\.ru |
| Twurly Crawler | seo | — | Twurly |
| Ubermetrics Crawler | seo | — | um-LN |
| Ubt Crawler | seo | — | Streamline3Bot\/ |
| Upflow Crawler | seo | — | Upflow |
| Uptime Monitor | monitoring | — | www\.uptime\.com |
| Uptimebot Monitor | monitoring | — | Uptimebot\.org |
| Uptimerobot Monitor | monitoring | UptimeRobot | UptimeRobot |
| Urlclassification Crawler | seo | — | KStandBot |
| Usine Nouvelle Crawler | seo | — | UsineNouvelleCrawler |
| Utexas Crawler | archiver | — | UT-Dorkbot |
| Utorrent Crawler | content_fetcher | — | BTWebClient |
| Vebidoo Crawler | seo | — | vebidoobot |
| Veooz Crawler | seo | — | Veoozbot |
| Vercel Crawler | monitoring | Vercel | Vercelbot |
| Vercel Monitor Preview | monitoring | Vercel | vercel-screenshot |
| Verisign Ips Agent | security | Verisign | ips-agent |
| Viber Crawler | page_preview | Viber | Viber |
| Vigil Crawler | security | — | Vigil\/ |
| Vipnytt Crawler | seo | — | jpg-newsbot |
| Virustotal Crawler | security | VirusTotal | virustotal |
| Vkrobot Crawler | social_media | VK | VKRobot |
| Vkshare Crawler | social_media | VK | vkShare |
| Vuhuv Crawler | scraping_framework | — | vuhuvBot |
| W3c Validator Css | monitoring | W3C | W3C_CSS_Validator |
| W3c Validator Feed | monitoring | W3C | FeedValidator |
| W3c Validator Html | monitoring | W3C | W3C_Validator |
| W3c Validator Html Nu | monitoring | W3C | Validator\.nu |
| W3c Validator I18n | monitoring | W3C | W3C_I18n-Checker |
| W3c Validator Links | monitoring | W3C | W3C-checklink |
| W3c Validator Mobile | monitoring | W3C | W3C-mobileOK |
| W3c Validator Unified | monitoring | W3C | W3C_Unicorn |
| Warebay Crawler | seo | Warebay | wbsearchbot |
| Wayback | archiver | Internet Archive | Wayback |
| Webarchive Crawler | archiver | — | web-archive-net\.com\.bot |
| Webceo Crawler | seo | WebCEO | online-webceo-bot |
| Webcompany Crawler | content_fetcher | — | webcompanycrawler |
| Webdatastats Crawler | seo | — | WebDataStats |
| Webeaver Crawler | seo | — | BehloolBot |
| Webmeup Crawler | seo | Webmeup | BLEXBot\/ |
| Webmon | monitoring | — | webmon |
| Webpagetest Crawler | monitoring | WebPageTest | (^| )PTST\/ |
| webzio | seo | Webzio | webzio |
| Webzio Crawler | search_engine | Webzio | webzio\s |
| Wellknown Crawler | search_engine | — | WellKnownBot |
| Wesee Crawler | search_engine | WeSEE | WeSEE:Search |
| Wget | tool | — | [wW]get |
| Whatsapp Crawler | social_media | ||
| Wocodi Crawler | seo | Wocodi | wocbot |
| Woorank Crawler | seo | Woorank | woobot |
| Woorank Crawler Review | seo | Woorank | woorankreview\/ |
| Wordpress Crawler | content_fetcher | WordPress | WordPress\/ |
| Wordpress Crawler Rss | feed_fetcher | — | feedbot |
| Wordup Crawler | seo | — | WordupInfoSearch |
| Worio Crawler | seo | Worio | woriobot |
| Wotbox Crawler | seo | Wotbox | wotbox |
| WPScan | security | WPScan | WPScan |
| Xenu Crawler | monitoring | — | Xenu Link Sleuth |
| Xovibot Crawler | seo | Xovibot | xovibot |
| Yacy Crawler | search_engine | Yacy | yacybot |
| Yahoo Crawler | search_engine | Yahoo | Slurp |
| Yahoo Crawler Japan | search_engine | Yahoo | Y!J |
| Yahoo Preview | search_engine | Yahoo | Yahoo Link Preview |
| Yamanalab Crawler | archiver | Yamanalab | Sonic |
| Yandex Crawler | search_engine | Yandex | yandex\.com\/bots |
| Yandex Crawler Javascript | search_engine | Yandex | YandexRenderResourcesBot\/ |
| Yanga Crawler | seo | Yanga | yanga |
| Yellowbp Crawler | advertising | Yellowbrand | Yellowbrandprotectionbot |
| Yext Bot | tool | Yext | YextBot |
| YextBot/ | seo | Yext | YextBot\/ |
| Yisou Crawler | scraping_framework | Yisou | YisouSpider |
| Yooz Crawler | seo | Yooz | yoozBot |
| Zabbix Monitor | monitoring | Zabbix | Zabbix |
| Zgrab | tool | Zgrab | zgrab |
| ZmEu | security | ZmEu | ZmEu |
| Zoominfo Crawler | seo | Zoominfo | ZoominfoBot |
| Zum Crawler | seo | Zum | ZumBot |
| Zuperlist Crawler | seo | Zuperlist | ZuperlistBot\/ |