Browse and search all 1650 known bot patterns in the classifier database.
| Name | Category | Organization | Pattern |
|---|---|---|---|
| SiemensWebservicesCrawler | internal_service | Siemens | SiemensWebservicesCrawler |
| AgentDataBotLLM | ai | AgentData | AgentDataBot |
| AgentTimesLLM | ai | The Agent Times | AgentTimes |
| Ai Search BotLLM | ai | Ai Search | AISearchBot |
| Ai2 CrawlerLLM | ai | AI2 | AI2Bot\s |
| Ai2 Crawler DolmaLLM | ai | AI2 | Ai2Bot-Dolma |
| AI2BotLLM | ai | Ai2 | AI2Bot |
| AI2Bot-DeepResearchEvalLLM | ai | Ai2, a non-profit AI research institute | AI2Bot-DeepResearchEval |
| Aihit CrawlerLLM | ai | Aihit | aiHitBot |
| AIWebIndexLLM | ai | Lyrenth | AIWebIndex |
| Amazon Bedrock Agentcore BrowserLLM | ai | Amazon | Amazon-Bedrock-AgentCore-Browser |
| Amazon CrawlerLLM | ai | Amazon | Amazonbot |
| Amazon SearchbotLLM | ai | Amazon | Amzn-SearchBot |
| Amazon UserLLM | ai | Amazon | Amzn-User |
| amazon-kendraLLM | ai | Amazon | amazon-kendra |
| amazon-QBusinessLLM | ai | Amazon | amazon-QBusiness |
| AmazonBuyForMeLLM | ai | Amazon | AmazonBuyForMe |
| AndibotLLM | ai | Andi | Andibot |
| AnomuraLLM | ai | Direqt | Anomura |
| Anomura CrawlerLLM | ai | Anomura | Anomura\/ |
| Anthropic AiLLM | ai | Anthropic | anthropic-ai |
| Anthropic CrawlerLLM | ai | Anthropic | [cC]laude(?:[bB]ot|-[Ww]eb) |
| Anthropic Crawler SearchLLM | ai | Anthropic | Claude-SearchBot |
| Anthropic Crawler UserLLM | ai | Anthropic | Claude-User |
| Apify BotLLM | ai | Apify | ApifyBot |
| ApifyWebsiteContentCrawlerLLM | ai | Apify | ApifyWebsiteContentCrawler |
| Applebot-ExtendedLLM | ai | Apple | Applebot-Extended |
| Aranet-SearchBotLLM | ai | Aranet | Aranet-SearchBot |
| Atlassian BotLLM | ai | Atlassian | atlassian-bot |
| AwarioLLM | ai | Awario | Awario |
| Azure Ai SearchbotLLM | ai | Microsoft | AzureAI-SearchBot |
| bedrockbotLLM | ai | Amazon | bedrockbot |
| bigsur.aiLLM | ai | Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for busi… | bigsur\.ai |
| BixelBotLLM | ai | Bixel | BixelBot |
| Botify CrawlerLLM | ai | Botify | botify |
| BravebotLLM | ai | https://safe.search.brave.com/help/brave-search-crawler | Bravebot |
| BrightbotLLM | ai | Bright | Brightbot |
| Brightbot 1.0LLM | ai | https://brightdata.com/brightbot | Brightbot 1\.0 |
| BuddyBotLLM | ai | BuddyBotLearning | BuddyBot |
| Bytedance CrawlerLLM | ai | ByteDance | Bytespider |
| Ccbot CrawlerLLM | ai | CCBot | CCBot\/ |
| Channel3 BotLLM | ai | Channel3 | Channel3Bot |
| ChatGLM-SpiderLLM | ai | ChatGLM | ChatGLM-Spider |
| ChatGPT AgentLLM | ai | OpenAI | ChatGPT Agent |
| Claude-CodeLLM | ai | Anthropic | Claude-Code |
| Claude-WebLLM | ai | Anthropic | Claude-Web |
| ClaudeBotLLM | ai | Anthropic | ClaudeBot |
| Cloudflare Ai SearchLLM | ai | Cloudflare | Cloudflare-(?:AI-Search|AutoRAG) |
| Cloudflare-AutoRAGLLM | ai | Cloudflare | Cloudflare-AutoRAG |
| CloudflareBrowserRenderingCrawlerLLM | ai | Cloudflare that returns rendered website content for research, monitoring, and AI data workflows | CloudflareBrowserRenderingCrawler |
| CloudVertexBotLLM | ai | CloudVertex | CloudVertexBot |
| CodeLLM | ai | Code | Code |
| Cohere CrawlerLLM | ai | Cohere | cohere-ai |
| cohere-training-data-crawlerLLM | ai | Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products | cohere-training-data-crawler |
| Commoncrawl CrawlerLLM | ai | Commoncrawl | CCBot |
| CragCrawlerLLM | ai | CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services | CragCrawler |
| Crawl4aiLLM | ai | Crawl4AI | crawl4ai |
| Crawl4AILLM | ai | Crawl4AI | Crawl4AI |
| CrawlspaceLLM | ai | Crawlspace | Crawlspace |
| CursorLLM | ai | Cursor | Cursor |
| Datenbank CrawlerLLM | ai | Datenbank | Datenbank Crawler |
| DeepSeekBotLLM | ai | DeepSeek | DeepSeekBot |
| DevinLLM | ai | Devin AI | Devin |
| DiffbotLLM | ai | Diffbot | Diffbot |
| Diffbot CrawlerLLM | ai | Diffbot | Diffbot\/ |
| Diffbot-UserLLM | ai | Diffbot | Diffbot-User |
| DoubaoBotLLM | ai | ByteDance | DoubaoBot |
| Duckassist BotLLM | ai | DuckAssist | DuckAssistBot |
| Echobot BotLLM | ai | Echobox | Echobot Bot |
| EchoboxBotLLM | ai | Echobox | EchoboxBot |
| ERNIEBotLLM | ai | Baidu | ERNIEBot |
| ExaBotLLM | ai | Exa | ExaBot |
| Facebook CrawlerLLM | ai | Meta | Face(book){0,1}[Bb]ot |
| Facebook Share CrawlerLLM | ai | Meta | facebookexternalhit |
| FacebookBotLLM | ai | Meta/Facebook | FacebookBot |
| Factset_spyderbotLLM | ai | Factset | Factset_spyderbot |
| Firecrawl AgentLLM | ai | Firecrawl | FirecrawlAgent |
| Flyriver BotLLM | ai | Flyriverbot | Flyriverbot |
| FriendlycrawlerLLM | ai | Friendlycrawler | FriendlyCrawler |
| GeistHaus-PageFetcherLLM | ai | GeistHaus, a company developing AI systems for therapy and psychological assessment | GeistHaus-PageFetcher |
| Glutenfreepleasure CrawlerLLM | ai | Glutenfreepleasure | Gluten Free Crawler\/ |
| Google AgentLLM | ai | Google-Agent | |
| Google Crawler CloudvertexLLM | ai | Google-CloudVertexBot | |
| Google Gemini Deep ResearchLLM | ai | Gemini-Deep-Research | |
| Google Gemini NotebookLLM | ai | Google-(?:GeminiNotebook|NotebookLM) | |
| Google-ExtendedLLM | ai | Google-Extended | |
| Google-FirebaseLLM | ai | Google-Firebase | |
| Google-Gemini-CLILLM | ai | Google-Gemini-CLI | |
| Google-NotebookLMLLM | ai | Google-NotebookLM | |
| GoogleAgent-MarinerLLM | ai | GoogleAgent-Mariner | |
| GoogleAgent-URLContextLLM | ai | Google that retrieves web content on behalf of Gemini API users | GoogleAgent-URLContext |
| GoogleOther-ImageLLM | ai | GoogleOther-Image | |
| GoogleOther-VideoLLM | ai | GoogleOther-Video | |
| HenkBotLLM | ai | Henk | HenkBot |
| Iask CrawlerLLM | ai | Iask | iaskspider |
| iAskBotLLM | ai | Iask | iAskBot |
| iaskspider/2.0LLM | ai | iAsk | iaskspider/2\.0 |
| Imagesift CrawlerLLM | ai | Imagesift | ImagesiftBot |
| ImagespiderLLM | ai | imageSpider | imageSpider |
| Img2datasetLLM | ai | Img2dataset | img2dataset |
| Infegy CrawlerLLM | ai | Infegy | collection@infegy\.com |
| Integralads CrawlerLLM | ai | Integralads | IAS crawler |
| ISSCyberRiskCrawlerLLM | ai | ISS-Corporate | ISSCyberRiskCrawler |
| Kagi FetcherLLM | ai | Kagi | kagi-fetcher |
| Kangaroo BotLLM | ai | Kangaroo | Kangaroo Bot |
| KeenableBotLLM | ai | Keenable | KeenableBot |
| Kimi BotLLM | ai | Kimi | KimiBot |
| Kimi SearchbotLLM | ai | Kimi | Kimi-SearchBot |
| Kimi UserLLM | ai | Kimi | Kimi-User |
| Kimi-AgentLLM | ai | Moonshot AI that browses websites while carrying out user-directed research and other tasks | Kimi-Agent |
| KlaviyoAIBotLLM | ai | Klaviyo | KlaviyoAIBot |
| KunatoCrawlerLLM | ai | Kunato | KunatoCrawler |
| Laion Huggingface ProcessorLLM | ai | Laion-Huggingface | laion-huggingface-processor |
| LAIONDownloaderLLM | ai | Large-scale Artificial Intelligence Open Network | LAIONDownloader |
| LCCLLM | ai | LCC | LCC |
| Leadcrunch CrawlerLLM | ai | Leadcrunch | CrunchBot |
| LightpandaLLM | ai | Anyone who downloads the Lightpanda client. Possibly being used by a Grok-adjacent organization's botnet. | Lightpanda |
| LinerBotLLM | ai | Liner | LinerBot |
| Linkup BotLLM | ai | Linkup | LinkupBot |
| Manus-UserLLM | ai | Butterfly Effect, a company based in China | Manus-User |
| Mediatoolkit CrawlerLLM | ai | Mediatoolkit | Mediatoolkitbot |
| Meta CrawlerLLM | ai | Meta | [Mm]eta-[Ee]xternal[Aa]gent |
| Meta Crawler UserLLM | ai | Meta | meta-externalfetcher |
| meta-externalagentLLM | ai | Meta | meta-externalagent |
| Meta-ExternalAgentLLM | ai | Meta | Meta-ExternalAgent |
| meta-externalagent/LLM | ai | Meta | meta-externalagent\/ |
| Meta-ExternalFetcherLLM | ai | Meta | Meta-ExternalFetcher |
| Mistral Ai IndexLLM | ai | MistralAI | MistralAI-Index |
| Mistral Ai TrainingLLM | ai | MistralAI | MistralAI-Training |
| Mistral Ai UserLLM | ai | Mistral | MistralAI-User |
| MistralAI-User/1.0LLM | ai | Mistral AI | MistralAI-User/1\.0 |
| Mozilla TabstackLLM | ai | Mozilla | Mozilla-Tabstack |
| MyCentralAIScraperBotLLM | ai | MyCentralAI | MyCentralAIScraperBot |
| NagetBotLLM | ai | Naget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts) | NagetBot |
| netEstate Imprint CrawlerLLM | ai | netEstate | netEstate Imprint Crawler |
| newsaiLLM | ai | newsai | newsai |
| NewsaiLLM | ai | newsai | newsai\/ |
| NotebookLMLLM | ai | NotebookLM | |
| NovaActLLM | ai | NovaAct | NovaAct |
| Ntent CrawlerLLM | ai | Ntent | NTENTbot |
| OAI-AdsBotLLM | ai | OpenAI | OAI-AdsBot |
| Omgili CrawlerLLM | ai | Omgili | omgili |
| omgilibotLLM | ai | Webz.io | omgilibot |
| OpenAILLM | ai | OpenAI | OpenAI |
| Openai CrawlerLLM | ai | OpenAI | GPTBot |
| Openai Crawler SearchLLM | ai | OpenAI | OAI-SearchBot |
| Openai Crawler UserLLM | ai | OpenAI | ChatGPT-User |
| opencodeLLM | ai | OpenCode | opencode |
| OperatorLLM | ai | Operator | Operator |
| PanguBotLLM | ai | the Chinese company Huawei | PanguBot |
| PanscientLLM | ai | Panscient | Panscient |
| panscient.comLLM | ai | Panscient | panscient\.com |
| Perplexity CrawlerLLM | ai | Perplexity | PerplexityBot\/ |
| Perplexity UserLLM | ai | Perplexity | Perplexity-?User |
| Perplexity-UserLLM | ai | Perplexity | Perplexity-User |
| PerplexityBotLLM | ai | Perplexity | PerplexityBot |
| Petalsearch CrawlerLLM | ai | Petalsearch | PetalBot |
| Phind BotLLM | ai | phind | PhindBot |
| Poggio-CitationsLLM | ai | Poggio, a company that provides AI sales enablement tools for creating tailored narratives, business cases, and account plan… | Poggio-Citations |
| Poseidon Research CrawlerLLM | ai | Poseidon Research | Poseidon Research Crawler |
| Primal CrawlerLLM | ai | Primal | Primalbot |
| Python ScrapyLLM | ai | Scrapy | Scrapy |
| qodercliLLM | ai | Qoder that helps developers build, debug, and modify software from the terminal | qodercli |
| QualifiedBotLLM | ai | Qualified | QualifiedBot |
| Querit-SearchBotLLM | ai | Querit that indexes web content for their search API service, which is designed to provide real-time search results for larg… | Querit-SearchBot |
| QueritBotLLM | ai | Querit, a company providing a search API for large language model integration | QueritBot |
| QuillBotLLM | ai | Quillbot | QuillBot |
| quillbot.comLLM | ai | Quillbot | quillbot\.com |
| QwenBotLLM | ai | Alibaba | QwenBot |
| ReflectionbotLLM | ai | Reflection | Reflectionbot |
| Safedns CrawlerLLM | ai | SafeDNS | SafeDNSBot |
| Sbintuitions BotLLM | ai | SBIntuitions | SBIntuitionsBot\/ |
| SBIntuitionsBotLLM | ai | SB Intuitions | SBIntuitionsBot |
| Searchatlas CrawlerLLM | ai | SearchAtlas | SearchAtlas |
| Semanticscholar CrawlerLLM | ai | SemanticScholar | SemanticScholarBot |
| SemrushBot-OCOBLLM | ai | Semrush | SemrushBot-OCOB |
| SemrushBot-SWALLM | ai | Semrush | SemrushBot-SWA |
| Sentione CrawlerLLM | ai | Sentione | SentiBot |
| Shap-UserLLM | ai | Shap | Shap-User |
| ShapbotLLM | ai | Parallel | ShapBot |
| Sidetrade indexer botLLM | ai | Sidetrade | Sidetrade indexer bot |
| SpiderLLM | ai | Spider | Spider |
| Storygize CrawlerLLM | ai | Storygize | StorygizeBot |
| Tavily BotLLM | ai | Tavily | TavilyBot |
| Terra CottaLLM | ai | Terra Cotta | Terra Cotta |
| TerraCottaLLM | ai | Ceramic AI | TerraCotta |
| ThinkbotLLM | ai | Thinkbot | Thinkbot |
| Timpi CrawlerLLM | ai | Timpibot | Timpibot |
| TongyiBotLLM | ai | Alibaba that fetches web content for the Tongyi Qianwen assistant and related Qwen-generated answers | TongyiBot |
| TraeLLM | ai | Trae | Trae |
| TwinAgentLLM | ai | Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa… | TwinAgent |
| UseAILLM | ai | UseAI | UseAI |
| Velen CrawlerLLM | ai | Velen | VelenPublicWebCrawler |
| WARDBotLLM | ai | WEBSPARK | WARDBot |
| Webzio Crawler AiLLM | ai | Webzio | webzio-extended |
| Webzio-ExtendedLLM | ai | Webzio | Webzio-Extended |
| wpbotLLM | ai | QuantumCloud | wpbot |
| WRTNBotLLM | ai | WRTN | WRTNBot |
| YaKLLM | ai | Meltwater | YaK |
| YandexAdditionalLLM | ai | Yandex | YandexAdditional |
| YandexAdditionalBotLLM | ai | Yandex | YandexAdditionalBot |
| YiyanBotLLM | ai | Baidu that fetches web content for the yiyan | YiyanBot |
| You CrawlerLLM | ai | You | YouBot |
| Zanista BotLLM | ai | Zanista | ZanistaBot |
| ;Owler | advertising | Owler | ;Owler |
| 008/ | seo | 008 | 008\/ |
| 7Siters | search_engine | 7Siters | 7Siters |
| A360-Search | advertising | A360 | A360-Search |
| A6corp Crawler | seo | A6corp | A6-Indexer |
| aAcunetix | security | Acunetix | [aA]cunetix |
| Aasa Bot | search_engine | AASA | AASA-Bot |
| Abeval Bot | monitoring | ABEval | ABEvalBot |
| Aboundex Crawler | seo | Aboundex | Aboundex |
| abusexmcofr | security | abusexmcofr | abuse\.xmco\.fr |
| Academicbot Rtu | archiver | AcademicRTU | AcademicBotRTU |
| Acapbot | seo | Acapbot | acapbot |
| Accessible Web Bot | monitoring | Accessible Web | Accessible Web Bot |
| AccessStatus | monitoring | AccessStatus | AccessStatus |
| Acoon Crawler | seo | Acoon | acoonbot |
| Acquia optimize | monitoring | Acquia | Acquia optimize |
| ActiveComply | monitoring | ActiveComply | ActiveComply |
| Adagio Crawler | advertising | Adagio | Adagiobot\/ |
| AdagioBot | seo | Adagio | AdagioBot |
| Adbeat Crawler | advertising | Adbeat | adbeat_bot |
| adbeatcom | advertising | Adbeat | adbeat\.com |
| Addsearch Crawler | search_engine | Addsearch | AddSearchBot |
| Addthis Crawler | content_fetcher | Addthis | AddThis |
| AdkernelTopicCrawler | advertising | Adkernel | AdkernelTopicCrawler |
| Admantx Crawler | advertising | Admantx | ADmantX |
| AdminLabs | seo | AdminLabs | AdminLabs |
| AdsBot-IAB | advertising | IAB | AdsBot-IAB |
| Adscanner Crawler | advertising | Adscanner | adscanner |
| Adstxtcrawler | advertising | Adstxt | AdsTxtCrawler |
| advanced_crawler | seo | advanced | advanced_crawler |
| Advbot Crawler | advertising | Advbot | AdvBot |
| Adventurer | seo | Adventurer | Adventurer |
| Adyen Webhook | webhooks | Adyen | Adyen HttpClient |
| AGAKIDSBOT | seo | AGAKIDS | AGAKIDSBOT |
| AgencyAnalyticsBot | seo | AgencyAnalytics | AgencyAnalyticsBot |
| Ahrefs Crawler | search_engine | Ahrefs | AhrefsBot\/ |
| Ahrefs Site Audit | advertising | Ahrefs | AhrefsSiteAudit\/ |
| AhrefsBotSiteAudit | seo | Ahrefs | Ahrefs(Bot|SiteAudit) |
| airai/scanning | ai | airai | air\.ai\/scanning |
| AkismetBot | content_fetcher | Akismet | AkismetBot |
| Alertsite Monitor | monitoring | AlertSite | AlertSite |
| alexa site audit | seo | Alexa | alexa site audit |
| Alexandria Crawler | search_engine | Alexandria | AlexandriaOrgBot |
| Algolia Crawler | search_engine | Algolia | Algolia Crawler\/ |
| Algolia Crawler | search_engine | Algolia | Algolia Crawler |
| alienfarm | seo | alienfarm | alienfarm |
| ALittle Client | tool | ALittle | ALittle Client |
| AliyunSecBot | security | Aliyun | AliyunSecBot |
| AllAfrica | content_fetcher | AllAfrica | AllAfrica |
| allOrigins | seo | allOrigins | allOrigins |
| Alphaseobot Crawler | seo | Alphaseobot | AlphaBot |
| Amazing-SearchBot | advertising | Amazing | Amazing-SearchBot |
| Amazon Adbot | content_fetcher | Amazon | AmazonAdBot |
| Amazon Alexa Crawler | content_fetcher | Dassault Systèmes | exabot |
| Amazon Cloudfront | content_fetcher | Amazon | Amazon CloudFront |
| Amazon Product Discovery | content_fetcher | Amazon | AmazonProductDiscovery |
| Amazon Route53 Health Check | content_fetcher | Amazon | Amazon Route 53 Health Check Service |
| Amazon Seller Initiated Listing | content_fetcher | Amazon | AmazonSellerInitiatedListing |
| Anchor Browser | tool | Anchor | Anchor Browser |
| Anderspink Crawler | seo | Anderspink | AndersPinkBot |
| Antibot | security | Antibot | antibot |
| AP3A240617008 | advertising | AP3A | AP3A\.240617\.008 |
| Apercite Crawler | seo | Apercite | Apercite |
| Apple Crawler | search_engine | Apple | Applebot |
| Apple Feedfetcher | feed_fetcher | Apple | i[Tt][Mm][Ss] |
| AppSiteAssociation | seo | AppSiteAssociation | AppSiteAssociation |
| Ara Crawler | search_engine | Ara | arabot |
| Aragog/ | seo | Aragog | Aragog\/ |
| Aranea Crawler | archiver | Aranea | Aranea |
| Araturka Crawler | seo | AraTurka | Bot\.AraTurka\.com |
| Archive It | archiver | Archive-It | Archive-It |
| Archivebox | archiver | ArchiveBox | ArchiveBox |
| Archiveorg Archiver | archiver | Archive.org | archive\.org_bot |
| ArenaUnfurlBot | page_preview | ArenaUnfurl | ^ArenaUnfurlBot |
| Arocom Crawler | content_fetcher | Arocom | drupact |
| Arquivo-web-crawler | archiver | Arquivo-web | Arquivo-web-crawler |
| ArquivoBot | archiver | Arquivo | ArquivoBot |
| Artemis Web Reader | feed_fetcher | Artemis | artemis web reader |
| ArtemisBot | scraping_framework | Artemis | ArtemisBot |
| Asana/ | seo | Asana | Asana\/ |
| Ask Crawler | search_engine | Ask | Teoma |
| asnriskscorer | monitoring | asnriskscorer | asnriskscorer |
| Aspiegel Crawler | seo | Aspiegel | AspiegelBot |
| Attracta | seo | Attracta | Attracta |
| AtVowBot | monitoring | AtVow | AtVowBot |
| Audigent Adbot | advertising | Audigent | AudigentAdBot |
| Audisto Crawler | seo | Audisto | Audisto Crawler |
| AudistoBot | seo | Audisto | AudistoBot |
| Authory | content_fetcher | Authory | Authory |
| Autoconfig Test from USTC | archiver | USTC | Autoconfig Test from USTC |
| AutomatonNewsify Feed Fetcher | monitoring | AutomatonNewsify | Automaton|Newsify Feed Fetcher |
| Avira Crawler | search_engine | Avira | SafeSearch microdata crawler |
| Awario Crawler | seo | Awario | AwarioBot |
| Awario Crawler Rss | feed_fetcher | Awario | AwarioRssBot |
| Awario Crawler Smart | seo | Awario | AwarioSmartBot |
| AwarioRendererBot | advertising | Awario | AwarioRendererBot |
| Awesomecrawler | scraping_framework | Awesomecrawler | awesomecrawler |
| Azure App Insights | search_engine | Microsoft | AppInsights |
| B2bbot | seo | B2B | B2B Bot |
| Backlinktest Crawler | seo | Backlinktest | backlinkcrawler |
| Bad Neighborhood Header Detector | security | Bad Neighborhood | Bad Neighborhood Header Detector |
| Baidu Cloud Watch | monitoring | Baidu | Baidu-YunGuanCe |
| Baidu Crawler | search_engine | Baidu | Baiduspider |
| BaiduAdsBot | advertising | Baidu | BaiduAdsBot |
| Bazqux Feedfetcher | feed_fetcher | Bazqux | BazQux |
| BDBot/ | seo | BD | BDBot\/ |
| BDFetch | seo | BD | BDFetch |
| BeeperBot | seo | Beeper | BeeperBot |
| BestChange | advertising | BestChange | BestChange |
| Betabot | scraping_framework | Beta | betaBot |
| Betterstack Monitor | monitoring | Betterstack | Better Stack |
| Betteruptime Monitor | monitoring | BetterUptime | Better Uptime Bot |
| BetterUptimeBot | monitoring | BetterUptime | BetterUptimeBot |
| Bibliotheque Nacional de France Crawler | archiver | Bibliotheque Nacional de France | Bibliotheque Nacional de France Crawler |
| Bidswitch Crawler | advertising | Bidswitch | bidswitchbot |
| Bigdatacorp Crawler | content_fetcher | Bigdatacorp | BDCbot |
| Biglotron | scraping_framework | Biglotron | BIGLOTRON |
| BigUpDataBot | seo | BigUpData | BigUpDataBot |
| BinaryCanary | monitoring | BinaryCanary | BinaryCanary |
| Bing Ads | search_engine | Microsoft | Adidxbot |
| Bing Crawler | search_engine | Microsoft | bingbot |
| Bing Office Store | search_engine | Microsoft | officestorebot |
| Bing Preview | search_engine | Microsoft | BingPreview\/ |
| Binlar | seo | Binlar | binlar |
| Bitbucket-Webhooks | seo | Bitbucket | Bitbucket-Webhooks |
| bitdiscovery | seo | bitdiscovery | bitdiscovery |
| Bitly Crawler | page_preview | Bitly | bitlybot |
| Bitsight Crawler | security | Bitsight | BitSightBot\/ |
| Blackboard Crawler | archiver | Blackboard | Blackboard |
| Blackbox Exporter/ | monitoring | Blackbox | Blackbox Exporter\/ |
| BlackDuck-FD | security | BlackDuck | BlackDuck-FD |
| blekkobot | search_engine | blekkobot | blekkobot |
| Bling ERP | monitoring | Bling | Bling ERP |
| BlingERP | tool | Bling | BlingERP |
| Blockaid | monitoring | Blockaid | Blockaid |
| Bloglines | feed_fetcher | Bloglines | Bloglines |
| Blogmura Crawler | content_fetcher | Blogmura | blogmuraBot |
| Blogtraffic Feedfetcher | feed_fetcher | Blogtraffic | BlogTraffic\/\d\.\d+ Feed-Fetcher |
| Blogtrottr | feed_fetcher | Blogtrottr | Blogtrottr |
| BlogVault | monitoring | BlogVault | BlogVault |
| Blp Bbot | seo | Blp | BLP_bbot |
| Bluesky Preview | page_preview | Bluesky | BlueskyPreviewBot |
| bluesky-domain-status-classifier | seo | bluesky-domain-status-classifier | bluesky-domain-status-classifier |
| Bluesky/ | social_media | Bluesky | Bluesky\/ |
| bluk_lddc_bot | scraping_framework | bluk_lddc | bl\.uk_lddc_bot |
| bluk_ldfc_bot | scraping_framework | bluk_ldfc | bl\.uk_ldfc_bot |
| bnees_bot | scraping_framework | bnees | bne\.es_bot |
| Bnf Crawler | seo | bnf.fr | bnf\.fr_bot |
| BnFBot | archiver | BnF | BnFBot |
| BoardGamePricesBot | seo | BoardGamePrices | BoardGamePricesBot |
| Bombora Crawler | advertising | Bombora | BomboraBot |
| BotPoke | seo | BotPoke | BotPoke |
| Boxcar Crawler | social_media | Boxcar | BoxcarBot |
| br-crawler | search_engine | br | br-crawler |
| Brainobot | ai | Brainobot | brainobot |
| Brandonmedia Crawler | seo | Brandonmedia | BrandONbot |
| Brandverity Crawler | monitoring | Brandverity | BrandVerity |
| Brandwatch | seo | Brandwatch | Brandwatch |
| Brandwatch Crawler | seo | Brandwatch | magpie-crawler |
| BraveBot | search_engine | Brave | BraveBot |
| Brightedge Crawler | seo | Brightedge | BrightEdge Crawler |
| brokenlinkcheckcom | seo | brokenlinkcheck | brokenlinkcheck\.com |
| BrowserBot-Observer | scraping_framework | BrowserBot | BrowserBot-Observer |
| Bublup Crawler | seo | Bublup | BublupBot |
| Buffer Link Preview | page_preview | Buffer | BufferLinkPreviewBot |
| Bugsnag | monitoring | Bugsnag | Bugsnag |
| Builtwith Crawler | seo | BuiltWith | ^BW\/ |
| Bushbaby | seo | Bushbaby | Bushbaby |
| Butterfly | seo | Butterfly | Butterfly |
| Buttondown | feed_fetcher | Buttondown | Buttondown |
| Buzzstream Crawler | seo | Buzzstream | buzzbot |
| BW/ | seo | BW | BW\/ |
| CaliberBot | seo | Caliber | CaliberBot |
| CapitalOneBot | advertising | CapitalOne | CapitalOneBot |
| CapitalOneShopping | seo | CapitalOne | CapitalOneShopping |
| Capsulink Crawler | seo | Capsulink | CapsuleChecker |
| CapterraBot | monitoring | Capterra | CapterraBot |
| Carbon Umbrella Bot | monitoring | carbon-umbrella | carbon-umbrella-bot |
| Careerx Crawler | content_fetcher | Careerx | careerbot |
| Catchpoint | monitoring | Catchpoint | Catchpoint |
| caveman-hunter | monitoring | caveman-hunter | caveman-hunter |
| cClaudebBot | seo | Claude | [cC]laude[bB]ot |
| cCludo | monitoring | Cludo | [cC]ludo |
| CDSCbot | archiver | CDSC | CDSCbot |
| Censys Inspect | monitoring | Censys | CensysInspect\/ |
| CentComBot/ | seo | CentCom | CentComBot\/ |
| Centro Adstxt Crawler | advertising | Centro | Centro Ads\.txt Crawler |
| Centurybot | scraping_framework | Century | centurybot |
| centurybot9 | seo | Century | centuryb\.o\.t9 |
| CERT PL | security | CERT PL | CERT PL |
| CertChief | security | CertChief | CertChief |
| certytags | seo | certytags | certytags |
| Changedetection Crawler | monitoring | Changedetection | changedetection |
| channable | advertising | channable | channable |
| ChargeBeeBot | seo | ChargeBee | ChargeBeeBot |
| Charlotte | seo | Charlotte | Charlotte |
| Chatwork LinkPreview | page_preview | Chatwork | Chatwork LinkPreview |
| CheckHost | monitoring | CheckHost | CheckHost |
| Checkly Monitor | monitoring | Checkly | Checkly |
| Checkmarknetwork Crawler | seo | Checkmarknetwork | CheckMarkNetwork\/ |
| Chirpgotosocial | monitoring | Chirp | Chirp|gotosocial |
| Chlooe Crawler | content_fetcher | Chlooe | bot-pge\.chlooe\.com |
| Chrome Privacy Preserving Prefetch Proxy | seo | Chrome Privacy Preserving Prefetch Proxy | |
| Cincrawdata Crawler | content_fetcher | Cincrawdata | Cincraw |
| CirrusExplorer | seo | Cirrus | CirrusExplorer |
| Cispa Crawler | archiver | CISPA | CISPA Webcrawler |
| Citeseerx Crawler | archiver | Citeseerx | citeseerxbot |
| Citoid | archiver | Citoid | Citoid |
| Clarity Bot | seo | Clarity | ClarityBot\/ |
| Classla Web Crawler | archiver | CLASSLA | CLASSLA-web |
| Clearscopebot | seo | Clearscope | Clearscopebot |
| Clickagy Crawler | seo | Clickagy | Clickagy |
| ClickUpLinkUnfurler | page_preview | ClickUp | ClickUpLinkUnfurler |
| Cliqz Crawler | seo | Cliqz | Cliqzbot\/ |
| Cloudflare Archiver | archiver | Cloudflare | CloudFlare-AlwaysOnline |
| Cloudflare Healthchecks | monitoring | Cloudflare | Cloudflare-Healthchecks |
| Cloudflare Prefetch | seo | Cloudflare | CloudFlare-Prefetch |
| Cloudflare Security Center | monitoring | Cloudflare | https:\/\/developers\.cloudflare\.com\/security-center\/ |
| Cloudflare Ssl Detector | monitoring | Cloudflare | Cloudflare-SSLDetector |
| Cloudflare Traffic Manager | monitoring | Cloudflare | Cloudflare-Traffic-Manager |
| cloudflare-csup | monitoring | Cloudflare | cloudflare-csup |
| Cloudflare-Custom-Error-Page-Crawler | monitoring | Cloudflare | Cloudflare-Custom-Error-Page-Crawler |
| Cloudflare-Custom-Hostname-Verification | seo | Cloudflare | Cloudflare-Custom-Hostname-Verification |
| Cloudflare-Radar-Scanner | security | Cloudflare | Cloudflare-Radar-Scanner |
| Cloudflare-SpeedTest | monitoring | Cloudflare | Cloudflare-SpeedTest |
| Cloudflare-Stream-Hook | seo | Cloudflare | Cloudflare-Stream-Hook |
| Cloudflare-Stream-Webhook | seo | Cloudflare | Cloudflare-Stream-Webhook |
| Cloudflare-Validator | monitoring | Cloudflare | Cloudflare-Validator |
| CloudflareRadarURLScanner | monitoring | Cloudflare | CloudflareRadarURLScanner |
| Cloudsystemnetworks Crawler | content_fetcher | Cloudsystemnetworks | Nimbostratus-Bot |
| Cloudtrellis | seo | Cloudtrellis | Cloudtrellis |
| CMSChecker | seo | CMSChecker | CMSChecker |
| Coccoc Crawler | search_engine | Coccoc | coccoc |
| Cocolyze Crawler | seo | Cocolyze | Cocolyzebot |
| Coda Server Fetcher | http_library | Coda | Coda-Server-Fetcher |
| CodaBot | social_media | Coda | CodaBot |
| Code/1 | tool | Code | Code\/1\. |
| Codewise Crawler | advertising | Codewise | VoluumDSP-content-bot |
| cognitiveSEO Bot | seo | cognitiveSEO | cognitiveSEO Bot |
| Cognitiveseo Crawler | seo | Cognitiveseo | James BOT |
| cognitiveseocom | seo | cognitiveseo | cognitiveseo\.com |
| Collapsify | scraping_framework | Collapsify | Collapsify |
| colly | scraping_framework | colly | colly |
| CommaFeed | feed_fetcher | CommaFeed | CommaFeed |
| Companybook Crawler | content_fetcher | Companybook | Companybook-Crawler |
| Conductor Crawler | seo | Conductor | Caliperbot |
| Content Crawler Spider | scraping_framework | Content Crawler | content crawler spider |
| contentkingapp | seo | contentking | contentkingapp |
| Contextad Crawler | advertising | Contextad | ContextAd Bot |
| ContextualBotsSoutcomesnet | advertising | ContextualBot | ContextualBot[\s\S]*outcomes\.net |
| Contxbot | seo | Contx | contxbot |
| Convera Crawler | seo | Convera | convera |
| Convermax | seo | Convermax | Convermax |
| cookie-maestro | advertising | cookie-maestro | cookie-maestro |
| Cookiebot Crawler | security | Cookiebot | Cookiebot |
| CookieHub Bot | advertising | CookieHub | CookieHub Bot |
| Cookiehub Scan | tool | Cookiehub | CookieHubScan |
| CookieHubVerify | advertising | CookieHub | CookieHubVerify |
| CookieYesbot | advertising | CookieYes | CookieYesbot |
| Corporama matcher | monitoring | Corporama | Corporama matcher |
| Cotoyogi Crawler | seo | ROIS | Cotoyogi |
| Coveobot | search_engine | Coveo | Coveobot |
| Cozi-iCalendar-FeedReader | feed_fetcher | Cozi | Cozi-iCalendar-FeedReader |
| crawler_eb_germany | seo | eb | crawler_eb_germany |
| Crawlson | seo | Crawlson | Crawlson |
| Crazy Egg | monitoring | Crazy Egg | Crazy Egg |
| Creativecommons Crawler | archiver | Creativecommons | CC Metadata Scaper |
| Criteo Crawler | advertising | Criteo | CriteoBot |
| Critical CSS Bot | monitoring | Critical CSS | Critical CSS Bot |
| Criticalcsscom | seo | Criticalcss | Criticalcss\.com |
| cron-joborg | monitoring | cron-job | cron-job\.org |
| Cronless | monitoring | Cronless | Cronless |
| CrunchBot | seo | Crunch | ^CrunchBot |
| Crusty Crawler | seo | crusty | crusty\/ |
| Crystalsemantics Crawler | seo | CrystalSemantics | CrystalSemanticsBot |
| csirtcz | security | csirtcz | csirt\.cz |
| Curebot Crawler | seo | Curebot | [Cc]urebot |
| Curl | tool | curl project | ^curl |
| CurrentsSRSS Reader | feed_fetcher | Currents | Current[\s\S]*RSS Reader |
| CustomAsyncHttpClient | http_library | CustomAsyncHttpClient | CustomAsyncHttpClient |
| Cutbot Crawler | link_checker | Cutbot | Cutbot |
| Cxense Crawler | seo | Cxense | cXensebot |
| CXK_Bot | seo | CXK | CXK_Bot |
| Cybaa Bot | advertising | Cybaa | CybaaBot |
| CyberFindCrawler | search_engine | CyberFind | CyberFindCrawler |
| Cyberpatrol Crawler | security | CyberPatrol | CyberPatrol |
| CyotekWebCopy | archiver | Cyotek | CyotekWebCopy |
| cypexai/scanning | ai | cypexai | cypex\.ai\/scanning |
| Cypress | tool | Cypress | [Cc]ypress |
| Dareboost Crawler | seo | DareBoost | DareBoost |
| Dark Visitor | monitoring | Dark Visitor | Dark Visitor |
| DaspeedBot | seo | Daspeed | DaspeedBot |
| Datadog Agent | monitoring | Datadog | Datadog Agent |
| Datadog Monitor Synthetics | monitoring | Datadog | Datadog(?:\/{0,1}Synthetics| Synthetic) |
| Datadog Synthetic | monitoring | Datadog | Datadog Synthetic |
| DatadogSynthetics | monitoring | Datadog | DatadogSynthetics |
| Datafeedwatch Crawler | content_fetcher | Datafeedwatch | Datafeedwatch |
| Dataforseo Crawler | seo | DataForSeo | DataForSeoBot |
| Datagnion Crawler | seo | Datagnion | datagnionbot |
| Datanyze Crawler | seo | Datanyze | Datanyze |
| Dataprovider Crawler | content_fetcher | Dataprovider | Dataprovider\.com |
| Datenbutler Crawler | seo | mindUp | mindUpBot |
| Daum Crawler | search_engine | Daum | Daum\/ |
| daumoa | search_engine | daumoa | daumoa |
| Dazzle Bluesky Bot | page_preview | Dazzle BlueSky | Dazzle BlueSky Bot |
| dbot | scraping_framework | dbot | dbot\) |
| dcrawl | seo | dcrawl | ^dcrawl |
| Dcrawl | tool | Dcrawl | dcrawl |
| dDirBbuster | scraping_framework | DirBuster | [dD]ir[Bb]uster |
| Dead Link Checker | seo | Dead Link Checker | Dead Link Checker |
| Deadlinkchecker | monitoring | Deadlinkchecker | deadlinkchecker |
| DeepCrawl | seo | DeepCrawl | DeepCrawl |
| Deepnoc Crawler | seo | Deepnoc | deepnoc |
| Deskyo Bot | seo | Deskyo | Deskyobot |
| Detectify | security | Detectify | Detectify |
| Determ | monitoring | Determ | Determ |
| Deusu Crawler | seo | DeuSu | DeuSu\/ |
| DF Bot | seo | DF | DF Bot |
| Digg Crawler | social_media | Digg | Digg Deeper |
| DigiCert DCV | security | DigiCert | DigiCert DCV |
| Digincore Crawler | seo | Digincore | Digincore bot |
| Digitaldragon Crawler | scraping_framework | CrawlyProject | CrawlyProjectCrawler |
| DingTalkBot-LinkService | page_preview | DingTalk | DingTalkBot-LinkService |
| Discord Crawler | page_preview | Discord | Discordbot |
| Discourse Onebox | page_preview | Discourse | Discourse Forum Onebox |
| Discoveryengine Crawler | seo | DiscoveryEngine | discobot |
| Disqus Crawler | monitoring | Disqus | Disqus |
| Dlc/ | seo | Dlc | Dlc\/ |
| dlvrit | content_fetcher | dlvrit | dlvr\.it |
| Dmbot | seo | Dm | Dmbot |
| DMBrowser | monitoring | DMBrowser | DMBrowser |
| DnBCrawler | seo | DnB | DnBCrawler |
| DNSScanner | security | DNSScanner | DNSScanner |
| Dnyz Crawler | seo | Dnyz | DnyzBot |
| Do Not Track Verifier | security | Do Not Track Verifier | Do Not Track Verifier |
| Domaincrawler Crawler | seo | Domaincrawler | domaincrawler |
| Domainreanimator Crawler | seo | Domainreanimator | Domain Re-Animator Bot |
| Domainsbot Crawler | seo | Domainsbot | Pandalytics |
| Domainsproject Crawler | archiver | Domainsproject | Domains Project\/ |
| Domainstats Crawler | seo | Domainstats | DomainStatsBot |
| Domaintools Crawler | seo | Domaintools | SurveyBot |
| Domcop Bot | seo | DomCop | DomCopBot |
| DominicBot | monitoring | Dominic | DominicBot |
| Dotcom-Monitor | monitoring | Dotcom-Monitor | Dotcom-Monitor |
| Dotnetdotcom Crawler | content_fetcher | Dotnetdotcom | ezooms |
| Dow Jones Searchbot | monitoring | Dow Jones | Dow Jones Searchbot |
| Download Ninja | scraping_framework | Download Ninja | Download Ninja |
| downnotifiercom | monitoring | downnotifiercom | downnotifier\.com |
| DowntimeDetector/ | monitoring | DowntimeDetector | DowntimeDetector\/ |
| Dragonmetrics Crawler | seo | Dragonmetrics | Dragonbot |
| DrataAutopilot | monitoring | Drata | DrataAutopilot |
| Dratabot | seo | Dratabot | Dratabot |
| DreamHost Data Team | monitoring | DreamHost | DreamHost Data Team |
| Driftnet Crawler | monitoring | Driftnet | InternetMeasurement |
| Drupalbot | monitoring | Drupal | Drupalbot |
| ds9 | advertising | ds9 | ds9 |
| Dubbot Crawler | monitoring | Dubbot | Dubbotbot |
| Duckduckgo Crawler | search_engine | DuckDuckGo | DuckDuckBot |
| Duckduckgo Crawler Favicons | page_preview | DuckDuckGo | DuckDuckGo-Favicons-Bot |
| Duedil Crawler | seo | Duedil | electricmonk |
| DVbot | advertising | DV | DVbot |
| Dynatrace Monitor | monitoring | Dynatrace | RuxitSynthetic |
| EasyBib AutoCite | archiver | EasyBib | EasyBib AutoCite |
| easybill-ImportManager | seo | easybill | easybill-ImportManager |
| Easycron Monitor | monitoring | EasyCron | EasyCron\/ |
| easyDNS Monitoring | monitoring | easyDNS | easyDNS Monitoring |
| Ec2linkfinder | link_checker | Ec2linkfinder | ec2linkfinder |
| EchoboxBot/ | seo | Echobox | EchoboxBot\/ |
| EcoVadisSustainabilityBot | advertising | EcoVadis | EcoVadisSustainabilityBot |
| edansbot | seo | edans | edansbot |
| Edgewatch Monitor | monitoring | EdgeWatch | EdgeWatch |
| Edister Crawler | scraping_framework | Edister | edisterbot |
| Elisabot | scraping_framework | Elisabot | elisabot |
| elmahio Uptime Monitoring | monitoring | elmahio | elmah\.io Uptime Monitoring |
| elmahio-uptimebot | monitoring | elmahio | elmahio-uptimebot |
| EmailWolf | scraping_framework | EmailWolf | EmailWolf |
| Embedly Crawler | page_preview | Embedly | Embedly |
| eMoney Advisor | monitoring | eMoney | eMoney Advisor |
| eMoneyBot | seo | eMoney | eMoneyBot |
| Entireweb Crawler | search_engine | Entireweb | speedy |
| Epfl Crawler | archiver | Epfl | A Patent Crawler |
| Epictions Crawler | seo | Epictions | epicbot |
| EpivozCrawler | seo | Epivoz | EpivozCrawler |
| Erepublik Tools Crawler | seo | eRepubliktools | eRepublik\.tools |
| Eright Crawler | seo | Eright | eright |
| Europarchive Crawler | archiver | Europarchive | europarchive\.org |
| Eventures Crawler | content_fetcher | Eventures | e\.ventures Investment Crawler |
| Eventures Crawler Batch | content_fetcher | Eventures | evc-batch |
| EvernoteRichLinkBot | page_preview | Evernote | EvernoteRichLinkBot |
| everyfeed-spider | feed_fetcher | everyfeed | everyfeed-spider |
| Everyonesocial Crawler | social_media | Everyonesocial | EveryoneSocialBot |
| EvoUptimeBot | monitoring | EvoUptime | EvoUptimeBot |
| Exa Searchbot | search_engine | Exa | ExaSearchBot |
| Exensa Crawler | seo | Exensa | Bark[rR]owler |
| Exodus Movement Crawler | seo | ExodusMovement | ExodusMovement |
| Experibot Crawler | seo | Experibot | Experibot |
| ExteContextCrawl | seo | ExteContext | ExteContextCrawl |
| Extlinks Crawler | seo | Extlinks | ExtLinksBot |
| Eyeota Crawler | seo | Eyeota | Eyeotabot |
| Ezgif | seo | Ezgif | Ezgif |
| Ezid Crawler | archiver | Ezid | EZID |
| EzLynx | advertising | EzLynx | EzLynx |
| Ezoic Bot | advertising | Ezoic | EzoicBot |
| Facebook Catalog | social_media | Meta | facebookcatalog\/ |
| factset_spyderbot | seo | FactSet | factset_spyderbot |
| Fast Crawler | search_engine | FAST | FAST-WebCrawler |
| Fast Crawler Enterprise | search_engine | FAST | FAST Enterprise Crawler |
| FastDAST | monitoring | FastDAST | FastDAST |
| Fastmail Preview | page_preview | Fastmail | FastmailUA |
| FDL Stats Bot | seo | FDL | FDL Stats Bot |
| Fedica Bot | seo | Fedicabot | Fedicabot |
| FediDB | monitoring | FediDB | FediDB |
| FediIndex | search_engine | FediIndex | FediIndex |
| FediList Agent | monitoring | FediList | FediList Agent |
| Fedineko | archiver | Fedineko | Fedineko |
| Fedistats Crawler | monitoring | fedistats | fedistatsCrawler |
| Fedoraplanet Crawler | content_fetcher | Fedoraplanet | fedoraplanet |
| Fedreporter Bot | seo | FedReporter | FedReporterDataBot |
| FedReporter Bot for FFIEC | monitoring | FedReporter | FedReporter Bot for FFIEC |
| Feed Image Audit | seo | Feed Image Audit | Feed Image Audit |
| Feedafever Crawler | feed_fetcher | Feedafever | Fever |
| Feedbin Crawler | feed_fetcher | Feedbin | Feedbin |
| Feedburner | feed_fetcher | FeedBurner | FeedBurner |
| Feeder / | feed_fetcher | Feeder | Feeder \/ |
| Feeder Co | feed_fetcher | feederco | feeder\.co |
| FeedFlow | feed_fetcher | FeedFlow | FeedFlow |
| Feedly Feedfetcher | feed_fetcher | Feedly | Feedly |
| Feedpresso Content Index Bot | feed_fetcher | Feedpresso | Feedpresso Content Index Bot |
| Feedsearch Bot | feed_fetcher | Feedsearch | Feedsearch Bot |
| Feedsearch-Crawler | feed_fetcher | Feedsearch | Feedsearch-Crawler |
| Feedspot Feedfetcher | feed_fetcher | Feedspot | Feedspot |
| Feedwind | feed_fetcher | Feedwind | Feedwind |
| Femtosearch Crawler | search_engine | Femtosearch | FemtosearchBot |
| fidget-spinner-bot | seo | fidget-spinner | fidget-spinner-bot |
| Findfiles Crawler | search_engine | FindFilesnet | FindFiles\.net |
| Findthatfile Crawler | content_fetcher | Findthatfile | findthatfile |
| fiperbot | search_engine | fiper | fiperbot |
| FirmoGraph | seo | FirmoGraph | FirmoGraph |
| Flamingosearch Crawler | search_engine | Flamingosearch | Flamingo_SearchEngine |
| Fleebs Bot | search_engine | Fleebs | FleebsBot |
| Flipboard Proxy | monitoring | FlipboardProxy | |
| FlipboardRSS | feed_fetcher | FlipboardRSS | |
| Fluffy | scraping_framework | Fluffy | fluffy |
| Fluid | feed_fetcher | Fluid | Fluid |
| Foregenix | seo | Foregenix | Foregenix |
| Fr Crawler | seo | Fr | fr-crawler |
| Freespoke Crawler | search_engine | Freespoke | Freespoke\/ |
| Freewebmonitoring Monitor | monitoring | Freewebmonitoring | FreeWebMonitoring SiteChecker |
| Freshbot | monitoring | Freshbot | Freshbot |
| Freshrss Feedfetcher | feed_fetcher | Freshrss | FreshRSS |
| Freshworks Monitor | monitoring | Freshworks | FreshpingBot |
| Friendly testing bot | seo | Friendly | Friendly testing bot |
| friendly-spider | seo | friendly | friendly-spider |
| FriendlyCrawler/ | seo | FriendlyCrawler | FriendlyCrawler\/ |
| Fuelbot | advertising | Fuelbot | fuelbot |
| FullStoryBot/ | seo | FullStory | FullStoryBot\/ |
| Funnelback | search_engine | Funnelback | Funnelback |
| FuseonBot/ | seo | Fuseon | FuseonBot\/ |
| FyndSearchEngine-Crawler | search_engine | FyndSearchEngine | FyndSearchEngine-Crawler |
| FyndSearchEngine-ReCrawler | search_engine | FyndSearchEngine | FyndSearchEngine-ReCrawler |
| Fyrebot | ai | Fyrebot | Fyrebot |
| G00g1e Crawler | search_engine | g00g1e\.net | |
| G2reader Crawler | feed_fetcher | G2reader | g2reader-bot |
| G2webservices Crawler | content_fetcher | G2webservices | G2 Web Services |
| Gabanzabot/ | search_engine | Gabanza | Gabanzabot\/ |
| Gaisbot | archiver | Gais | Gaisbot |
| Garlik Crawler | seo | Garlik | GarlikCrawler |
| gdnpluscom | seo | gdnplus | gdnplus\.com |
| Geedo Crawler | search_engine | Geedo | GeedoBot |
| Geedo Crawler Products | search_engine | Geedo | GeedoProductSearch |
| Geedo Crawler Shop | search_engine | GeedoShop | GeedoShopProductFinder |
| Genieo Crawler | advertising | Genieo | Genieo |
| GenomeCrawlerd | monitoring | Genome | GenomeCrawlerd |
| getthitcom | seo | getthit | getthit\.com |
| GG PeekBot | page_preview | GG | GG PeekBot |
| Ghost Inspector | tool | Ghost Inspector | Ghost Inspector |
| GIFTEDVISITOR SCAN | seo | GIFTEDVISITOR | GIFTEDVISITOR SCAN |
| Gigablast Crawler | search_engine | Gigablast | Gigabot |
| Gigablast Crawler Oss | tool | Gigablast | Gigablast |
| Ginger Crawler | seo | Ginger | GingerCrawler |
| Github Camo | page_preview | GitHub | github-camo |
| GlobalWebSearch | search_engine | GlobalWebSearch | GlobalWebSearch |
| Gnam Gnam Spider | scraping_framework | Gnam Gnam | gnam gnam spider |
| Gnowit Crawler | content_fetcher | Gnowit | GnowitNewsbot |
| Go Http | http_library | Go | Go-http-client |
| Golfe/ | search_engine | Golfe | Golfe\/ |
| Goo Crawler | search_engine | Goo | ichiro |
| Goodreads | content_fetcher | Goodreads | Goodreads |
| Goodzer | seo | Goodzer | Goodzer |
| Google Ads Conversions | search_engine | Google-Ads-Conversions | |
| Google Adsbot | search_engine | AdsBot-Google([^-]|$) | |
| Google Adsbot Mobile | search_engine | AdsBot-Google-Mobile | |
| Google Adsense | search_engine | Mediapartners-Google | |
| Google Adsense Googlebot | search_engine | Mediapartners \(Googlebot\) | |
| Google Adwords | advertising | Google-Adwords-Instant | |
| Google Appengine | search_engine | AppEngine-Google | |
| Google Apps Script | seo | Google-Apps-Script | |
| Google Association Service | search_engine | GoogleAssociationService | |
| Google Certificates Bridge | search_engine | Google-Certificates-Bridge | |
| Google Crawler | search_engine | Googlebot\/ | |
| Google Crawler Image | search_engine | Googlebot-Image | |
| Google Crawler Mobile | search_engine | Googlebot-Mobile | |
| Google Crawler News | search_engine | Googlebot-News | |
| Google Crawler Other | search_engine | GoogleOther | |
| Google Crawler Safety | search_engine | Google-Safety | |
| Google Crawler Store | search_engine | Storebot-Google | |
| Google Crawler Video | search_engine | Googlebot-Video | |
| Google Cws | search_engine | Google-CWS | |
| Google Favicon | search_engine | Google Favicon | |
| Google Feedfetcher | search_engine | Feedfetcher-Google | |
| Google Image Proxy | search_engine | GoogleImageProxy | |
| Google Inspection Tool | search_engine | Google-InspectionTool | |
| Google Lighthouse | search_engine | Chrome-Lighthouse | |
| Google Messages | search_engine | GoogleMessages | |
| Google Physical Web | search_engine | Google-PhysicalWeb | |
| Google Pinpoint | search_engine | Google-Pinpoint | |
| Google Preview | search_engine | Google Web Preview | |
| Google Producer | search_engine | GoogleProducer | |
| Google Push Notifications | search_engine | APIs-Google | |
| Google Read Aloud | search_engine | Google-Read-Aloud | |
| Google Site Verification | search_engine | Google-Site-Verification | |
| Google Stackdriver Monitoring | search_engine | GoogleStackdriverMonitoring | |
| Google Structured Data Testing Tool | search_engine | Google-Structured-Data-Testing-Tool | |
| Google Trust Services | monitoring | Google Trust Services | |
| Google Web Snippet | search_engine | developers\.google\.com\/\+\/web\/snippet | |
| Google Xrawler | search_engine | google-xrawler | |
| Google-AdWords-Express | advertising | Google-AdWords-Express | |
| Google-Area120 | seo | Google-Area120 | |
| Google-Trust-Services/ | content_fetcher | Google-Trust-Services\/ | |
| GoogleAssociationService | seo | GoogleAssociationService$ | |
| GoogleAssociationService/ | seo | GoogleAssociationService\/ | |
| Googlebot-IA/ | search_engine | Googlebot-IA\/ | |
| GoogleDocs | seo | GoogleDocs | |
| Goparser Bot | tool | GoParser | GoParserBot |
| GoPay | seo | GoPay | GoPay |
| Got | tool | Sindresorhus | sindresorhus\/got |
| GotSiteMonitorcom | monitoring | GotSiteMonitor | GotSiteMonitor\.com |
| Gowiki Crawler | http_library | Gowikibot | Gowikibot |
| Grafana Synthetic Monitoring Agent | monitoring | Grafana | synthetic-monitoring-agent |
| Grammarly/ | seo | Grammarly | Grammarly\/ |
| Grapeshot Crawler | seo | Grapeshot | GrapeshotCrawler |
| gregcrawler | seo | greg | gregcrawler |
| Greppr Web Crawler | search_engine | Greppr | Greppr Web Crawler |
| Grob Crawler | seo | Grob | Grobbot |
| GroovinaAdsbot/ | advertising | Groovina | GroovinaAdsbot\/ |
| Grouphigh Crawler | seo | GroupHigh | GroupHigh\/ |
| Groupme Crawler | social_media | GroupMe | GroupMeBot |
| Grover/ | seo | Grover | Grover\/ |
| Grub | search_engine | Grub | grub\.org |
| gsa-crawler | search_engine | gsa | gsa-crawler |
| Gslfbot | seo | Gslfbot | gslfbot |
| Gtmetrix Monitor | monitoring | GTmetrix | GTmetrix |
| GuestpostsBot/ | seo | Guestposts | GuestpostsBot\/ |
| Gulper Web Bot | archiver | Gulper | Gulper Web Bot |
| Gwene Crawler | content_fetcher | Gwene | Gwene |
| HaloBot | monitoring | Halo | HaloBot |
| Hanalei Bot | monitoring | Hanalei | HanaleiBot |
| Haosou Crawler | search_engine | 360 | 360Spider |
| HappyWing | seo | HappyWing | HappyWing |
| Hardenize | security | Hardenize | Hardenize |
| harsilbot/ | seo | harsil | harsilbot\/ |
| Hatena Crawler | content_fetcher | Hatena | Hatena |
| HawaiiBot | seo | Hawaii | HawaiiBot |
| Hcard Validator | seo | hCardValidator | hCardValidator |
| Headless Chrome | tool | Chrome | HeadlessChrome |
| Headline Crawler | content_fetcher | ev-crawler | ev-crawler\/ |
| HeadOnlyScraper | monitoring | HeadOnlyScraper | HeadOnlyScraper |
| Hello World | seo | Hello World | Hello World |
| HelloworkJobPostingBot/ | seo | Hellowork | HelloworkJobPostingBot\/ |
| Hetrixtools Monitor | monitoring | HetrixTools | HetrixTools |
| hgfAlphaXCrawl/ | archiver | hgfAlphaX | hgfAlphaXCrawl\/ |
| HIFIBot/ | seo | HIFI | HIFIBot\/ |
| Hlidam Robot | seo | Hlidamto | Hlidam\.to robot |
| Honeybadger Bot | monitoring | Honeybadger | HoneybadgerBot |
| Honeybadger Uptime Check | monitoring | Honeybadger | Honeybadger Uptime Check |
| HostTracker/ | monitoring | HostTracker | HostTracker\/ |
| Hotjar | seo | Hotjar | Hotjar |
| Hoyer Crawler | social_media | Friendica | Friendica |
| hstspreload-bot | security | hstspreload | hstspreload-bot |
| Http Get | http_library | http_get | http_get |
| Httpie | tool | HTTPie | ^HTTPie\/ |
| Httrack | archiver | HTTrack | HTTrack |
| Hubspot Crawler | monitoring | HubSpot | HubSpot |
| Huckabot/ | seo | Huckabot | Huckabot\/ |
| Hydrozen Monitor | monitoring | Hydrozen.io | Hydrozen\.io\/ |
| Hydrozenio | monitoring | Hydrozen.io | Hydrozen\.io |
| Hype Machine/ | seo | Hype Machine | Hype Machine\/ |
| Hypefactors Crawler | seo | Buck | Buck\/ |
| Hypestat Crawler | seo | Hypestat | hypestat |
| Hyscore Crawler | monitoring | hyscore.io | hyscore\.io |
| Ia Archiver | archiver | Internet Archive | ia_archiver |
| iAskBot/ | search_engine | iAsk | iAskBot\/ |
| IBM Crawler | seo | IBM | IBM Crawler |
| Ibou Bot | search_engine | Ibou | IbouBot |
| Ideasandcode Crawler | seo | Ideasandcode | ICBot\/ |
| IDG/EU | seo | IDG | IDG\/EU |
| Iframely Preview | page_preview | Iframely | Iframely |
| IFTTT/ | seo | IFTTT | IFTTT\/ |
| ImageFetcher/ | seo | ImageFetcher | ImageFetcher\/ |
| ImageMind | content_fetcher | ImageMind | ImageMind |
| Imessage Preview | search_engine | Facebot | Facebot |
| imgproxy/ | seo | imgproxy | imgproxy\/ |
| Impactcom Agent | advertising | Impactcom | Impact\.com Agent |
| impendoom-bot/ | seo | impendoom | impendoom-bot\/ |
| Indeed Crawler | content_fetcher | Indeed | IndeedBot |
| IndeedJobBot | search_engine | Indeed | IndeedJobBot |
| Indy Library | seo | Indy Library | Indy Library |
| Inetdex Crawler | seo | INETDEX-BOT | INETDEX-BOT |
| Infoo Crawler | seo | Infoo | infoobot |
| Innguma Fetcher | seo | Innguma | Innguma\/ |
| Innologica | seo | Innologica | Innologica |
| Inoreader Aggregator | feed_fetcher | Inoreader | inoreader |
| Instapaper | seo | Instapaper | Instapaper\/ |
| Integromat/ | seo | Integromat | Integromat\/ |
| Integromedb Crawler | seo | Integromedb | integromedb |
| Intelium Crawler | seo | Intelium | intelium_bot |
| intelxio_bot | search_engine | intelxio | intelx\.io_bot |
| Internetarchive Crawler Oss | archiver | Internetarchive | heritrix |
| internetVista monitor | monitoring | internetVista | internetVista monitor |
| Ionos Crawler | content_fetcher | Ionos | IonCrawl |
| Ip Web Crawler | scraping_framework | Ip Web | ip-web-crawler\.com |
| Ipip Crawler | tool | Ipip | HTTP Banner Detection |
| Irc Archivebot | archiver | Irc Archive | ArchiveBot |
| IRLbot | archiver | IRL | IRLbot |
| Irokezcz monitoring | monitoring | Irokezcz | Irokez\.cz monitoring |
| IsDownBot/ | monitoring | IsDown | IsDownBot\/ |
| Iskanie Crawler | seo | Iskanie | iskanie |
| Isonsearch Crawler | search_engine | Isonsearch | IsonSearchBot\/ |
| IsonSearchBot | search_engine | IsonSearch | IsonSearchBot |
| Iss Crawler | seo | Iss | filterdb\.iss\.net\/crawler |
| ISSCyberRiskCrawler/ | security | ISS | ISSCyberRiskCrawler\/ |
| It2media Crawler | seo | It2media | it2media-domain-crawler |
| Itinfluentials Crawler | seo | Itinfluentials | FindITAnswersbot |
| iubenda-radar/ | seo | iubenda | iubenda-radar\/ |
| Jamiembrown Crawler | scraping_framework | Jamiembrown | Jamie's Spider |
| Java Apache Httpclient | http_library | Java Apache | ^Apache-HttpClient |
| Java Asynchttpclient | http_library | Java Async | AHC\/ |
| Java Crawler4j | http_library | Java Crawler4j | crawler4j |
| Java Httpunit | http_library | Java Httpunit | httpunit |
| Java Jersey | http_library | Java Jersey | HttpUrlConnection |
| Java Jetty | http_library | Java Jetty | Jetty |
| Java Okhttp | http_library | Java Okhttp | okhttp |
| Java Snacktory | http_library | Java Snacktory | Snacktory |
| Javascript Axios | http_library | Javascript Axios | axios |
| Javascript Node Fetch | http_library | Javascript Node | node-fetch |
| Javascript Phantom | http_library | Javascript Phantom | PhantomJS |
| jetmon/ | monitoring | jetmon | jetmon\/ |
| Jetpack Monitor | monitoring | Jetpack | jetmon |
| Jetslide Crawler | seo | Jetslide | Jetslide |
| Jobboerse Crawler | seo | Jobboerse | JobboerseBot |
| Jobswithgpt Bot | search_engine | JobsWithGPT | jobswithgptcom-bot |
| Jooble Crawler | seo | Jooble | Jooblebot |
| Jumio | seo | Jumio | Jumio |
| Jusprog Crawler | seo | Jusprog | Jugendschutzprogramm-Crawler |
| Jyxo Crawler | seo | Jyxo | jyxobot |
| K7computing Crawler | seo | K7computing | K7MLWCBot |
| Kagi Crawler | search_engine | Kagi | Kagibot |
| Kagibot/ | search_engine | Kagibo | Kagibot\/ |
| kaikkiorg-digital-archive | archiver | kaikki.org | kaikki\.org-digital-archive |
| KangarooBot/ | seo | Kangaroo | KangarooBot\/ |
| KargoBot-Artemis | advertising | Kargo | KargoBot-Artemis |
| kazbtbot/ | seo | kazbt | kazbtbot\/ |
| kbdk_bot | content_fetcher | kb.dk | kb\.dk_bot |
| Kemvi Crawler | seo | Kemvi | Kemvibot |
| KendraBot | search_engine | Amazon | KendraBot |
| Keybase Bot | page_preview | Keybase | KeybaseBot |
| keycdn-tools/ | monitoring | Keycdn | keycdn-tools\/ |
| Keydropio | tool | Keydrop | Keydrop\.io |
| keys-so-bot | seo | keys-so | keys-so-bot |
| kinsta-bot | seo | Kinsta | kinsta-bot |
| Klaviyo/ | seo | Klaviyo | Klaviyo\/ |
| Known Agent | monitoring | Known | Known Agent |
| Komodia Crawler | seo | Komodia | KomodiaBot |
| Konturbot/ | seo | Kontur | Konturbot\/ |
| Kosmio Crawler | seo | Kosmio | KosmioBot |
| KrawlerBot | scraping_framework | Krawler | KrawlerBot |
| Kukeieu-Bot/ | seo | Kukei | Kukei\.eu-Bot\/ |
| Kuma Monitor | monitoring | Uptime-Kuma | Uptime-Kuma |
| L9explore | tool | L9explore | l9explore |
| l9scan | security | l9scan | l9scan |
| LAC_IAHarvester | archiver | LAC_IA | LAC_IAHarvester |
| Landaumedia Crawler | seo | Landaumedia | Landau-Media-Spider |
| larbin | scraping_framework | larbin | larbin |
| Laserlike Crawler | seo | Laserlike | Laserlikebot |
| LastModBot/ | seo | LastMod | LastModBot\/ |
| Law Unimi Crawler | archiver | Law Unimi | BUbiNG |
| Lb Spider | scraping_framework | Lb | lb-spider |
| LegalMonster | seo | LegalMonster | LegalMonster |
| Leiki Crawler | seo | Leiki | Leikibot |
| Leipzig Findlinks | archiver | Leipzig | findlink |
| Leipzig Lcc | archiver | Leipzig | ^LCC |
| Lemmy Crawler | social_media | Lemmy | Lemmy |
| Let's Encrypt | security | Let's Encrypt | Let's Encrypt |
| Level9SearchBot/ | search_engine | Level9 | Level9SearchBot\/ |
| Library Of Congress Web Archiving | archiver | Library Of Congress | Library Of Congress Web Archiving |
| Lightspeedsystems Crawler | seo | Lightspeedsystems | LightspeedSystemsCrawler |
| Line Crawler | seo | Line | Linespider |
| LinerBot/ | social_media | Liner | LinerBot\/ |
| Linguee Crawler | search_engine | Linguee | Linguee Bot |
| LinkAce/ | seo | LinkAce | LinkAce\/ |
| Linkapedia Crawler | seo | Linkapedia | linkapediabot |
| Linkarchiver | archiver | Linkarchiver | LinkArchiver |
| Linkchecker Pro | tool | LinkChecker | LinkCheckerBot |
| Linkdex Crawler | seo | Linkdex | linkdex |
| Linkedin Crawler | social_media | LinkedInBot | |
| Linkfluence Crawler | seo | Linkfluence | YaK\/ |
| Linkis Crawler | seo | Linkis | LinkisBot |
| linko | monitoring | linko | linko |
| LinkpadBot | monitoring | Linkpad | LinkpadBot |
| linkReader/ | feed_fetcher | linkReader | linkReader\/ |
| Linksindexer Bot | seo | LinksIndexer | LinksIndexerBot\/ |
| LinkTiger | seo | LinkTiger | LinkTiger |
| LinkWalker/ | seo | LinkWalker | LinkWalker\/ |
| Lipperhey Crawler | seo | Lipperhey | lipperhey |
| Listgem Bot | page_preview | ListGem | ListGemBot |
| Livelap Crawler | seo | Livelap | Livelap[bB]ot |
| LMArenaUnfurlBot | page_preview | LMArenaUnfurl | LMArenaUnfurlBot |
| locgov/programs/web-archiving | archiver | locgov | loc\.gov\/programs\/web-archiving |
| LogicMonitor | monitoring | LogicMonitor | LogicMonitor |
| Logly Crawler | seo | Logly | zenback bot |
| LoomlyBot | social_media | Loomly | LoomlyBot |
| Loop Crawler | seo | Loop | netresearchserver |
| loopimprovementscom | monitoring | loopimprovements | loopimprovements\.com |
| Lssbot | seo | Lss | lssbot |
| Lssbot Rocket | seo | Lssbot | lssrocketcrawler |
| Ltx71 Crawler | seo | Ltx71 | ltx71 |
| Luminator Crawler | seo | Luminator | Luminator-robots |
| lwp-trivial | tool | lwp | lwp-trivial |
| Lyonl Crawler | search_engine | lyonl | lyonl-crawler |
| lyonl-asset-proxy | search_engine | lyonl | lyonl-asset-proxy |
| Macocu Crawler | archiver | Macocu | MaCoCu |
| Macrobondbot | seo | Macrobond | Macrobondbot |
| Madb Bot | seo | MADB | MADBbot\/ |
| Magellan | seo | Magellan | Magellan |
| MagiBot | search_engine | Magi | MagiBot |
| magicsearchdev/ | seo | magicsearchdev | magicsearchdev\/ |
| Magnet Me Crawler | seo | Magnetme | Magnet\.me-web\/ |
| MagnetmeBot | monitoring | Magnetme | MagnetmeBot |
| MagpieRSS | feed_fetcher | Magpie | MagpieRSS |
| Magus Bot | monitoring | Magus | Magus Bot |
| MailChimp | advertising | Mailchimp | MailChimp |
| mailru | search_engine | Mail.ru | mail\.ru |
| Mailru Crawler | search_engine | Mailru | Mail\.RU_Bot |
| MainWP/ | seo | MainWP | MainWP\/ |
| Majestic Crawler | search_engine | Majestic | MJ12bot |
| Make/ | seo | Make | Make\/ |
| ManageWP | seo | ManageWP | ManageWP |
| Mappydata Crawler | seo | Mappydata | mappydata |
| Marginalia Crawler | search_engine | Marginalia | search\.marginalia\.nu |
| MarketGoo/ | seo | MarketGoo | MarketGoo\/ |
| MarketingMiner | seo | MarketingMiner | MarketingMiner |
| masscan | security | masscan | masscan |
| Mastodon Crawler | page_preview | Mastodon | Mastodon |
| MatchorySearch | search_engine | Matchory | MatchorySearch |
| Mattermost-Bot/ | social_media | Mattermost | Mattermost-Bot\/ |
| Mauibot | search_engine | Mauibot | MauiBot |
| Mavifinds | seo | Mavifinds | Mavifinds |
| MB-LinkChecker | seo | MB | MB-LinkChecker |
| Mcontextual Bot | advertising | Mcontextual | McontextualBot |
| Media Monitoring Bot | seo | MediaMonitoring | MediaMonitoringBot\/ |
| MedialogiaBot | seo | Medialogia | MedialogiaBot |
| MediavineMetadataParser/ | advertising | Mediavine | MediavineMetadataParser\/ |
| Mediumbot-MetaTagFetcher | page_preview | Medium | Mediumbot-MetaTagFetcher |
| Megaindex Crawler | search_engine | MegaIndex | MegaIndex |
| Meltwater Crawler | seo | Meltwater | MeltwaterNews |
| MergadoBot | seo | Mergado | MergadoBot |
| Meta Externalads | advertising | Meta | meta-externalads\/ |
| Meta Externalhit | social_media | Meta | Meta-ExternalHit |
| Meta Webindexer | social_media | Meta | meta-webindexer |
| meta-externalfetcher/ | content_fetcher | Meta | meta-externalfetcher\/ |
| Meta-ExternalHit/ | seo | Meta | Meta-ExternalHit\/ |
| Metadatalabs Crawler | seo | Metadatalabs | mlbot |
| MetaIAB Facebook | advertising | Meta | MetaIAB Facebook |
| Metajob Crawler | seo | Metajob | MetaJobBot |
| Metauri Crawler | seo | Metauri | MetaURI |
| Metorik | seo | Metorik | Metorik |
| Metricstools Crawler | seo | Metricstools | MTRobot |
| MgidBot | advertising | Mgid | MgidBot |
| Microsoft Preview | search_engine | Microsoft | MicrosoftPreview\/ |
| Microsoft Research Crawler | search_engine | Microsoft | msrbot |
| Mignify Crawler | seo | Mignify | memorybot |
| Mignify Imrbot | seo | Mignify | imrbot |
| Miniatureio/ | page_preview | Miniature | Miniature\.io\/ |
| Miniflux Feedfetcher | feed_fetcher | Miniflux | Miniflux\/ |
| Minoru's Fediverse Crawler | monitoring | Minoru | Minoru's Fediverse Crawler |
| mirrorwebcom | archiver | MirrorWeb | mirrorweb\.com |
| MirrorWebCrawler | archiver | MirrorWeb | MirrorWebCrawler |
| MissinglettrBot/ | seo | Missinglettr | MissinglettrBot\/ |
| mithril-crawler | archiver | mithril | mithril-crawler |
| Mixnode Cache | content_fetcher | Mixnode | MixnodeCache\/ |
| Mixrank Bot | advertising | Mixrank | MixrankBot |
| mMechanize | scraping_framework | mMechanize | [mM]echanize |
| Moat Crawler | advertising | Moat | moatbot |
| ModatScanner | monitoring | Modat | ModatScanner |
| ModernizeBot | seo | Modernize | ModernizeBot |
| ModularConnector/ | seo | Modular | ModularConnector\/ |
| Mojeek Crawler | search_engine | Mojeek | MojeekBot\/ |
| Mollie HTTP client | seo | Mollie | Mollie HTTP client |
| Monibot | seo | Monibot | Monibot |
| monitis - | monitoring | monitis | monitis - |
| Monitorbacklinks Crawler | monitoring | Monitorbacklinks | MBCrawler |
| monitoring360bot/ | monitoring | monitoring360 | monitoring360bot\/ |
| MonitoRSS/ | feed_fetcher | MonitoRSS | MonitoRSS\/ |
| Monsido Crawler | seo | Monsido | Monsidobot |
| MonSpark/ | monitoring | MonSpark | MonSpark\/ |
| Montastic Monitor | monitoring | Montastic | MontasticMonitor|montastic-monitor |
| montastic-monitor | monitoring | Montastic | montastic-monitor |
| MontasticMonitor | monitoring | Montastic | MontasticMonitor |
| MonToolscom | seo | MonTools | MonTools\.com |
| Moodle Crawler | content_fetcher | Moodle | MoodleBot |
| Moreover Crawler | content_fetcher | Moreover | Moreover |
| Motominer Bot | seo | MotoMiner | MotoMinerBot\/ |
| Moz Crawler | seo | Moz | dotbot |
| Moz Site Audit | seo | Moz | rogerbot |
| MRGbot/ | seo | MRG | MRGbot\/ |
| Msn Crawler | search_engine | Microsoft | msnbot |
| Muckrack Crawler | seo | Muckrack | MuckRack |
| Multiviewbot | seo | Multiviewbot | Multiviewbot |
| MxToolbox | monitoring | MxToolbox | MxToolbox |
| my-tiny-bot | seo | my-tiny | my-tiny-bot |
| MyBot/ | seo | MyBot | MyBot\/ |
| Nagios Check Http | tool | Nagios | check_http |
| Nanointeractive Crawler | advertising | NanoInteractive | NanoInteractive |
| NapBot | search_engine | NapBot | NapBot |
| Naver Bot | search_engine | Naver | NaverBot |
| Naver Crawler | search_engine | Naver | Yeti |
| Naver Crawler Rss | feed_fetcher | Naver | NAVER Blog Rssbot |
| nbertaupete95 | seo | nbertaupete95 | nbertaupete95 |
| Neeva Crawler | search_engine | Neeva | Neevabot |
| Nerdbynature Crawler | content_fetcher | Nerdbynature | NerdByNature\.Bot |
| Nerdybot Crawler | scraping_framework | Nerdybot | nerdybot |
| Nessus | security | Nessus | Nessus |
| Nestdaddy Bot | search_engine | NestDaddy | NestDaddybot |
| NetAPI | seo | NetAPI | NetAPI |
| Netcraft Crawler | seo | Netcraft | NetcraftSurveyAgent |
| Netcraft SSL Server Survey | security | Netcraft | Netcraft SSL Server Survey |
| Netcraft Web Server Survey | advertising | Netcraft | Netcraft Web Server Survey |
| Netestate Crawler | search_engine | Netestate | netEstate NE Crawler |
| Neticle Crawler | social_media | Neticle | Neticle Crawler |
| NetpeakCheckerBot/ | seo | Netpeak | NetpeakCheckerBot\/ |
| Netseer Crawler | advertising | NetSeer | NetSeer crawler |
| NetShelter ContentScan | advertising | NetShelter | NetShelter ContentScan |
| Netsystemsresearch Crawler | seo | Netsystemsresearch | NetSystemsResearch |
| Netumonetumo | monitoring | Netumo | Netumo|netumo |
| Netvibes Crawler | feed_fetcher | Netvibes | Netvibes |
| NETVIGIE | monitoring | NETVIGIE | NETVIGIE |
| New York Times Newsgathering | monitoring | New York Times | New York Times Newsgathering |
| Newrelic Monitor | monitoring | New Relic | NewRelic(?:bot\/|Synthetics) |
| NewRelicbot/ | monitoring | New Relic | NewRelicbot\/ |
| NewRelicSynthetics | monitoring | New Relic | NewRelicSynthetics |
| Newsblur Aggregator | feed_fetcher | Newsblur | NewsBlur |
| Newsharecounts Crawler | seo | Newsharecounts | newsharecounts |
| NewsNow/ | seo | NewsNow | NewsNow\/ |
| Newspaper | content_fetcher | Newspaper | newspaper\/ |
| NewsRoomBI | advertising | NewsRoom | NewsRoom\.BI |
| Nextcloud Crawler | content_fetcher | Nextcloud | NextCloud |
| Nicecrawler Archive | archiver | Nicecrawler | Nicecrawler |
| NicheIndex | seo | NicheIndex | NicheIndex |
| Nict Crawler | archiver | Nict | ICC-Crawler |
| Niki Bot | social_media | Niki | niki-bot |
| Nikto | security | Nikto | Nikto |
| Ning Crawler | content_fetcher | Ning | NING\/ |
| Ninjabot | scraping_framework | Ninjabot | NINJA bot |
| Nitro Bot | seo | Nitro | NitroBot |
| Nitro- | seo | Nitro | Nitro- |
| Nixstats Crawler | monitoring | Nixstats | NIXStatsbot |
| NLNZ_IAHarvester | archiver | NLNZ | NLNZ_IAHarvester |
| NLUX_IAHarvester | archiver | NLUX | NLUX_IAHarvester |
| Nmap | tool | Nmap | Nmap Scripting Engine |
| Noah Bot | monitoring | Noah | NoahBot |
| NodePing | monitoring | NodePing | NodePing |
| Noibu | monitoring | Noibu | Noibu |
| nomore404com robot | seo | nomore404com | nomore404\.com robot |
| noorobot | seo | noorobot | noorobot |
| Nooshub/ | seo | Nooshub | Nooshub\/ |
| NostoCrawlerBot | advertising | Nosto | NostoCrawlerBot |
| Notabot | seo | Notabot | Notabot |
| Notion/ | content_fetcher | Notion | Notion\/ |
| Novaact/ | seo | Novaact | Novaact\/ |
| Novellum | seo | Novellum | Novellum |
| NsToolsBot/ | seo | NsTools | NsToolsBot\/ |
| Nutch | tool | Nutch | Nutch |
| Nuzzel Crawler | feed_fetcher | Nuzzel | Nuzzel |
| nvdorz | seo | nvdorz | nvdorz |
| nyt_scraping | seo | nyt | nyt_scraping |
| Observepoint Monitor | monitoring | ObservePoint | ObservePoint |
| Ocarinabot | scraping_framework | Ocarinabot | Ocarinabot |
| Odin; | seo | Odin | Odin; |
| Offline Explorer | archiver | Offline Explorer | Offline Explorer |
| OhDear/ | monitoring | OhDear | OhDear\/ |
| OICrawler | search_engine | OI | OICrawler |
| Okru Crawler | social_media | Okru | OdklBot\/ |
| Omnisend/ | seo | Omnisend | Omnisend\/ |
| Oncrawl | seo | Oncrawl | OnCrawl\/ |
| OneTrust | monitoring | OneTrust | OneTrust |
| Online Domain Tools | seo | Online Domain Tools | Online Domain Tools |
| OnlineOrNotcom_bot | monitoring | OnlineOrNot.com | OnlineOrNot\.com_bot |
| opencode-smartfetch | content_fetcher | opencode | opencode-smartfetch |
| Opengraphcheck Crawler | page_preview | Opengraphcheck | OpenGraphCheck |
| OpenGraphio/ | page_preview | OpenGraph.io | OpenGraph\.io\/ |
| Openhose Crawler | content_fetcher | Openhose | OpenHoseBot |
| Openindex Crawler | search_engine | Openindex | openindexspider |
| OpenRSS | feed_fetcher | OpenRSS | OpenRSS |
| Openthebox Bot | search_engine | OpenTheBox | OpenTheBoxBot |
| OpenVAS | security | OpenVAS | OpenVAS |
| Operator/ | seo | Operator | Operator\/ |
| Optimizer | seo | Optimizer | Optimizer\) |
| Orange Crawler | seo | Orange | OrangeBot\/ |
| Orange Ftgroup Crawler | content_fetcher | Orange Ftgroup | voilabot |
| Orbbot/ | seo | Orb | Orbbot\/ |
| Orlo-LinkPreview/ | page_preview | Orlo | Orlo-LinkPreview\/ |
| Outbrain Link Checker | monitoring | Outbrain | outbrain |
| Outclicks Crawler | seo | Outclicks | OutclicksBot |
| OutsellURLValidator | seo | Outsell | OutsellURLValidator |
| Overcast/ | content_fetcher | Overcast | Overcast\/ |
| Owler owseu | search_engine | Owler | Owler \(ows\.eu |
| OWLer-W | monitoring | OWLer | OWLer-W |
| Page To Rss | feed_fetcher | Page To Rss | page2rss |
| page-preview-tool | seo | page-preview-tool | page-preview-tool |
| Pagepeeker Crawler | monitoring | Pagepeeker | PagePeeker\/ |
| Pagespeed/ | monitoring | Pagespeed\/ | |
| Pagething Crawler | seo | Pagething | PageThing |
| Paloaltonetworks Crawler | security | Paloaltonetworks | Expanse |
| Panopta | monitoring | Panopta | Panopta |
| Panscient Crawler | seo | Panscient | panscient |
| Paperli Crawler | seo | Paperli | PaperLiBot |
| Paqle Bot | seo | Paqlebot | Paqlebot\/ |
| parsely scraper/ | seo | Parsely | parse\.ly scraper\/ |
| ParselySharesBot | monitoring | Parsely | ParselySharesBot |
| PayPal/ | seo | PayPal | PayPal\/ |
| PDF24 URL To PDF | seo | PDF24 | PDF24 URL To PDF |
| peer39_crawler | advertising | peer39 | peer39_crawler |
| Penthouse Critical Path CSS Generator | seo | Penthouse | Penthouse Critical Path CSS Generator |
| Perl Libwww | http_library | Perl Libwww | libwww-perl |
| Perl Pcore | http_library | Perl Pcore | Pcore-HTTP |
| PerplexityUser | search_engine | Perplexity | PerplexityUser |
| Php Curlclass | http_library | Php Curlclass | ^PHP-Curl-Class |
| Php Phpcrawl | http_library | Php Phpcrawl | phpcrawl |
| Php Simple Scraper | http_library | Php Simple Scraper | SimpleScraper |
| Php Simplepie | http_library | Php Simplepie | SimplePie |
| Phxbot | scraping_framework | Phx | PhxBot |
| Picsearch Crawler | search_engine | Picsearch | psbot |
| PingAdminRu/ | monitoring | PingAdminRu | PingAdmin\.Ru\/ |
| Pingdom Crawler | monitoring | SolarWinds | [pP]ingdom |
| pingpingio/ | monitoring | pingpingio | pingping\.io\/ |
| Pinterest Crawler | social_media | Pinterest(bot)?\/ | |
| Pintrest Crawler | social_media | pinterest\.com\/bot | |
| Pipl Crawler | search_engine | Pipl | PiplBot |
| Pixalatecom | advertising | Pixalate | Pixalate\.com |
| Plagaware Bot | monitoring | PlagAware | PlagAwareBot |
| PlayStore-Google | search_engine | PlayStore-Google | |
| Playwright | http_library | Playwright | Playwright |
| Plesk screenshot bot | seo | Plesk | Plesk screenshot bot |
| Pocket Crawler | feed_fetcher | PocketParser | |
| PocketCasts/ | content_fetcher | PocketCasts | PocketCasts\/ |
| PodchaserParser | scraping_framework | Podchaser | PodchaserParser |
| Podimo | content_fetcher | Podimo | Podimo |
| Poduptime | monitoring | Poduptime | Poduptime;\s*\+fediverse\.observer |
| Poduptime | monitoring | Poduptime | Poduptime |
| Pomothy Bot | monitoring | Pomothy | Pomothy-Bot |
| Postman | tool | Postman | PostmanRuntime\/ |
| Postrank Crawler | content_fetcher | Postrank | postrank |
| Potions/ | seo | Potions | Potions\/ |
| Prcy Crawler | seo | Prcy | PR-CY\.RU |
| PressEngineBot | seo | PressEngine | PressEngineBot |
| PricedroneShoppingBot/ | seo | PricedroneShopping | PricedroneShoppingBot\/ |
| PriEcoBot/ | seo | PriEco | PriEcoBot\/ |
| PrintFriendlycom | seo | PrintFriendly | PrintFriendly\.com |
| Privacore Crawler | seo | Privacore | findxbot |
| Privacyaware Crawler | seo | Privacyaware | PrivacyAwareBot |
| Pro-Sitemaps/ | seo | Pro-Sitemaps | Pro-Sitemaps\/ |
| ProbelySPDR/ | seo | Probely | ProbelySPDR\/ |
| Productsup Crawler | advertising | productsup | productsup\.io\/crawler |
| Profound Crawler | seo | Profound | urlappendbot |
| ProjectShield-UrlCheck | security | ProjectShield | ProjectShield-UrlCheck |
| Protopage/ | search_engine | Protopage | Protopage\/ |
| Proximic Crawler | content_fetcher | Proximic | proximic |
| Prtg Monitor | monitoring | Prtg | PRTG(?:CloudBot\/| Network Monitor) |
| PRTG Network Monitor | monitoring | Paessler | PRTG Network Monitor |
| PRTGCloudBot/ | monitoring | Paessler | PRTGCloudBot\/ |
| PS_Daily/ | seo | PS | PS_Daily\/ |
| Pulsepoint | advertising | PulsePoint | Pulsepoint |
| Pulsepoint Crawler | advertising | Pulsepoint | Pulsepoint |
| PulsePoint-Crawler | advertising | PulsePoint | PulsePoint-Crawler |
| pulseticcom | monitoring | pulseticcom | pulsetic\.com |
| Puppeteer | http_library | Puppeteer | Puppeteer |
| Pure Crawler | seo | Pure | purebot |
| PWABuilderHttpAgent | seo | PWABuilder | PWABuilderHttpAgent |
| Python Aiohttp | http_library | Python | aiohttp |
| Python Bitbot | http_library | Python | BitBot |
| Python Httpx | http_library | Python | httpx |
| Python Opengraph | http_library | Python | python-opengraph |
| Python Requests | http_library | Python | python-requests |
| Python Urllib | http_library | Python | Python-urllib |
| Pywikibot/ | seo | Pywikibot | Pywikibot\/ |
| qcbot | advertising | qcbot | qcbot |
| Qualified Bot | seo | Qualified | QualifiedBot\/ |
| Qualys | monitoring | Qualys | Qualys |
| Quantcast Crawler | advertising | Quantcast | Quantcastbot\/ |
| Quora-Bot | social_media | Quora | Quora-Bot |
| Qwant Crawler | search_engine | Qwant | Qwant(?:ify|bot) |
| Qwantbot | search_engine | Qwant | Qwantbot |
| Qwantify | search_engine | Qwant | Qwantify |
| Qwarrybot | advertising | Qwarry | Qwarrybot |
| Rackspace Monitoring/ | monitoring | Rackspace | Rackspace Monitoring\/ |
| Rakuten Image extraction bot | monitoring | Rakuten | Rakuten Image extraction bot |
| Rakuten Usa Image Bot | seo | Rakuten | rakutenusabot-image\/ |
| Rankactive Crawler | seo | Rankactive | RankActiveLinkBot |
| RankurBot/ | seo | Rankur | RankurBot\/ |
| RavenCrawler/ | seo | Raven | RavenCrawler\/ |
| rawweb-bot | scraping_framework | rawweb | rawweb-bot |
| Readable/ | seo | Readable | Readable\/ |
| Recurly Webhooks/ | seo | Recurly | Recurly Webhooks\/ |
| RED/ | monitoring | RED | RED\/ |
| Reddit Crawler | social_media | redditbot | |
| Reelevant/ | seo | Reelevant | Reelevant\/ |
| Refind Crawler | feed_fetcher | Refind | Refindbot |
| removebg/ | seo | remove.bg | remove\.bg\/ |
| Repo Lookout Bot | seo | RepoLookout | RepoLookoutBot |
| ResearchBot | archiver | ResearchBot | ResearchBot |
| researchscancomsysrwth-aachende | archiver | researchscan.comsys.rwth-aachen.de | researchscan\.comsys\.rwth-aachen\.de |
| Retool/ | seo | Retool | Retool\/ |
| Retrevo Page Analyzer | seo | Retrevo | RetrevoPageAnalyzer |
| Retroliste Crawler | seo | RetroListeCOM | RetroListeCOM\/ |
| RevvimGort/ | seo | RevvimGort | RevvimGort\/ |
| reward-gateway | seo | reward-gateway | reward-gateway |
| Ridder Crawler | seo | Ridder | RidderBot |
| Riddler http://riddlerio | ai | Riddler | Riddler \(http:\/\/riddler\.io |
| Rigor | monitoring | Rigor | Rigor\) |
| Rivva Crawler | seo | Rivva | Rivva |
| RobotsChecker/ | seo | RobotsChecker | RobotsChecker\/ |
| RSiteAuditor | seo | RSiteAuditor | RSiteAuditor |
| rss-is-deadlol web bot | feed_fetcher | rss-is-deadlol | rss-is-dead\.lol web bot |
| rss-parser | feed_fetcher | rss-parser | rss-parser |
| rss2tg | feed_fetcher | rss2tg | rss2tg |
| RSSAPI/ | feed_fetcher | RSSAPI | RSSAPI\/ |
| Rssbot Feedfetcher | feed_fetcher | Rssbot | rssbot\/ |
| Rssing Crawler | feed_fetcher | RSSing | RSSingBot |
| Rssmicro Feedfetcher | feed_fetcher | Rssmicro | Atom Feed Robot |
| RssReaderBot | feed_fetcher | RssReader | RssReaderBot |
| RSSSocial | feed_fetcher | RSSSocial | RSS\.Social |
| Ruby Metainspector | http_library | Ruby | MetaInspector |
| Ryte Crawler | seo | Ryte | RyteBot |
| s4a-probe-bot/ | seo | s4a-probe | s4a-probe-bot\/ |
| s4a/ | seo | s4a | s4a\/ |
| Salesforcecom | content_fetcher | Salesforce | Salesforce\.com |
| SandobaCrawler/ | seo | Sandoba | SandobaCrawler\/ |
| Sansec Security Monitor/ | security | Sansec Security Monitor | Sansec Security Monitor\/ |
| Scan Interfax Crawler | seo | Interfax | InterfaxScanBot |
| Schema-Markup-Validator | seo | Schema-Markup-Validator | Schema-Markup-Validator |
| Schmorp Crawler | tool | Schmorp | AnyEvent |
| Scoopit/ | seo | Scoopit | Scoop\.it\/ |
| Scope3 Crawler | advertising | Scope3 | Scope3 |
| Scour Rss Bot | feed_fetcher | ScourRSSBot | ScourRSSBot\/ |
| Scoutjet Crawler | seo | Scoutjet | ScoutJet |
| ScrapeheroBot/ | seo | ScrapeheroBot | ScrapeheroBot\/ |
| scraping@nytimescom | advertising | The New York Times | scraping@nytimes\.com |
| Screamingfrog Crawler | seo | Screaming Frog | Screaming Frog SEO Spider |
| screeenly-bot | seo | Screeenly | screeenly-bot |
| Scribd Crawler | content_fetcher | Scribd | scribdbot |
| Scritch Crawler | content_fetcher | Scritch | Fetch\/ |
| Scrubby | search_engine | Scrubby | Scrubby |
| Scrunch Bot | monitoring | Scrunch | Scrunchbot |
| Searcherweb | search_engine | Searcherweb | Searcherweb |
| Searcherxweb | search_engine | Searcherxweb | Searcherxweb |
| SearchExpress | search_engine | SearchExpress | SearchExpress |
| SEBot-WA | seo | SEBot | SEBot-WA |
| SecurityHeaders | security | SecurityHeaders | SecurityHeaders |
| Seekbot | search_engine | Seekbot | ^Seekbot |
| Seekbot Crawler | seo | Seekbot | seekbot |
| Seekport Crawler | search_engine | Seekport | Seekport |
| Seewithkids Crawler | content_fetcher | Seewithkids | seewithkids |
| Selenium | http_library | Selenium | Selenium |
| semaltbot/ | seo | Semalt | semaltbot\/ |
| semantic-visions | seo | Semantic Visions | semantic-visions |
| Semanticaudience Crawler | seo | Semanticaudience | semanticbot |
| Sempitech Crawler | seo | Sempitech | sempi\.tech |
| Semrush Crawler | seo | Semrush | S[eE][mM]rushBot |
| SendGrid Event API | seo | SendGrid | SendGrid Event API |
| SENTINEL-LinkCheck | monitoring | SENTINEL | SENTINEL-LinkCheck |
| Sentry Crawler | monitoring | Sentry | (^| )sentry\/ |
| Sentry Uptime Monitor | monitoring | Sentry | SentryUptimeBot\/1\.0 |
| SentryUptimeBot/ | monitoring | Sentry | SentryUptimeBot\/ |
| Senuto Crawler | seo | Senuto | SenutoBot |
| seo-audit-check-bot/ | seo | seo-audit-check | seo-audit-check-bot\/ |
| seo4ajaxcom | seo | seo4ajax | seo4ajax\.com |
| Seobility Crawler | seo | Seobility | SeobilityBot |
| Seokicks Crawler | seo | Seokicks | SEOkicks |
| Seolizer Crawler | seo | Seolizer | SEOlizer |
| seoLyt | seo | seoLyt | seoLyt |
| Seoprofiler Crawler | seo | Seoprofiler | spbot |
| Seoscanners Crawler | seo | Seoscanners | seoscanners |
| SeoSiteCheckup | seo | SeoSiteCheckup | SeoSiteCheckup |
| Seostar Crawler | seo | Seostar | seostar\.co |
| SeoulBot | seo | Seoul | SeoulBot |
| Seozoom Crawler | seo | Seozoom | ZoomBot |
| SequelWP | monitoring | SequelWP | SequelWP |
| SERankingBacklinksBot | seo | SERanking | SERankingBacklinksBot |
| Serendeputy Crawler | feed_fetcher | Serendeputy | SerendeputyBot |
| Serpstatbot Crawler | seo | Serpstatbot | serpstatbot\/ |
| SERPtimizerBot | seo | SERPtimizer | SERPtimizerBot |
| Server Density Service Monitoring | monitoring | Server Density | Server Density Service Monitoring |
| ServerDensity | monitoring | Server Density | ServerDensity |
| ServerHunterSpider/ | seo | ServerHunter | ServerHunterSpider\/ |
| Seznam Crawler | search_engine | Seznam | SeznamBot |
| SeznamHomepageCrawler/ | search_engine | Seznam | SeznamHomepageCrawler\/ |
| SFDC-Callout/ | seo | SFDC | SFDC-Callout\/ |
| Shopify-Captain-Hook | seo | Shopify | Shopify-Captain-Hook |
| ShortPixel | advertising | ShortPixel | ShortPixel |
| Shortwave Image Fetcher | content_fetcher | Shortwave | Shortwave Image Fetcher |
| Silk/ | seo | Silk | ^Silk\/ |
| Silktide | monitoring | Silktide | Silktide |
| Similartech Crawler | seo | Similartech | smtbot |
| Simple Crawler | scraping_framework | Simple | SimpleCrawler |
| SinceraSyntheticUser/ | seo | Sincera | SinceraSyntheticUser\/ |
| Sindup | monitoring | Sindup | Sindup |
| Sirdata Bot | advertising | Sirdata | SirdataBot |
| Sistrix 007ac9 Crawler | seo | Sistrix | 007ac9 Crawler |
| Sistrix Crawler | seo | Sistrix | (sistrix|SISTRIX) [cC]rawler |
| Site24x7 | monitoring | Site24x7 | Site24x7 |
| SiteAuditBot/ | seo | SiteAudit | SiteAuditBot\/ |
| Sitebot Crawler | seo | Sitebot | sitebot |
| Sitebulb | seo | Sitebulb | sitebulb |
| SiteCheck-sitecrawl | seo | SiteCheck | SiteCheck-sitecrawl |
| Sitechecker Crawler | seo | Sitechecker | SiteCheckerBotCrawler |
| Siteexplorer Crawler | seo | Siteexplorer | siteexplorer\.info |
| Siteimprove Crawler | seo | Siteimprove | Siteimprove\.com |
| SiteLock | monitoring | SiteLock | SiteLock |
| SiteScoreBot | seo | SiteScore | SiteScoreBot |
| SiteSearch360/ | search_engine | SiteSearch360 | SiteSearch360\/ |
| SitesOverPagesBot | seo | SitesOverPages | SitesOverPagesBot |
| SiteSucker | scraping_framework | SiteSucker | SiteSucker |
| SiteUptimecom | monitoring | SiteUptime | SiteUptime\.com |
| Skroutz ImageBot | seo | Skroutz | Skroutz ImageBot |
| SkroutzBot | seo | Skroutz | SkroutzBot |
| Skype Preview | search_engine | Microsoft | SkypeUriPreview |
| SkyworkSpider | seo | Skywork | SkyworkSpider |
| SlaccaleBot | seo | Slaccale | SlaccaleBot |
| Slack Crawler | page_preview | Slack | Slackbot |
| Slack Image Proxy | page_preview | Slack | Slack-ImgProxy |
| SleepBot | monitoring | Sleep | SleepBot |
| SlickBot/ | seo | Slick | SlickBot\/ |
| SmarshBot | social_media | Smarsh | SmarshBot |
| Smartology Bot | advertising | Smartology | SmartologyBot\/ |
| SMTnetPMBot | search_engine | SMTnetPM | SMTnetPMBot |
| Snap Preview | page_preview | Snap | Snap URL Preview Service |
| Snapchat Adsbot | advertising | Snapchat | SnapchatAds |
| SnapchatAds/ | advertising | Snapchat | SnapchatAds\/ |
| SnapURLPreview/ | page_preview | Snap | SnapURLPreview\/ |
| Snipcart/ | seo | Snipcart | Snipcart\/ |
| Socialrank Crawler | seo | Socialrank | SocialRankIOBot |
| Softbytelabs Crawler | seo | Softbytelabs | SBL-BOT |
| Software-Security-Research | archiver | Software-Security-Research | Software-Security-Research |
| Sogou Crawler | search_engine | Sogou | Sogou |
| solarwinds/ | monitoring | SolarWinds | solarwinds\/ |
| Sora POS/ | seo | Sora | Sora POS\/ |
| Sosospider | search_engine | Soso | Sosospider |
| SottopopNone | content_fetcher | Sottopop | SottopopNone |
| SparkPost | seo | SparkPost | SparkPost |
| SparkShipping | seo | SparkShipping | SparkShipping |
| Spawning-AI | ai | Spawning | Spawning-AI |
| Specificfeeds | feed_fetcher | Specificfeeds | Specificfeeds |
| Spectate Monitor | monitoring | Spectate | Spectate\/ |
| SpiderLing | archiver | SpiderLing | SpiderLing |
| SpidersSspidercom | scraping_framework | SpidersSspider | Spider[\s\S]*spider\.com |
| splash Version/ | tool | splash | splash Version\/ |
| Splunk | monitoring | Splunk | Splunk |
| SpringserveBot | advertising | Springserve | SpringserveBot |
| sqlmap | security | sqlmap | sqlmap |
| SQWatcher | monitoring | SQWatcher | SQWatcher |
| SSL Labs | security | SSL Labs | SSL Labs$ |
| SSL Labs https://wwwssllabscom | security | SSL Labs | SSL Labs \(https:\/\/www\.ssllabs\.com |
| SSSSBot/ | seo | SSSS | SSSSBot\/ |
| Stape/ | seo | Stape | Stape\/ |
| Startme Crawler | feed_fetcher | Startme | startmebot\/ |
| StartpagePrivateImageProxy/ | search_engine | Startpage | StartpagePrivateImageProxy\/ |
| Statabot/ | seo | Statabot | Statabot\/ |
| StatistikAustria/ | seo | StatistikAustria | StatistikAustria\/ |
| StatsDroneBot | seo | StatsDrone | StatsDroneBot |
| Statuscake Monitor | monitoring | Statuscake | StatusCake |
| Statusnest Crawler | seo | StatusNest | StatusNestBacklinkSpider |
| Steam Preview | page_preview | Valve | Valve\/Steam |
| Stepstone Crawler | search_engine | Stepstone | stepstoneCrawlBot |
| Stract Crawler | search_engine | Stract | StractBot |
| Stripe Crawler | http_library | Stripe | Stripebot\/ |
| Stripe Webhook | webhooks | Stripe | Stripe\/ |
| Stuttgart Crawler | seo | Stuttgart | RegionStuttgartBot |
| Sucuri | security | Sucuri | Sucuri |
| Summaly Crawler | tool | Summaly | SummalyBot |
| Summify Crawler | social_media | Summify | summify |
| Supabase Paired Crawler | seo | Supabase | Supabase Paired Crawler |
| Superfeedr Crawler | feed_fetcher | Superfeedr | Superfeedr |
| Surly Crawler | monitoring | Surly | SurdotlyBot |
| SustainabilityCrawler | seo | Sustainability | SustainabilityCrawler |
| sv-watchagent | monitoring | sv-watchagent | sv-watchagent |
| Svix-Webhooks/ | seo | Svix | Svix-Webhooks\/ |
| Swiftbot | search_engine | Swift | Swiftbot |
| SwifteqLinkChecker | seo | Swifteq | SwifteqLinkChecker |
| Swimgbot | seo | Swimg | SWIMGBot |
| Swisscows | search_engine | Swisscows | Swisscows |
| Synapse Crawler | page_preview | Synapse | Synapse |
| Synthesi Bot | monitoring | Synthesi | SynthesiBot |
| synthetic-monitoring-agent/ | monitoring | synthetic-monitoring-agent | synthetic-monitoring-agent\/ |
| Sysomos Crawler | social_media | Sysomos | Sysomos |
| T3versions Crawler | seo | T3versions | t3versionsBot |
| Taboola Crawler | advertising | Taboola | Taboolabot |
| TactiScout/ | seo | TactiScout | TactiScout\/ |
| Tagoo Crawler | seo | Tagoo | tagoobot |
| Tangiblee Crawler | seo | Tangiblee | TangibleeBot |
| TaraGroup Intelligent Bot | seo | TaraGroup | TaraGroup Intelligent Bot |
| tchelebi/ | seo | tchelebi | tchelebi\/ |
| Telegram Crawler | page_preview | Telegram | TelegramBot |
| Termly | monitoring | Termly | Termly |
| Test Certificate Info | security | Test Certificate Info | Test Certificate Info |
| test-bot | seo | test-bot | test-bot |
| Testcrawler | seo | Testcrawler | Testcrawler |
| TestLocally/ | tool | TestLocally | TestLocally\/ |
| Testomato Crawler | monitoring | Testomato | minicrawler |
| TestURI | seo | TestURI | TestURI |
| TextRazor | seo | TextRazor | TextRazor |
| The Knowledge AI | ai | The Knowledge AI | The Knowledge AI |
| TheInternetSearchx | search_engine | TheInternetSearchx | TheInternetSearchx |
| Theoldreader Crawler | feed_fetcher | Theoldreader | theoldreader\.com |
| thesis-research-bot | archiver | thesis-research | thesis-research-bot |
| Thinklab Crawler | seo | Thinklab | Thinklab |
| ThousandEyes | monitoring | ThousandEyes | ThousandEyes |
| Tiger Crawler | seo | Tiger | tigerbot |
| Tiktok Crawler | social_media | ByteDance | TikTokSpider |
| Tineye Crawler | seo | TinEye | TinEye |
| Tiscali Crawler | seo | Tiscali | IstellaBot |
| TLS tester | security | TLS tester | TLS tester |
| Tombascraper Crawler | scraping_framework | Tomba | TombaPublicWebCrawler |
| top100ramblerru crawler | search_engine | top100ramblerru | top100\.rambler\.ru crawler |
| Toplist Crawler | seo | Toplist | toplistbot |
| Torus Crawler | scraping_framework | Torus | ReverseEngineeringBot |
| Toutiao Crawler | search_engine | Toutiao | ToutiaoSpider |
| Traackr Crawler | seo | Traackr | Traackr\.com |
| Tracemyfile Crawler | seo | Tracemyfile | tracemyfile |
| tracking-quality-spider | advertising | tracking-quality | tracking-quality-spider |
| Trae/ | scraping_framework | Trae | Trae\/ |
| Trafilatura | tool | trafilatura | trafilatura |
| Trellis-Services | seo | Trellis | Trellis-Services |
| Trendiction Crawler | seo | Trendiction | trendictionbot |
| Trendsmap Crawler | archiver | Trendsmap | TrendsmapResolver |
| trentwiles | seo | trentwiles | trentwil\.es |
| Trove Crawler | archiver | Trove | Trove |
| Trovit Crawler | seo | Trovit | trovitBot |
| Trustly/ | seo | Trustly | Trustly\/ |
| Tsm Bot | monitoring | TSM | TSMbot |
| TSM-turingos | scraping_framework | TSM | TSM-turingos |
| Ttd Content | advertising | TTD | TTD-Content |
| Ttrss Feedfetcher | feed_fetcher | Tiny Tiny RSS | Tiny Tiny RSS |
| Tumblr/ | social_media | Tumblr | Tumblr\/ |
| Turnitin Crawler | archiver | Turnitin | Turnitin |
| Tweakers | seo | Tweakers | Tweakers |
| Tweetedtimes Crawler | social_media | Tweetedtimes | tweetedtimes |
| Tweetmemebot | social_media | Tweetmeme | TweetmemeBot |
| Twenga Crawler | seo | Twenga | twengabot |
| TwilioProxy/ | webhooks | Twilio | TwilioProxy\/ |
| Twingly Crawler | seo | Twingly | Twingly |
| TwinWaveScanner | security | TwinWave | TwinWaveScanner |
| Twitter Crawler | page_preview | Twitterbot | |
| Twoip Crawler | seo | 2ip | 2ip bot |
| Twoip Crawler Cms | seo | 2ip | 2ip\.ru |
| Twurly Crawler | seo | Twurly | Twurly |
| Uas Link Checker | seo | UASlinkChecker | UASlinkChecker\/ |
| Ubermetrics Crawler | seo | Ubermetrics | um-LN |
| Ubt Crawler | seo | Ubt | Streamline3Bot\/ |
| UGAResearchAgent | archiver | UGA | UGAResearchAgent |
| Uip Bot | advertising | uip | uipbot |
| um-FC | advertising | um | um-FC |
| um-IC | advertising | um | um-IC |
| UnshortenIt! | link_checker | UnshortenIt | Unshorten\.It\! |
| upday/ | content_fetcher | upday | upday\/ |
| updownio | monitoring | updownio | updown\.io |
| Upflow Crawler | seo | Upflow | Upflow |
| Uptime Monitor | monitoring | UptimeRobot | www\.uptime\.com |
| Uptime/ | monitoring | Uptime | Uptime\/ |
| Uptimebot Monitor | monitoring | Uptimebot | Uptimebot\.org |
| UptimeBot/ | monitoring | UptimeBot | UptimeBot\/ |
| uptimedoctor | monitoring | UptimeDoctor | uptimedoctor |
| Uptimerobot Monitor | monitoring | UptimeRobot | UptimeRobot |
| UptimeStatistics | monitoring | UptimeStatistics | UptimeStatistics |
| Uptimia | monitoring | Uptimia | Uptimia |
| uptrends | monitoring | Uptrends | uptrends |
| UrlBeeBot | seo | UrlBee | UrlBeeBot |
| Urlcheckr/ | seo | Urlcheckr | Urlcheckr\/ |
| Urlclassification Crawler | seo | Urlclassification | KStandBot |
| Urlsuma Crawler | monitoring | UrlSuMade | UrlSuMa\.de crawler |
| URLSuMaBot | seo | URLSuMa | URLSuMaBot |
| useeBookChecker/ | seo | useeBookChecker | useeBookChecker\/ |
| Usine Nouvelle Crawler | seo | Usine Nouvelle | UsineNouvelleCrawler |
| Utexas Crawler | archiver | Utexas | UT-Dorkbot |
| Utorrent Crawler | content_fetcher | Utorrent | BTWebClient |
| Vagabondo/ | seo | Vagabondo | Vagabondo\/ |
| VaultPress | content_fetcher | VaultPress | VaultPress |
| Vebidoo Crawler | seo | Vebidoo | vebidoobot |
| Veooz Crawler | seo | Veooz | Veoozbot |
| Vercel Crawler | monitoring | Vercel | Vercelbot |
| Vercel Monitor Preview | monitoring | Vercel | vercel-screenshot |
| Verisign Ips Agent | security | Verisign | ips-agent |
| Verispider | seo | Verispy | Verispider |
| Verity/ | advertising | Verity | Verity\/ |
| Viber Crawler | page_preview | Viber | Viber |
| videootv Bot | monitoring | Videootv | videootv Bot |
| videootvBot | seo | Videootv | videootvBot |
| Vigil Crawler | security | Vigil | Vigil\/ |
| Vipnytt Crawler | seo | Vipnytt | jpg-newsbot |
| Virustotal Crawler | security | Virustotal | virustotal |
| visionheightcom/scan | seo | VisionHeight | visionheight\.com\/scan |
| Vkrobot Crawler | social_media | VK | VKRobot |
| Vkshare Crawler | social_media | VK | vkShare |
| vmcrawl | monitoring | Vm | vmcrawl |
| VsuSearchSpider/ | search_engine | Vsu | VsuSearchSpider\/ |
| vu-server-health-scanner/ | archiver | Vu | vu-server-health-scanner\/ |
| Vuhuv Crawler | scraping_framework | Vuhuv | vuhuvBot |
| W3c Validator Css | monitoring | W3C | W3C_CSS_Validator |
| W3c Validator Feed | monitoring | W3C | FeedValidator |
| W3c Validator Html | monitoring | W3C | W3C_Validator |
| W3c Validator Html Nu | monitoring | Validator.nu | Validator\.nu |
| W3c Validator I18n | monitoring | W3C | W3C_I18n-Checker |
| W3c Validator Links | monitoring | W3C | W3C-checklink |
| W3c Validator Mobile | monitoring | W3C | W3C-mobileOK |
| W3c Validator Unified | monitoring | W3C | W3C_Unicorn |
| WadooBot | search_engine | Wadoo | WadooBot |
| WanscannerBot | security | Wanscanner | WanscannerBot |
| Ward Bot | seo | WARD | WARDBot\/ |
| Warebay Crawler | seo | Warebay | wbsearchbot |
| Watchbot monitoring robot | monitoring | Watchbot | Watchbot monitoring robot |
| Watchfor Bot | monitoring | WatchFor | WatchForBot |
| Watchful | monitoring | Watchful | Watchful |
| WatchMouse | monitoring | WatchMouse | WatchMouse |
| watchTowr | monitoring | watchTowr | watchTowr |
| Wayback | archiver | Wayback | Wayback |
| Web Measure/ | archiver | Web Measure | Web Measure\/ |
| Web Screen Service By hyperhost | seo | hyperhost | Web Screen Service By hyperhost |
| Webarchive Crawler | archiver | Webarchive | web-archive-net\.com\.bot |
| WebCapture | archiver | WebCapture | WebCapture |
| Webceo Crawler | seo | Webceo | online-webceo-bot |
| WebCEO Online/ | seo | WebCEO | WebCEO Online\/ |
| Webcompany Crawler | content_fetcher | Webcompany | webcompanycrawler |
| WebCopier | archiver | WebCopier | WebCopier |
| Webdatastats Crawler | seo | Webdatastats | WebDataStats |
| Webeaver Crawler | seo | Webeaver | BehloolBot |
| Webflow | seo | Webflow | Webflow |
| webgains-bot | advertising | webgains | webgains-bot |
| Webmeup Crawler | seo | Webmeup | BLEXBot\/ |
| Webmon | monitoring | Webmon | webmon |
| weborama-fetcher | advertising | Weborama | weborama-fetcher |
| Webpagetest Crawler | monitoring | Webpagetest | (^| )PTST\/ |
| webprosbot/ | seo | webpros | webprosbot\/ |
| Website Info Robot | seo | Website-infonet | Website-info\.net-Robot |
| WebsiteOps | monitoring | WebsiteOps | WebsiteOps |
| websitepulse | monitoring | websitepulse | websitepulse |
| WebSniffer/ | seo | WebSniffer | WebSniffer\/ |
| webspidermount | advertising | webspidermount | webspidermount |
| WebwikiBot/ | seo | Webwiki | WebwikiBot\/ |
| webzio | seo | Webzio | webzio |
| Webzio Crawler | search_engine | Webzio | webzio\s |
| WebZIP | archiver | WebZIP | WebZIP |
| WEDOS OnLine | monitoring | WEDOS | WEDOS OnLine |
| Wellknown Crawler | search_engine | Wellknown | WellKnownBot |
| Wepch Crawler | search_engine | Wepch | WepchSearchEngine |
| Wesee Crawler | search_engine | Wesee | WeSEE:Search |
| Wget | tool | wget project | [wW]get |
| Whatsapp Crawler | social_media | ||
| WhatsMyIPorg | seo | WhatsMyIP | WhatsMyIP\.org |
| WhatWeb/ | security | WhatWeb | WhatWeb\/ |
| Wheregoescom | seo | Wheregoes | Wheregoes\.com |
| wheresitupcom/ | monitoring | wheresitup | wheresitup\.com\/ |
| WikiDo | search_engine | WikiDo | WikiDo |
| WireReaderBot/ | seo | WireReader | WireReaderBot\/ |
| WISEbot | monitoring | WISE | WISEbot |
| WJHRO/ | seo | WJHRO | WJHRO\/ |
| wknd-bot | advertising | wknd | wknd-bot |
| wmtipscom/ | seo | wmtipscom | wmtips\.com\/ |
| Wocodi Crawler | seo | Wocodi | wocbot |
| Woorank Crawler | seo | Woorank | woobot |
| Woorank Crawler Review | seo | Woorank | woorankreview\/ |
| WordCountBot/ | seo | WordCount | WordCountBot\/ |
| Wordpress Crawler | content_fetcher | Wordpress | WordPress\/ |
| Wordpress Crawler Rss | feed_fetcher | Wordpress | feedbot |
| Wordup Crawler | seo | Wordup | WordupInfoSearch |
| Wordup-1 | seo | Wordup | Wordup-1 |
| Worio Crawler | seo | Worio | woriobot |
| workona-favicon-service/ | seo | workona | workona-favicon-service\/ |
| WorldBot | seo | World | WorldBot |
| WormlyBot | monitoring | Wormly | WormlyBot |
| Wotbox Crawler | seo | Wotbox | wotbox |
| WOVN Crawler | monitoring | WOVN | WOVN Crawler |
| WovnCrawler/ | seo | Wovn | WovnCrawler\/ |
| wowLink Crawler/ | seo | wowLink | wowLink Crawler\/ |
| WP Time Capsule | seo | WP | WP Time Capsule |
| wpbot/ | seo | wpbot | wpbot\/ |
| WPMU DEV Broken Link Checker | seo | WPMU DEV | WPMU DEV Broken Link Checker |
| WPMU DEV Hub | monitoring | WPMU DEV | WPMU DEV Hub |
| WPMUDEV Uptime Monitor | monitoring | WPMUDEV | WPMUDEV Uptime Monitor |
| WPScan | security | WPScan | WPScan |
| WPSec/ | security | WPSec | WPSec\/ |
| WPUmbrella | monitoring | WPUmbrella | WPUmbrella |
| WSM/ | seo | WSM | WSM\/ |
| WTotem | advertising | WTotem | WTotem |
| Xenu Crawler | monitoring | Xenu | Xenu Link Sleuth |
| Xing Bot | social_media | Xing Bot | |
| XML Sitemaps Generator | seo | XML Sitemaps | XML Sitemaps Generator |
| Xovibot Crawler | seo | Xovibot | xovibot |
| XoviOnpageCrawler | seo | Xovi | XoviOnpageCrawler |
| Xy Archive Compliance | archiver | XY | XY-Archive-Compliance |
| Yacy Crawler | search_engine | Yacy | yacybot |
| Yahoo Ad monitoring | advertising | Yahoo | Yahoo Ad monitoring |
| Yahoo Ad Monitoring | advertising | Yahoo | Yahoo Ad [Mm]onitoring |
| Yahoo Crawler | search_engine | Yahoo | Slurp |
| Yahoo Crawler Japan | search_engine | Yahoo | Y!J |
| Yahoo Mail Proxy | page_preview | Yahoo | YahooMailProxy |
| Yahoo Preview | search_engine | Yahoo | Yahoo Link Preview |
| YahooCacheSystem | seo | Yahoo | YahooCacheSystem |
| Yamanalab Crawler | archiver | Yamanalab | Sonic |
| Yandex Crawler | search_engine | Yandex | yandex\.com\/bots |
| Yandex Crawler Javascript | search_engine | Yandex | YandexRenderResourcesBot\/ |
| Yandori Rss Bot | feed_fetcher | Yandori | YandoriRSSBot |
| Yanga Crawler | seo | Yanga | yanga |
| Yellowbp Crawler | advertising | Yellowbp | Yellowbrandprotectionbot |
| yelpspider | search_engine | Yelp | yelpspider |
| Yext Bot | tool | Yext | YextBot |
| YextBot/ | seo | Yext | YextBot\/ |
| Yisou Crawler | scraping_framework | Yisou | YisouSpider |
| YLT Chrome | monitoring | YLT | YLT Chrome |
| YokoyGroupAG/ | seo | YokoyGroupAG | YokoyGroupAG\/ |
| Yooz Crawler | seo | Yooz | yoozBot |
| YoudaoBot | search_engine | Youdao | YoudaoBot |
| Yuuperbot | seo | Yuuperbot | Yuuperbot |
| Zabbix Monitor | monitoring | Zabbix | Zabbix |
| Zapier | seo | Zapier | Zapier |
| Zeal Bot | search_engine | Zealbot | Zealbot |
| zebra-v2-bot | seo | zebra-v2 | zebra-v2-bot |
| Zendesk Webhook | seo | Zendesk | Zendesk Webhook |
| Zgrab | tool | Zgrab | zgrab |
| ZmEu | security | ZmEu | ZmEu |
| Zombie Bot | seo | Zombiebot | Zombiebot\/ |
| Zoominfo Crawler | seo | Zoominfo | ZoominfoBot |
| ZoomInfo- | search_engine | ZoomInfo | ZoomInfo- |
| ZoteroTranslationServer/WMF | archiver | ZoteroTranslationServer | ZoteroTranslationServer\/WMF |
| Zum Crawler | seo | Zum | ZumBot |
| Zuperlist Crawler | seo | Zuperlist | ZuperlistBot\/ |
| ZyBorg | search_engine | ZyBorg | ZyBorg |
| zzhbot | seo | zzhbot | zzhbot |