blob: ec366ad9e897e539f04ec8bd478610a95eee70cd (
plain) (
blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
|
# OpenAI
+https://openai.com/gptbot
+https://openai.com/bot
OAI-SearchBot/
# Claude
+claudebot@anthropic.com
ClaudeBot/
# Facebook
facebookexternalhit/
facebookcatalog/
meta-externalagent/
meta-externalfetcher/
# Other (by linked help domain)
+http://yandex.com
+https://developer.amazon.com
+https://search.brave.com
+https://opensiteexplorer.org
+https://awario.com/
+https://help.seranking.com/en/blex-crawler
+https://app.twingly.com/public-docs/crawler
+http://duckduckgo.com/duckassistbot.html
mailto:fruitsaladisland@gmail.com
# Other (mozilla/5.0)
Mozilla/5.0 zgrab/
Mozilla/5.0; Keydrop.io/
"'Mozilla/5.0 (compatible; GenomeCrawlerd/1.0; +https://www.nokia.com/genomecrawler)'"
"Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
"Mozilla/5.0 (compatible; Odin; https://docs.getodin.com/)"
# "Security scanners"
AliyunSecBot/Aliyun (AliyunSecBot@service.alibaba.com)
FH Muenster/Security-Scanner/fh-muenster.de
l9explore/1.2.2
abuse.xmco.fr"
http://mj12bot.com/
SeznamBot/
SemanticScholarBot
NetcraftSurveyAgent
ALittle Client
scaninfo@paloaltonetworks.com
Scrapy/
CensysInspect/
ModatScanner/
AppleBot
AhrefsBot
# Unknown and nondescript
"link_checker/0.1.0"
"SuperFastScraper/
ThinkBot/
Thinkbot/
oii-research/
|
