Search Tech Blog

User-Agents of the Top 10 Web-Crawler

U

There are thousends of bots and web crawlers working the internet but below is my list of the 10 popular search engines user-agents.

If you browse the logfiles of your website, you will always see the access to a file called “robots.txt”. These are usually calls from search engines. Their web crawlers with there user-agents that read the robots.txt file (hopefully you have one). They check if a visit is allowed, which folders are not allowed and which delay is desired after each page call.

Here is a list of all user agents for the major, leading search engines. I often use this information to analyze my log files, so I thought it would be useful to publish the information online for the benefit of others. It can be useful to have user agents for these popular bots in one place. Each search engine contains references and a list of the most common user agents.

Search Engine Bot Names

  1. Google = Googlebot
  2. Bing/MSN = Bingbot
  3. Yahoo = Slurp
  4. DuckDuckGo = DuckDuckBot
  5. Baidu = Baiduspider
  6. Yandex = YandexBot
  7. Sogou = Sogou
  8. Exalead = Exabot
  9. Facebook = facebot
  10. Alexa = ia_archiver

Full User-Agent Strings

1
2
3
4
5
6
7
8
9
10
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Sogou ... spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
GraphicsSC / Pixabay

About the author

I. Gaffling

I would like to introduce myself, my name is Igor Gaffling, I was born in 1968 and have more than 30 years of experience in the IT- and new-media industry. In this blog I write about how search engines work, facts, ideas, code experiments and the possibility to develop a simple search engine from scratch that can handle a few million entries at an acceptable speed.

Add comment

Search Tech Blog

Latest posts

Latest comments

Categories

Tag Cloud