Home » dmyalktq » On the major search engine spiders those things

On the major search engine spiders those things

do SEO every day to deal with most of the search engine is put out of the small robot, also known as search engine spiders, we have to do is to wait for them to be good, according to their intention to vote good.

a list of the names of the major search engine spiders

Google: Googlebot, Baidu spider spider: Baiduspider, Sogou: spider, soso Sogou spider spider: Sosospider, Yahoo: slurp, Alexa spider spider: ia_archiver, MSN: MSNBot, Altavista spider spider: scooter, Lycos: lycos_spider_ (T-Rex), spider spider: fast-webcrawler inktomi, AlltheWeb slurp, a spider spider:: YodaoBot hot spot: Adminrtspider and OutfoxBot, the spider. Of course, this is just a list of symbols, even if we do not see the analysis of the log is not too strange, or you want to ban them crawl your site is also possible.

two, a sentence summed up the habit of spiders

Google spider: do not love to climb, but love included.

Baidu spider: crawling cautious, included more cautious.

search spider: love climbing pictures, often around to the dynamic address.

YAHOO spider: abide by the rules, every time is to climb robots.txt.

other do not pay too much attention, not much to say.

three, support for robots.txt

analysis may not be realistic, here take Disallow: / *? * (Forbidden dynamic web pages) analysis.

Google performance: written ban will no longer climb, it will be listed in the Google webmaster tools it wants to climb you blocked, as follows:


Baidu’s performance: written after the ban rarely climb, but occasionally will climb, I believe it is working, because less and less, a few times a day before a few days now.

Sogou spider: can be said to be basically not obedient, also do not know not to eat this rule, said it was not eat it will also eat a little, but the dynamic address question, and then according to climb, a climb is large, it doesn’t know it can climb out of what, as shown below:


search with YAHOO spider spider like almost, the feeling was quite effective, prohibited "

Leave a Reply

Your email address will not be published. Required fields are marked *