Site Slow
Posted on Sunday, November 14, 2004
I don’t know if enough people read this to care BUT – my site has been extremely slow all day and I just figured out why. It seems that all the major search engines have decided to crawl it AT THE SAME TIME. Checking into my logs look who has been requesting everything:
Googlebot/2.1 (+http://www.google.com/bot.html) msnbot/0.3 (+http://search.msn.com/msnbot.htm) Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; iebar; .NET CLR 1.1.4322) SurveyBot/2.3 (Whois Source) Jetbot/1.0
These guys along with a bunch of news aggregators have been demolishing my web server since about midnight last night.