A big fraud.. or was I too lucky?
During some debugging, I was checking the access log on my web server.
Accidently, I noticed following (and many other similar) entries. Why would google, yahoo and microsoft all get interested in refreshing their index of my photo album at exactly the same time? Perhaps, some crawler out there is faking its identity.
I tried nslookup for these ips, and the info seem to correspond to microsoft, yahoo and google respectively. My network admin tried to convince me that I was lucky that all 3 search giants are simultaneously interested in me.
Likelihood of such an event is so small.. Assuming the refresh cycle of 30 days, and the scheduling epoch of one hour, the probability that all 3 companies would have scheduled a particular page in the same hour is something like (30*24)^(-3) = 3 out of a billion.
My probability estimation is missing some terms. You need another division by 20 Billion to account for "a particular page", and a multiplication with 30*24 for "the same hour".
I dont believe in so much of coincidence. Either there is a big fraud going on.. or the big giants have some collaborative projects going on..
65.55.209.52 - - [22/Jan/2007:19:15:25 -0600] "GET /~gkabra2/publish/summer2005/navahoPass/slides/P1040659.html HTTP/1.0" 200 14002 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
74.6.86.107 - - [22/Jan/2007:19:29:34 -0600] "GET /~gkabra2/publish/summer2005/helen/slides/IMG_1050.html HTTP/1.0" 200 13570 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp )"
66.249.66.171 - - [22/Jan/2007:19:31:27 -0600] "GET /~gkabra2/publish/summer2005/4thJuly/slides/P1050115.html HTTP/1.1" 304 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html)"
Accidently, I noticed following (and many other similar) entries. Why would google, yahoo and microsoft all get interested in refreshing their index of my photo album at exactly the same time? Perhaps, some crawler out there is faking its identity.
I tried nslookup for these ips, and the info seem to correspond to microsoft, yahoo and google respectively. My network admin tried to convince me that I was lucky that all 3 search giants are simultaneously interested in me.
Likelihood of such an event is so small.. Assuming the refresh cycle of 30 days, and the scheduling epoch of one hour, the probability that all 3 companies would have scheduled a particular page in the same hour is something like (30*24)^(-3) = 3 out of a billion.
My probability estimation is missing some terms. You need another division by 20 Billion to account for "a particular page", and a multiplication with 30*24 for "the same hour".
I dont believe in so much of coincidence. Either there is a big fraud going on.. or the big giants have some collaborative projects going on..
65.55.209.52 - - [22/Jan/2007:19:15:25 -0600] "GET /~gkabra2/publish/summer2005/navahoPass/slides/P1040659.html HTTP/1.0" 200 14002 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
74.6.86.107 - - [22/Jan/2007:19:29:34 -0600] "GET /~gkabra2/publish/summer2005/helen/slides/IMG_1050.html HTTP/1.0" 200 13570 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us
66.249.66.171 - - [22/Jan/2007:19:31:27 -0600] "GET /~gkabra2/publish/summer2005/4thJuly/slides/P1050115.html HTTP/1.1" 304 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot
posted by Govind Kabra at 11:57 PM
1 Comments:
I think its more about competition. Maybe once you update your page, a message is sent to these search engines to update.
All of them want to be in on the game as quickly as possible. Imagine if it were a news event or something like that.
Post a Comment
<< Home