Sat, 01/03/2008 - 01:52 — peterv

What a difference ... While between 11:00 am 23:00 today. Googlebot visited all 923 individual job openings for Leuven (on http://allejobsinleuven.be) and all list pages, the other spiders where very lame ...

Compare this:

~/logs$ grep -v javascript nginx-access.log | grep -i bot | grep -i google | wc -l
1188
~/logs$ grep -v javascript nginx-access.log | grep -i Yahoo | wc -l
23
~/logs$ grep -v javascript nginx-access.log | grep -i EnaBot | wc -l
8
~/logs$ grep -v javascript nginx-access.log | grep -i msn | wc -l
4
~/logs$ grep -v javascript nginx-access.log | grep -i WebAlta | wc -l
4

Maybe the other spiders will come too (with a delay ?).