Should this concern me? (bots)
Ok, I am *very* new to this, so I need some guidance on bots. I did create the robots.txt file, as suggested in another thread. This morning I checked my stats, and saw this:
Robots/Spiders visitors (Top 25) - Full list - Last visit
3 different robots* Hits Bandwidth Last visit
Googlebot 41 239.85 KB 16 Dec 2007 - 15:09
Unknown robot (identified by 'bot/' or 'bot-') 2+1 42.31 KB 17 Dec 2007 - 04:27
Unknown robot (identified by hit on 'robots.txt') 0+1 1.64 KB 16 Dec 2007 - 15:13
I am fine with the googlebot. What do the other 2 lines tell me? Do I need to do something to further restrict access?
Any insight appreciated!
|