Log in

View Full Version : Should this concern me? (bots)


dawn100
12-17-2007, 02:45 PM
Ok, I am *very* new to this, so I need some guidance on bots. I did create the robots.txt file, as suggested in another thread. This morning I checked my stats, and saw this:

Robots/Spiders visitors (Top 25) - Full list - Last visit
3 different robots* Hits Bandwidth Last visit
Googlebot 41 239.85 KB 16 Dec 2007 - 15:09
Unknown robot (identified by 'bot/' or 'bot-') 2+1 42.31 KB 17 Dec 2007 - 04:27
Unknown robot (identified by hit on 'robots.txt') 0+1 1.64 KB 16 Dec 2007 - 15:13

I am fine with the googlebot. What do the other 2 lines tell me? Do I need to do something to further restrict access?

Any insight appreciated!

pcoskat
12-17-2007, 02:49 PM
You might want to also check vbulletin.com's board for posts on bots. There are some great threads on it. (especially check out the ones on the Yahoo bots)