Robots.txt works only for those bots that respect it, and identifying every single bot to put into a robots.txt can also be time consuming. There is also a performance overhead associated with robots.txt. I block all my bad bots directly in the webserver, but all the "good bots" to crawl my site.
However, everything that I've written above (and your question) relating to blocking bots from the site.
This mod simply removes the spiders from the totals, allowing you to get a better idea of how many "real" guests you have on your forum. If you keep statistics and graphs as I do this gives you a clearer picture of growth, etc.
It also saves questions from users saying "Wow, there were 800 guests online last night, this forum is really popular".
|