[Max Taxable]

Originally Posted by pitzerwm

Since I have tried to block all bots with robots.txt and obviously not working, is there another way to block this?

You don't want to block all bots, most of them are your friends. Like Google, Yahoo, MSN and etc, these obey robots.txt. You just want to do away with the bad actors. Like Baidu, for one.

Robots.txt is merely instructions for the bots, it doesn't block anything.

Here's a Mod that lets you block spiders by user agent:

https://vborg.vbsupport.ru/showthread.php?t=264932

Here's advanced spider detection, and guest visitor tracking - identifies spiders for you in online.php and also tracks all guests visits. Very handy:

https://vborg.vbsupport.ru/showthread.php?t=201214

--------------- Added [DATE]1326398877[/DATE] at [TIME]1326398877[/TIME] ---------------

Originally Posted by pitzerwm

Is there a way to block these? I would think that CF would do this automatically.

No, in CF it is blocking and challenging known bad actors, using the Project Honey Pot database as reference. For custom blocking, you can enter that information by IP, IP range, or even by country.

"China" is the first one you need to block, this wipes out Baidu for ya.