Miscellaneous Hacks - Ban Spiders by User Agent

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost....&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost....&postcount=137

How does it work?
https://vborg.vbsupport.ru/showpost....&postcount=381

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost....&postcount=318
https://vborg.vbsupport.ru/showpost....7&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost....&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost....&postcount=259
https://vborg.vbsupport.ru/showpost....&postcount=224
https://vborg.vbsupport.ru/showpost....&postcount=281

Tested on vb3.7.x, vB3.8.x , vB4.x.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

1. Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
2. Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Hook changed extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.1 uploaded, prevented spiders being counted when mod turned off.
17th December 2014 Version 3.1.2 uploaded, due to rogue code from another mod
The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

[Simon Lloyd]

Ok, i've checked and i dont see any of these bots in your native vbulletin WOL, the other mods you have for statistics and total visitors...etc WILL log these as visiting because the bots are directly accessing a url, the logging is done before the url loads completely, my mod also bans them at this point so both mods are working

Just as a note, you're using create a thread, you can quickly get thousands of threads, it's better to use the output.txt logging

Note to all!:
If you have Simon in your ban list this will ban the following:
simon
SimonLloyd
Lloyd simon
thisisanincrediblylongsimonwordhere

Get the idea?, you dont need to add all those to your ban list, simply because the mod looks for the string "simon" (case doesn't matter) in the entire string, so, if you'd used this in your list:
Simon*\Lloyd
It would NOT ban:
Simon
Simon Lloyd
thisissimonlloydinastring
but it WOULD ban
Simon*\Lloyd-in(this.string)
thisstringSimon*\Lloydhere
....etc

Hope you all understand this better now and can get to removing duplicates from your list.

@tricksodave, you can delete the temp account for me now thanks, also if you read the above please prune your list.

If any of you have any trouble with editing your lists let me know and i'll help with anything you're stuck with

[Disco_Dave]

Thanks Simon, That's helped me understand it a bit better. Thanks again...

[TheSupportForum]

Simon Lloyd

i c wat u done there

haha

[CAG CheechDogg]

Simon does this also block Facebook's scrapper? I am getting slammed by Facebook IP's and spiders:

facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

I did it through htaccess but this blocks the ability for me to post any articles to facebook with a thumbnail.

Here is a link: http://www.botopedia.org/user-agent-...k-external-hit

[CAG CheechDogg]

Or is there a way to slow these guys down with crawl-delay like this:

User-Agent: *
Crawl-Delay: 10

I read you should use the agent by name instead of the above, if you know how or does facebook follow the above?

[CAG CheechDogg]

Here is something else on facebook's bot, spiders or what ever they really are. Facebook claims they are not spiders or bots but instead scrapers, but I have been getting 500 server side errors and I check my error logs and during or around the time they are hitting my site over 100 times sometimes within 2 minutes I see Facebook IPs in the error logs....sigh...

Help? lol....

[Simon Lloyd]

Is there only Facebook in the error log? As for banning both or whoever read my post above, as you see it all depends on the UAs of each bot, banning is a personal thing, most both don't recognise the delay command in robots. Maybe look at their ip range and ban some of their ips you can use my other mod for that.

[Simon Lloyd]

CAG you haven't downloaded or marked this as installed!

[CAG CheechDogg]

Hey! I did downloaded but I didn't hit installed! lol...Sorry ...

As for banning the ips I have done that, but that blocks the ability to post the articles with the right info on facebook, so I have to make a decision here on whether facebook will help my site or not.

I was just asking the question really about the crawl-delay which shouldn't have been asked here Simon, I apologize for that.

[Simon Lloyd]

Banning ips are only for incoming unless you've banned them in cpanel or htaccess. As for asking about the delay there's no problem i like to help where i can.