Miscellaneous Hacks - Ban Spiders by User Agent

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost....&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost....&postcount=137

How does it work?
https://vborg.vbsupport.ru/showpost....&postcount=381

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost....&postcount=318
https://vborg.vbsupport.ru/showpost....7&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost....&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost....&postcount=259
https://vborg.vbsupport.ru/showpost....&postcount=224
https://vborg.vbsupport.ru/showpost....&postcount=281

Tested on vb3.7.x, vB3.8.x , vB4.x.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

1. Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
2. Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Hook changed extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.1 uploaded, prevented spiders being counted when mod turned off.
17th December 2014 Version 3.1.2 uploaded, due to rogue code from another mod
The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

[ForceHSS]

Aboundex/0.2
seems to be a new one here is the full thing Aboundex/0.2 (http://www.aboundex.com/crawler/)
ip is 173.193.219.168-static.reverse.softlayer.com
I have checked the ip and it has come back that a spam bot is using it

If someone wants to run checks see if it needs added to the list. I am not 100% sure if it is this is why it needs checked first

[CAG CheechDogg]

Originally Posted by Simon Lloyd

Banning ips are only for incoming unless you've banned them in cpanel or htaccess. As for asking about the delay there's no problem i like to help where i can.

Yeah I used htaccess to ban them completely. I need to find out exactly what IPs I can band and still allow facebook to work properly when I post links to articles or posts...sigh...lol

But hanks for understanding and helping out, it is very much appreciated Simon

[Max Taxable]

Originally Posted by CAG CheechDogg

Yeah I used htaccess to ban them completely. I need to find out exactly what IPs I can band and still allow facebook to work properly when I post links to articles or posts...sigh...lol

But hanks for understanding and helping out, it is very much appreciated Simon

In my experience, that's the only time the FB external hit bot comes to your site - when you or someone else posts a link to your site, on facebook. It's your friend. Same with twitterbot and all of its affiliates. I don't mess with those at all.

[CAG CheechDogg]

Originally Posted by Max Taxable

In my experience, that's the only time the FB external hit bot comes to your site - when you or someone else posts a link to your site, on facebook. It's your friend. Same with twitterbot and all of its affiliates. I don't mess with those at all.

Max it's weird because I have the facebook like buttons off on my forums. I do have rssgraffiti but I don't see why that would be hitting pages like the mood and status module and other unrelated pages.

[Max Taxable]

Originally Posted by CAG CheechDogg

Max it's weird because I have the facebook like buttons off on my forums. I do have rssgraffiti but I don't see why that would be hitting pages like the mood and status module and other unrelated pages.

Some autospam bots do spoof their user agents as facebook or even googlebot.

[CAG CheechDogg]

Originally Posted by Max Taxable

Some autospam bots do spoof their user agents as facebook or even googlebot.

Great! now you tell me ! lol...Thanks again Max I will have to take a careful look at the IPs and try to see if they match facebooks then.

[Max Taxable]

From what I've seen over the years facebook's bots have good behavior and only come to see you when something is posted there, from your site. Then they don't crawl around and they SURE don't go anywhere suspicious.

[CAG CheechDogg]

Yeah, nothing suspicious about facebook's crawlers, scrapers or bots , what ever they are. But it has caused my forums to pop the 500 internal server error a bunch of times , I check around the time those errors happen and are reported to me and its facebook's ips around the times of the 500 errors.

[CAG CheechDogg]

Well I decided to completely deny Facebook crawlers, scrapers, spiders or bots to crawl my site.

I deleted all their active sessions from my database through phpMyAdmin and added "facebook" to the list and I haven't gotten one single facebook critter on my site since.

Sucks because I can no longer share anything on facebook from my forums but I just had to do it. Facebook wont reply and doesn't seem to care about eating up bandwidth with their crawlers.

Oh well.

[TheSupportForum]

Originally Posted by CAG CheechDogg

Well I decided to completely deny Facebook crawlers, scrapers, spiders or bots to crawl my site.

I deleted all their active sessions from my database through phpMyAdmin and added "facebook" to the list and I haven't gotten one single facebook critter on my site since.

Sucks because I can no longer share anything on facebook from my forums but I just had to do it. Facebook wont reply and doesn't seem to care about eating up bandwidth with their crawlers.

Oh well.

a wise choice if that's happening to you, as they will eat up your bandwidth