vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 4.x Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=245)
-   -   Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)

Max Taxable 03-14-2012 03:08 PM

Quote:

Originally Posted by meaters (Post 2309374)
Awesome mod, thanks!

Saved our community from Baidu, hundreds of bots were online persistenly to the point of crashing our server.

And with only the addition of, per line:

MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6

You end 99.9% of all spam bot registration attempts and cut garbage traffic even further.

Here's my entire ban list for this Mod:

baiduspider
beta.statsit.com
statsit
SiteIntel
Yandex
GomezAgent
FunWebProducts
MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6
w3m

Simon Lloyd 03-14-2012 03:25 PM

Are you dead sure on those early IE's?

Max, could you mark as installed please?

Max Taxable 03-14-2012 03:39 PM

Quote:

Originally Posted by Simon Lloyd (Post 2309394)
Are you dead sure on those early IE's?

I am dead sure the percentage of human beings still using these dinosaurs is infinitesimally small, so small they're not worth worrying about losing. (None of my 3,200+ users have these, for example)

I am also dead sure that entering these into your Mod doesn't interfere with IE 7,8,9 etc. Tested and verified.

I am also dead sure that the IsBot Mod I have is still working, but that since I put the dinosaur IE's in your Mod - it went from catching 40-50 bot registration attempts per day to catching only one or two!

The early IE's are 99.9% of the spam bot problem on the web, because these are easily infected to become botnet zombies. Human spammers are extremely rare, because think about it - if you have to pay someone to spam it kind of defeats the purpose of spamming.

I used to get 1,500 or so visits a day from these early IE computers, and spent months analyzing them and their origins. Never found one that looked like a Human. It is the 21st Century already, and I think it is high time webmasters not only stopped supporting early IE, but should also take steps to just plain block them. If the FBI and Microsoft really wanted to stop the botnet problem, MS would revoke the registration of these, or automatically upgrade them.

I used to use a script that did just that - would detect early IE and install the latest version of firefox, making it the default browser on that computer - using the same exploits that made them botnet zombies in the first place. I virtually wiped out a entire botnet that way, back in 2006 while one of my sites was undergoing a DDoS attack from one.

Your Mod is by far the best weapon against the botnets yet, and I have been studying them and fighting them for at least 10 years.
Quote:

Max, could you mark as installed please?
I did, on the 3.8.x version I run.

Simon Lloyd 03-14-2012 05:59 PM

:), thanks and thanks! ;)

manning 03-16-2012 02:05 AM

Quote:

Originally Posted by Wayne Luke (Post 2232334)
I banned them at the server level. Not catering to the Chinese or Asian market and never will cater to the Chinese or Asian market so don't need them to index my site.

Interesting idea - my forum really doesnt cater to Asian markets either no Russian or pretty much any place other than USA maybe UK ... What if I add ALLOW for those IPS and deny for everyone else... that makes htaccess huge - what affect will that have on load time? Course if they use a proxy in one of the other locations theyd still get in..... damn idiots!

BadgerDog 03-16-2012 11:04 AM

Just for my clarity ... :)

I still get spiders appearing in PaulM's guest list and I understand from previous posts why. I also still see spiders active in my "Who's On-line" listing, but I understand that doesn't mean they actually are on the site, but have showed and been redirected?

As a test, I turned ON for a few minutes the post in thread option, captured a few posts and then turned it OFF.

Here's a typical thread it started:

Quote:

Activity from Bot No. 7 (Baiduspider) in your list

Date and Time: 03-16-2012 06:57:28
Associated Username (if any): Unregistered
Matched bots[7]: Baiduspider
With User Agent: MOZILLA/5.0 (COMPATIBLE; BAIDUSPIDER/2.0; +HTTP://WWW.BAIDU.COM/SEARCH/SPIDER.HTML)
Does this mean that in fact that the Baidu spider has been caught by this mod and redirected elsewhere? Does it mean that the mod is actually working, in spite of what appears in the "Who's On-line" listing?

Thanks .. :)

Regards,
Doug

Max Taxable 03-16-2012 02:40 PM

BadgerDog that's strange, I never see any of the banned user agents either in who's online or in Paul's Track Guest Visits Mod.

Simon Lloyd 03-16-2012 06:51 PM

Quote:

Originally Posted by BadgerDog (Post 2309990)
Just for my clarity ... :)

I still get spiders appearing in PaulM's guest list and I understand from previous posts why. I also still see spiders active in my "Who's On-line" listing, but I understand that doesn't mean they actually are on the site, but have showed and been redirected?

As a test, I turned ON for a few minutes the post in thread option, captured a few posts and then turned it OFF.

Here's a typical thread it started:



Does this mean that in fact that the Baidu spider has been caught by this mod and redirected elsewhere? Does it mean that the mod is actually working, in spite of what appears in the "Who's On-line" listing?

Thanks .. :)

Regards,
Doug

It may be that the mod is conflicting with some other mod, if you want to pm me admin access with permissions i'll take a look for you :)

baileyjojoms 03-16-2012 08:32 PM

Just a hint to anyone with Baidu Spider issues. This Mod works great, but after getting 30,000 spider bans I had enough. I contact Baidu via their Spider Complaint section on their webpage, and they have halted crawling my site. This request was processed within 3 working days. I haven't seen a hint of Baidu since then.

Simon Lloyd 03-16-2012 09:00 PM

Thats great news, i cnat believe you actually logged all those denials :), great info anyway as Baidu doesn't follow robots.txt (which they claim it does).


All times are GMT. The time now is 08:08 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01577 seconds
  • Memory Usage 1,748KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (6)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (3)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete