vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 3.8 Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=235)
-   -   Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=264932)

accludetuner 12-12-2011 08:53 AM

Trying the ban IP mod now.

On a side note, I help several people run other VB sites and I've been installing this spider ban mod as a default thing on all of their sites and putting in my list of user agents to ban/block. I recently installed it on a 3.6 site and it doesn't seem to be working. Is there a different hook I need to use for it to work with 3.6? Thought you might know.

EDIT: Maybe it is working and I spoke too soon. I have that pesky Baidu spider banned due to the amount of indexing it does without any regard to server resources. There were about 50 Baidu bots crawling the site but the number seems to be gradually going down (at about 20 right now). I'll keep checking it to see if Baidu disappears for good and then I'll know for sure whether it's working on 3.6 or not.

lgpaul 01-10-2012 12:18 PM

It works! Thanks!

pitzerwm 01-13-2012 05:27 PM

Thank you for creating a great mod, and thanks to Max for turning me on to it.

A question: in Who is online, in the IP column, right now I have 2 guest where the IP is totally blank. I suppose that there is an explanation for this, but is there anything that you can do about it?

Midohash 01-17-2012 12:53 AM

Fantastic ... marked as Installed ... many thanks
Just a little query! ... If I need to block unwanted visitor or member, should I put his/her IP in the spider list? ... And what is the difference between denying the IP address on cpanel or .htaccess? ... Also can I block a proxy server through this mod?

Simon Lloyd 01-17-2012 07:04 AM

For IP's you should use my other mod https://vborg.vbsupport.ru/showthread.php?t=268146 as this mod processes the useragent string and not the IP address, if you deny in cpanel or .htaccess then the user of that IP will never get to see your site or files, my mod redirects them before your site loads properly. Also with cpanel and .htaccess you have to of course log in and edit them each time you want to ban or remove a ban, mines a little more user friendly :)

Simon Lloyd 01-17-2012 07:07 AM

Quote:

Originally Posted by pitzerwm (Post 2287361)
Thank you for creating a great mod, and thanks to Max for turning me on to it.

A question: in Who is online, in the IP column, right now I have 2 guest where the IP is totally blank. I suppose that there is an explanation for this, but is there anything that you can do about it?

Sorry, again you need this https://vborg.vbsupport.ru/showthread.php?t=268146 for IP's, this deals with the user agent string not IP's, if their IP is blank then they are cloaking it somehow, if you choose to see the User Agent (via the dropdown at the bottom of the WHo's Online window you can copy their user agent string, pop it in the list to ban and then watch them disappear :)

Midohash 01-17-2012 09:02 AM

Quote:

Originally Posted by Simon Lloyd (Post 2288886)
For IP's you should use my other mod https://vborg.vbsupport.ru/showthread.php?t=268146 as this mod processes the useragent string and not the IP address, if you deny in cpanel or .htaccess then the user of that IP will never get to see your site or files, my mod redirects them before your site loads properly. Also with cpanel and .htaccess you have to of course log in and edit them each time you want to ban or remove a ban, mines a little more user friendly :)

Thanks Simon :up:, regarding your other mod is it possible to be modified in the future to block users from anonymous proxy? :confused: ... That will be a great add on :cool: ... However I note a significant drop in the number of my site visitors after installing the mod, is that attributed to blocking bad bots only or could be also some normal visitors are not able now to access my forum? ... Also is the drop in the visitor numbers would affect my over all alexa rank? ... Regards,

pitzerwm 01-17-2012 06:50 PM

I installed this and with Ban IPs apparently have removed a large percentage of my issues.

I was wondering if I get this as the "guest" dhcp-0-24-b2-58-41-4a.cpe.eaglecable.net

Can I put this total "agent" in, and it will ban this person?

Thanks

photonetau 01-17-2012 09:16 PM

I installed this about 9 hours ago and baidu is still hitting at the rate of 200 hits an hour is that usual , how long before they give up?
BTW they are redirected to the default site.

Simon Lloyd 01-18-2012 09:15 AM

Quote:

Originally Posted by Midohash (Post 2288919)
Thanks Simon :up:, regarding your other mod is it possible to be modified in the future to block users from anonymous proxy? :confused: ... That will be a great add on :cool: ... However I note a significant drop in the number of my site visitors after installing the mod, is that attributed to blocking bad bots only or could be also some normal visitors are not able now to access my forum? ... Also is the drop in the visitor numbers would affect my over all alexa rank? ... Regards,

It could be modified to ban those without IP and this one could be modified to ban those without user agent but would need a lot of testing!

Yes, your visitor drop is due tothe banning, you'll find (especially if you installed the spiders on forumhome mod) that most of the visitors were bots in the list like Baiduspider, they're insatiable. Bots don't make up part of your alexa ranking, believe it or not Alexa makes it's assumptions from the amount of visitors to your site that have the Alexa toolbar installed :)

Quote:

Originally Posted by pitzerwm (Post 2289106)
I installed this and with Ban IPs apparently have removed a large percentage of my issues.

I was wondering if I get this as the "guest" dhcp-0-24-b2-58-41-4a.cpe.eaglecable.net

Can I put this total "agent" in, and it will ban this person?

Thanks

Yes of course, the exact agent string will help zero in on that person/bot...etc, i added partial matching because of bots changing their UAs..etc but this mod was always built on using the entire string :)

Quote:

Originally Posted by photonetau (Post 2289142)
I installed this about 9 hours ago and baidu is still hitting at the rate of 200 hits an hour is that usual , how long before they give up?
BTW they are redirected to the default site.

I guess you are still seeing Baidu because you are using Paul M's who visited mod, both his mod and mine are doing their job, his picks them up because they make a direct call to a thread or post...etc and get logged but as soon as they attempt to access the thread or post they are redirected to whichever way you choose with a 301 redirect header :)

If you are seeing Baiduspider and not using Paul's mod then pm me your site url and access details and we'll get you sorted!


All times are GMT. The time now is 06:48 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01257 seconds
  • Memory Usage 1,751KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (5)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (1)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete