vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 4.x Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=245)
-   -   Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)

Simon Lloyd 12-08-2014 06:52 AM

Quote:

Originally Posted by ozzy47 (Post 2526232)
I would not say a conflict, but perhaps a better hook to execute the mod. That is if the testing continues to provide the desired results.

Just bear in mind that any other hook you choose will need to be sufficient to perform the other tasks of the mod if you wished like sending the email or creating the threads. Some of the other runtime hooks will give errors or not work as expected especially with the thread creation, also keep in mind you need to redirect them before anything has loaded as it's this that is the basis of the mod - keeping resources for your members and not the bots :)

CAG CheechDogg 12-08-2014 06:55 AM

Quote:

Originally Posted by Simon Lloyd (Post 2526254)
I agree in part, when you didnt have the block they were calling on every resource...php, mysql, cpu and ram, with the block they pretty much are just using ram as cpu and php time and response is minmal and as you are not loading anything else the ram isn't being maxed either. If you have whole country blocks that doesn't take as much checking as full octet ips like 192.161.0.1, if you have plenty of those then they are checked against each request, if you are blocking just 192.161 then its just one check against each request.

Im probably not explaining myself too well (it reads much better in my head :)).


No my Man, you actually are explaining yourself very well lol ...

All I know is that I have not had any negative effects from doing the blocks and I also have just a list of single IPs ..... and let me tell you , that list is long as hell lol ....

princesspepper 12-08-2014 06:57 AM

Quote:

Originally Posted by Max Taxable (Post 2526230)
Just don't redirect to any of your own pages - feedback loop danger.

EDIT TO ADD: I was right about the hook conflict with some other mod(s) Ozzy?

Thanks, but I'm still unsure what the benefit would be to redirect back to the source. Would it make them aware you don't want them sooner?

Simon Lloyd 12-08-2014 08:42 AM

Quote:

Originally Posted by princesspepper (Post 2526259)
Thanks, but I'm still unsure what the benefit would be to redirect back to the source. Would it make them aware you don't want them sooner?

It really doesn't matter, they are redirected with a 301 which is a permanent redirect, so they will always see the url they tried to crawl as the one you send them to. Like i said, i coded that in to send them back to themselves so they have less resources to be crawling other peoples sites - it's only fair! :)

Alan_SP 12-08-2014 09:17 PM

Quote:

Originally Posted by CAG CheechDogg (Post 2525962)
the best thing to do for me was also add a huge IP block to my htaccess file that completely blocks all of China and a couple other Asian countries from accessing my site ...

Would you share you CIDR list? Not in this thread, but maybe make a new thread?

princesspepper 12-08-2014 10:14 PM

Quote:

Originally Posted by Simon Lloyd (Post 2526269)
i coded that in to send them back to themselves so they have less resources to be crawling other peoples sites - it's only fair! :)

Thanks, that is all I wanted to know. :)

Simon Lloyd 12-09-2014 03:47 AM

Princesspepper could you mark this as installed please :)

CAG CheechDogg 12-09-2014 03:49 AM

Quote:

Originally Posted by Alan_SP (Post 2526371)
Would you share you CIDR list? Not in this thread, but maybe make a new thread?

You can get it here my Man : https://vborg.vbsupport.ru/showthrea...134.184.0%2F21

Gadget_Guy 12-09-2014 06:18 PM

Hey Ozzy,

Are we any closer to an alternative or modification to this so that we can get better blocking in place?

I am willing to test on my site as I am still getting hit hard by spiders even with the mod in place.

D.

Simon Lloyd 12-09-2014 07:33 PM

Hey Gadget Guy, no disrespect but the mod here is mine and isn't marked as reusable code, Ozzy may post what he's tried or done but wont necessarily be added to this mod, however Ozzy has developed one like this with other measures, you can get it at his site.

If you are being hit by spiders with this mod in place it will be because there is an anomaly in your list, this list isn't exhaustive but here's a few reasons why:
Entry in list has a leading or trailing space
Entry has a typo of some sort
Entry doesn't actually represent the bot you think it does (i.e Ahrefsbot I believe has a different name in the UA)
Mod run order may conflict with another mod using the same hook
There are other reasons but those should get you going! :)


All times are GMT. The time now is 04:37 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01833 seconds
  • Memory Usage 1,746KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (7)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (3)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete