vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 4.x Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=245)
-   -   Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)

Max Taxable 10-31-2012 11:45 PM

Quote:

Originally Posted by simonhind (Post 2377383)
for those who want to block blocks accesses through Facebook external hit

facebookexternalhit/1.0

I don't block that because it's just facebook getting image and text information from a thread or a post someone has posted to facebook. It's self defeating to block this, it's your friend. Same with twitterbot.

Simon Lloyd 11-01-2012 07:39 AM

Quote:

Originally Posted by Max Taxable (Post 2377401)
I have had no problems, Ahrefsbot is in my collection, and I never see it hit my site anymore.

Pretty sure the OP is saying he sees "ahrefsbot" in his WOL after adding it to your mod.

Well if thats the case and its actually in vbulletins standard WOL then it will either disappear after his online timeout setting in admincp or the Ahrefsbot is using a UA that doesn't have Ahrefs in it, if he is using Paul M's Who has visited or a similar mod then it will appear always as both mods are doing their job!, i've mentioned this a few times.

EDIT: this is actually mentioned in the FAQ thats referrenced in the mods description.

Disco_Dave 11-01-2012 08:14 AM

Quote:

Originally Posted by Max Taxable (Post 2377326)
It blocks that spider, how did you add it?

There user agent was comingup as this: choopa: choopa.net: I placed both of these in. They had at least 20 different ip addies.

Simon Lloyd 11-01-2012 08:37 AM

You only need to enter choopa if that displays in their UA and they'll be banned immediately and will disappear from WOL after the WOL timeout, the ip's are of no consequence, i have a ban ip mod but this one ban's the string found in the UA so they can use 100 different ips for the same UA and still they will be banned :)

Disco_Dave 11-01-2012 08:48 AM

That's what I did yesterday morning, but in the afternoon I had around 36 ahrefsbots under the UA Choopa.net I added choopa in the morning to your mod but they where still there in the afternoon that's way I asked. I haven't seen them this morning though..

Simon Lloyd 11-01-2012 09:06 AM

Do you use any mod for who visited statistics? if you dont the only other explanation is that they had already accessed a thread or area prior to you banning them, the only time they can be banned is when they release that thread or area to move to another, after that they're history :)

Disco_Dave 11-01-2012 09:15 AM

I have boofo's mod for displaying spiders. I'm not sure if any spiders can view our site you need to be registered to view any content on our site.

Great mod and thanks for your help..

Simon Lloyd 11-01-2012 09:45 AM

It doesn't matter that they cannot view any content, your thread url's are being indexed which is why you are being crawled, naturally they see the same as guest, just viewed your site you should also turn off displaying WOL for guests, it will save you queries and bandwidth :)

Disco_Dave 11-01-2012 10:00 AM

Cheers Simon :D

The feckers are still getting in: AhrefsBot Spider 02:15 PM / Viewing Index NIRC: 173.199.115.83.choopa.net

Simon Lloyd 11-01-2012 03:05 PM

If thats form the logging then yes you will get that until every thread they have tried to index previously becomes a 301 permanent redirect, if not, if you want to pm me temp admin access with all permissions i'll take a look and see what i can do for you.


All times are GMT. The time now is 08:20 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01928 seconds
  • Memory Usage 1,740KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (3)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (2)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete