vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 4.x Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=245)
-   -   Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)

Simon Lloyd 09-19-2011 07:05 AM

Your second UA can access your site (i just used the UA simulator from the link i posted above)

Simon Lloyd 09-19-2011 07:14 AM

post just to get to page 5 as this thread appears to be faulty!

niteflyer32 09-19-2011 07:44 AM

Thanks for the help on the UA list.

When I tested the 2nd UA I listed above with your UA test website http://www.botsvsbrowsers.com/SimulateUserAgent.asp I see our forum but I also see a "Request Status: 500 : Internal Server Error" at the top. Is that caused by our US block list or is that a server setting issue I need to talk to our webhost about? I also get a "500 : Internal Server Error" when trying to verify we have our Comscore analytic code (like Google Analytics) inserted on the forum.

What is the status of the spiders Majestics MJ12bot Spider, Speedy Spider and Voila Spider? I've searched them and get info that is dated or conflicting on if they are good or bad. They are hitting our site pretty often.

The page 5 weirdness of this thread you're seeing appears to be okay on my end.

BadgerDog 09-19-2011 11:51 AM

1 Attachment(s)
Installed with thanks .. :)

Unfortunately, I'm still getting Baidu spiders, even 4 days after installing this.

Attached is screen pic. What am I doing wrong?

Regards,
Doug

Simon Lloyd 09-19-2011 12:59 PM

For now there is an issue when banning bots and having one of the notification enabled (either create thread or Output.txt file on server), so if you have those enabled please disable them, to ban bots for now you must only have the mod activated and ban bots in list selected (and of course bots that you want to ban, do that and all will be good :)

You will recieve notice when i solve the "working together" issue.

Simon Lloyd 09-19-2011 01:04 PM

@niteflyer32, try turning the mod off then trying the UA at the UA simulation site and see what it returns, the mod shouldn't cause a 500 error and then show you the site, if you don't have access you simply get redirected so you wouldn't see your site.

I haven't researched the spiders, i built this mod to cut down on my server load as Baidu were hammering it, i have between 200 and 350 Baidu at my site at any one time, and because the index so vigourously their demand on the server is huge (although while im working on the issues with the notification i am allowing all bots at the moment).

BadgerDog 09-19-2011 01:48 PM

Quote:

Originally Posted by Simon Lloyd (Post 2247842)
For now there is an issue when banning bots and having one of the notification enabled (either create thread or Output.txt file on server), so if you have those enabled please disable them, to ban bots for now you must only have the mod activated and ban bots in list selected (and of course bots that you want to ban, do that and all will be good :)

You will recieve notice when i solve the "working together" issue.

Thanks .. :)

I've turned OFF logging (email notifications were already off) and I'll monitor it now ...

Regards,
Doug

BadgerDog 09-22-2011 10:09 AM

Quote:

Originally Posted by BadgerDog (Post 2247868)
Thanks .. :)

I've turned OFF logging (email notifications were already off) and I'll monitor it now ...

Regards,
Doug

That doesn't work either.... :confused:

Still getting lots of Baidu and Yandex spiders ...

I'm not sure this mod is working at all, regardless of any options set, or turned ON or OFF ... ;)

Regards,
Doug

smirkley 09-22-2011 04:08 PM

Still testing but I can say so far,... NICE !!

Thank you.

I am only banning 4 useragnts at the moment, but I wish to ask is there a condensed version of 'must ban' useragents off that list here, as compared to the whole list? I dont want to go crazy and ban too much especially if it hurts my membership or adsense rev.

So far I ban:

Baidu
Yeti
Twiceler
Yandex

Simon Lloyd 09-22-2011 04:52 PM

Quote:

Originally Posted by BadgerDog (Post 2248891)
That doesn't work either.... :confused:

Still getting lots of Baidu and Yandex spiders ...

I'm not sure this mod is working at all, regardless of any options set, or turned ON or OFF ... ;)

Regards,
Doug

if you want to pm me admin access details and url i'll take a look :)


All times are GMT. The time now is 01:50 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01613 seconds
  • Memory Usage 1,741KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (3)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (2)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete