vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 4.x Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=245)
-   -   Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)

gigawiz 11-05-2011 12:16 AM

Quote:

Originally Posted by Simon Lloyd (Post 2264480)
Your entire forum folder isn't being given "that sort of access", it's simply one text file all restictions that all your other files have are and have never been unchanged, if i get time (really bogged down with working 2 jobs at the moment) i'll add a custom box so that you can set where the file is written but if you CHMOD that to read only then how can it write to it?

The contents of the output.txt do not give any information about your site that you cannot get from browsing your site or its users, the fact that it's just bot information makesit even less desirable info :)

Glad you like the mod ;)

OK now I feel a right idiot, it never occurred to me to just create the needed file in my forum root directory and CHMOD just the file for read/write access. I can't see the woods for the trees!

On a side note due to me making a slight error while setting up the hack I have threads made by the hack in all sorts of places, the threads don't actually exist and just make the forum look a mess. Any idea on how to remove them? Somebody mentioned about a SQL thing to do but I have no idea about that.

Thanks for the support.

Oh and I forgot to mention that I am running v3.8.5 of vBulletin.

gigawiz.

EDIT - I currently have a specific forum for the threads created by this hack and if I don't put them there then they end up everywhere. How do I set it so as no threads are made at all and just the output.txt file is made?

Simon Lloyd 11-05-2011 06:32 AM

Firstly this is the vb4 thread so specific version questions should be in the thread for that version, however, to clean up just go to admincp>maintainance>Update Counters then update forum information.

To NOT create threads (which of course was my recommendation) then simply uncheck the radio button for "Create Thread" :)

gigawiz 11-05-2011 12:24 PM

Quote:

Originally Posted by Simon Lloyd (Post 2264649)
Firstly this is the vb4 thread so specific version questions should be in the thread for that version, however, to clean up just go to admincp>maintainance>Update Counters then update forum information.

To NOT create threads (which of course was my recommendation) then simply uncheck the radio button for "Create Thread" :)

You sir are a gentleman and a scholar! That cleanup bit did just the trick, sorry for posting in the wrong version thread I will look for the other one? Should I re-post my previous questions over in that thread don't want to be seen as double posting type thing.

Thanks again for your continued support!

gigawiz.

Simon Lloyd 11-05-2011 04:47 PM

gigawiz, no need to post in the other thread now, the mods are the same but different versions can give different erros which is why i have versions of this for vb3.7 and vb3.8

Ath3na 11-06-2011 06:56 PM

Awsome mod, thanks so much for this.

Spent hours trying to get rid of Baiduspider via htaccess and robots.txt then found this.
Twenty minutes after having it turned on no crappy unwanted bots.

Voted for MOTM

One quick question. I turned on the logging to the output.txt file that shout be in my forum root but I didn't see it generated in my httpdocs folder once the bots were removed?

I then turned logging off after just twenty minutes of having the mod installed. Does the log take a while to generate?

Thanks for this mod, it is really helpfull

Simon Lloyd 11-06-2011 07:05 PM

The output.txt is generated as bots found in your list attempt to call a forum or thread, there's no time lag and the file should be created straight away. If you have no cms then the file should be available at www.mysite.com/output.txt if forum is in a folder then something like www.mysite.com/forum/output.txt

Any issues post back and i'll deal with them for you :)

Ath3na 11-06-2011 07:40 PM

Ah ok, I will turn the logging back on and let you know. Should be fine though.

Thanks

bigtree 11-07-2011 03:06 PM

Just installed this, very cool, thank you!
I'm using the full list but I see many are not. I don't care about most Asian traffic. Actually, I only care about the main bots, the rest can go you know where. Is the full list recommended then?


I don't need a log, notifications or to create threads etc. I just want to turn this on and have it work without having to dump logs etc. I've set it to the top 3 and pointing to www.klikhierniet.net Is this enough?

Thanks again!

Simon Lloyd 11-07-2011 05:55 PM

Firstky glad you like it :)

You don't have to have any logging of any sort, that stuff was added by request so folk could monitor things...etc. The FULL list isn't exhaustive and there are many missing off it, denying bots/spiders is a personal thing, just use the names of those that you don't want to see your site (if you are using the full list remove DA and Custo as these may cause issues with real users), remember you are banning bots/spiders by user agent and what you see in WOL isn't necessarily in the UA, if you go to WOL and then chose the option for displaying useragents aswell it will help you.

I personally ban:
Yandex
Yeti
Youdao
Sogou
SoSo
Baidu
spinn3r
psbot
SBIder
exabot
speedy
omgili
wget

Amongst a few others, like i said, you can ban agressively as you like :)
EDIT: You can use the automatic option of redirecting each spider/bot to their own IP address instead of redirecting to a site!

bigtree 11-07-2011 10:26 PM

This is such a great Mod! You are king!

Quote:

RE: You can use the automatic option of redirecting each spider/bot to their own IP address instead of redirecting to a site!
What does the most damage to them without helping the bot to learn from this?


All times are GMT. The time now is 07:44 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01733 seconds
  • Memory Usage 1,747KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (3)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (3)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete