vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 4.x Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=245)
-   -   Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)

Simon Lloyd 12-29-2011 01:18 PM

Well i suppose the secondd of those two links has more explanation but i didn't get that from your post, i thought you wanted links to lists posting, but for my money that can be a bad thing, people would then blindly enter all the bots to the banning list when really, dependant on their content, they will want some of those scraping or visiting their site, it's a personal preference really.

I built this because the bots were killing my bandwidth and making the forum slow so i ban the more agressive ones like Baidu, why on earth would they need 215 bots indexing my site?, so ban them in favour for the less aggressive :)

Anyway as an information post this link explains what they are http://en.wikipedia.org/wiki/Spambot

Max Taxable 12-29-2011 01:49 PM

Quote:

Originally Posted by Simon Lloyd (Post 2281916)
i ban the more agressive ones like Baidu, why on earth would they need 215 bots indexing my site?

Yeah, 215 Chinese bots leeching resources, to allegedly index your site for people who most likely can never see it. Most US sites are blocked in China. Baidu makes no sense at all, it certainly doesn't help anybody. By far the worst behaving bot out there, it totally ignores robots.txt.

ForceHSS 12-30-2011 03:39 PM

Quote:

Originally Posted by Boofo (Post 2281437)
A link to those lists would be a good addition, also.

https://vborg.vbsupport.ru/showpost....&postcount=224

Simon Lloyd 12-30-2011 03:42 PM

Lol, thanks Force ;)

spillage 01-22-2012 08:02 PM

Since upgrading to vB4.1.10, I've noticed (occasionally) some spiders in the list still showing up online.
Anyone else having this issue, and any ideas what's going on?

TiA

Simon Lloyd 01-22-2012 08:53 PM

You shouldn't unless they are using a different UA, the spiders xml that you can download from Mosh's site (vb.com thread) may have them in the list with the same name but their UA may be different to that in the list :)

spillage 01-22-2012 09:32 PM

A couple of examples of ones in my list that (recently) show as online;
AhrefsBot
Exabot
gigabaz
Heritrix
Majestics MJ12bot

I though this hack picked up on all things similar (name based)?... ie "Majestics" should also take out "Majestics MJ12bot", regardless of the software they're using to access the site.

How do we include same name spiders in our list that are using different UA's?

Simon Lloyd 01-23-2012 04:40 AM

You are right entering Exabot should prevent that bot viewing your site, however entering "Majestics MJ12bot" will not stop a bot with either "Majestics" or "MJ12bot" it will only stop bots with that entire phrase as part of their UA, so if you want to ban that bot enter the seperated values.

Can you give me a link to your site?

spillage 01-24-2012 02:07 AM

Simon, I thought about that and included the individual entries when the issue first arose.

nscale.net ... however, the WGO block is only visible to members... PM me if you want temporary access.

TombstoneWarrior 01-24-2012 10:30 AM

what does this option mean? " Create new thread for each UA detection (be aware that this can cause hundreds of threads at first until spiders get the mesage!)
You can use this even if you aren't using the ban option above" also what does this option do. will it creat a thread in my forum?

ALSO WHY IN A DISCRIPTION FROMA MEMBER DO THEY HAVE THIS OPTION SET TO YES??


All times are GMT. The time now is 03:47 AM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01863 seconds
  • Memory Usage 1,737KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (2)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (3)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete