vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 4.x Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=245)
-   -   Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)

Max Taxable 03-28-2012 07:50 PM

Quote:

Originally Posted by Alan_SP (Post 2314388)
Thanks for the info. Before I didn't noticed this. I'll wait till it shows again (hopefully never again).

I also installed vB Bad Behavior: https://vborg.vbsupport.ru/showthread.php?t=261498

EDIT: I found this info about it:

Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)

I now use only this string in spider list settings: MJ12bot.

I hope it will stop it, maybe Majestics MJ12bot was too much

On separate lines.

But I am curious - why do you want to ban this bot? It's one of the better behaved ones out there and it doesn't flood. It obeys robots.txt as well.

How can I block MJ12bot?

MJ12bot adheres to the robots.txt standard. If you want the bot to prevent website from being crawled then add the following text to your robots.txt:

User-agent: MJ12bot
Disallow: /

Simon Lloyd 03-28-2012 09:49 PM

Quote:

Originally Posted by Alan_SP (Post 2314388)
I hope it will stop it, maybe Majestics MJ12bot was too much

Yes it was because it looks for the entire string you entered and as that doesn't appear in the UA it doesn't get banned :)

BTW, Max is dead right :)

Alan_SP 03-29-2012 06:01 PM

Because I don't have use of spiders that no one is using. At least I don't know anyone that uses Majestics.

People usually use Google, Bing, Facebook... Other search engines or spiders don't interest me, as they don't interest my users. Same goes for Baidu or similar search engines.

Does people use Majestics? And for what purposes?

Baf_Jams 03-29-2012 09:04 PM

Installed Thanks :)

Max Taxable 03-29-2012 09:36 PM

Quote:

Originally Posted by Alan_SP (Post 2314784)
Because I don't have use of spiders that no one is using. At least I don't know anyone that uses Majestics.

People usually use Google, Bing, Facebook... Other search engines or spiders don't interest me, as they don't interest my users. Same goes for Baidu or similar search engines.

Does people use Majestics? And for what purposes?

The link you posted earlier nicely explains it.

It is a friendly, well behaved bot that helps your presence on the web. Baidu is none of these, it is a aggressive, leeching, unfriendly bot attached to a Chinese search engine. There's no comparison between the two.

Ban whatever bots you like, no one's telling you not to. But MJ12 isn't hurting you, it's helping you and it is friendly.

Alan_SP 03-30-2012 04:43 PM

Quote:

Originally Posted by Max Taxable (Post 2314845)
But MJ12 isn't hurting you, it's helping you and it is friendly.

Thanks for your explanation. It's helpful, not with just this post. :)

S_E_A 04-14-2012 10:33 AM

Thank you for a great mod, Simon.

I want to block Deepnet Explorer Spiders. To do this I enter 'Deepnet Explorer' into the spide list? It's okay to block Deepnet Explorer?

Simon Lloyd 04-14-2012 10:43 AM

Blocking spiders is all about personal choice, do a little research and find out whether you want to cater for that country and whether they add value to your site!, when Deepnet Explorer are visiting go to who's online and at the bottom there's a dropdown box for "Show Useragent?" select Yes, then check out their useragent, you can enter any or all of the UA string, so if they actually do have Deepnet in the UA then you just enter that on its own line in the list :)

Max Taxable 04-14-2012 01:48 PM

My current (updated) list of banned user agents entered into this Mod:

baiduspider
beta.statsit.com
statsit
SiteIntel
Yandex
GomezAgent
FunWebProducts
MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6
Nesotebot
DCPbot
Opera/1
Opera/2
Opera/3
Opera/4
Opera/5
Opera/6
Opera/7
Opera/8
AOL Advertising R&D
DataCha0s
aiHitBot
Apache-HttpClient
Zend_Http_Client
ReverseGet

ForceHSS 04-18-2012 06:32 PM

xpymep.exe
start.exe
I seen these two as hosts so I added them to the list look strange


All times are GMT. The time now is 09:08 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01716 seconds
  • Memory Usage 1,745KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (4)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (3)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete