vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 3.8 Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=235)
-   -   Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=264932)

Dan49 07-24-2013 01:19 PM

What does MSIE 7 stand for?
What about Mozilla/4.0?

Is there a list somewhere that explains the meaning?

Simon Lloyd 07-24-2013 02:39 PM

Microsoft Internet Explorer
Mozilla is firefox, use google to find out what these terms are if you are unsure, there's not a list that I know of or have even searched for but you can use the tools I've shown in the first post to help you find some info around these :)

Max Taxable 07-25-2013 06:49 PM

Quote:

Originally Posted by Simon Lloyd (Post 2435071)
I know I sound like a broken record but it really is a personal choice about who you want to block, less fortunate countries will have older browsers and it's probably safe to block up to MSIE6, later browsers will still be in use, so you really need to check your demographics and make sure you are catering for your intended audience.

Sorry but that's the best advice I can give.

That's the main point really.

Max Taxable 07-25-2013 06:51 PM

Quote:

Originally Posted by Dan49 (Post 2435103)
What does MSIE 7 stand for?
What about Mozilla/4.0?

Is there a list somewhere that explains the meaning?

Do NOT enter the term "Mozilla" into this mod! You will be blocking just about every windows computer on the planet if you do!

Here's a fairly typical Windows computer user agent string:

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727

ALL windows computer user agent strings are going to have the word "Mozilla" in them no matter what browser the computer actually uses.

Dan49 07-25-2013 07:08 PM

Entering Mozilla/4.0 is different than entering Mozilla correct? If so, is blocking Mozilla/4.0 unreasonable? All the Russian IPs are using it and no one else.

Max Taxable 07-25-2013 07:22 PM

Quote:

Originally Posted by Dan49 (Post 2435367)
Entering Mozilla/4.0 is different than entering Mozilla correct? If so, is blocking Mozilla/4.0 unreasonable? All the Russian IPs are using it and no one else.

NO Mozilla AT ALL!

Most ALL windows computers gonna have that "Mozilla4.0" yours probably does too! Have you studied your OWN user agent string?

Dan49 07-25-2013 11:49 PM

But Simon's original list of UA that comes with the mod contains:

Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler

And my understanding is that will only block those particular strings? It looks like this is true because they're on my list and there are plenty of Mozilla/5.0 and Mozilla/4.0 in my forum.

So if I understand correctly; adding Mozilla/4.0 (which I will not do as per your recommendation) will not affect Mozilla /5.0?

Max Taxable 07-25-2013 11:52 PM

Quote:

Originally Posted by Dan49 (Post 2435390)
But Simon's original list of UA that comes with the mod contains:

Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler

And my understanding is that will only block those particular strings? It looks like this is true because there are plenty of Mozilla/5.0 and Mozilla/4.0 in my forum.

I never saw that on his list or any other list, but yes as long as "mozilla" isn't on a line by itself you got no problems.

But what i am telling you is, blocking "Mozilla/4.0" gonna block billions of computers, not just the Russian ones.

Find something unique in their strings to block, if you want to use this to block them. You might not find any such.

Dan49 07-26-2013 12:32 AM

Understood, thanks.

Dan49 07-26-2013 04:03 AM

Max, you may be interested in this. I had a legit user blocked out of my forum. I checked and rechecked his UA and could not figure out why he was blocked
Code:

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)
Of course turning off the mod unblocked him, so I knew it had to be something.

On a hunch I looked at the MSIE entries I copied from your old post:

MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6

I realized that MSIE 1 is the culprit being the first part of MSIE 10.0;

Have you since removed MSIE 1 from your list?

Is there another way to make MSIE 1 work without affecting MSIE 10? As well as the other MSIE, like MSIE 2 for when IE gets to version 20 :)?

Simon Lloyd 07-26-2013 05:45 AM

I doubt there's a pc on the planet that still uses MSIE 1,2 or 3 :)

Remember this mod was built to ban spiders or bots which usually display their bot name in their UA, using the mod outside of that scope is bound to take you in to some grey areas, make sure you use the tools shown in the original post of this thread to check new UA's before you ban them.

Dan49 07-26-2013 11:24 AM

Is there a way to block MSIE 1 without affecting MSIE 10?

Simon Lloyd 07-26-2013 02:19 PM

All you can do is add a space after it, but again all it will do is look for MSIE 1_ where _ is the space (it doesn't physically look like _ that was for illustration purposes), but if it doesn't appear in the UA with a space after it, it wont ban it.

Dan49 07-26-2013 02:56 PM

Thanks.

Another general question. I understand that malicious bots can fake the UA. Why don't they all fake it to display as Google? This way anybody would be hesitant to block them? BTW I do use your IP ban mod also.

Max Taxable 07-26-2013 03:55 PM

Quote:

Originally Posted by Dan49 (Post 2435415)
Max, you may be interested in this. I had a legit user blocked out of my forum. I checked and rechecked his UA and could not figure out why he was blocked
Code:

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)
Of course turning off the mod unblocked him, so I knew it had to be something.

On a hunch I looked at the MSIE entries I copied from your old post:

MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6

I realized that MSIE 1 is the culprit being the first part of MSIE 10.0;

Have you since removed MSIE 1 from your list?

Is there another way to make MSIE 1 work without affecting MSIE 10? As well as the other MSIE, like MSIE 2 for when IE gets to version 20 :)?

Yes this was covered earlier in the thread, and have since removed MSIE 1 from my list. This was months ago. Like Simon said, the number of surviving computers on the planet still using MSIE1 is so tiny, it's not worth blocking.

Max Taxable 07-26-2013 03:57 PM

Quote:

Originally Posted by Dan49 (Post 2435482)
Thanks.

Another general question. I understand that malicious bots can fake the UA. Why don't they all fake it to display as Google? This way anybody would be hesitant to block them? BTW I do use your IP ban mod also.

Some of them do so, but it's very easy to spot those fakes since Google only uses one or two IP addresses. I am pretty sure there's a Mod for blocking spoofed google UA's somewhere.

Dan49 07-26-2013 07:42 PM

I found this information about verifying Google bot. And used this tool http://ipadmin.junkemailfilter.com/rdns.php. Is this the best way? Or is there a list of the IP addresses google uses?

I'd appreciate a link to the mod you mentioned, I searched and can't find it.

valdet 07-26-2013 08:46 PM

Hi Simon,

Hope you got your forum problems sorted :)

I noticed that the mod will not create new threads on private forums, where only admins can post/read.
I am using my admin account as thread creator, so it has permissions to create threads. The threads are created only on public forums where other members and guests can see them, which of course isn't nice.

The output.txt file writing is also not logging anything.

But from my server logs, I see this mod has been doing incredibly well in blocking some nasty spiders (mainly Magpie and Spinn3r).

For those who don't check their server logs, I would advise to check them and see if Magpie is sucking up to 40% of bandwidth as it was until recently in one of my sites.

Max Taxable 07-26-2013 08:48 PM

Quote:

Originally Posted by Dan49 (Post 2435539)
I found this information about verifying Google bot. And used this tool http://ipadmin.junkemailfilter.com/rdns.php. Is this the best way? Or is there a list of the IP addresses google uses?

I'd appreciate a link to the mod you mentioned, I searched and can't find it.

I only vaguely remember seeing such, and it was sometime back. And I might even be wrong.

Simon Lloyd 07-26-2013 09:15 PM

Quote:

Originally Posted by valdet (Post 2435544)
Hi Simon,

Hope you got your forum problems sorted :)

I noticed that the mod will not create new threads on private forums, where only admins can post/read.
I am using my admin account as thread creator, so it has permissions to create threads. The threads are created only on public forums where other members and guests can see them, which of course isn't nice.

The output.txt file writing is also not logging anything.

But from my server logs, I see this mod has been doing incredibly well in blocking some nasty spiders (mainly Magpie and Spinn3r).

For those who don't check their server logs, I would advise to check them and see if Magpie is sucking up to 40% of bandwidth as it was until recently in one of my sites.

Glad it's working for you, on my forum I simply added a forum where only staf had permissions to view and made the threads there, I also had no issue with logging.

I did get my forum issues sorted, turns out it was a half assed script kiddie attempt to kill it but got it sorted (one of my forums allowed html in the post environment....really bad idea!). When I get chance (which will be after Thursday as my wife is having an op) I will retest the mod from scratch and post specifics here.

jl255 10-06-2013 05:00 AM

i suppose it shld be pretty safe to use the default list of spider list to ban on this plugin? i've no idea which to prune.....

Simon Lloyd 10-06-2013 06:12 AM

There are quite a few lists posted throughout this thread, take a look at them, if you use the built in one then
Quote:

Originally Posted by Simon Lloyd in Mod Description
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :


Max Taxable 10-06-2013 01:59 PM

Quote:

Originally Posted by jl255 (Post 2450315)
i suppose it shld be pretty safe to use the default list of spider list to ban on this plugin? i've no idea which to prune.....

See if the default one helps the issue you're having, first. Is my suggestion. The default list is a good list.

Wajow-community 10-11-2013 11:30 AM

Is there a mod for vb 4.x.x

Max Taxable 10-11-2013 11:36 AM

Found on the developer's profile:

https://vborg.vbsupport.ru/showthread.php?t=268208

qpurser 11-09-2013 02:58 PM

Really love this mod a lot and works great on 4.2.1

Finally Baidu is not showing up anymore in my "online users" list.

I saw another bot recently searching my forum and from some research it seems to be a bad one also: AhrefsBot http://blocklistpro.com/content-scra...o-spybots.html

I added this to my list:
Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
and
Mozilla*Ahref

Was this the correct way to do it?

Max Taxable 11-09-2013 03:51 PM

Quote:

Originally Posted by qpurser (Post 2459434)
I added this to my list:
Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
and
Mozilla*Ahref

Was this the correct way to do it?

NO!!!



Including "Mozilla" and "compatible" in this list blocks just about the entire world!

Get rid of that entry and simply put: "AhrefsBot" in instead!

qpurser 11-09-2013 05:57 PM

Quote:

Originally Posted by Max Taxable (Post 2459439)
NO!!!

Including "Mozilla" and "compatible" in this list blocks just about the entire world!
Get rid of that entry and simply put: "AhrefsBot" in instead!

Thank you for the quick reply.
I was reading post #127 and 128 and thought as long there is something behind the "mozilla" in the same line there wouldn't be an issue. My mistake I guess

Max Taxable 11-09-2013 07:10 PM

Quote:

Originally Posted by qpurser (Post 2459463)
Thank you for the quick reply.
I was reading post #127 and 128 and thought as long there is something behind the "mozilla" in the same line there wouldn't be an issue. My mistake I guess

You might be right but 1.) I wouldn't risk it, and 2.) it's not necessary.

Simon Lloyd 11-09-2013 08:31 PM

When banning whatever you add to the list will be looked for in its entirety, so if you enter "today" then it will ban:
Today
thisdayistoday
wheretodayis
one day today

if you entered "here today" then it will not ban:
one day today
thisdaytoday
today....etc

but it will ban:
we were here today
allhere todayagain....etc

When banning bots make sure you go to WOL and check out their UA as it may not contain their name in the UA.

DemOnstar 11-11-2013 08:15 AM

Had this mod for a while now, forgot to mark installed..

Cheers, does the trick...

Simon Lloyd 11-15-2013 05:57 AM

Thanks for the feedback! :)

waterart 02-07-2014 04:59 PM

ok not sure what i am missing here but i dont see the screenshots that others are showing where it lists the area of setting the spiders to block etc... where do i go in the admin to manage this mod?

Simon Lloyd 02-08-2014 01:21 PM

Go to ban spiders by user agent in vbulletin options :)

And the pic was way back in the thread https://vborg.vbsupport.ru/attachmen...7&d=1329854992

waterart 02-08-2014 02:22 PM

thanks i saw the pic and i expanded the entire menu on the left search for ban or spider and it doesnt show up anywhere?

then i installed spam hammer which does show a menu item and shows a link for options and I ended up finding it there.

thanks

Simon Lloyd 02-09-2014 02:25 PM

it would have been found in vbulletin options, if you've added spam hammer uninstall my mod as spam hammer has added pretty much what this mod does to its code, you don't need both.

SEAda 04-17-2014 04:54 PM

Had a legitimate user blocked, removed mod and user still gets redirected.

Simon Lloyd 04-17-2014 05:25 PM

If you've removed the mod then its not this mods fault!, it doesn't keep any database....etc on which UA strings are blocked, if it blocked a legitimate user it's down to what and how you entered the string to ban.

EDIT: You've never downloaded this mod!!!!!

SEAda 04-17-2014 06:08 PM

Quote:

Originally Posted by Simon Lloyd (Post 2493608)
If you've removed the mod then its not this mods fault!, it doesn't keep any database....etc on which UA strings are blocked, if it blocked a legitimate user it's down to what and how you entered the string to ban.

EDIT: You've never downloaded this mod!!!!!

Yes I did and I have a user who was blocked after installing this mod, and was redirected to the website that is in there by default.

Was hoping for help solving the issue.

SEAda 04-17-2014 06:15 PM

Just because someone is asking for help with your Mod does not mean that that are accusing you of wrong doing. Chill.


All times are GMT. The time now is 04:36 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01432 seconds
  • Memory Usage 1,836KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (2)bbcode_code_printable
  • (14)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (3)pagenav_pagelink
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (40)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete