Did you read that? There is nowhere in that link that says that Amazon themselves crawl websites. Can you even explain why a hosting company would want to catalog data from every website on the internet?

I'm wondering if there is some confusion on what a user agent is and does. The UA is the remote web crawlers way of tell you that it is there cataloging your site. It's not required that a crawler send you a UA at all. Instead, its just considered polite. If someone wanted to, they could send a completely random UA every time or not send one at all.

Since Amazon AWS is in the hosting business, they have no need to crawl websites at all. However, this doesn't PREVENT people from buying their own server from Amazon and crawling your website. If someone were to do this, the UA would be whatever they wanted it to be, not some form of "AmazonAWS".

Assuming what you're really trying to do is prevent anyone from buying a server from Amazon and accessing your website, you'll need to find all the IP blocks that AWS owns and block those. However, that is outside the scope of this mod.

For reference here's what a user agent is and some extra info http://en.wikipedia.org/wiki/User_agent. All this mod is designed to do is stop bots from eating up your bandwidth by redirecting them before any content loads. To be honest you can never stop anyone who is intent on scraping your site from doing so.

The "amazonaws" crawlers have that designation in their UA string. Anything else coming from Amazon has it in its host description.

The rest of your missive, I am well aware of.

ok.

I have a confusing question...
Ok I have a very small member site...like 24 members...
So when I noticed I had 35 users online most of the time and I started seeing more and more baidu spiders
I decided to do something about it...

I installed this mod.
almost instantly ...well within say 3 hours my users online soared to well over 150 on busy times like Now...tonight.
I had
Most users ever online was 247, 1 Day Ago at 12:58 AM.

With only one new account created, and maybe me or one other registered user online...

My question is this. what happened when I installed this mod to make such a drastic change in the users on my site and why?

I do not understand this and I read that the server load increases...
I find it hard to believe that anyone is finding my site via a search engine since it is a brand new .cc name and it has only been online for two months now...

Is there something about pushing away Baidu that enables more sites to come, or Spam bots?
attempting to register and what not, many are in areas that there would not be a normal user.

I see many attempts a registering and yet no more new users.,.. so I believe those are bots locking...

Please advise...

What's happening is (and you'll probably find this) is because Baidu can't get in with the spiders/ip's they were using they are now trying a rotation of other ip's and bots, i use this mod myself although i don't ban the bots as i monitor their visits to further enhance any mod i make against them, i currently have 236 baidu bots (and 140 other bots/search engines) at my site.

With the mod in place and redirection working you'll find that these bots that you have banned will slowly drop off as they all get the message of the 301 permananet redirect to wherever you've decided to send them, your server load will lessen and things will be more normal :)

Also do you have your robots.txt set up correctly to stop the search engines or bots that obey robots.txt from indexing pages on your site that they shouldn't like register.php, members.php ....etc?

I did not understand how to do the text part since I am what I even call very green in this aspect of Vbulleting...
so I just installed the mod...
I can wait and see if it drops off and report back...
Thanks for the help in understanding...

Ok, what you need to do is upload the attached to your forum root, however if your forum is at this level www.mysite.com/ then edit the attached to remove /forums if your forum is at this level www.mysite.com/forums then you can just upload it to that folder.

You can add any page or file to robots.txt that you wish, just follow the same structure :)

Well thanks Simon...
Thats really nice...
I will do so immediately.
Nice to see someone really help out the Noob...lol
Thanks again I appreciate this very much...
I will report back.

So I think what you are telling me is this...
Since my site forum is at root level to edit as follows...
This...Disallow: /forums/albums.php
to This...Disallow: /albums.php

yes if your forum isn't in a folder but simply "on your server" so you dont need to access a folder to get to it then thats correct!

After being only installed 10 minutes, I've seen a 20% drop in server load already. I was already blocking them with .htaccess but they were still getting in. According to AWstats bots have been hitting my server MILLIONS of times per month.

Thank you very much from the bottom of my heart, you're very talented!

You're welcome, dont forget to remove them from /htaccess now as they will be adding load just being there :)

Thank you good mod

Glad you like it :)

Been running this for a little over 8 months now.

This past month it blocked 6,659 bad bots. Which is very close to what it blocked on the first month I had it installed.

Baidu finally stopped coming after about 4 months. They were originally hitting the site at over 10 times an hour. Yandex is still coming but they are down to once or twice a day instead of multiple times an hour.

Most Popular blocked User Agents currently:
FunWebProducts, MSIE 6, MSIE 7, Nutch, Yandex

My Full Blocked User Agent list:

This new one just showed up and has been attempting to ping my site on average around a hundred times a day (started about 15 days ago):

Seems some bots come and go, just glad this mod is here!

Im very glad you've found this useful, thanks for posting your updated bot list it may help others decide which to block, however i still have to mention that banning bots is a personal thing and you have to decide what it is you want to acheive from the banning and will anything you block prevent legitimate people from viewing your site.

In the above you block MSIE 7, whilst this may be good for you others may want users who still only have IE7 to be able to view their site. All i'm saying to people is think before you block :)

What is your take on "MSIE 6"? I seem to also be getting quite a few hits from that browser as well.

Personally unless you're catering for developing countries (computerwise i mean like eastern block...etc) i'd ban MSIE 6 but again have to stress it's a personal choice.

thanks
installed your mod

Thanks a lot for the great mod!

I will try it out and see how things go...

Cheers

My personal take is, there aren't very many actual humans using it. It's almost always a botnet zombie computer.

And if it is a human using that dinosaur, I really don't want his/her traffic anyway.

I came across this a couple weeks back:

http://www.ie6countdown.com/


It's a worldwide countdown Microsoft is doing tracking Internet Explorer 6 usage. They are tracking the percentage of users worldwide still using ie6.


Excluding China the percentage of users worldwide still using ie6 is much less than 1% and in China it is currently 24%. To me that is just one more reason to block "MSIE 6".

Yup. Like I said, people who use that dinosaur aren't desirable to have on the site, and definitely aren't worth support for their archaic, garbage browser that should be erased from the web.

WARNING: For those that have that use the vBulletin Mobile Application, this plugin can and will prevent your app from being publish. if you have the UserAgent banned. I think it is the MSIE 6.

Solution: When you got to publish you app just disable this product until you have published you app. Then enable the product after words. If you have this active when trying to publish and you have it posting in a forum, look for the post that targes the API file. Then you will know what the UserAgent is that you have that is locking it down and preventing it from getting your site's information. Don't worry, when you go to publish you will know instantly.

Why on Earth would you ban the IE6 user agent?

Because just about the only use for IE6 anymore is spambot networks, botnet zombie computers.

If some real, actual human is still using IE6 I don't want them on my site.
But, there really aren't.

Installed and testing.

One question, the pre-filled redirect url should be left intact?

Thanks

You can change that if u want

Thank you... I will leave as is for the present...

You have the option to redirect to a site (i.e the one already installed) or directly back to the ip of the banned useragent, its all about choice really :)

How do I know if this is working?

Haven't seen any evidence so far...What do I look for?

it will take over 30 minutes to start to see differences in the WOL as the spiders get the message and a bit longer until they stop trying altogether.

The easiest way to see it working is to turn on writing to the log file, or if you dare have threads made in a forum of your choice, i advise against it as you can get thousands of posts quickly!!!!! it's only there for test purposes.

Smoke. Smoke starts coming out of your hard drive. :D

Thats only if you're a power user :D

Should I make any changes to my robot.txt file?
Right now it is blank.

Hi Gary this thread has nothing to do with robots.txt files, the mod bans anything whose useragent contains any string you enter in to it.

And as a standard you should have something in your robots file as you've been shown here https://vborg.vbsupport.ru/showthread.php?t=304164, there are many threads here that contain details of robots.txt.

Oh, I guess I need to learn more about user agents and robots.

Thanks'

Hi Gary, all that you need to know about useragents...etc is in the thread description. Not all bots follow the robots.txt so, with this mod you can block those bots completely and many others. What you need to do is identify your target audience, so if you are not catering for China then you'd want to block Chinese traffic, to sort the bots out you can block the likes of Baidu Sogou....etc.

I'll try and help you with whatever you need along the way so that you get to keep your bandwidth for more important users :)