That is it !
They go by those generic names only, nothing else.

Im ,y mod you are banning any useragent that has any occurrence of one of the strings in your list, i very much doubt that \wbot[\/\-] is found in any useragent as the looks like a regular expression, in my mod simply wbot will do if thats in their useragent.

Thats just stats from cpanel awstats and mean nothing other than spiders were identified with those in their identifier.

They are not their useragents, read some of the links that i took time and trouble to post in the mod description on hpw to find the useragent.

It won't block MSIE Beta 10, but it will block MSIE 10. Best to remove MSIE 1 from your list now. I haven't seen any references in the output.txt file of anything prior to MSIE 5 so removing MSIE 1 probably won't cause any problems.

https://vborg.vbsupport.ru/external/2012/11/15.jpg


Edit: If you were wondering what the IP Address 108.2.106.107 is, it is Verizon's Search Engine (www.verizon.net) definitely not something you would want to block.

Using the time based Mod in my signature, I often see user agents with early IE, such as 3, 4 and 5, but you're right - I haven't seen any IE 1 or 2 in so long, there's no doubt it would be good to remove MSIE 1 from the ban list.

Thanks for the information!

It will block MSIE Beta 10 if it appears like that in the useragent, this mod will block ALL instances where the exact string in your list is found in the UA.

Okay, I've installed this and it seems to be working fine, but I have some questions.

At first the Baidu spider was hammering us, hitting the site about every 8 seconds, but now it seems to have more or less given up on us. Yandex, on the other hand, seems to have intensified it's attempted crawls. At first it was hitting the site about every 25 seconds but over the course of this install has ranged everywhere from every 1 second to where it's at now, about every 1.5 minutes. The 1.5 minute attempts have only occurred in the last couple of days.

A couple of days ago when Yandex was hitting us every 1 second, we had some server load issues. I don't know if this is related but it seems it might be and I'm wondering if logging the blocks to a text file might be counterproductive in that area, writing to the file every second or so.

Also, Yandex is using the same IP address every time, so I thought it might be best to just block it using the .htaccess file, but that doesn't seem to have had any effect. Is this mod redirecting Yandex before it has a chance to read the .htaccess file or is Yandex simply ignoring it?

Your server would be the one to decide if Yandex is blocked by the .htaccess file or not.
If the ip is denied in htaccess, your server will block before any vb module loads, or any page opther for that matter.

Smirkley is right, .htaccess is loaded before anything else. As for my mod using Yandex as a blocking string will block it, if it is still showing then thats because either they dont actually have yandex in the user agent or you've entered more than just Yandex as the string, remember, my mod bans anything that has exactly your string to look for (including spaces), so if your strings in my mod look like this:
Baidu
Yandex
SoSo
.....etc
then yandex will be blocked, however if it looks like this:
Baidu
Yandex123
SoSo
....etc
then any bot with just yandex in their string or any other kind of yandex like YandexWorld will not be blocked, but anything containing yandex123 will be blocked.

As for writing to a file, just turn that bit off, it's there simply for test purposes, trouble shooting or checking on individual user agents.

Your mod is blocking Yandex, but I just figured since it was using the same IP address every time it might be better to just go ahead and block that IP.

I guess I didn't do something right in the .htaccess file because Yandex was getting past it and was getting blocked by the mod.

yandex doesn't always use the same IP, somewhere earlier in the pages for this mod i think i posted how to do it in .htaccess if you wish.

Hi Simon,

I have recently installed your mod and whilst testing options managed to block my home PC agent from accessing my site (Im new to this game sorry)! My question is after removing my agent details i still appear to be blocked from my home PC - How can i rsolve this any advice would be appreciated....

Try clearing your browser cache and also cache from admincp/Maintenance

It will be your browser cache, you can also try (on your pc) Start>Run>ipconfig /flush dns (the space is intentional)

I'll caution on the above - you really don't want to enter any parts of user agent strings that might be common stuff - you have the potential of not only blocking yourself, but literally tens of millions of computers and/or other devices.

It's one of the best Mods there is for vBulletin but, USE WITH CAUTION.

Thanks for that, this mod was only ever built to stop bots eating up your bandwidth which is why i recommend using actual bot names found in the useragents :)

:) Thanks Simon I will try this once I get back from work

Thanks for the advice!!

:up: Sorted thanks again!!

:up: Thanks for the heads up on this!!

Fantastic little mod. Works great! :up:

well done, mark installed

Glad its helped you:)

I'm looking for feedback guys!
Would it be beneficial to automatically ban bots that exceed x number of bots at any one time?

So, the likes of Baiduspider send around 200 at any one time, so if i entered say 150 (in place of x) in a settings box then they would automatically get added to the ban list, let me know your views as i'm not going to work on something nobody feels is needed :)

It's not a bad idea but is probably not needed. I think most any bot that would "exceed a certain number" would probably be a bot we are already blocking by name with this. I guess it would be nice to have a second log of the bots that are coming through if that is possible, then we could tell if it was necessary.

One thing I wouldn't mind seeing is options to choose both filename and directory for the bot output file(s). An option to be able to show the most recent bots at the top of the file (reverse of how it saves now) is something I would like too, not sure what everyone else thinks.

I like this idea and would most definitely like to see this feature added. :)

Hi,

I would like to ban Amazon AWS EC2. I have tried AmazonAWS and Amazon AWS EC2. Any suggestions please?

Thank you.

Check out the links i've given in the mod description above entitled : How do i ban a bot?
it should explain how to find out their exact user agent :)

That's a hosting service. Why are they spidering your site? Are you sure that's correct?

Based on research a number of people recommend blocking AmazonAWS. What do people on here recommend?

I suspect accounts held on some of their servers are of no use to your forum and are scrapping content or emails..etc

Banning bots, as i've always said is a personal thing :)

Okay, but if I recall correctly this only bans by user agent, not IP block and therefore would be ineffective to ban 'AWS'.

Why would it be ineffective banning them? every device that accesses the internet...etc has a UserAgent, you just need to find the useragent and i show you how to do that in the links in the mod description.

Read this: http://www.webmasterworld.com/search...rs/4368965.htm

If you really want to ban ip's then https://vborg.vbsupport.ru/showthread.php?t=268146

Because since Amazon runs a cloud hosting service, anyone can own an AWS server. Hell, I have one. There is no ONE service and user agent on AWS, so its not possible to ban all AWS servers by user agent.

But not ALL AWS users are bad, are you? :), agreed you cannot ban a server but every bot, spider, person or device that comes your way will have a UA that you can ban.

The request was to ban AWS servers by user agent. That's not possible.

(And technically you don't even have to send a user agent.)

The request wasn't specifically to ban the servers by UA :), if you send a malformed or blank UA then you can ban those too ;)

As a side note i noticed that you haven't downloaded the latest version of this mod or marked it installed, have you uninstalled it, if so could i ask why? just helps me develop more robust things in the future.

I started with this mod. However, my server at the time was so resource starved that I needed to block the spiders before it got to PHP/MYSQL. Nothing wrong with it. It worked well. I just couldn't afford the resources.

Yes it is. Enter it exactly like it appears in the user agent string.

"amazonaws"

Yes, but no one is using that UA. Amazon has no reason to crawl *any* site.

Amazon AWS is their hosting they sell. And yes they also crawl the web: http://aws.amazon.com/search-engines/

I have it blocked as well, using this Mod.

Here is how I decide what UAs I block:

1.) Is it beneficial for my site to have it crawling?

2.) Does it behave nicely? Does it obey robots.txt?

3.) If in any way suspicious, it goes in this Mod.

Like the developer says, it's all about personal choice.