Believe it or not you can actually go to their site and ask them not to crawl your site :)

you say i am correct but i see also these spiders in whois online :( why ? also i see i link showing like this
showthread?=1' ???
i mean 1 spider see this link showthread?=1' ???

Where are you seeing that? make sure that your list of bad bots has no leading or trailing spaces on each name. If you are still having trouble you can pm me temporary admin login details with rights to administer plugins and i'll take a look :)

It's not a no crawl rule, it is a outright block, with this mod. There's no obeying or disobeying. It's not robots.txt. When you install this mod and with baidu on the list, it should be blocked. Gone.

I believe for the incidents of it still appearing in v4s after this mod is installed, must have something to do with interference from some other mod. Else, how to explain my v4 getting NO appearances by baidu after I installed this?

I'll add to the above - I suspect a hook conflict actually. One that only happens when some other mod calls the same hook, and it's not all the time because from what we saw at OzzModz, Baidu was greatly reduced in appearances by this mod, just not totally gone. Occasionally one or two of them slip through during the time of the hook conflict.

It doesn't happen in v3 at all, for the same reason I believe.

Anything in particular I can look for in terms of a hook that might conflict?

Would a list of my mods help?

D.

In v3 and v4 this mod calls "style_fetch."

Well I haven't had Baidu appear on my site in over 2 years ... way before I even installed this mod which has helped me a lot regardless of how it does it lol ... my point is that "I" haven't had Baidu for a very long time ...

Well sure if you block China and other countries via .htaccess you probably won't see Baidu.

I used to have a massive .htaccess with country blocks.

banning by .htaccess is fine if you only have a few things in it because it is read with every single server request, so if you have 10 blocks in your .htaccess and lets say you have a web page with 30 elements (icons, css, containers, includes.....etc) then each one of those that tries to access that page has 30 checks made just to load that page.

Now consider your own landing page and check how many things load to make that page up and you'll soon see why having a lot of bans in your .htaccess can be detrimental particularly if you are on shared hosting or limited vps.

@Gadget_Guy & Max Taxable
The hook is style_fetch, you can try changing the hook for one of the others that loads before all the others but you may not see the result your looking for, doesn't hurt to try :)

Yeah I would try and stay away from ip blocking totally.

Ha Ha Ha, I have been testing this out for the past 10hrs or so Simon, to early to tell yet, but so far looking good.

Installed on VB4.2.2 PL2.

One question that I can't seem to find in the first few posts of this thread that usually explain stuff.... Why would you choose to redirect the bot back to itself? What function does this have over redirecting to a url?

It really makes no difference where you send them, it is just user choice. :)

For me it was giving them a taste of their own medicine, they drain our resources so we send them back to drain theirs :)

Well so far it seems to be going as planned, I will wait another 24 - 48 hrs, and if it is working, I'll let you know exactly what i did. Which hook I used, and what additional plugin I added. :)

Whoot!

Looking forward to your findings Ozzy!

D.

I have had the ip blocks in my htaccess for over 5 years my Man and I haven't ran into any problems in those 5 years ..

If an IP is blocked on your server it's not allowing the page or any page to load, so I am a bit confussed about "so if you have 10 blocks in your .htaccess and lets say you have a web page with 30 elements (icons, css, containers, includes.....etc) then each one of those that tries to access that page has 30 checks made just to load that page."


As a matter of fact, when I didn't have these IP blocks in my htaccess file I was constantly getting emails from my host that my site was being suspended ... by blocking these IPs I am keeping them from even accessing anything on my website or forums ...thus the usage of resources went down ...

Just don't redirect to any of your own pages - feedback loop danger.

EDIT TO ADD: I was right about the hook conflict with some other mod(s) Ozzy?

I would not say a conflict, but perhaps a better hook to execute the mod. That is if the testing continues to provide the desired results.

I agree in part, when you didnt have the block they were calling on every resource...php, mysql, cpu and ram, with the block they pretty much are just using ram as cpu and php time and response is minmal and as you are not loading anything else the ram isn't being maxed either. If you have whole country blocks that doesn't take as much checking as full octet ips like 192.161.0.1, if you have plenty of those then they are checked against each request, if you are blocking just 192.161 then its just one check against each request.

Im probably not explaining myself too well (it reads much better in my head :)).

Just bear in mind that any other hook you choose will need to be sufficient to perform the other tasks of the mod if you wished like sending the email or creating the threads. Some of the other runtime hooks will give errors or not work as expected especially with the thread creation, also keep in mind you need to redirect them before anything has loaded as it's this that is the basis of the mod - keeping resources for your members and not the bots :)

No my Man, you actually are explaining yourself very well lol ...

All I know is that I have not had any negative effects from doing the blocks and I also have just a list of single IPs ..... and let me tell you , that list is long as hell lol ....

Thanks, but I'm still unsure what the benefit would be to redirect back to the source. Would it make them aware you don't want them sooner?

It really doesn't matter, they are redirected with a 301 which is a permanent redirect, so they will always see the url they tried to crawl as the one you send them to. Like i said, i coded that in to send them back to themselves so they have less resources to be crawling other peoples sites - it's only fair! :)

Would you share you CIDR list? Not in this thread, but maybe make a new thread?

Thanks, that is all I wanted to know. :)

Princesspepper could you mark this as installed please :)

You can get it here my Man : https://vborg.vbsupport.ru/showthrea...134.184.0%2F21

Hey Ozzy,

Are we any closer to an alternative or modification to this so that we can get better blocking in place?

I am willing to test on my site as I am still getting hit hard by spiders even with the mod in place.

D.

Hey Gadget Guy, no disrespect but the mod here is mine and isn't marked as reusable code, Ozzy may post what he's tried or done but wont necessarily be added to this mod, however Ozzy has developed one like this with other measures, you can get it at his site.

If you are being hit by spiders with this mod in place it will be because there is an anomaly in your list, this list isn't exhaustive but here's a few reasons why:
Entry in list has a leading or trailing space
Entry has a typo of some sort
Entry doesn't actually represent the bot you think it does (i.e Ahrefsbot I believe has a different name in the UA)
Mod run order may conflict with another mod using the same hook
There are other reasons but those should get you going! :)

Crazy how a leading or trailing space can jack things up lol .. I remember trying to block facebook completely when I first installed your mod here Simon and it wasn't working ..why? the damn trailing space!! lol

Sorry Simon,

I meant no disrespect... I thought you guys were working together.

My apologies.

Could I send you my list via PM and a screenshot of what I am seeing in terms of the spiders appearing?

D.

No I have not developed a mod similar to this. I recommend this mod to everyone. If you remember correctly, I was one of your beta testers way back in the day.

My actual intention with what I have done, was to PM it to you first, and get your input, before I said anything to anyone else, or maybe you update the mod with it.

So let me know what you want me to do. :)

Thank you, I tried to like your post, but there's a limit in this.

Anyway, I see you put this in htaccess, I think it's better added to firewall rules for denying hosts. It works much faster. Of course, if you have access to firewall in that way.

Yeah, if you have a dedi, or a vps, you would add it to the firewall there, stop them cold.

I thought you had developed something that's why planned improvements for this have been shelved (no point in duplication :)), in that case Ozzy PM away :)

Yes of course :)

No sir we have been trying to solve the mystery of why Baidu gets through on some v4 installations, but not all and never a v3, and my hook conflict idea opened a new can of worms for investigation, and Ozz found something very interesting.

What interesting thing was found

Yeah ...Yeah .. what "something very interesting" do you speak of .....