How can I stop Unknown robot, spider, crawler

[dacho]

Hello mate

How can we stop the Unknown robot, spider, crawler ?
They probably skipped and not refer to robot.txt
In robot.txt I made some bots are enabled and blocking all others bots, and still without permission they scan me

[Dave]

I believe it usually takes a few days up to a week before those robots "look" at the robots.txt file and update it on their side.
By the way you named it robots.txt right? With the s after robot?

[dacho]

Yes my friends, it is clear that file name robots.txt and sits on Root.
It's not about takes a few days up to a week before those robots "look" at the robots.txt file and update it on their side

It probably really are a lot of bots, spiders and crawlers pass through the robots.txt and go inside without identified

The Solution probably need to combine the robots.txt and .htaccess

See the following guide :
http://michael.langley.id.au/blog/posts/28

[Max Taxable]

Robots.txt isn't a law. Bots aren't required to follow it or even index it. Only the legitimate SE bots will. Bad bots simply ignore it. It's just a sign basically, politely saying "please don't crawl these areas of my site." It's alot like gun laws in the US - criminals simply ignore them.

For the others, you want the "Ban Spiders by User Agent" Mod.

[ForceHSS]

<a href="https://vborg.vbsupport.ru/showthread.php?t=268208" target="_blank">https://vborg.vbsupport.ru/showthread.php?t=268208</a>
This will stop them as soon as you install it

[Lee G]

This mod by Paul M will help you track them
https://vborg.vbsupport.ru/showthread.php?t=232182

There used to be an updated spider list for vbulletin on the net. No idea what or if that's still shared

You can then track the heavy hitters and block them

[Max Taxable]

There used to be an updated spider list for vbulletin on the net. No idea what or if that's still shared[/B]

It is:

Updated vBulletin spiders list (http://ozzmodz.com/showthread.php/943-vBulletin-Spiders-List-Updated)


But that is NOT the block list for the ban spiders mod, there's never been a "official" list for that as the Mod is blank when first installed..

[ForceHSS]

When was the last time that list was updated Max

[ozzy47]

It was updated in January of this year. It had over 200 spiders added.

[ForceHSS]

So the last few in the last few posts have not been added? Can they be added

[ozzy47]

They will once I get a few more, actually this weekend I can push out a update, I think I have about 30 more added since the last release.

[ForceHSS]

Good can you tell me when its updated so I can download the new one

[ozzy47]

If you subscribe to the thread on my site, you will know when I push the update. Then you can unsubscribe after that. :)

[ForceHSS]

Will do

[Max Taxable]

If you subscribe to the thread on my site, you will know when I push the update. Then you can unsubscribe after that. :)Actually it would be helpful Force, to stay subscribed and help us add new spiders by posting them in the thread.

[ForceHSS]

Have never seen any new ones as I don't get them much on my site

[Max Taxable]

Every now and then I run into one that's not on the list.