How can I stop Unknown robot, spider, crawler
 

Hello mate

How can we stop the Unknown robot, spider, crawler ?
They probably skipped and not refer to robot.txt
In robot.txt I made some bots are enabled and blocking all others bots, and still without permission they scan me

I believe it usually takes a few days up to a week before those robots "look" at the robots.txt file and update it on their side.
By the way you named it robots.txt right? With the s after robot?

Yes my friends, it is clear that file name robots.txt and sits on Root.
It's not about takes a few days up to a week before those robots "look" at the robots.txt file and update it on their side

It probably really are a lot of bots, spiders and crawlers pass through the robots.txt and go inside without identified

The Solution probably need to combine the robots.txt and .htaccess

See the following guide :
http://michael.langley.id.au/blog/posts/28

Robots.txt isn't a law. Bots aren't required to follow it or even index it. Only the legitimate SE bots will. Bad bots simply ignore it. It's just a sign basically, politely saying "please don't crawl these areas of my site." It's alot like gun laws in the US - criminals simply ignore them.

For the others, you want the "Ban Spiders by User Agent" Mod.

<a href="https://vborg.vbsupport.ru/showthread.php?t=268208" target="_blank">https://vborg.vbsupport.ru/showthread.php?t=268208</a>
This will stop them as soon as you install it

This mod by Paul M will help you track them
https://vborg.vbsupport.ru/showthread.php?t=232182

There used to be an updated spider list for vbulletin on the net. No idea what or if that's still shared

You can then track the heavy hitters and block them

It is:

Updated vBulletin spiders list


But that is NOT the block list for the ban spiders mod, there's never been a "official" list for that as the Mod is blank when first installed..

When was the last time that list was updated Max

It was updated in January of this year. It had over 200 spiders added.

So the last few in the last few posts have not been added? Can they be added