How can I stop Unknown robot, spider, crawler

[dacho]

Yes my friends, it is clear that file name robots.txt and sits on Root.
It's not about takes a few days up to a week before those robots "look" at the robots.txt file and update it on their side

It probably really are a lot of bots, spiders and crawlers pass through the robots.txt and go inside without identified

The Solution probably need to combine the robots.txt and .htaccess

See the following guide :
http://michael.langley.id.au/blog/posts/28