SPider question

[JonUrban]

A few days ago, I got "visited" by a spider that used 1Gb of bandwidth! My Control Panel shows this:

Unknown robot (identified by 'crawl')
HITS: 43632
Bandwidth: 1007.05 MB

I have changed my board so that unregistered users cannot view, and I added a robots.txt file.

Is there anything else I should do?

How can I find out what spider this one is?

Should the robots.txt file be in "public_html", or "public_html/forums".

Thanks

:-jon

[lasto]

should be in main root im sure of it

also is this what is in your robot file

# go away
User-agent: *
Disallow: /

as this indicates that no robots should visit this site further

1gb of your bandwidth is bad esp for a robot you dont want in the first place.

Instead of us blocking em it should be if there is no file there saying allow they should move on.

[filburt1]

<a href="https://vborg.vbsupport.ru/showthread.php?s=&threadid=55092" target="_blank">https://vborg.vbsupport.ru/showt...threadid=55092</a> !

[Erwin]

Don't block all spiders! :) You may only want to block specific ones from smaller less significant search engines.

[lasto]

how about a list of spiders to block then or ones that seem to want to use all your bandwidth as to be honest i aint got a clue what spidering and wot aint.