Log in

View Full Version : blocking spiders


HakkieDEV
02-19-2006, 09:06 AM
Hi all,

I've been using a small file, robots.txt to prevent spiders from searching my webpage (costs too much bandwith).

Now, it has been working quite well, until recently I saw a spider called "e-collector" was still spidering my webpage.

How do I stop it from doing that?

This is the robots.txt file I used up till now:
===================
User-agent: *
Disallow: /

Paul M
02-19-2006, 10:22 AM
robots.txt will only work if the spider in question actually bothers to read it, not all do.

HakkieDEV
02-20-2006, 09:43 AM
any idea about how I could block it without a robots.txt file?