Bots

[Irishnotsane]

How i prevent bots from crawling on my website/forum?

[kh99]

There's a bunch of mods having to do with spiders, just search for spider or spiders. There's this one for instance: www.vbulletin.org/forum/showthread.php?t=268208 which I think allows you to ban only the "bad" ones.

[Irishnotsane]

Thanks, but is there a way to prevent them fullstop, as opposed to banning only the bad ones?

[kh99]

I think you can create a file called robots.txt and upload it. Here's a link to a site about that: http://www.robotstxt.org/robotstxt.html . To be honest it's been so long since I've dealt with that stuff myself that I don't know how well it works. But like it says on that site, it depends on bots voluntarily obeying your rules. If they choose not to, then I don't see that you have any choice other than to try to ban them by user agent, or maybe ip (although I'm guessing that doesn't work all that well).

[Irishnotsane]

Ah right, so by adding

User-agent: *
Disallow: /

In my robots.txt file in my ROOT directory, it'll disallow all bots (unless of course they choose to ignore it)

So by having this, plus that product, it should get rid of 90% (or so) of bots?

[kh99]

Originally Posted by Irishnotsane

So by having this, plus that product, it should get rid of 90% (or so) of bots?

I'm not sure about that, but I'm pretty sure lots of other people around here will be able to comment....

[Videx]

These days what you're talking about are generally called "spiders", with 'bots' being spam bots. You rarely want to ban spiders - they are your friends.

Spiders Display on Forumhome and Who's Online for vb4

[Irishnotsane]

What do these "Spiders" even do?

I don't really want them on my forum regardless of if they're my friends or not, I want it to be just me and my members so I'd rather stay off Google and other search engines.. I own a small community based forum where users are invited in, so pointless being on Google anyway.

[ForceHSS]

the link in post 2 is the best to stop bots the bad ones that is