Google AdSense Spiders

[WolfsongNevan]

We have many, many spiders visit our forums daily (hourly), but one that keeps concerning me is the Google Adsense one.

I frequently see it trying to access things that no spider should be accessing ... like moderating a thread, trying to read PMs of other members ....

They're always getting an error message, but it just flabbergasts me that these bots are trying to infiltrate the security of our site. Why is it necessary to try and access what no regular member would EVER have the ability to do, like moderating a thread? No other spider tries to do these things.

Does anyone else have this concern, or am I just being paranoid?

[tgreer]

They are just following links. You can control this with robots.txt. Here's mine, which I think I got from a post from Scott on vbulletin.com. If your forum is in a sub-folder, such as "forum", then you'd need "/forum/attachment.php", etc.

User-agent: *
Disallow: attachment.php
Disallow: avatar.php
Disallow: editpost.php
Disallow: member.php
Disallow: member2.php
Disallow: misc.php
Disallow: moderator.php
Disallow: newreply.php
Disallow: newthread.php
Disallow: online.php
Disallow: poll.php
Disallow: postings.php
Disallow: printthread.php
Disallow: private.php
Disallow: private2.php
Disallow: report.php
Disallow: search.php
Disallow: sendtofriend.php
Disallow: threadrate.php
Disallow: usercp.php
Disallow: /admin/
Disallow: /images/
Disallow: /mod/

[Dan]

The GoogleAdsense spider is actually looking at your site to see what Ads to display in your Google Adsense banner, if you block them they won't be able to see the content and show ads relivant to the page.

[tgreer]

There are no ads relevant to your moderation page, or admincp...

[WolfsongNevan]

Dan, I understand that ... my point was they're not just viewing threads, they're trying to moderate threads, read the PMs of other members .... I have no problem with them viewing content they should be so that we get relevant ads, I just have a problem with them trying to do things no regular member would have access to. It's a little .... disturbing.

[Damian]

tgreer already explained this. The bots are just following links. For example, you have links all over the forum to "Send A PM". Of course they can't actually send a PM. The bot would follow that link the same as a link to a new thread, they don't know the difference. But you’re "Who’s Online" would show the boot "Creating Private Message"
They follow the same forum permissions as Unregistered / Not Logged In.

[ConqSoft]

In other words, just get over it. It's fine...