Search engine spiders/bots in their own usergroup

[trilljester]

Hi all,

I used to remember this as a feature in IPB, but it's not in VB at all. I'd like to add this feature into VB as a hack/mod but need a little direction on where to look.

I'd like to have Search engine robots/spiders put into their own special usergroup so they can browse the board as a registered user as opposed to a guest (which they are treated as now). I searched VB.org, but could not find a hack or anything that explains how to accomplish this.

Any suggestions on where I should look to make this modification? Thanks!

[Ramsesx]

That would be great if this could be done.

[trilljester]

I'm pretty sure I could do it, just not completely sure where to start. I'm currently reading the API documentation, but it's not giving me a lot of clues. :disappointed:

[Snake]

Yeah this would be a great feature to have on our boards. I used to run an IPB forum and I'm really looking forward to this!

/me subscribes to this thread

[Marco van Herwaarden]

This would mean breaking the TOS of all major search engines, and your site will be removed from their listings.

[Snake]

And could you explain how?

[Marco van Herwaarden]

By presenting them a different page then a normal guest, this is called cloaking.

An example from the google FAQ:

Alternately, your page may have been manually removed from our index if it didn't conform with the quality standards necessary to assign accurate PageRank. We won't comment on the individual reasons a page was removed, and we don't offer an exhaustive list of practices that can cause removal. However, certain actions such as cloaking, writing text that can be seen by search engines but not by users, or setting up pages/links with the sole purpose of fooling search engines may result in permanent removal from our index. You may want to review our Quality Guidelines for more guidance. If you think your site may fall into this category, you might try "cleaning up" the page and contacting us with a re-inclusion request. We don't make any guarantees about if or when we'll re-include your site

[trilljester]

Hrm... that's really shaky. I couldn't see what I'm working on as "fooling" the search engine, more like helping it. I would imagine fooling it would involve loading up a page full of keywords written in white on a white background so normal users can't see but it fools the bot into thinking that there's tons of content there. I'll email Google and ask them if I do this, if I would get removed from their results because of subterfuge.

I just want the robots to see my site as a registered user would. I've been to a couple of forums where this happens and they have excellent search engine positioning. None of them were VB-based boards, however, and it's a feature that VB is lacking a lot in.

[treasureman]

I used this feature for over 2 years with Invision board and never had any problems, I was always well listed with the search engines.

I see this feature requested over and over, both here and Vbulletin.com, come on PLEASE add it or make a mod for it!

Those that don't want to use it wouldn't have too, and of course those that did want to use it would use at their own risk (which you do anyway when you use any mod)

Thanks!

[trilljester]

After reviewing a lot of VB's code, I think this will be more of a code modification and not a plugin, unfortunately. The plugin system probably won't work here, since this will mess with sessions (changing the bot's session from a guest session to more like a member's session).

I could write a custom hook, and make that the only code modification. Yeah, that might be better.

Quick update: I've got the system now recognizing a bot from a human very early in the startup process. Now if I can take that info and have it apply it to the permissions, I should be OK. This might a great addition to VB if I can get it to work.