Spider Conditional?

[tgillespie]

Just curious if anyone has ever looked into making a search engine spider conditional? I have certain aspects on my web site, such as text links and buttons that I would rather not let spiders visit. With vBulletin's engine, would it be possible to create a conditional that sorted spiders by hostmask or IP? For example:

<if conditional="host8.google.com"><else />My Friends Site</if>

Not looking for someone to make it, just checking if its possible. Most logical condition would be based on hostmask or ip range.

If done, I think it could greatly help the ability to better SEO your forums.

[amykhar]

I'll do it today if I have time. I can use code from my spider's hack to set it up.

Amy

[xStylezx]

Id be interested in knowing how to do this in vbulletin myself

[amykhar]

A word of warning - I'm releasing this for 3.6 because that's what I'm running and that's what I can test on. I suspect, however, that the 3.6 version will work on 3.5 as well.

[tgreer]

No plugin or special coding is required. All of this can be accomplished via a standard "robots.txt" file. A simple web search should reveal all you need to know about this standard technique. Also, I believe there is a thread already on vbulletin.com which lists some of the standard pages within vbulletin that spiders should not try to access.

[amykhar]

Tgreer, you are correct. I misunderstood his post to read that he had certain portions of his individial pages that he wanted to exclude from spiders.

[tgreer]

Just out of curiosity, then... wouldn't excluding certain portions of pages from spiders be considered "spoofing", and negatively affect SERPS?

[amykhar]

Probably. But, it would have been a fun academic exercise to code :D

[tgreer]

No doubt, but the original poster was wanting it for SEO purposes. I don't know for certain, but I'm pretty sure that hiding certain sections of pages from a search engine would have a negative impact on SEO.

But then again, since the conditional would be implemented in PHP, I'm not sure how a search engine would KNOW...

[tgillespie]

No doubt, but the original poster was wanting it for SEO purposes. I don't know for certain, but I'm pretty sure that hiding certain sections of pages from a search engine would have a negative impact on SEO.

But then again, since the conditional would be implemented in PHP, I'm not sure how a search engine would KNOW...I'm pretty sure it wouldn't hurt SERPS because the HTML is just never rendered and the spider wouldn't know. The same technique is used in vBSEO's signature hide hack which disables signatures for guests which also disables it for spiders. Used properly, a conditional like this would greatly benifit boards/sites.

amykhar, I would really appreciate it. 3.6 is fine as I am sure it will most likely be backwards compatible, if not, I'm sure I could manage.

Well seeing as it might be possible, I guess I'll start a mods request thread.

[xStylezx]

Im very interested in seeing how you do this amykhar.Im learning to code now in my spare time and think i could learn some stuff from this(being as that im an SEO addict as well.)

[amykhar]

I plan on swiping code from my spiders on forumhome hack. Basically look to the useragent and see if it's a spider. If it is, set a variable isspider to true within the userinfo array.

Then, I can do a check on $vbulletin->userinfo[isspider] within the templates. That's the general premise anyway.

[xStylezx]

yeah,that sounds like a good plan.pretty much how i envisioned,will be looking forward to this.

[tgillespie]

Any updates on this Amy?