Sure thing treasureman, I too had it with my old IPB installation and I missed it!

I guess the argument is pretty moot, but that was my understanding of cloaking as well.

The spiders are not handled as registered member they get same before "acces denied". Have a new spiders_vbulletin.xml and vbulletin recognize and show them as spiders. Any help about that?

Things to check:
-- Did you make the code change to init.php in the includes/ directory?

-- Make sure the plugin is active.

-- Double check that the group ID you're placing them in exists and has permission to view whatever they're trying to view.

Yeah I created a new group just for spiders and made that group unable to post because if you set the spiders to use the Registered Users group then someone could change their user agent to the spider user agent and post on your board without registering. Works great!

Thanks, checked everything, usergroup for registered members is 2, but it is not working.

What exactly are the robots trying to do that they're getting permission denied errors on? Some functions won't work because they don't have a real user id.

vba inks directory and vbgallery

I have Photopost VBGallery, let me check a couple of things out, it might not be fully compatible with 3rd party addons, but I'll check.

Yes, these 3rd party addons have their own permission set, so this mod won't work with them. Sorry about that.

ok, thank you

No prob. I'll probably look into making it work with them in the future, but as it stands now, they don't mesh. But don't give up hope! I won't let it get too far from my radar!

:D

Hi

If bots are members then how come mine are viewing the No Permissions Message?
Is this right?

It depends on what the bots are trying to look at. What location are the bots seeing the No Permission message?

Hi, please see image, it was trying to read a thread in the lounge thats open to all

Hmmm, I'm pretty sure that MKPortal is causing the issue. Note that this mod is not compatible with any 3rd party addons that mess with permissions/sessions. Are there any forums on your site that normal guests can't see?

Hi

Yes there are a few that normal guests cant see prob about 5 or so

Installed and works fine. But I'm wondering if there's a way to make these bots showable in the members who have visited the forum in the last 24 hours list.

No, not via this hack. You'd have to ask whoever did that hack to see if they can add bots into it. Not a bad idea, gives you an idea how often bots are visiting.

Excellent idea for a hack. Ive installed it, and am having an issue. Ive edited the core file, init.php, and now the end of the file looks like this :

My issue is that its not detecting the spiders. Ive changed my user agent in Mozilla to Google, Yahoo slurp, MSN, Teoma, etc, and they still dont have proper access. I view my forum with user-agent set to GOOGLEBOT, create a thread, and it says my username is Unregistered. Shouldnt it be "registered"?

I upload http://www.nytimes.com/2006/02/14/po...hitehouse.html
in /includes/xml and I set $vbulletin->userinfo['usergroupid'] = 2;
and #2 = my registered group ID ,,
and still Now I see in (online.php)
Yahoo! Slurp Spider (watch no presmmistion message)
Yahoo! Slurp Spider (watch no presmmistion message)
MSNBot Spider (watch no presmmistion message)
Google Spider (watch no presmmistion message)

whyyyyyyyyyyyyyy ??

Just a comment. You may want to consider changing the title of this mod to use the word spiders instead of bots. Most people here consider a bot a script that automatically posts something to the forums, while a search engine agent is a spider. More people will search for spider when they want this mod.

this mod not work guys :(

All of you saying this is not working had me worried so I made some changes to mine...

I made a new group that works like my registered group but it is named
"Search Bots" plus I updated my spider list... Next I assigned my bots to be in this new group and saw the bots were in the forums...

I shut their access down to access forums, threads, etc... and the bots all got no permissions errors....

So, I then went back in and updated my settings so they can see forums and threads BUT I made it so they can't see profiles (Can't harvest E-mails that way) and locked them out of forums I didn't want them to see based on the user group...

Works great for me!

Thanks!

yeah, pretty much. it seems strange to me that there is an auto-response bot for phpBB & not for vBulletin.

This is too risky for my blood..... Showing one page to a spider and one page to a guest is extreamly dangerous.... Often search engines will send "secret spiders" to test and see if you are serving different pages.....

This is not a definitive clarification of the issue in any way. Google has basically said they don't discuss individual cases or divilge their indexing or verfication processes (apart from saying it's done by spiders not humans), and that you'll find the answers to all your questions in their webmasters' FAQ. It's left completely to the individual to interpret the TOS and decide what's black hat.
To be fair, the BMW case involves very different issues. BMW.de was using quite blatant and crude cloaking: spiders were redirected to pages chocked full of car related keywords at a much greater density than on the pages shown to ordinary visitors in an attempt to boost page rank. Very old, tired strategies that Google can spot at 100 MegaPaces.


I face the usual competing goals of retaining control of my site's privacy and member priviledges while ensuring good SE ranking and the delivery of relevant GoogleAds (and hence healthy GoogleAds revenues). One added problem in all this is that if you have restricted the forum viewing access to logged-in visitors, installed the mod and had your forum pages indexed by Google (essentially the archive URL versions of them), the Google search results will reveal the content via the "Cached" link for each result. If unlogged-in visitors from the Google results click the ordinary link, they will be confronted by a login screen but not if they click the "Cached" link. To avoid this hole in your security you need to add a robots no-cache meta tag to your headerinclude template:
This says to all spiders that visit this page "you may index this page, and any pages linked to it via hyperlinks but you may not store the content of any of them in the Googlesearch engine cache".

You should also edit/install a robots.txt file in your root or forum folder with the following content:
These settings say to the 2 spiders Googlebot and Mediapartners-Google (the GoogleAds spider) "roam my site at will" but tells all others to not crawl any part of my site. It should be noted that only spiders that play by the rules read the robots.txt file. All the big names obey the robots.txt directives but there are lots of smaller rogue spiders who ignore them. Another good reason to edit your spiders_vbulletin.xml file to define which spiders will be accepted into your forum by this mod.


Nice mod but it's operating in the grey zone and until Google definitively clarifies whether it's OK for a spider to see my forum content while an unlogged-in visitor can not, I will not be installing it. I can't afford the risk of losing my significant monthly GoogleAds income or my healthy SE ranking.

I will write to Google and report here if I get a clearer answer than trilljester received. :)

I don't understand how the point hasn't been clarified by the major search engines. Cloaking is when the spiders see content a person casually going to the site doesn't see for the purposes of gaining ranking. If the default usergroup is Unregistered with ability to read posts restricted, and the bots see much more its cloaking...

The bottom line is that its not OK.

Imagine you are Google or Yahoo trying to get the most relevant search results. The archive is a duplicate content / optimization issue... they may rank you lower or higher for duplicate content depending on duplication but theyre not going to terminate you. Now if you added 100 content relevant keywords at the end of every archive page to gain better rank, they probably would.

The same goes for this - if there are some general discrepencies between what registered/unregistered people see its not a big deal. But turning spiders into forum members... :rolleyes: Combine this with the vBPopup mod and someone might as well start cheesyforum.org

Hmm...seems like this isnt working for me either. I'm lookin at the bots and its saying that they are getting "no permissions" error. Any definite fix to this yet?

My guests can see everything that my members can see. So I'm thinking that I don't need this hack. Unless there is some other advantage to using it?

I also have welcome headers installed. From reading parts of this thread, it seems that welcome headers may be what the google bot is reading when crawling my pages. If so, is there a way to make google ignore the welcome header?

Thanks for any advice.

Reg

Corbynt, I'm not sure mine is working either. I've adjusted my useragent to "Googlebot/2.X (http://www.googlebot.com/bot.html)" and gone to my site (not logged in as a user, the access I get is contolled by the user group setting, so thats working, its just that I don't appear in the whos online list as googlebot, only as guest. ? any suggestions whats wrong.

Reg Car, I also need to stop spiders from seeing the welcome headers, as they are ruining my search results.

All suggestions welcome

Well, they will not commit to any further elaboration so I say act on the side of caution and beware.

I have settled on a compromise: I will allow all guests (including spiders) to view the forums and thread titles but stop short of showing thread content. Hopefully most thread titles will be informative (and not of the "Help!", "One more thing" and "Well what do you know..." variety) and so entice Google search engine users to come and register.

You can achieve this with the built-in permissions options by setting the forum permissions for each forum for the Unregistered/Not Logged In usergroup as follows:
Can View Forum Yes
Can View Thread Content No
Can View Others' Threads Yes

vB can't tell who the guest is (whether or not they have posted in the forum before) and so all threads are by someone else. But they are still not allowed to view the content.

The quickest way to implement this for all your forums is to do it once manually for one forum and then use the Forum-Based Permission Duplicator . You can find a link to that duplicator at the top of the page:
Admin Control Panel -> Forums and Moderators -> Forum Permissions

I hope this helps. :)

Do you have Googlebot defined as such in your AdminCP - Spider Identification?

ie: Does googlebot show up in your Who's Online ever?

seems a good hack , thanks

donno if its working for me..??

but i see that it has no permissions :S

now guests have permission to view profiles n stuff :s.... so do memebers...?

Just wondering, has anyone actually seen their rank improved as a result of this hack?

I have something like 11 members and 2 guests show up quite a bit. When I enter the whos online page, it will say one of the guests - I can tell its googlebot by the browser id only.

googlebot is listed in the spider id xml file if thats what you mean?

Dear trilljester & everybody,

I didn't try this mod yet but after reviewing all of your discussions here, I have some ideas :

1. This mod is alright with Google, you guys don't need to worry. Because, very simple, you can set for Guests permission to view all of pages on your forums. In this mod, you set for Spiders permission to view all of pages, there is no problem from Google or whether.

2. I recommend that you guys should create a separate usergroup for Bots, with viewing permissions only. This is good in case if someone try to be .. a bot. In this moment, I am willing to be a spider to make some posts annoy your forums because Bots are in Registered usergroup.

Just small ideas.

Additional, a short question : Is there any accident if we use this mod and vbBOL ?

boeserwolf klicks install

It´s very useful for my community as a lot of members would complain if we would allow guests to read the threads. We have a legal topic in our board, but we also want to improve our search engine ranking, so I think this is one of the best hacks to fullfil both needs.

Thanks a lot for this

Markus

You don't appear to understand.

By allowing SE bots to read these 'member-only' threads, you ARE allowing Guests to read them.

a) by viewing the cached page in the SE results.
b) by changing their Useragent to match that of a known bot, thereby allowing an actual Guest to read the thread.