vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 3.8 Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=235)
-   -   Board Optimization - KX - Spider Permissions (https://vborg.vbsupport.ru/showthread.php?t=196420)

unp 12-09-2008 04:28 AM

webmasterworld.com used this technique for years, results and description would appear in google but clicking the links would take you to a login page,
They were never penalized,

ShawneyJ 12-09-2008 06:28 AM

Quote:

Originally Posted by eJM (Post 1681035)
Why don't you provide the same sort of "proof?"

IP.Boards AdminCP
Treat spider/bot as part of which group?
This is useful as you can specify group settings for a 'bot' group which can restrict access to certain areas, etc. just like Ziki's addon. Please note: you can make a bot group and give bots there own permissions.
https://vborg.vbsupport.ru/external/2008/12/83.jpg

Force spider/bot to use skin
This is useful as you can create a 'lite' optimized skin for your spiders to use to reduce bandwidth and increase the depth of the bot crawling.
https://vborg.vbsupport.ru/external/2008/12/84.jpg

Spider Bot User-Agent
Use this section to list which search engine user-agents you'd like to be treated as a search engine visit along with a 'human' name for the online list in a bot=Human Name format.(one per line)
Example: googlebot=Google.com
https://vborg.vbsupport.ru/external/2008/12/85.jpg

Quote:

Originally Posted by eJM (Post 1681035)
based on the content of your avatar and signature, you're prob'ly around a teenager. I don't know how much experience you have as a webmaster, but mine goes back to about the time you were born (in other words, in the teens). I'm old and I hope I am wise in some matters. My personal rulZZZ: don't argue with children. I'm done with this conversation.

well you didnt learn much in that time lol.
did you no that Avril is an Adult which mite make you think im one to lol.
actually im not going to flam anyone like you are, if im wrong i dont cry like
a little girl. no offense but your just plain wrong.
you started something you couldnt finish.


also, sorry Ziki, your thread has been attacked, its a sweet hack and keep up the good work m8.
this will be my last post unless i need support for your addon.
but at least now everyone can see that IP.Boards/invision comes with
this built in and can see how useful your addon/hack is ;)
cheers.

eJM 12-09-2008 07:46 AM

Quote:

Originally Posted by jaycob (Post 1681449)
IP.Boards AdminCP
Treat spider/bot as part of which group?
This is useful as you can specify group settings for a 'bot' group which can restrict access to certain areas, etc. just like Ziki's addon. Please note: you can make a bot group and give bots there own permissions.

Notice that is says "restrict access" not gain access.
Quote:

Force spider/bot to use skin
Spider Bot User-Agent
Neither of these are a violation of any rule.

Quote:

well you didnt... ssssniiiip
As you kids like to say, whateverrrrr.

Quote:

also, sorry Ziki, your thread has been attacked...
BS! An issue has been brought to light. You can delude yourself all you want, but it is still an issue. Your proof is not. Ask IPBoard support whether that feature is designed to give access to Google when it blocks others. It doesn't. You can use it to gain access to restricted areas if you want, but it was NOT designed to violate any Google or other search engine terms of service. I am familiar with the product. I've been managing message boards of all types since 1992.

If you want to continue this discussion further, then honor the mod author by complying with his request. Start a new thread somewhere else about it.

Jim

Ziki 12-09-2008 11:57 AM

Ok that does it.I had to report both of you.If you can't discuss in a mature way about the topic (Is this modification SE friendly?) and just insult yourself,please do it in a private discussion.This is not a place for such conversation.

Marco van Herwaarden 12-09-2008 02:07 PM

Quote:

I am sure that this does not conflict with Google TOS,as spiders can see the same content as the board's users can.
Sorry to burst the bubble, but those warnings are correct.

Serving different content (layout is not an issue) to spiders and to visitors (and not registered members) is considered cloaking and can get you blacklisted on SE's.

Leo Brazil 12-09-2008 03:00 PM

Quote:

Originally Posted by eJM (Post 1680384)
It is a violation of Google's rules. They may consider this "cloaking." One of the definitions of cloaking is:

It's likely that putting a spider in a special position to see content that is not available to unregistered users would be considered cloaking ind it will get your website removed from the Google indexed pages.

In fact this is kind of violation of Google's rules. But it takes time to Google punish you but trust me they'll.
And don't forget you're playing like "My Game, my rules", I mean Google can simply ban you from their lists and you'll have to do whatever they want. The problem is it takes a lot of time to get back to their lists if you were banned.

I can tell you that because I've seen this before, a friend of mine had an issue with Google about cloaking. His board was on the web about two years with a kind of cloaking, he always tried to convince us that wasn't against rules and something. One day he was notified by Google and he took almost one year to get back to their lists.
The problem is if you are listed due cloaking there's no argue with them at all, you're guilt and that's it and have to do whatever they want.

I'm not telling this because I think this is a BS, waste of time and nothing like this. In fact I think it's got potential. This this is for who really wants what they want and the risks for.

Orakk 12-19-2008 06:08 AM

Quote:

Originally Posted by Zia (Post 1667360)
it can be done via robotx.txt .........

That text file is just a polite request which many, if not most bots ignore. One only need check their log files to see this is true and is common knowledge via google search the thing is close on useless.

Fair enough, if you don't want to risk up setting google then so be it but there are many of us who see being indexed as a waste of time and invasion of our privacy.

Thanks again for this plugin, home is much calmer now.. :up:

hscorp 01-28-2009 12:27 AM

ziki i really appreciate this mod and i hope it work

but it doesn't look to work with me

see the wol pic and setting pic

https://vborg.vbsupport.ru/

https://vborg.vbsupport.ru/

Ziki 01-28-2009 06:54 AM

Try placing them into a usergroup.

CableSux 03-17-2009 06:01 PM

Quote:

Originally Posted by dirtyfeast (Post 1667156)
I am not sure if this works. All my spiders still get the "Viewing 'No Permission' Message & Viewing Error Message". I have installed, set permissions, and it is active. Did I miss something. Are there other edits, or permissions that must be made. Is there a special spiders mod that must be installed. Let me know. Thanks for your time.

Quote:

Originally Posted by Ziki (Post 1667291)
No this should be all.Can someone also report this as not working?

Quote:

Originally Posted by Ziki (Post 1726946)
Try placing them into a usergroup.

I'm having the same problem with the 3.7 version.
https://vborg.vbsupport.ru/showpost....4&postcount=33
I'm getting ready to upgrade to 3.8. Has anything changed in 3.8 to make it work?
.

intell_iut 04-09-2009 10:01 AM

not work for me too
maybe better choice is that put bot in usergroup base on it's IP , googlebot and yahoo slurp have constant IP

Welshy2008 04-11-2009 03:56 PM

So Gentlemen,

What happens when your board is open to the public (Full view or limited views before registering)? I mean, SE's rank it and it's contents.

Should you decide later on that you wish to Stop those views for a little while, What happens then? Is that against SE terms etc?

Genuine Question.

Welshy2008 04-11-2009 03:58 PM

Until I know for certain, I will not be installing it.

But - nethertheless - A great mod, Congratulations. And Good Luck to all who use it. :)

drupy 04-11-2009 07:29 PM

Installed. Thank you!

mpoorrajab 06-18-2009 03:42 PM

ty very much Ziki https://vborg.vbsupport.ru/external/2009/10/7.gif

https://vborg.vbsupport.ru/external/2009/10/5.gif

Orakk 06-19-2009 03:54 AM

1 Attachment(s)
Quote:

Originally Posted by hscorp (Post 1726698)
ziki i really appreciate this mod and i hope it work

but it doesn't look to work with me

see the wol pic and setting pic

http://img156.imageshack.us/img156/7494/01ti9.jpg

http://img156.imageshack.us/img156/2946/02bt4.jpg

I'm guessing you'll be going to get a google.bot added to the spiders_vbulletin.xml so you'll need block them via .htaccess but for those bots on the list we use a separate spidieDungeon.group (id:101) with zero permissions.

TheSupportForum 08-11-2009 02:00 PM

i am currently using tubenow link checker, and downloaded this mod that is needed for it to work

if anyone has got this and tubenow link checker works please advise me, i followed everthing on www.tubenow.net website and it dont work

here what happens
http://tubenow.net/link-checker/?url...thread.php?t=8

http://www.tutorials4you.co.uk/images/Untitled.jpg

i get the following message
Error: Unable to find download links.[ Try again ]

Manoel J?nior 08-11-2009 10:35 PM

Me too!

:(

final kaoss 10-28-2009 12:03 AM

<a href="http://tubenow.net/link-checker/?url=tutorials4you.co.uk/showthread.php?t=8#" target="_blank">http://tubenow.net/link-checker/?url...hread.php?t=8#</a>

Try removing the pound sign

TheChaosFactor 10-28-2009 09:13 AM

For all those who read this thread, as I did, and were still unsure of what to think, I bring you:

Quote:

Cutts explained that when Google's crawlers visited a BMW page, it saw blocks of text with repeated key search words such as "neuwagen," which means "new car" in German. However, when a user visited the listed page they would be automatically redirected to another page with less text and more pictures, which was more attractive than the page the crawler saw, but would have scored lower in Google's PageRank system.

"This is a violation of our Webmaster quality guidelines, specifically the principle of 'Don't deceive your users or present different content to search engines than you display to users,'" Cutts' blog said.
Full article:

It took BMW about a week's time, and who knows how much to their team of lawyers, to get re-listed. Don't think you'll get nearly the same treatment without being able to blame an SEO company, and have some lawyers do some fancy talk for you. Google is srs bsns.

And I'm not hating. I couldn't have written a mod like this myself if my life depended on it. :cool:

Marek58 12-02-2009 01:52 PM

1 Attachment(s)
Polish version

Meola 12-10-2009 07:00 PM

I know, phpBB has an ability to define different permission for the Bots and guests, and I've never heard any phpBB forum has been banned by Google yet.

mxracer95 12-16-2009 03:04 AM

Quote:

Originally Posted by TheChaosFactor (Post 1907335)
Cutts explained that when Google's crawlers visited a BMW page, it saw blocks of text with repeated key search words such as "neuwagen," which means "new car" in German. However, when a user visited the listed page they would be automatically redirected to another page with less text and more pictures, which was more attractive than the page the crawler saw, but would have scored lower in Google's PageRank system.

"This is a violation of our Webmaster quality guidelines, specifically the principle of 'Don't deceive your users or present different content to search engines than you display to users,'" Cutts' blog said.

That supports the interpretation that this mod is OK.

It seems obvious to me that Google's intent is to ensure people don't mislead search engines with content that doesn't exist, such as when you click on a google result you get taken to a malicious website, not the website and content you wanted to see.

Locking content behind a username/password is not changed, malicious or misleading content.

As was mentioned earlier in this thread, newspapers and television stations regularly do this. A news story will be visible in a search result, but when you click the link to the website, you can't see it unless you register and log in.



That said, I have a question about this mod.

My website requires users to log in. This mod allows you to assign search bots to a usergroup, but does it bypass the login requirement? In otherwords, does it show bots private forums without them logging in?

mohammadxxx 02-10-2010 05:59 PM

Hi

I installed this product and change all you say .

#
ACP -> vBulletin Options -> KX - Spider -> Select yes on view threads.
#
ACP -> Style Manager -> Your Template -> BB Code Layout -> bbcode_code
and
# accomplish this change too :
edit "/includes/xml/spiders_vbulletin.xml"


in topic, above the links show HyperLink Check Download Links , but when i click on Check Download Links in new page (on your website) it don't check any of my links in my topic .
why ?
please help

kind regards
mohammadxxx

Alfa1 02-17-2010 04:09 AM

Google specifically asks webmasters to give Googlebot access to restricted pages. I have seen this at various Google webmaster blogs and webmaster help pages. However, you must show the whole access page/thread to the visitors. Any other pages may be restricted from the visitors.

See here:
http://googlewebmastercentral.blogsp...eb-search.html

Quote:

Webmasters wishing to implement First Click Free should follow these guidelines:
  • All users who click a Google search result to arrive at your site should be allowed to see the full text of the content they're trying to access.
  • The page displayed to all users who visit from Google must be identical to the content that is shown to Googlebot.
  • If a user clicks to a multi-page article, the user must be able to view the entire article. To allow this, you could display all of the content on a single page—you would need to do this for both Googlebot and for users. Alternately, you could use cookies to make sure that a user can visit each page of a multi-page article before being asked for registration or payment.

First Click Free is also what webmasterworld.com is using.

I hope this clears it up.

Ziki would you consider to add First Click Free functionality?

Ricsca 04-17-2010 12:34 PM

for vbulletin 4.0.x version?

ibanez32 05-29-2010 11:06 AM

Need VB4 verssion, thx in advance

MadK 06-11-2010 12:30 AM

Likewise, looking for vB4 version! ;)

hawa.ly 06-12-2010 03:19 PM

thank you

Rick7C2 07-18-2010 07:14 PM

Need this for vb 4.0.x

ins3 08-15-2010 12:05 PM

Also need this for vB 4.0.x

Brandon Sheley 08-15-2010 12:58 PM

allowing spiders access to content guest can't see = cloaking = google ban
Cloaking, sneaky Javascript redirects, and doorway pages

Quote:

If your site contains elements that aren't crawlable by search engines (such as rich media files other than Flash, JavaScript, or images), you shouldn't provide cloaked content to search engines. Rather, you should consider visitors to your site who are unable to view these elements as well. For instance:
stay away from this mod ;)

Greight 09-03-2010 07:50 AM

Quote:

Originally Posted by Loco.M (Post 2084217)
allowing spiders access to content guest can't see = cloaking = google ban
Cloaking, sneaky Javascript redirects, and doorway pages



stay away from this mod ;)

Does it talk about illegality of asking people to register on your page for free, then showing them the page? I think we should put a stop to this cloaking idea r lets say nonsense. Spiders separate permissions is something standard and working in phpBB and as said above no one ever never has heard of a page of phpBB based forums to be blocked by any search engine.

BTW waiting for vB 4 version impatiently.

BirdOPrey5 09-04-2010 06:02 PM

It seems very clear to me there is valid use for this mod and potential for abuse. If you use this mod to restrict what googlebot can see you're 100% in the clear. If you use this mod to give googlebot access to private forums then you are playing with fire.

That said I don't care about googlebot, I'm wondering does anyone know how I could use this mod to give only the google adsense bot access to "private" forums? Basically I want ad-sense to act like a logged in registered user, this is not against any TOS, in fact it is encouraged. Google's own authentication/log in system doesn't seem to work with vBulletin.

josantos 09-08-2010 03:08 PM

I?m waiting for vB 4 version impatiently too :up::up:

Super-six 10-13-2010 06:42 AM

Quote:

Originally Posted by eJM (Post 1681235)
If you want to have a debate about what Google and other search engines like and don't like, start a new topic.

You should follow your own advice and take the depression-fueled
moral & ethical crusade with you. This wasnt your thread.

Quote:

Originally Posted by eJM (Post 1681235)
I'm taking a wild guess here that, based on the content of your avatar and signature, you're prob'ly around a teenager. I don't know how much experience you have as a webmaster, but mine goes back to about the time you were born (in other words, in the teens). I'm old and I hope I am wise in some matters. My personal rulZZZ: don't argue with children. I'm done with this conversation.

Quote:

Originally Posted by eJM (Post 1681235)
I am prejudiced against children. I don't like them. So sue me.
Jim

Wow. Who's the "child" here? :erm:
Its you that actually sounds like a toddler on a rampage.

onealien 10-13-2010 11:24 PM

i am trying to install the linkchecker add-on...in the instruction text doc is says to first add this mod....wat do they have to do with each other?

is there any other link checker add-ons that someone can recommend where you do not have to mess with all of this?

thanks,

ouch 12-06-2010 10:38 AM

is there a VB4 version of this

Ricsca 03-20-2011 11:54 PM

Foru vbulletin version 4 ?

madshark 05-25-2011 03:14 AM

It works fine with VB4 from what I know.


All times are GMT. The time now is 01:34 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01665 seconds
  • Memory Usage 1,852KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (21)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (2)pagenav_pagelink
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (40)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete