vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)
-   -   google spiders 60% of my threads (https://vborg.vbsupport.ru/showthread.php?t=46201)

pimpingfools 12-02-2002 07:34 AM

I second that last post..

Erwin 12-02-2002 09:03 PM

Quote:

Originally posted by Velocd
What is your version of this Erwin? Is it a hack that could be released, or is it integrated very deeply into your site that to prepare a hack for it would be too troublesome?

I took a quick browse through your forums Erwin, and you did a great job with the rewrite, and how it's all organized ;)

Essentially, fastforward's hack, but applied to every page in every thread, also to memberlist, avatarlist, search pages and changed most PHP files so that sessionhash and dynamic URLs are eliminated except for latest posts etc. - too extensive to release. I don't really look forward to vB3 when I have to recode all the files again. :)

Thanks for the compliment too.

indiamike 12-02-2002 09:09 PM

Though it's a little off topic...and I just want to chime in a little bit here....
I use vbportal and google spiders my site almost everyday (usually 20 robots at a time), somtimes twice a day. It doesn't list all the posts in the search engine, at one time it did but dropped a bunch, however the big pitfall is that Google uses up around one gig of my bandwidth a month. I have a small site and that is still way to high.

I may start blocking Google just because of this....just a little warning if your inviting Google.

Cheers Anyway
Mike

NTLDR 12-02-2002 09:12 PM

Perhaps you should create a non-graphical style that is used when the googlebot visits? I'm sure this wouldn't be too hard to hack in.

SkuZZy 12-02-2002 09:42 PM

Quote:

Originally posted by indiamike
Though it's a little off topic...and I just want to chime in a little bit here....
I use vbportal and google spiders my site almost everyday (usually 20 robots at a time), somtimes twice a day. It doesn't list all the posts in the search engine, at one time it did but dropped a bunch, however the big pitfall is that Google uses up around one gig of my bandwidth a month. I have a small site and that is still way to high.

I may start blocking Google just because of this....just a little warning if your inviting Google.

Cheers Anyway
Mike

Good point being made here. I can see how some people might be concerned with this. I should point out, the archive scripts i'm releasing are very, very small in size the way they come. Of course, if you add images and styles into them, then they get much bigger. But using them the way they are, the bandwidth used will be virtually un-noticable. When it comes to getting google to archive your posts, the simpler = the better ;)

Erwin 12-02-2002 10:14 PM

You can use robots.txt to block Google or any other spiders spidering specific sections of your site or certain files like .gif etc.

Smoothie 12-03-2002 01:21 AM

Quote:

and there is no feature in it right now to block out private forums selectively. I could probably add those small features to it though
SkuZZy- This is something I need to have before using this. I only have one private forum, but it's the mod forum. Some pretty heavy stuff goes on there. I would need to block this forum.

Also, in your instructions;
Quote:

Add a link to the archive on your main forums page, otherwise google won't know it exists and therefore won't spider it!
Whats the best way to do this? Just a plain old link on forumhome? Then other members will be able to view it?

SkuZZy 12-03-2002 01:24 AM

Quote:

Originally posted by Smoothie
SkuZZy- This is something I need to have before using this. I only have one private forum, but it's the mod forum. Some pretty heavy stuff goes on there. I would need to block this forum.

Also, in your instructions;Whats the best way to do this? Just a plain old link on forumhome? Then other members will be able to view it?

I'll see about adding something to block out private forums, shouldn't be too hard. About the link, just add a text link to it at the bottom of your forums or whatever. You don't need to do this, you could just add the site to google (http://www.google.com/addurl.html) but linking it on your site is the best and fastest way to get it spidered... plus it will inherit your "PR" which will help the pages rank higher.

DrkFusion 12-03-2002 01:38 AM

Doesn't Xenon's Hack block all private forums to guests, and normal members.

Smoothie, just add a link at the bottom :)

Smoothie 12-03-2002 01:55 AM

Ok, probably a dumb question, but after I add the link to my forums, what link do I submit to google?


All times are GMT. The time now is 08:34 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01020 seconds
  • Memory Usage 1,740KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (5)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete