Go Back   vb.org Archive > vBulletin Article Depository > Read An Article > General Articles
[an article] [vB 2 - 3] Getting better results from search engines
Brad
Join Date: Nov 2001
Posts: 4,765

 

Show Printable Version Email this Page Subscription
Brad Brad is offline 07-05-2003, 10:00 PM

Search engines, how they index your forum; and getting better results from them.

This article is a brief overview of search engines, how they work and what you can do as an admin to improve your results on sites like google.com

This thread is open to replies so fell free to if you have any questions.

What are search engines and 'bots'?

Search engines are your friend, without search engines there would be no way to do research online. Search engines also help you (the webmaster) by bringing in hits.

Search engines use things called bots to 'crawl' your site, these bots visit your website and index them so the info from your site can later be searched via their engines.

Below are sections that explain how to get better results from bots and how to control what they crawl on your web-site.

Tip 1: Link exchange/submit your site

When bots spider sites they get around like any human, they click the links on the current page they are viewing. So, a good way to ensure more hits from a search engine bot is having your site linked from anther. This is why starting up a link exchange with sites similar to yours is always a good idea.

You may also submit your URL directly to search engines so they may index them, here are a few links:

Submit to Google
Submit to AltaVista
Submit to Inktomi
Submit to AlltheWeb


Tip 2: Meta tags

Meta tags are used to allow search engines to index your pages more relevantly. We will only be dealing with two types of meta tags in this article, 'keywords' and 'descriptions'.

Keywords are just that, common words that someone might input into a search engine that relate to your site.

A description is also just that, a description of the content hosted on your web-site.

Say for example I was building a site based on soda, my meta tags might read something like this:

Code:
<meta name="keywords" content="soda,pepsi,coke" />
<meta name="description" content="This is a site all about soda ." />
--editing the meta tags in vBulletin v2

see: this thread

--editing the meta tags in vBulletin v3

v3 includes a way to do this from the admincp (without editing the templates), the options for controlling the meta tags are under the general settings in the vBulletin options area of the admin cp.

Tip 3: Spider Friendly URL's

this section is mostly for vB v2 users

Most search engines can not spider urls with query stings in them (? and & for example), vBulletin makes use of these in nearly every url it generates. However in vbulletin the session hash causes most of our troubles, and because spiders (bots) are guest at our forums they will get the session hash in there urls (vB 2 only, v3 strips them stock).

To solve these problems two hacks have been made, one converts all urls into friendly ones, the other strips the session hash but nothing more.

Also, both these hacks will not make the pages optimized for search engines, check out the next tip for more on that.

The spider friendly links hack

strip session hash from urls - google bot only

Tip: 4: Archives

Archives are wonderful tools that make a spider bots life a lot easier! They are search engine optimized to allow the bots to index you more quickly.

For an example of an archive check out vBulletintemplates here: [click me]

Notice how that page has a simple layout, dose not use images, and uses friendly url's. All these things make this page 'optimized' for spider bots.

You can have an archive of your own by turning the option on in the control panel (vB 3 only), or installing an archive hack if you are running vB v2.

Tip 5: using robots.txt to save bandwidth

Like I have mentioned before, spiders (bots) are just like any guest. They will see everything a normal user would see including images.

But we can control what a bot can 'see' with a file called robots.txt. robots.txt is a file any spider checks for before it begins to spider the site, the spider is only allowed to do what it is told to in robots.txt

note: robot.txt is always uploaded to the ROOT public_html folder

So, with robots.txt we can dis-allow a bot from reading a file, for example the code below would dis-allow any bot from loading images on my site:

Code:
User-agent: *
Disallow: /forum/images/
Fell free to play around with this, you can also disallow files like so:

Code:
User-agent: *
Disallow: /forum/usercp.php
And multiple things at once are also accepted:

Code:
User-agent: *
Disallow: /forum/attachment.php
Disallow: /forum/avatar.php
Disallow: /forum/editpost.php
Disallow: /forum/member.php
Disallow: /forum/member2.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/private2.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/sendtofriend.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/admin/
Disallow: /forum/images/
Disallow: /forum/mod/

Tip 6: Don?t cheat!

The last thing you every want to do is cheat spiders, this can get you banned from many index's. Here are a few things you should never do.

- Hidden link to your archives
- Image link to archive without alt tag
- Do not use automatic search engine submitters
Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 07:43 PM.


Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2024, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.03475 seconds
  • Memory Usage 2,197KB
  • Queries Executed 16 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (4)bbcode_code
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)modsystem_article
  • (1)navbar
  • (4)navbar_link
  • (120)option
  • (1)post_thanks_box
  • (1)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (1)post_thanks_postbit_info
  • (1)postbit_onlinestatus
  • (1)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete