vbArchive - Search Engine Indexer for vBulletin

[high]vbArchive v1.3 Released[/high]
240,000 Google pages (and counting...) indexed so far. Congratulations to all users!

WHAT'S NEW IN VERSION 1.3:
I added the number of threads and posts for each forum.
The problem with a static page like the main archive page (as well the category ones) was that it never changed.
Now, every time a crawler visit the page, it will see new elements changed, since the number of threads and posts will always change.
Take a look at my archive, until FireFly updates the vBulletin.org one.

I also added a new meta tag for crawlers:
[high]<meta name="robots" content="index,follow">[/high]

To upgrade from 1.2, simply upload the new archive.txt and forumdisplay.txt files.
Then REVERT to original all templates and run the NEW installer script to un-install/install the templates.

[high]IMPORTANT[/high]
If you already installed version 1.3, check the "archive_forumtitle" template and see if you have there a variable called "$archiveurl".
If you do, clean your browser temp files and redownload the package, it was a mistake I made in the code.
The file is now updated with the new code.

Not convinced this is a [high]good[/high] script? A picture and it's Google results is worth 1000 words (thanks xiphoid).
Is funny that few people rated [high]1[/high] my script, we wonder who they might be?

[high]DEMO WEB SITE:[/high] vBulletin.org Archive (Google Results for [high]TeckWizards.com Archive[/high])
ESTIMATED INSTALL TIME: 2 minutes

Script Information
IF YOU WANT TO READ [HIGH]fastforward's[/HIGH] EVALUATION ABOUT THE TECHNIQUE USED, READ MORE HERE.

This script will install the Search Engine Indexer Add-On for vBulletin.
Is the little brother of vbHome (lite) Archive Add-On.

[high]NOTE[/high]
The script uses Apache's [high]ForceType[/high] directive. Most of Apache servers have it installed by default.
Check with your host to make sure the module is installed.
If you use another server then Apache, the script will NOT work.
The only solution I found for IIS is the ISAPI Rewrite module.

The script uses only 6-12 queries, depending on what page you view, and it works with any [high]2.2.x[/high] vBulletin version.

[high]Some of it's cool features are:[/high]
- vBulletin 3 style
- listings based on forum/thread permissions
- forums architecture followed [example]
- classic .html extension usage
- dynamic meta tags (unique for each forum/thread/post, [high]extremely important[/high] for good indexing)
- no broken links while using no_permission functions
- navigation bar
- multiple pages (200 threads or 100 posts per page) [example]
- template based, so you can edit it's look easy
- installer included

You will ask: So what the script does?
It makes all your forum URL's as search engine friendly, so they can be easy indexed by all search engines.
For example, the URL:
[high]forum/forumdisplay.php?forumid=9&daysprune=365&sortorder =&sortfield=lastpost&perpage=25&pagenumber=2[/high]
will look like:
[high]forum/forumdisplay/f-9-p-2.html[/high]
That will allow any search engine to index properly all your forum contents, in no time.

[high]IMPORTANT[/high]
Do NOT get "creative" and start adding crazy stuff (popups, etc.) and links to the actual templates.
The most 2 important things for your pages are:
1. good meta tags
2. clean html code that won't upset the crawlers

The script was optimized to perform at it's best the way it is now, so crawlers gravitate only onto the archives files, not outside.
You can edit the [high]archive_homekeytag[/high] to enter your web site key words.

The link to forums page [high]is needed[/high] (image logo), because some search engines might consider this as URL cloacking, if you don't link it back to your actual forums. Don't worry about the rest, if you performed the first 3 steps in [high]Forums Optimizations[/high] (listed below), they will go back and forward to the archives, without any problems.
Also, follow the readmefirst.htm instructions carefully.

Upgrade from previous version (lower then 1.2)
Estimated time for uninstall-install process: 5 minutes
Follow these steps (clear your browser temporary files before you download the new file):
1. Revert to original all archive templates.
2. Run the OLD Installer script and un-install the script components.
3. Follow the NEW instructions in the readmefirst file, included in v1.2 package.
NOTE: Overwrite the OLD code with the NEW one, in functions.php file.

Other similar scripts
Those scripts are alternatives to my code. Have your pick for the one it suit better your taste or forum performance.
SkuZZy's vB Easy Archive - another script coded by Xenon
fastforward's Spider Friendly URL's - it uses the mod_rewrite

Forums Optimizations
You MUST perform also some the mods listed below if you want your forums optimized properly for search engines indexing.
Steps 1 to 3 are [high]vital[/high], the rest is optional.

1. TO STRIP THE [high]sessionhash[/high] FROM TEMPLATES (ONLY FOR CRAWLERS), READ MORE HERE.
2. TO BLOCK CRAWLERS GO TO CERTAIN PAGES, READ MORE HERE.
3. TO LINK EACH FORUM/THREAD DIRECTLY TO ARCHIVE FILES, READ MORE HERE.
4. TO DISPLAY NICE LOCATIONS, THE FIX FOR [high]online.php[/high] FILE IS HERE.
5. TO DISPLAY CRAWLER NAME INSTEAD OF GUEST ON [high]FRONTPAGE AND ONLINE PAGE[/high], READ MORE HERE. (mod by Inphinity and xiphoid)
6. TO DISPLAY CRAWLER NAME INSTEAD OF GUEST ON [high]ONLINE PAGE only[/high], READ MORE HERE.
7. IF YOU WANT THE MAIN ARCHIVE FILE TO HAVE A [HIGH].php[/HIGH] EXTENSION, READ MORE HERE.
8. TO CHANGE THE [high]threads/posts per page[/high] NUMERIC VALUES, READ MORE HERE.
9. TO DISPLAY THE SMILIES AS [high]image parsed[/high], READ MORE HERE (mod by Logician).

[high]IMPORTANT[/high]
Kill crawler918.com! READ MORE HERE.

Other Users Demo's
Feel free to post your archive link, so I can display it here.

TeckWizards.com Archive
eva2000's Anime Boards archive
overgrow's Edge Forums archive
glenvw's Yes-Its-Free archive
codeweb's Code Webs archive
xiphoid's Open Forum archive
Hwulex's Xaprief archive
BiggieSwolls' Steroidology archive
GearedUp's FitnessGeared archive
saint_seiya's VG City archive

Search Engine Submission
You should follow these guidelines to get listed in every major search engine:
(also [high]visit[/high] those forums for more information)

DO NOT CHEAT
- do not use URL cloaking
- do not use automatic search engine submitters, do it manually
- do not use 1 pixel images to link your archive file
- do not make invisible your link text, by masking it with the same background color
- do not use 1-4 pixels text at the top of your page, to display the site contents
- do not link your archives file to an image without using the [HIGH]alt=""[/HIGH] tag

GOOGLE INDEX STATUS
To see how your site is doing, related to links, go to Google Web Site and type:
site:[high]yourwebsite.com[/high] archive

GOOGLE FACTS
1. Google uses a crawler named Googlebot which crawls the web approximately every thirty days.
2. It is not necessary to submit any page to Google. If you do submit, submit only your most important page to this search engine.
3. Googlebot is a deep crawler and should crawl all of your pages.
4. Google supplies ranking results for placement in Netscape Search, the ODP, Anzwers, Yahoo! and Ilor.
5. Google can crawl pages in ASP, JSP, CFM, PHP, Excel, Microsoft Word, newsgroups, PDF and PostScript files, Power Point and Rich Text formats.
6. Google loves sites with a high number of legitimate, relevant incoming links.
7. Google hates spam.

GOOGLE TECH SUPPORT E-MAIL [LINKS ARE [HIGH]"DROPPED"[/HIGH]? NO]
If your site is new, or hasn't shown up in Google for long, it may because our "fresh crawl" (which runs each day) was finding your site instead of our main crawl (which runs about once a month). Our "fresh crawl" is a newer feature, and we're still experimenting with which pages to crawl, how deeply to crawl, etc. We even reserve the right to (gasp!) not do a fresh crawl on some days because we're doing tests or reviewing new code. Someone wrote in recently and said "my site got in Google three weeks ago, and you've dropped me four times!" Nope, it's just that we don't always crawl the same pages in our fresh crawl, and we don't always crawl to the same depth. As we do a full crawl of the web, we find most of the sites from our fresh crawl and put them in our regular index. My advice on our fresh crawl is to view it as a nice "bonus" on top of Google's deep index. Users can always search our full index, but sometimes we can serve up even fresher pages as an extra nicety.

What does this mean for the average webmaster? In the word of the great Hitchhiker's Guide, "Don't Panic."
Just do the normal things you should do:
1. Create a great site.
2. Submit your site to Google on our "add url" form.
3. Get a link from the Open Directory Project or other directories (Yahoo, etc.).
4. Don't panic if your site takes a little while to show up in Google. Be patient, and start to look around the web--there's lots of great advice about improving your site for users and search engines.

Hope this helps,
xxxxxxxxxx

RECOMMENDED SEARCH ENGINES
1. Google - The largest and best index at the time.
Submit your link here.

2. Inktomi - This is the database that feeds iWon, 4anything, AOL Search, HotBot, GoTo, ICQ, LookSmart, MSN Search & Snap.
Submit your link here and here.

3. Fast / AllTheWeb - This Norwegian index is almost as good as Google.
Submit your link here.

4. AltaVista - Still one of the big guns, despite its temperamental ranking system.
Submit your link here.

5. Walhello - Mysterious new index with great results. Get listed, it is on its way up.
Submit your link here.

6. Non-English Indexes - The people of these countries use their own search engines. It helps if your site is in their language, because they will be searching for keywords in their own language.
Caloweb France - Submit your link here.
Caloweb Germany - Submit your link here.
Caloweb Spain - Submit your link here.

DEAD ENGINES
There is no point trying to submit to these search engines:
Excite - dead, now uses pay-per-click results
Direct Hit - will be retired shortly
Northern Light - no longer available to the general public
Lycos - now use AlltheWeb's index

Once you done all this, watch the incoming traffic that will arrive to your site.
Good luck.


[high]Copyright Permissions[/high]
1. You ARE NOT allowed to REMOVE or MODIFY the copyright text at the bottom of the page.
The copyright MUST be in a distinctive color and easy readable by visitors.
2. You ARE NOT allowed to ALTER in any way the URL links listed in the copyright text.
The Search Engine Indexer link pointing to TeckWizards.com MUST stay intact.
You can remove ONLY the vBulletin version or replace the direct link to vBulletin site with your referral link.
3. You ARE NOT allowed to DISTRIBUTE the contents of downloaded .zip file.
4. You ARE NOT allowed to COPY ANY PARTS of the code and use it for distribution.

[Gutspiller]

Where does the VBarchive.php script put it's contents? I'm working on tweaking some text sizes and I can't find where the script put the content. Can somebody please help me out?

Also can anybody please help me with a way to either get smilies / icons to work or to just should the text. Just not the redXs.

Please help.

Thanks.

[michaelab]

All I get when I go to my archive directory is a 404 Not Found :ermm:

See here: http://www.zerogain.com/forum/archive/

My server uses Apache and the mod_mime module (containing ForceType) is installed. Any ideas??

Don't know if it's relevant but my host has something in the FAQ that goes like this:

By default .php files are parsed as php4 on Oneandone Servers.

However there are still a lot of .php3 scripts out there, as standard these will not work. What you need to do is either create a new .htaccess file or edit an existing one.

Please create a .htaccess file in your main directory containing the following:

Create a .htaccess file in your main directory containing the following:

Upload this file into this file into the root of your domain

The changes take effect immediatly.

I've tried changing the ForceType directives to use x-mapp-php4 instead of x-httpd-php but that didn't make any difference. I also tried having an the vBarchive .htaccess file in my root - no joy

Are there any indexers that don't require ForceType?

Michael.

[Gutspiller]

07-19-03 at 07:28 PM NexDog said this in Post #667
No red X's here. I just see the text representation of the smilies. You can add a header and a footer in th email archive template.

In this thread: http://www.theforumz.com/forumz/showthread/t-58064.html there are read Xs. Because those smilies are part of html code. I need it to not allow the <img tag in the archive in the threads part. Also, it looks like I'm getting some header error code here at home on XP. I just checked it today at work on 98SE and it looked fine... I think.

Anybody help me please. :dead:

Oh yeah, and I checked it in Netscape and it's completely fookered.

[jc2002]

I just installed the hack and now my vbhome lite homepage gives an error message:
Fatal error: Cannot redeclare archive_nopermission() (previously declared in /home/bbschine/public_html/global.php:378) in /home/bbschine/public_html/bbs/admin/functions.php on line 2447

Can someone help??? I searched the functions.php file and did not find archive_nopermission() declared there.

[Andreas Raht]

07-08-03 at 07:26 PM Onkel_Tom said this in Post #647
Hi,
I got the problem that a 404-error comes up when goint to the /archive file.
I asked my Provider to check the force directive of Apache and they mentioned I have to change my .htaccess file as follows and now it works.

I had to replace:

PHP Code:

<files archive>
  ForceType application/x-httpd-php
</files>

<files forumdisplay>
  ForceType application/x-httpd-php
</files>

<files showthread>
  ForceType application/x-httpd-php
</files> 


with:

PHP Code:

<files archive>
  ForceType x-mapp-php4
</files>

<files forumdisplay>
  ForceType x-mapp-php4
</files>

<files showthread>
  ForceType x-mapp-php4
</files> 


It's just a hint for other persons which has the same problem as me.

Now it work fine Thank you for this great hack !

THANK YOU VERY MUCH!!!!

[gsk8]

Pls forgive me if I missed this in the long thread. I have a forum called "admin" that I have password protected w/htaccess.

Obviously I don't want the contents and threads crawled, so is this sufficient?

User-agent: *
Disallow: /forum/forumdisplay.php?s=&forumid=16

[Gutspiller]

Can somebody please help me with my problem? grrr

Where does the archive script put the information like the text size, thread background color, etc.?

In this thread in my archives you can see that the background of the threads are the wrong color for some reason as well as html image tags not showing up. Can somebody please help me get it so that the images either 1) show up and display the picture or 2) only show the code to the image, but not the red X.

If nothing else, is it possible for me to edit the vbarchive.php file and then reinstall over the top of my existing to get the right setup I want?

Please help. I've been trying to get help on this hack for a good solid month.

[Kaelon]

So, it has been several weeks since I added my site (www.domain.com/forums/archive) to Google, but I still don't have any of the Archive threads themselves showing up in Google.

Is this normal? Did I do something wrong with the submission process?

[Till]

Yesterday at 04:46 PM Kaelon said this in Post #678
So, it has been several weeks since I added my site (www.domain.com/forums/archive) to Google, but I still don't have any of the Archive threads themselves showing up in Google.

Is this normal? Did I do something wrong with the submission process?

No, perfectly normal. It sometimes takes weeks. Just link your archive and submit it to the search engines.

[jdmuniverse]

I've installed this and it works great! Thanks!

However, when I go to click on #3 addon, it comes up with this error :

3. TO LINK EACH FORUM/THREAD DIRECTLY TO ARCHIVE FILES, READ MORE HERE.

Fatal error: Maximum execution time of 15 seconds exceeded in /www/vbulletin.org/htdocs/forum/admin/functions.php on line 1244