Cache Cannon - Search Engine friendly + plus archive tool.

This is a first release beta.

I had hoped to finish it before now, but the past week of vb 2.2.3 > 2.2.3b > 2.2.3c > 2.2.4 means that I have lost nearly all my spare time in upgrading boards and re-applying hacks!!! Thanks vb!

Anyhow... the cache cannon is a simple script that will loop through the database and will, in essence, splurge every post onto the docroot in the form of a static file.

In this first beta the functionality of splurging onto the docroot is intact, as is the control panel interface.

Still to be done is indexing the resultant directory structures (for the search engine spiders to follow) and prettying up the resultant file.

This is released here as a first beta only, and is for developers only to aid it's improvement.

I do aim to finish this within the next fortnight, but in light of my lost time though it wiser to just get it out there for the meantime!

Cheers

David K

PS: Should've stated more clearly it's purpose! It's designed to perform this splurging primarily to aid with sites being indexed by search engines. Because they are static files they can be indexed quicker, and you can tweak the format to enhance the hit rate.

A second purpose of it is as a static archive of your board. It you are going to upgrade... want to start over, or are just spring cleaning and would like to archive posts in a static method outside of the database, then this will also serve you.

Edited for new version on 2002-03-20 21:11 GMT

[eva2000]

Originally posted by buro9


eva2000, I shall endeavour to make sure that this does not generate files for private forums. This will be perfect for you since entering private forum id's would not be possible, since the files are static. Though it should be noted that as this will generate static files... should you later turn a public forum private then you would have to delete those files manually, hence including the $forumId in the proposed folder structure.

Proposed storage:

The folder structure...

$forumpath/archive/$forumId/$year/$month/$day

For the file names...

$postId.htm

I shall start this on Tuesday next week, and hope to have it finished by Saturday next week (I'd do it sooner, but it's my birthday and this isn't that important!).

The files will be standalone and I shall develop them with vb v2.2.2 though as I shall only be accessing user, post, thread and forum (I guess... I'll have to look at the schema) this should be backwards compatible to at least 2.x boards. Though I will only be supporting the latest version at any time.

If I run into trouble or need assistance with the schema I shall let you know.

i knew there's a reason i should use email notifications

how about having the ability to rerun a script to regenerate the static html files if you later change a public forum to private ? you could have a forumid setting in the script which you can edit to either

1. remove all static html files and recreate the static files based on new forumid settings

OR

2. remove only the static html files for the forumid which when private

would be nice to be able to have the option to set the path to where the static html files are to placed

[nuno]

any demos out there?
seems a nice hack

[buro9]

OK, one step further... I've sketched out the indexer and shall be trying to apply it early next week (massive party in Taunton this weekend means that I will be too drunk to be let near code of any sort.. generally means I look when I'm sober and spend hours unpicking each line I added!).

The basis of this goes like this:

The directory and file names will follow this naming convention

k_id

Where k = key, it can be one of these:

f = forum
p = post
t= thread
y = year
m = month
d = date, e.g. 31, as in day of month

id = primary key or data.

if k = f, then id = forumid, e.g. f_2 is forum 2
if k = y, then id = year, e.g. y_2002 is year 2002

You get the idea.

This way the indexer can be relatively dumb, but knows that anything before the underscore is the key to the data after the underscore.



The next step is the recursive index proc.

One function will simply index a single directory, based upon the above rules. Templates will be provided for the display end of the index.

Another function will wrap the 'index this directory' function... and that will simply call that function for each subfolder, this in turn is called from the 'index this directory' function, resulting in a recursive indexing.

I don't suppose I explained that well, but I've done it before in TCL and ASP, so I know how it works



Once the indexer is in place, hopefully before the end of next week, I shall ask you all to try it out and see if there are any improvements that could be made at this point. Once we're happy we'll launch version 1.



I have in mind that there will be three versions.

Version 1: Splurge and Index
Version 2: More file management, deletion of pre-indexed items.
Version 3: Possibility for more than one archive, additional templates for views (these allow you to archive for support on other platforms).

A very last feature that I'll leave someone else to consider is the possibility that the Cache Cannon could be fired in single instances when a post is made on the board... thus never requiring firing manually. But without Oracle and Triggers, I'm not sure how clean adding this feature may be... could possibly be a very nasty hack, whereas at the moment it's quite elegant and standalone.

Cheers

David K

[buro9]

New version uploaded now includes indexer.

Please test it and let me know what you think.

I have also activated the auto-redirect stuff, and put all of the meta tags into the headers.

It is pretty ready in this state... but I would like to apply the forumhome default style to all these templates (just the stylesheet and replacement variables) for the sake of prettiness... so if anyone can tell me how the hell to access this stuff i'd be grateful... spent a few hours poking around and haven't conclusively got an answer yet (don't want to just hit replacementset for the -1 default set.).

Cheers

David K

PS: Attachment is on the first post of the thread. I've just updated it.

[rawnet]

David - you are a star.

My FTP has just died, but I'll be giving this a go as soon as I can.

As well as Posts, could you consider writing out Members (along with their related userfields), so people searching for information they've entered in their userfields will come to the boards. It would be useful for me as I have a lot of user fields, but I'm sure others would benefit too!!

[fastforward]

buro9,

Nice work!

I successfully installed this tonight and test archived a few forums. I'm going to edit the templates and during the quiet periods over the weekend I'll start archiving everything.

I'm on an older version of vB so I had to make a few modifications to get the forum select dropdown working and the control panel redirect.

I also changed the .htm extension to .html. Other than that, it worked flawlessly.

Thanks.

[buro9]

Now that the major part of the application is in place, there is no reason why you couldn't add additional functionality such as caching member information too.

It would only take minor changes to add this functionality.... which would probably be best under a folder of it's own (../archive/members/) and then a file for each members details named u_userid, so as an admin you'd likely be u_1.htm.

A seperate button on the cache cannon page would fire caching of members... or it could be tied to the end of the posts firing mechanism.

Indexing would be done at the same time as the rest of the indexing.

Yeah, that all makes sense

I'm not going to start such additions though, not until what is currently there is finished to a higher spec. I'm not one of those developers who never quite finish something before moving on. My work is used widely because I do finish it properly, and I hope that is true of this too.

Remaining on Version 1 is merely the ability to create a single stylesheet *.css file and place a reference to that in each of the created files. This stylesheet I want based upon the default style for the forum... so please, anyone who knows the best way to get hold of this from within a hack... let me know

Once it's created, a change to the amount of data in the posts file (add more, such as user details for who posted), and applying the stylesheets that would've been created.

After that I'll launch version 1.

Again, plan is:

Version 1 = Splurge posts, index them.
Version 2 = File management, flush archive, etc.
Version 3 = Additional templates for prettiness, additional functionality.

That's just the order in which I feel functionality will be desired.
Each version will be a finished hack implementable safely.

Cheers

David K

[jamesdasher]

Two Changes, that have to be made

1. It is indexing Private Forums...this has got to be changed, my moderator and administrator forums can be read...I understand it is for indexing but it shouldn't be accessible.

2. Strip Smilies...you really need to strip the smilies out of the pages...they are coming up as missing images.

James

[buro9]

Private forums, I understood that it wasn't, but I shall create a private forum to test this and then stop it from doing so.

Regards the smilies, I was converting them to the images, but it's probably best to strip them totally.

I'll put both of these on my todo list.

Cheers and thanks for the feedback.

David K

[rawnet]

Thanks David. We all appreciate the effort you are putting into perfecting this.