Mysql 4 Search hack

MySQL 4 Search hack for Vbulletin 2.x.x - version 1.2 beta
Author: PineappleClock (pineappleclock@clockcrew.cc) aim: Janitor61
-------------------------------------

Overview
The MySQL 4 search hack modifies the search functionality in Vbulletin to take advantage of the new SQL functions available in MySQL 4.

Synopsis
This hack allows MySQL to index the posts and threads in your BBS instead of requiring vbulletin to do it.
This hack, when installed, will change the BBS in the following ways:

• Vbulletin will not require the 'word' or 'searchindex' tables anymore - saving lots of space - instead it will use the 'post' table as an index.
• Vbulletin will not catalog, reference and index every word of every post upon insertion into the database - saving tons of processing.
• No need to regenerate the search index ever.
• The 'post' table will increase in size by about 35%
• Searches will be considerably faster, and posting will be somewhat faster.
• Search strings will be able to contain phrases, like "lets share a soda", which can include small words that would of been blocked in vbulletin 2.x searches.
• Advanced searches, containing included words, optional words, excluded words and exact phrases.

File Listing
install.txt -- the installation howto
simplesearch.txt -- a block of code that replaces a block of code in search.php
searchtemplate.txt -- a vbulletin template used for drawing the search form.

Requirements
MySQL 4.0.1 or newer - required
vbulletin 2.x.x - required (i've only tested it on 2.3.0 though)

How to check your MySQL version:
execute the following query:

SELECT @@global.version

Notes
The installation instructions assumes that you have access to database management software, like phpMyAdmin. If you can't edit your database directly, I can make a php script that will modify your database if there's enough demand.

The Vbulletin team aren't going to put in support for MySQL 4 until version 3.1 - so this hack should be obsolete by then.
If you'd like to see the modified search engine, go here:

http://www.clockcrew.cc/bbs/search.php

I was originally going to distribute the whole modified search.php file, but after reading the rules, im only distributing the modified part. Please let me know if you spot any bugs, etc. I've tested this, but there still are probably bugs i need to work out.

update [Apr 17]: added instructions on uninstalling

good luck!

[jbourke]

The mysql docs state that anything 3 or less characters is not indexed.

Jim

[mute]

Well there you have it, thanks Jim!

/me rtfms

[jbourke]

Hmmm...this hack only returns the first few pages of results.

I'm going to have to revert to the old search engine if there isn't a way to fix this.

Jim

[jbourke]

These two lines:

PHP Code:

    $posts=$DB_site->query("SELECT postid FROM post WHERE MATCH(title) AGAINST ('$querywc' IN BOOLEAN MODE) ORDER BY dateline DESC LIMIT 25"); 


and

PHP Code:

    $posts=$DB_site->query("SELECT postid FROM post WHERE MATCH(pagetext) AGAINST ('$querywc' IN BOOLEAN MODE) ORDER BY dateline DESC LIMIT 100"); 


both have limit statements.

Is there a problem with taking these out?

I don't mind experimenting but I'm a bit nervous because I have a pretty large forum.

Jim

[Brad]

Jim, increse them as you see fit, if you think you've gone to high revert them.

[Da`Nacho]

A few problems I am noticing, please someone verify with their setup:

When viewing a thread contained in search results the search terms are only hilighted on the first page of the thread.

When searching for a PHRASE and viewing a thread contained in those results, the search terms are not hilighted at all in the results and when you click to visit page 2 in the thread it reloads the first page and the &hilight part of the URL is empty.

I don't recall these things happening with the original search schema.

[Da`Nacho]

Gah it's driving me crazy... somehow spaces or something are being inserted in the query string, thats whats causing the screwups but I don't understand where it's coming from.

IE a normal one word search will end up having a + behind it... hard to explain.

If I remove the hack it's totally fine...

I got so frustrated that I totally reinstalled a clean copy of vBulletin 2.3.0 and reperformed the hack - same results.

[Tigga]

Just installed the hack and it seems to be working pretty good. Searches are much better, and posting does seem a little faster.

I can confirm Da`Nacho's problems though. It doesn't seem to highlight the words, and when you search for an exact phrase, the page numbers no longer work. It seems the link to page 2 (or any other) on showthread.php is somehow missing the "pagenumber=X" part of the url. After you search and go to the thread though, if you take out the "&highlight=xxxx" in the url and reload the page, it seems to work fine.
Other than that though it seems to work great.

[Da`Nacho]

Today at 05:31 PM Tigga said this in Post #38
Just installed the hack and it seems to be working pretty good. Searches are much better, and posting does seem a little faster.

I can confirm Da`Nacho's problems though. It doesn't seem to highlight the words, and when you search for an exact phrase, the page numbers no longer work. It seems the link to page 2 (or any other) on showthread.php is somehow missing the "pagenumber=X" part of the url. After you search and go to the thread though, if you take out the "&highlight=xxxx" and reload the page, it seems to work fine.
Other than that though it seems to work great.

Yeah, I spent 5 hours last night redoing and redoing this hack and I just dont' know what the problem is. Anyone who wants to see what we mean can visit www.candidforums.com and use the search (it's enabled for guests).

We are running MySQL 4.0.12-0 and php 4.3.1 on Apache 1.3.27.

Anyone have any clues?

[Tigga]

Well I can tell you where the problem is coming from... After you perform a search, if you hold your mouse over the thread, you'll see it's linking to something like:
http://yoursite.com/forum/showthread.php?s=&threadid=x&highlight="what+you+searched+for"
The " marks in the url seem to be what's causing the problem. When you click on the thread it replaces the " marks with %2B in your browsers url. If you remove the %2B from the url then it seems to work as it should. Not sure how to fix it yet, but hopefully that gives traekwon a little more insight as to what the problem is.

I also just noticed another strange problem that seems to be coming from the %2B being in the url... After I performed the search and went to one of the threads, a smilie image there had border="0" alt=""> after the image. When I viewed the source there it has this for the html code...


<td bgcolor="#160056" colspan="2" align="left" valign="top"><p><font class="nf">he saw 'the ring'<br /><font color="red"></font>
<font color="red"></font><br /><font color="red"></font>
<font color="red"></font><img src="/forum/images/smilies/frown.gif"<font color="red"></font> border="0"<font color="red"></font> alt="<font color="red"></font>"><font color="red"></font></font></p></td>

Obviously it's trying to highlight quite a few things there that it shouldn't be.