Sphinx Search

[snakes1100]

Confirmed as working on vBulletin 3.7.2 Patch Level 1

cron:
postdelta - 5min
threaddelta - 5min
reindexed nightly

Seem to ran into one issue tho of only returning 500 results when searching for posts by a user with 15k, found a hard limit set to 500 in sphinx_search, raised that limit to 20000 to match everything else.

Odd enough it now returned only 1000 results, which still isn't right, as there is no other limit set to 1000, will have to chk it out some more.

Full index & restart didn't help.

Note, found the 1000 hard limit set in sphinxapi.php

New search now returned 7100 posts by that user, missing about 8k in posts, back to the drawing board.

Showing results 1 to 25 of 7074
Search took 0.52 seconds.

6 million post board w/about 2500 online.

Add-on:
Seems i may have found a bug someplace, ill post part of the error, if orban or anyone wants the full error msg i have it saved, but doing a search for a keyword that has only about 3500 results produced a duplicate key entry and a db error page, after, the browser reported a script still running error and wouldnt let me refresh the page, FF stopped the script and then a refresh cured the db error page, doing the search again for the same keyword resulted in a good search & no error.

Invalid SQL:

				INSERT INTO search
					(userid, titleonly, ipaddress, personal, query, searchuser, forumchoice, 

/*prefixchoice,*/ sortby, sortorder, searchtime, showposts, orderedids, dateline, searchterms, displayterms, searchhash, 

completed)
				VALUES
					(641509,
					0,
					'',
					1,
					'bollywood',
					'',
					'',
					'lastpost',
					'DESC',
					0.10879, 0,

### SAVE ORDERED IDS TO SEARCH CACHE ###;

MySQL Error   : Duplicate entry '0089875ee33efd9d611cfff6ca28d9e4-lastpost-DESC' for key 2
Error Number  : 1062

[HFB]

It appears that Sphinx will be included in vB 4.0
http://www.vbulletin.com/forum/showp...92&postcount=7

[RedWingFan]

Originally Posted by RedWingFan

Not random. If I choose "View Results as Posts", I get the errors. If I view results as threads, no errors.

Warning: strpos() [function.strpos]: Empty delimiter in [path]/includes/functions_search.php on line 829

The proper search results do display below, however...

More info, if anyone cares to debug this. For now, I just put error_reporting(0); as the first line in includes/search_functions.php to kill the error message. It doesn't cure the problem, but at least removes a small amount of end-user panic.

Here is how to duplicate it:

1) Use no search terms
2) Enter a valid username
3) Choose "Threads started by user", and check Exact Name
4) View results as posts

The function's affected lines are here (strpos is the affected line):

PHP Code:

// ###################### Start process_quote_removal #######################
function process_quote_removal($text, $cancelwords)
{
    $lowertext = strtolower($text);
    foreach ($cancelwords AS $word)
    {
        $word = str_replace('*', '', strtolower($word));
        if (strpos($lowertext, $word) !== false)
        {
            // we found a highlight word -- keep the quote
            return "\n" . str_replace('\"', '"', $text) . "\n";
        }
    }
    return '';
}
// 


So, I'm thinking the $cancelwords array is empty when being passed to this function. (Follow my logic?) Not knowing enough about this or the vB search function, or the Sphinx file I uploaded, I don't know where I'd begin troubleshooting this. The function is used at around line 2817 in search.php:

PHP Code:

            $post['pagetext'] = preg_replace('#\[quote(=("|"|\'|)??.*\\2)?\](((?>[^\[]*?|(?R)|.))*)\[/quote\]#siUe', "process_quote_removal('\\3', \$display['highlight'])", $post['pagetext']);
//
// 


If there's something that'll work, to fix this, I'm all ears.

Beyond that, the search is working fine on our test forum...and I'm anxious to run this on our live forum as soon as I can get this ironed out.

[Jah-Hools]

Sorry for being a total dunce but..

Does the Sphinx search display its results in an "in line" VB format? (so that in line moderation actions may be applied to selected results?)

Reason I ask is - right now I have Google custom search + the regular VB search + TAG search..

Newbies probably just use the Google CS as its the first box in the drop down menu - but that doesnt give moderators any tools to work with the results and do any 'in line' moderation (merging, renaming, changing prefixes, tagging etc)

Power users probably use both..

Regular users are probably confused about what to use...

The Google Custom Search is great for searching phrases

But its not a very clear solution with the various options - its a bit irksome that I may have to write a how to search guide that advises when to use what search engine..:down:

2 + million posts - is it time for Sphinx?

Is it a good 'all in one solution with an "in line" moderation functionality?

Server speed / load isn't an issue here, my quest is purely to improve search quality for the forum members.

Thanks in advance.. Sorry if this butts in... or is OT.



--------------- Added [DATE]1219333047[/DATE] at [TIME]1219333047[/TIME] ---------------

Originally Posted by HFB

It appears that Sphinx will be included in vB 4.0
http://www.vbulletin.com/forum/showp...92&postcount=7

Holey moley!:up:

[Jon]

We just upgraded from vBulletin 3.6.10 / Sphinx 0.9.7 to vBulletin 3.7.3 / Sphinx 0.9.8 using the scripts provided here. The sort order when displaying results as threads seems to be messed up now. I believe in our old setup we had this fixed by using the sort_search_items() function on the result set. Is this function still available in vBulletin 3.7.3? If so, how do I implement it?

[eoc_Jason]

Jah-Hools - Yes, the search hack here basically uses Sphinx indexes to find the results, which are then returned to the vB search process and displayed like internal search results...

If performance isn't an issue, then probably not worth trying to implement unless you have some spare time and are bored. Yes with sphinx you can use phrases and various expressions to make a much more powerful search (read the sphinx docs online, it goes into detail about that). However, you still occationally run into the issue of vBulletin liking to filter out some search results it thinks are not relevant...

Jon - Yes that function is still in the functions_search.php file.

I just came here to download the code someone posted using the new sphinx code + tag search... If I can intergate tag searching I will be happy...

I'm still using the old code that uses a separate file and edits the search.php to make it run. I'm not sure if I'm ready to switch over or not, like I said I'm about to take a look. I had the sort_search_items in my existing code, and plan to keep that functionality in my new code. I'll post a file or instructions once I'm done. I can't believe I've waited this long to upgrade to vB 3.7.x, but I suppose it's time...

I think it's great the vB team are finally addressing their search shortcomings and integrating in alternate search technologies... It doesn't take long for a site to reach a few million posts (especially with a lot of Off-Topic sites).

[ferreo]

Originally Posted by NickCat

You guys have to remember... we aren't the majority, we are sadly the very small minority of their install base, and frankly we are their target audience any longer. We've already bought the software, we pay a measly $30 a year for upgrades, so that puts us in the category of don't fix it.

It is really sad though. We are pushing 21.2 millions posts, and without this hack the site would have eaten itself. As it was, people would have to wait an average of 15-60 seconds when someone would post a really nasty search. Towards the ends I had literally hundreds is disallowed common, but still very useful, words on the search just to limp by.

Being probably just 1%, or less, of their customer base puts us at a real disadvantage when it comes to solving issues like this.

It seems as though the development team over there is resistant to change these days. The smallest and silliest example I saw the other day was an argument on .com about the feasibility of adding a gtalk IM to user profiles. The still don't believe enough people use it to add it in. Maybe that's changed on 3.7.0, but I'm still on the 3.6 branch while I wait for the rest of our hacks we use to be ported.

I know this thread isn't for solicitation, but I know Teck's posts were removed as he was deemed to be "vending." I personally would be very interested in his packaged commercial solution if he actually develops it.

Sphinx saved my butt, and I'd like to ensure that I have the option for many years to come.

I would like to chip in to this thread, especially with the aim to provide a real customer testimonial to TECK's sphinx solution.

Our forums had been struggling with the vb-internal search at about 1 million posts, the mysql fulltext search gave in at around 2.5 million. At that point, the processes on our front ends would be just eating up resources, waiting for the responses from the sql backends, a sphinx solution had to come into play, stat. A nice chap that I found on here installed an implementation that followed the instructions from this thread, the solution was sufficient, lacked a lot of deep functionality though. I do not recall what branch I upgraded vB to in May, but the old sphinx implementation was just not cutting it anymore.

I emailed TECK first about server optimization work, as my boxes were struggling badly during peak traffic times. It turned out that he had a sphinx solution of his own, he promised a commercial solution for large vB installs. I agreed to pay a premium price for it, which turned out to be the best investment I had yet done for my business and community.

I realize that it is not within every webmaster's means to pay several thousand dollars for a custom software solution. However, considering the performance of the product and TECK's dedication to his customers, I have no regrets whatsoever. Due to the painful circumstance that my host doesn't give root to managed boxes, the install of the search and optimizing my servers was a real pain in the f***ing ass. TECK often spent long nights until 3am working to get the solution working in my tedious environment. If I am to spend thousands for code, I expect proper customer service, and TECK delivered.

Now I have a perfectly working vB install and search solution with sphinx 0.9.8, my servers have now plenty of resources and room for future growth, and our members are delighted to have a fast and accurate, rock-solid search. Money well spent.

Feel free to contact me if you have any other questions.

[Jon]

Originally Posted by eoc_Jason

Jon - Yes that function is still in the functions_search.php file.

I just came here to download the code someone posted using the new sphinx code + tag search... If I can intergate tag searching I will be happy...

I'm still using the old code that uses a separate file and edits the search.php to make it run. I'm not sure if I'm ready to switch over or not, like I said I'm about to take a look. I had the sort_search_items in my existing code, and plan to keep that functionality in my new code. I'll post a file or instructions once I'm done. I can't believe I've waited this long to upgrade to vB 3.7.x, but I suppose it's time...

Jason, I would appreciate that (as would a significant number of my members ).

And you're not alone in postponing upgrades. It took me years to upgrade from vBulletin 2.x.. I believe SA.com is still running that version. Heavily modified I'm sure, but still

[Kaelon]

I've been reading through these posts, but am wondering if there is a concise listing of the latest updates/guides to using Sphinx with vBulletin. I see that the first post was last modified in May 2008; have there been any bug-fixes/improvements on Sphinx with vBulletin since then?

[xnetco]

Has anyone had an issues with not all posts being indexed?

Tim