Sphinx Search

[kontrabass]

I'm going to get some guys on this - VB4.0's search is way too slow for our forum of 5M posts. It'll take us some time though, we just started playing around with 4.0 in a test enviro. Floren's commercial sphinx solution (Searchlight) will have 4.0 support in the future as well.

[RedWingFan]

We have close to 4.5 million posts, and even in our private test board with only around 170,000 posts, search drags. I can't see going back to MySQL fulltext search any time soon. (Plus, there are too many other issues with 4.0 that will keep us from upgrading for quite a few months.)

[snakes1100]

Originally Posted by RedWingFan

We have close to 4.5 million posts, and even in our private test board with only around 170,000 posts, search drags. I can't see going back to MySQL fulltext search any time soon. (Plus, there are too many other issues with 4.0 that will keep us from upgrading for quite a few months.)

I'd guess it's those Sheep wondering over from Novi & clogging your server up w/ wool!

[RedWingFan]

Originally Posted by snakes1100

I'd guess it's those Sheep wondering over from Novi & clogging your server up w/ wool!

True dat!



--------------- Added [DATE]1262125966[/DATE] at [TIME]1262125966[/TIME] ---------------

P.S. I'd blame the Steelers, actually--our server is in Pittsburgh. It's just the boneheaded server admin who lives near the Sheep...erm, Whalers, Dead Wings, etc.

[eoc_Jason]

Yeah, but for most of us Searchlight is not very affordable...

I'm still super pissed at Jelsoft for not having any sort of sphinx plug-in / switch for vB 4.0... From all the previous development talk I was led to believe they were going to implement it (finally)...

It's like they don't really care about the big boards... They are implementing all these other crazy features that maybe 0.2% of people *might* use, but not taking a hard look at the core of their product... *sigh*

I'm about to install vB 4.0 on my test server. At the very least I'm going to try and code sphinx for the basic search stuff. To standardize things I'm going to try and code it with the PECL::Sphinx library.

[RedWingFan]

Originally Posted by eoc_Jason

Yeah, but for most of us Searchlight is not very affordable...

I'm afraid to ask how much... We run on donation funds ONLY.

Originally Posted by eoc_Jason

I'm still super pissed at Jelsoft for not having any sort of sphinx plug-in / switch for vB 4.0... From all the previous development talk I was led to believe they were going to implement it (finally)...

I only got the idea that the search system was more "modular" now (??), meaning it would be easier to support alternate search engines like Sphinx. All I really noticed in the AdminCP was that you only have one option called "DB Search" to choose from, and there is NO clue as to whether this is a built-in vB search or fulltext or something new entirely. It couldn't be that difficult to code for Sphinx over MySQL fulltext...and to be honest, I'd rather have it done officially vs. using a third party hack or plugin. (Not knocking them, since I'm more than guilty of using many third party plugins myself.)

Originally Posted by eoc_Jason

It's like they don't really care about the big boards... They are implementing all these other crazy features that maybe 0.2% of people *might* use, but not taking a hard look at the core of their product... *sigh*

Then again, the percentage of "big boards" is small also. But those of us who need Sphinx and use it are now at the point where we can't just fall back to fulltext without possibly bringing a server to its knees. We are at that point now. Sphinx also seems to have a lot more available in terms of configuring user searches AND indexing.

Originally Posted by eoc_Jason

I'm about to install vB 4.0 on my test server. At the very least I'm going to try and code sphinx for the basic search stuff. To standardize things I'm going to try and code it with the PECL::Sphinx library.

I don't know if there's any code I can "borrow" from the current Sphinx hack I'm using. But I do know that the two deal breakers for us are Sphinx and iTrader (which gets a lot of use in our Marketplace area)...all the other plugins and template mods we're using can be easily replaced. Definitely will wait a few versions, too, since the 4.0.0 seems rushed and I'm finding at least a dozen problems with it already...

[Kevlar]

I purchased vB4 on the premise that the new search engine would work for big forums, after playing with it and inquiring with "the people" I have heard otherwise. Even the search on their own site with vB4 is slow and it is no where the size of my forum. If that is the case, I couldn't go back to vB search even if I wanted to ... my forum would come to a grinding halt.

So while I do have vB4 purchased... I'm currently in a holding pattern.

[RedWingFan]

Originally Posted by Kevlar

So while I do have vB4 purchased... I'm currently in a holding pattern.

We are as well. I put it on our private test forum so the staff could get used to it, but it'll be quite a few months before we can put it on our main forum. The only good thing is that we had an expired license, so I can at least upgrade to 3.8.x and get us current in that area.

Is it just me, or do some parts of vB4 appear unfinished? IMHO it seems like it could have gone through another RC version or two before going gold. I've got a list of about a dozen issues with it so far--really, some are very minor, but still make me hesitant to recommend it even for new applications just yet.

[Kevlar]

vB4 seems unfinished... but then again, most software the comes out these days is far from finished. Is it ready for production environments? Probably if you are within the 70% of the average users of vB. I think I (we) fall outside of the average hence why we notice all the little problems more. I would have no problem installing vB4 on other small forums that I used to run, but those were smaller unmodified cookie cutter forums... nothing like the monster I deal with now.

[eoc_Jason]

I dunno what the average size vB forum is... But you would think if you are selling a forum product, you *want* your clients to be successful. If that means they are going to hit a brick wall after a few million posts in their DB, then you are severely limiting your market. There are quite a few sites that are running very old versions of VB, opting to just patch and write custom code on their own then wait for jelsoft to come up with the official features.

FYI, Searchlight base cost is $2,000 USD...

I haven't had a chance to install 4.0 yet, but I'll check through the code to see what they did to eliminate the table locking. It's still moot though because if search results don't return within a second then people start getting click-happy or give up (both unacceptable).

Imagine if Google took 5 minutes to return every search you make? How many people would be googling?