Search: How does VB4 "optimize" searching? Is Sphinx still required for big boards?

[RedWingFan]

Originally Posted by nitra1000

Don't shoot me but you could always implement google custom search...

Originally Posted by kontrabass

We've done that for years as well (in conjunction with sphinx). Unfortunately it's extremely feature limited. Search by username? Order results by thread date? last post date? Search a specific forum category? You're out of luck.

To add to what kontrabass writes: Google Search (any version) can't work properly, as it is not a real-time search directly on our database tables. Google can only search on the content it spiders over the course of time. That could be daily, weekly, whatever. And because of this, it can't support searches by username, thread title searches, date-based searches, etc.

Having said that, we DO offer a Google search option, as for some of the phrases our visitors search for in older posts, Google's results are far more accurate and usable. Google's search also gets around common stopwords: on a music forum that discusses rock music, NO built-in search via vB or MySQL can find the title of the album by The Who called "Who's Next", as both are stopwords (unless you custom-configure MySQL).

And to help with server loads, we have asked members to give Google search a try. We have it in the search drop-down on the forum as an option. But, that still precludes using any of the search functions built into vB that we know (and...umm...love). It's a good additional tool for members, but it can't replace vB's search.

Originally Posted by kontrabass

The "good" news is that table locking has been removed from the equation in 4.0.

How did they pull that one off?

Originally Posted by kontrabass

I think VB is hesitant to use Sphinx because they don't have control over it. It's not *their* product. The thinking is probably that if Sphinx for some reason becomes unsupported or ceases to exist, VB is crippled.

But I say, if they don't want to incorporate Sphinx, fine - at least create a plugin that is easy to implement, so that if their big-boarding clients have no other option, at least they can upgrade to VB 4.0 instead of moving to IPB.

%< snip >%

....... Sure, the user needs to have root server access (probably another reason VB doesn't officially support Sphinx), but what big-boarder seriously is not going to have root access???? No board with 5+ million posts is going to be on a shared $10/month hosting plan.

We don't have root access on our dedicated server, BUT...I can still install quite a few packages myself, and I successfully compiled Sphinx and installed it on our box without needing a root login. I could do this on our shared accounts at the same host, but like you say...NO board with as many posts as ours would live long on a shared account! (We're around 4.5 million posts ourselves right now.)

And for that matter, vB doesn't control vB and MySQL, yet they depend on them to make vB run. I agree with you: Sphinx support should be a vB-supplied plugin, or an option we can choose from admincp. Just like in vB 3.x, where we could choose GD or Imagemagick support based on what we had available to us. The only hiccup I can see is that Sphinx does require a bit of additional setup beyond installation, and you need access to cron (or similar mechanism) to run the indexing at regular intervals. Even there, though, admins with large forums like ours already run on dedicated hosts and know how to do this already.

It's been said before that it's not even vB so much as it is MySQL that is limiting us. One of our fellow members here on vB.org rewrote parts of vB3 to use PostreSQL and it ran far better. If the freebie forums like phpBB can offer multiple schemas, I don't see why the same couldn't have been done with vB when it was redesigned from the ground up for vB4. I would pay extra at our host to get PostgreSQL installed if I could use it with our forum.

Originally Posted by eoc_Jason

Searching is the one crucial feature of a forum (or heck, even the Internet... Google anyone?) and I am appalled at how the vB team has been putting search features so low on their priority list (if it has even made the list)... Once a forum gets to a certain size, the built-in search becomes totally unusable because of table locking...

We tried fulltext search and had to go back to MyISAM table types, and the table locking was killing us. Back to built-in vB search for us! Lesser of two evils at the time, until I decided to bite the bullet and put Sphinx on our production forum. Still not ideal, but the forum (and especially our search) became...usable.

I don't know if I'm wrong for saying this, but isn't Search the most intensive operation on the forum, and the one that brings most of our big boards to their knees? If so, then, why aren't we given more options to choose alternate search engines? Granted, us big board admins are a small minority of users, but this also can't look good to visitors who see a busy forum branded with "vB" and find that yet another vB forum drags on for minutes at a time during searches.

Originally Posted by eoc_Jason

Likewise it is not very feasible for a webmaster to get a second server exclusively as a slave search DB. Sphinx on the other hand can return results in a fraction of a second, and the program is FREE... *sigh*...

We run on donations only. We did try a second server for a database-only server, but the additional fee killed us and we almost had to shut down entirely. Fortunately a hardware upgrade saved us. But we can't afford to keep throwing additional hardware at the problem just because ONE function of the forum--search--is dragging us down, and a solution is already available and used by some big-board admins already. Sphinx just seems to solve SO much for us, and yet we still have to resort to using plugins and/or hacks that need a fair amount of tweaking to fit our individual needs.

I hate to keep posting about this, but maybe we just need to make enough noise so that Sphinx finally gets official support and easy integration with vB that we can trust.