Sphinx Search

[RedWingFan]

I have it running OK now.

Really basic (OK, stupid ) question. What do I need to turn OFF in vB to keep it from using its own internal search engine? IOW, aren't the built-in vBulletin search index tables going to be populated with the newest posts as they are made? From the way I see it, Sphinx bypasses the whole vB system, so vB shouldn't need to store any of the search terms anymore, unless another search process uses it. I've already dropped the fulltext indexes and changed the search type from fulltext back to vB internal, and am switching selected tables over to InnoDB to clear up the load issues we're having.

[amcd]

Originally Posted by RedWingFan

I have it running OK now.

Really basic (OK, stupid ) question. What do I need to turn OFF in vB to keep it from using its own internal search engine? IOW, aren't the built-in vBulletin search index tables going to be populated with the newest posts as they are made? From the way I see it, Sphinx bypasses the whole vB system, so vB shouldn't need to store any of the search terms anymore, unless another search process uses it. I've already dropped the fulltext indexes and changed the search type from fulltext back to vB internal, and am switching selected tables over to InnoDB to clear up the load issues we're having.

No.

First set the search type back to fulltext. This will tell VB not to populate the search tables (word and another one). The onus of maintaining search data now shifts to MySQL.

Then drop the fulltext indices from post and thread tables.

[RedWingFan]

I'll give that a try, thanks! My post and thread tables are converted to InnoDB, so they can't accept fulltext anyway.

Will doing this generate any kind of error, either from MySQL or vB? I don't think it will, but our visitors have a way of shaking out any type of rare bug or hiccup, when I least expect it.

[amcd]

If your tables are innodb, there will certainly be an error when you try to set search back to fulltext.

[RedWingFan]

Just looked at this again: if I try to change it back to fulltext, it will attempt to change the InnoDB tables back to MyISAM. Don't want that!

In the settings table, "fulltextsearch" has its "value" column set to "0". There are other parameters in that row. Think I'm safe to change "value" to "1" (and rebuild the datastore cache)?

I just have to comment that since changing tables back to InnoDB and implementing Sphinx, our forum runs SO much better now! I had fulltext previously. Searching for "Steve" as a search word, it would take 35-40 seconds to get results. (Steve is our forum's owner, so his name appears in most threads.) With Sphinx, the searches come back on average around 0.4 seconds. And with the InnoDB change, we don't have stacks of queries waiting in the queue anymore.

The only hiccup I've had is that I once again had a set of *.new.* files, this time for my post indexes. I killed and restarted searchd, and reran the update, and it all rotated properly. My clue was a forum member saying he couldn't search for his posts for the past few days. Sure enough, the stale indexes were dated around the time he was unable to find his posts. I may have to run a cron job to check for any *.new.* files in that directory, and possibly put together a shell script to kill and restart searchd.

[amcd]

I may have to run a cron job to check for any *.new.* files in that directory, and possibly put together a shell script to kill and restart searchd.

I would rather look for the cause of the malfunction. If most people do not get such problems, why should you?

--------------- Added [DATE]1250056638[/DATE] at [TIME]1250056638[/TIME] ---------------

Think I'm safe to change "value" to "1" (and rebuild the datastore cache)?

I have no idea. Maybe someone from Jelsoft can answer. Or you can try reading the AdminCP PHP script which changes search type.

[UK Jimbo]

Originally Posted by amcd

I would rather look for the cause of the malfunction. If most people do not get such problems, why should you?

I agree on this one.

RedWingFan do you have a log of what the indexer is doing incuding any messages from the cronjobs?

[Shamil.]

Is this verified to work with the latest vB, 3.8.4 ?

[RedWingFan]

Originally Posted by amcd

I have no idea. Maybe someone from Jelsoft can answer. Or you can try reading the AdminCP PHP script which changes search type.

Well, I changed the value in the database, regenerated the options datastore, and it's working properly (as though I had chosen "Fulltext"). So that's a success!

Originally Posted by UK Jimbo

RedWingFan do you have a log of what the indexer is doing incuding any messages from the cronjobs?

Memory allocation error during rotate. It happened again last night, and I found the memory allocation error in the log. I changed this value in my conf file: "seamless_rotate=0". Searching will be interrupted a bit when the indexes rotate, but since I do this during off-hours, it won't affect many users at all. (Apparently, Sphinx's indexer will load indexes into RAM so that users can still search while the old indexes are being replaced by the new.) The mem_limit is at 512MB right now, but I don't want to increase that and possibly starve everything else on the server. http://www.sphinxsearch.com/docs/cur...eamless-rotate

One other setting looks a bit deceptive: max_matches is set for 1000 in my conf file, but I'm only pulling in 500 in vB's search. What Sphinx does, for max_matches, is send back the BEST 1000 matches, not necessarily just running the search and returning the FIRST 1000 matches it finds. My point here is that I should be setting the maximum search results in the conf file and in vB to be the same number. Basically I'm searching for 1000 best matches, but throwing away 500 of them for visitors. I will probably bump the forum to display those 1000 matches. Visitors will think they're getting a bonus. http://www.sphinxsearch.com/docs/cur...nf-max-matches

Here is something else: http://www.sphinxsearch.com/docs/cur...nf-enable-star . You can search Sphinx using the asterisk ("star") as a wildcard. I have thought of enabling this, but I am thinking that the search_sphinx.php file (or vB itself) would strip out the asterisk and make no difference during searches. This would be a neat addition. Has anyone else here tried it?

I came across another option for the indexer. When indexing, there is a --merge option for the indexer, which will merge your delta indexes with your main indexes, rather than generating new indexes once per day. I have it running now where the indexes are regenerated once each night. Would there be any disadvantage to using merge? If it goes correctly, it should work just as well, since you have essentially the same indexes when you're finished. But I can also see a tiny opportunity for the main indexes to get corrupted. Otherwise, --merge takes less time and CPU cycles, which is attractive. http://www.sphinxsearch.com/docs/cur...#index-merging

Finally, I see that Sphinx also has a plugin for MySQL, where you can specify using SphinxSE as an additional engine in MySQL. It will not do us much good here, I know, but I could see a future use for it between vB and Sphinx. http://www.sphinxsearch.com/docs/current.html#sphinxse

IMHO, given how much Sphinx's popularity is growing, and after having pored over the documentation this afternoon, it is disappointing that there apparently will not be any built-in support for Sphinx in vB 4.0. Sphinx can do a lot, and is a lot more flexible, than the built-in vB search, as well as MySQL's own fulltext indexing. A shame we'll probably still have to patch these hacks together to use Sphinx...

[amcd]

I came across another option for the indexer. When indexing, there is a --merge option for the indexer, which will merge your delta indexes with your main indexes, rather than generating new indexes once per day. I have it running now where the indexes are regenerated once each night. Would there be any disadvantage to using merge? If it goes correctly, it should work just as well, since you have essentially the same indexes when you're finished. But I can also see a tiny opportunity for the main indexes to get corrupted. Otherwise, --merge takes less time and CPU cycles, which is attractive. http://www.sphinxsearch.com/docs/cur...#index-merging

moved/merged/edited/deleted posts are a huge reason to regenerate the whole index periodically.