Sphinx Search

[xnetco]

Hello.

I'm not sure if this is a bug but maybe someone could help. I've installed sphinx on a board but although the administrators can search users can't search. Anyone seen this before?

Tim

[jwksite]

Well, I seem to have gotten everything installed OK... but sorting by Relevancy returns 0 results...

Any reason why it should be doing this?

I am using Sphinx 0.9.8 and it did give me a few warnings about things in my sphinx.cron being deprecated, but the other parts of the search work just fine.

I followed the steps properly, so should I be looking in sphinx.php for the error or search.php? Thanks.

Resolved... I just used a different sphinx.php file. The one that Weeno supplied is brilliant! I was using a pretty hacked up one, I guess.

Actually.. I guess the only thing that does work is relevancy (aside from dateline)? I thought the sort_search_items() fix supplied a few pages ago was meant to allow searching with all the different sort options again. So I guess sorting by Thread, Username, etc is still not possible?

OK, I guess just anything numeric works, so sorting by dateline, replies, views, and relevancy works, but not the others. *edits the search template again*

--------------- Added [DATE]1201276186[/DATE] at [TIME]1201276186[/TIME] ---------------

Now... My next question is are we supposed to (or is it even needed) make use of "SphinxSE" if we have MySQL 5? I was reading that it plugs into MySQL for faster something-or-other but I'm not quite sure I get what it does.

Thanks everyone who has contributed! I'm really tickled to be able to search through 3M posts again!

[Xorlev]

Originally Posted by jwksite

Well, I seem to have gotten everything installed OK... but sorting by Relevancy returns 0 results...

Any reason why it should be doing this?

I am using Sphinx 0.9.8 and it did give me a few warnings about things in my sphinx.cron being deprecated, but the other parts of the search work just fine.

I followed the steps properly, so should I be looking in sphinx.php for the error or search.php? Thanks.

Resolved... I just used a different sphinx.php file. The one that Weeno supplied is brilliant! I was using a pretty hacked up one, I guess.

Actually.. I guess the only thing that does work is relevancy (aside from dateline)? I thought the sort_search_items() fix supplied a few pages ago was meant to allow searching with all the different sort options again. So I guess sorting by Thread, Username, etc is still not possible?

OK, I guess just anything numeric works, so sorting by dateline, replies, views, and relevancy works, but not the others. *edits the search template again*

--------------- Added [DATE]1201276186[/DATE] at [TIME]1201276186[/TIME] ---------------

Now... My next question is are we supposed to (or is it even needed) make use of "SphinxSE" if we have MySQL 5? I was reading that it plugs into MySQL for faster something-or-other but I'm not quite sure I get what it does.

Thanks everyone who has contributed! I'm really tickled to be able to search through 3M posts again!

I personally don't use SphinxSE. It's just another interface for Sphinx.

It is possible to sort by thread, username, and forum name. It just requires sorting on a ordinal attribute (converts text to ord for sorting). It'll be in mine at any rate.

[jwksite]

Ah, well I found sorting by forum actually works off the ID, as it groups them correctly. Not sure how the original sorting worked, whether it was by name (alphabetically) or numerically.

Thread and username would be nice though..

[TECK]

Once you change a forum or modify it and your ID's are not ordered properly, your forum name sorting is gone. Use the forum.title sorting to achieve the proper results, do not use the forumid. Also, you will never be able to properly sort by thread title, no matter what. This is the only feature I have it disabled in vBulletin 3.7, anything else is working perfectly.

You should upgrade to 0.9.8 r1065, there are a bunch a new cool features that will make your programmer life very easy.
I never had any file edits with vBulletin and Sphinx 0.9.7... but it helps to toss a bunch of code that can slow down your cluster performance. I removed more then half of the old PHP code, once I upgraded to latest Sphinx version.

It is really cool how smart the engine got. Now I can search for queries like:
search engine (query) OR
search engine (query) + nginx (tag) OR
*engi* (query) + Floren (username) + nginx (tag) OR
Floren (username) + nginx (tag) OR
nginx (tag) OR
Floren (username)

There is a bug with the wild cards, but you address it with this patch:

sphinx.cpp:3418
                bool bSpecial =
                        ( iCode & FLAG_CODEPOINT_SPECIAL ) &&
- !( ( iCode & FLAG_CODEPOINT_DUAL ) && m_iAccum );
+ !( ( iCode & FLAG_CODEPOINT_DUAL ) && m_iAccum ) && m_pCur[-1] != '*';

sphinxquery.cpp:900

                int iSpecial = sToken
                        ? ( IsSpecial(sToken[0]) ? sToken[0] : 0 )
                        : QUERY_END;
                assert ( !( iSpecial>0 && sToken[1]!=0 ) );

+ if(sToken && sToken[0] == '*') iSpecial = 0;

All searches are sorted by:
- Relevancy
- Number of Replies
- Number of Views
- Thread Start Date
- Last Posting Date
- User Name
- Forum Name AND
* ASC/DESC sorting filters

As you can see, the only sorting option missing is Thread Title.
Since this is a place where Sphinx is already looking for data to pull, I would not recommend it to manipulate it through the ordinal column. The stored data is already huge as we speak, the ATTRS array has 13 keys to index the search data. From my tests, with the current array you get this ratio: 5 posts/1Kb, where each post has an average of 20-30 words. If we do a simple math operation, for 5 million posts you will inflate your data to a whopping 1GB of search data. The problem is not the speed, Sphinx can shred 1 terabyte of data in less then 3 seconds... but still, expect to lose few Gigs on your disks, if you manage a large board.

If you deal with a board that have over 5 million posts, make sure you experiment with the max_iosize and max_iops settings. So far those are the optimal values:

mem_limit	= 64M
max_iosize	= 1048576
max_iops	= 40

Another nice feature I like the the seamless rotation you can perform now with the indexer.
Very elegant. I am truly impressed with Andrew's latest version.

[jwksite]

Is the command for this new rotation any different? Or is the same command just performed differently? Just curious since I'm running the latest version and surely want the best performance out of it.

Also, I have only close to 3M posts, and I thought for sure the indexer data was already approaching 1GB if not just barely exceeding it. Care to share your .conf file?

And, like amcd said earlier on "What do we have to do to convince you to share your work?"

I would certainly like figuring out the proper optimizations for 0.9.8. I'm using a deprecated sphinx.conf file but I was getting too impatient with getting Sphinx to work in the first place to go in and use the new attributes. Now that I know it's working very well, I think I can finally get around to the tweaking stuff.

[TECK]

Sorry, I cannot share my work because it is part of my optimization package I offer as a paying service to my clients (like running 3 queries on the vBulletin frontpage without file edits). I spent countless hours through white nights to test various scenarios to get it working perfectly. Basically, it is a combination of server optimizations and PHP code modifications. I also tested the optimization package into several large boards with over 5,000 users online. It improved the overall vBulletin performance and saved them a lot of money on cluster upgrades:

Optimizations Enabled
500 online users, 10 seconds parallel test

779 fetches, 500 max parallel, 1.564e+07 bytes, in 10.0024 seconds
20077 mean bytes/connection
77.8815 fetches/sec, 1.56363e+06 bytes/sec
msecs/connect: 1.50359 mean, 131.5 max, 0.07 min
msecs/first-response: 1674.58 mean, 8914.71 max, 97.031 min

1000 online users, 10 seconds parallel test

633 fetches, 1000 max parallel, 1.26093e+07 bytes, in 10.0019 seconds
19919.9 mean bytes/connection
63.2879 fetches/sec, 1.26069e+06 bytes/sec
msecs/connect: 124.028 mean, 3007.64 max, 0.073 min
msecs/first-response: 1971.85 mean, 8686.33 max, 1.099 min

Optimizations Disabled
500 online users, 10 seconds parallel test

269 fetches, 500 max parallel, 5.34661e+06 bytes, in 10.0094 seconds
19875.9 mean bytes/connection
26.8747 fetches/sec, 534159 bytes/sec
msecs/connect: 0.904112 mean, 85.94 max, 0.074 min
msecs/first-response: 3783.43 mean, 9272.46 max, 1.373 min

1000 online users, 10 seconds parallel test

263 fetches, 1000 max parallel, 5.32541e+06 bytes, in 10.0039 seconds
20248.7 mean bytes/connection
26.2897 fetches/sec, 532332 bytes/sec
msecs/connect: 0.824681 mean, 78.576 max, 0.071 min
msecs/first-response: 3868.94 mean, 8879.29 max, 165.277 min

Test Explained
The test emulated the matching number of users who tried to access as many times as possible the forums, from various locations.
1000 online users, 10 seconds parallel test means that 1000 users visited without interruption the forums. for 10 seconds while navigating as fast as they can through pages.
63.2879 fetches/sec means that those 1000 users managed to visit 63 pages per second, for a total of 633 fetches per user... pretty fast those guys on their browser.

Test Comparison

OPTIONS		Optimizations Enabled	Optimizations Disabled
-----------------------------------------------------------------
Users		1000	500		1000	500
-----------------------------------------------------------------
Fetches		633	779		263	269
Fetches/sec	63.28	77.88		26.28	26.87

In other words, with the optimizations enabled, the overall server performance was improved by 290%. That is almost 3 times faster compared to the optimizations disabled.

As I said before, I do not use Orban's code approach in any way, shape of form... I don't have one single line of PHP code identical to his and the Sphinx configuration file is exactly the opposite what you think it should look like.

[amcd]

Well, this is a community forum for sharing. If you are only going to brag about your code and not share it, IMHO it should not be allowed.

[TECK]

Originally Posted by amcd

Well, this is a community forum for sharing.

If it is a community forum for sharing, why they allow here paid services?

Originally Posted by amcd

If you are only going to brag about your code and not share it, IMHO it should not be allowed.

I did not bragged about it. I actually posted valuable information for programmers. Why should I not be allowed to talk about code? It is a coding forum, after all... not a communist zone where moderators dictate what regular users can post. As you noticed already, there are many people who post what they achieved with Sphinx, without sharing any code... I'm not the only one. Any experienced programmer will be able to adapt to vBulletin the Sphinx API and it's search engine in no time... because it is very simple to implement. Not to mention that you do not need any file edits.

[andrewkhunn]

Originally Posted by TECK

Sorry, I cannot share my work because it is part of my optimization package I offer as a paying service to my clients (like running 3 queries on the vBulletin frontpage without file edits).

Do you have a link or any more information about your optimization services?