Sphinx Search

[Mickie D]

anyone know why its not getting the full amount of results ?

if i search with command line its getting houndreds and if i search from the forums its finding 1 or 2 things which i know there is more of ?

[raywjohnson]

Originally Posted by orban

if you're interested in an implementation
https://vborg.vbsupport.ru/showpost.php?p=1104866

Thank you for the great instructions. A few hours work ( I am very new to vBulletin ), plenty of Google searching, and another successful installation of Sphinx! Forum "write-lock" issues have disappeared. Thanks again!

Forgot to ask, is there any kind of setting to block indexing before a certain date? One of my admins informed me that he cannot find any post prior to 2005.

Later, RayJ

[raywjohnson]

Originally Posted by jason|xoxide

I wouldn't classify this as a bug, just more of an oversight. The max_matches variable in the sphinx.conf file is ignored when using the PHP API script. It doesn't matter if it's left at the default 1000, the 1500 that orban's file has been modified to use, or 1000000 as the config file says not to do. If you want to change the number of results returned then you need to change line 15 of 'includes/sphinx.php'.

Old:

PHP Code:

$cl->SetLimits ( 0, $vbulletin->options['maxresults'] ); 


New (replace '2500' with the number you want):

PHP Code:

$cl->SetLimits ( 0, $vbulletin->options['maxresults'], 2500 ); 


Otherwise, great work! This has really sped up searching on the forums I have used for testing (6K posts, 820K posts, and 3.2M posts).

Little help!

Most likly I am just misunderstanding the results.
Here are the results from a command line search: (displaying matches: snipped)

[root@MYSERVER ~/]# search test

Sphinx 0.9.7-RC2
Copyright (c) 2001-2006, Andrew Aksyonoff

index 'mypostidx': query 'test': returned 1000 matches of 191296 total in 0.029 sec

words:
1. 'test': 191296 documents, 475585 hits
index 'mypostidxdelta': query 'test': returned 56 matches of 56 total in 0.000 sec

words:
1. 'test': 56 documents, 134 hits
index 'mythreadidx': query 'test': returned 1000 matches of 2847 total in 0.154 sec

words:
1. 'test': 2847 documents, 2879 hits
index 'mythreadidxdelta': query 'test': returned 0 matches of 0 total in 0.000 sec

But, searching in the forum: (show posts/search entire post)
Search: Key Word(s): test Showing results 1 to 40 of 392

(I created a huge test forum and it has many more posts with the word "test" than 392)

And using test.php
php test.php test
Query failed: searchd error: index 'mythreadidx': incompatible schemas: non-virtual attributes count mismatch: 4 in schema '/var/sphinx/mythreadidx', 5 in schema '/var/sphinx/mypostidx'.

On the forum, the search never returns any more that 400 results (i.e. Showing: 40 of 400). I cannot find a setting that cuts off the results at 400.

I have read through this thread (twice!) and made suggested changes to sphinx.php [$cl->SetLimits()] and sphinx.conf (max_matches) with no change to the results.
(I also searches the "Common Forum" at sphinxsearch.com, no luck!)

Any insight into this would be most appreciated!

Later, RayJ

[orban]

Have you restarted searchd?

[UK Jimbo]

There were some problems with the assert function around post #280.

My solution to these was to turn assert warnings off using the single line of code:

PHP Code:

assert_options(ASSERT_ACTIVE, false); 


You can do this in the api file or in sphinx.php

I've just re-indexed with the post table at a min word length of 3. As you can see from the command line the process was niced at 20 and there were 400 active users on the site. I'm hugely impressed by this implemention.

sphinx@new [/usr/local/etc]# nice -n 20 indexer --config /usr/local/etc/sphinx.conf --rotate --all
Sphinx 0.9.7-RC2
Copyright (c) 2001-2006, Andrew Aksyonoff

using config file '/usr/local/etc/sphinx.conf'...
indexing index 'post'...
collected 4180518 docs, 1218.2 MB
sorted 165.8 Mhits, 100.0% done
total 4180518 docs, 1218224054 bytes
total 632.617 sec, 1925689.34 bytes/sec, 6608.29 docs/sec
indexing index 'post_delta'...
collected 22 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 22 docs, 6315 bytes
total 0.012 sec, 545974.95 bytes/sec, 1902.05 docs/sec
indexing index 'thread'...
collected 253498 docs, 7.0 MB
sorted 0.8 Mhits, 100.0% done
total 253498 docs, 6961147 bytes
total 4.818 sec, 1444710.55 bytes/sec, 52610.76 docs/sec
indexing index 'thread_delta'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.014 sec, 0.00 bytes/sec, 0.00 docs/sec
skipping index 'fulltext_post' (distributed indexes can not be directly indexed)...
skipping index 'fulltext_thread' (distributed indexes can not be directly indexed)...
rotating indices: succesfully sent SIGHUP to searchd (pid=8972).

[UK Jimbo]

Another post from me (might get auto-merged)...

I wanted to easily see the query.log that searchd creates, I always think it's good to try to give something back to a project that you like too

• Place the attached file sphinx_search_log.php to your admincp directory, edit the reference to query.log if necessary
• Place the attached file cpnav_sphinx_search_log.xml to your includes/xml directory

It's really that easy.

Now look in your AdminCP menu system under Statistics & Logs for Sphinx Search Log

[raywjohnson]

Originally Posted by orban

Have you restarted searchd?

Thanks for the help!

After I read your post I restarted searchd and preformed command line and forum search again, but the results were the same. I indexed -all as well and tried again, no joy!

Do you know of any settings in vBulletin that would limit the search results?

Later, RayJ

[eoc_Jason]

Originally Posted by kontrabass

Thanks for the report! Mind if I ask a couple questions (gathering info to see where I would stand): Are you running a slave DB server for searches? What kind of hardware is behind your database? Thanks

No, just one single server. Dual Woodcrest (4 cores total) & 4GB RAM. A slave server was an initial consideration, but even if there was a slave server the mysql fulltext searches would not have run any faster really. It would of only alieviated the locking on the master db.

Depending on what people searched before, queries could of taken several minutes (and we all know nobody waits that long for a web page to load). Queries like that would cause the post table to be locked and thus anyone trying to post would of also been sitting waiting until the search compelted. Usually people got impatient too and would click the search button several times, only queuing up the searches even more.

Until I installed this sphinx search mod, the only course of action was to have a custom script that would kill any search queries that took over 60 seconds (to prevent the issues above).

Yes everything on the server was extremely optimized, and I even had to set the mysql fulltext min characters to 5, and max to like 12-15 I think it was.

Basically the only two things out there is sphinx & mnogosearch mods for vB. I chose this one because it was the most transparent. Now searches are usually done in way under 1 second, and even though the results could sometimes be out of (date) order, it still works a million times better than before.

I plan on installing this on my own forum too, as searches are starting to cause issues.

Searching is one of the last weak points of vBulletin and really needs to be addressed.

Originally Posted by raywjohnson

Thanks for the help!

After I read your post I restarted searchd and preformed command line and forum search again, but the results were the same. I indexed -all as well and tried again, no joy!

Do you know of any settings in vBulletin that would limit the search results?

Did you increase your max results in both the sphinx and the other php api file? Also I think vBulletin might have some limits in the control panel. I don't know which all are used for searching as I haven't had time to really look through all the underlying code with sphinx. I just got it up and running and have been letting it do it's own thing.

vB also does a lot of weighting and will toss out low results (irritating as it can produce no results even when there are). I do not know if this is still used with sphinx though, if it is then that probably explains your issue.

[raywjohnson]

Originally Posted by eoc_Jason

Did you increase your max results in both the sphinx and the other php api file? Also I think vBulletin might have some limits in the control panel. I don't know which all are used for searching as I haven't had time to really look through all the underlying code with sphinx. I just got it up and running and have been letting it do it's own thing.

vB also does a lot of weighting and will toss out low results (irritating as it can produce no results even when there are). I do not know if this is still used with sphinx though, if it is then that probably explains your issue.

I did make the changes to both files, still no change. But... your post promoted me to look deeper into a possible limit imposed by vBulletin.

Eureka! vBulletin Control Panel -> vBulletin Options -> Message Searching Options -> Maximum Search Results to Return (was set to 400 now set to 9000) Worked perfectly!

Thank you! And thanks to all who worked on helping get Sphinx Search working on vBulletin! Extra thanks to orban!

Later, RayJ

[eoc_Jason]

Glad to hear you found it, I guess I should go back and check what I have it set to also.

Ah, post #301 was what I was referring to before. Anyhow, glad you figured out what it was. I guess all three of those settings need to be the same for the most optimal results.

This is a true must-have for any large forum, the mysql fulltext index search goes painfully slow after you exceed a certain number of posts, and the other vB search feature never worked all that wll for me.

This would really be the next big thing I would like to see vB integrate into new versions. They already support things like other datastore caches, why not other search engines?

Anyhow, I'm about to tackle another install, this time on my forum. Should go smoother than the first time now that I know all the ins & outs. The biggest thing is just making sure you rename everything properly in the config files.