Sphinx Search

[eoc_Jason]

Originally Posted by Jah-Hools

See - at 2 million posts search speed isnt a problem for me (so far) - what IS a problem is the forum members deep, deep unhappiness with the way the VB search works.. Google is the best way to search my forum (it handles 2 character content way better) so I am about to pull the trigger on Google Custom Business edition.. I wanted to make a last check to see if Sphinx would do the job better.. But all I can glean from these posts is 'use sphinx if VB search is slow' I don't see any raves about improved user functionality.. Does Sphinx simply improve speed for big forums and reduce the extra machines in a cluster of servers?

Its search functionality I am after.. Does Sphinx add functionality to the forum users experience - or just add speed?

One big problem with the stock vB code is that it likes to do a lot of post-filtering of the returned results. It happened way too often (to me and others) that searches where I know a keyword should appear (because I put the word in a specific place to do the test) vB will return "No results found" because it drops the score so low that it rejects it, even if it was the ONLY result...

Sphinx has it's own scoring system, and depending on how you set it up it can be extremely powerful. But if you want to gain the full benefit you will also need to probably peel through the search.php file and rip out some of the vB scoring so that it doesn't remove relevant results.

Also with sphinx it will return results in the same thread (or post) like view. So having results returned by date or a certain user or whatever is a lot more powerful than what google could probably do.

[rebelde]

I was desperate to get tag search working, so I've worked it out myself. I used Orban's 0.1 "plug-in" version as a base, with Sphinx-0.9.8-rc2 on vBulletin 3.7.1. Only small changes were necessary to support tags.

I don't use prefixes in my forum, so the prefix implementation is incomplete. I just added the prefixes in the full-text index so they are searchable just like any other text in a thread or post. I would be happy to try to walk somebody (with basic PHP skills) through what I think is necessary to get it to work properly.

I have attached my versions of Orban's files. Here are the details of my changes:

In Orban's search_sphinx.php,

after:
#$vbulletin->GPC['prefixchoice']

add:
 if (!empty($verified_tag ))
        $cl->SetFilter('tagid', array($verified_tag['tagid']) );


This isn't required, but I prefer 'extended' mode so people can use quotes in searches to find phrases:

Replace:
if (!$vbulletin->config['Sphinx']['mode']) $vbulletin->config['Sphinx']['mode'] = 'SPH_MATCH_ALL';
With:
if (!$vbulletin->config['Sphinx']['mode']) $vbulletin->config['Sphinx']['mode'] = 'SPH_MATCH_EXTENDED';



In sphinx.config, I have made a number of changes.
 - added tags and prefixes to the indexes
 - changed post.visible=1 to post.visible > 0 so moderators can see deleted posts (bug fix)
 - use a minimum word length of 1 (instead of 4).  Sphinx can handle it without problem.

(Please excuse my new username. I haven't posted in this thread before, but I have been a member of vB.org for four years under a different username.)

And finally, thank you Orban! If you are around, turn on private messaging and I'll help however I can.

[orban]

Thanks rebelde, adding prefixes to the full text index is a great idea, I don't know why I haven't considered this. Also re-enabled PMs

[ivanp]

Nice job orban and rebelde! It works great.

I think this code in search_sphinx.php:

if (!$vbulletin->config['Sphinx']['host']) $vbulletin->config['Sphinx']['host'] = 'localhost';

should be changed to:

if (!$vbulletin->config['Sphinx']['host']) $vbulletin->config['Sphinx']['host'] = $vbulletin->config['MasterServer']['servername'];

Two questions:
1) Is there a fix to make wildcards work?
2) Is there a transliteration support for Sphinx?

[rebelde]

I'm glad it is working for you.

Originally Posted by ivanp

1) Is there a fix to make wildcards work?

From what I can tell from the Sphinx docs, you need to add:
min_prefix_len = 3
prefix_fields = title (maybe more?)
enable_star = 1

I'm not sure where you add these lines, though. I would think that they go in:
index thread : post - first two
index threaddelta : post - first two
searchd - enable_star = 1

Let us know if you find out where they go, and if it slows searches much.

Transliteration? I'm not sure. Check the Sphinx docs.

[ivanp]

/search.php?do=finduser&u=... is not working properly.

It returns either:

Sorry - no matches. Please try some different terms. %1$s

or returns very old results.

--------------- Added [DATE]1212666226[/DATE] at [TIME]1212666226[/TIME] ---------------

Originally Posted by rebelde

e Sphinx docs, you need to add:
min_prefix_len = 3
prefix_fields = title (maybe more?)
enable_star = 1

I am afraid of this:

http://www.sphinxsearch.com/doc.html

However, indexing prefixes will make the index grow significantly (because of many more indexed keywords), and will degrade both indexing and searching times.

Waiting to hear feedback if moderators really need this feature.

[BillP]

We are running the new plugin with 3.7.1

When one clicks on a username and chooses "Find More Posts By <user>", the following warning appears on the "Your search is in progress" page:

Warning: in_array() [function.in-array]: Wrong datatype for second argument in [path]/search_sphinx.php on line 160

Then it refreshes, and returns numerous errors followed by the results:

Warning: strpos() [function.strpos]: Empty delimiter in [path]/includes/functions_search.php on line 829

Warning: strpos() [function.strpos]: Empty delimiter in [path]/includes/functions_search.php on line 829

Warning: strpos() [function.strpos]: Empty delimiter in [path]/includes/functions_search.php on line 829

Warning: strpos() [function.strpos]: Empty delimiter in [path]/includes/functions_search.php on line 829

Warning: strpos() [function.strpos]: Empty delimiter in [path]/includes/functions_search.php on line 829

Warning: strpos() [function.strpos]: Empty delimiter in [path]/includes/functions_search.php on line 829

Any ideas on how to fix this?

[rebelde]

Originally Posted by BillP

When one clicks on a username and chooses "Find More Posts By <user>"

Originally Posted by ivanp

/search.php?do=finduser&u=... is not working properly.

I tested it, and have no trouble at all with these searches.
/search.php?do=finduser&u=1
/search.php?do=finduser&u=1&starteronly=1
It works properly and shows the threads in descending order for me.

[BillP]

Originally Posted by rebelde

I tested it, and have no trouble at all with these searches.
/search.php?do=finduser&u=1
/search.php?do=finduser&u=1&starteronly=1
It works properly and shows the threads in descending order for me.

Did you use the search_sphinx.php directly from the download on the first post of this thread, or have you edited it with fixes that orban (or others) have suggested along the way?

We aren't running any other hacks in the system. These ARE warnings, maybe it's just the warning vs. error setting in PHP and these can be ignored?

[DaiTengu]

Turning off warnings is a cosmetic fix, but the underlying issues still exist.


Did you update your sphinx.conf with the new settings suggested here: https://vborg.vbsupport.ru/showpost....&postcount=541 ?