Sphinx Search

[TECK]

Good to see the thread is back! The need queries are still here... Hoooray!

[TheComputerGuy]

I think I am going to give this a try. This should be awesome if it works.

[MentaL]

Guys, Im really stuck, im trying to set this up and i just run into trouble, if anyone can gimme a good guide, thats great, but im stuck at this point.

root@mental [~/sphinx-0.9.7]# indexer --config sphinx.conf --all
Sphinx 0.9.7
Copyright (c) 2001-2007, Andrew Aksyonoff

using config file 'sphinx.conf'...
indexing index 'post'...
ERROR: index 'post': failed to open sphinx-data/fulltext.tmp0: No such file or directory.
total 0 docs, 0 bytes
total 0.023 sec, 0.00 bytes/sec, 0.00 docs/sec
indexing index 'postdelta'...
ERROR: index 'postdelta': failed to open sphinx-data/fulltextdelta.tmp0: No such file or directory.
total 0 docs, 0 bytes
total 0.017 sec, 0.00 bytes/sec, 0.00 docs/sec
indexing index 'thread'...
ERROR: sql_connect: Unknown MySQL server host 'db.local' (1) (DSN=mysql://:***@db.local:3306/).
ERROR: index 'thread': (no error message).
total 0 docs, 0 bytes
total 5.149 sec, 0.00 bytes/sec, 0.00 docs/sec
indexing index 'threaddelta'...
ERROR: sql_connect: Unknown MySQL server host 'db.local' (1) (DSN=mysql://:***@db.local:3306/).
ERROR: index 'threaddelta': (no error message).
total 0 docs, 0 bytes
total 1.451 sec, 0.00 bytes/sec, 0.00 docs/sec
distributed index 'fulltext' can not be directly indexed; skipping.
distributed index 'threadtitles' can not be directly indexed; skipping.
root@mental [~/sphinx-0.9.7]#

[amcd]

Post your configuration file.

From what you posted, it appears that you have 2 problems, at least.
1. sphinx-data does not exist or the sphinx user does not have proper permissions to that directory.
2. sphinx is not able to connect to mysql because the connection parameters are not correct.

[MentaL]

Originally Posted by amcd

Post your configuration file.

From what you posted, it appears that you have 2 problems, at least.
1. sphinx-data does not exist or the sphinx user does not have proper permissions to that directory.
2. sphinx is not able to connect to mysql because the connection parameters are not correct.

Managed to fix that error but Now i have this error.

root@mental [~/sphinx-0.9.7]# indexer --config sphinx.conf --all
Sphinx 0.9.7
Copyright (c) 2001-2007, Andrew Aksyonoff

using config file 'sphinx.conf'...
indexing index 'post'...
ERROR: index 'post': raw_hits: write error: 5941 of 262023 bytes written.
total 2337609 docs, 630070420 bytes
total 89.155 sec, 7067095.78 bytes/sec, 26219.46 docs/sec
indexing index 'postdelta'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.010 sec, 0.00 bytes/sec, 0.00 docs/sec
indexing index 'thread'...
ERROR: index 'thread': raw_hits: write error: 156494 of 262021 bytes written.
total 215804 docs, 5689335 bytes
total 2.430 sec, 2341232.40 bytes/sec, 88806.04 docs/sec
indexing index 'threaddelta'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.010 sec, 0.00 bytes/sec, 0.00 docs/sec
distributed index 'fulltext' can not be directly indexed; skipping.
distributed index 'threadtitles' can not be directly indexed; skipping.
root@mental [~/sphinx-0.9.7]#

[epheph]

Hello, I have installed this mod and sphinx into a sizable vBulletin 3.6.5 installation (6M+ posts) and the performance so far in testing is very good. The only issue I have is that 'show as posts' doesn't work all the time. It seems if the search term has many results, I get

"Sorry - no matches. Please try some different terms."

But everything else works great (by user, etc). If I intentionally search for a term I know will be quite low, like misspellings, the "show by posts" works just fine. Any ideas?

[kontrabass]

Well I've finally got this up and going. Install went well, thanks to all the input here. Took me a while to figure out why a search term with thousands of matches returned only 180 vbulletin threads (1000 post matches gets distilled down into the 180 threads that hold them!). So I upped the max results to 2000 and get satisfactory result sets.

I was disappointed with not being able to do "phrase matches" - until I changed this line in sphinx.php:

$cl->SetMatchMode ( SPH_MATCH_ALL );

to

$cl->SetMatchMode ( SPH_MATCH_EXTENDED );

Extended mode enables all this stuff (including "phrase matching") :

http://www.sphinxsearch.com/doc.html#extended-syntax

(sorry if this is repeated)

However it seems that unless you can teach your users to use pipes and ampersands, "AND" and "OR" text boolean operators will not work

Edit: Actually, even pipes and apmersands don't work I guess ("The search term you specified (|) is under the minimum word length (3) and therefore will not be found. Please make this term longer."). Lol

Is everyone using this search doing ok without any boolean operators? Every search is treated as an "AND" search I guess.

[BillP]

Originally Posted by Jon

No, the indexer works fine. There could be up to hundreds of posts not included. Say I'm searching for a term, with a match on posts, displayed as posts, I first get 3 pages of matches on title, ranging from 2007 to 2005. After that I get matches on post, ranging from 2005 to 2003. Recent posts are nowhere to be found.

We are getting strange results as well. Looking at results via sphinx "search" CLI shows me that there are hits in the right date range, but in our case we have lots of data since August, and spotty or missing data going back to 2003, light data back from there to 2001, and then a whole lot of data again. It's like the index is "light" for a few years.

=========

Bzzt! I forgot to update search.php after the last VB upgrade (it was on my to-do list, I just muffed it)

If you have weird data being returned, take a look at your search.php to see if you accidentally upgraded it and lost your edits!

[eoc_Jason]

Didn't want to see this thread die as I still heavy rely on Sphinx for my vB search. It's disappointing that the vB team *still* has not come up with a built-in solution for searching that is acceptable for large forums.

One bit of advice - The code in the vB search does a lot of weighting & filtering. I have many instances where I search for a specific word that is in a post, and doing a raw search I can find it, but after vB works its magic it will give a 'no results found'. So don't think something is broken, I guess technically it is, but it's 'by design'.

Anyhow, here's some bits of code that might help get your sphinx running a little smoother.

One thing you really should do is run the search daemon & indexer under a non-root user, I use 'sphinx'. If yours is different than simply adjust the files accordingly. you will probably need to make a few directories mentioned in the scripts below and have them owned by your user that sphinx is running as. (like for the lock file & log)

I run on Redhat / CentOS, here's my script that goes in the /etc/rc.d/init.d/ directory. I call it 'searchd'. Simply use 'chkconfig' to add it and have it start up when you system boots.

#!/bin/sh
#
# searchd   This script starts and stops the sphinx search engine
#
# chkconfig: - 80 15
#
# description: Stand Alone Search Engine
# processname: searchd
# config: /usr/local/etc/sphinx.conf
# pidfile: /var/run/searchd/searchd.pid


# Source function library.
. /etc/rc.d/init.d/functions

RETVAL=0

start() {
        echo -n "Starting Sphinx: "
        sudo -u sphinx /usr/local/bin/searchd --config /usr/local/etc/sphinx.conf > /dev/null 2>&1
        RETVAL=$?
        if [ $RETVAL -eq 0 ]; then
          success startup
          touch /var/lock/subsys/searchd
        else
          failure startup
        fi
        echo
        return $RETVAL
}

stop() {
        echo -n "Shutting Down Sphinx: "
        kill `cat /var/run/searchd/searchd.pid`
        RETVAL=$?
        if [ $RETVAL -eq 0 ]; then
          success shutdown
          rm -f /var/lock/subsys/searchd /var/run/searchd/searchd.pid
        else
          failure shutdown
        fi
        echo
        return $RETVAL
}

restart() {
        stop
        start
}

case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart)
        restart
        ;;
  status)
        status searchd
        ;;
  *)
        echo $"Usage: $0 {start|stop|restart|status}"
        exit 1
esac

exit $RETVAL

Second, here is my cron script. I made a directory called 'cron.quarterly' that runs every 15min. Similar to the cron.hourly, cron.daily, etc... You can do whatever. If you make that directory be sure to edit your /etc/crontab file accordingly too. I added a time-check so once a day sphinx will do a full-reindex, and it makes a lock file so if yours takes a long time to reindex you won't run into issues. Basically at 5am it will do a re-index. I chose that time because it's a low usage period for my forum and server. By default the cron.daily scripts run at 4am so you wouldn't want to do it then since everything else will be eating CPU cycles.

#!/bin/sh

# the lockfile is not meant to be perfect, it's just in case the
# two sphinx cron scripts get run close to each other to keep
# them from stepping on each other's toes.

LOCKFILE=/var/lock/subsys/sphinx_indexer

# If the lockfile exists then exit!

if [ -f $LOCKFILE ]; then
  echo "Lockfile already exists, not running sphinx indexer!"
  exit
fi;

touch $LOCKFILE

compareh=$(date +%k)
comparem=$(date +%M)

if [ $compareh -eq "5" ] && [ $comparem -le "14" ]; then
  sudo -u sphinx /usr/local/bin/indexer --config /usr/local/etc/sphinx.conf --rotate --all > /dev/null 2>&1
else
  sudo -u sphinx /usr/local/bin/indexer --config /usr/local/etc/sphinx.conf --rotate post_index_delta thread_index_delta > /dev/null 2>&1
fi;

rm -f $LOCKFILE

exit 0

I also have my logs in their own /var/log/searchd/ directory, you set this in the sphinx config (most people probably just have them in the /var/log dir). Again that directory will need to be owned by the sphinx user that you use.

/var/log/searchd/*.log {
        missingok
        compress
        postrotate
          if test -n "`ps acx|grep searchd`"; then
            /sbin/service searchd restart  2> /dev/null > /dev/null || true
          fi
        endscript
}

Oh, and for people wanting to know how to implement the sort_search_items(), here's some code. It goes in the includes/sphinx.php file.

At the end where there is the following:

                        if ($vbulletin->GPC['titleonly'] == $vbulletin->GPC['showposts'])
				$orderedids[$docinfo['attrs'][$sphinx_switch_fields]] = $docinfo['attrs'][$sphinx_switch_fields];
                        else
				$orderedids[] = $doc;
                }       
        }       
        else
		$orderedids = array();
}

Replace that with:

                        if ($vbulletin->GPC['titleonly'] == $vbulletin->GPC['showposts'])
                        {
                                $orderedids[$docinfo['attrs'][$sphinx_switch_fields]] = $docinfo['attrs'][$sphinx_switch_fields];
                                $itemids[$docinfo['attrs'][$sphinx_switch_fields]] = $docinfo['attrs'][$sphinx_switch_fields];
                        }
                        else
                        {
                                $orderedids[] = $doc;
                                $itemids["$doc"] = true;
                        }
                }
        }
        else
        {       
                $orderedids = array();
        }
        
        // #############################################################################
        // now sort the results into order
        // #############################################################################
        if (!$vbulletin->GPC['titleonly'] OR $vbulletin->GPC['showposts'])
        {       
                // sort by database field
                if ($vbulletin->GPC['sortby'] == 'post.dateline' || $vbulletin->GPC['sortby'] == 'lastpost')
                {       
                        if (empty($itemids))
                        {       
                                $errors[] = array('searchnoresults', $displayCommon);
                        }
                        else
                        {       
                                // remove dupes and make query condition
                                $itemids = iif($vbulletin->GPC['showposts'], 'postid', 'threadid') . ' IN(' . implode(',', array_keys($itemids)) . ')';
                                
                                // sort the results and create the final result set
                                $orderedids = sort_search_items($itemids, $vbulletin->GPC['showposts'], $vbulletin->GPC['sortby'], $vbulletin->GPC['sortorder']);
                        }
                }
        }
        
// END Results  
}

Be sure to leave the unset line at the very very bottom.


I'm about to make the upgrade to vB 3.6.8 whatever, and I'm running php 5.2.4, so I'll let you know how the upgrade goes. I want to look over everyones modifications and updates in this thread. Then I'll post some more files and stuff if necessary.

[mute]

Thanks for that search results sort order fix, it seems to be working nicely.

All that really stands between this being a perfect solution is finding a way to handle all of the "Find all posts by user" type searches.