Go Back   vb.org Archive > Community Discussions > Forum and Server Management
  #411  
Old 09-06-2007, 10:22 PM
TECK's Avatar
TECK TECK is offline
 
Join Date: Nov 2001
Location: Canada
Posts: 4,182
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Good to see the thread is back! The need queries are still here... Hoooray!
Reply With Quote
  #412  
Old 09-06-2007, 11:59 PM
TheComputerGuy's Avatar
TheComputerGuy TheComputerGuy is offline
 
Join Date: Oct 2001
Location: TX
Posts: 580
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

I think I am going to give this a try. This should be awesome if it works.
Reply With Quote
  #413  
Old 09-15-2007, 01:00 AM
MentaL's Avatar
MentaL MentaL is offline
 
Join Date: Jan 2003
Posts: 550
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Guys, Im really stuck, im trying to set this up and i just run into trouble, if anyone can gimme a good guide, thats great, but im stuck at this point.


Code:
root@mental [~/sphinx-0.9.7]# indexer --config sphinx.conf --all
Sphinx 0.9.7
Copyright (c) 2001-2007, Andrew Aksyonoff

using config file 'sphinx.conf'...
indexing index 'post'...
ERROR: index 'post': failed to open sphinx-data/fulltext.tmp0: No such file or directory.
total 0 docs, 0 bytes
total 0.023 sec, 0.00 bytes/sec, 0.00 docs/sec
indexing index 'postdelta'...
ERROR: index 'postdelta': failed to open sphinx-data/fulltextdelta.tmp0: No such file or directory.
total 0 docs, 0 bytes
total 0.017 sec, 0.00 bytes/sec, 0.00 docs/sec
indexing index 'thread'...
ERROR: sql_connect: Unknown MySQL server host 'db.local' (1) (DSN=mysql://:***@db.local:3306/).
ERROR: index 'thread': (no error message).
total 0 docs, 0 bytes
total 5.149 sec, 0.00 bytes/sec, 0.00 docs/sec
indexing index 'threaddelta'...
ERROR: sql_connect: Unknown MySQL server host 'db.local' (1) (DSN=mysql://:***@db.local:3306/).
ERROR: index 'threaddelta': (no error message).
total 0 docs, 0 bytes
total 1.451 sec, 0.00 bytes/sec, 0.00 docs/sec
distributed index 'fulltext' can not be directly indexed; skipping.
distributed index 'threadtitles' can not be directly indexed; skipping.
root@mental [~/sphinx-0.9.7]#
Reply With Quote
  #414  
Old 09-15-2007, 05:28 AM
amcd amcd is offline
 
Join Date: Oct 2004
Posts: 218
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Post your configuration file.

From what you posted, it appears that you have 2 problems, at least.
1. sphinx-data does not exist or the sphinx user does not have proper permissions to that directory.
2. sphinx is not able to connect to mysql because the connection parameters are not correct.
Reply With Quote
  #415  
Old 09-15-2007, 09:54 AM
MentaL's Avatar
MentaL MentaL is offline
 
Join Date: Jan 2003
Posts: 550
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by amcd View Post
Post your configuration file.

From what you posted, it appears that you have 2 problems, at least.
1. sphinx-data does not exist or the sphinx user does not have proper permissions to that directory.
2. sphinx is not able to connect to mysql because the connection parameters are not correct.
Managed to fix that error but Now i have this error.

Code:
root@mental [~/sphinx-0.9.7]# indexer --config sphinx.conf --all
Sphinx 0.9.7
Copyright (c) 2001-2007, Andrew Aksyonoff

using config file 'sphinx.conf'...
indexing index 'post'...
ERROR: index 'post': raw_hits: write error: 5941 of 262023 bytes written.
total 2337609 docs, 630070420 bytes
total 89.155 sec, 7067095.78 bytes/sec, 26219.46 docs/sec
indexing index 'postdelta'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.010 sec, 0.00 bytes/sec, 0.00 docs/sec
indexing index 'thread'...
ERROR: index 'thread': raw_hits: write error: 156494 of 262021 bytes written.
total 215804 docs, 5689335 bytes
total 2.430 sec, 2341232.40 bytes/sec, 88806.04 docs/sec
indexing index 'threaddelta'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.010 sec, 0.00 bytes/sec, 0.00 docs/sec
distributed index 'fulltext' can not be directly indexed; skipping.
distributed index 'threadtitles' can not be directly indexed; skipping.
root@mental [~/sphinx-0.9.7]#
Reply With Quote
  #416  
Old 09-26-2007, 05:44 AM
epheph epheph is offline
 
Join Date: Nov 2005
Posts: 1
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Hello, I have installed this mod and sphinx into a sizable vBulletin 3.6.5 installation (6M+ posts) and the performance so far in testing is very good. The only issue I have is that 'show as posts' doesn't work all the time. It seems if the search term has many results, I get

"Sorry - no matches. Please try some different terms."

But everything else works great (by user, etc). If I intentionally search for a term I know will be quite low, like misspellings, the "show by posts" works just fine. Any ideas?
Reply With Quote
  #417  
Old 10-21-2007, 08:44 PM
kontrabass kontrabass is offline
 
Join Date: Feb 2002
Posts: 139
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Well I've finally got this up and going. Install went well, thanks to all the input here. Took me a while to figure out why a search term with thousands of matches returned only 180 vbulletin threads (1000 post matches gets distilled down into the 180 threads that hold them!). So I upped the max results to 2000 and get satisfactory result sets.

I was disappointed with not being able to do "phrase matches" - until I changed this line in sphinx.php:

$cl->SetMatchMode ( SPH_MATCH_ALL );

to

$cl->SetMatchMode ( SPH_MATCH_EXTENDED );

Extended mode enables all this stuff (including "phrase matching") :

http://www.sphinxsearch.com/doc.html#extended-syntax

(sorry if this is repeated)

However it seems that unless you can teach your users to use pipes and ampersands, "AND" and "OR" text boolean operators will not work

Edit: Actually, even pipes and apmersands don't work I guess ("The search term you specified (|) is under the minimum word length (3) and therefore will not be found. Please make this term longer."). Lol

Is everyone using this search doing ok without any boolean operators? Every search is treated as an "AND" search I guess.
Reply With Quote
  #418  
Old 10-23-2007, 04:08 AM
BillP BillP is offline
 
Join Date: Nov 2001
Posts: 38
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by Jon View Post
No, the indexer works fine. There could be up to hundreds of posts not included. Say I'm searching for a term, with a match on posts, displayed as posts, I first get 3 pages of matches on title, ranging from 2007 to 2005. After that I get matches on post, ranging from 2005 to 2003. Recent posts are nowhere to be found.
We are getting strange results as well. Looking at results via sphinx "search" CLI shows me that there are hits in the right date range, but in our case we have lots of data since August, and spotty or missing data going back to 2003, light data back from there to 2001, and then a whole lot of data again. It's like the index is "light" for a few years.

=========

Bzzt! I forgot to update search.php after the last VB upgrade (it was on my to-do list, I just muffed it)

If you have weird data being returned, take a look at your search.php to see if you accidentally upgraded it and lost your edits!
Reply With Quote
  #419  
Old 11-01-2007, 03:05 PM
eoc_Jason's Avatar
eoc_Jason eoc_Jason is offline
 
Join Date: Dec 2001
Location: Houston, TX
Posts: 493
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Didn't want to see this thread die as I still heavy rely on Sphinx for my vB search. It's disappointing that the vB team *still* has not come up with a built-in solution for searching that is acceptable for large forums.

One bit of advice - The code in the vB search does a lot of weighting & filtering. I have many instances where I search for a specific word that is in a post, and doing a raw search I can find it, but after vB works its magic it will give a 'no results found'. So don't think something is broken, I guess technically it is, but it's 'by design'.

Anyhow, here's some bits of code that might help get your sphinx running a little smoother.

One thing you really should do is run the search daemon & indexer under a non-root user, I use 'sphinx'. If yours is different than simply adjust the files accordingly. you will probably need to make a few directories mentioned in the scripts below and have them owned by your user that sphinx is running as. (like for the lock file & log)

I run on Redhat / CentOS, here's my script that goes in the /etc/rc.d/init.d/ directory. I call it 'searchd'. Simply use 'chkconfig' to add it and have it start up when you system boots.

Code:
#!/bin/sh
#
# searchd   This script starts and stops the sphinx search engine
#
# chkconfig: - 80 15
#
# description: Stand Alone Search Engine
# processname: searchd
# config: /usr/local/etc/sphinx.conf
# pidfile: /var/run/searchd/searchd.pid


# Source function library.
. /etc/rc.d/init.d/functions

RETVAL=0

start() {
        echo -n "Starting Sphinx: "
        sudo -u sphinx /usr/local/bin/searchd --config /usr/local/etc/sphinx.conf > /dev/null 2>&1
        RETVAL=$?
        if [ $RETVAL -eq 0 ]; then
          success startup
          touch /var/lock/subsys/searchd
        else
          failure startup
        fi
        echo
        return $RETVAL
}

stop() {
        echo -n "Shutting Down Sphinx: "
        kill `cat /var/run/searchd/searchd.pid`
        RETVAL=$?
        if [ $RETVAL -eq 0 ]; then
          success shutdown
          rm -f /var/lock/subsys/searchd /var/run/searchd/searchd.pid
        else
          failure shutdown
        fi
        echo
        return $RETVAL
}

restart() {
        stop
        start
}

case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart)
        restart
        ;;
  status)
        status searchd
        ;;
  *)
        echo $"Usage: $0 {start|stop|restart|status}"
        exit 1
esac

exit $RETVAL
Second, here is my cron script. I made a directory called 'cron.quarterly' that runs every 15min. Similar to the cron.hourly, cron.daily, etc... You can do whatever. If you make that directory be sure to edit your /etc/crontab file accordingly too. I added a time-check so once a day sphinx will do a full-reindex, and it makes a lock file so if yours takes a long time to reindex you won't run into issues. Basically at 5am it will do a re-index. I chose that time because it's a low usage period for my forum and server. By default the cron.daily scripts run at 4am so you wouldn't want to do it then since everything else will be eating CPU cycles.

Code:
#!/bin/sh

# the lockfile is not meant to be perfect, it's just in case the
# two sphinx cron scripts get run close to each other to keep
# them from stepping on each other's toes.

LOCKFILE=/var/lock/subsys/sphinx_indexer

# If the lockfile exists then exit!

if [ -f $LOCKFILE ]; then
  echo "Lockfile already exists, not running sphinx indexer!"
  exit
fi;

touch $LOCKFILE

compareh=$(date +%k)
comparem=$(date +%M)

if [ $compareh -eq "5" ] && [ $comparem -le "14" ]; then
  sudo -u sphinx /usr/local/bin/indexer --config /usr/local/etc/sphinx.conf --rotate --all > /dev/null 2>&1
else
  sudo -u sphinx /usr/local/bin/indexer --config /usr/local/etc/sphinx.conf --rotate post_index_delta thread_index_delta > /dev/null 2>&1
fi;

rm -f $LOCKFILE

exit 0
I also have my logs in their own /var/log/searchd/ directory, you set this in the sphinx config (most people probably just have them in the /var/log dir). Again that directory will need to be owned by the sphinx user that you use.

Code:
/var/log/searchd/*.log {
        missingok
        compress
        postrotate
          if test -n "`ps acx|grep searchd`"; then
            /sbin/service searchd restart  2> /dev/null > /dev/null || true
          fi
        endscript
}
Oh, and for people wanting to know how to implement the sort_search_items(), here's some code. It goes in the includes/sphinx.php file.

At the end where there is the following:
Code:
                        if ($vbulletin->GPC['titleonly'] == $vbulletin->GPC['showposts'])
				$orderedids[$docinfo['attrs'][$sphinx_switch_fields]] = $docinfo['attrs'][$sphinx_switch_fields];
                        else
				$orderedids[] = $doc;
                }       
        }       
        else
		$orderedids = array();
}
Replace that with:

Code:
                        if ($vbulletin->GPC['titleonly'] == $vbulletin->GPC['showposts'])
                        {
                                $orderedids[$docinfo['attrs'][$sphinx_switch_fields]] = $docinfo['attrs'][$sphinx_switch_fields];
                                $itemids[$docinfo['attrs'][$sphinx_switch_fields]] = $docinfo['attrs'][$sphinx_switch_fields];
                        }
                        else
                        {
                                $orderedids[] = $doc;
                                $itemids["$doc"] = true;
                        }
                }
        }
        else
        {       
                $orderedids = array();
        }
        
        // #############################################################################
        // now sort the results into order
        // #############################################################################
        if (!$vbulletin->GPC['titleonly'] OR $vbulletin->GPC['showposts'])
        {       
                // sort by database field
                if ($vbulletin->GPC['sortby'] == 'post.dateline' || $vbulletin->GPC['sortby'] == 'lastpost')
                {       
                        if (empty($itemids))
                        {       
                                $errors[] = array('searchnoresults', $displayCommon);
                        }
                        else
                        {       
                                // remove dupes and make query condition
                                $itemids = iif($vbulletin->GPC['showposts'], 'postid', 'threadid') . ' IN(' . implode(',', array_keys($itemids)) . ')';
                                
                                // sort the results and create the final result set
                                $orderedids = sort_search_items($itemids, $vbulletin->GPC['showposts'], $vbulletin->GPC['sortby'], $vbulletin->GPC['sortorder']);
                        }
                }
        }
        
// END Results  
}
Be sure to leave the unset line at the very very bottom.


I'm about to make the upgrade to vB 3.6.8 whatever, and I'm running php 5.2.4, so I'll let you know how the upgrade goes. I want to look over everyones modifications and updates in this thread. Then I'll post some more files and stuff if necessary.
Reply With Quote
  #420  
Old 11-01-2007, 03:34 PM
mute mute is offline
 
Join Date: Dec 2002
Location: Phoenixville, PA
Posts: 152
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Thanks for that search results sort order fix, it seems to be working nicely.

All that really stands between this being a perfect solution is finding a way to handle all of the "Find all posts by user" type searches.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 05:52 AM.


Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2024, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.04486 seconds
  • Memory Usage 2,282KB
  • Queries Executed 12 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (7)bbcode_code
  • (2)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (2)pagenav_pagelinkrel
  • (10)post_thanks_box
  • (10)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (10)post_thanks_postbit_info
  • (10)postbit
  • (10)postbit_onlinestatus
  • (10)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete