Sphinx Search

[RedWingFan]

Originally Posted by UK Jimbo

Is the cronjob that's running the indexer creating any output?

Do you see any output if you run the indexer from the command line?

Can you copy/paste the command line you're using along with any output back here?

Here is the command line I use (within cron):

/usr/home/xxxx/sphinx/bin/indexer --config /usr/home/xxxx/sphinx/bin/sphinx_rr.conf --rotate postdelta threaddelta

I just ran it with this output:

Sphinx 0.9.8-rc2 (r1234)
Copyright (c) 2001-2008, Andrew Aksyonoff

using config file '/usr/home/xxxx/sphinx/bin/sphinx_rr.conf'...
indexing index 'postdelta'...
collected 10 docs, 0.0 MB
collected 10903 attr values
sorted 0.0 Mvalues, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 10 docs, 2932 bytes
total 0.019 sec, 152923.38 bytes/sec, 521.57 docs/sec
indexing index 'threaddelta'...
collected 2 docs, 0.0 MB
collected 100 attr values
sorted 0.0 Mvalues, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 2 docs, 40 bytes
total 0.010 sec, 4000.00 bytes/sec, 200.00 docs/sec
rotating indices: succesfully sent SIGHUP to searchd (pid=605).

It's frustrating because it is an intermittent error--I kept trying to reproduce it but had no luck.

The output looks OK. But, it's updating the threaddelta.new.* files, not the threaddelta.* files (which remain at zero bytes, dated 5/29/09).

It's not a huge deal, since I can dump and regenerate all the indexes, but I just don't want this to happen if we put this on our "production" forum, and was curious to know how it happened.

[UK Jimbo]

Those .new files relate to the --rotate option and the indexer read more about it here: http://sphinxsearch.com/docs/current.html#ref-indexer

I would suspect that something preventing the indexer from rotating the files out perhaps the file permissions?

I'd suggest deleting all of the threaddelta files and then re-indexing.

[RedWingFan]

Originally Posted by UK Jimbo

Those .new files relate to the --rotate option and the indexer read more about it here: http://sphinxsearch.com/docs/current.html#ref-indexer

That was what I suspected--something like a temporary file, IOW.

Originally Posted by UK Jimbo

I would suspect that something preventing the indexer from rotating the files out perhaps the file permissions?

Nothing has changed on the server, so I'd attribute it to some kind of glitch in one (or all?) of those files dating back to May. Permissions and ownership matches all the others in the same directory, which was the first thing I checked.

I just tried deleting the threaddelta files, and reran the command line:

/usr/home/shtv/sphinx/bin/indexer --config /usr/home/shtv/sphinx/bin/sphinx_rr.conf threaddelta

This created a new set of threaddelta.* files. OK, so far so good. But then I go and retry with the --rotate option, and we're back to having the .new.* files, and the threaddelta.* files don't get updated.

In the same directory, the post.* and postmeta.* files are all working properly.

Deeply weird...I could see if I had changed the configuration of this mess awhile ago, but I actually haven't touched it since July last year, when I first installed it. No other changes on the server, and we have plenty of disk space.

Still poring over the Sphinx docs you pointed to...but am not seeing much else helpful yet.

Thanks!

[UK Jimbo]

have you tried restarting searchd and then trying it all over again?

Sounds like a strange one. Perhaps worth posting on the forum over at sphinxsearch.com

[RedWingFan]

Originally Posted by UK Jimbo

have you tried restarting searchd and then trying it all over again?

I'm willing to try anything. Although, the problem is with the indexer...unless something in searchd is somehow preventing the files from rotating properly (maybe searchd reporting that it's "busy", in other words, so the .new.* files don't get rotated in).

Originally Posted by UK Jimbo

Sounds like a strange one. Perhaps worth posting on the forum over at sphinxsearch.com

I'll do a search over there--thanks! I'll probably try dumping the entire index, regenerating a new one, restart searchd, etc., and start with a clean slate before pestering them too much over there.

[UK Jimbo]

Originally Posted by RedWingFan

I'm willing to try anything. Although, the problem is with the indexer...unless something in searchd is somehow preventing the files from rotating properly (maybe searchd reporting that it's "busy", in other words, so the .new.* files don't get rotated in).

In my mind it's 50:50 whether it's the indexer or searchd which is causing the problem. The indexer seems to be creating the index happily with the .new file name. After that it's the job of the search daemon to rotate the new index in.

With some of those .new files there what happens if you signal a restart to searchd with (not tested but I think this is correct).

killall -HUP searchd

That's the same method that the indexer uses to signal a rotate to searchd. I can't remember if searchd keeps a system log of it's activity. if so the restarts (and any possible problems) might be reported.

Good luck!

[RedWingFan]

I did:

searchd --config /.../my.conf --stop

...to stop, then restarted searchd. First time I ran the indexer, the .new.* files all were deleted. One hour and about 20 rotations later, they're still gone.

Thanks much--it worked!

[UK Jimbo]

Afraid I've not trawled the whole thread for this but here's a product that makes the install of the two small plugins even easier. Use at your own risk, etc...

[RedWingFan]

Originally Posted by UK Jimbo

Afraid I've not trawled the whole thread for this but here's a product that makes the install of the two small plugins even easier. Use at your own risk, etc...

Thanks! I'll try it out and let you know how it works. I'm about to pull the plug on fulltext and try Sphinx on our main (production) forum. Can't live w/o InnoDB tables, as I'm finding. I'll report back here, good or bad.

The hardest part is trying to remember what I did to get Sphinx running on our test forum. Working my way through the posts here, and my own notes...I think I'm getting it.

[UK Jimbo]

Originally Posted by RedWingFan

The hardest part is trying to remember what I did to get Sphinx running on our test forum. Working my way through the posts here, and my own notes...I think I'm getting it.

I'd recommend (this is true of any production roll-out):

• Backup test site
• Place a full copy of the live code (and db dump too if you think it's needed)
• Go through migration/roll-out process on your test platform making notes
• Repeat the above until you're confident of the steps you need to make
• Repeat on production. It'll be easy as you've just done it on the test platform!