Sphinx Search

[amcd]

my forum is 200k threads and 10mil posts, vb 3.8.4. I have only one database (no slave), nginx webserver, Core I7 with 12GB RAM

It is time for you to move to dual servers - one for webserver/PHP and another for MySQL.

[kris]

It is time for you to move to dual servers - one for webserver/PHP and another for MySQL.

no money making here just spend

I think spliting big post table to smaller read only archived tables will be cheaper and even better solution and of couse Sphinx for search.

[mute]

Originally Posted by kris

mute, can you share how did you archive post table ? What changes did you do in code and MySQL ? I want to move my old posts to another post_archive table but I am not sure how can I join those tables from vbulletin code.

It's REALLY nasty. I really don't think you want to do it. In fact, I'm thinking about abandoning it on our site.

Back when we wrote it, we were probably at like 25 million posts, on (if I remember right), like a dual xeon with HT. Now, we're on a Quad Quad xeon box with 16gb of ram. We have 30 million in our archived tables (10x3 mill posts each) + 30 million in our post table. I'm not seeing any slowdowns against the post table, which has me wondering if we'd be seeing any slowdowns if I was pulling against all 60 million in one table, given how much faster our CPUs have gotten and how much ram we have sitting around.

[masons]

Hi,

I have sphinx installed for my wiki since a few days, and now am looking to get it working with my vbulletin setup,

But, my server load went a bit overboard this morning (120+) and I have no idea how to work with that,..... any tips on taking some presure of the load? Before I add this to vbull?

Some server stats (dedicated)
Processor #1 Vendor: GenuineIntel
Processor #1 Name: Intel(R) Core(TM)2 Duo CPU E8300 @ 2.83GHz
Processor #1 speed: 1998.000 MHz
Processor #1 cache size: 6144 KB

Processor #2 Vendor: GenuineIntel
Processor #2 Name: Intel(R) Core(TM)2 Duo CPU E8300 @ 2.83GHz
Processor #2 speed: 1998.000 MHz
Processor #2 cache size: 6144 KB

[amcd]

go through this thread over at vb.com

[eoc_Jason]

Originally Posted by mute

We have 2 post indexes, one or our live post table, and one for our archived post table. They each have 30 million posts each. I don't see a point in sharding the post indexes aside from being able to take advantage of multiple CPUs when indexing.

The way I see it, if I can keep the old indexes online while I do a full reindex, I don't really care how long the full reindex takes since (at least in our case), the search server is just a slave database server and not our primary.

Splitting up the posts index into several sources has more advantages that just re-indexing. As kmike eluded to, you can setup "agents" on your server so that when a person does a search it will search all the sources in parallel utilizing one source per CPU.

For most people though, even large indexes we are all probably just using 1 CPU without realizing it and still maintaining less than one second search times...

This week I'm going to be swamped working on some new AMD processor reviews, so work on the new sphinx code will have to be set aside until Tuesday next week.

I'm super pumped though, after diving into the documentation and code, I realized it's really not that bad at all...

[kris]

Splitting up the posts index into several sources has more advantages that just re-indexing. As kmike eluded to, you can setup "agents" on your server so that when a person does a search it will search all the sources in parallel utilizing one source per CPU.

eoc_Jason, can you share how to split post table index into several sources ?

[kmike]

FYI, realtime index updates in Sphinx are near:
http://twitter.com/sphinxsearch/statuses/8179531738
Looks like so far the realtime indexes are only accessible via SphinxQL.

[weeno]

hey guys,

Can someone post their sphinx.conf file for sphinx version 0.9.9? I'm still running into a few problems so wanted to compare.

arn

[boggseric]

Was post number 1 supposed to contain some XML mods? Or am I missing something here.

edit: Never mind, there are no xml files, you have to create the plugins manually and copy and paste the code into them.