Sphinx Search

[Spinball]

Originally Posted by mlx

I see.

Still ... the point is that the free solution can be setup the same way. Our search index is updating every 5 minutes without any significant load too. Rebuilding all indexes is a totally different thing. That setup is totally based on the free information within this thread.

That's probably the reason for the slightly offensive post earlier today ... sorry for that

No problem . Still the search keyword suggestions are cool, don't you think? Some useful info there for marketing purposes also.

[Simetrical]

Originally Posted by Spinball

Just wanted to post a note here to thank the folks who pointed me to Floren Munteanu's Sphinx solution which is now installed at www.AVForums.com.
I was so impressed that I made a video at http://www.youtube.com/watch?v=zJkVHcqfEBQ (choose the HD version).
Thanks, guys.

I'm curious, did he suggest that you post here?

Originally Posted by Spinball

Ok it was every day when I last used it. Have things changed since then? Apologies if so. Please let me know and I will change the video.
As I understand it, though the Sphinx reindex in the free solution was quite server intensive and took a while?

No. I run it every hour and don't even notice the load, either from indexing or from searching. I could probably run it every five minutes with no problem.

Originally Posted by Spinball

(By the way there is need to be offensively defensive about the free solution - we used it for ages and I'm not running down the efforts that people have put in for free).

I think people are not so much defensive about the free solution as annoyed at TECK for continually advertising his paid solution for personal profit in a thread devoted to the free solution. He had a bunch of posts in this thread deleted some time ago for advertising, and since then you're the second or third person to bump the thread and e-mail everyone watching it to advertise his solution as well.

Nothing wrong with paid solutions for those who have trouble with the free ones, but I don't think this is the place to repeatedly advertise them, which is what's been happening in this thread over the past few months. It's off-topic, this thread is about the free hack. People can put this stuff in their signatures if they want to advertise it, or run Google ads, or mention it as an option when they're giving free help or otherwise posting on-topic.

[Spinball]

Originally Posted by Simetrical

I'm curious, did he suggest that you post here?

No. I run it every hour and don't even notice the load, either from indexing or from searching. I could probably run it every five minutes with no problem.

I think people are not so much defensive about the free solution as annoyed at TECK for continually advertising his paid solution for personal profit in a thread devoted to the free solution. He had a bunch of posts in this thread deleted some time ago for advertising, and since then you're the second or third person to bump the thread and e-mail everyone watching it to advertise his solution as well.

Nothing wrong with paid solutions for those who have trouble with the free ones, but I don't think this is the place to repeatedly advertise them, which is what's been happening in this thread over the past few months. It's off-topic, this thread is about the free hack. People can put this stuff in their signatures if they want to advertise it, or run Google ads, or mention it as an option when they're giving free help or otherwise posting on-topic.

Ok I understand your point completely and I want to stress that I made my own mind up to thank the people who directed me towards the other solution. I posted here merely cos I was so enthusiastic and maybe I let that get the better of me.
I understand also that you would get annoyed at someone pimping their solution and I don't want to be accused of being associated with that so I will not post in here any more about it.
If you could reindex every 5 minutes then I would suggest that you do. Seems to me that users will want their posts to appear in the search ASAP.

[Simetrical]

Ah, it looks like it's actually every 20 minutes that I have the deltas built. No one's ever complained that it takes 10 minutes on average for their posts to appear in the index, so I'm fine with leaving it as it is.

It takes under 200 ms for rotate.sh to run, so actually I could probably run it every minute if I really felt like it.

[TECK]

Originally Posted by mlx

Sure, re-indexing all configured indexes will take some time and might be quite server intense on a database with millions of posts. So are you saying that the paid solution is running "indexer --all --rotate" to re-index all configured indexes every 20 minutes in no time?

Or is it updating the delta indexes every 20 minutes pretty much as described here in 2007?

If you do an "indexer --all --rotate" every 20min, you will blow your servers. It takes in average 20-30min to rebuild the indexes from scratch on a 10GB database. In my product, the indexes are updated every 10min, regardless of the database size. It takes in average 30sec-1min to refresh the data, all the time with no "indexer --all --rotate" commands used ever. Of course, that includes the threads/posts that were deleted or edited. What is the point to store a deleted post into indexes. Not to mention that if you edit the contents of a post, the deleted keywords should not be available into search. When you rebuild the indexes, all "errors" are gone... but that occurs every 24hrs for the vB.org product.

Also, I don't know if you consider this advertising... because I don't. It is a logical explanation on how my system works. It is like talking about Community Server at vBulletin... yet I don't see anyone chasing the people who reply about an open asked question related to their product and replied by one of their reps.

[mlx]

Yeah I'm sure your product is great for those who can afford it and probably superior compared to the free solution. However we knew that before. No need to repeat it over and over again. I guess that's why some people become pissed off here. Easy as that.

Just my 2 cents though.

[TECK]

<i>> No need to repeat it over and over again.</i>
I totally agree. In the previous posts, I only replied to your technical question and passed the general questions.

Regards,

[Simetrical]

Originally Posted by TECK

If you do an "indexer --all --rotate" every 20min, you will blow your servers. It takes in average 20-30min to rebuild the indexes from scratch on a 10GB database. In my product, the indexes are updated every 10min, regardless of the database size. It takes in average 30sec-1min to refresh the data, all the time with no "indexer --all --rotate" commands used ever. Of course, that includes the threads/posts that were deleted or edited. What is the point to store a deleted post into indexes. Not to mention that if you edit the contents of a post, the deleted keywords should not be available into search. When you rebuild the indexes, all "errors" are gone... but that occurs every 24hrs for the vB.org product.

That doesn't seem like a huge advantage to me. The deltas seem to work well enough. I don't actually know how Sphinx works, though ― you're saying that the solution posted here, rotating only the deltas, doesn't pick up edited posts? Any other disadvantages to it?

[TECK]

Personally, I think it is important to have accurate results. If deleted or edited posts still show in search results (when it should not) as well that once a day you perform a full reindex (when you should not), that affects the overall search accuracy as well the server performance. Think of it this way: you have several threads in a forum where the users change a price for their "to be sold" items published few days ago. Because your search index is not updated right away (10min max), other forum users will never know that the price on certain items was revised until next day when indexes are rebuilt entirely...

A better example, related to search accuracy. Let's presume you use the default vBulletin search, query the entire posts for 'spaghetti' (most intensive) and display the results as threads. Then, you perform the same type of search with the vB.org Sphinx search. You will notice the number of missing results very easy, while performing the search with vB.org Sphinx product. There are many other aspects that I rather not cover, because it will sound like I'm trying to advertise my product... Fell free to ask more questions in my forum.

[charlie71]

Its a great piece of software!
May one of you can help me with my problem:

Search for word -> No results
Search for the same word again -> Results are found