External search engine + vBulletin?

[UK Jimbo]

Has anyone considered using an external search engine for indexing/searching forum archives?

I've got some experience with some opensource software called amberfish and have used it to index large chunks of XML. Amberfish's author plans to index a terabyte of data with it. It's possible to do native XML indexing so it's aware of the different fields within an XML document.

The process would have to involve dumping out the posts into XML files (probably one file for many posts - say 10,000 but possibly single files) then indexing that XML data.

The content could then be searched, with field constraints if required (i.e. only look in the thread title), then return an XML representation of the matching posts. This XML result set could then be parsed and used to create the search results page. Alternatively the result set could just return and ID list then you could query the database and fetch the thread/post data from there.

I'd be interested in any comments or thoughts about this suggestion and details of anyone's tried dabbling with this before.