Search Engine Spiders [Archive]

View Full Version : Search Engine Spiders

tHE DSS

11-13-2002, 10:45 PM

I'm noticing that search engines crawl our forums, but don't actually index anything other than a top level forum... never the threads.

I know they have a problem with URL parameters, like ? &, etc. etc.... and have seen a hack in the Hacks database here that attempts to sort it out.

The one that i've seen changes URLs to a more search crawler friendly form, but, i've seen vB forums around the Internet that seem to get indexed with their normal URLs in tact.

How?

okrogius

11-13-2002, 11:49 PM

Forum posts are not considered relevant high quality content.

Erwin

11-14-2002, 02:23 AM

Google does index URLs with ? in them, but not as well as if the URL doesn't have them. My forums now has over 30,000 pages indexed because I use the SE-friendly hack. It works. :) Before, I only had 100 pages in.

tHE DSS

11-14-2002, 08:43 PM

Where do I find this "SE-friendly" hack? I've searched the database, and only really find 1 hack that attempts to sort out the search engine thing.

To be honest, though it looks great, i'd rather not use it, because it tampers with all the URLs. I want our URLs to look normal, but I want us also getting a good listing in places such as Google.

Cheers.