Does Google hate big boards?
 

Hi, I was wondering what makes some big boards be excluded from Google. Does Google penalize these boards for some reason?

Gaia Online, 1,074,929,715 posts:
Google: about 132,000 from gaiaonline.com

FaceTheJury, 135,719,579 posts:
Google: about 33,100 from forums.facethejury.com

Offtopic.com, 79,896,449 posts:
Google: about 141 from forums.offtopic.com
(!!!)

nope,
www.atraf-f.net - 1,200,000~ posts
Results 1 - 10 of about 172,000 from www.atraf-f.net. (0.19 seconds)

in google israel we have 233,000 results

That's the thing. Smaller forums seem to have more indexed pages that bigger ones.

The forum you mentioned has just over 1 million posts, and still has more indexed pages that the forum with 1 BILLION posts.

And that's not the only example. I've heard Google banning some sites with inappropriate content, but all big forums can't fall into this category... And still... Big postcount, small number of indexed pages.

???

I think some big boards ask google to exclude them from their index - because they don't WANT to get bigger.

Hmm, I never thought of that... This sounds resonable...

some sites exclude web crawlers in general not only because of the advertisement, but also because it is resource expensive...

when a web crawler visit your site, it hits hundreds of pages in a minute, compared to a member who can hit 3 or 4 pages per minute...

i have on my site yahoo that crawl my site once per day, 58 to 100 hits per minute... and go after 10 minutes... i have 0 members... just infos... imagine a site with 1 million posts... yahoo would crawl the site like 1000 times the count i have... webhosts hates that...

not at all.

I know yahoo has this horrible habbit of sending 50-100 bots at a time, had 50 the other day on my page.

Maybe bad SEO ;)

Most of Offtopic.com's forum are not visible to guests, so it makes sense that they don't have many indexed pages. I don't know about the rest of them though.

i have a small site... year old w/ 1000 members 4000 threads and 50000 posts. i had 772 yahoo critters on there at once a few days ago... my record stats went though the roof. i thought my site was broke.

Ya, exactly. Or they don't want to use up even more bandwidth.

Many forums that are setup for good SEO don't allow spiders to index showpost.php, so whole threads are indexed via showthread.php. On any big board that's setup like that, the indexed pages listed in Google will be much lower than the total posts on the forum.

Sites like Gaia Online just don't have to care about google. They've got more than they need, and probably more coming every day.

It's just good to not getting any extra load from spiders etc.

It makes me wonder about the capability of google to efficiently spider large boards. But a good sitemap.org solution should fix that.

Google uses artificial intelligence to spider pages, and may be up to 300 times more efficient than other spiders, using traditional algorithms. This efficiency is actually the main reason why Google needs so few bots compared to many other spiders. No matter what however, the bandwidth is the same ;)

No , Google hates sites with bad SEO. Of course, these huge boards don't care mutch about SEO, they got all the members ands visitors they need.

Well said.

Offtopic.com doesn't show guests threads. Which is why their number is so low.