robots.txt question

[David Rose]

Hello,
I just opened my first Vbulletin site a few weeks ago and I was suprised to see that it wasn't yet indexed by Google.

I added the webmaster tools just now and I already received the next message:
We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.
I know basically what the robots.txt file is but I wasn't even thinking on using it at the moment.
Is that something I did? Is there an option I need to disable? How do I take it off?

Thanks !

[Brandon Sheley]

you should add a robots.txt file
here is mine if you want an example
http://www.general-forums.com/robots.txt

[David Rose]

Thanks for your reply,
at the moment I'd like the Search Engines to crawl through what ever they want.

In that case, is there a robots.txt file that always exists and I need to take it off?

Or in other words, how do I disable this and allow Google to crawl through anything?

Thanks.

[Brandon Sheley]

I wouldn't advise letting them "crawl what they want"
you'll do far better if you restrict the bots to crawling the actual content, not pages like the member pages, or faq for example..

[David Rose]

Yeah I know what you mean but I think that I'll decide in a later stage, the forum is compeltly new and it doesn't matter to me right NOW.

Anyways, what should I do in order to get rid of it for now? I want Google to index the homepage and it won't because of that.

I actually was surprised that it's there, is this file coming automatically with the VBulletin ?

[StarBuG]

Connect to your forum root via FTP and remove the robots.txt

Or chmod it 644 so google can read it.
The problem isn't the restriction inside, it is that google is unable to access/read it at all.

[David Rose]

Thank you, well seems like this problem was solved but now I have a new one.
When adding new pages to the Sitemap, I receive Sitemap errors and warnings:

Sitemap is HTML
Your Sitemap appears to be an HTML page. Please use a supported sitemap format instead.


Any idea what's wrong?

Thanks !

[jkcerda]

Originally Posted by Loco.M

you should add a robots.txt file
here is mine if you want an example
http://www.general-forums.com/robots.txt

ok, let me get this straight, all you do is make the robots.txt file and load it to the root folder?
like http://www.vbulletin.com/forum/showt...obots.txt-file

[your24hourstore]

if you want a robots.txtx file that is what you must do :>

they dont put themselves up! LOL

--------------- Added [DATE]1274183085[/DATE] at [TIME]1274183085[/TIME] ---------------

Originally Posted by David Rose

Thank you, well seems like this problem was solved but now I have a new one.
When adding new pages to the Sitemap, I receive Sitemap errors and warnings:

Sitemap is HTML
Your Sitemap appears to be an HTML page. Please use a supported sitemap format instead.


Any idea what's wrong?

Thanks !

if you read at all in here you have seen my rants about the sitemaps generated by this software. not only does it create weird errors , sometimes it wont work at all, if you are going to run it, i suggest you run xml sitemap generator paid version, or if you want vbseo.

but vbull has a real problem with its sitemaps, if you use it, you need to set it run create sitemap daily, that way you don't have a messed up sitemap for days or weeks.

instead of running xmlsitemap.php
go to xml sitemap generators google it, and use theirs to generate a xml sitemap, then upload it to your root and point webmaster to that file.

I have never been able to find anyone here or on vbull forums that will address this issue its like your invisible if you report errors in sitemap, but go ahead and start a ticket over at vbull and get ready for them to tell you its your fault.

[jkcerda]

Originally Posted by harleyparts

if you want a robots.txtx file that is what you must do :>

they dont put themselves up! LOL



if you read at all in here you have seen my rants about the sitemaps generated by this software. not only does it create weird errors , sometimes it wont work at all, if you are going to run it, i suggest you run xml sitemap generator paid version, or if you want vbseo.

but vbull has a real problem with its sitemaps, if you use it, you need to set it run create sitemap daily, that way you don't have a messed up sitemap for days or weeks.

instead of running xmlsitemap.php
go to xml sitemap generators google it, and use theirs to generate a xml sitemap, then upload it to your root and point webmaster to that file.

I have never been able to find anyone here or on vbull forums that will address this issue its like your invisible if you report errors in sitemap, but go ahead and start a ticket over at vbull and get ready for them to tell you its your fault.

thanks HR
now sitemaps
here?
http://www.xml-sitemaps.com/