The Arcive of Official vBulletin Modifications Site.It is not a VB3 engine, just a parsed copy! |
|
#1
|
|||
|
|||
robots.txt question
Hello,
I just opened my first Vbulletin site a few weeks ago and I was suprised to see that it wasn't yet indexed by Google. I added the webmaster tools just now and I already received the next message: We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely. I know basically what the robots.txt file is but I wasn't even thinking on using it at the moment. Is that something I did? Is there an option I need to disable? How do I take it off? Thanks ! |
#2
|
||||
|
||||
you should add a robots.txt file
here is mine if you want an example http://www.general-forums.com/robots.txt |
#3
|
|||
|
|||
Thanks for your reply,
at the moment I'd like the Search Engines to crawl through what ever they want. In that case, is there a robots.txt file that always exists and I need to take it off? Or in other words, how do I disable this and allow Google to crawl through anything? Thanks. |
#4
|
||||
|
||||
I wouldn't advise letting them "crawl what they want"
you'll do far better if you restrict the bots to crawling the actual content, not pages like the member pages, or faq for example.. |
#5
|
|||
|
|||
Yeah I know what you mean but I think that I'll decide in a later stage, the forum is compeltly new and it doesn't matter to me right NOW.
Anyways, what should I do in order to get rid of it for now? I want Google to index the homepage and it won't because of that. I actually was surprised that it's there, is this file coming automatically with the VBulletin ? |
#6
|
||||
|
||||
Connect to your forum root via FTP and remove the robots.txt
Or chmod it 644 so google can read it. The problem isn't the restriction inside, it is that google is unable to access/read it at all. |
#7
|
|||
|
|||
Thank you, well seems like this problem was solved but now I have a new one.
When adding new pages to the Sitemap, I receive Sitemap errors and warnings: Sitemap is HTML Your Sitemap appears to be an HTML page. Please use a supported sitemap format instead. Any idea what's wrong? Thanks ! |
#8
|
|||
|
|||
Quote:
like http://www.vbulletin.com/forum/showt...obots.txt-file |
#9
|
|||
|
|||
if you want a robots.txtx file that is what you must do :>
they dont put themselves up! LOL --------------- Added [DATE]1274183085[/DATE] at [TIME]1274183085[/TIME] --------------- Quote:
but vbull has a real problem with its sitemaps, if you use it, you need to set it run create sitemap daily, that way you don't have a messed up sitemap for days or weeks. instead of running xmlsitemap.php go to xml sitemap generators google it, and use theirs to generate a xml sitemap, then upload it to your root and point webmaster to that file. I have never been able to find anyone here or on vbull forums that will address this issue its like your invisible if you report errors in sitemap, but go ahead and start a ticket over at vbull and get ready for them to tell you its your fault. |
#10
|
|||
|
|||
Quote:
thanks HR now sitemaps here? http://www.xml-sitemaps.com/ |
Thread Tools | |
Display Modes | |
|
|
X vBulletin 3.8.12 by vBS Debug Information | |
---|---|
|
|
More Information | |
Template Usage:
Phrase Groups Available:
|
Included Files:
Hooks Called:
|