PDA

View Full Version : Question: Robots.txt File


Narutoboy
09-29-2013, 03:02 AM
Is there anything that I've disallowed that will hurt me as far my rankings in Google?


User-agent: *
Disallow: /admincp/
Disallow: /arcade/
Disallow: /archive/
Disallow: /cgi-bin/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customgroupicons/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /includes/
Disallow: /modcp/
Disallow: /packages/
Disallow: /signaturepics/
Disallow: /vb/
Disallow: /ajax.php
Disallow: /album.php
Disallow: /announcement.php
Disallow: /asset.php
Disallow: /assetmanage.php
Disallow: /attachment.php
Disallow: /attachment_inlinemod.php
Disallow: /blog_ajax.php
Disallow: /blog_external.php
Disallow: /blog_inlinemod.php
Disallow: /blog_report.php
Disallow: /blog_search.php
Disallow: /blog_subscription.php
Disallow: /blog_usercp.php
Disallow: /calendar.php
Disallow: /clear.gif
Disallow: /converse.php
Disallow: /cron.php
Disallow: /editor.php
Disallow: /editpost.php
Disallow: /external.php
Disallow: /faq.php
Disallow: /global.php
Disallow: /group_inlinemod.php
Disallow: /groupsubscription.php
Disallow: /image.php
Disallow: /infraction.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /list.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderation.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /picture.php
Disallow: /picture_inlinemod.php
Disallow: /picturecomment.php
Disallow: /posthistory.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /sendtofriend.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /visitormessage.php
Disallow: /widget.php

CAG CheechDogg
09-29-2013, 03:59 AM
Leave your admincp, modcp out of the robots.txt file, those are areas you don't even want the bots to know where they are ...

If anything put a htacess password on them

Narutoboy
09-29-2013, 03:05 PM
Leave your admincp, modcp out of the robots.txt file, those are areas you don't even want the bots to know where they are ...

If anything put a htacess password on them

Thanks, didn't think about that.

socialteenz
09-29-2013, 05:25 PM
Hello, feel free to include your sitemap at the end of the robots.txt.

Also, it is better to add something like this at the top, since we know google bot's name.


User-agent: Mediapartners-Google
Disallow:

User-agent: *
Crawl-delay: 30
Add your list here.

Sitemap: http://sitemapurl

You can validate your sitemap here (http://tool.motoricerca.info/robots-checker.phtml).