This is what my Robots.txt looks like:
Code:
User-agent: thetechgenius-Google
Disallow:
User-agent: *
Disallow: /cache.php
User-agent: *
Disallow: /go/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /attachment.php
Disallow: /search.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /editpost.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /login.php
Disallow: /subscription.php
Disallow: /private.php
Disallow: /report.php
Disallow: /sendmessage.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /postings.php
Disallow: /sendtofriend.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /showgroups.php
Disallow: /announcement.php
Disallow: /downloads.php
Disallow: /downloads.html
Disallow: /**********/ #<- Renamed AdminCP Directory -- Blanked out for security reasons
Disallow: /*********/ #<- Renamed ModCP Directory -- Blanked out for security reasons
Sitemap: https://thetechgenius.net/dbseositemap.php #<- This is my Sitemap from the DBSEO Addon --
As for the Sitemap, you have to put the URL that generates the Sitemap. If you go to my Sitemap URL, it will Generate and Start downloading the Sitemap. I'm not sure how vBSEO handles Sitemaps.
10-25-2014, 04:52 PM
Threaded Mode