vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vB4 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=251)
-   -   What should my robot.txt file look like? (https://vborg.vbsupport.ru/showthread.php?t=315043)

SpottySocks 10-24-2014 09:29 PM
What should my robot.txt file look like?
 
Hi guys, I have had an alert from Google Adsense saying the google robot has been denied. This is out of the blue as I haven't had a problem before.

Instructions say to remove the following from my robot.txt file:

Code:
User-agent: Mediapartners-Google
Disallow: /
This is my current robot.txt file content:

Code:
User-agent: *
User-agent: Mediapartners-Google
Disallow:
Disallow:/admincp/
Disallow:/clientscript/
Disallow:/cpstyles/
Disallow:/customavatars/
Disallow:/customprofilepics/
Disallow:/images/
Disallow:/modcp/
Disallow:/ajax.php
Disallow:/attachment.php
Disallow:/calendar.php
Disallow:/cron.php
Disallow:/editpost.php
Disallow:/global.php
Disallow:/image.php
Disallow:/inlinemod.php
Disallow:/joinrequests.php
Disallow:/login.php
Disallow:/member.php
Disallow:/memberlist.php
Disallow:/misc.php
Disallow:/moderator.php
Disallow:/newattachment.php
Disallow:/newreply.php
Disallow:/newthread.php
Disallow:/online.php
Disallow:/poll.php
Disallow:/postings.php
Disallow:/printthread.php
Disallow:/private.php
Disallow:/profile.php
Disallow:/register.php
Disallow:/report.php
Disallow:/reputation.php
Disallow:/search.php
Disallow:/sendmessage.php
Disallow:/showgroups.php
Disallow:/subscription.php
Disallow:/threadrate.php
Disallow:/usercp.php
Disallow:/usernote.php

   
#Begin Attracta SEO Tools Sitemap. Do not remove
sitemap: http://cdn.attracta.com/sitemap/3406644.xml.gz
#End Attracta SEO Tools Sitemap. Do not remove

#Begin Attracta SEO Tools Sitemap. Do not remove
sitemap: http://cdn.attracta.com/sitemap/3406643.xml.gz
#End Attracta SEO Tools Sitemap. Do not remove
Shall I remove what Google Adsense says to move? How should my robot.txt file look?

I use vBSEO so not sure if I should do something via that in the AdminCP or if I should edit the robot.txt file manually.

Thanks all.

thetechgenius 10-25-2014 04:52 PM
This is what my Robots.txt looks like:

Code:

User-agent: thetechgenius-Google
Disallow:

User-agent: *
Disallow: /cache.php

User-agent: *
Disallow: /go/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /attachment.php
Disallow: /search.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /editpost.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /login.php
Disallow: /subscription.php
Disallow: /private.php
Disallow: /report.php
Disallow: /sendmessage.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /postings.php
Disallow: /sendtofriend.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /showgroups.php
Disallow: /announcement.php
Disallow: /downloads.php
Disallow: /downloads.html
Disallow: /**********/ #<- Renamed AdminCP Directory -- Blanked out for security reasons
Disallow: /*********/  #<- Renamed ModCP Directory --  Blanked out for security reasons

Sitemap: https://thetechgenius.net/dbseositemap.php  #<- This is my Sitemap from the DBSEO Addon --
As for the Sitemap, you have to put the URL that generates the Sitemap. If you go to my Sitemap URL, it will Generate and Start downloading the Sitemap. I'm not sure how vBSEO handles Sitemaps.


All times are GMT. The time now is 08:43 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01176 seconds
  • Memory Usage 1,723KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (3)bbcode_code_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (2)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete