robots.txt

[binkuang]

hi...anyone here know what is the best code for the robots.txt? what should Disallow cause the best result for fast craw and index the page?

[Jase2]

I use the following:

User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/album.php
Disallow: /forum/announcement.php
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/faq.php
Disallow: /forum/showgroups.php
Disallow: /forum/images/
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/members/list/
Disallow: /forum/misc.php
Disallow: /forum/modcp/
Disallow: /forum/moderator.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showpost.php
Disallow: /forum/subscription.php
Disallow: /forum/subscriptions.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php

Modified it a little; as I use vBSEO hehe.

[Lynne]

You realize that anyone can read a robots.txt file, right? It might not be a good idea to put the directory names of your modcp and admincp in there (they should not be called that anyway).

[binkuang]

User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/album.php
Disallow: /forum/announcement.php
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/faq.php
Disallow: /forum/showgroups.php
Disallow: /forum/images/
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/members/list/
Disallow: /forum/misc.php
Disallow: /forum/modcp/
Disallow: /forum/moderator.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showpost.php
Disallow: /forum/subscription.php
Disallow: /forum/subscriptions.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php

if you use that robots.txt.

Disallow: /forum/showpost.php and Disallow: /forum/newthread.php

is that the spider can crawl the newthread and post or not if i put this robots. txt? my forum is new not so many index page. thanks

[Lynne]

Originally Posted by binkuang

if you use that robots.txt. is that the spider can crawl the newthread and post or not if i put this robots. txt? my forum is new not so many index page. thanks

They will still crawl the showthread page which has all the posts on it. The showpost page just shows a single post and the newthread page is used to make new pages - there is no content on it.

[Jase2]

^^ Correct! Therefore, the bots will spend more time crawling/indexing your threads/posts.

[binkuang]

your mean if i use the robots.txt below, the bots will crawling and indexing my thread/posts better. right?

User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/album.php
Disallow: /forum/announcement.php
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/faq.php
Disallow: /forum/showgroups.php
Disallow: /forum/images/
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/members/list/
Disallow: /forum/misc.php
Disallow: /forum/modcp/
Disallow: /forum/moderator.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showpost.php
Disallow: /forum/subscription.php
Disallow: /forum/subscriptions.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php