google bots
 

I have my board set so members must register to view it. Therefore, all new content cannot be crawled by the google bots. The google website says:

"Google needs to be able to crawl your content without filling out a registration form. The easiest way to do this is to configure your webservers not to serve the registration page to our crawlers (when the User-Agent is "Googlebot") so that Googlebot can crawl these pages successfully. You can choose to allow Googlebot access to some restricted pages but not others."

Can anyone tell me how to do this? I'm on a shared server and I cannot find where I might accomplish this in the admin CP. Can I change my robots.txt file to do this?