perfect robots.txt for vb 3.7?

[logicuk]

Does anyone care to share the perfect robots.txt for vb 3.7

here is what i have

User-agent: *
Disallow: /admincp/
Disallow: /archive/
Disallow: /backup/
Disallow: /announcement.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /faq.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /misc.php
Disallow: /modcp/
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /subscriptions.php
Disallow: /threadrate.php
Disallow: /usercp.php

[Boofo]

You should have that in the domain root.

[Bradley_Wint]

don't mind me sounding like a newbie here but i added the content to a robots.txt file, upped it to my server html root folder like 8 hrs ago..now when i check my who's online list, i still see google bots murdering my calender.php, login.php...etc...

anything i did wrong?

[blike2]

Google only checks robots.txt every day or two, I think. Give it a little time.

[Bradley_Wint]

Originally Posted by blike2

Google only checks robots.txt every day or two, I think. Give it a little time.

thanks mate

[SEOvB]

once every 24hrs

[Bradley_Wint]

yes apparently it worked like 5 minutes after i posted... only the important files like forumdisplay, showpost, showthread etc we being scanned. thanks for this mate. cheers.

[jambo_1969]

Make sure you add in

User-agent: Fasterfox
Disallow: /

I managed to catch a thread on my board where people were discussing have Fasterfox set as turbo charged...my loads went through the roof !!

[Boofo]

Can we combine these 2?

User-Agent: Googlebot-Image
Disallow: /

User-agent: Fasterfox
Disallow: /

Like this?

User-Agent: Googlebot-Image
User-agent: Fasterfox
Disallow: /

[jambo_1969]

Originally Posted by Boofo

Can we combine these 2?



Like this?

Not sure...

This is how mine looks (part of it anyway)

User-agent: Fasterfox
Disallow: /
User-agent: * 
Disallow: /.k.k.k./

?