Disallow robots - .htaccess & robots.txt

[Zia]

helo...
as we know we can control bots/spider using robots.txt

we know to disallow bot we use
we use:

User-agent: BotName
Disallow: /

but to allow any particular bot for all places can we use bellow???

User-Agent: Googlebot
Allow: /

source:http://www.mattcutts.com/blog/new-robotstxt-tool

and using .htaccess we are currently using to disaalow badbot

RewriteCond %{HTTP_USER_AGENT} ^mister\ pix [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR]
RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
RewriteRule !^http://[^/.]\.golpo.net.* - [F,L]

<Files php.ini>
Order allow,deny
Deny from all
</Files>

here what the mean of NC & FL ?

-- in other palce i read

SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot

<Limit GET POST>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

---
we are really confused...any one can give us some tips...what is what ?
and is perfect?
which .htaccess is good to disallow badbots?

[Zia]

helo..
any one can tell me...

about it..

i also want to BAN some bots too...

[optrex]

There is more than one way to do this. I use

   SetEnvIfNoCase User-Agent "libwww-perl/" bad_bot
   Order Allow,Deny
   Allow from all
   Deny from env=bad_bot

[Zia]

GOOD TO KNOW..

Also can u tell...how many bot i can ban/disallow using .htaccess ?
any limitations ? or as much as i want..

[Crow]

Can't you just use..

User-agent: *
Disallow: /

And that work for all bots? and all pages?

[optrex]

Yes but then you would end up blocking yahoo and google etc, NOT a very good idea!

btw, zia, afaik there are no restrictions in htaccess

[Crow]

Originally Posted by optrex

Yes but then you would end up blocking yahoo and google etc, NOT a very good idea!

btw, zia, afaik there are no restrictions in htaccess

Sorry I mis-read the post, for some odd reason I thought he wanted to ban all bots.. Must be lack of sleep.:tired:

[SaN-DeeP]

As much as you want but banning from .htaccess = increase on server loads, try adding this values to your httpd.conf on apache webserver.

Interesting discussion here.

[Zia]

Originally Posted by SaN-DeeP

As much as you want but banning from .htaccess = increase on server loads, try adding this values to your httpd.conf on apache webserver.

Interesting discussion here.

Im infact curious about this Server laod?

here im attaching my portion..yet not i implemeted it..

planning to do...

All the way i want to disallow bad-bots..

any one around here to have a look on it..?

Any comments about the bots ?

httpd.conf --hmm i gez i have to contact with service provider tho on a vps