vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   Community Lounge (https://vborg.vbsupport.ru/forumdisplay.php?f=13)
-   -   Disallow robots - .htaccess & robots.txt (https://vborg.vbsupport.ru/showthread.php?t=133735)

Zia 12-11-2006 04:35 PM
Disallow robots - .htaccess & robots.txt
 
helo...
as we know we can control bots/spider using robots.txt

we know to disallow bot we use
we use:
Quote:
User-agent: BotName
Disallow: /
but to allow any particular bot for all places can we use bellow???

Quote:
User-Agent: Googlebot
Allow: /
source:http://www.mattcutts.com/blog/new-robotstxt-tool

and using .htaccess we are currently using to disaalow badbot



Quote:
RewriteCond %{HTTP_USER_AGENT} ^mister\ pix [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR]
RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
RewriteRule !^http://[^/.]\.golpo.net.* - [F,L]

<Files php.ini>
Order allow,deny
Deny from all
</Files>
here what the mean of NC & FL ?

-- in other palce i read

Quote:
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot

<Limit GET POST>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>
---
we are really confused...any one can give us some tips...what is what ?
and is perfect?
which .htaccess is good to disallow badbots?

Zia 12-13-2006 01:12 PM
helo..
any one can tell me...

about it..

i also want to BAN some bots too...

optrex 12-13-2006 01:21 PM
There is more than one way to do this. I use

Code:
  SetEnvIfNoCase User-Agent "libwww-perl/" bad_bot
  Order Allow,Deny
  Allow from all
  Deny from env=bad_bot

Zia 12-13-2006 02:08 PM
GOOD TO KNOW..

Also can u tell...how many bot i can ban/disallow using .htaccess ?
any limitations ? or as much as i want..

Crow 12-13-2006 02:54 PM
Can't you just use..

Code:
User-agent: *
Disallow: /
And that work for all bots? and all pages?

optrex 12-13-2006 02:57 PM
Yes but then you would end up blocking yahoo and google etc, NOT a very good idea!

btw, zia, afaik there are no restrictions in htaccess

Crow 12-13-2006 04:46 PM
Quote:
Originally Posted by optrex (Post 1137816)
Yes but then you would end up blocking yahoo and google etc, NOT a very good idea!

btw, zia, afaik there are no restrictions in htaccess
Sorry I mis-read the post, for some odd reason I thought he wanted to ban all bots.. Must be lack of sleep.:tired:

SaN-DeeP 12-15-2006 12:40 AM
As much as you want but banning from .htaccess = increase on server loads, try adding this values to your httpd.conf on apache webserver.

Interesting discussion here.

Zia 12-15-2006 03:44 PM
1 Attachment(s)
Quote:
Originally Posted by SaN-DeeP (Post 1138908)
As much as you want but banning from .htaccess = increase on server loads, try adding this values to your httpd.conf on apache webserver.

Interesting discussion here.
Im infact curious about this Server laod?

here im attaching my portion..yet not i implemeted it..

planning to do...

All the way i want to disallow bad-bots..

any one around here to have a look on it..?

Any comments about the bots ?

httpd.conf --hmm i gez i have to contact with service provider tho on a vps


All times are GMT. The time now is 02:30 AM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01195 seconds
  • Memory Usage 1,732KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (2)bbcode_code_printable
  • (6)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (9)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete