Go Back   vb.org Archive > vBulletin Modifications > Archive > vB.org Archives > vBulletin 2.x > vBulletin 2.x Beta Releases
Reload this Page robots.txt Manager
FAQ Community Calendar Today's Posts Search

Reply
Page 3 of 5 12 3 45
 
Thread Tools
robots.txt Manager Details »»
robots.txt Manager
MOD Information
The Developer
 
Version: 1.00, by MUG MUG is offline
Developer Last Online: Nov 2023 Show Printable Version Email this Page
Version: 2.2.x Rating:
Released: 02-07-2003 Last Update: Never Installs: 15
Is in Beta Stage  
No support by the author.

This script allows you to easily create a dynamically generated robots.txt file, based on specified rules.

If you use this hack, please click 'Install'

Screenshots will be attached...

Show Your Support

  • This modification may not be copied, reproduced or published elsewhere without author's permission.

Comments
  #22  
Old 02-09-2003, 11:26 AM
MUG MUG is offline
 
Join Date: Apr 2002
Posts: 104
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Can this thread be moved to the Full Releases forum?
Reply With Quote
  #23  
Old 02-09-2003, 04:31 PM
Velocd's Avatar
Velocd Velocd is offline
 
Join Date: Mar 2002
Location: CA University
Posts: 1,696
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
I have a slight problem with googlebots, and that is they storm my forum by huge numbers. Currently, for example, I have 7 googlebots crawling my forum. That seems purely excessive to me, and I would like to somehow limit the amount of googlebots to maybe 2.

What is the command line for robots.txt to do this? Or maybe there is some other alternate method.

Thanks
Reply With Quote
  #24  
Old 02-09-2003, 04:42 PM
MUG MUG is offline
 
Join Date: Apr 2002
Posts: 104
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Quote:
Originally posted by Velocd
I have a slight problem with googlebots, and that is they storm my forum by huge numbers. Currently, for example, I have 7 googlebots crawling my forum. That seems purely excessive to me, and I would like to somehow limit the amount of googlebots to maybe 2.

What is the command line for robots.txt to do this? Or maybe there is some other alternate method.

Thanks
Honestly, I don't think that is possible with robots.txt. If you created something that would dynamically insert text into a robots.txt file based on the number of Googlebots spidering your site, Google might "take the hint" and never come back. :ermm:
Reply With Quote
  #25  
Old 02-10-2003, 01:43 AM
Velocd's Avatar
Velocd Velocd is offline
 
Join Date: Mar 2002
Location: CA University
Posts: 1,696
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Drat.. :ermm:

Wish it were possible somehow, oh well. My current bandwidth is being consumed quicly by these googlebots, so I guess I'll simply have to restrict them from the threads.
Reply With Quote
  #26  
Old 02-10-2003, 11:40 AM
Automated Automated is offline
 
Join Date: Sep 2002
Posts: 266
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Quote:
Originally posted by Velocd
Drat.. :ermm:

Wish it were possible somehow, oh well. My current bandwidth is being consumed quicly by these googlebots, so I guess I'll simply have to restrict them from the threads.
restricting them from the threads whats the point of getting spidered then ?
Reply With Quote
  #27  
Old 02-11-2003, 08:04 PM
djr's Avatar
djr djr is offline
 
Join Date: Nov 2001
Location: Amsterdam
Posts: 220
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
We have two different domains, but only one MySQL-database. Is it possible to place the robots.php on both the domains (and thus using the same tables)?

- djr
Reply With Quote
  #28  
Old 02-13-2003, 08:46 AM
djr's Avatar
djr djr is offline
 
Join Date: Nov 2001
Location: Amsterdam
Posts: 220
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Already found it. Just rename the robots_log table to robots_log_domain1 and create another one with _domain2 and update changes in robots.php.

- djr
Reply With Quote
  #29  
Old 02-16-2003, 02:48 PM
mheinemann's Avatar
mheinemann mheinemann is offline
 
Join Date: May 2002
Posts: 57
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Installed, works great!
Reply With Quote
  #30  
Old 02-16-2003, 10:54 PM
MUG MUG is offline
 
Join Date: Apr 2002
Posts: 104
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Glad that you like it.

Any suggestions?
Reply With Quote
  #31  
Old 02-17-2003, 12:19 PM
mheinemann's Avatar
mheinemann mheinemann is offline
 
Join Date: May 2002
Posts: 57
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
The only suggestion I can think of is being able to import your current robots.txt

I had disallowed "turnitin" and would like to be able to still block them.
Reply With Quote
Reply
Page 3 of 5 12 3 45

« Previous Thread | Next Thread »

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 11:53 AM.

Contact Us - Archive - Top

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.05884 seconds
  • Memory Usage 2,297KB
  • Queries Executed 25 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (2)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)modsystem_post
  • (1)navbar
  • (6)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (11)post_thanks_box
  • (11)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (11)post_thanks_postbit_info
  • (10)postbit
  • (11)postbit_onlinestatus
  • (11)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 
Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete