The Arcive of Official vBulletin Modifications Site.

robots.txt Manager · **Released**: 02-07-2003

This script allows you to easily create a dynamically generated robots.txt file, based on specified rules.

If you use this hack, please click 'Install'

Screenshots will be attached...

djr · #12 02-08-2003, 08:50 PM

Is it suppose to write a new robots.txt file everytime or do the bots see the robots.php file as robots.txt?

If your answer is it's suppose to write a new robots.txt file, it isn't working for me :-(

And: do I still need a robots.txt file?

MUG · #13 02-08-2003, 09:06 PM

Quote:

Originally posted by djr
Is it suppose to write a new robots.txt file everytime or do the bots see the robots.php file as robots.txt?

If your answer is it's suppose to write a new robots.txt file, it isn't working for me :-(

And: do I still need a robots.txt file?

It uses mod_rewrite to send requests to robots.php. You have to create an .htaccess file with the following:

RewriteEngine on
RewriteRule robots.txt /robots.php (note: this is for the old version. read the new install file

)

Upload robots.php to the root web directory (usually public_html). Make sure you run robots.sql using phpMyAdmin.

djr · #14 02-08-2003, 09:33 PM

I did that already

So the robots are redirected to robots.php which is feeding them a perfectly rendered robots.txt file? Sorry 'bout asking, but I don't want to break my (high) ranking(s).

- djr

MUG · #15 02-08-2003, 10:10 PM

Quote:

Originally posted by djr
I did that already
So the robots are redirected to robots.php which is feeding them a perfectly rendered robots.txt file? Sorry 'bout asking, but I don't want to break my (high) ranking(s).

- djr

Yup. The only difference is the X-Powered-By header generated automatically by PHP.

BigCheeze · #16 02-08-2003, 10:16 PM

Thanks! I just installed it. See if I can control those bot's a little more!!

#17 02-09-2003, 02:09 AM

very nice!

***installs

djr · #18 02-09-2003, 09:51 AM

Hi MUG,

Can you add another column 'Owner' and 'Origin' (or whatever you might want to call it) where we can add the owner and origin of the spider?

For example:

Code:

googlebot | Googlebot/2.1 (+http://www.googlebot.com/bot.html) | 216.239.46.19 | Google | http://www.google.com | 4 Edit - Delete |

Not every spider describes itself fully. e.g. Mercator-2.0 is one of Altavista's robots, but there's no link to Altavista whatsoever.

Thanks,
- djr

djr · #19 02-09-2003, 09:54 AM

I found some good overviews of spiders here and here. If anyone has more of these lists, please add them to this thread.

Thanks,
- djr

MUG · #20 02-09-2003, 10:24 AM

Quote:

Originally posted by djr
Hi MUG,

Can you add another column 'Owner' and 'Origin' (or whatever you might want to call it) where we can add the owner and origin of the spider?

For example:

Code:

googlebot | Googlebot/2.1 (+http://www.googlebot.com/bot.html) | 216.239.46.19 | Google | http://www.google.com | 4 Edit - Delete |

Not every spider describes itself fully. e.g. Mercator-2.0 is one of Altavista's robots, but there's no link to Altavista whatsoever.

Thanks,
- djr

Ooh, thanks. I was wondering what Mercator-2.0 was.

aranoid:

I'll add a description field, but there's not enough room for it to show on the main page so you'll have to click edit to view it.

MUG · #21 02-09-2003, 10:40 AM

Version 1.0 final released.

irate:

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.04911 seconds Memory Usage 2,296KB Queries Executed 25 (?)
More Information
Template Usage: (1)SHOWTHREAD (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)ad_showthread_beforeqr (2)bbcode_code (3)bbcode_quote (1)footer (1)forumjump (1)forumrules (1)gobutton (1)header (1)headinclude (1)modsystem_post (1)navbar (6)navbar_link (120)option (1)pagenav (1)pagenav_curpage (3)pagenav_pagelink (11)post_thanks_box (11)post_thanks_button (1)post_thanks_javascript (1)post_thanks_navbar_search (11)post_thanks_postbit_info (10)postbit (10)postbit_onlinestatus (11)postbit_wrapper (1)spacer_close (1)spacer_open (1)tagbit_wrapper Phrase Groups Available: global inlinemod postbit posting reputationlevel showthread	Included Files: ./showthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/functions_bigthree.php ./includes/class_postbit.php ./includes/class_bbcode.php ./includes/functions_reputation.php ./includes/functions_post_thanks.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete showthread_start showthread_getinfo forumjump showthread_post_start showthread_query_postids showthread_query bbcode_fetch_tags bbcode_create showthread_postbit_create postbit_factory postbit_display_start post_thanks_function_post_thanks_off_start post_thanks_function_post_thanks_off_end post_thanks_function_fetch_thanks_start post_thanks_function_fetch_thanks_end post_thanks_function_thanked_already_start post_thanks_function_thanked_already_end fetch_musername postbit_imicons bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete postbit_display_complete post_thanks_function_can_thank_this_post_start pagenav_page pagenav_complete tag_fetchbit_complete forumrules navbits navbits_complete showthread_complete
Messages:

The Arcive of Official vBulletin Modifications Site.

It is not a VB3 engine, just a parsed copy!