robots.txt Manager
 

This script allows you to easily create a dynamically generated robots.txt file, based on specified rules.

If you use this hack, please click 'Install' :)

Screenshots will be attached...

Control Panel

Editor (All Robots)

Editor (Specific Robot)

Generated File (:banana:)

Excellent G.

Wow very nice!

]very good job, tho' I can edit one txt file by hands :) No offence, again, very good job :)

May I ask you if you have the list with IPs of some major/all search engines? I kinda need it :) Thanx a lot and again, a nice hack!

The point of the hack is to make administration easier. It keeps track of robots requesting the robots.txt file, allowing you to ban or restrict a bot without having to dig through the server logs. I wrote the hack today, so the only bots included in the database already are the ones that spidered my site during that time. The list is in the .sql file.

]thanx for the answer :)

Is it suppose to write a new robots.txt file everytime or do the bots see the robots.php file as robots.txt?

If your answer is it's suppose to write a new robots.txt file, it isn't working for me :-(

And: do I still need a robots.txt file?

It uses mod_rewrite to send requests to robots.php. You have to create an .htaccess file with the following:

RewriteEngine on
RewriteRule robots.txt /robots.php (note: this is for the old version. read the new install file :))

Upload robots.php to the root web directory (usually public_html). Make sure you run robots.sql using phpMyAdmin. :)

I did that already ;)
So the robots are redirected to robots.php which is feeding them a perfectly rendered robots.txt file? Sorry 'bout asking, but I don't want to break my (high) ranking(s).

- djr

Yup. The only difference is the X-Powered-By header generated automatically by PHP.

Thanks! I just installed it. See if I can control those bot's a little more!!

very nice!

***installs

Hi MUG,

Can you add another column 'Owner' and 'Origin' (or whatever you might want to call it) where we can add the owner and origin of the spider?

For example:
Not every spider describes itself fully. e.g. Mercator-2.0 is one of Altavista's robots, but there's no link to Altavista whatsoever.

Thanks,
- djr

I found some good overviews of spiders here and here. If anyone has more of these lists, please add them to this thread.

Thanks,
- djr

Ooh, thanks. I was wondering what Mercator-2.0 was. :paranoid:

I'll add a description field, but there's not enough room for it to show on the main page so you'll have to click edit to view it.

Version 1.0 final released. :pirate:

Can this thread be moved to the Full Releases forum?

I have a slight problem with googlebots, and that is they storm my forum by huge numbers. Currently, for example, I have 7 googlebots crawling my forum. That seems purely excessive to me, and I would like to somehow limit the amount of googlebots to maybe 2.

What is the command line for robots.txt to do this? Or maybe there is some other alternate method.

Thanks ;)

Honestly, I don't think that is possible with robots.txt. If you created something that would dynamically insert text into a robots.txt file based on the number of Googlebots spidering your site, Google might "take the hint" and never come back. :ermm:

Drat.. :ermm:

Wish it were possible somehow, oh well. My current bandwidth is being consumed quicly by these googlebots, so I guess I'll simply have to restrict them from the threads.

restricting them from the threads :confused: whats the point of getting spidered then ?

We have two different domains, but only one MySQL-database. Is it possible to place the robots.php on both the domains (and thus using the same tables)?

- djr

Already found it. Just rename the robots_log table to robots_log_domain1 and create another one with _domain2 and update changes in robots.php.

- djr

Installed, works great!

Glad that you like it. :cool:

Any suggestions? :)

The only suggestion I can think of is being able to import your current robots.txt

I had disallowed "turnitin" and would like to be able to still block them.

I thought that I already included TurnitinBot in the .sql file?
Good idea... it shouldn't be too hard to implement. :)

And maybe being able to manually edit it as well.

My current robots.txt file is not being overwritten when I click submit?

I made the changes to the .htaccess file... is there anything else i should look @ ??

Thanx

1.1 Beta released. It includes the following bug fixes / additions:[list=1][*]Stripping of comments from generated file (although it is in the robots.txt specification, some bots choke on them)[*]Repairs newlines in generated file (old version sometimes produced \r\r\n)[*]Cleaner interface for control panel[*]Several other things I can't remember :confused:[/list=1]

thanks for this hack its very useful and i think it should be a full release :)

i have one problem and it has nothing to do with your script :)

where can i find info on what bots i should ban i never had turnit in bot banned b4 .. why is that bot bad ???

very cool, now I don't need to do this manually all the time ;) thank you for sharing :)

I get an error after i updated to 1.1

Fatal error: db_connect(): Failed opening required '' (include_path='') in /home/name/public_html/robots.php on line 63

Did you change $vB_Config_Path to the correct path?

Great addition.
Ar their any plans for a new version?

-Fish

nice little hack this :)