[DaveAK]

I have the following robots.txt file in both my web root and forums directory:

User-agent: * # Applies to all robots
Disallow: / # This should stop all robots from everything.
Disallow: /cgi-bin # Index all pages except those in cgi-bin
Disallow: /logs # Index all pages except those in logs
Disallow: /stats # Index all pages except those in stats

(I know the last lines should be redundant, I just left them in from the file I was using to set it up.)

I just moved my forum to a new server, and now I'm being hit by spiders. I also upgraded from 3.6.3 to 3.8.4. THe move did not go smoothly and I originally forgot to put a copy of robots.txt in the web root. I had a Yahoo spider show up so I fixed that by putting the file in the web root, and I thought I was good to go, but now I'm getting a Google Spider instead, even with the robots.txt in both places.

What gives? Have I messed something up in moving to the new server? Is there something different in 3.8.4 that I'm not aware of? I haven't made any changes to vBulletin settings since the upgrade, neither have I addressed any template issues yet. I was up til 6:00AM doing the migration and now I'm back to clean up the mess.

Anyone have any suggestions? I've entered a similar question on the Google help forum as well.

Thanks!