Log in

View Full Version : No more spider visits. What is wrong? Please help.

fpattberg
06-08-2008, 10:51 AM
Hello.

I have a had a strange thing happening to me. I run a vbulletin forum and when I checked this morning I noticed that there where no visits by spiders (Yahoo, Google, etc) since the early hours of the 7th. And this seems to continue. I only have my visitors but no spiders.

This is my forum URL: http://www.sustainabilityforum.com/forum

I am running 3.6.7 PL1 and use the Whos is online mod and the country display for Whos online. I did not make any changes that could have caused this. I also run vbseo and doe xml sitemaps that which was successfully send to google this morning.

Does anyone have an idea what could have caused this or has anyone had the same issue?

Please help! :(

Thanks.

Fabian
Boofo
06-08-2008, 11:12 AM
Are you using the Spider Display mod?
fpattberg
06-08-2008, 12:41 PM
HI.

I am using this mod: https://vborg.vbsupport.ru/showthread.php?t=131314

And I have another mistery that I have a robots.txt being displayed but I never did a robots file but it still shows up: http://www.sustainabilityforum.com/robots.txt/

And it says to disallow siders. But I dont want that.

Waht can I do? And where can I find this robot file?

Thanks.
Fabian
Lynne
06-08-2008, 01:30 PM
HI.

I am using this mod: https://vborg.vbsupport.ru/showthread.php?t=131314

And I have another mistery that I have a robots.txt being displayed but I never did a robots file but it still shows up: http://www.sustainabilityforum.com/robots.txt/

And it says to disallow siders. But I dont want that.

Waht can I do? And where can I find this robot file?

Thanks.
Fabian
It looks like the robots.txt file is located in your root directory. Just login via ftp and remove it. You might want to try to find out who put that there though if you didn't.
fpattberg
06-08-2008, 01:53 PM
Hi Lynne.

Yes, I have looked for the folder robots.txt but there is none on the root directory or anywhere else. I have looked through every file.

In ftp it needs to be under "/" right as a .txt right?

Thanks.
Fabian
Lynne
06-08-2008, 02:04 PM
It is not a folder. It should be at the same level as your "forum" folder though. And it is called "robots.txt"