PDA

View Full Version : What Robot txt should i have? Google Webmaster blocking 11 000 URL


Weeds
05-29-2011, 06:09 AM
I wonder what google Robot.txt i should i have. At the moment i have a Robot.txt but in the root it is just blocking my wordpress that i have on blog.domain.com.

Google webmaster is blocking this kind of threads

/tags.php?tag=investment+in+thailand URL restricted by robots.txt

/threads/2207-Obama-Visit-To-Indonesia?p=2389 URL restricted by robots.txt


Should the VB block these?

Brandon Sheley
05-29-2011, 06:30 AM
so you're asking if you should blog tags or one thread?
you can look at mine (http://general-forums.com/robots.txt) as an example....

Weeds
05-29-2011, 07:57 AM
I do not even have a robot for my VB and i do not know why it blocking these URL there is 11 000 now. I see you not blocking tags and threads.