I have some LDM pages which have no restrictions to guests, and human guests are able to access them correctly. However I have noticed in the 'Who's Online' display that yahoo / google crawlers are being shown an error message, which is shown by an exclamation mark in the 'Who's Online' display (when placing cursor over this it says the googlebot is 'Viewing Error Message'). I think this means the google crawler is being denied access to those pages despite guests having permission to access them. I have no restrictions set in my 'robots.txt' file, ie. all crawlers/ bots are allowed.
Any idea why this could be occuring?
|