![]() |
Bing, Facebook and Baidu bots do cause those issues, for me it was facebook's bots not Bings or Baidu's.
|
they sent me these instructions to do
You can also prevent robots from crawling parts of your site, while allowing them to crawl other sections. The following example would request search engines and robots not to crawl the cgi-bin folder, the tmp folder, and the junk folder and everything in those folders on your website. If you want to stop bots from visiting you site and stop search engines from ranking you, use this code: #Code to not allow any search engines! User-agent: * Disallow: / 2nd choice which i used and iam asking is this enough # Blocks robots from specific folders / directories User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /junk/ |
That only works if the bots obey the robots.txt and many dont!
|
thanks
but how they donot obey??? also they suspend again and said to me to enable caching how to enable them??? --------------- Added [DATE]1359576211[/DATE] at [TIME]1359576211[/TIME] --------------- does this topic help https://vborg.vbsupport.ru/showthread.php?t=139151 iam sorry iam annoying you but help me really iam depressed from recurrent suspension |
Try the article in my signature, as for bingbot I think you can regulate it's crawling via the bing webmaster tools.
|
my version is 3.8
|
Some of the tips also work for 3.8. The ones that don't clearly state that they don't.
|
1 Attachment(s)
Once you sign in to BWT & verify your site ownership, navigate to
Dashboard > Configure My Site > Crawl Control Less bars = Lower Crawl Speed |
1 Attachment(s)
for sorry
i cannot find crawl control attached what i can see |
Well if you like you can tell it to block urls & force it to stop visiting completely.
|
All times are GMT. The time now is 09:47 AM. |
Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information | |
---|---|
|
|
![]() |
|
Template Usage:
Phrase Groups Available:
|
Included Files:
Hooks Called:
|