vb_taggregate_temp_... table grow!

[Simon Lloyd]

a lot of bots dont recognise robots.txt but follow some of the advice here http://antezeta.com/news/avoid-search-engine-indexing. For .htaccess use this

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider|yandex|anywordyoulike|like|bing) [NC]
RewriteRule .* - [R=403,L]

just keep adding names to the string or use my mod for banning bots.

Your .htaccess file must reside in the forum root for this (you might have to set your control panel to view hidden files if you dont see it)

[Ghostt]

1. i dont want block all useragent especially not google....

i need to block them from only 1101 forumdisplay what ive done with this robot.txt line (hope this is correct):
Disallow: /*forumdisplay.php?f=1101&order=desc&page=*

but this is the emergency solution because its not a real fix of the bad performance of very big forumdisplay pages.
and how i see no one can realy help here .

and i allready use your ban spider addon. thanks for it, but we have sometimes problem with it.
if server crashes the cloudflare.com 502 error page is shown, after user refresh page. user get blocked/redirected from this addon.
you know why?

[Simon Lloyd]

It will be to do with cloudflare caching i would imagine as my mod doesn't store any user/visitor details. As for your search engines try this, put this in your header template (or even forumdisplay but i think it must always use the header template)

<if condition="in_array($forumid, array(X,Y,Z)) AND "$show['search_engine']">
<meta HTTP-EQUIV="REFRESH" content="0; url=http://www.mysite.com">
</if>

Change x,y,z to be whichever forumid's you want to protect, change mysite.com to any url you want to redirect the spiders to and you should be golden

[Ghostt]

just tested htaccess code on the wordpress site on same server. connections/sec dropped from 200 to 10
with blocking this bastards: Baiduspider|yandex|anywordyoulike|like

so my robot.txt code not allways work you say?
seems to. but i will test your code tomorrow and check querys. thanks,

[Simon Lloyd]

I put these in anywordyoulike|like to show you can include anything in the string, they're not actual bots Robots.txt is only obeyed by good legitimate (but maybe unwanted) bots, for the likes of Baidu...etc you can actually get in contact with them and ask them to stop indexing your site.

[Ghostt]

so baidu ignore robots.txt?

[Simon Lloyd]

It seems that way as do many others like AhrefsBot, sosospider, Aboundex and even Bing to name but a few!

--------------- Added [DATE]1403918146[/DATE] at [TIME]1403918146[/TIME] ---------------

For a more complete .htaccess block look here http://wpsecure.net/bad-bot-list/

[Ghostt]

i test it . SetEnvIfNoCase or rewrite Rules better?
both codes from the link dont work i think coz i see in cloudflare baidu still crawling.... first rewrite gives an error.
can i use this bots listed there for your write here?
or any better list. in your plugin ive a very big list but i think it will not work with that list because there are not complete spider names

RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider|yandex|bing) [NC]
RewriteRule .* - [R=403,L]