Sudden increase in server load at specific times

[abdobasha2004]

Within 6-11 pm GMT I get high load averages on my forum, my webhost says the load comes from showthread.php and MySQL.
I have tried disabling all mods but also load is very high.
My site is on a dedicated server.
This happens between 6-11 pm GMT when online are >13k.
Any other time the load is fine and the website is working fine.

url : nmisr.com
Any help?

[Simon Lloyd]

You need to check your statistics..etc to see who is on your site at those times, which ip's and how many connections per ip.

[CAG CheechDogg]

Lately there has been a rash of IPs hitting vbulletin sites for what ever reasons and most of them are Chinese IPs. I just helped another member here who was getting hit with tons of IPs and we blocked them through htaccess.

It is going to take a few hours of monitoring your site but you will get it under control. Here is the link to that member's thread :

https://vborg.vbsupport.ru/showthread.php?t=302628

If you need some help let me know, but it will require me to have access to your cPanel.

[abdobasha2004]

thanks for help CAG and Simon, but all hacking or attacks possibilities are excluded
all the 13k online are random arab visitors !

[CAG CheechDogg]

Ok good stuff..then just block all the ips from those areas and you should be good ...

[Gripi]

It is very normal that MySQL and showthread.php consume lots of resource.

13k online in the same time? my site have 1200-1500 user online in the same time (showing from realtime google analytics), i'm using dual e2620 with memcache + apc, and 64gb ram, raid 10 ssd. the CPU load average is around 7.00 to 10.00.

As long as the site load time still fast, i think nothing to be worry about it.

[abdobasha2004]

Lately there has been a rash of IPs hitting vbulletin sites for what ever reasons and most of them are Chinese IPs. I just helped another member here who was getting hit with tons of IPs and we blocked them through htaccess.

It is going to take a few hours of monitoring your site but you will get it under control. Here is the link to that member's thread :

https://vborg.vbsupport.ru/showthread.php?t=302628

If you need some help let me know, but it will require me to have access to your cPanel.

Unbelievable !
When I added the code in htaccess the load markedly decreases, but it is still high about 50 with 14k online.
so do you have any updated list for these ips to block?

[CAG CheechDogg]

Which one did you add the one from the thread I gave you?

Also use Simon's Ban Spiders by User Agent to block those bots we do not need on our forums and sites:

https://vborg.vbsupport.ru/showthread.php?t=268208&highlight=ban+spiders

Here is the list I use, I block facebook, bing, msn, alexa and some others, why? I don't need them and they always seem not to play nice when they crawl my forums....I recommend you replace the list that comes with this mod with this one below:



almaden
Anarchie
Artabus
ASPSeek
attach
autoemailspider
BackWeb
Baidu
Bandit
BatchFTP
BlackWidow
Bot\mailto:craftbot@yahoo.com
Buddy
bumblebee
CherryPicker
ChinaClaw
CICC
Collector
Copier
Copyscape
Crescent
DIIbot
DISCo
DISCo\Pump
dotbot
Download\Demon
Download\Wonder
Downloader
Drip
DSurf15a
eCatch
EasyDL/2.99
EirGrabber
email
EmailCollector
EmailSiphon
EmailWolf
Express\WebPictures
ExtractorPro
EyeNetIE
FileHound
FlashGet
FrontPage
GetRight
GetSmart
GetWeb!
gigabaz
GNIP
Go\!Zilla
Go!Zilla
Go-Ahead-Got-It
gotit
Grabber
GrabNet
Grafula
grub-client
HMView
HTTrack
httpdown
.*httrack.*
ia_archiver
Ichiro
Image\Stripper
Image\Sucker
Indy*Library
Indy\Library
InterGET
InternetLinkagent
Internet\Ninja
InternetSeer.com
Iria
JBH*agent
JetCar
JOC\Web\Spider
JustView
larbin
LeechFTP
LexiBot
lftp
Link*Sleuth
likse
//Link
LinkWalker
Mag-Net
Magnet
Magpie
magpie
Mass\Downloader
Memo
Microsoft.URL
MIDown\tool
Mirror
Mister\PiX
Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler
MS\FrontPage*
MSFrontPage
MSIECrawler
MSProxy
Navroad
NearSite
NetAnts
NetMechanic
NetSpider
Net\Vampire
NetZIP
NICErsPRO
Ninja
Nutch
Octopus
Offline\Explorer
Offline\Navigator
omgili
Openfind
PageGrabber
Papa\Foto
pavuk
pcBrowser
Ping
PingALink
Pockey
psbot
Pump
QRVA
RealDownload
Reaper
Recorder
ReGet
Scooter
Seeker
Siphon
sitecheck.internetseer.com
SiteSnagger
SlySearch
SmartDownload
Snake
sogou
Soso
SpaceBison
speedy
Spinn3r
sproose
Stripper
Sucker
SuperBot
SuperHTTP
Surfbot
Szukacz
tAkeOut
Teleport\Pro
URLSpiderPro
Vacuum
VoidEYE
Web\Image\Collector
Web\Sucker
WebAuto
[Ww]eb[Bb]andit
webcollage
WebCopier
Web\Downloader
WebEMailExtrac.*
WebFetch
WebGo\IS
WebHook
WebLeacher
WebMiner
WebMirror
WebReaper
WebSauger
Website
Website\eXtractor
Website\Quester
Webster
WebStripper
WebWhacker
WebZIP
Wget
Whacker
Widow
WWWOFFLE
x-Tractor
Xaldon\WebSpider
Xenu
Yandex
Yeti
YOUDAOBOT
Zeus.*Webster
Zeus
baiduspider
beta.statsit.com
statsit
SiteIntel
Yandex
GomezAgent
FunWebProducts
Nesotebot
DCPbot
AOL Advertising R&D
DataCha0s
aiHitBot
Apache-HttpClient
Zend_Http_Client
ReverseGet
XXX bot Content
vBSEO
spbot
OffByOne
thyroidbuzz
AcoonBot
coccoc
xpymep
proxyproxy2884
AppEngine
start.exe
Semiocast HTTP client
Firefox/3.6.23
TurnitinBot
curl
SwpLc/1.6
GrepNetstat.com
news bot
AskTbPTV
checks
panopta
App3le
PhantomJS
AlwaysOnline
SISTRIX
proximic
CRAWL-E/0.6.4
WebMoney
Maxthon
HTMLParser
oBot
UnisterBot
ERACrawler
Butterfly
Topsy
Butterfly Topsy Crawler
Ezooms
Deepnet
Alexa
Bitlybot
Seznam
Fulltext
Facebook
Sunrise Communications AG
crawl
Crawl
MJ12bot
Bimbot
Snapbot
thunderstone
Thunderstone
grub-client
Bing
MSN

--------------- Added 1380574201 at 1380574201 ---------------

I also use this in my robots.txt file to block them before they even hit my forums:

User-agent: Baiduspider
Disallow: /

User-agent: Baidu
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Baiduspider-video
Disallow: /

User-agent: Baiduspider-image
Disallow: /

User-agent: Baiduspider-favo
Disallow: /

User-agent: Baiduspider-news
Disallow: /

User-agent: Baiduspider-mobile
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: aipbot
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: Alexibot
Disallow: /

User-agent: Aqua_Products
Disallow: /

User-agent: asterias
Disallow: /

User-agent: b2w/0.1
Disallow: /

User-agent: BackDoorBot/1.0
Disallow: /

User-agent: becomebot
Disallow: /

User-agent: BlowFish/1.0
Disallow: /

User-agent: Bookmark search tool
Disallow: /

User-agent: BotALot
Disallow: /

User-agent: BotRightHere
Disallow: /

User-agent: BuiltBotTough
Disallow: /

User-agent: Bullseye/1.0
Disallow: /

User-agent: BunnySlippers
Disallow: /

User-agent: CheeseBot
Disallow: /

User-agent: CherryPicker
Disallow: /

User-agent: CherryPickerElite/1.0
Disallow: /

User-agent: CherryPickerSE/1.0
Disallow: /

User-agent: Copernic
Disallow: /

User-agent: CopyRightCheck
Disallow: /

User-agent: cosmos
Disallow: /

User-agent: Crescent
Disallow: /

User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /

User-agent: DittoSpyder
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

User-agent: EmailWolf
Disallow: /

User-agent: EroCrawler
Disallow: /

User-agent: ExtractorPro
Disallow: /

User-agent: FairAd Client
Disallow: /

User-agent: Fasterfox
Disallow: /

User-agent: Flaming AttackBot
Disallow: /

User-agent: Foobot
Disallow: /

User-agent: Gaisbot
Disallow: /

User-agent: GetRight/4.2
Disallow: /

User-agent: Harvest/1.5
Disallow: /

User-agent: hloader
Disallow: /

User-agent: httplib
Disallow: /

User-agent: HTTrack 3.0
Disallow: /

User-agent: humanlinks
Disallow: /

User-agent: IconSurf
Disallow: /
Disallow: /favicon.ico

User-agent: InfoNaviRobot
Disallow: /

User-agent: Iron33/1.0.2
Disallow: /

User-agent: JennyBot
Disallow: /

User-agent: Kenjin Spider
Disallow: /

User-agent: Keyword Density/0.9
Disallow: /

User-agent: larbin
Disallow: /

User-agent: LexiBot
Disallow: /

User-agent: libWeb/clsHTTP
Disallow: /

User-agent: LinkextractorPro
Disallow: /

User-agent: LinkScan/8.1a Unix
Disallow: /

User-agent: LinkWalker
Disallow: /

User-agent: LNSpiderguy
Disallow: /

User-agent: lwp-trivial
Disallow: /

User-agent: lwp-trivial/1.34
Disallow: /

User-agent: Mata Hari
Disallow: /

User-agent: Microsoft URL Control
Disallow: /

User-agent: Microsoft URL Control - 5.01.4511
Disallow: /

User-agent: Microsoft URL Control - 6.00.8169
Disallow: /

User-agent: MIIxpc
Disallow: /

User-agent: MIIxpc/4.2
Disallow: /

User-agent: Mister PiX
Disallow: /

User-agent: moget
Disallow: /

User-agent: moget/2.1
Disallow: /

User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: NetAnts
Disallow: /

User-agent: NICErsPRO
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Openbot
Disallow: /

User-agent: Openfind
Disallow: /

User-agent: Openfind data gatherer
Disallow: /

User-agent: Oracle Ultra Search
Disallow: /

User-agent: PerMan
Disallow: /

User-agent: ProPowerBot/2.14
Disallow: /

User-agent: ProWebWalker
Disallow: /

User-agent: psbot
Disallow: /

User-agent: Python-urllib
Disallow: /

User-agent: QueryN Metasearch
Disallow: /

User-agent: Radiation Retriever 1.1
Disallow: /

User-agent: RepoMonkey
Disallow: /

User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /

User-agent: RMA
Disallow: /

User-agent: searchpreview
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: SpankBot
Disallow: /

User-agent: spanner
Disallow: /

User-agent: SurveyBot
Disallow: /

User-agent: suzuran
Disallow: /

User-agent: Szukacz/1.4
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: Telesoft
Disallow: /

User-agent: The Intraformant
Disallow: /

User-agent: TheNomad
Disallow: /

User-agent: TightTwatBot
Disallow: /

User-agent: toCrawl/UrlDispatcher
Disallow: /

User-agent: True_Robot
Disallow: /

User-agent: True_Robot/1.0
Disallow: /

User-agent: turingos
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: TurnitinBot/1.5
Disallow: /

User-agent: URL Control
Disallow: /

User-agent: URL_Spider_Pro
Disallow: /

User-agent: URLy Warning
Disallow: /

User-agent: VCI
Disallow: /

User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /

User-agent: Web Image Collector
Disallow: /

User-agent: WebAuto
Disallow: /

User-agent: WebBandit
Disallow: /

User-agent: WebBandit/3.50
Disallow: /

User-agent: WebCapture 2.0
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: WebCopier v.2.2
Disallow: /

User-agent: WebCopier v3.2a
Disallow: /

User-agent: WebEnhancer
Disallow: /

User-agent: WebSauger
Disallow: /

User-agent: Website Quester
Disallow: /

User-agent: Webster Pro
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebZip
Disallow: /

User-agent: WebZip
Disallow: /

User-agent: WebZip/4.0
Disallow: /

User-agent: WebZIP/4.21
Disallow: /

User-agent: WebZIP/5.0
Disallow: /

User-agent: Wget
Disallow: /

User-agent: wget
Disallow: /

User-agent: Wget/1.5.3
Disallow: /

User-agent: Wget/1.6
Disallow: /

User-agent: WWW-Collector-E
Disallow: /

User-agent: Xenu's
Disallow: /

User-agent: Xenu's Link Sleuth 1.1c
Disallow: /

User-agent: Zeus
Disallow: /

User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /

User-agent: Zeus Link Scout
Disallow: /

User-agent: Bimbot/1.0
Disallow: /

User-agent: Snapbot/1.0
Disallow: /

User-agent: snap.com beta crawler v0
Disallow: /

User-agent: magpie-crawler
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: thunderstone
Disallow: /

User-agent: grub-client
Disallow: /

[abdobasha2004]

unfortunately problem is not yet solved :( :(

[ozzy47]

How many users are online at that time, following what was posted above?

[Simon Lloyd]

4,571 right now!

[ozzy47]

Well that should not be to bad then?

[abdobasha2004]

How many users are online at that time, following what was posted above?
now about 4k but the problem happens when online are 13k or more

Well that should not be to bad then?
what should not be bad?!

[ozzy47]

4K should not be bad, so adding all the above suggestions, and you still get 13k or more?

[abdobasha2004]

yes, 95% are normal users not bots and at that time load increase exponentially !

--------------- Added 1381283598 at 1381283598 ---------------

I have a big database with big posts table, should this table be partitioned?

[ozzy47]

What are the specs on your server? Does this load slow down the site, or stop it from running?

[abdobasha2004]

2x4 core
32g RAM
500 Giga hard disk space
The load first makes site slow down then stops running completely, then server auto restart and the cycle repeated

[snakes1100]

As that dive is 500GB & most likely SATA, its disk i/o is going to kill SQL.

What CPU is it running?

cat /proc/cpuinfo

RAM is fine, but your disk & cpu is lacking for the amount of users & size of DB.

[abdobasha2004]

As that dive is 500GB & most likely SATA, its disk i/o is going to kill SQL.

What CPU is it running?

cat /proc/cpuinfo

RAM is fine, but your disk & cpu is lacking for the amount of users & size of DB.

my host says : The server is running e3 1240v2
we could upgrade you to a SAS or SSD or even a SAS raid or SSD raid server.

what do you recommend?

[snakes1100]

Can you take a snapshot of "top" running when its under heavy load?

Hit 1 after top starts so we can see all the core's.

[abdobasha2004]

Can you take a snapshot of "top" running when its under heavy load?

Hit 1 after top starts so we can see all the core's.

now load is 40

the cause of this load is mysql

[mysqld]
query_cache_size=256M
table_cache=80000
max_connections=5000
open_files_limit=5400
log-slow-queries="/var/lib/mysql/slow.log"
innodb_file_per_table=1
thread_cache_size=4
interactive_timeout=60
wait_timeout=60
default-storage-engine=MyISAM
max_heap_table_size = 32M
local-infile=0
max_allowed_packet = 16MB


for my.cnf

and this is what mysql tuner says.

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.32-cll
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 1G (Tables: 333)
[--] Data in InnoDB tables: 224K (Tables: 14)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[--] Data in MEMORY tables: 16M (Tables: 111)
[!!] Total fragmented tables: 19

-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 20h 0m 46s (32M q [447.256 qps], 2M conn, TX: 1387B, RX: 9B)
[--] Reads / Writes: 63% / 37%
[--] Total buffers: 424.0M global + 2.8M per thread (5000 max threads)
[OK] Maximum possible memory usage: 13.8G (59% of installed RAM)
[OK] Slow queries: 0% (979/32M)
[OK] Highest usage of available connections: 33% (1686/5000)
[OK] Key buffer size / total MyISAM indexes: 8.0M/1.1G
[OK] Key buffer hit rate: 99.0% (1B cached / 9M reads)
[OK] Query cache efficiency: 79.3% (16M cached / 21M selects)
[!!] Query cache prunes per day: 11559
[OK] Sorts requiring temporary tables: 0% (279 temp sorts / 1M sorts)
[OK] Temporary tables created on disk: 1% (3K on disk / 217K total)
[OK] Thread cache hit rate: 93% (182K created / 2M connections)
[OK] Table cache hit rate: 92% (15K open / 16K opened)
[OK] Open file limit used: 9% (15K/165K)
[OK] Table locks acquired immediately: 99% (16M immediate / 16M locks)
[OK] InnoDB data size / buffer pool: 224.0K/128.0M

-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours - recommendations may be inaccurate
Increasing the query_cache size over 128M may reduce performance
Variables to adjust:
query_cache_size (> 256M) [see warning above]

https://vborg.vbsupport.ru/external/2013/10/58.png

[Simon Lloyd]

I think you may have a corrupt table as that server right now is hardly under load.

[snakes1100]

I think you may have a corrupt table as that server right now is hardly under load.

What picture are you looking at?

-----------------------------------------

As your server's CPU is using over 60% on each core & your going defunct on almost every php pid, i would start by removing suPHP & running apache/php as a DSO.

You would also need to optimize that server.

Your my.cnf settings are badly done, if your host is the one doing it, i would put a stop to that, as they dont have a clue on how to optimize a server.

This variable (thread_cache_size=4) isnt even used under the [mysqld] section & its set wrong anyways.

[Simon Lloyd]

What picture are you looking at?.His cpu usage and memory usage are fairly low, i suspect the load shown at the top is the entire load of the shared server.

There again i'm not wearing my glasses :)

--------------- Added 1381342875 at 1381342875 ---------------

I'm also interested to know why you have mixed tables, you have a few InnoDB tables was that by choice, by install, your hosts...etc?

[abdobasha2004]

What picture are you looking at?

-----------------------------------------

As your server's CPU is using over 60% on each core & your going defunct on almost every php pid, i would start by removing suPHP & running apache/php as a DSO.

You would also need to optimize that server.

Your my.cnf settings are badly done, if your host is the one doing it, i would put a stop to that, as they dont have a clue on how to optimize a server.

This variable (thread_cache_size=4) isnt even used under the [mysqld] section & its set wrong anyways.
Do you have a recommended configuration plz
?

[snakes1100]

Is this a shared server?

Simon, in his first post, he states dedicated.

As far as recommendations, i would start by removing suPHP & running apache/php as a dso, you really need to find out whats going on.

As far as SQL goes, its not as simple as me telling you what to use here, the stats you posted arent enough.

As to mixed tables, there is nothing wrong with that.

[abdobasha2004]

It is dedicated
are these the recommended settings to start with?

[snakes1100]

Start by removing suPHP & running apache/php as a DSO.

Then you can start checking your server logs for more details as to whats going on.

There is really no way to give you recommendations, as you havent provided much to work with in terms of settings on the server, apache, php, mysql & the OS settings for things like open file limits or if your running any caching programs etc.

Tuning a server entails much more than just simple data posted in the forums, as there are many things to look at.

Besides the fact that your server is using all its RAM & states only about 200MB free

table_cache=80000 <-- Thats way to high
max_connections=5000 <-- Way to high
open_files_limit=5400 <-- Doesnt matter unless the OS allows that many
thread_cache_size=4 <-- Wrong section & set wrong

Edit to add:
If your host is setting your server details, i would fire them on the spot for poor workmanship.......

--------------- Added 1381349605 at 1381349605 ---------------

Looks like you down now for server load.

Run this cmd in ssh when this happens & post the results:
netstat -plan|grep :80|awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -nk 1

[abdobasha2004]

For all thanks for your help
It was discovered that the problem was a ddos attack + flooding
We used cloudflare business and problem solved

Thanks God

:)