HELP: Huge number of simultaneous connections to the db server
 

My forum is running very slowly and even after disabling all plugins. The issue seems to be stemming from the huge number of connections that are being made to the database server. Currently there are over 5000 simultaneous connections and while there are only about 65 PHP processes, these are taking up a large amount of resources - possibly because of poor queries being made to the MySQL server.

I did check the queries that are running on MySQL and they are not running slowly - the server is responding well, but the forum server is just opening so many connections for whatever reason.

Can I get some assistance on this? My forum version is 3.8.7. Tks.

--------------- Added [DATE]1380994648[/DATE] at [TIME]1380994648[/TIME] ---------------

i've checked google analytics and it reports only 80 concurrent users. Though on my forum homepage, it reports 20000 current active users, most of which are guests.

A link might help

What do you see in Who's Online (with it set to see guests)? Recently I saw on my little forum about quarter of the users were a(?) guest(s) all viewing the same thread with the same browser. All I remember of the user agent is it was Mozilla 4 and MSIE.

This is what I see right now:

Currently Active Users: 21729 (107 members and 21622 guests)
View Who's Online
Most users ever online is 24,817, 30-09-2012 at 04:30 PM.

This looks suspiciously like a DDoS attack. It might also be some Baidu bombing, do you block bad spiders?

Install this will help with all them bad bots

My server admin checked and it doesn't seem like its a ddos. All the connections are started internally within the site, so it's vb that is opening all the connections.

What is Baidu bombing?

What you are dealing with is a ton of bots, spiders, search engines crawling your forums all at the same time.

What do you have your session timeout set at? If you have it really high it would be a good indication of why you have so many guests in your whos online. using Simon Lloyd's Ban Spiders by User Agent will help you a lot to help combat this.

You will also have to use some IP blocks in your htaccess to stop Chinese bots from hitting your site, which is what Baidu is , they just don't care about rules and will just hit your site or forums like they are going out of style.

You should have this taken cared of in a couple of hours, if you need help let me know, take a look at this thread to help you more if you feel you can do what we did for Dave's website.

https://vborg.vbsupport.ru/showthrea...53#post2448953

tks everyone. am thinking of installing the ban spiders plugin as recommended. however, i've no idea what to ban or who. how shld i approach this problem?

and what is a recommended session timeout setting? mine is currently set at 7200

Once you install the mod you can replace the default list with this one :

Please note that I block facebook, MSN, Bing, Alexa and a couple others that you might not want to block, just remove them from the list and you are good to go ....

almaden
Anarchie
Artabus
ASPSeek
attach
autoemailspider
BackWeb
Baidu
Bandit
BatchFTP
BlackWidow
Bot\mailto:craftbot@yahoo.com
Buddy
bumblebee
CherryPicker
ChinaClaw
CICC
Collector
Copier
Copyscape
Crescent
DIIbot
DISCo
DISCo\Pump
dotbot
Download\Demon
Download\Wonder
Downloader
Drip
DSurf15a
eCatch
EasyDL/2.99
EirGrabber
email
EmailCollector
EmailSiphon
EmailWolf
Express\WebPictures
ExtractorPro
EyeNetIE
FileHound
FlashGet
FrontPage
GetRight
GetSmart
GetWeb!
gigabaz
GNIP
Go\!Zilla
Go!Zilla
Go-Ahead-Got-It
gotit
Grabber
GrabNet
Grafula
grub-client
HMView
HTTrack
httpdown
.*httrack.*
ia_archiver
Ichiro
Image\Stripper
Image\Sucker
Indy*Library
Indy\Library
InterGET
InternetLinkagent
Internet\Ninja
InternetSeer.com
Iria
JBH*agent
JetCar
JOC\Web\Spider
JustView
larbin
LeechFTP
LexiBot
lftp
Link*Sleuth
likse
//Link
LinkWalker
Mag-Net
Magnet
Magpie
magpie
Mass\Downloader
Memo
Microsoft.URL
MIDown\tool
Mirror
Mister\PiX
Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler
MS\FrontPage*
MSFrontPage
MSIECrawler
MSProxy
Navroad
NearSite
NetAnts
NetMechanic
NetSpider
Net\Vampire
NetZIP
NICErsPRO
Ninja
Nutch
Octopus
Offline\Explorer
Offline\Navigator
omgili
Openfind
PageGrabber
Papa\Foto
pavuk
pcBrowser
Ping
PingALink
Pockey
psbot
Pump
QRVA
RealDownload
Reaper
Recorder
ReGet
Scooter
Seeker
Siphon
sitecheck.internetseer.com
SiteSnagger
SlySearch
SmartDownload
Snake
sogou
Soso
SpaceBison
speedy
Spinn3r
sproose
Stripper
Sucker
SuperBot
SuperHTTP
Surfbot
Szukacz
tAkeOut
Teleport\Pro
URLSpiderPro
Vacuum
VoidEYE
Web\Image\Collector
Web\Sucker
WebAuto
[Ww]eb[Bb]andit
webcollage
WebCopier
Web\Downloader
WebEMailExtrac.*
WebFetch
WebGo\IS
WebHook
WebLeacher
WebMiner
WebMirror
WebReaper
WebSauger
Website
Website\eXtractor
Website\Quester
Webster
WebStripper
WebWhacker
WebZIP
Wget
Whacker
Widow
WWWOFFLE
x-Tractor
Xaldon\WebSpider
Xenu
Yandex
Yeti
YOUDAOBOT
Zeus.*Webster
Zeus
baiduspider
beta.statsit.com
statsit
SiteIntel
Yandex
GomezAgent
FunWebProducts
Nesotebot
DCPbot
AOL Advertising R&D
DataCha0s
aiHitBot
Apache-HttpClient
Zend_Http_Client
ReverseGet
XXX bot Content
vBSEO
spbot
OffByOne
thyroidbuzz
AcoonBot
coccoc
xpymep
proxyproxy2884
AppEngine
start.exe
Semiocast HTTP client
Firefox/3.6.23
TurnitinBot
curl
SwpLc/1.6
GrepNetstat.com
news bot
AskTbPTV
checks
panopta
App3le
PhantomJS
AlwaysOnline
SISTRIX
proximic
CRAWL-E/0.6.4
WebMoney
Maxthon
HTMLParser
oBot
UnisterBot
ERACrawler
Butterfly
Topsy
Butterfly Topsy Crawler
Ezooms
Deepnet
Alexa
Bitlybot
Seznam
Fulltext
Facebook
Sunrise Communications AG
crawl
Crawl
MJ12bot
Bimbot
Snapbot
thunderstone
Thunderstone
grub-client
Bing
MSN

Tks alot dude. Have installed and activated it using the list you recommended without Bing, MSN and Alexa. Let's see if it works.

One thing i don't understand is why like this all of a sudden??

jl255, lately man a lot of people have been seeing a huge increase in hits from search bots for what ever reason, who really knows.

Who is your host, if it's hostgator like me, their recent sale of the company to some idiot would explain why, they just finished a huge migration of servers and it looks like those servers saw fresh blood and decided to hit them with all they have...

If you have access to your phpMyAdmin I would recommend you go in their look for your session table and empty it, that's only if you are comfortable with doing that.

What is your session timeout set at?

One thing i would never do is have wordpress as a front end to vbulletin

Was this a "tech support" person from your host? They are infamous for not knowing diddly squat.

perhaps, but they have been pretty knowledgable in the past 5yrs i've been with them.

--------------- Added [DATE]1381074746[/DATE] at [TIME]1381074746[/TIME] ---------------

what is a good session timeout setting??

mine is currently set at 7200

But, their first and only instinct is to blame, you. And that's their default answer. And I'm yet to find even one of them who understands vBulletin. His answer to you is a crock of crap, to put it bluntly..That's TWO HOURS! I use 15 minutes, 900 seconds.

BUT - see, if your host is right and the connections aren't visitors, but some unknown mystery glitch making vB itself duplicate connections, it wouldn't matter what your session time is.

Plus, this setting doesn't affect visitors, only users logged in. Your problem is with visitors.

I note in passing, you're now suddenly looking much better with:

Currently Active Users: 5194 (23 members and 5171 guests)

Looks like the "Ban Spiders by User Agent" mod is helping alot with the mystery, nonexistent "internal" connections. :D

it was still hovering at 20k 1hr ago. then i changed the session settings to 900 about 30mins ago and it came down.

I suppose you haven't thought about disable the hook system?.

what do you mean? i tried disabling all plugins, and it didn't seem to improve things much.

The session timer sets the cookie time for those who are LOGGED IN. It also determines how long visitors will appear in who's online.

Have you analyzed the user agent strings of the mass visitors? Five minutes logged in as admin I can tell you precisely what is going on.


EDIT: Looking at your "who's online" as a guest myself, I can see many guests are trying to reply to threads, trying to start threads - these are SPAM BOTS. If I could see their user agent strings i could tell you alot more.

--------------- Added [DATE]1381079015[/DATE] at [TIME]1381079015[/TIME] ---------------

I've never seen a more horrendous web page test from Meehan. Look at this mess - 16 seconds first byte time? And your host thinks this is acceptable? Note: It is not. This server is undergoing a DDoS attack.

http://www.webpagetest.org/result/131006_X6_DJC/

http://www.webpagetest.org/result/13...DJC/1/details/


Your board is loading 217 requests per pageload when it should be more like 60. You have many 302 errors and even some 404 errors. It's loading over 1MB when it should be around 600kb. You have big problems independent and separate of the attack you're experiencing.

thanks alot max. you were right. there was a ddos attack. after the IP was blocked, everything pretty much went back to normal. really appreciate the assistance and the effort you took to help.

As a side note you should show guests as little as possible, one thing thats a load on your server is showing the "who's Online" to guests, disable it for them. Try to cut down on the amount of things you let them see, also with my mod, if they still exist in your list, prune out "custo" and "DA".

Mostly when you see a bot/spider on your forum you can enter it's name (one name on one line) in the list and after your whos online timeout period has expired the offending bot will be no more :), some bots have variations of their name in their useragent and some dont have their display name in the useragent string at all!

make sure you make use of these tools in the mod description here https://vborg.vbsupport.ru/showthread.php?t=264932

cool tips. i've disabled who's online for guests now. contemplating whether to disable it altogether....

Your members generally like to see it and browse it so maybe put up a poll and ask your memebers to vote whether they want to see it or not?

BTW i'm on your site as a guest and i can see whos on line!!, i'd also turn off stats for guests too, that will save plenty of queries and overheads.

You have not. You have only removed the link to online.php from the main page. Disable permission to view online.php in usergroup permissions if you really want it gone to visitors and guests.

http://forums.sgclub.com/online.php <--- Anyone going here can still get the who's online page load.

I note SIGNIFICANT improvement in both number of visitors and also page load speed!

But you still have many problems:

http://www.webpagetest.org/result/131007_1W_M2J/

First byte time is still terrible - this is your host. You're still loading a huge number of requests (277, that's you.)

I agree with Simon, you should dump the advanced stats for visitors. But if you keep it, make it only five instead of 12. Also you should restore the "What's going on" on main page, this tells legitimate humans you have a busy site and encourages more registrations.

is it really necessary to disable the online.php? pretty sure most visitors won't deliberately go to that php without any link?

my images are already served from CDN. not sure how else i can improve FBT?

isn't the 'whats going on' still there on the homepage?

again, appreciate all the suggestions and feedback. i'm seriously looking into it.

Bots and search engines don't need the link in order to crawl it, most bots have a preprogrammed list of urls to start at. You can leave online.php enabled including the links but you turn off all guest privilidges to view it, do the same for the stats.

I think Max_Taxable is making reference mostly to your host, your system, provided by the host, is delivering the first byte, or if you like shaking hands with the person requesting the page, in far to long a time.

As for your requests either reduce the number of stats you show and auxiliary bits n pieces or use some type of compression, try to amalgamate js files and minify them, compress your css files. Change your images from being a lot of individual images to one sprite and then deliver them from co-ordinates on the sprite.

There really is a ton of stuff to do if you know how and can be bothered ( I don't say that like you're not I say it because sometimes it all gets a bit to much and can put you off)!

Actually, no they are not.

Having the URL be "cdn.mysite/image" is NOT the same thing as having a CDN.

Here's every image your site loads, none of them are coming from a actual CDN.

http://www.webpagetest.org/pageimage...run=1&cached=0

Here are the details of your pageload, every call and how it served. No CDN here either.

http://www.webpagetest.org/result/13...M2J/1/details/

Bingo.

tks for the tips. online.php permissions have been turned off for guests as suggested.

i have contacted my host to see what they can do about the first byte issue.

i am contemplating a redesign of the forum and will definitely get the coder to look into your suggestions on compression and sprites. :)


hmm, cdn.xxxx.com is setup to point to my amazon cdn account where a copy of the image files reside as well. how do you tell if they come from the actual CDN?

There's no actual, real Content Delivery Network (CDN) involved in your pageload, is all I know. You can put images in a 3rd party server all day long and point the script to those files on that server, doesn't make it a cloud server or a CDN. Look at the webpage test results - it's clear and evident that your "CDN" not only is causing many more unnecessary calls, it's also actually slowing your page load down. See that big black X at the upper right hand corner of the test page under "Effective use of CDN?"

I doubt he would have to worry about using a cdn after implementing one or more steps from my guide (its in the sig). Unless his website consumes over 5GB worth of bandwidth daily, then he might want to pursue the cdn issue. Even then, there are some free cdn's he can use if needed.