Multiple server Nic cards-public and private

[alexi]

I have seen a lot of questions on this subject and I happend to have some graphs handy so I thought I would put up a post that should help all "big" boards understand this a little better.
In a multi server setup the web server needs to talk to 2 different places. The internet so users can come and get their data and the database server to get the information they are requesting. This diagram shows that relationship:
http://www.wdwinfo.com/alexmeetpics/webservernics.jpg

The web server should have 2 seperate NIC cards, one facing the internet and 1 facing the database server. Even if your traffic is not that high trying to do this over 1 nic card is not a good idea because database requests will have to wait for the web requests.
The database server NIC will handle far more traffic than the public NIC. Let's look at some graphs. This graph shows 24 hours on my web server. That would be about 300 users at the low and 2200 simoultaneous users at peak

http://www.wdwinfo.com/alexmeetpics/eth0.jpg

The blue line represents the amount of data going out to the users, the green line represents the data coming in. Notice that there is far more going out as the web server serves up the pages. The "95th percentile" a measure of how much bandwidth you use is 4.97 mbits or megabits per second so out to the users a 10 based connection would be more than enough.

Here is the same graph between the webserver and the database server:

http://www.wdwinfo.com/alexmeetpics/eth1.jpg

In this case the blue line, way at the bottom represents the data from the web server to the database server. The green lines are the database server returning data to the web server. Notice how much more data goes over this connection than actually goes out to the users. That is one of the reasons it is so important to have it on a seperate nic card. Also note that the 95th percentile is 38.8 mbit so you would not be able to run a 10 based nic card you need a 100 based to not create a bottleneck. It is not neccesary to run a gigabit card although you would still see some improvement from that as it would let stuff get "off the wire" quicker at peak load.

Hope this helps!

[Paul M]

Nicely demonstrated. :)

[kmike]

That's some crazy backend traffic. Do you have the attachments stored in the database?

[alexi]

Nope, avatars and the like but we don't do any downloadable attachments

[libertate]

This is good information.

What I love to see is the correlation between how much goes out to the user vs. how much the server needs.

Why is there a need for the DB server to ship 40Mbits to the Web server, when the Web server only serves up, at most 4Mbits? Clearly a lot of the data is discared.

This is where stored routines on the DB server would come very handy. Instead of requesting a record, manipulating, then requesting an other record, manipulating... etc. 4,000 times - it could be shifted to the DB server.

Anyone experiment with rewriting some of the more DB intensive routines as stored routine?

[kmike]

On every forum page request, a web server requests from a db server:

1. datastore (896KB here), includes forum cache which may be quite big if you have lots of forums (we do)
2. style data (20+KB here)
3. whole set of templates for that forum page - up to 50 templates for a single showthread.php! it's difficult to estimate their size, assuming every template is 1KB, it adds another 50KB for every request
4. some other relevant data: session info, user info, forum/thread/post info, threads/posts themselves, etc

So for a single 50KB showthread page web server slurps about 1MB from the database. Scary, huh? And no, stored procedures won't help here as you need all that information on the web server to properly format and output the pages.

The proper solution would be to cache datastore, styles and templates in some kind of memory cache (like memcached, eaccelerator, apc).

[Erwin]

At the moment I cache the datastore using eAccelerator which helps a lot using the inbuilt config.php info (bearing in mind that it is not supported by vB as this is buggy).

[kmike]

We cache the datastore, the style data and templates in eAccelerator's shared memory, too. Helps to decrease a backend traffic a lot.

[hidjra]

We cache the datastore, the style data and templates in eAccelerator's shared memory, too. Helps to decrease a backend traffic a lot.
This sounds interresting. How did you accomplish this?

Grtz,
Hidjra :bunny:

[kmike]

Obviously, by the great deal of hacking.

[khalid]

Very informative alexi and nicely presented.

Thanks.

[FlyBoy73]

Great example.
I was billed almost $1,900.00 in overage expenses from my host one month because the PNet was not setup properly and my dual server setup was talking publicly over the net. The communication bandwidth was huge between our web server and our database server. I thought it was inaccurate, and not to contest the billing, but because I did not know they "chattered" that much..

Thankfully, they dropped the overage because it was partly their mistake. Needless to say they were happy to supply a crossover cable after this at no charge and give the data base server the single connection to the web server only.

[GNDI]

this is the teori, how do you do this for real whit freebsd 6.1?

my case:

I have a lovely server whit two nics. I want to "abuse" them both :)

The first nic should handle the normal www traffic (The outsite card)
The second nic should route to the DB server.

My OS is FreeBSD 6.1-RELEASE
www= Apache 2.2

The DB is Mysql 5.x

My IP-Addresses is 87.52.171.128/24 and the IP to the first nic is
87.52.171.131

How do I set this up to force the second nic only to get to the DB server
(Only one nic installed)

Other hardware to use for networking build is two swicthes One fare better
than the second, therefore I'll prefere to only use that one. One Cisco
router.

I have been reading a lot on this site
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-routing.html
but i didn't understand it :/ mayby I have been reading to little.

Please give me the exact scipting lines and setup (For dummies)

[DevilYellow]

I will soon have 2 boxes each with 2 Gbit NIC's. On the first box I bonded both NIC's together. The new box could be set up almost the same. I was thinking of connecting the DB server directly to the Webserver accross one of these NIC's and not do the bonding - but then I'd have to maintain the DB server thru the Webserver.

Thoughts?

Surely 2 bonded Gbit NIC's will have no problem talking to each other thru a Switch. The Firewall would block all traffic directly to the DB box and I would VPN in to maintain it.

[gabrielt]

Hi there,

Thanks for this idea, I have never thought about this before.

We have two webservers and one DB server here. How could we implement this idea here, since in your example you had only one webserver?

I though about adding two network cards on each server, i.e. on each webserver we would have one NIC connected to the Internet and the other NIC connected to the DB server, and on the DB server two NICs, one connected to each webserver. Would this work?

Thanks,
Gabriel.

[alexi]

That would work but I'm not sure I would do it that way. The DB server would have to have 2 ip addresses that would make configuring things a lot tougher.
What I would do is put in a gigabit switch and run 1 cable to the the database server and each web server. If your database server had dual nics you could team them.
Do you need access to the database server from the web for administration? If so you could use a low speed nic for that

[gabrielt]

Hi Alexi,

Thanks a lot for your input. I think this idea is really better.
No, we do not access our DB server via www.

Cheers,
Gabriel

[gabrielt]

Any ideas on how to measure the NIC performance, i.e. how to create a chart similar to the one posted on this thread? I wish to analyse how is the traffic on our NIC right now.

Thanks,
Gabriel.

[alexi]

Google MRTG, it's free

[gabrielt]

Thanks for the tip. I installed MRTG and also created some scripts for monitoring and the network traffic is around 10-12 Mbps right now on each web server and around 20 Mbps on MySQL server.

Thanks
Gabriel

[taydu]

How would I set up four servers to run vbulletin?

2x running Apache
2x running Mysql

- I wanted to be able to manage the 2 mysql server via ssh not thru the apache server.

I did some reading but not quite sure, so I better ask the expert ;)

- So each of my server need dual NIC ?

- The two web server will need dual Gigabits NIC (1 to the internet and one to the DB?)

- The two Mysql serve also need 1 gigbits NIC and 1 100mbps (giga to web server, and 100mbps to the internet for ssh connection?)

- 1x Load balancer for the 2 web server ? (which load load balancer is good)

- 1x gigabits switch to connect two web and 2 mysql servers together ?

thanks guys

[alexi]

You may need another switch to tie the servers together to go out to the internet for management depending on how many ports your host gives you. The load balancer may also be able to do this depending on the number of ports
So you would end up with internet line coming in
switch
Load balancer
web boxes
Database 10/100 cards off that switch

Then in the back end all 4 boxes on 1 switch

I'm not an expert on load balancers but I have heard good things about the Coyote's you might want to have a look

[taydu]

I'm not really good at this, if possible can you demonstrate it using diagram?

What you said about is the Hardware setup, what about the configuration on each box? The reason i'm asking this is because I will not set it up. I will either request the server provider tech to configure it or pay some admin to do the set up so i want to have it clear for them to understand what i'm trying to do.

Thanks

[Marco van Herwaarden]

I strongly suggest you post your question in the Server Configuration forum at vbulletin.com, where Eva2000 (our server configuration expert) might be able to help you a bit more.