/clientscripts over to Amazon S3/Cloudfront?
 

We have moved most of our static images over to Amazon Cloudfront/S3 content delivery network (CDN) over the past few months with good results.

Now, I am considering moving all the vB clientside Javascript in the /clientscript directory over to Cloudfront as well.

Anyone else using a CDN for the JS in the /clientscript directory?

Any issues other than keeping in sync with updates anyone can think of?

Why would you want to do this?

Because these (clientside) scripts account for (approx) 1/3 of the traffic, so moving them to a CDN will reduce the load on the server and also increase performance (just as it did when we moved most of all the static images).

You should consider adding "Expires" HTTP header to .js files, so the user browsers won't have to re-check these files for updates on every page load. And by the way, the same could be done for the static images.

For the record, .js files (with the Expires header) account for 0.9% of our bandwidth usage. Forum images take another 1.20%.

I would think putting your money in a better setup/server is better then putting your money into another host, in this case the cloud setup.

I agree that static images and .js have little to no impact on any of my servers or sites.

There is more to be had with a good server and proper setup.

We have a good server and have run benchmarks with Amazon S3/CF. You cannot get the same global performance from a single server or local cluster than you can with a global CDN.

While your "opinion" is valuable. It is also incorrect because we have alread run the performance numbers and know the gain realized from the global Amazon CDN.

OBTW, Amazon CloudFront services are not "another host", so it seems you enjoy an "opinion" about something you do not understand. CloudFront is a Content Delivery Network with (seven I think, at last count) global geographic locations. It is not a hosting solutions.

We run over 500 metrics on our configuration and know the performance better than an uninformed third party who, frankly speaking, is just "hip shooting".

In fact, in the vBSEO forum, there are a number of users, like us, who server static content off the main server. Some use other hosts, others, like us, use a global CDN.

Sorry to be so direct. Facts are facts.

Cheers.

Not wanting to argue here, but if they are storing your images and then serving them and you are paying for this service, then in fact they are a host... lol

The only benifit I an see is one that you mentioned, being globally located for faster service to people on the other side of the world.

Other then that you are still paying for something you should be able to do yourself.

This all changes as the size of the forum has a large impact on these things as well, but since you didn't mentione this I went with what the aerage user would need and this service for the average user is useless imo.

Let's not argue, because you are shooting from the hip and will just keep shooting yourself in the foot my friend :-) I posted here, not do debate performance, but to look for configuration issues related to vBulletin, but I was able to work with the folks on the vBSEO board to fine tune all those issues.

Here are some published data related to CF/S3 performance, you can review yourself.

Amazon CloudFront / S3 Small Object Test Result

Huge client side gains for using CloudFront (across the board), some as high as 95% improvement.

On a different note, related to server side apache GETs (which effects apache workers, load, etc.)

As I posted earlier in this ill-fated thread, our /clientscript traffic accounted for 30% of all our GET requests (before I moved them to S3/CF yesterday). (The static gifs are mostly gone off the server and were already served from CF globally).

Regarding the reason I posted here, I was not looking for a performance discussion, as I know the performance intimately, serving millions of PVs per month.

We are all set now..... We had a very useful discussion over at vBSEO on this.

Cheers.

I would be interested in the outcome of this strategy.

The basic outcome has already been published:

Amazon CloudFront / S3 Small Object Test Result

I can confirm that moving the /clientscript also results in better server performance, especially if you have a lot of .js code (from mods, plugins, etc.).

Globally, pulling static objects like images and Javascript results in faster response time for end users (unless they are next to the server, of course, see results above), decreased Apache workers, decreased bandwidth (out from the main server, increased at CF/S3 of course) and a decrease in load average (load results depend on a number of factors and is not easily qualified).

Please note that vB/Jelsoft uses the same strategy (for a long time) in the Server Settings and Optimization Options:

So, you can see that Jelsoft/vB already use this strategy for the YUI scripts to increase performance, etc. Using a global CDN (like Amazon S3/CF, Limelight, etc). for static content (images, clientside scripts, etc) is the same strategy (but on a larger scale).

:-) Maybe we can convince Jelsoft to serve their scripts in S3/CF and permit all vB customers to pull from the global CDN as part of the license :-)

Not knowing all of the fine details involved, is it difficult to set-up to work through vbulletin?
Is the process very intense and is it easy to set-up & test before you actually 'launch' the cloud?
I have set-up the vbulletin to work on my shared server with very few problems I just don't know if this is going to be very intense, or a matter of simply uploading the files to amazon and it will serve them.

Are there timeout issues when the local amazon server is serving images and such quicker then the main server is handling the rest of the process? I am afraid of users timing out on posting new threads and replies.
Sorry if this is to simple minded but I want the best experience for my visitors even if I don't possess the knowledge others do.

There's no advantage to the cloud when compared to regular servers, in fact you will overcomplicate yourself having to learn how that specific cloud works and adapt to it's limitations, and learn how to scale with it. Pricing wise, the bandwidth is very expensive, and will be cheaper to go with a server.

This is definately not the case.

We use S3/CF on a site with around 4M PVs per month, and found using S3/CF was so easy to set it, it was almost trivial, and our users notice faster downloads.

In addition, a good server, no matter how gigantic, cannot outperform a globally distributed content delivery network (CDN) which is what Amazon CloudFront is.

I think Mr. Royo is confusing "cloud computing" with Amazon's CloudFront, which is a global CDN, not a "cloud computing infrastructure".

In addition, I am not posting from "theory", we actually run it, serving millions of users each month from over 200 countries :cool:

silkroad - I recently moved just about all my js files and images to amazon s3 w/ cloudfront - results are good so far. This took an additional 4-5gb a day off of the database server (where they were previously being served)

Basically all I did is search templates for "clientscript" and added the CNAME I created for cloudfront in front of /clientscript

I would like to figure out how to move avatars and other 'dynamic' images over to s3. I read through this thread wich looks like didn't result in much: http://www.vbulletin.com/forum/showthread.php?t=302300

I posted my experiences with CDN in that thread.

I used replacement variables rather than change the templates, and am using SimpleCDN on a test basis for now. I've noticed the bandwidth usage drop, so I know it's working.

On our server, avatars are in /forums/images/customavatars, so they are served by the CDN thanks to my directing the entire /images directory to SimpleCDN. The difference, though, is that SimpleCDN uses a "mirror" type of delivery where, if an image is requested from the CDN, and it is not cached there, it will grab it from the server. That is why I can safely push the avatars to visitors via SimpleCDN.

I don't know if Amazon has any mechanism like that, or if you could use something like rsync with your avatar directory to an extent where new avatars would appear after a short delay. (If I did it, I would change text to tell visitors that their new avatar would be active within five minutes, and have cron run rsync every five minutes to push the files out to the CDN.) Not ideal, of course.

Just some random thoughts (from a mind that is currently half awake ;) ).

I actually saw your post, redwing, and thought a replacement variable is a better idea. I don't believe Amazon has a mirror type system like SimpleCDN. It may be even simpler for that reason.

I'm curious about the Amazon solution, so I'll be reading up on that in the future. (I have too many projects going on right now to think about it.) For our purposes, SimpleCDN's service works well enough, and it's low-maintenance enough that I don't have to worry about setting anything up to sync our images between the forum and CDN. (We're all volunteer, so the less time I have to spend on it, the better. ;) ) Even so, it's a bargain. My "free" $15 lasted quite awhile, at least 6 weeks or so.

Yes, we noticed a drop in bandwidth usage (of course) and also a faster user response time, especially our users in Asia Pacific where there Amazon CF has CDN nodes:

• United States
  1. Ashburn, VA
  2. Dallas/Fort Worth, TX
  3. Los Angeles, CA
  4. Miami, FL
  5. Newark, NJ
  6. Palo Alto, CA
  7. Seattle, WA
  8. St. Louis, MO
• Europe
  1. Amsterdam
  2. Dublin
  3. Frankfurt
  4. London
• Asia
  1. Hong Kong
  2. Tokyo

Whoooo, that is very nice! I like that feature very much. The issue we had (have) with SimpleCDN is that they do not have international nodes. We like the Amazon CF CDN because of their 8 CDN nodes in the US, 4 in Europe and 2 in Asia (and they continue to expand and add more). I would consider SimpleCDN if they had a global reach like CF.

Yes, there are plenty of command line tools to update Amazon S3 and sync the images between CF/S3 and the server. We currently don't do it, but might consider it in the future. I like the SimpleCDN method on pulling missing files from the original server. That is very nice. So, I'll take this up with Amazon CF.

Cheers.

--------------- Added [DATE]1251136521[/DATE] at [TIME]1251136521[/TIME] ---------------

We user RR for this.. Worked great.

--------------- Added [DATE]1251137222[/DATE] at [TIME]1251137222[/TIME] ---------------

Another alternative, of course, is to use mod_rewrite and 301 over to the CDN of your choice, and not bother with editing templates or adding RRs.

..... FWIW

--------------- Added [DATE]1251139047[/DATE] at [TIME]1251139047[/TIME] ---------------

FYI:

http://developer.amazonwebservices.c...threadID=35547

I think SimpleCDN had one or two in Europe, but I don't remember exactly. But, they did have plans for expanding in the near future, so I'm sure they'll have more resources as time goes on. I figured that having even just a couple of servers spread around the U.S. was better than our single web server, which serves from only one location. Most of our traffic is U.S./Canada based, with a sizeable chunk from Europe, but we only have just a few visitors from elsewhere in the world.

SimpleCDN calls ours a Mirror Bucket. $0.039 US per GB of transfer, with no other fees for storage or setup. So last week, it cost us a whole $1.56 to deliver just over 40GB of files. Now you can see why we like it. ;) Since I have some usage stats, I should compare the costs to Amazon CF.

One thing some users may not like is that SimpleCDN is evolving...quickly. When I signed up over a month ago, the services had different names, but now they have reverted back to their "bucket" naming for their service levels. Their pricing has changed for Mirror Buckets too, although in a good way: once you deliver so much content, your pricing drops to a lower tier. My only fear is that there will be a price JUMP, and knock it out of affordability. But heck, they give you $15 in "play money" to try it out for free. That's why I'm not ruling out Amazon--if anything should happen and we need to host these files somewhere else, I'd like to be prepared with a backup plan.

AWS CF charges in many ways ways:

(1) Uploading files to S3 storage (bandwidth).
(2) Pulling from CF to S3 (bandwidth and per request) and
(3) Pulling from the CF CDN (per request and bandwidth).

There are probally more charges. AWS CF/S3 is not cheap.

I wonder if there is a price comparision on the net?

--------------- Added [DATE]1251143701[/DATE] at [TIME]1251143701[/TIME] ---------------

I was checking.... I think I'm going to give SimpleCDN a try and compare with Cloudfront...

Thanks for the tip about "Mirror Bucket" :)

--------------- Added [DATE]1251149179[/DATE] at [TIME]1251149179[/TIME] ---------------

OK, I see, you are 301'ing your entire image directory :-)

That explains how you get the customavatars in the CDN. I searched the templates and the phases and could never find how to set a CDN URL for avatars (and a few other hard coded image paths).

I don't understand why Jelsoft hard coded the customavatar domain as the forum domain and made it necessary to 301 those over to a CDN :(

At first I was editing the templates, but then when I wanted to try getting /clientscript over to the CDN, I discovered replacement variables. From that point on, I went in and changed my templates back to their original directories, then redirected the entire /images directory to CDN using the replacement variables. That also solved the issue with avatars. (My "recipe" is in that thread linked above, but I could repost it here. It was just a handful of entries in replacement variables.)

I have not attempted attachments, though, as we don't have a direct URL that we access them with. (They are stored in the filesystem, however.) I'm sure there's something in vB's code I could modify to retrieve images from the CDN, but since we're just doing this on a trial basis, I'm not that motivated yet.

I think I asked that very question on vb.com when I was looking for a way to change the avatar directory, and came up empty. And I think someone from vB's staff pointed me to the replacement variables which, at that point, I had no clue as to what they did, or how they worked. Had I known... ;)

Default setup for custom avatars is in /customavatars ..... (if you don't change it in the CP) so you would have to either (1) move the user avatars over to the image directory and 301 everything in images or (2) 301 custom avatars, or (3) use RR and replace $post[avatarurl] with http://yourcdn.net/$post[avatarurl]

Anway, I'll read your post where you outlined how you did it....

We used RR for /clientscript when we moved to AWS Cloudfront, then we changed the StyleVars in the templates for the style images. I hand coded a new phrase $vbphase['cdn_image_bucket'] and put that in a few places.....

Guess I'll RR or 301 the custom avatars over to SimpleCDN and give it a go :-)

--------------- Added [DATE]1251153995[/DATE] at [TIME]1251153995[/TIME] ---------------

Note, for customavatars in postbit you can either use an RR or edit the template to change:

To

Assuming you have your CDN URL in $vbphrase[image_bucket_url] .....

Cheers.

FYI, I managed to trim a couple more seconds off page loads by modifying this:
http://www.vbulletin.com/forum/showthread.php?t=306573

If you wanted to you could add a line in there to make the css file read off of a dynamic bucket simplecdn. This way you wouldn't have to upload a new css file every time you want to change something:

but then, like mentioned the WYSIWYG editer doesn't work correctly - I can't figure that one out.

We pull the CSS files from SimpleCDN without any issues at all. The reason is that SimpleCDN has the concept of "Mirror Buckets" so if a file is requested from the CDN and does not exist in the CDN cache, it will get the missing file from the origin-server.

Because of the excellent ways to use SimpleCDN Mirror Bucket, we have moved completely off Amazon CloudFront/S3. You might find this post interesting (more details about Mirror Bucket):

On Demand Files for "Cache Hit" Misses for CF/S3 - Feature Request

Right - are you sure your WYSIWYG editors are working properly?

If you don't use $style[css] at all in headerinclude, css for the editors and vbulletin_important.css do not load. The code above partially corrects this, but for some reason my (and I believe other's) editors don't load a block of css when the main css is hosted on a CDN. I've tried serveral different things with no solution to getting the editor to load properly.

Yes, all everything is fine. We have been running in this configuration for nearly a week, no problems at all.

I did have to manually edit some of the Javascript and CSS files.

The way I did it was to simply look at the Apache2 log files and look for any "oddities" or 404 errors from the CDN and then track down the problem and fix it.

One of the "beauties" of Mirror Bucket is that you can see the 404 errors on your Apache server when it tries to grab a file that is missing from the CDN.

When I was debugging, I ran a tail -f on the Apache log file and used grep (egrep with regular expressions, actually) to fine tune the JavaScript paths in the templates and CSS files.

It works, absolutely, but it take a bit of analysis to track down all the hard coded paths in the vB templates, JS and CSS file includes.

Having said that, I think we might have to edit the CSS files manually if we make a change to a vB style, but this is not an issue for us, since we don't change the style CSS often.

--------------- Added [DATE]1251831991[/DATE] at [TIME]1251831991[/TIME] ---------------

As I recall, I used phpMyAdmin to edit the css field in the style table.

PS: I forgot to mention, I don't think we use a WYSIWYG editor, we just use the standard (Advanced and Quick Reply editors - the same editors here in this forum).

Maybe that explains it?

--------------- Added [DATE]1251832273[/DATE] at [TIME]1251832273[/TIME] ---------------

Ah! Sorry, We are not pulling this file from the CDN:

Maybe that's the reason.... I think that is one of the few files I have not moved to the CDN

Yeh, that would explain it - to be more specific the class "wysiwyg" doesn't load when the main css is not on the same server. Really strange. I can't track down where this class is coming from.
that file doesn't contain the wysiwyg class, but it does have a couple others.

OK, I checked the logs again for you....

The only CSS file that is not pulled from the CDN (currently) is ... (interestingly enough!)


/clientscript/vbulletin_important.css

--------------- Added [DATE]1251832733[/DATE] at [TIME]1251832733[/TIME] ---------------

It can be found in the css field of the style table in the dB.

What we did was use the search function in phpMyAdmin to track down these guys.

Accidentally, this is the only file we did not manually edit or change a stylevar or use a RR, etc.

See attached.... screen shot ....

Right - the code above actually points the hard coded location of vbulletin_important.css to a different location as well.

However, "wysiwyg" class is not contained in that file. This is what it looks like:
btw, it's not in vbulletin_editor.css either (path of this can be changed in editor_css template)

Edit: it looks like it's coming out of the database - but that doesn't explain why it won't load when the main css is on a different server.

--------------- Added [DATE]1251835311[/DATE] at [TIME]1251835311[/TIME] ---------------

I think I figured it out - The editor is loaded within an iframe (vB_Editor_001_iframe). It won't load .wysiwyg because it's calling from that iframe and is expecting the class css to be on the root domain.

I have been "overly vocal" lately about vBulletin not being "CDN friendly" .... It take a lot of detective work to track down all the hard coded paths to static files.

In addition, if you run vBSEO, vBSEO hard codes a "/" into some of their image rewrites so even when you change a basic vBSEO config var (not all, but one or two) you end up with "/http://... blah blah... "

It is a lot of work to move vB static object to a CDN, so much is hard coded without phrasing or style vars. Some are hard coded in the dB. Some plugins are even worse. I could not get a 100% solution, but I'm happy with the 99% solution for now.

Yeh, I've run into many similar problems. I really wish it was easier to change over attachments.

I did not move attachments over yet ... same problem, code not CDN friendly.

The main goal with CDNs, the way I understand it, is to reduce network latency by reducing the distance between the end user and the content.

CDNs reduce the distance over the wires that the data needs to travel, reduces the number of network hops and helps to avoid your data needing to traverse over congested network equipment (or the chance that it might hit congested equipment on the networks that are skipped).

Reducing latency is very helpful for videos and especially streaming media, which many of the CDN networks like to focus on (more bandwidth income for them with video).

Cloud hosting is defined differently by many hosts. The inconsistency makes it a bit difficult to compare. Not all clouds are the same and not all clouds are also content delivery networks.

Actually that is only one "main goal" of a CDN.

We find that using a CDN also significantly reduces the load on the primary server. Here as some example stats from our CDN provider for our site, and this is off peak:

Taking 43 hits per second off any server improves performance. So, CDNs provide both better delivery globally, and better origin server performance.

Also, there is a huge difference between "the theory of using a CDN" and actually using one. I find it a more-than-funny having folks who do not use a CDN to be explaining to me, using a CDN that serve over 20 million objects a week, the benefits of a CDN.

Just a quick note that 43 hits per second for a seldom changing and relatively small (a few hundred of megabytes in total) static content is nothing for the modern multiplexing web servers such as nginx or lighttpd. They are able to serve tens of thousands concurrent connections, effectively solving the C10K problem. This kind of load may pose a problem for Apache which isn't really suitable for the static content serving, but a few hundred hits per second for the static content residing in the OS cache (so there is no disk I/O) is peanuts for nginx.

Not trying to downplay your accomplishment, just pointing out there is another way to reduce server load - by switching to the more modern web server software.

No offense taken, don't worry. So please do not take offense to my direct reply.

First of all, you are replying, making numerous assumptions that might be relevant to your site, but not relevant to another site. So please don't be offended, but when I read your reply it seems you are not talking at all to me and our configuration requirements, but are just talking, to make a statement about web server optimization. Why?

Because you are simply promoting a few high performance web servers without considering the bigger picture. For example, did you consider that we may have solid reasons running Apache2 and that make extensive use of mod_rewrite and mod_geoip and other Apache mods?

Did you consider there are real costs of porting an entire site over to another web server that may or may not have the feature we need versus a few dollars a month for CDN services? A discussion about performance trade-offs without considering costs and other trade-offs is simply academic and generally meaningless.

Furthermore, off-loading static content to a CDN reduces the load on any web server. Even if you run nginx or lighttpd (which have have looked at and frankly do not like them) or Apache or anything under the sun, you will gain performance. Less hits means less load on any web server, independent and orthogonal to other server optimizations.

So, my impression to your post, nothing personal, is that you want to talk past me and make technical statements about web server software without considering the bigger picture (O&M, cost-benefit, features, etc.) of our requirements. The bottom line is that if we wanted to run nginx or lighttpd we would be running them. They are not "state secrets" ROTFL. We prefer Apache server for many reasons and yes, we have looked at both nginx and lighttpd. We prefer Apache2.

And.... as I said earlier, it is pointless (and also technically incorrect) for anyone to argue that moving static content off a server to a CDN is good for Apache but not for nginx or lighttpd. It is good for any configuration. The discussion gets muddled when we are discussing apples and someone wants to talk pasta and wheat bread.

So, if you intend to reply to me about CDN benefits and tradeoffs, then please do so. However, if you just want to talk and advocate your favorite web server technology or optimization strategy, that is a difference story. Go ahead, advocate nginx or lighttpd, but that has little or nothing do to with the discussion of the benefits of a CDN. They are orthogonal discussions.

CDNs benefit all configurations and software, modern or antique from the days of iron Gone-Daddy-Gone. Discussions about web server performance optimization, while interesting, are orthogonal to a discussion on the various benefits of moving static content off the server, any server, to a CDN.

Cheers.

I agree from an end-user standpoint but was commenting in regards to the business need that most CDNs appear to be working to solve. All of them seem to be focused on bringing the content as close to the end-user as possible in order to reduce network latency. Running fast non-overloaded equipment is pretty much a necessity for them, however there are times when CDN equipment does get overloaded.

They could easily solve the load problem by having a huge cluster of servers & equipment in a single datacenter, but that does not solve the network problem where latency can cause stuttering with streaming media, etc. This is why the CDNs have as many edge locations as possible, in order to address the network issues.

For example, you could host your images anywhere else regardless of quality and that would still reduce the load on your webserver. VPS, another dedicated server or even a clustered file hosting provider such as http://www.blueturbo.com/file-hosting.php Blue Turbo solves the server load issue, but not the network issue as they run a cluster from a single datacenter. The network issues generally are not nearly as significant for images as they are for video, though, which is why Blue Turbo makes sense for many (I considered using them, but I do not).

To see the real power and benefits of a CDN, we can use Highwinds for an example. They streamed coverage of the presidential inauguration to a peak of 625,000 concurrent visitors using 310Gbps bandwidth. They could have done this from a single datacenter, but the viewing would not have been optimal for some viewers due to the network issue.

In my opinion, that's what CDNs are really being created for. But, I use a CDN for basically the same reason as you. Last month I offloaded just under 50 million image views to a CDN. The sole reason that I began using a CDN wasn't about the network latency benefit but was to reduce load on my webserver until my new server is built & colocated. Even after my new server is online I might keep images running from a CDN but am not certain about that just yet.

Actually we use a CDN for a number of reasons (not one reason).

• Global delivery from the CDN network (network performance, end user experience globally)
• Deliver static content from "other than" origin server (server performance, user experience globally) reducing load (CPU, memory, and device IO) on origin-server.
• Network robustness, more secure, content delivered from many nodes globally, not one node or a geographic cluster.

When I started this thread I was using Amazon CloudFront, but now we have ported to SimpleCDN. CloudFront performance is much better than SimpleCDN, but SimpleCDN is much, much cheaper, the near-real time stats are better, cache control is better and getting non-CDN-cached files from the origin server is much better than the upload to S3 AWS model.

OBTW, SimpleCDN mirror bucket service is now cheaper (and must less work) than most budget web hosting based solutions to server static content (and the mirror bucket concept is better that server-server object sync-based solutions, IMHO)

PS: I am now considering GeoIP-based DNS in the future, for distributing origin servers globally, but there is no rush for this at the moment.

I wish one of the free DNS providers like SiteLutions offered GeoIP-based DNS services. Do you know of any?

Cheers!

--------------- Added [DATE]1252535963[/DATE] at [TIME]1252535963[/TIME] ---------------

RE: Highwinds

Highwinds does not appear to publish their pricing model like Amazon and SimpleCDN.

Also, since we are taking vBulletin forums, most forums serve small images (buttons, attachments, icons, avatars), CSS files and JavaScript (as static content).

Forums are not really applications that serve streaming coverage of presidential inaugurations (that is not really vBulletin). Forums, like vBulletin et al, generally have PHP/MySQL origin-server requirements and static content of small images (avatars, buttons, icons, attachments), CSS files and (clientside) JavaScript.

Bringing massive streaming content requirements into the discussion has very little to with a "normal" vBulletin application, frankly speaking, which would be 99.99+ vBulletin sites.

Did you go the replacement variable route, or did you do something else? I realize that I'm probably missing a handful of files, but at least I have the majority of them on the CDN. My solution may be 95%... ;)

P.S. I'm glad it worked out for you! I, too, like the Mirror Bucket method vs. uploading, as it also takes into account what I'd call "semi static" images like avatars, where members are uploading new avatars all day long. No worries on how to sync them.

One drawback (not really a drawback, but...) is that we have our static files out there on the CDN, served out quickly, but the rest of the site is still hosted on a single server. We really aren't exploring the full capabilities of CDN, but if it's serving our needs and doing what we need it to do, who's to argue? ;)

Something that happened last week sort of left me amazed at how some systems administrators are not using CDNs, that could be. When the Beatles remasters were released to radio last week, using the Play MPE service, you can imagine how bogged down their servers got when everyone started their downloads. To fix the problem, they had their host sell them more bandwidth, and IIRC, they may have also had to beef up their hardware.

If this wasn't an application tailor-made to a CDN, I can't think of another! Flexible bandwidth delivery, available on demand. And I'm sure with so many edge servers, delivery itself would have been faster as well.

While they were all FLAC files, still...figure 14 or 15 CDs worth of FLACs being downloaded by hundreds of radio stations...nothing this big is going to come along in the near future, and they'll find their new configuration is overkill for what they would normally use. More expense. With CDN (or at least SimpleCDN), it's essentially "pay as you go". You pay for what you use. They could have handled the spike without needing to do anything on their end.

But what do I know? I just run a lowly music forum on someone's behalf... :D