External Attachment Storage

[SaintDog]

I know this has been asked before and generally points to a specific release here at vBulletin.org, however I am looking for a simple modification that will allow me to do the same without having to integrate a different download/attachment system.

Has anyone been able to successfully have their attachments ported to a seperate server that would mind sharing the code or Ext/Plugin to do so?

[Zachery]

I know that erwin had done it in the past, there are possibly several ways of doing this. Either ftping files to a remote server during the upload, or rysyncing the attachments folder to the remote server. Then uploading nessary files to the remote server to process things, like attachment.php global.php config.php etc.

[SaintDog]

I knew that Erwin had accomplished this, however I wasn't sure if anyone had made such coding public as of yet. It'd be a great feature to have as a side option or even in future versions of vBulletin.

I may have to take a look at those files and see what I can come up with, though if anyone else has already completed this task, I'd be very interested to speak with them and see how they accomplished this.

[Zachery]

Any reason you want to store files externally?

[SaintDog]

Any reason you want to store files externally?

Specifically, to lessen the load on a single server and off-load it to another. While on new communities, it wouldn't mean much, though on more active communities it could drop the load. The less activity specifically placed to a single server, the more speed available for other comsuming resources and queries.

Overall, to save CPU cycles, save memory and drop the load altogether. Why strain the server any more than it needs to be when you could simply offset the load to another if you have it available?

[Zachery]

You would probally notice a better preformance increasae by offlodaing all of your static files to an external server, or even another services. lighttpd is a great server, I've moved 95% of the static content on one of the sites i help run to an lighttpd server and the server was able to survive the E3 barrage, both the apache machine and the image/content servers.

[SaintDog]

Seperating the images & attachments from the static PHP files to an external server would be doing just that, however, remaining on an Apache based system.

The static files themselves do not force a load on the server, however utilizing and storing the attachments to the database or by using the file system does when there is such high usage to these areas of the community.

Offsetting the load of the attachments and images specifically to another server will reduce the load since a great deal of the activity is there.

This is not so much for your average community with posts, threads and users online, it's for an approach using vBulletin more as a CMS and download-specific tool.

[Zachery]

You'd be suprised how big of a difference offloading the other static content (not attachments) makes. We went from loads of 5,10,20, duruing preE3 week to 1~ the entire week of E3 while our servers were hammered. (we had more traifc after E3 as well). Apache is not the greatest static file server when you've got 4000 people trying to download large files.

[SaintDog]

With the current setup, everything resides on the same server and while the loads are not high by any means, there's room for improvement and seperating that which gains the most traffic is the best starting point. In this case, it's the images and attachments.

The other static files simply aren't hit enough, or even used as much to warrant moving them to their own server. It's kind of the opposite end of the spectrum.

Most people use vBulletin as a community software, in this case, more of a download and image manager.

[Zachery]

But you are ignoreing the fact of how often images are called, even if cached. Theres quite a few images here that apache would not have to deal with serving. Since i left your hosting I've moved from my own hosting company to consulting for lots of bigboards now and have been doing fieldtests of differnt methods to improve website preformance under high usage loads. I'm currently looking into not only clustering but also in general spacing systems to do specific tasks and working on finding programs that better accomplish tasks. Apache is a decent program for processing php, but lighttpd is doing a much much better job serving static content (images etc) and I'll be testing how it does with php in the naer future. However there are already a few very large boards who've made this transition and its been wonderful.

[SuperJETT]

The reason I'd love this feature is because I'm on a host with excellent support/uptime/etc but not the most bandwidth/storage. For $10/month you can get a ton of storage/bandwidth at other hosts that may have marginal support. What I'd like to do is keep a $10 account at Dreamhost for my attachments but stay at my real host for the vb processing since Dreamhost is a terrible vb host.

($10 account gets 1Tb bandwidth which climbs weekly and 20Gb storage which also climbs weekly, just don't use them for vb because of their cpu usage issues)

[Veer]

The reason I'd love this feature is because I'm on a host with excellent support/uptime/etc but not the most bandwidth/storage. For $10/month you can get a ton of storage/bandwidth at other hosts that may have marginal support. What I'd like to do is keep a $10 account at Dreamhost for my attachments but stay at my real host for the vb processing since Dreamhost is a terrible vb host.

($10 account gets 1Tb bandwidth which climbs weekly and 20Gb storage which also climbs weekly, just don't use them for vb because of their cpu usage issues)
I'm 100% agree with you! and im facing this issue. :(

[Zachery]

What happens when they acidently rm-rf your account and you have no attachments?

[SuperJETT]

What happens when they acidently rm-rf your account and you have no attachments?

Same as if the attachments were on the primary host and it were done there?

[Zachery]

It makes things much more complicated to have hosting at multiple sites and such, not to mention backups and you'd be sending possibly private data though the internet a second time by uploading to the remote site

[SuperJETT]

It makes things much more complicated to have hosting at multiple sites and such, not to mention backups and you'd be sending possibly private data though the internet a second time by uploading to the remote site

No more complicated than setting up Coppermine, bridging, and maintaining that to allow users to upload large files which are automatically resized to save server space.

Yes it does add to the mix, but for my situation it's a tradeoff well worth it.

[SaintDog]

Websites store and send private data almost everyday, sending from one server to another is really no different. What can be viewed/accessed on one server can be viewed/accessed on another. Enabling the option to port the attachments to another server is really no different than housing them on the same, security and privacy-wise.

If you happen to be sending extremely private data from server to server, chances are you're not going to be sending it through a forum, or storing the data as an attachment to a forum.

You're data is only as secure as the server it resides on, so while there may be a privacy or security concern, these concerns are more so ones that those using cheap budget hosts should be concerned about. For those using dedicated servers, or multi-servers, I hardly think that would be a concern as the server is in your hands and the network in the other.

[downlo]

I agree I would like this feature. We get more downloads than anything else and the bandwidth and space taken up gets rediculous. I would like to place the attachment files on another server.

[zlos]

I agree I would like this feature. We get more downloads than anything else and the bandwidth and space taken up gets rediculous. I would like to place the attachment files on another server.
Yeah. First "big please" to vB.com about that comes from middleage of two years ago. Since then nothing happened...

[Mickie D]

zachery would not hosting lighthttp to handle images and statics on another server be the same as hosting downloads on another server for security it will all get passed about ?

unless its just called from another server and not passed around ?

but i think its a good idea to host downloads somewhere else as some sites have large downloads and is benificial to host them on another server and close one call, i get around 500 people online and have two servers one for database and one for php/apache etc

it runs nice now but i am considering hosting the attachments elsewhere and also your idea of lighttp for static images maybe both on the same server.

but its a good idea i agree, and if erwins done it i am 100% sure its a good idea you seen his forum lol

Regards
Mickie D

[Zi55]

Anyone know how i can connect Erwin , I try to send him emails and PM but he didn't answer .. :( i need to do external attachment storage too much important .
Thanks

[Marco van Herwaarden]

Erwin is hardly active on vB.org anymore.

[Zi55]

Anyway to get coder to hack my attachment code ?
Thanks

[doc_sameer]

I would really want to know how this can be accomplished... please someone, devise a way of doing it!

VB should have this integrated in the future versions... With the availability of cheap but unreliable hosting & reliable but costly hosting... & the difference in their prices, its common sense to run VB on the reliable server & attchments on the cheap server.

[Zi55]

Please HELP

[Lynne]

I don't think this is a simply thing to do and so I think your best option is going to be to pay someone to do it. You can post in this forum for a paid service: https://vborg.vbsupport.ru/forumdisplay.php?f=30

[acyk]

hi ,guys

there is another idea for this

using vBSEO to rewrite the attachement URL and let the CDN do the rest of things. (such as reverse proxy )

http://www.vbseo.com/f23/rewrite-rules-attachment-cdn-34701/

[COBRAws]

hi ,guys

there is another idea for this

using vBSEO to rewrite the attachement URL and let the CDN do the rest of things. (such as reverse proxy )

http://www.vbseo.com/f23/rewrite-rules-attachment-cdn-34701/
That's not quite the original idea of external attachment store as requested on this thread.