Attachments not in DB

[VirtueTech]

Hello,

I think it would be great if we could move attachments out of the DB so that on my high traffic site it doesn't load the graphic everytime a user views it.

I want it to be cached.

This would be a mod like Kier's custom avatar mod.

[tubedogg]

There are some disadvantages to this.

First, if you allow html, php, or other code files (js, asp, etc.), and they weren't stored in the database, they could be executed on your server by clicking the attachment link. This could be devastating if someone knew what they were doing.

Well I guess that's really the only disadvantage I can think of (I thought there were others...) but that's a pretty major one in my book.

[VirtueTech]

I would only be using it for jpg's anf gif's

Think you can do it?

[TechTalk]

This is a much needed feature. I for the life of me cant understand the point of filling our databases with images.

My db size is roughly 15megs and I have 50 members :(

[Wayne Luke]

Attachments aren't just for images. In my opinion, you don't even need attachments for that you can use the image tag but I digress.

If someone allows attachments of PHP, HTML or Javascript files which would be very handy on my board, it would be a big security risk to store these in a web accessible area on your site.

[SpankMe]

Well looks like we need separate attachment code just for images. My forums also use attachments heavily. The attachment table is 55Meg and growing. I run my server over a 128k ADSL line and this problem with image attachments not being cached is a real bandwidth sucker :(

[Wayne Luke]

use the [img] BBCode instead.

[SpankMe]

The problem with using [img] or <img> tags is that the user has to put the image on the web somewhere and most free web space providers don't allow images on their space to be displayed on other web sites.

Allowing them to store the image on my server guarantees them that the image will always be there.

[Jawelin]

Hi. I would renew the request with a little security option:
there should be a pseudo-automated way to edit (by Admin, for example) any post with attachments to store individually the attached files onto the server, redirecting the link from db to url.
Everything in an interactive way, so I could - for instance - check the attached file nature and store it on a web-server only if safe...

Another problem should be to save an attachment out of the DB directly (automatically) to the server which hosts the DB itself in a configurable directory....

What do you (ALL OF YOU) think about ?

Thanks a lot.
Happy new year!

[Martz]

How easy would it be have the attachments stored in another database, for example on a remote server. Attachments are as much of a priority for me, mySQL space on a high performance (shared) server isloads of $£$£$ - I have cheaper mySQL space I can use remotely.

HTH, I was going to make a new thread about this.. if anyone else is interested in this I would appreciate any feedback.

[ethank]

Couldn't you store the attachments in a non HTTP served directory and have the attachment.php page load it up from there?

Ethan

[auto]

yea, you should be able to - great point ethan :)

I REALLY need this hack as soon as possible, my attachments table is over 450 megs :eek:

[Jawelin]

Is there any FAST way to throw out the files from the db and store them to any dir on the same server ??
From php, of course...

Thanks

Bye

[Scott MacVicar]

I've already started work on this hack after some nameless admin here suggested to someone i would be able to do it ;) thanks heh

I've done the modifications to the functions that deal with the upload but i need to make more adjustments because all attachments are uploaded to a dir below root and they have a funky random name to stop people trying to execute them, so randomstring.file is what i've been calling them and then the attachment.php file opens them sends the header and then sends the file.

Hopefully I'll be able to get this finished soon.

[Jawelin]

GREAT!! :D

Probably should the following rows in attachment.php do the job if configured in settings ... header("Cache-control: max-age=31536000");
header("Expires: " . gmdate("D, d M Y H:i:s",time()+31536000) . "GMT");
header("Last-Modified: " . gmdate("D, d M Y H:i:s",$attachmentinfo[dateline]) . "GMT");
header("Content-disposition:$attachment filename=$attachmentinfo[filename]");
header("Content-Length: ".strlen($attachmentinfo[filedata]));
$extension=strtolower(substr(strrchr($attachmentin fo[filename],"."),1));
I.e. to store (echo $attachmentinfo[filedata];) the same stream instead of into DB to a configurable path.... and modify the url to the path for example in a new field of the attachment table...

;)

Yeah. My problem was exactly: how to write the bin output to the server ?

I also tried to follow the same path of Kier's hack about avatars (look here (http://kier.3dfrontier.com/vbhacks/avatarasfile.php) ), and I think another way could be to intercept the file, even with the random name that some php SHOULD know, when it's still on the server in the TMP directory...
Move it from there to the requested dir and bypass all the attachment.php job but the url creation linking to that directory.

For security reasons, anyway, I think should be better to make ONLY an AdminCP explicit feature to save the attachment file on the server, remove it from the DB and relink the post to new location. Nothing else...
(an automated url-location of executable files could be dangerous....)

What do you (all of you) think about ?

Thanks a lot for your work.

Bye

[Jawelin]

What about creating subdirs on a download dir of the server by random md5 hashes (tailed to 8 chars, better....) and keep the original filename ?
Like for example Java SDK is downloadable from the support site.

This way, any user should pass through the attachment php and couldn't be able to download the file directly as he doesn't know the full path ... !?!?

Thanks

[Scott MacVicar]

thats almost what i've done
new Attachment table structure
attachmentid
userid
dateline
filename
visible
data
hash

when you upload a file it creates a random 8 character string then hashes it to 32, the file then becomes HASH.file within the attachments folder. I've been running the folder under the document root just to be sure.

I'll be lookin for some beta testers.

I'll also look into making a file to remove files from database to folder at a later date.

[Jawelin]

Originally posted by PPN
thats almost what i've done
[...]
I'll be lookin for some beta testers.

I'll also look into making a file to remove files from database to folder at a later date.
HERE I'M ! :)
I'd like a lot also this last mentionned option to get out attachments from DB. Do you think that flow could be reversed Admin-uploading the file into ?

LMK if I could help you someway.
Thnx

P.S.: I think a 32-char dir shoul be too long for some kind of servers. It isn't an high-security matter, so an 8-char hash would be nicer. For example it could be, instead of a random number, also an algo of the filename... It's enough don't tell it to anyone ! ;)

[Scott MacVicar]

I'm positive all operating systems support up to a 32character filename, the hash is gonna be the name of the actualy physical file.

say the hash is
bc28af6f750004729474ccbb403bd0ee

and i upload earl.gif (my avatar :))

the file is moved from the temp location it is uploaded to the folder specified in the admin panel and the file is called bc28af6f750004729474ccbb403bd0ee.file instead of earl.gif, this prevents people from trying to guess the location and the fact that it has a different extension should stop people from trying to execute it, even if they could find it.

[Jawelin]

Ok, but when the file is stored w/ this name in a server dir and the browser accesses it to download, who actually changes its name to save locally with the right one ?
Thnx

[Scott MacVicar]

the browser never access's the file directly

attachment.php will send headers and then the content ot the file, they will never know what the hash was or where the file is located.

[Jawelin]

Ah... ;)

Well, where do you plan to 'intercept' the flow ? Before the file is inserted into db directly from the temp dir or just later or selectable by Admin ?
I mean, the attachment could be easly go into database and the Admin could have an option - near the attachment moderating - to save selectable files from table to directories; this way, the link on posts shouldn't be affected at all, as its target is attachment.php with a query_string.... and only this php should do the entire work.

This was my idea when started thinkin' about, but absolutely I'm not able to throw out the db data field to a file.... :pirate:

:)

[InvictuZ]

Any updates on this, or a download linky. :)

[Scott MacVicar]

@Jawelin:
I plan to intercept the file as soon as its in the /tmp directory and use the vBulletin functions that are used to move the files from the temp directory to the avatar or smiley directory.

@InvictuZ:
Its being updated daily about what section I am at, as soon as its finished it will be mentioned here and a thread started in the full release forum.

[Jawelin]

So the file will never go actually into db.... ;)
Attachment table would contains only those fields you described above, not the actual file data.... :noid:
Thnx

[Scott MacVicar]

In beta testing look at the link in my sig

[Jawelin]

GREAT! Sorry for the delay but I was away... (I spoke just like you were waiting something)
Very good job, as I can see by now.

I'll install ASAP and let you know any idea, if you'd like.
Thnx