expressjs
How to get file contents before/after S3 upload?
Hi,
I'm using multer-s3 as a middleware to automatically upload files to an S3 bucket. However, I also need to get the file contents either before or after the S3 upload for further processing. I'd like to avoid downloading the file from S3 as it is supposed to be already in the server's memory. Could not figure out how to get the file contents.
Any idea would be appreciated!
@papaiatis - I don't think at any stage the data is entirely in the server's memory, once the upload channels are established, it flows (hint: pipe) into the destination file like a stream.
If you use memory storage the data is held in memory buffer, but then it inhibits disk storage. If you want both, you can workaround it using something like this:
var express = require ('express')
var fs = require('fs')
var app = express()
var multer = require ('multer')
var v = multer({ storage: multer.memoryStorage()}).single('foo')
app.post('/', (req, res, next) =>{
v(req, res, (e) => {
fs.writeFileSync(`uploads/${req.file.originalname}`, req.file.buffer)
console.log(req.file.buffer.toString()) // or whatever you want to do with the data
res.end('done!')
})
});
app.listen(8000);
basically setup a memory multer, and when the data is available, write it into the disk, as well as process it for your in-memory need, manually.
hope this helps!
@gireeshpunathil
Thanks for your response.
Actually I came up with something very similar.
I'm also using memoryStorage to keep the file on the server (temporarily), do some pre-processing and then I'm uploading the file to an S3 bucket manually (instead of multer-s3).
It would be nice if I could pipe one multer middleware to another or combine them together.
let us ask @LinusU to see if this is something in the roadmap. I strongly agree with the idea of a pipeline (just like next middleware primitive in express).
