bodyParser ignores maxBodySize and buffers whole file in memory for file uploads

[romandecker]

The bodyParser middleware seems to buffer the whole file in memory for file uploads.

Check the following minimum example:

'use strict';

var os = require( 'os' );
var restify = require( 'restify' );

var server = restify.createServer( {} );

server.use( restify.bodyParser( {
  // this should supposedly limit maximum upload size to 10 KiB
  maxBodySize: 10 * 1024,
  mapParms: true,
  mapFiles: true,
  keepExtensions: true,
  uploadDir: os.tmpdir()
} ) );

server.get( '/', function( req, res, next ) {
  res.json( { message: 'hello' } );
  next();
} );

server.post( '/upload', function( req, res, next ) {
  console.log( 'Received files:', req.files );
  res.send( 200 );
  next();
} );

server.listen( 8000 );

Then, upload a really big file to it:

# bigfile.dat has a couple of GiB
curl -F "file=@/path/to/bigfile.dat" localhost:8000/upload

This will send the process into accumulating tons of memory until it eventually runs out of memory and crashes with:

FATAL ERROR: JS Allocation failed - process out of memory

Am I making a mistake in setting up the middleware or is this a bug?

[micahr]

Looks like there are two pieces in play here, the maxBodySize doesn't limit the upload, rather it limits the body size regardless of how much data is uploaded. The second piece is running out of memory due to Node reading in the file.

Restify could abort the request and return a HTTP 413 status code in that case. @yunong or @DonutEspresso what are your thoughts?

[yunong]

Makes sense that this is a bug. It should respect the body size.

[itsOmidKarami]

so what happens? how should we limit uploaded file size?

[retrohacker]

Hey @rkarami! It doesn't look like there currently is a way to effectively limit the upload size at the front door. If you are interested in taking a pass at a PR I would be happy to offer help!

[richardkiene]

@micahr, @retrohacker, and @yunong FWIW I have only been able to reproduce this on Restify versions <= 2.x.x. It seems like the current code for the body reader reads in chunks until it is done or when a given chunk exceeds maxBodySize, which seems effective but it's possible I'm missing something.

I tested with different Node versions (0.10.48, 4.8.5, 6.14.3) and different Restify versions (2.0.0, 3.0.0, 4.0.0, 4.3.3). I used the exact code listed in this issue and an 8GB file uploaded from a separate machine to the test application. Only restify 2.0.0 hit ENOMEM and ENOSPC.