Presence of middleware prevents access to raw request bodies greater than or equal to 16,384 bytes (16 KiB)

[jhahn]

Verify canary release

• [X] I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
      Platform: darwin
      Arch: arm64
      Version: Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000
    Binaries:
      Node: 18.7.0
      npm: 8.15.0
      Yarn: 1.22.19
      pnpm: 7.8.0
    Relevant packages:
      next: 12.2.4-canary.9
      eslint-config-next: N/A
      react: 18.2.0
      react-dom: 18.2.0

What browser are you using? (if relevant)

N/A

How are you deploying your application? (if relevant)

N/A

Describe the Bug

When attempting to upload a file over a few kilobytes (e.g. sending a POST request with a binary body and a Content-Type of multipart/form-data) via fetch or curl, the request stalls, then fails with the error:

error - Error: aborted
    at connResetException (node:internal/errors:704:14)
    at abortIncoming (node:_http_server:700:17)
    at socketOnClose (node:_http_server:694:3)
    at Socket.emit (node:events:525:35)
    at TCP.<anonymous> (node:net:757:14) {
  middleware: true
}

This occurs only for API pages with…

export const config = {
  api: {
    bodyParser: {
      bodyParser: false,
    },
  },
}

…and only when middleware is present; even something as basic as:

import {NextRequest, NextResponse} from "next/server"

export async function middleware(req: NextRequest) {
    return NextResponse.next()
}

Removing the middleware fixes the issue. Of note, very small request bodies (e.g. < 1kb files) work even in the presence of middleware.

Expected Behavior

Sending a POST request to an API endpoint with a Content-Type of multipart/form-data along with a reasonably sized (~200kB) binary payload should work and not stall.

Link to reproduction

https://github.com/jhahn/nextjs-upload-issue

To Reproduce

pages/index.tsx

import type { NextPage } from 'next'
import Head from 'next/head'
import Image from 'next/image'
import styles from '../styles/Home.module.css'

const Home: NextPage = () => {
  const uploadFile = async (files: FileList | null) => {
    if (!files) return
    const formData = new FormData()
    formData.append("file", files[0])
    const response = await fetch("/api/hello", {
      method: "POST",
      body: formData,
    })
    console.log(await response.json())
  }

  return <input type="file" onChange={(e) => uploadFile(e.target.files)} />
}

export default Home

pages/api/hello.ts:

import type { Readable } from 'node:stream';
import type { NextApiRequest, NextApiResponse } from 'next'

export const config = {
  api: {
    bodyParser: {
      bodyParser: false,
    },
  },
}

async function buffer(readable: Readable) {
  const chunks = [];
  for await (const chunk of readable) {
    chunks.push(typeof chunk === 'string' ? Buffer.from(chunk) : chunk);
  }
  return Buffer.concat(chunks);
}

export default async function (req: NextApiRequest, res: NextApiResponse) {
  if (req.method === 'POST') {
    const buf = await buffer(req.body);
    const rawBody = buf.toString('utf8');

    // Can do something here...
    res.json({ rawBody });
  } else {
    res.setHeader('Allow', 'POST');
    res.status(405).end('Method Not Allowed');
  }
}

The code for pages/api/hello.ts was adapted from https://vercel.com/support/articles/how-do-i-get-the-raw-body-of-a-serverless-function. However, I had to change const buf = await buffer(req); to const buf = await buffer(req.body);

[Evilscaught]

Sorry, I'm not necessarily here to answer your issue, but I did notice a potential bug when skimming through your code,

export const config = {
  api: {
    bodyParser: {
      bodyParser: false,
    },
  },
}

According to these NextJS' guides, shouldn't this be:

export const config = {
  api: {
    bodyParser: false,
  },
}

[jhahn]

@Evilscaught good catch! (The perils of hacking away until well after midnight…)

I’ve updated the code in the sample repo; this is what the corrected API page looks like:

pages/api/hello.ts:

import type { Readable } from 'node:stream';
import type { NextApiRequest, NextApiResponse } from 'next'

export const config = {
    api: {
        bodyParser: false,
    },
}

async function buffer(readable: Readable) {
    const chunks = [];
    for await (const chunk of readable) {
        chunks.push(typeof chunk === 'string' ? Buffer.from(chunk) : chunk);
    }
    return Buffer.concat(chunks);
}

export default async function (req: NextApiRequest, res: NextApiResponse) {
    if (req.method === 'POST') {
        const buf = await buffer(req);
        const rawBody = buf.toString('utf8');

        // Can do something here...
        res.json({ rawBody });
    } else {
        res.setHeader('Allow', 'POST');
        res.status(405).end('Method Not Allowed');
    }
}

Fixing the config export also meant changing back to const buf = await buffer(req);, bringing the code into alignment with the example provide by Vercel.

Unfortunately, the issue as described originally is still present; any sort of middleware seems to prevent the processing of a raw body over 1-2kB.

[levinhtin]

Same issue, post a multipart/form-data is prevented by middleware. When I remove middleware.ts file, it's work.

[jhahn]

After some additional digging, I’ve realized the multipart/form-data bit is probably a red herring.

I’ve reproduced the issue more precisely with a second test case at https://github.com/jhahn/nextjs-upload-issue/blob/main/pages/large-post.tsx:

/** Add your relevant code here for the issue to reproduce */
export default function Home() {
    const largePost = async () => {
        const formData = new FormData()
        const succeeds = new Uint8Array(16383);
        const fails = new Uint8Array(16384);
        self.crypto.getRandomValues(succeeds);
        self.crypto.getRandomValues(fails);
        const response = await fetch("/api/hello", {
            method: "POST",
            body: fails,
        })
        console.log(await response.json())
    }

    return <button onClick={largePost}>Send Large Post</button>
}

A POST of the succeeds array, with a size of 16,383 bytes, works when middleware is present.

Adding a single byte (the fails array) for a size of 16,384 bytes, causes the POST to hang when middleware is present.

(Again – when there’s no middleware, things work as they should)

Fascinatingly, 16,384 bytes (16 KiB) corresponds exactly to the default highWaterMark option for NodeJS Readable and Writable streams. (https://nodejs.org/api/stream.html#new-streamreadableoptions)

[jhahn]

I’ve updated the repo to 12.2.4-canary.11; this issue is still present.

[jhahn]

I’ve updated the repo to 12.2.4-canary.12 and confirmed the issue is still present.

[jhahn]

I’ve updated the repo to 12.2.5-canary.0 and confirmed the issue is still present.

[Marcelx8]

Could the issue be related to this change? https://github.com/vercel/next.js/commit/d7e83704bf8c20cc885bd3e5c82e99d0a23afa97

[dmgawel]

Apparently, it worked in v12.0.0 (middleware introduced). I did a bisect to narrow down which version introduced this bug and I can confirm it stopped working in 12.1.1-canary.1. Looking at the version changelog, it's almost certainly caused by https://github.com/vercel/next.js/pull/34519.

I did some testing and while sending application/json requests the issue is even more pronounced because it's present even with an empty or very small body. Again, without the middleware requests hits the API endpoint correctly.

Example request:

const response = await fetch("/api/hello", {
    method: "POST",
    headers: {
        "Content-Type": "application/json",
    },
    body: JSON.stringify({}),
});

---

EDIT: Looks like that small JSON payload issue was resolved by https://github.com/vercel/next.js/pull/35131, but the original issue still persists.

Following up, in my testing the issue occurs regardless of bodyParser setting.

[jhahn]

🤖 I’ve updated the repo to 12.2.5-canary.1 and confirmed the issue is still present. (I hope confirming with each canary release is somewhat helpful!)

[Marcelx8]

🤖 I’ve updated the repo to 12.2.5-canary.1 and confirmed the issue is still present. (I hope confirming with each canary release is somewhat helpful!)

@jhahn - I believe everyone following this issue appreciates you testing it with every release, thank you. It might be helpful to have a look at the canary release changelog to see whether there were any changes made to the related affected area.

[Pamboli]

This problem appears on the 12.2.3 version. In the 12.2.2 there is no problem. I hope this information may be useful.

[Marcelx8]

This problem appears on the 12.2.3 version. In the 12.2.2 there is no problem. I hope this information may be useful.

Verified on my end and working when I changed the version to 12.2.2

Thank you @Pamboli !

[dmgawel]

Great find @Pamboli, looks like the issue was first introduced in 12.1.1-canary.1, then fixed, and then a similar bug was reintroduced in 12.2.3-canary.17 (see new commits comparing to canary 16).

[jhahn]

I’ve updated the repo to the final release of 12.2.5 and confirmed the issue is still present.

I also did a bit of additional digging and confirmed the findings of @dmgawel:

• The last official release that doesn’t exhibit the issue described here is 12.1.0
• The last pre-release that doesn’t exhibit the issue is 12.1.1-canary.0

If I had to hazard a guess, https://github.com/vercel/next.js/pull/34519 is the PR that introduced the issue. https://github.com/vercel/next.js/issues/34966 looks relevant, as does its fix https://github.com/vercel/next.js/pull/35131.

[sloonz]

This occurs only for API pages with…

  api: {
    bodyParser: false,
  },
}```

Can you confirm this part ? Commenting away bodyParser: false (or the config) does not solve the issue for me. In fact it’s not even for API pages only, I can reproduce the issue on a test page ("/test" just returning a page containing "Hello world") fine.

[jhahn]

@sloonz good catch; thanks! While our use-case (uploaded files) requires access to the raw request body, I can confirm your findings that the issue occurs with or without the bodyParser config set.

[hmdnprks]

I have the same issue with http-proxy in my API route and a middleware file. The request will stall if I upload a file larger than 64kb. I think it's the default chunk size for the stream mentioned here. Removing the middleware file will resolve the issue.

[Gawdfrey]

Thanks for updating @jhahn ! Sadly downgrading did not help me, a bit unsure why but I will have to conduct further tests tomorrow. Removing the middleware worked great, but is necessary for the project I am working on so that is not an option.

[jhahn]

@Gawdfrey FWIW, the workaround proposed by @Pamboli (downgrading to 12.2.2) did not work for us, either.

[Gawdfrey]

Actually, I just found and attempted with the matcher filter and I was able upload a 295Kb file without deleting the middleware on version 12.2.4! This is of course only an option if you can do without middleware on the paths you exclude.

export const config = {
  runtime: "nodejs",
  matcher: "/about/:path*",
}

A bit late so I will confirm further tomorrow.

[jhahn]

I’ve updated the sample repo to 12.2.6-canary.1 and confirmed this issue is still present.

I can also confirm @Gawdfrey’s "matcher filter" workaround solves the issue, assuming you don’t need middleware on the excluded path(s).

[jhahn]

I’ve updated the sample repo to 12.2.6-canary.2 and confirmed the issue is still present.

[filipedeschamps]

@jhahn thank you for all your reports.

In the meantime, looks like this bug only happens in development and not on Vercels infra. We noticed this with our integration tests, it started to complain for posts above that limit, but in the preview environment, it worked.

[jhahn]

@filipedeschamps 😱 and 💯 thanks for that very interesting data point!

We’re not hosting on Vercel, but we didn’t even think to try what we assumed was non-functioning code in our production environment. We’ll do some additional testing on this end and update the issue with our findings.

[filipedeschamps]

@filipedeschamps 😱 and 💯 thanks for that very interesting data point!

This was discovered by @aprendendofelipe 🤝

[dmgawel]

I can confirm we experienced this issue in a production environment, not hosted on Vercel.

[ambrauer]

Building on the workaround found by @Gawdfrey (🙏), if you simply wanted to exclude all api routes from middleware, seems you could do something like this:

export const config = {
  matcher: ['/', '/((?!api/).*)'],
};

[Gawdfrey]

Thanks for the snippet works great for my use case! @ambrauer

[noc2spam]

Same issue here. Only downgrading to 12.2.2 helps. After downgrading as well as applying

export const config = {
    api: {
      bodyParser: {
        sizeLimit: '4mb',
      },
    },
  }

the API works as expected.