shutdown hooks stuck waiting for dispose call

[ssilve1989]

Bug Report

When the process receives a SIGINT, the application gets stuck waiting for the httpAdapter.close() in the dispose() call in the shutdown lifecycles. It never reaches the shutdown hooks.

Current behavior

When enabling shutdown hooks and having an http server receiving requests from a polling source, like Prometheus, this behavior can be seen when sending a SIGINT into the process. The terminal prompt may be returned, but the process is still alive and providing output to the current terminal context.

Input Code

https://github.com/ssilve1989/nest-sigint-no-exit

Expected behavior

The http server should exit properly. I am not sure if this is a bug in Nest or Express, but it only seems to happen when the process is in the midst of receiving requests.

Possible Solution

Environment

Nest version: 8.0.8 and 7.16.18

 
For Tooling issues:
- Node version: 14.17.3 
- Platform:  Mac and Linux

Others:

[ssilve1989]

After some further investigation, I was able to reproduce this with a barebones express app instantiated in a similar way to how Nest is doing it. I had to add a call to removeAllListeners for it to properly exit the application when binding a SIGINT handler. Adding this

this.httpServer.removeAllListeners();

in the close() function of the express adapter helps. This eventually ends up resolving the dispose call and running the shutdown hooks, but still it returns the prompt immediately on SIGINT, and then sometime later runs the shutdown hooks.

There is still something funny happening as the next execution cycle after the httpAdapter.close() call seems to take too long after the close() function has ended.

[micalevisk]

looks like this only happens if we use nest start (and express):

https://user-images.githubusercontent.com/13461315/148582617-32b0b78c-cb23-4088-9b5f-71e0fcc36688.mp4

With node dist/main:

https://user-images.githubusercontent.com/13461315/148582675-5d35f573-4722-4031-a026-875bba64af4c.mp4

[kamilmysliwiec]

Would you like to create a PR for this issue @micalevisk?

[micalevisk]

I didn't investigate it enough :/ I would start by looking at https://github.com/nestjs/nest-cli/blob/master/actions/start.action.ts but I don't have time to tackle this soon.

[ssilve1989]

@micalevisk I don't use nest start in my current project, we are using the Bazel build system. So I don't think its specifically related to the Nest CLI tool.

[micalevisk]

@ssilve1989 so, to run the project, you're using node directly? I didn't understand why it worked with node dist/main.js but didn't with nest start tbh

[ssilve1989]

@micalevisk

so, to run the project, you're using node directly?

that's right

I didn't understand why it worked with node dist/main.js but didn't with nest start tbh

i'll try to find some time to try it that way as well with the reproduction repo I provided

[tolgap]

FYI, this should've been fixed with the introduction of the forceCloseConnections option: https://github.com/nestjs/nest/pull/10345

If you create your app using

const app = NestFactory.create({ forceCloseConnections: true });
app.enableShutdownHooks();

Nest will keep track of Connection: Keep-Alive HTTP requests and dispose them correctly.