src: add `--env-file-if-exists` flag

[BoscoDomingo]

Add a new flag, --env-file-if-exists that allows loading .env files without throwing an error if they don't exist.

Achieved by returning an env_file_data struct instead of just the paths of the .env files. This tells the loader if the file itself is required or not, thus allowing to not throw an error if the file doesn't exist and is optional, without breaking existing behaviours with --env-file (we could maybe show a warning if an optional file is missing?).

Usage:

node --env-file A.env --env-file-if-exists B.env

If B.env exists, it will load its environment variables and overwrite those that conflict with A.env. If it doesn't, node will continue as normal.

Fixes: https://github.com/nodejs/node/issues/50993 Refs: https://github.com/nodejs/node/issues/51451

[nodejs-github-bot]

Review requested:

• [ ] @nodejs/startup

[BoscoDomingo]

I think we shouldn't introduce a new flag. We should make the default --env-file not throw and revert the change.

@anonrig Are you sure? I am personally fine with #50588, and at this stage, we might be breaking people's CI processes and whatnot by allowing processes to run that will be missing environment variables. Multiple people in #49148 agree with this stance, too. I'd like to understand your reasoning

[anonrig]

@anonrig Are you sure? I am personally fine with #50588, and at this stage, we might be breaking people's CI processes and whatnot by allowing processes to run that will be missing environment variables. Multiple people in #49148 agree with this stance, too. I'd like to understand your reasoning

This is an "active development" non stable feature. therefore, breaking changes are expected.

IMHO, Instead of throwing an error, we can print a warning. I don't see why we should create a new flag rather than changing the current one.

[BoscoDomingo]

@anonrig Are you sure? I am personally fine with #50588, and at this stage, we might be breaking people's CI processes and whatnot by allowing processes to run that will be missing environment variables. Multiple people in #49148 agree with this stance, too. I'd like to understand your reasoning

This is an "active development" non stable feature. therefore, breaking changes are expected.

IMHO, Instead of throwing an error, we can print a warning. I don't see why we should create a new flag rather than changing the current one.

@anonrig I do see value in having a "hard" and a "soft" version, to allow users to decide what behaviour they desire. Plus the underlying implementation will be the same and thus barely any complexity is added while greatly improving UX/DX.

In my case, there are multiple instances where I'd want the node process to be aborted if the .env file is not present.

Is there any team we should tag to get their input on this as well?

[anonrig]

Is there any team we should tag to get their input on this as well?

There is no specific team responsible for dotenv. Originally, I implemented the dotenv parser and but we can always ask @nodejs/tsc for their opinion.

I think https://github.com/nodejs/node/pull/50588 was a mistake. @mcollina and several others have expressed their opinion on this as well, but it never hurts to ask for their opinion one more time :-)

FYI, I didn't block this pull-request, any collaborator can come in and approve your pull-request, and land this pull-request.

[BoscoDomingo]

Is there any team we should tag to get their input on this as well?

There is no specific team responsible for dotenv. Originally, I implemented the dotenv parser and but we can always ask @nodejs/tsc for their opinion.

I think #50588 was a mistake. @mcollina and several others have expressed their opinion on this as well, but it never hurts to ask for their opinion one more time :-)

FYI, I didn't block this pull-request, any collaborator can come in and approve your pull-request, and land this pull-request.

Agree on asking others. I'm not an expert on this matter either, so happy to hear other POVs. I do err on the side of keeping --env-file as is for now, though

[GeoffreyBooth]

If we add this, the flag name needs to start with env-file like the others.

I feel pretty strongly that the current behavior is correct. We should not be changing --env-file. I'm also against adding this new flag but I feel less strongly about that if there's a reasonable need for it. I don't consider "it's annoying to make my command understand the difference between dev and prod" a persuasive enough reason.

[mbrevda]

I'm also against adding this new flag

There seems to be a consensus that:

1. No new flags should be added
2. There should be a mechanism to signal not to error if the env file isn't found

Is there any precedent for loader-style arguments? Perhaps prefixing (or suffixing) the path with a symbol (say, ?) can meet all the above requirements?

E.g. --env-file=?.optional.env

[BoscoDomingo]

@GeoffreyBooth Fair enough. You can see in #51451 the different workarounds we currently need to do. I believe for this to compete with dotenv, bun or other alternatives out there, the UX has to be at least matching, if not better. They all currently march on when the .env file is not found. We can do better.

In terms of changing the flag, good on my side 👍🏼

@mbrevda I feel like that's too much complexity both in use (I'd never seen such syntax) as in implementation (what happens with --env-file? path/to/file). It'd be a first AFAIK and I'm not sure such a small flag warrants such a divergence from the norm, but again, open to change my mind!

[BoscoDomingo]

Updated code according to the failed CI jobs. Should all be good now

@GeoffreyBooth Updated flag to start with --env-file

[GeoffreyBooth]

I will block on changing --env-file. We should not be silently ignoring failing to load a specified file, even if other tools do.

I'm also fine with no solution landing for this problem. I'm not convinced this is something that Node should solve, as opposed to users better managing their commands for production versus development.

[anonrig]

I will block on changing --env-file. We should not be silently ignoring failing to load a specified file, even if other tools do.

We can always convert it into a warning.

[BoscoDomingo]

I agree with @GeoffreyBooth on this one. I'd much rather Node doesn't allow the program to run than to start it with the wrong assumptions.

In my experience, startup warnings are rarely read in production environments, and usually not even properly marked as such by log services. It's far worse to let systems crash in prod due to missing files than to not start them to begin with.

However, the pain for users having to manage multiple scripts for whether they want to load x file or not will be a major pain (imagine all the possible combinations) when it's an easy fix for us. Therein lies the value IMHO.

Edit: @anonrig I like the idea of a warning for missing optional files

[lewismoten]

I would opt to have the flag not fail when the file is missing, have an option to say that the file is optional, or simply detect that a .env file exists without specifying a flag. With .env files, the industry standard is to proceed when its missing because environment variables may already exist. The main issue I see is with build servers that don't have the .env file. You end up adding extra steps in your build to create the file with each build in order to support projects that use it when environment variables are already available on the build server.

[Shudrum]

Hello there.

Basically, both of the solutions are going to achieve the same result. But I would also prefer to have the flag to not fail.

I think every people using .env files are currently using motdotla/dotenv, and its behavior is to not fail. Like said @lewismoten it's kind of an industry standard. The reasons I think of are:

• It is strongly discouraged to commit .env files. So should not be mandatory.
• It's named "environment" because well ... it should depends of the environment. And it should allow developpers to set their environments locally.
• Many teams use environments to define default values for their applications, having a default one commited to their repositories. Having it missing is not frequent, and should be tested because...
• ...they frequently defines empty values, for the database for example. Throwing an error will not ensure of a proper configuration.
• Having two dot files for overriding is frequent, making one of both optionnal.
• I don't have the real statistics, but I'm sure at least 80% of deployed applications have their productions values defined from the running environment, like with Kube, AWS Fargate / ECS, GCP Run, Heroku, etc...
• Secrets are rarely wrote into an env file.

I know it is still possible to use dot files for, maybe, all my examples, but with more boilerplates.

Personnaly, I never used dot env files outside of local developments. With this solution, for example, I cannot run the tests with the same npm script on my CI and locally. That's why I stay with dotenv.

I love to minimize libraries on my microservices, I really hope it will stop to throw.

BTW thank you for the contribution and all my thanks to the team!

[BoscoDomingo]

Just for reference: this issue will be discussed in the next TSC meeting since a consensus has not been reached. More info in the counterpart PR by @anonrig

Thank you all for the input, and we'll see how this one goes!

[GeoffreyBooth]

I assume the intended usage of this PR is:

node --env-file=maybe-exists.env --env-file-optional app.js

What if instead it’s:

node --env-file-optional=maybe-exists.env app.js

And if both --env-file and --env-file-optional are passed, Node throws an error that they can’t be used together.

[BoscoDomingo]

I assume the intended usage of this PR is:

node --env-file=maybe-exists.env --env-file-optional app.js

What if instead it’s:

node --env-file-optional=maybe-exists.env app.js

And if both --env-file and --env-file-optional are passed, Node throws an error that they can’t be used together.

@GeoffreyBooth No, the idea is to have --env-file-optional behave exactly like --env-file, but not throw an error when the file isn't found. E.g.

node --env-file this_exists.env --env-file-optional this_may_not.env

The flag was --optional-env-file originally, but I was asked to change it (so both show up together in the docs and to maintain consistency I assume)

[aduh95]

I would be in favor of either non-boolean flag (e.g. --missing-env-file=throw), or a more explicit one (e.g. --throw-on-missing-env-file); my idea is that it would be good to have a flag to which we can flip the default while keeping it readable (because --no-env-file-optional is not very clear, --no-throw-on-missing-env-file is much better, and --missing-env-file=warn is shorter). (I don't know if we are going to flip the default, but it's good to keep that option open)

[BoscoDomingo]

I would be in favor of either non-boolean flag (e.g. --missing-env-file=throw), or a more explicit one (e.g. --throw-on-missing-env-file); my idea is that it would be good to have a flag to which we can flip the default while keeping it readable (because --no-env-file-optional is not very clear, --no-throw-on-missing-env-file is much better, and --missing-env-file=warn is shorter). (I don't know if we are going to flip the default, but it's good to keep that option open)

@aduh95 I personally am not a big fan of flags that affect others, plus it doesn't really solve any of the issues why I proposed this change to begin with. One would still need multiple scripts for files that may or may not exist, and need to be aware of the context in which each is run to decide if the file will exist.

[aduh95]

I would be in favor of either non-boolean flag (e.g. --missing-env-file=throw), or a more explicit one (e.g. --throw-on-missing-env-file); my idea is that it would be good to have a flag to which we can flip the default while keeping it readable (because --no-env-file-optional is not very clear, --no-throw-on-missing-env-file is much better, and --missing-env-file=warn is shorter). (I don't know if we are going to flip the default, but it's good to keep that option open)

@aduh95 I personally am not a big fan of flags that affect others, plus it doesn't really solve any of the issues why I proposed this change to begin with. One would still need multiple scripts for files that may or may not exist, and need to be aware of the context in which each is run to decide if the file will exist.

I guess I misunderstood the way the flag worked. Maybe --env-file-if-exists would be clearer in that case?

[BoscoDomingo]

I guess I misunderstood the way the flag worked. Maybe --env-file-if-exists would be clearer in that case?

@aduh95 Sounds good to me

[mbrevda]

@aduh95 we've come full circle 😄: https://github.com/nodejs/node/issues/50993

[aduh95]

@BoscoDomingo can you make the flag emit a warning when the file doesn't exist? Also consider renaming the flag to --env-file-if-exists which seems to have consensus.

[BoscoDomingo]

@BoscoDomingo can you make the flag emit a warning when the file doesn't exist? Also consider renaming the flag to --env-file-if-exists which seems to have consensus.

Sure thing. I'll take care of it this weekend 👍🏼

[BoscoDomingo]

@anonrig Thanks for the tips! Not used to C++ so I'll implement the changes requested

[BoscoDomingo]

@aduh95 Not sure how to make the flag return a warning. When the file options are processed (InitializeNodeWithArgsInternal) no warnings are available, only errors, which if there are any, process is stopped.

Is there a way to do this without adding an extra parameter to InitializeNodeWithArgsInternal?

Edit: Wait, am I that dumb and all it takes is a simple fprintf()?

[BoscoDomingo]

Stuck on this:

I've changed the path to be a string_view, but I can't seem to get it to correctly print out. Using c_str() yields weird results. However not using it doesn't compile. Should I just use std::string and call it a day? Let someone else do the string_view handling (I'm extremely inexperienced with C++, so this is 100% skill issue)

[anonrig]

@BoscoDomingo You should use file_data.path.data() or std::string(file_data.path).c_str() if you want to copy it.

[BoscoDomingo]

@BoscoDomingo You should use file_data.path.data() or std::string(file_data.path).c_str() if you want to copy it.

@anonrig That's what I've been using to no avail, hence why it was so surprising to me. I'll give it a go this week if I have time, but I'm going on a 2-week vacation where I won't have access to a machine to develop, so this may be stuck for a bit until I'm back.

The path printed includes a bunch of weird characters I'm not sure where they come from: /home/bosco/repos/node/out/Release/node: #٠��U\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00nv/optional.env\x00env: not found\n

Keeping it as a string works, so I'm pushing that for now 👍🏼