--env-file does not support inner quotes (does not behave like dotenv)

[macrozone]

Version

v20.16.0

Platform

Darwin <redacted> 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:14:38 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6020 arm64

Subsystem

No response

What steps will reproduce the bug?

inspired by this comment: #50814 (comment)

1. create an .env file:

   INNER_QUOTES="1: foo'bar"baz`qux"
   INNER_QUOTES_WITH_NEWLINE="2: foo bar\ni am "on" newline, 'yo'"

2. test with dotenv:

   // envtest-dotenv.js
   require("dotenv").config();
   console.log(process.env.INNER_QUOTES);
   console.log(process.env.INNER_QUOTES_WITH_NEWLINE);

   $ node envtest-dotenv.js

3. test with --env-file

   // envtest.js
   console.log(process.envconsole.log(process.env.INNER_QUOTES);
   console.log(process.env.INNER_QUOTES_WITH_NEWLINE);

   $ node --env-file=.env envtest.js

How often does it reproduce? Is there a required condition?

always

What is the expected behavior? Why is that the expected behavior?

outputs should be the same.

// dotenv output:

1: foo'bar"baz`qux
2: foo bar
i am "on" newline, 'yo'

What do you see instead?

output are different, node native terminates at the first occurrence of the double quote:

// --env-file output
1: foo'bar
2: foo bar
i am

compare that to dotenv:

1: foo'bar"baz`qux
2: foo bar
i am "on" newline, 'yo'

Additional information

• this makes it impossible to have env-variables that contain all three quotes (", ' and `) and newlines.
• There is no alternative to inner quotes, because escaping quotes have been rejected src: add support for escaping quotes with escape slash in --env-file #50814
• It's very problematic in many cases since depending on the quotes you are using, you need to use different outer quotes
• i noticed that because i am writing some tooling that creates those .env files based on some other configuration. I just used double quotes as outer quotes and noticed that it fails when the string contains double quotes
• be aware that dotenv also has a bug, when your string contains an actual newline (not "\n") AND has inner quotes, it will cut the string off at the line break. It works if the string does not contain inner quotes. It also works if you instead use "\n" instead of an actual line break. The output will still be the same and will have a real line break in it (as long as you put outer quotes around the string. without quotes, the \n will be literally in the output as "\n")

And

• since the format is so problematic, maybe its time for a better format?

[anonrig]

Pull requests are welcome

[YutongZhuu]

Pull requests are welcome

Hi, this is my first time contributing to Node.js, I will take this issue as my first contribution. I think this is a good starting place.

[macrozone]

added additional information around dotenv and the implications of the bug

[marekpiechut]

For reference, here are parsing rules from dotenv project page:

What rules does the parsing engine follow?

The parsing engine currently supports the following rules:

• BASIC=basic becomes {BASIC: 'basic'}
• empty lines are skipped
• lines beginning with # are treated as comments
• # marks the beginning of a comment (unless when the value is wrapped in quotes)
• empty values become empty strings (EMPTY= becomes {EMPTY: ''})
• inner quotes are maintained (think JSON) (JSON={"foo": "bar"} becomes {JSON:"{\"foo\": \"bar\"}")
• whitespace is removed from both ends of unquoted values (see more on trim) (FOO= some value becomes {FOO: 'some value'})
• single and double quoted values are escaped (SINGLE_QUOTE='quoted' becomes {SINGLE_QUOTE: "quoted"})
• single and double quoted values maintain whitespace from both ends (FOO=" some value " becomes {FOO: ' some value '})
• double quoted values expand new lines (MULTILINE="new\nline" becomes

source: https://github.com/motdotla/dotenv?tab=readme-ov-file#what-rules-does-the-parsing-engine-follow

[marekpiechut]

There is one more invalid use case:

MP_#CRAZY_COMMENT="2: foo bar\ni am "on" newl\nine, 'yo'"

is parsed into env by Node, but dotenv skips it.

[marekpiechut]

@anonrig @macrozone How strict do we want to be with dotenv compatibility? I've got a working fix that is handling all tests of dotenv. It improves compatibility and simplifies parser, but behaves differently with multiline "" values that have unbalanced ".

Covering all edge-cases of dotenv without using their regexp is pretty crazy.

[anonrig]

Covering all edge-cases of dotenv without using their regexp is pretty crazy.

Agreed. We started following dotenv through their tests, but I think it's ok to diverge from there for extreme edge cases. I don't think we need to be 100% compliant with dotenv.

[macrozone]

@anonrig @macrozone How strict do we want to be with dotenv compatibility? I've got a working fix that is handling all tests of dotenv. It improves compatibility and simplifies parser, but behaves differently with multiline "" values that have unbalanced ".

Covering all edge-cases of dotenv without using their regexp is pretty crazy.

I actually personally don't care about the compatiblitiy, but at the moment some env var values are absolutly impossible to declare. Namly one that contains: a line break, a double quote a backtick and a single quote. There is no way to declare such a env var.

Allowing to escape quotes would also solve it, but that was attempted and rejected because "its not compatible with dotenv".

I am also fine when it breaks with actual line breaks, since thats also bugged in dotenv. When using quotes you can encode line breaks with \n.

So if this works, it would be fine for me:

MY_VAR="singlequote: ', double quote: ", a line break: \n(i am on newline) and a backtick: `. that is all i need"