Awkward quoting behaviour when splitting option values

[dbear496]

My Expectation

In my experience, when quoting is applied, and only when it is applied, exactly one quote level is removed. Every programming language I know of follows this rule and especially shells make use of quoting that behaves according to this rule. So I would expect picocli to follow this rule too. However, it does not -- regardless whether trimQuotes is enabled. In fact, trimQuotes just makes the situation worse.

Specific Surprising Behavior

Assume the basic @Option definition from part 11.13 of the manual:

@Option(names = "-x", split = ",")
String[] x;

1. With trimQuotes disabled, quoting is applied without removing a quote level.
   Actual behavior is "1,2,3" => ["1,2,3"]; quoting prevented splitting (i.e. it is applied), but the quote level is not removed.
   Expected behavior is "1,2,3" => [1,2,3]; a quote level is removed if and only if it prevents splitting.
2. With trimQuotes enabled, a level of quoting may be removed without applying any quoting.
   Actual behavior is "1,2,3" => [1, 2, 3]; a quote level is removed, but it did not prevent splitting.
   Expected behavior is "1,2,3" => [1,2,3]; a quote level is removed if and only if it prevents splitting.
3. With trimQuotes enabled, quoting may or may not be applied depending on specific locations of quotes.
   Actual behavior is "11,22,33" => [11, 22, 33] and "11,22,33",44 => [11,22,33, 44]; quotes did not prevent splitting in the former case, but did in the latter.
   Expected behavior is either "11,22,33" => [11,22,33] (preferred) or "11,22,33",44 => [11, 22, 33, 44]; behavior is consistent so it is predictable and intuitive for end users.
4. With trimQuotes enabled, a quote level may or may not be removed depending on specific locations of quotes.
   Actual behavior is "11,22,33",44 => [11,22,33, 44] and "11,22,3"3,44 => ["11,22,3"3, 44]; a quote level was removed in the former case, but not in the latter.
   Expected behavior is either "11,22,33",44 => ["11,22,33", 44] or "11,22,3"3,44 => [11,22,33, 44] (preferred); behavior is consistent so it is predictable and intuitive for end users.
5. Intuitively, the name "trimQuotes" indicates a variation in whether quotes appear in the result -- not a variation in splitting behavior.
   Actual behavior is "1,2,3" => ["1,2,3"] with trimQuotes disabled and "1,2,3" => [1, 2, 3] with trimQuotes enabled; trimQuotes changes whether splitting is performed.
   Expected behavior is "1,2,3" => [1,2,3] with trimQuotes enabled; trimQuotes stays true to its name and only changes whether quotes appear in the result and does not change splitting behavior.
6. Intuitively, the name "trimQuotes" indicates that quotes will be trimmed when it is enabled, but this is not always the case.
   Actual behavior is "11,22,3"3,44 => ["11,22,3"3, 44] with trimQuotes enabled; trimQuotes did not trim the quotes.
   Expected behavior is "11,22,3"3,44 => [11,22,33, 44] with trimQuotes enabled; trimQuotes trims the quotes that are used to prevent splitting.

Potential Remedy

Since this behavior, though surprising, awkward, unuseful, and broken in my opinion, is documented behavior, I don't expect it to be changed because that could break downstream applications that depend on this behavior. But perhaps a new configuration could be added that fixes these issues. trimQuotes being disabled (which is the picocli default) is certainly lesser broken than otherwise and merely fails to remove a quote level. So a configuration that removes a quote level whenever the split regex is non-empty would implement expected behavior.

I don't expect anything to ever be done about points 5 and 6, but I figured I'd mention them for completeness' sake.

A Question

As I explained at the beginning, nothing else handles quoting like this that I know of -- both when trimQuotes is enabled and when disabled. So I am curious why the picocli developers chose to implement these behaviors in the first place. Are there languages or applications I do not know of that behave in these ways that picocli quoting is modeled from?

[dbear496]

Another minor annoyance to highlight what I believe is the atrocity of trimQuotes enabled. Suppose an end user wants an entire option value to not be split and it includes literal quotes or gasp literal backslashes. Their shell command line would look something like this:

> command -x "\"\\\"\\\\\"11\\\\\", \\\\\"22\\\\\", \\\\\"33\\\\\", \\\\\"\\\\\\\\\\\\\\\\\\\\\"\\\"\""

It looks almost as bad as M4.

[remkop]

Thank you for the feedback. I agree with many of your points.
I am thinking to introduce a new method trimQuotesAfterSplit.

In this mode, the input is split first, where the split character is ignored within quoted sections, and quotes are removed from the resulting values if trimQuotesAfterSplit == true. Also, it could potentially remove quotes embedded in a value (as opposed to only remove quotes from the beginning and end of a value).

Examples:

                  trimQuotesAfterSplit                        trimQuotes
Input             true                false                   true                false
"1,2,3"           [1,2,3]             ["1,2,3"]               [1, 2, 3]           ["1,2,3"]
"11,22,33",44     [11,22,33, 44]      ["11,22,33", 44]        [11,22,33, 44]      ["11,22,33", 44]
"11,22,3"3,44     [11,22,33, 44]      ["11,22,3"3, 44]        ["11,22,3"3, 44]    ["11,22,3"3, 44]
a,b,"c,d",e,"x,y" [a, b, c,d, e, x,y] [a, b, "c,d", e, "x,y"] [a, b, c,d, e, x,y] [a, b, "c,d", e, "x,y"]

What do you think?

[dbear496]

Yes. Your examples of trimQuotesAfterSplit=true is exactly the behavior I would expect, i.e. quotes always prevent splitting and a quote level is always removed. I look forward to (hopefully) seeing this option is a future release.