Non-standard evaluation of empty-matching patterns

[GfEW]

As of ugrep v7.3.0, the manual states:

The empty pattern "" matches all lines.  Other empty-matching patterns
do not.  For example, the pattern `a*' will match one or more a's.
Option -Y forces empty matches for compatibility with other grep tools.

First off, I'm new to ugrep and quite excited there's -Y to ensure standard matching behavior, allowing it to be used as a drop-in replacement for e. g. GNU grep.

This question is rather about the deviating default, in the first place.
As I derive from #287 , it's mainly performance motivated. However, doesn't 'a+' readily do the same thing?

If it does, why would a regexp matching tool assume by default that users running ugrep 'a*' (slower) actually mean to run ugrep 'a+' instead (faster, but different), as if they couldn't do just that equally well wherever desired? The core effect of (mis)?taking 'a*' for 'a+' that I can see is annihilating the difference between the two, thereby harming expressibility (and standard compliance) for no real use.

Please note, this question is not answered by one of them being much more commonly encountered than the other. If by alias hyena=dog, you interpret two originally non-synonymous expressions as synonyms, you take away expressibility from their distinct meanings: speakers won't be able to refer to a hyena by its name anymore (even if more elaborate expressions might still do), for the questionable merit of henceforth being able to refer to a dog by both, "a dog" or "a hyena".

What am I missing?

[genivia-inc]

This question is rather about the deviating default, in the first place. As I derive from #287 , it's mainly performance motivated. However, doesn't 'a+' readily do the same thing?

Yes, but that is a specific toy example with only one subpattern. What about x*y*z* for which x+y+z+ is not the same? And no, this feature is not only performance motivated.

For example, formulating an alternative for x*y*z* to match non-empty patterns is cumbersome with GNU grep: x+y*z*|x*y+z*|x*y*z+. With $n$ subpatterns using a * or ? quantifier we will end up with a regex that could be as long as $n^2$ because of subpattern duplication.

If it does, why would a regexp matching tool assume by default that users running ugrep 'a*' (slower) actually mean to run ugrep 'a+' instead (faster, but different), as if they couldn't do just that equally well wherever desired?

I think this is a tipsy-turvy view of things, because users use a+ to match any number of a. Why would they use a* instead? That is not the point of the non-empty match feature. Instead, consider practical and complicated patterns for which it quickly becomes much harder to formulate a regex that matches non-empty patterns. Any use of ? and * in such a pattern, including nested * and ?, may have to be reformulated and combined with other parts of the regex to form a longer regex that does not match empty. All that effort to write such a regex to avoid an empty match is unnecessary with ugrep.

[genivia-inc]

I'll update the man page to be more clear about the benefit:

    -Y, --empty
            Empty-matching patterns match all lines.  Normally, empty matches
            are not output, unless a pattern begins with `^' or ends with `$'.
            With this option, empty-matching patterns, such as x*y?, x*y*z*,
            and x*(y?|z) match all lines instead of the intended pattern.
            This option is enabled by default in grep compatibility mode,
            in that case specify option --no-empty to prevent matching all.

Note: --no-empty explicitly disables -Y when using ugrep as a grep replacement when you want to use this feature instead.

[GfEW]

Thank you for your answer. Some remarks on the man page:

            With this option, empty-matching patterns, such as x*y?, x*y*z*,
            and x?|y|z for example match all lines instead of the intended
            pattern (enabled by default in grep compatibility mode).

• I appreciate the rewording, those examples do a much better job at getting your point across and illustrating its motivation.
• The term intended still feels pretty assuming, IMHO it tells more about your idea of the users' intentions than about the latter. How about

            With this option, empty-matching patterns, such as x*y?, x*y*z*,
            and x?|y|z for example, match all lines instead of non-empty 
            lines, only. (Enabled by default in grep compatibility mode).

(Note there are two other, minor changes: The (Enabled ...) has moved to a separate sentence to make clear it refers to the previous sentences as a whole, rather than possibly to the last clause preceding it. And there's one additional comma.)

• When it comes to the merits of least surprise, approaches that assume less about intentions, and focus more on the established standard, tend to outweigh the merits of deviations from the standard that optimize for special case applications, but inevitably consume extra attention to be dealt with (and still can be confusing) - think e. g. of using different tools most of which do follow the standard, or doing pattern tests if you're on the way to understanding regular expressions. Whilst there are occasions where breaking a set standard can't be avoided for essential progress to happen, I beg to differ on the point that regular expressions that match empty lines (and are neither just the empty string, nor enclosed in ^...$) constitute such a case.
• That been said, the issue at hand is obviously at your decision and has been decided long ago. I highly respect the awesome tools you've given to the world and keep on improving.

[genivia-inc]

An appeal to a "subjective intent" in the phrasing is almost unavoidable to bring the message across. I would argue that "expected" rather than "intended" is preferable IMO, with a subtle hint to the POLS: "[...] match all lines, not only the expected pattern."