How to match all word pairs (with overlap/look-behind)?

[kenorb]

I've list of words and I would like to capture all word pairs (including these between the matches). I've tried the following:

% echo "foo bar baz qux" | rg -o '(\w+)\s(\w+)'
foo bar
baz qux

however it doesn't capture all of them (I expect 3 lines in total, not 2). I've tried to use --pcre2, but I'm not sure of the syntax.

[BurntSushi]

It seems to me like if you want all word pairs, then you necessarily must also have the ability for matches to overlap. PCRE almost seems like it could give you that with look-around, but look-around doesn't actually change the bounds of the match itself. Overlapping matches are typically not supported by most general purpose regex engines.

Despite my heavy involvement in implementing regex engines, I am not particularly creative in their use and haven't spent a lot of time with PCRE features. So I think either someone else will need to answer this (if there's even a solution) or this question would probably be better asked somewhere where there are more PCRE folks.

---

I am working on adding support for overlapping matches to ripgrep's underlying regex engine. (Albeit, with no specific intent on exposing that in ripgrep proper.) So I decided to see what an answer to this would look like. The main problem with overlapping regexes is that they can be really weird. So for example, just applying your regex in overlapping mode doesn't quite give what you would expect:

$ regex-cli find hybrid regex 'foo bar baz qux' '(\w+)\s(\w+)' --matches -kall -Koverlapping
    build hybrid regex time:  42.447029ms
    create regex cache time:  27.169µs
                search time:  21.605µs
cache clear count (forward):  0
cache clear count (reverse):  0
                     counts:  [9]

PatternID(0): [0, 5)
PatternID(0): [0, 6)
PatternID(0): [0, 7)
PatternID(0): [4, 9)
PatternID(0): [4, 10)
PatternID(0): [4, 11)
PatternID(0): [8, 13)
PatternID(0): [8, 14)
PatternID(0): [8, 15)

That's because (\w+)\s(\w+) matches foo b, foo ba, foo bar, and so on. So you wind up with a lot more matches. Instead, you've got to re-work your regex to make sure it only matches what you want completely. I suspect word boundaries are the simplest thing here. Let's try that:

$ regex-cli find hybrid regex 'foo bar baz qux' '(?-u:\b)(\w+)\s(\w+)(?-u:\b)' --matches -kall -Koverlapping
    build hybrid regex time:  42.477143ms
    create regex cache time:  30.391µs
                search time:  20.05µs
cache clear count (forward):  0
cache clear count (reverse):  0
                     counts:  [3]

PatternID(0): [0, 7)
PatternID(0): [4, 11)
PatternID(0): [8, 15)

Yup, that looks right!

(The regex-cli tool is a development tool that I use to expose the internal library regex interfaces to the command line for quick iteration. It's not actually meant to be a search tool.)

[kenorb]

I've found the following workaround by printing missing pairs manually (in use case when they go together in 4 pairs):

$ echo "foo bar baz qux" | rg -o '(\w+)\W+(\w+)\W+(\w+)\W+(\w+)' -r "$(printf '$1 $2\n$2 $3\n$3 $4')"
foo bar
bar baz
baz qux

but with more words, it'll keep skipping the pairs along the edges (after qux).

This is with further improvements:

$ echo "11 22 33 44 55 66 77" | tee >(rg -o '(\w+)\W+(\w+)' -r "$(printf '$1 $2')") >(rg -o '(\w+)\W+(\w+)\W+(\w+)' -r "$(printf '$1 $2\n$2 $3')") >/dev/null | cat
11 22
22 33
44 55
55 66
11 22
33 44
55 66

but it's not perfect either (skipping the last pair) and giving one duplicate. But it's better than nothing. There is still area for improvements.