OTP-28.0-rc1: re:run failes on some binaries, but not on others

[RoadRunnr]

Describe the bug

I'm using jesse to validate a JSON document with a fairly large schema (https://github.com/jdegre/5GC_APIs/blob/Rel-18/TS29510_Nnrf_NFManagement.yaml).

As part of that validation the IPv6 address ::1 is checked against the regex ^((:|(0?|([1-9a-f][0-9a-f]{0,3}))):)((0?|([1-9a-f][0-9a-f]{0,3})):){0,6}(:|(0?|([1-9a-f][0-9a-f]{0,3})))$.

This used to work fine, but suddenly fails when switching to 28.0-rc1.

The code that does the validation is in https://github.com/for-GET/jesse/blob/master/src/jesse_lib.erl#L127-L128.

What it does is essentially:

Subject = <<"::1">>,
RE = <<"^((:|(0?|([1-9a-f][0-9a-f]{0,3}))):)((0?|([1-9a-f][0-9a-f]{0,3})):){0,6}(:|(0?|([1-9a-f][0-9a-f]{0,3})))$">>,
Opts = [{capture,none},unicode,ucp],
re:run(Subject, RE, Opts).

When I run that snippet from a small test module or from the CLI, then it works. However, running the exact same Subject and RE as part of large project results in a nomatch. Compiling the regex first does not change the result.

However, when I replace the re:run expression with: re:run(Subject, binary:copy(RE), Opts) then it also works within the large project.

So far trying to inspet RE with io:format and erlang:display has yield nothing. Is there anything I can do to provide more information?

To Reproduce

unknown

Expected behavior

successfully matches

Affected versions
28.0-rc1

[frazze-jobb]

We suspect that the state is wrong when returning from a yield, while executing the regex match function. We tried to replicate this by force, by making re:run() yield all the time and trigger a garbage collect in between. But we were not able to replicate the issue. What happens if you run re:run(Subject, RE, Opts) twice in your project, does it work the second time?
If you run

re:run(Subject, RE, Opts)
re:run(Subject, binary:copy(RE), Opts)

or

re:run(Subject, binary:copy(RE), Opts)
re:run(Subject, RE, Opts)

or

re:run(Subject, RE, Opts)
re:run(Subject, RE, Opts)

Do you get consistent results that the one with binary:copy(RE) always works, and the other one fails?

Tracer = spawn(fun F() -> receive M -> io:format("~p~n",[M]), F() end end),
Session = trace:session_create(my_session, Tracer, []),
trace:process(Session, self(), true, [garbage_collection, running_procs]),
re:run(Subject, RE, Opts),
re:run(Subject, RE, Opts),
trace:session_destroy(my_session).

Hopefully we see a gc happening when it is failing.

[RoadRunnr]

@frazze-jobb sorry for late answer, I didn't see your reply..

If tried multiple version of the re:run and the results are consistent. re:run with the binary:copy fails every time, no matter how often it is repeated, with binary:copy is works every time, mixing the two version in any order does no change the outcome.

Variant 1:

re:run(Subject, RE, Opts)    --> nomatch
re:run(Subject, binary:copy(RE), Opts)  --> match

Variant 2:

re:run(Subject, binary:copy(RE), Opts)  --> match
re:run(Subject, RE, Opts)    --> nomatch

Variant 3:

re:run(Subject, binary:copy(RE), Opts)  --> match
re:run(Subject, binary:copy(RE), Opts)  --> match

Variant 4:

re:run(Subject, RE, Opts)    --> nomatch
re:run(Subject, RE, Opts)    --> nomatch

I've now build a release with RCRE2 DEBUG_SHOW_PARSED enabled (https://github.com/erlang/otp/blob/master/erts/emulator/pcre/pcre2_compile.c#L68) and it has some interesting output.

When it works, the output is something like this:

+++ Pre-scan complete:
+++ 00 80090000 META_CIRCUMFLEX
+++ 01 80080001 META_CAPTURE 1
+++ 02 80080002 META_CAPTURE 2
+++ 03 0000003a :
+++ 04 80010000 META | 0
+++ 05 80080003 META_CAPTURE 3
+++ 06 00000030 0
+++ 07 803a0000 META ?
+++ 08 80010000 META | 0
+++ 09 80080004 META_CAPTURE 4
+++ 10 800a0000 META [
+++ 11 00000031 1
...
+++ 96 80160000 META_DOLLAR
+++ 97 80000000 META_END

That does corresponds to the regex and is what I would expect.

But when it fails, the output is:

+++ Pre-scan complete:
+++ 00 0000007b {
+++ 01 0000000a 
+++ 02 00000020 
+++ 03 00000020 
+++ 04 00000022 "
+++ 05 0000006f o
+++ 06 00000070 p
+++ 07 00000065 e
+++ 08 0000006e n
+++ 09 00000061 a
+++ 10 00000070 p
+++ 11 00000069 i
+++ 12 00000022 "
+++ 13 00000020 
+++ 14 0000003a :
+++ 15 00000020 
+++ 16 00000022 "
+++ 17 00000033 3
+++ 18 80170000 META_DOT
+++ 19 00000030 0
+++ 20 80170000 META_DOT
+++ 21 00000030 0
+++ 22 00000022 "
+++ 23 0000002c ,
+++ 24 0000000a 
+++ 25 00000020 
+++ 26 00000020 
+++ 27 00000022 "
+++ 28 00000069 i
+++ 29 0000006e n
+++ 30 00000066 f
+++ 31 0000006f o
+++ 32 00000022 "
+++ 33 00000020 
+++ 34 0000003a :
+++ 35 00000020 
+++ 36 0000007b {
+++ 37 0000000a 
+++ 38 00000020 
+++ 39 00000020 
+++ 40 00000020 
+++ 41 00000020 
+++ 42 00000022 "
+++ 43 00000064 d
+++ 44 00000065 e
+++ 45 00000073 s
+++ 46 00000063 c
+++ 47 00000072 r
+++ 48 00000069 i
+++ 49 00000070 p
+++ 50 00000074 t
+++ 51 00000069 i
+++ 52 0000006f o
+++ 53 0000006e n
+++ 54 00000022 "
+++ 55 00000020 
+++ 56 0000003a :
+++ 57 00000020 
+++ 58 00000022 "
+++ 59 00000065 e
+++ 60 00000072 r
+++ 61 00000047 G
+++ 62 00000057 W
+++ 63 00000020 
+++ 64 00000041 A
+++ 65 00000050 P
+++ 66 00000049 I
+++ 67 00000020 
+++ 68 00000074 t
+++ 69 0000006f o
+++ 70 00000020 
+++ 71 00000069 i
+++ 72 0000006e n
+++ 73 00000074 t
+++ 74 00000065 e
+++ 75 00000072 r
+++ 76 00000061 a
+++ 77 00000063 c
+++ 78 00000074 t
+++ 79 00000020 
+++ 80 00000077 w
+++ 81 00000069 i
+++ 82 00000074 t
+++ 83 00000068 h
+++ 84 00000020 
+++ 85 00000065 e
+++ 86 00000072 r
+++ 87 00000047 G
+++ 88 00000057 W
+++ 89 00000022 "
+++ 90 0000002c ,
+++ 91 0000000a 
+++ 92 00000020 
+++ 93 00000020 
+++ 94 00000020 
+++ 95 00000020 
+++ 96 00000022 "
+++ 97 0000006c l
+++ 98 00000069 i
+++ 99 00000063 c
+++ 100 00000065 e
+++ 101 0000006e n
+++ 102 00000073 s
+++ 103 00000065 e
+++ 104 00000022 "
+++ 105 80000000 META_END

That is exactly the text at the beginning of JSON Schema, which is:

{
  "openapi" : "3.0.0",
  "info" : {
    "description" : "erGW API to interact with erGW",
    "license" : {

The regex is somewhat deep in that schema.

This feels like a reference to a subbinary is somehow getting mangled.

EDIT: I've managed to build a reproducer!!!

The code is mostly generated with the OpenAPI generator and the schema is public too. I can therefore share the code. You can find it at https://github.com/RoadRunnr/ergw_oapi_h_oam_nrf

Run it with:

 rebar3 shell
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling ergw_oapi_h_oam_nrf
Erlang/OTP 28 [RELEASE CANDIDATE 1] [erts-16.0] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit:ns]

Eshell V16.0 (press Ctrl+G to abort, type help(). for help)
1> otp_9438:test().

On my OTP-28.0-rc1 kerl build installation, it crashes with:

** exception throw: {wrong_param,'NFProfile',schema,
                                 #{error =>
                                       {all_schemas_not_valid,[{data_invalid,#{<<"pattern">> =>
                                                                                   <<"^((:|(0?|([1-9a-f][0-9a-f]{0,3}))):)((0?|([1-9a-f][0-9a-f]{0,3})):){0,6}"...>>},
                                                                             no_match,<<"::1">>,
                                                                             [<<"ipv6Addresses">>,0]}]},
                                   type => data_invalid,
                                   path => [<<"ipv6Addresses">>,0],
                                   schema =>
                                       #{<<"allOf">> =>
                                             [#{<<"pattern">> =>
                                                    <<"^((:|(0?|([1-9a-f][0-9a-f]{0,3}))):)((0?|([1-9a-f][0-9a-f]{0,3})):){0,6}(:|(0?|([1-9"...>>},
                                              #{<<"pattern">> =>
                                                    <<"^((([^:]+:){7}([^:]+))|((([^:]+:)*[^:]+)?::(([^:]+:)*[^:]+)?))$">>}],
                                         <<"description">> =>
                                             <<"String identifying an IPv6 address formatted according to clause 4 of RFC5952. The mixed IPv"...>>,
                                         <<"example">> => <<"2001:db8:85a3::8a2e:370:7334">>,
                                         <<"type">> => <<"string">>}}}
     in function  ergw_oapi_h_oam_nrf_api:validation_error/3 (/usr/src/erlang/ergw/otp_9438/ergw_oapi_h_oam_nrf/src/ergw_oapi_h_oam_nrf_api.erl, line 331)
     in call from ergw_oapi_h_oam_nrf_api:'-validate_response_body/4-lc$^0/1-0-'/3 (/usr/src/erlang/ergw/otp_9438/ergw_oapi_h_oam_nrf/src/ergw_oapi_h_oam_nrf_api.erl, line 158)

Adding the binary:copy to the re:run in jesse_lib.erl line 103 makes it pass.

[RoadRunnr]

I've managed to minimize the reproducer to:

-module(otp_9438).

-export([test/0]).

test() ->
    String = ~"Prefix^((:|(0?|([1-9a-f][0-9a-f]{0,3}))):)((0?|([1-9a-f][0-9a-f]{0,3})):){0,6}(:|(0?|([1-9a-f][0-9a-f]{0,3})))$",
    <<_:6/bytes, P1/binary>> = String,
    io:format("RegEx: ~p~n", [P1]),

    Subject = ~"::1",
    Opts = [{capture,none},unicode,ucp],

    Match = re:run(Subject, binary:copy(P1), Opts),
    Match = re:run(Subject, P1, Opts).

With OTP-27.2.2, the output is:

Erlang/OTP 27 [erts-15.2.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit:ns]

Eshell V15.2.2 (press Ctrl+G to abort, type help(). for help)
1> otp_9438:test().
RegEx: <<"^((:|(0?|([1-9a-f][0-9a-f]{0,3}))):)((0?|([1-9a-f][0-9a-f]{0,3})):){0,6}(:|(0?|([1-9a-f][0-9a-f]{0,3})))$">>
match

With OTP-28-rc1 it fails:

Erlang/OTP 28 [RELEASE CANDIDATE 1] [erts-16.0] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit:ns]

Eshell V16.0 (press Ctrl+G to abort, type help(). for help)
1> otp_9438:test().
RegEx: <<"^((:|(0?|([1-9a-f][0-9a-f]{0,3}))):)((0?|([1-9a-f][0-9a-f]{0,3})):){0,6}(:|(0?|([1-9a-f][0-9a-f]{0,3})))$">>
** exception error: bad argument
     in function  re:run/3
        called as re:run(<<"::1">>,
                         <<"^((:|(0?|([1-9a-f][0-9a-f]{0,3}))):)((0?|([1-9a-f][0-9a-f]{0,3})):){0,6}(:|(0?|([1-9a-f][0-9a-f]{0,3})))$">>,
                         [{capture,none},unicode,ucp])
        *** argument 2: could not parse regular expression
                        missing closing parenthesis on character 105
     in call from otp_9438:test/0 (otp_9438.erl, line 14)

NOTE: it fails on the second re:run, the one without the binary copy.

[frazze-jobb]

Thank you, I have reproduced the problem and have a fix for it soon