ct_doctest.erl

%%
%% %CopyrightBegin%
%%
%% SPDX-License-Identifier: Apache-2.0
%%
%% Copyright Ericsson AB 2025-2026. All Rights Reserved.
%% 
%% Licensed under the Apache License, Version 2.0 (the "License");
%% you may not use this file except in compliance with the License.
%% You may obtain a copy of the License at
%%
%%     http://www.apache.org/licenses/LICENSE-2.0
%%
%% Unless required by applicable law or agreed to in writing, software
%% distributed under the License is distributed on an "AS IS" BASIS,
%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
%% See the License for the specific language governing permissions and
%% limitations under the License.
%% 
%% %CopyrightEnd%
%%
-module(ct_doctest).
-moduledoc """"
`ct_doctest` runs doctests on documentation examples. Using `ct_doctest` ensures that the examples
in the documentation are correct, up to date, and stylistically consistent.

The tested examples can be either in a module (normally written using
[documentation attributes](`e:system:documentation.md`)) or in files.
By default `ct_doctest` looks for markdown code blocks and runs any Erlang
code block found that looks like a shell session.

The doctest parser looks for examples that are formatted as if they were run in the
Erlang shell, using prompts of the form `N>`, where `N` starts at `1` for each block.
The expected output is written on the lines following the prompt. For example:


    -doc """
    This is an example of a doctest:
    
    ```
    1> 1+2.
    3
    ```
    """.

`ct_doctest` can be used in Common Test suites to validate documentation examples as part of your
test runs. Normal usage is to call `module/1` with a module name. For example:

```
all() ->
    [doctests].
doctests(_Config) ->
    ct_doctest:module(my_module).
```

## Prompt format rules

For a code block to run as a doctest:

- prompts must start at `1>` for each block
- each subsequent prompt must increment (`2>`, `3>`, ...)
- continuation lines must be indented
- `%` style comment lines are allowed in prompt blocks
- mismatched prompt numbering causes a doctest parse error

## Troubleshooting

If a doctest fails unexpectedly:

- use `verbose` to print per-block execution details
- verify that expected output matches the shell output exactly
- verify prompt numbering and continuation-line indentation

## Examples

Below are examples of supported formats for the code blocks in the documentation. The parser
is quite flexible and supports various styles, including multi-line expressions, comments,
and even prebound variables.

### Basic example

```
1> 1+2.
3
```

### Basic example using Erlang code

This example uses an explicit Erlang code block. That is,

    ```erlang
    1> 1+2.
    3
    ```

instead of the previous one which is a generic code block. Both formats are supported.

```
1> 1+2.
3
```

### Multi-line prompt

Use multiline prompts for expressions that span multiple lines by starting the prompt with `>` and indenting the continuation lines. For example:

```
1> 1
  +
  2
  .
3
```

### Multi-line with comma

It is possible to have multiple expressions in the same prompt, separated by commas. For example:

```
1> A = 1,
  A + 2.
3
```

### Multi-line match

The expected output can span multiple lines. For example:

```
1> [1, 2].
[
 1
 ,
 2
 ]
```

### Multiple prompts

Examples can have multiple prompts. For example:

```
1> 1 + 2.
3
2> 3 + 4.
7
```

### Defining variables

Any variable defined in the examples will be available in the following prompts. For example:

```
1> A = 1 + 2.
3
2> A + 3.
6
```

### Prebound variables

If the documentation examples rely on certain variables being prebound, you can provide these
bindings when calling `module/3`. For example, if you have a module
doc that uses a variable `Prebound`, you can set it up like this:

```
1> Prebound.
hello
```

and then in your test suite:

```
binding_test(_Config) ->
    Bindings = [{moduledoc, #{'Prebound' => hello}}],
    ct_doctest:module(my_module, Bindings, []).
```

### Ignore result

To ignore the results of a prompt, just skip writing the expected output. For example:

```
1> 1 + 2.
2> 3 + 4.
7
```

### Matching exceptions

Examples of failures can be tested by writing the expected exception after the prompt. For example:

```
1> hello + 1.
** exception error: an error occurred when evaluating an arithmetic expression
     in operator  +/2
        called as hello + 1
2> lists:last([]).
** exception error: no function clause matching lists:last([])
```

The simplest way to know what output to write is to run the example in the shell and copy the output,
including the `** exception` line.

If you don't want to include the entire exception message, use only the start of the message.

```
1> hello + 1.
** exception error
```

### Comments

Comments can be inserted anywhere in the code block. For example:

```erlang
%% A comment before the first prompt
1> [1,
%% A comment between prompts
  2].
[1,
%% A comment in a match
 2]
2> [1,
  %% Indented comment between prompts
  2].
[1,
 %% Indented comment in a match
 2]
3> """
  %% A comment in a string is not a comment
  
  """.
"""
%% A comment in a string is not a comment

"""
4> 1 + a.
** exception error: an error occurred when evaluating an arithmetic expression
%% Comments
     in operator  +/2
     %% in exceptions
        called as 1 + a
%% are ignored
```

### Matching of maps

When matching on maps, it is possible to use shell syntax, that is, `=>` and not `:=`, as in
normal Erlang code. For example:

```
1> #{ a => b }.
#{ a => b }
```

### Matching of ...

It is possible to use `...` in the expected output to indicate that the rest of the output
should be ignored. This is useful for outputs that are large or contain non-deterministic elements.

```
1> lists:seq(1,100).
[1, 2, 3, ...]
2> #{ a => b }.
#{ a => ... }
3> <<1, 0:1024>>.
<<1, 0, 0, 0, ...>>
```

### Compiling modules

ct_doctest can also compile full module code examples. It then looks for a
`-module` declaration to determine the module name and compiles the code as
if it were in a file. For example:

```
-module(my_module).
-export([foo/0]).
foo() ->
    {?MODULE, ?FUNCTION_NAME, ?LINE}.
```

The module is then available for use in following prompts. For example:

```
1> my_module:foo().
{my_module, foo, 4}
```

### Edge cases

The following are examples that are not supported by the parser and will be ignored.

```
a> should not be tested
```

```
 1> should not be tested
```

```
> should not be tested
```

```
should not be tested
1> 
```
"""".
-moduledoc(#{since => ~"OTP 29.0"}).

-compile([{nowarn_possibly_unsafe_function, {erlang, list_to_atom, 1}}]).

-include_lib("kernel/include/eep48.hrl").

-export([module/1, module/2, module/3, file/1, file/2, file/3]).

-doc """
Options for doctest execution.

* `parser` - Use this option to plug in an external documentation parser. The
  parser callback must be a `fun/1` and return a list of Erlang code block binaries.
  The code blocks are then checked to determine whether they should be run as doctests.
  If no parser is provided, a built-in markdown parser will be used.

* `skipped_blocks` - Sets the exact number of Erlang code blocks that are allowed
  to be skipped because no runnable shell prompts were found. It does not count blocks
  in any function listed in `missing_tests`. It defaults to `false`.

* `missing_tests` - A list of `{Function, Arity}` pairs that are expected to have
  documentation but no doctests. When this option is set, `ct_doctest` will fail
  if any documented function lacks doctests and is not in this list (i.e., a new
  function was added without doctests), and also fail if a function in this list
  now has doctests (i.e., the list is stale and should be updated). Defaults to
  not checking.

* `skip_tests` - A list of doc entries whose doctests should be skipped. Each entry
  is either `moduledoc` or a `{Kind, Name, Arity}` tuple where `Kind` is
  `function`, `type`, or `callback`. For example,
  `[moduledoc, {function, foo, 1}]` skips the moduledoc and the `foo/1` function.

* `verbose` - Print detailed information while running doctests, including each
  block run and skipped block details.
""".
-type options() :: [{parser, fun((unicode:unicode_binary()) -> [unicode:unicode_binary()] | {error, term()}) } |
                    {skipped_blocks, non_neg_integer() | false} |
                    {missing_tests, [{atom(), arity()}]} |
                    {skip_tests, [moduledoc |
                                  {function | type | callback, atom(), arity()}]} |
                    {verbose, boolean()} |
                    {compile_options, [compile:option()]}].

-record(options,
        { parser = fun parse_markdown_builtin/1 :: fun((unicode:unicode_binary()) -> term()),
          skipped_blocks = false :: non_neg_integer() | false,
          missing_tests = false :: [{atom(), arity()}] | false,
          skip_tests = [] :: [moduledoc |
                              {function | type | callback, atom(), arity()}],
          verbose = false :: boolean(),
          compile_options = [] :: [compile:option()] }).

-doc #{equiv => module(Module, [])}.
-spec module(module()) ->
          ok | {comment, string()} | {error, term()} | no_return().
module(Module) ->
    module(Module, []).

-doc #{equiv => module(Module, [], Options)}.
-spec module(module(), options()) ->
          ok | {comment, string()} | {error, term()} | no_return().
module(Module, Options) ->
    module(Module, [], Options).

-doc """
Run tests for the documentation in a module with EEP-48 docs.

When calling `module/3`, `ct_doctest` looks for documentation in the specified module and
runs any examples found there. The module, function, type, and callback documentation
are all checked for examples.

The function returns `ok` if all tests pass, or `{comment, Comment}` if all tests pass but one or more
functions lack tests. If any test fails, an exception in the form of `error({N, errors})` is raised,
where `N` is the number of failed tests. The details of each failure are printed to the console.

Use `Bindings` to provide prebound variables for a specific doc entry. Use
`moduledoc` for module docs and `{function, Name, Arity}` (or corresponding
`type`/`callback` keys) for entry-specific bindings.

See `t:options/0` for available options.
""".
-spec module(Module :: module(), Bindings, Options :: options()) ->
          ok | {comment, string()} | {error, term()} | no_return()
          when 
            KFA :: {Kind :: function | type | callback, atom(), arity()},
            Bindings :: [{KFA | moduledoc, erl_eval:binding_struct()}].
module(Module, Bindings, OptionsList) ->
    Options = options(OptionsList),
    HasParserKey = proplists:is_defined(parser, OptionsList),
    case code:get_doc(Module) of
        {ok, #docs_v1{ format = ~"text/markdown" } = Docs} when not HasParserKey ->
            run_module_docs(Docs, Bindings, Options);
        {ok, #docs_v1{} = Docs} when HasParserKey ->
            run_module_docs(Docs, Bindings, Options);
        {ok, _} ->
            {error, unsupported_format};
        Else ->
            Else
    end.

-doc #{equiv => file(File, [], [])}.
-spec file(file:filename()) ->
          ok | {error, term()} | no_return().
file(File) ->
    file(File, []).

-doc #{equiv => file(File, [], Options)}.
-spec file(file:filename(), options()) ->
          ok | {comment, string()} | {error, term()} | no_return().
file(File, Options) ->
    file(File, [], Options).

-doc """
Run doctests for a markdown file.

The function returns `ok` if all tests pass. If any test fails, an exception in the form of
`error({N, errors})` is raised, where `N` is the number of failed tests. The details of each
failure are printed to the console.

Use `Bindings` to provide prebound variables. Bindings are global for all files, so take
care to avoid any naming conflicts.

You can run doctests on non-markdown files by providing a custom parser that extracts the
code blocks to be tested.

See `t:options/0` for available options.
""".
-spec file(File :: file:filename(), Bindings :: [{atom(), term()}], Options :: options()) ->
          ok | {comment, string()} | {error, term()} | no_return().
file(File, Bindings, OptionsList) ->
    Options = options(OptionsList),
    case file:read_file(File) of
        {ok, Content} ->
            try
                Blocks = inspect(parse(Content, Options#options.parser)),
                {_RunResult, Skipped} = run_blocks(Blocks, Bindings,
                                {file, File}, Options),
                ensure_skipped_blocks(Options#options.skipped_blocks, Skipped),
                ok
            catch
                throw:{error, Error} ->
                    format_error({file, File, Error}),
                    error({1, errors});
                C:R:ST ->
                    io:format("Uncaught exception in file ~ts~n", [File]),
                    erlang:raise(C, R, ST)
            end;
        {error, _} = Error ->
            Error
    end.

run_module_docs(#docs_v1{ docs = Docs, module_doc = MD } = DocsV1,
                Bindings, Options = #options{ skip_tests = SkipTests }) ->
    
    MDRes = case lists:member(moduledoc, SkipTests) of
                true -> [];
                false -> parse_and_run(moduledoc, MD, Bindings, Options)
            end,
    Equiv = sets:from_list([KFA || {KFA, _Anno, _Sig, _EntryDocs, Meta} <- Docs,
                                   is_map_key(equiv, Meta)]),
    Res =
        lists:append(
          [parse_and_run(KFA, EntryDocs, Bindings, Options) ||
              {KFA, _Anno, _Sig, EntryDocs, _Meta} <- Docs,
              is_map(EntryDocs),
              not lists:member(KFA, SkipTests)]),
    Errors =
        [{{T,F,A},E} || {{T,F,A},[{error,E}],_} <- Res] ++
        [{moduledoc,E} || {moduledoc,[{error,E}],_} <- MDRes],
    _ = [io:put_chars(format_error(E)) || E <- Errors],
    case length(Errors) of
        0 ->
            MissingKFAs = case Options#options.missing_tests of
                              false -> [];
                              MTs -> [{function,F,A} || {F,A} <- MTs]
                          end,
            Skipped = lists:sum([Count || {KFA, _, Count} <- MDRes ++ Res,
                                         not lists:member(KFA, MissingKFAs)]),
            verbose_log(Options,
                        "module complete; total skipped blocks: ~p (expected ~p)",
                        [Skipped, Options#options.skipped_blocks]),
            ensure_skipped_blocks(Options#options.skipped_blocks, Skipped),

            ensure_skipped_tests(Options#options.skip_tests, DocsV1),

            NoTests = lists:sort([{F,A} ||
                                     {{function,F,A},[],_} <- Res,
                                     not sets:is_element({function,F,A}, Equiv)]),
            HasTests = lists:sort([{F,A} ||
                                      {{function,F,A},RunResult,_} <- Res,
                                      RunResult =/= []]),
            DocFuns = [{F,A} || {{function,F,A}, _Anno, _Sig, EntryDocs, _Meta} <- Docs,
                                is_map(EntryDocs)],
            ensure_missing_tests(Options#options.missing_tests,
                                 NoTests, HasTests, DocFuns);
        N ->
            error({N,errors})
    end.

format_error({moduledoc, Context}) ->
    [a_test_failed("moduledoc", Context), format_error_context(Context)];
format_error({{Type, Name, Arity}, Context}) ->
    [a_test_failed(io_lib:format("~p ~p/~p", [Type, Name, Arity]), Context), format_error_context(Context)];
format_error({file, Path, Context}) ->
    [a_test_failed(io_lib:format("file ~ts", [Path]), Context), format_error_context(Context)].

a_test_failed(Where, Context) ->
    LineNo =
        if is_map_key(line, Context) ->
            [" on line ", integer_to_list(maps:get(line, Context))];
           true ->
            ""
        end,
    io_lib:format("A test failed in ~ts~ts:~n~n", [Where, LineNo]).

format_error_context(#{ test := {test, Index, Test, Match}, message := Message }) ->
    io_lib:format("~ts> ~ts~n~ts~n~n~ts~n", [Index, Test, string:trim(Match), string:trim(Message)]);
format_error_context(#{ message := Message, context := Context }) ->
    io_lib:format("~ts~n~n~ts~n", [string:trim(Context), string:trim(Message)]);
format_error_context(#{ message := Message }) ->
    io_lib:format("~ts~n", [string:trim(Message)]).

parse_and_run(_, hidden, _, _) -> [];
parse_and_run(_, none, _, _) -> [];
parse_and_run(KFA, #{} = Ds, Bindings, Options) ->
    [do_parse_and_run(KFA, D, Bindings, Options) || _ := D <- Ds].

do_parse_and_run(KFA, Docs, Bindings, Options) ->
    try
        InitialBindings = proplists:get_value(KFA, Bindings, []),
        Blocks = inspect(parse(Docs, Options#options.parser)),
        {RunResult, Skipped} = run_blocks(Blocks, InitialBindings,
                                          {module, KFA}, Options),
        {KFA, RunResult, Skipped}
    catch
        throw:{error,_}=Error ->
            {KFA, [Error], 0};
        C:R:ST ->
            io:format("Uncaught exception in ~p~n", [KFA]),
            erlang:raise(C, R, ST)
    end.

run_blocks(Blocks, Bindings, Context, Options) ->
    {_Index, Result} =
        lists:foldl(fun(Test, {Index, {Acc, Skipped}}) ->
                            {Result0, NewSkipped} = test_block(Test, Bindings,
                                                               Context, Index, Skipped, Options),
                            {Index + 1, {Acc ++ Result0, NewSkipped}}
                    end, {1, {[], 0}}, Blocks),
    Result.

test_block(Code, Bindings, Context, Index, Skipped, Options) when is_binary(Code) ->
    ContextLabel = context_label(Context),
    FirstLines = first_lines(Code),
    verbose_log(Options, "running block ~p in ~ts:~n~ts",
                [Index, ContextLabel, Code]),
    try run_test(Code, Bindings, Options) of
        [] ->
            verbose_log(Options, "skipped block ~p in ~ts (no runnable prompt, ~p skipped): ~ts",
                        [Index, ContextLabel, Skipped + 1, FirstLines]),
            {[], Skipped + 1};
        Result ->
            verbose_log(Options, "passed block ~p in ~ts", [Index, ContextLabel]),
            {Result, Skipped}
    catch
        throw:{error, ErrorContext} = Error ->
            verbose_log(Options,
                        "failed block ~p in ~ts:~n~ts~n",
                        [Index, ContextLabel,
                         format_error_context(ErrorContext)]),
            throw(Error);
        C:R:ST ->
            verbose_log(Options,
                        "failed block ~p in ~ts with ~p:~tp~nblock snippet:~n~ts",
                        [Index, ContextLabel, C, R, Code]),
            erlang:raise(C, R, ST)
    end;
test_block(Other, _Bindings, _Context, _Index, _Skipped, _Options) ->
    throw({error, {invalid_code_block, Other}}).

context_label({module, moduledoc}) ->
    ~"moduledoc";
context_label({module, {Kind, Name, Arity}}) ->
    lists:flatten(io_lib:format("~p ~p/~p", [Kind, Name, Arity]));
context_label({file, Path}) ->
    unicode:characters_to_binary(Path);
context_label(Other) ->
    lists:flatten(io_lib:format("~tp", [Other])).

first_lines(Code) ->
    Lines = string:split(Code, "\n", all),
    lists:join($\n, lists:sublist(Lines, 5)).

verbose_log(#options{ verbose = true }, Fmt, Args) ->
    Str = io_lib:format("ct_doctest(verbose): " ++ Fmt, Args),
    [First | Rest] = string:split(string:trim(Str), "\n", all),
    io:put_chars([First, [["\n    ", Line] || Line <- Rest], "\n"]);
verbose_log(_, _, _) ->
    ok.

parse(Content, ParserFun) when is_function(ParserFun, 1) ->
    validate_code_blocks(ParserFun(Content));
parse(_Content, Parser) ->
    Msg = io_lib:format("Invalid parser provided: ~p. Parser must be a fun/1.", [Parser]),
    throw({error, #{ message => Msg }}).

validate_code_blocks({error, Reason}) ->
    Msg = io_lib:format("Parser returned an error: ~p.", [Reason]),
    throw({error, #{ message => Msg }});
validate_code_blocks(Blocks) when is_list(Blocks) ->
    [validate_code_block(Block) || Block <- Blocks];
validate_code_blocks(Other) ->
    Msg = io_lib:format("Parser returned invalid result: ~p.", [Other]),
    throw({error, #{ message => Msg }}).

validate_code_block(Block) when is_binary(Block) ->
    Block;
validate_code_block(Other) ->
    throw({error, #{ message => io_lib:format("Invalid code block: ~p.", [Other]) }}).

options(OptionsList) ->
    lists:foldl(fun
                        (parser, Acc) ->
                            Acc#options{ parser = proplists:get_value(parser, OptionsList) };
                        (skipped_blocks, Acc)  ->
                            Acc#options{ skipped_blocks = proplists:get_value(skipped_blocks, OptionsList) };
                        (missing_tests, Acc) ->
                            Acc#options{ missing_tests = proplists:get_value(missing_tests, OptionsList) };
                        (skip_tests, Acc) ->
                            Acc#options{ skip_tests = proplists:get_value(skip_tests, OptionsList) };
                        (verbose, Acc) ->
                            Acc#options{ verbose = proplists:get_value(verbose, OptionsList) };
                        (compile_options, Acc) ->
                            Acc#options{ compile_options = proplists:get_value(compile_options, OptionsList) };
                        (_Key, Acc) ->
                            Acc
                end, #options{}, proplists:get_keys(OptionsList)).

parse_markdown_builtin(Markdown) ->
    extract_erlang_code_blocks(inspect(shell_docs_markdown:parse_md(Markdown))).

extract_erlang_code_blocks(Ast) when is_list(Ast) ->
    lists:append([extract_erlang_code_blocks(Item) || Item <- Ast]);
extract_erlang_code_blocks({pre, [], [{code, Attrs, [Code]}]}) when is_binary(Code) ->
    Class = proplists:get_value(class, Attrs, ~"language-erlang"),
    Tokens = string:split(unicode:characters_to_binary(Class), ~" ", all),
    case lists:member(~"language-erlang", Tokens) of
        true ->
            [Code];
        false ->
            []
    end;
extract_erlang_code_blocks({_Tag, _Attrs, Content}) ->
    extract_erlang_code_blocks(Content);
extract_erlang_code_blocks(_Other) ->
    [].

ensure_skipped_tests([moduledoc | T], #docs_v1{ module_doc = MD } = DocsV1) ->
    case is_map(MD) of
        true ->
            ensure_skipped_tests(T, DocsV1);
        false ->
            io:format("Module doc entry not found for skip_tests option"),
            error({skipped_tests_mismatch, moduledoc})
    end;
ensure_skipped_tests([KFA | T], #docs_v1{ docs = Docs} = DocsV1) when is_tuple(KFA) ->
    case lists:keyfind(KFA, 1, Docs) of
        {_, _, _, Doc, _} when is_map(Doc) ->
            ensure_skipped_tests(T, DocsV1);
        _ ->
            io:format("Doc entry ~p not found for skip_tests option", [KFA]),
            error({skipped_tests_mismatch, KFA})
    end;
ensure_skipped_tests([], _) ->
    ok.

ensure_missing_tests(false, NoTests, _HasTests, _DocFuns) ->
    case NoTests of
        [] ->
            ok;
        _ ->
            NoTestsFmt = lists:sort([io_lib:format("  ~p/~p\n", [F,A]) ||
                                        {F,A} <- NoTests]),
            io:format("The following functions have no tests:~n~n~ts~n",
                      [NoTestsFmt]),
            {comment,
             lists:flatten(io_lib:format("~p functions lack tests", [length(NoTests)]))}
    end;
ensure_missing_tests(Expected0, NoTests, HasTests, DocFuns)
  when is_list(Expected0) ->
    Expected = lists:sort(Expected0),
    %% Functions that lack tests but aren't in the expected list.
    NewMissing = NoTests -- Expected,
    %% Functions in missing_tests that now have doctests (stale entries).
    Stale = [FA || FA <- Expected, lists:member(FA, HasTests)],
    %% Functions in missing_tests that aren't documented at all.
    Invalid = [FA || FA <- Expected, not lists:member(FA, DocFuns)],
    case {NewMissing, Stale, Invalid} of
        {[], [], []} ->
            ok;
        {_, _, _} ->
            [io:format("Functions missing doctests "
                                   "(add doctests or update missing_tests option):~n"
                                   "  ~p~n", [NewMissing]) || NewMissing =/= []],
                    [io:format("Functions in missing_tests that now have doctests "
                                   "(remove from missing_tests):~n"
                                   "  ~p~n", [Stale]) || Stale =/= []],

                    [io:format("Functions in missing_tests that are not documented "
                                   "(update missing_tests option):~n"
                                   "  ~p~n", [Invalid]) || Invalid =/= []],
            error({missing_tests_mismatch,
                   [{missing, NewMissing}, {stale, Stale}, {invalid, Invalid}]})
    end.

ensure_skipped_blocks(false, _Actual) ->
    ok;
ensure_skipped_blocks(Expected, Actual) when is_integer(Expected), Expected >= 0 ->
    case Actual of
        Expected ->
            ok;
        _ ->
            error({unexpected_skipped_blocks, Expected, Actual})
    end.

-define(RE_CAPTURE, ~B"(?:(?'line_number'[0-9]+)(?'prefix'>\s)|(?'prefix'\-module\())?(?'content'.*)").
-define(RE_OPTIONS, [{capture, [line_number, prefix, content], binary}, dupnames, unicode]).

run_test(Code, InitialBindings, Options) ->
    Lines = string:split(Code, "\n", all),
    case lists:dropwhile(fun(Line) ->
                           re:run(Line, ~B"^\s*(%.*)?$", [unicode]) =/= nomatch
                       end, Lines) of
        [] ->
            [];
        [FirstLine | _] = LinesAfterIntro ->
            case re:run(FirstLine, ?RE_CAPTURE, ?RE_OPTIONS) of
                {match, [_Line_Number, _Prefix = <<"> ">>, _Code]} ->
                    ReLines = [re:run(Line, ?RE_CAPTURE, ?RE_OPTIONS) || Line <- LinesAfterIntro],
                    Tests = inspect(parse_tests(ReLines, [], 1)),
                    check_prompt_numbers(Tests),
                    _ = lists:foldl(fun(Test, Bindings) ->
                                            try run_tests(Test, Bindings, Options)
                                            catch throw:{error, Error} ->
                                                    throw({error, Error#{ test => Test}})
                                            end
                                    end, InitialBindings, Tests),
                    [ok];
                {match, [_Line_Number, _Prefix = <<"-module(">>, ModContent]} ->
                    [ModName | _] = binary:split(ModContent, [<<")">>]),
                    [compile_string(Code, ModName, Options#options.compile_options)];
                _ ->
                    []
            end
    end.

compile_string(Code, ModName, CompileOptions) ->
    FileName = unicode:characters_to_list(ModName) ++ ".erl",
    case compile:string(Code, [binary, return_errors, {source, FileName} | CompileOptions]) of
        {ok, Module, Binary} ->
            {module, Module} = code:load_binary(Module, FileName, Binary),
            ok;
        {error, Errors, Warnings} ->
            Messages = [begin [{_, M}] = sys_messages:format_messages(File, "", Msgs, []), M end || {File, Msgs} <- Errors ++ Warnings],
            throw({error,#{ message => Messages, context => Code}})
    end.

check_prompt_numbers(Tests) ->
    check_prompt_numbers(Tests, 1).

check_prompt_numbers([], _Expected) ->
    ok;
check_prompt_numbers([{test, LineNumber, _, _} = Test | T], Expected) ->
    case binary_to_integer(LineNumber) of
        Expected ->
            check_prompt_numbers(T, Expected + 1);
        Actual ->
            Message = io_lib:format("Bad prompt number ~p; expected ~p",
                                    [Actual,Expected]),
            throw({error,#{ message => Message, test => Test }})
    end.

parse_tests([], [], _) ->
    [];
parse_tests([], Cmd, No) ->
    [{test, No, lists:join($\n, lists:reverse(Cmd)), "_"}];
parse_tests([{match, [<<>>, <<>>, <<>>]}], Cmd, No) ->
    parse_tests([], Cmd, No);
parse_tests([{match, [<<>>, <<>>, <<>>]} | T], Cmd, No) ->
    parse_tests(T, [<<>> | Cmd], No);
parse_tests([{match, [PromptNo, <<"> ">>, NewCmd]} | T], [], _) ->
    parse_tests(T, [NewCmd], PromptNo);
parse_tests([{match, [PromptNo, <<"> ">>, NewCmd]} | T], Cmd, No) ->
    [{test, No, lists:join($\n, lists:reverse(Cmd)), "_"} | parse_tests(T, [NewCmd], PromptNo)];
parse_tests([{match, [<<>>, <<>>, <<FirstChar, _/binary>> = More]} | T], Acc, No)
        when FirstChar =:= $\s; FirstChar =:= $% ->
    parse_tests(T, [More | Acc], No);
parse_tests([{match, [<<>>, <<>>, NewMatch]} | T], Cmd, No) ->
    {Match, Rest} = parse_match(T, [NewMatch]),
    [{test, No, lists:join($\n, lists:reverse(Cmd)),
      lists:join($\n, lists:reverse(Match))} | parse_tests(Rest, [], No)].

parse_match([{match, [<<>>, <<>>, <<>>]}], Acc) ->
    parse_match([], Acc);
parse_match([{match, [<<>>, <<>>, More]} | T], Acc) ->
    parse_match(T, [More | Acc]);
parse_match(Rest, Acc) ->
    {Acc, Rest}.

run_tests({test, _Index, Test0, Match0}, Bindings, Options) ->
    Test1 = unicode:characters_to_list(Test0),
    Test = string:trim(Test1),
    case Match0 of
        [<<"** ", _/binary>> | _] ->
            Match = unicode:characters_to_list(Match0),
            run_failing(Test, Match, Bindings, Options);
        _ ->
            run_successful(Test, Match0, Bindings, Options)
    end.

run_successful(Test, Match, Bindings, Options) ->
    verbose_log(Options, "Running: ~ts = ~ts", [Match, Test]),
    Ast = parse_exprs(Test, Match),
    try
        {value, _Res, NewBindings} = inspect(erl_eval:exprs(Ast, Bindings)),
        NewBindings
    catch C:R:ST ->
            throw({error,#{ message => format_exception(C, R, ST) }})
    end.

run_failing(Test, Match, Bindings, Options) ->
    verbose_log(Options, "Running: ~ts", [Test]),
    Ast = parse_exprs(Test, "_"),
    try inspect(erl_eval:exprs(Ast, Bindings)) of
        {value, Res, _} ->
            Message = io_lib:format("Expected failure got ~ts",
                                    [Res]),
            throw({error,#{ message => Message, context => Match}})
    catch C:R:ST ->
            Actual = format_exception(C, R, ST),
            FailMatch = strip_comments(Match),
            case string:prefix(Actual,  FailMatch) of
                nomatch ->
                    Message = io_lib:format("Failure did not match:~n~w~n~n~w~n",
                                            [Actual, FailMatch]),
                    throw({error,#{ message => Message, context => FailMatch}});
                _ ->
                    Bindings
            end
    end.

strip_comments(Match) ->
    Lines = string:split(Match, "\n", all),
    NonCommentLines = [Line || Line <- Lines,
                        re:run(Line, ~B"^\s*%.*$", [unicode]) =:= nomatch],
    lists:join($\n, NonCommentLines).

%% As we allow <0.1.0> style matches we first need to check
%% if the match actually is a valid pattern. So we parse and
%% evaluate it first, and if that fails we convert it to a literal
%% and try again.
parse_exprs(Test0, Match0) ->

    %% First we check that we can parse the test correctly.
    %% This will throw if we fail.
    TestExprs = try_parse_exprs(Test0),

    %% Now we know that the test is parseable, we try to parse the match.
    try try_parse_match_exprs(Match0 ++ "\n = 1.") of
        MatchAst ->
            Match =
                try erl_eval:exprs(MatchAst, #{}) of
                    {value, _Res, _} ->
                        Match0
                catch
                    error:_ ->
                        maybe_convert_to_literal(Match0)
                end,
            
            %% Both the test and the match can end in a comment,
            %% so we parse each independently and then combine them
            [{match,Anno,LHS,_}|MaybeLiteral] = lists:reverse(try_parse_match_exprs(Match ++ "\n = 1.")),
            MaybeLiteral ++ [{match,Anno,LHS,{block,Anno,TestExprs}}]
    catch throw:_ ->
            try_parse_match_exprs(Match0 ++ "\n = 1.")
    end.

try_parse_match_exprs(Str) ->
    rewrite_match_ast(try_parse_exprs(Str)).
try_parse_exprs(Str) ->
    Cmd = lists:flatten(unicode:characters_to_list(Str)),
    maybe
        {ok, T, _} ?= erl_scan:string(Cmd, {1,1}, [text]),
        RewrittenToks = rewrite_tokens(T),
        {ok, Ast} ?= inspect(erl_eval:extended_parse_exprs(RewrittenToks)),
        Ast
    else
        {error, {{Line, _},_Mod,_Reason} = Err, _} ->
            [{_, Message}] = sys_messages:format_messages("", "", [Err], []),
            throw({error,#{ message => string:trim(Message, leading, ":"), line => Line}});
        {error, {{Line,_},_Mod,_Reason} = Err} ->
            [{_, Message}] = sys_messages:format_messages("", "", [Err], []),
            throw({error,#{ message => string:trim(Message, leading, ":"), line => Line}})
    end.

%% We rewrite ...>> to _/binary>> to match shell syntax better
rewrite_tokens([{'...', L1}, {'>>', L2} | T]) ->
    rewrite_tokens([{var, L1, '_'}, {'/', L1}, {atom, L1, binary}, {'>>', L2} | T]);
%% We rewrite , ... ] to | _ ] to match shell syntax better
rewrite_tokens([{',', L1}, {'...', L2}, {']', L3} | T]) ->
    rewrite_tokens([{'|', L1}, {var, L2, '_'}, {']', L3} | T]);
%% We rewrite ... to _ to match shell syntax better
rewrite_tokens([{'...', L} | T]) ->
    rewrite_tokens([{var, L, '_'} | T]);
rewrite_tokens([H | T]) ->
    [H | rewrite_tokens(T)];
rewrite_tokens([]) ->
    [].

%% Rewrite the AST to convert map field associations to exact matches on the LHS
rewrite_match_ast([{match, Ann, LHS, RHS}]) ->

    RewrittenLHS =
      erl_syntax_lib:map(
        fun(Tree) ->
                case erl_syntax:type(Tree) of
                    map_field_assoc ->
                        Name = erl_syntax:map_field_assoc_name(Tree),
                        Value = erl_syntax:map_field_assoc_value(Tree),
                        erl_syntax:map_field_exact(Name, Value);
                    _Else ->
                        Tree
                end
        end, LHS),
    [{match, Ann, erl_syntax:revert(RewrittenLHS), RHS}];
rewrite_match_ast([{match, _, _, _} = Match | Rest]) ->
    [Match | rewrite_match_ast(Rest)].

%% We do a little dance here in order to allow refs, pids and ports
%% to be matched as literals, since the shell prints them as literals
%% but erl_eval doesn't accept them as such.
%%
%% We traverse the AST and hoist only the literal-producing calls
%% (list_to_pid, list_to_port, list_to_ref) into variable bindings,
%% leaving the rest of the match pattern intact.
%%
%% For example, {ok, <0.1.0>} becomes: _L1 = <0.1.0>, {ok, _L1}
maybe_convert_to_literal(Match0) ->
    Match = unicode:characters_to_list(Match0),
    maybe
        {ok, Toks, _} ?= erl_scan:string(Match ++ ".", 0, [text]),
        RewrittenToks = rewrite_tokens(Toks),
        {ok, [Expr0]} ?= erl_eval:extended_parse_exprs(RewrittenToks),
        {Expr1, Acc} = hoist_literal_calls(Expr0),
        true ?= Acc =/= [],
        Expr = erl_syntax:revert(Expr1),
        Prefix = lists:join(", ",
            [lists:flatten(io_lib:format("~s = ~p", [V, L]))
             || {V, L} <- lists:reverse(Acc)]),
        ExprStr = lists:flatten(erl_pp:expr(Expr)),
        lists:flatten([Prefix, ", ", ExprStr])
    else
        _ ->
            Match
    end.

hoist_literal_calls(Expr) ->
    erl_syntax_lib:mapfold(
      fun(Tree, Acc) ->
              case is_literal_producing_call(Tree) of
                  {yes, Value} ->
                      VarName = "_L" ++ integer_to_list(
                                          erlang:unique_integer([positive])),
                      Var = erl_syntax:variable(list_to_atom(VarName)),
                      {Var, [{VarName, Value} | Acc]};
                  no ->
                      {Tree, Acc}
              end
      end, [], Expr).

is_literal_producing_call(Tree) ->
    case erl_syntax:type(Tree) of
        application ->
            Op = erl_syntax:application_operator(Tree),
            case erl_syntax:type(Op) of
                module_qualifier ->
                    Mod = erl_syntax:atom_value(
                            erl_syntax:module_qualifier_argument(Op)),
                    Fun = erl_syntax:atom_value(
                            erl_syntax:module_qualifier_body(Op)),
                    case {Mod, Fun} of
                        {erlang, F} when F =:= list_to_pid;
                                         F =:= list_to_port;
                                         F =:= list_to_ref ->
                            [Arg] = erl_syntax:application_arguments(Tree),
                            S = erl_syntax:string_value(Arg),
                            try {yes, erlang:F(S)} catch _:_ -> no end;
                        _ ->
                            no
                    end;
                _ ->
                    no
            end;
        implicit_fun ->
            %% Handles fun M:F/A references from shell syntax like M.F/A
            Reverted = erl_syntax:revert(Tree),
            {value, Val, _} = erl_eval:exprs([Reverted], #{}),
            {yes, Val};
        _ ->
            no
    end.

format_exception(Class, Reason, [Top,Next|_]) ->
    Stacktrace = [clean_stacktrace_item(Item) || Item <- [Top,Next]],
    Tag = "** ",
    I = iolist_size(Tag) + 1,
    PF = fun pp/2,
    SF = fun(Mod, _, _) -> Mod =:= erl_eval end,
    Enc = unicode,
    Str = erl_error:format_exception(I, Class, Reason, Stacktrace, SF, PF, Enc),
    Tag ++ string:trim(lists:flatten(Str), trailing).

clean_stacktrace_item({M,F,A,Info0}) ->
    Info = lists:keydelete(line, 1, Info0),
    {M,F,A,Info}.

pp(V, I) ->
    D = 30,
    io_lib_pretty:print(V, [{column, I}, {line_length, 120},
                            {depth, D}, {line_max_chars, 100},
                            {strings, true},
                            {encoding, unicode}]).

inspect(Term) ->
    %% Uncomment for debugging
    %% io:format("~tp~n",[Term]),
    Term.