Fail to run individual CT suite with data directory, but running all suites works fine

[AntoineGagne]

Environment

• Add the result of rebar3 report to your message:

Rebar3 report
 version 3.16.1
 generated at 2021-07-27T14:46:01+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: ct
Entered as:
  ct
-----------------
Operating System: x86_64-apple-darwin19.6.0
ERTS: Erlang/OTP 24 [erts-12.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]
Root Directory: /Users/a.gagne/.kerlenv/24.0
Library directory: /Users/a.gagne/.kerlenv/24.0/lib
-----------------
Loaded Applications:
bbmustache: 1.10.0
certifi: 2.6.1
cf: 0.3.1
common_test: 1.20.3
compiler: 8.0
crypto: 5.0
cth_readable: 1.5.1
dialyzer: 4.4
edoc: 1.0
erlware_commons: 1.5.0
eunit: 2.6.1
eunit_formatters: 0.5.0
getopt: 1.0.1
inets: 7.4
kernel: 8.0
providers: 1.8.1
public_key: 1.11
relx: 4.4.0
sasl: 4.1
snmp: 5.9
ssl_verify_fun: 1.1.6
stdlib: 3.15
syntax_tools: 2.6
tools: 3.5

-----------------
Escript path: /Users/a.gagne/.local/bin/rebar3
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report repos shell state tar tree unlock update upgrade upgrade upgrade version xref

Current behaviour

When I run a particular test suite, its corresponding data directory can not be found. However, when I run the entire test suites, the data directory is found and the test run correctly. This is the error that I get:

• rebar3 ct --suite=test/nested/example/example_SUITE:

%%% example_SUITE ==> init_per_suite: FAILED
%%% example_SUITE ==> {{badmatch,{error,enoent}},
 [{example_SUITE,init_per_suite,1,
                 [{file,"/Users/a.gagne/Documents/programmation/erlang/common-test-fun/test/nested/example/example_SUITE.erl"},
                  {line,19}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1380}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1224}]}]}

rebar3 ct:

===> Verifying dependencies...
===> Analyzing applications...
===> Compiling common_test_fun
===> Running Common Test suites...
%%% example_SUITE: .
All 1 tests passed.

Is there anything wrong with what I am doing?

---

Here are some information about my setup:

• Project structure:

.
├── LICENSE
├── README.md
├── rebar.config
├── src
│   ├── common_test_fun.app.src
│   ├── common_test_fun_app.erl
│   └── common_test_fun_sup.erl
└── test
    ├── example_SUITE_data
    │   └── foo
    └── nested
        └── example
            └── example_SUITE.erl

5 directories, 10 files

• rebar.config:

{erl_opts, [debug_info]}.

{deps, []}.

{profiles, [
    {test, [
        {deps, [
            {proper, {git, "https://github.com/manopapad/proper.git", {branch, "master"}}},
            {meck, {git, "https://github.com/eproxus/meck.git", {tag, "0.9.2"}}},
            {unite, {git, "git://github.com/eproxus/unite.git", {tag, "0.3.1"}}}
        ]},
        {erl_opts, [export_all, nowarn_export_all, debug_info]},
        {eunit_opts, [no_tty, {report, {unite_compact, []}}]},
        {extra_src_dirs, [
            {"test", [{recursive, true}]}
        ]}
    ]}
]}.

• example_SUITE.erl:

-module(example_SUITE).

-include_lib("common_test/include/ct.hrl").
-include_lib("eunit/include/eunit.hrl").

-compile(nowarn_export_all).
-compile(export_all).

all() ->
    [
        can_read_file
    ].

init_per_suite(Config) ->
    DataDirectory = ?config(data_dir, Config),
    FileName = filename:join(DataDirectory, "foo"),
    {ok, Binary} = file:read_file(FileName),
    [{binary, Binary} | Config].

end_per_suite(Config) ->
    Config.

%%%===================================================================
%%% Test cases
%%%===================================================================

can_read_file() ->
    [{doc, "When reading files from data directory, then reads file."}].

can_read_file(Config) ->
    ?assert(byte_size(?config(binary, Config)) > 0).

%%%===================================================================
%%% Internal functions
%%%===================================================================

Expected behaviour

I would expect the data directory to be in a single place and the suites to be runnable individually.

[ferd]

Hm, my guess is that there's possibly something about how modules can be found recursively. If I try to print out the path it extracts, it gives _build/test/lib/chk/test/nested/example/example_SUITE_data/ and _build/test/lib/chk/test/nested/example/example_SUITE_data/ is precisely the file that exists in standard cases, but not yours.

The issue you're getting is that the <name>_SUITE.erl file and <name>_SUITE_data/ directory should both be at the same level in directories when there's no recursive search. When using the form you use, where the directory and the file are not in the same path at all, rebar3 just goes and assumes that the longest path possible passed to --suite is the test directory and won't locate the data dir.

I believe you can work around that by combining arguments to properly locate the rebar3 path about what is the test directory. Try rebar3 ct --dir test/ --suite example_SUITE and see if that works. This should properly locate the data directory as living underneath test/ and still let the recursive search find the proper test suite.

[AntoineGagne]

The suggested fix of using the form rebar3 ct --dir test/ --suite example_SUITE seems to work. Thanks!

Quick note: At first, the data directory was at the same level as the test suite. However, the tests would not pass when running only rebar3 ct which is why I put the data directory back to the test/ directory level. Although, running the rebar3 ct --suite=test/nested/example/example_SUITE did work with this configuration (probably because of the reason you just gave me).

[ferd]

yeah, I think that's a mismatch between the compiler supporting recursive directories and Common Test not actually doing that, and finding the clash. When we move things to build for common test, we put the .beam files at the top-level of the final directory and the framework can find things fine.

I don't know we can actually fix this other than patching OTP itself to support similar recursion, and I don't know if that's the best of ideas. Funnily enough it won't show aside from data directories, so the --dir test --suite <name> trick might be the only way forwards :/