rebar3 ct forces `TEST` macro

[qzhuyan]

rebar3 ct forces TEST macro, it is used for compiling both test suite and the application code.

I want to run the ct suite against the prod release build of the application. I could not find a way to do that.

I know TEST macro is used by eunit, but some other places in the app, may use it for wrapping 'test' as well.

Do you have any solution? thanks!

[ferd]

Nope, no solutions. rebar3 ct always forces the test profile to be added. You'd probably need to look at ct_run and other OTP commands to run your test suites independently from the build steps required to support them (which may also download build-time dependencies like mocking libraries and whatnot)

[qzhuyan]

thanks for the reply!

I am thinking maybe we could intro a rebar ct arg that do not compile the application code with TEST macro defined.

[ferd]

The ct task changes more than just the TEST macro (it also configures a COMMON_TEST macro), it reconfigures log handlers, loads configuration in slightly different ways, runs different macros in your code (eg. ct:pal gets overwritten in non-test code), and may keep test-related dependencies in your runtime path that wouldn't exist in a real prod releases that can hide some specific failures.

If what you are writing are integration tests against a production release of an application, I would probably encourage you to write a dedicated test application that is given commands or code to test from another application in a more hermetic way.

[qzhuyan]

thanks for the info!

If some ct functions need to be overridden to adpat to rebar3 integration it is totally fine.

Write another integration test app is doable but there are engineering overheads I think.

I am just thinking maybe it is doable with rebar3 ct with different profiles.

in every profile I could specifiy the macro explicitly I want to use.

default profile is test, TEST macro is defined, which is ok.

rebar3 ct will just be the same as rebar3 ct as test

if I want to run 'integration test against prod build', I run with defined prod profile

rebar3 ct as prod

if I want to run some tests that app has special behaviour just for test some specific code/conditions, I could choice the profile to run

rebar3 ct as whitebox

rebar3 ct as strict_assertions

My issue is for whichever the profile I use the TEST macro is always defined while rebar3 ct compiles the application code.