Handling feature flags in test runs

[fionna-at-ocuco]

Hi all

I'm curious to know how others are handling feature flags when running tests with Reqnroll.

My initial plan is:

• tag affected scenarios with name of flag and required flag status
• BeforeTestRun - get list of all defined feature flags from FlagSmith
• BeforeScenario - if the scenario tags contain a flag name that had been returned from FlagSmith, check status of the flag in our System Under Test (and cache result, so we aren't constantly checking for the same flags)
  • if the scenario should be run (e.g. tag is flag-name-ON and that flag is indeed on in this environment), continue with test execution
  • if it should not be run, stop test execution

One of the reasons that I'm choosing this approach is that our feature flags are not especially long-lived, so having steps defined to check the flag status will mean more complicated test maintenance when a particular feature flag will retire.

It is mainly the last point that I'm having problems with. I can add in Assert.Inconclusive in the BeforeScenario hook, but then all of the other steps are logged as if they errored out. It's not the end of the world, but I can forsee having to explain what has happened over and over again.

I found Skipping Scenarios in the documentation, which seems exactly what I'm looking for, but that will only run in steps, not in hooks. It does make me think that people are using steps to check (or toggle) flag status though.

(We won't be toggling flag status during test run, as we have a lot of parallel executions that will be impacted)

[gasparnagy]

I think I am doing a very similar approach to yours. The only addition I have is that I have some scenarios that need to be run for "all except flag1", so I also have tags like @notSupported:flag1.

It is mainly the last point that I'm having problems with. I can add in Assert.Inconclusive in the BeforeScenario hook, but then all of the other steps are logged as if they errored out. It's not the end of the world, but I can forsee having to explain what has happened over and over again.

I found Skipping Scenarios in the documentation, which seems exactly what I'm looking for, but that will only run in steps, not in hooks. It does make me think that people are using steps to check (or toggle) flag status though.

Could you please elaborate on this a bit more in detail? Despite what the doc is saying (outdated?), I think this should generally work in hooks as well, at least I use it in a BeforeScenario hook in my project.

[fionna-at-ocuco]

The only addition I have is that I have some scenarios that need to be run for "all except flag1", so I also have tags like @notsupported:flag1.

Yup - I have @flag-name-ON and @flag-name-OFF for this

Could you please elaborate on this a bit more in detail? Despite what the doc is saying (outdated?), I think this should generally work in hooks as well, at least I use it in a BeforeScenario hook in my project.

Aha! I had taken the documentation at face value :) Thanks for the heads-up, everything is working for me now

[gasparnagy]

Fixed the docs.