Automatically tag scenario depending on their steps

[btvanhooser]

I currently have a use case where the best way to maintain my test suite (~7000 tests all split up between multiple projects) and some of those tests require pre-conditions to ensure they are ready to run.

An example to show the use case: we are using MockServer for our UI tests but only about 400 of them need this case and they require setting some default headers for only those cases. Across my team it's very easy to just forget adding that @mockserver tag on a random test and then it'd fail. We elected to make our own extension project to hook into the MSBuild target after Reqnroll generates the *.feature.cs files so we can add the Category attributes after parsing them out. The code was functional but dense and it seems to have broken recently when attempting to update from 2.0.3 to 3.2.0. Instead of trying to maintain that pattern, I was electing to try and follow Reqnroll patterns by making a generator plugin.

Ultimately I used the pattern on the ExternalData plugin to guide me and was successful on the pattern, but the performance is... lacking. A build that takes 30-40 seconds without the plugin now takes about 4 to 5 minutes. I'm not too familiar with the ins and outs of Reqnroll plugins and where the best spot to do my checks and modifications would be, so I'm looking for advice here. Admittedly, it makes sense that it's slow because the pattern on the ExternalData plugin is to let the parsing happen normally for getting the ReqnrollDocument instance, but then it more-or-less loops through all documents/features/scenarios/rules to check for qualifying steps and marks them accordingly by recreating the Scenario or ScenarioOutline with the tags concatenated. When doing some time tracking, it seems like the normal parsing is taking about 0.0001 - 0.0002 seconds per document and then the iterative pattern with adjustments can take anywhere from .05 - 1+ seconds per document.

Is there a recommendation to dive deeper into overriding the parser as a way to avoid the 2nd pass on the documents, or am I too low already and this would be better higher up? I tried overriding the NUnit generator implementation with this logic and that was pretty fast but ultimately only added the category attribute without adjusting anything else so that had a mismatch between NUnit and Reqnroll since the generated code that passes in the tags into the scenario creation methods never got adjusted.

I need some good resolution if I'm gonna be able to update and I'd really like to update if possible. Open to ideas here.

[clrudolphi]

Would it be possible to share more detail of your implementation? I'm not sure I'm understanding clearly the objective.
It sounds like you would like to ensure that certain features or scenarios are marked with a custom tag. Is that tag then used to drive a hook with a custom scope?

What indicators are you examining within a Scenario to identify those that need this tag?

If you can share the plug-in implementation, I can take a look and provide feedback.

[btvanhooser]

Unfortunately my company has legalities around sharing code that's committed to any internal source repositories, so I can't explicitly drop the direct implementation, but I can point you to the source inspiration from the Reqnroll repo and describe what I've done.

Based it on these files here:

• https://github.com/reqnroll/Reqnroll/tree/main/Plugins/Reqnroll.ExternalData/Reqnroll.ExternalData.ReqnrollPlugin/Transformation
• https://github.com/reqnroll/Reqnroll/blob/main/Plugins/Reqnroll.ExternalData/Reqnroll.ExternalData.ReqnrollPlugin/ExternalDataTestGenerator.cs

The one file the is different is the "IncludeExternalDataTransformation" as I'm not looking to add scenarios, but rather adjust those scenarios with new tags. That part is determined by configs that can be provided mapping what step wording to look for and what tag to apply, like so:

[
  ...
  {
    "tag": "MyTag",
    "conditions": [
      "some substring from a step"
    ]
  }
  ...
]

That config is loaded and then initialized by transforming those strings into compiled Regex expressions. Then in that "IncludeExternalDataTransformation" file I would be iterating through the steps for either the Scenario or ScenarioOutline at that point to see if any of them match the provided Regex. To help performance here, I include an in-memory Dictionary cache to map the step I checked with what the resulting matching configs were.

And that's essentially it. On a small scale it works great, but on a much larger scale it appears to scale very poorly with the performance numbers I provided. I only just started my attempt this week so honestly the implementation only differs by that much from the linked files. I can try to make "similar" code to highlight the point but my company is massive so I'm trying to not step on toes where I can

[clrudolphi]

I completely understand the constraint on not sharing proprietary code.

A few thoughts, hopefully you find them constructive.

Using the ExternalData plugin as the basis for your solution is feasible, but with a few caveats. Since it shims itself in to replace the built-in TestGenerator, it cannot be composed with other plugins that also want to do the same thing. That is, if your plugin replaces TestGenerator then you may never use the ExternalDataPlugin or anything else like it.

The execution cost of running your transformation against the parsed AST using the techniques shown in the ExternalData plugin are, by themselves, not all that expensive. The most expensive part is the object allocation of crafting replacement Scenarios. If you determine that a Scenario does not need your tag added, then follow the pattern of returning null from GetTransformedScenario() as that will avoid the cost of replacing that part of the object graph.

If you were very concerned about replacing all of those Scenario objects with new ones, then you might experiment with poking new values into the Tags collection of the Scenario AST object. By contract those are IEnumerable, but my guess is that they are either arrays or lists under the hood and you might be able to extend them with a new value. Yes, possibly breaking compatibility with some future change, but you gotta do what you gotta do.

My hunch is that the most expensive part of your algorithm is executing all of those regex expressions over your feature corpus.

A couple of alternatives to consider:

• Turn this into a build step that runs outside of Reqnroll as a job that finds only those feature files that have changed (since the last run) and then parse-rewrite those that need it. That would avoid running the regexes against all 7000 scenarios. Could this be run as a condition of checking in the feature file? Could this be run on a schedule (like once per day) to find those features for which the authors have forgotten to place the tag?
• At the other end of the spectrum, drop the idea of a tag and build a BeforeStep hook that matches current step text against your list of triggers and then fires your logic if it hasn't previously been run for the current Scenario.

If it were me, I'd pursue the idea of the check-in time job.

In the long-run though, think very hard about whether you can sustain maintaining another mapping artifact. As the text of steps keeps changing you'll have to keep updating that mapping. You'll have just another place for someone to forget to update something.

[btvanhooser]

All fantastic advice and I greatly appreciate it

The execution cost of running your transformation against the parsed AST using the techniques shown in the ExternalData plugin are, by themselves, not all that expensive. The most expensive part is the object allocation of crafting replacement Scenarios. If you determine that a Scenario does not need your tag added, then follow the pattern of returning null from GetTransformedScenario() as that will avoid the cost of replacing that part of the object graph.

Yes, I forgot to mention but I am ensuring to return null if I find no matches here.

If you were very concerned about replacing all of those Scenario objects with new ones, then you might experiment with poking new values into the Tags collection of the Scenario AST object. By contract those are IEnumerable, but my guess is that they are either arrays or lists under the hood and you might be able to extend them with a new value. Yes, possibly breaking compatibility with some future change, but you gotta do what you gotta do.

When I originally attempted this it was not letting me add, but I didn't even think about casting to their runtime type. I'll attempt this first as this should involve the least amount of change.

My hunch is that the most expensive part of your algorithm is executing all of those regex expressions over your feature corpus.

I was generic with my time checking, but I'll be more specific in my next adjustments to see if it's the scenario creation or the regex matching that's adding more time. I was ensuring that I create them as compiled regex which I figured would speed things up but perhaps that's still not enough of a savings.

Turn this into a build step that runs outside of Reqnroll as a job that finds only those feature files that have changed (since the last run) and then parse-rewrite those that need it. That would avoid running the regexes against all 7000 scenarios. Could this be run as a condition of checking in the feature file? Could this be run on a schedule (like once per day) to find those features for which the authors have forgotten to place the tag?

This could be worthwhile, so if the above doesn't bring the performance into a tolerable range for my consumers then I may have take this route.

At the other end of the spectrum, drop the idea of a tag and build a BeforeStep hook that matches current step text against your list of triggers and then fires your logic if it hasn't previously been run for the current Scenario.

Alright this is just smart and I'm sad I didn't think of that. Admittedly the other benefit of the tag approach was that we have a custom reporting site that we use to help search for tests and provide a history. The test searching side includes tag searching/grouping so being able to see how many test use MockServer is useful when we want to see the impact of such tooling on our test base.

In the long-run though, think very hard about whether you can sustain maintaining another mapping artifact. As the text of steps keeps changing you'll have to keep updating that mapping. You'll have just another place for someone to forget to update something.

Solid point. In this migration I already found and issue with the original solution and some misidentifications there so I could be setting myself up for the same gaps later. I think just because the team is used to this, I will try to clean it up enough to keep it alive, but I'll be considering something more dynamic as I go so that maybe we can have something that avoids this outright later on.

[btvanhooser]

Alright, ultimately dropping the re-creation of the Scenarios and re-assignment was a good boost, but honestly the biggest boost was just dropping regex. Just a basic substring search dropped the overall rebuild-time to 1.5 minutes and the build time (no clean) to 24 seconds which almost matches the original build time without the plugin. So huge win! Appreciate the call outs!

The only downside to this as a future note is that Tags is get-only and the IEnumerable is actually a perfectly sized array, so I did need to use reflection to add and re-assign which was unfortunate... but clearly the combination of that and the above out performed the previous pattern! So yay.

I leaned harder into the optimization route instead of the step matching because I remembered that skipped tests won't reach the step so the tagging would be variable. We run against 4 environments and 23 domains so there are tests that don't make sense to run if you have a certain set of configurations needed. To keep the reporting benefit I really wanted to make the tagging work.

Your advice was paramount though so I absolutely appreciate it. Thank you again @clrudolphi

[Code-Grump]

It looks like you've already got the answers you're looking for, but I wondered if the problem might be down to not preserving Regex instances between invocations. .NET's regular expressions implementation is very fast, but there's significant overhead to parsing the expression and compiling the matcher. If you find yourself needing to return to regular expressions, you may need to take a look at ways to re-use expression objects or to pre-compile them.

[btvanhooser]

Definitely was accounted for! I had a centralized static config that was pre-compiled. We'd loop through the provided config as described in the other thread and initialize them with the RegexOptions.Compiled flag passed in. Should have been fast enough, but it didn't seem to make the difference here. I agree that it's always been extremely fast for me in .NET as well but here it seems to not have been the right choice. Open to other paths to recapture the ability to use Regex, but losing it isn't backbreaking for me here.