-
Notifications
You must be signed in to change notification settings - Fork 12
Description
Jira: https://asfdaac.atlassian.net/browse/TOOL-3663
Note: The above link is accessible only to members of ASF.
We've historically done most/all of our integration testing for HyP3 API job validators in test/sandbox deployments and then summarized the results in PR/issue descriptions. We've treated validator functions as modular and independent of job type and typically only add comprehensive tests for the validator functions themselves, but not for the actual job types. The problems with this approach include:
-
It obscures the fact that when you're adding/modifying validator functions, you're also adding/modifying API behavior; e.g. you might change how a validator function raises exceptions, even if it's just revising an error message, without thinking about the fact that you're changing the behavior of the corresponding API responses, which is a breaking change.
-
We're not testing the order in which validators are executed for a given job type. It might be important to call one validator before another (e.g. if the second validator is slower, so you only want it to run if the first one succeeds). I believe they run in the same order that they're listed in the job spec, but we don't test this for all of our job types.
-
We're mocking the input data to the validator functions (job dict and list of CMR records), so our tests won't help us catch bugs that occur before the validator function is called (e.g. during CMR querying/filtering). See Validate granules using UMM_JSON metadata from CMR #2737 for why this is important.
I think we should start including comprehensive integration tests for API job validation going forward.