Skip to content

Add integration tests for HyP3 API job validation #2740

@jtherrmann

Description

@jtherrmann

Jira: https://asfdaac.atlassian.net/browse/TOOL-3663

Note: The above link is accessible only to members of ASF.


We've historically done most/all of our integration testing for HyP3 API job validators in test/sandbox deployments and then summarized the results in PR/issue descriptions. We've treated validator functions as modular and independent of job type and typically only add comprehensive tests for the validator functions themselves, but not for the actual job types. The problems with this approach include:

  1. It obscures the fact that when you're adding/modifying validator functions, you're also adding/modifying API behavior; e.g. you might change how a validator function raises exceptions, even if it's just revising an error message, without thinking about the fact that you're changing the behavior of the corresponding API responses, which is a breaking change.

  2. We're not testing the order in which validators are executed for a given job type. It might be important to call one validator before another (e.g. if the second validator is slower, so you only want it to run if the first one succeeds). I believe they run in the same order that they're listed in the job spec, but we don't test this for all of our job types.

  3. We're mocking the input data to the validator functions (job dict and list of CMR records), so our tests won't help us catch bugs that occur before the validator function is called (e.g. during CMR querying/filtering). See Validate granules using UMM_JSON metadata from CMR #2737 for why this is important.

I think we should start including comprehensive integration tests for API job validation going forward.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Jira TaskCreate a Jira Task for this issue

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions