Allow FAILED status for cal steps

[stscijgbot-jp]

Issue JP-4103 was created on JIRA by Melanie Clarke:

JP-3673 will introduce a possible "FAILED" status for the background subtraction step.  Previously, we have only allowed COMPLETE and SKIPPED values for the meta.cal_step attributes.  We should consider if there are other steps that should report FAILED instead of SKIPPED in some circumstances.  We should also update the related utility function that handles the status keywords to expect 3 possible values (jwst.stpipe.utilities.record_step_status).

[stscijgbot-jp]

Comment by David Law on JIRA:

Seems like a useful thing to be able to report back more generally, for steps in which some kind of failure may impact data quality but not need to halt pipeline processing.

[braingram]

FWIW roman uses "FAILED" for the same purpose so this change would be consistent with that mission:
https://github.com/spacetelescope/rad/blob/dffad6de9cb60067d2a993f1d28c504fb26fe1c7/latest/cal_step_flag.yaml#L13

[stscijirahub]

This issue is tracked on JIRA as JP-4103.

[stscijirahub]

Comment by Melanie Clarke on JIRA:

David Law Tyler Pauly - JP-4334 is requesting a "FAILED" status for clean_flicker_noise under some circumstances, so maybe now is a good time to get this more generally implemented.

Any reason I shouldn't just add "FAILED" to the allowed values for all cal_step statuses?  We can implement when it gets set for any particular step later, as needed.

[stscijirahub]

Comment by Melanie Clarke on JIRA:

And should we also allow "N/A" and "INCOMPLETE", as Roman does?  Those might also be useful in some situations.

[stscijirahub]

Comment by Tyler Pauly on JIRA:

I'm on board with adding FAILED as a valid value for all steps. The other two do not immediately seem as useful - I'd vote to wait on their addition until we have a clear use for them.

[stscijirahub]

Comment by Melanie Clarke on JIRA:

Sounds good. I'll go ahead and make this update, then.  Let me know if you object, David Law!

[stscijirahub]

Comment by David Law on JIRA:

Sounds good to me.

[stscijirahub]

Comment by Melanie Clarke on JIRA:

FAILED status added as an option in spacetelescope/stdatamodels#722

I have a follow-up PR to set the status to FAILED by default in some situations here: #10497