Differentiate “need manual review” status in "snapcraft upload" from errors

[Jehan]

Check existing issues

• I've verified that this request isn't described by any existing issues.

Request

The snapcraft upload CLI tool should return a different return value when it ends with “need manual review” status, compared to actual errors.

The problem it solves

A problem we encountered on GIMP's release pipeline is that this "need manual review" status broke our pipeline (as it apparently didn’t return 0). As a consequence:

1. it stopped after uploading one out of the 2 archs;
2. a few other steps in the release procedure were made more annoying (because for failed pipelines, Gitlab won’t generate some “easy” redirection URLs used by our scripts to publish other binaries to our download infra).

We still want to see real failures, but “need manual review” is not actually a failure, right? We just wait a bit to get the manual validation. So I’d like for our pipeline to still pass as success in this “manual review” case, which is why I wish to be able to react conditionally on the return value.

P.S.: as initially reported on the snapcraft forums.

[mr-cal]

We have 2 potential solutions and they aren't mutually exclusive:

1. Return different error codes for snapcraft upload responses #6081
2. Add a --format json that returns machine parsable output.
   • This will take a bit of design on how to structure the json. Ideally, it would be consistent across commands and craft apps.

[Jehan]

Thanks @mr-cal. The return codes are a bit easier to process (though there exists some CLI tools to parse json, this requires adding these tools to a CI environment and using these), assuming of course these are properly documented.

Though as you say, this is not exclusive. If the JSON output allows to pass more information about the process (what went wrong or not, or what is still in question, like a manual review!), calling code could either just care about the return value if they want to do something simple, then if they want to do some more advanced processing, they could additionally check the JSON output.

So I would advise to start with the return values change anyway as having a process return different values for a bit more status-discrimination (not just success/failure) is quite a common practice. Then for sure, you may add fancier output option for even better status reporting.