Enable synchronization of release candidate lifecycles #1239
Description
Is your feature request related to a problem? Please describe.
ASTM F3548-21 and F3411 recognize that interoperability testing is important -- that is, merely testing an individual USS in isolation is not sufficient to rigorously detect interoperability problems. Therefore, it prescribes a shared test environment in which a USS must provide a test instance with software that matches their production instance. This enables other USSs to test their release candidates for interoperability with the production software before releasing their candidates to production.
ASTM F3548-21 recognizes that it is important to test release candidates against the production versions of other USSs' software and therefore establishes GEN0305 (a USS's test-env instance must match the software version of its prod-env instance except when testing a release candidate). However, this can be a challenge when multiple USSs want to release a new software version at the same time. If they both promote a release candidate to the test environment simultaneously, neither one will have tested their release candidate against the production ecosystem software versions. This may be ok if the two USSs coordinate to promote their release candidates into production simultaneously following a successful test, but in the absence of such coordination, non-interoperable software may be released into production and/or test environments. For instance:
InterUSS's current GEN0305 automated test prevents this problem, but does not provide any means to resolve the underlying release race condition. This means that USSs racing to release new software must fall back to manual coordination in order to achieve successful releases gated on successful automated tests.
Describe the solution you'd like
In its role as providing synchronization for an ecosystem, it seems like the DSS could provide a simple "release candidate qualification" mutex to clients of the DSS pool. A USS wishing to qualify a release candidate would acquire this mutex (waiting until it could acquire the mutex). Then, it would perform all release activities including deploying its release candidate to the test environment, running an automated test, and promoting the candidate to production or rolling it back. Finally, the USS would release the mutex when finished with this round of its release activities.
Since the holder of this mutex would apply to the ecosystem served by a pool of DSS instances, it seems like an obvious location to find this information would be in the DSS pool information. We could add an aux endpoint to set the mutex holder, and add the current mutex holder to the information available on the pool information endpoint.
We could perhaps have two different scopes, something like:
interuss.pool_status.qualification_lock.manage: Allows the client to view the current mutex/lock ownership, acquire the lock for themselves only if it is not held by anyone, and release the lock if they hold it. We may need to decide if otherpool_statusinformation needs to be redacted for clients who provide just this scope.interuss.pool_status.qualification_lock.admin: Allows the client to manipulate the lock regardless of who holds it (e.g., to fix a lock stuck as acquired by a non-functional client).
Describe alternatives you've considered
The mutex endpoint could be separate from the aux API, but that seems unnecessary given that the concept seems to fit well with other pool information which is already available via the aux API.