Draft Artifact State doesn't behave as expected

[wolfchimneyrock]

We expected to be able to use the new Draft state introduced in #5330 in order to provide a mutable "dev" registry for our users, however as implemented this won't work for us:

Actual behavior

1. draft artifacts are assigned a fake random content hash, so that hash-based lookups don't work
2. GET /ids/... calls are explicitly blocked
3. content rules are explcitly not evaluated

this prevents us from exposing draft-mode to our users who may want to test deployments of their application without quickly burning through artifact version numbers:

1. producer applications that register their schema at runtime always register a new version content when using draft mode, even if a version with the same content already exists in the registry, due to the mangled contentHash. With distributed systems this quickly leads to a id# explosion, and breaks many users expectations that the newest version registered is actually the most recently updated schema.
2. consumer applications that fetch by contentId fail to see the version since the contentId lookup is explicitly blocked.

Expected behavior

• Draft versions should be created with their actual content and canonical sha256 hashes so that lookups by hash succeed and ifExists=FIND_OR_CREATE_VERSION returns the existing version instead of creating infinite new versions with the same content.
• I see there is a unique constraint on the content table contentHash. attempts to update a version's content to some content that already exists elsewhere would fail in order to maintain that uniqueness.
• GET /ids/... should successfully return draft state contents
• if the latest version of an artifact is in draft state, then any calls to create the artifact with ifExists=FIND_OR_CREATE_VERSION with some before unseen content could just mutate that latest draft version's content instead of creating a new version
• content rules should be evaluated when trying to create draft versions

maybe this different behavior could be behind a config? though I don't see any potential use with Draft mode as-is with fake hashes, since you can't actually run applications against it.

Description

Registry
Version: 3.0.6
Persistence type: postgresql

[EricWittmann]

Thanks for the post @wolfchimneyrock and sorry for the delay. We've got a bit of a backlog.

You've accurately described how the DRAFT feature is implemented. It definitely works that way by design, for a variety of reasons. It's not a feature that is intended to provide a mutable DEV version of the registry. Instead, it's intended to provide a way for tools to iteratively store more "temporary" content as that content is being actively worked on but before the content is ready to be used in any application. For example, we use it as the basis for persistence in Apicurio Studio 1.0.

@wolfchimneyrock could you perhaps share with us your use-case for wanting a mutable "dev" registry? What is the flow you would like to have? I'd like to understand your needs better, because it's possible that some other existing feature(s) of Registry (branches, delete/add, ifExists, etc) may be helpful to you. And if not, we may be able to shape a new feature that would fit your needs and make sense.

[wolfchimneyrock]

Thanks for the post @wolfchimneyrock and sorry for the delay. We've got a bit of a backlog.

You've accurately described how the DRAFT feature is implemented. It definitely works that way by design, for a variety of reasons. It's not a feature that is intended to provide a mutable DEV version of the registry. Instead, it's intended to provide a way for tools to iteratively store more "temporary" content as that content is being actively worked on but before the content is ready to be used in any application. For example, we use it as the basis for persistence in Apicurio Studio 1.0.

@wolfchimneyrock could you perhaps share with us your use-case for wanting a mutable "dev" registry? What is the flow you would like to have? I'd like to understand your needs better, because it's possible that some other existing feature(s) of Registry (branches, delete/add, ifExists, etc) may be helpful to you. And if not, we may be able to shape a new feature that would fit your needs and make sense.

Thanks Eric for looking into this, I appreciate it.

Our intent is to replace an existing schema registry built in-house with Apicurio. The existing registry works like this:

1. We have a DEV tier and a PROD tier. The DEV tier is "mutable" while the PROD tier is not.
2. Newly created schema versions are registered into DEV tier and assigned a sequential version number.
3. The contents of a version in DEV tier can be modified multiple times, until it is "promoted" to PROD and the contents "Frozen".
4. Compatibility rules when modifying content of a version are not evaluated against the current contents of the same version, only against the contents of prior versions (as if the current and past contents of the version being modified never existed)
5. a version that hasn't been promoted to PROD yet (i.e. in DEV mutable state) is visible to the client API indistinguishable from those that have been promoted / frozen.
   For this purpose I had looked at branches, however it would need to have a different transitive rule evaluation logic (as I raised in Branch Transitive Compatibility #5297)