External compatibility / validity rules

[steverice]

Hello! Just started looking into this project, like what I see!

I was a bit confused (as was #1696) at first about the validation rules, and somewhat surprised to see that the rules are themselves implemented within the registry code. Many of these schema languages have lots of tools available for validation and compatibility checks already; in addition, every use case will have different rules they want to enforce (imagine "style guide" type rules) that won't make sense for other users.

Instead of needing to implement every compatibility/validation/style check within the Registry code, perhaps a more flexible pattern would be to define a callback interface through which rules could be run. I could see this taking the form of an HTTP request made by the Registry to another service which is expected to respond with a defined response format. Or perhaps a CLI command to run that would do the same.

Of course, the same thing can be achieved by having whatever client is uploading schemas run the validations itself; however, it would be preferable for the schema registry (source of truth for schema versions) to also be the source of truth for whether an incoming schema is valid, even if that decision is being delegated to another tool.

What do you think?

[steverice]

@EricWittmann @famartinrh what do you think? Is this in line with your conception of what Apicurio Registry does?

[famarting]

hey! IMO it makes sense, some sort of admission/validation webhook maybe embedded in our validity/compatibility rules. it would be a nice and clean way to allow for extending registry functionality. I would say contributions are welcome :)

Not strictly related to this, but registry already have the ability to emit events when artifacts are created,... and this discussion reminds me we need to have a nice way (at runtime) to configure this webhooks. As well as add support for securing the connections of the webhook invocations.

[steverice]

I am thinking of a model broadly similar to the GitHub Checks API; chiefly, a webhook that is sent to registered checker when a new schema is uploaded, and an API the checker can POST back to to report the status of the request.

It could be nice to actually offer compatibility with the GitHub APIs in such a way that the large ecosystem of check tools that already integrate with GitHub could be plugged into Apicurio as well. There are a few key differences though, chiefly that:

1. GitHub already has a concept of the "pending change" (pull request) that requires subsequent action to be taken before the change is made. This provides the opportunity for a stage where checks can be reported back asynchronously and block/affect the final change. I don't think Apicurio has a similar concept as the schema is added to the repository asynchronously. Perhaps metadata could be used to mark the schema as passing or failing checks and consumers could look for these markers.
2. Consumers of the GitHub webhook are expected to retrieve the commit contents by calling back to the GitHub API. While the similar concept exists here (fetch artifact version) it would be tricky, if not impossible, to expose that in a GitHub-compatible way.

I think that deciding on an approach for the first point is the more pressing issue, as it's relevant to the check execution model regardless of GitHub compatibility. It's also a design that presumably would affect the way existing (built-in) checks operate.

[EricWittmann]

Just getting to read this now - and it's very interesting. I agree with the consensus that an external service would be a nice way to support custom checking of content. I can imagine doing that for both Validity and Compatibility, although the latter is complicated because it requires previous version(s) of the content and the new content.

In any case, I would prefer to see this done incrementally, rather than go straight for a GitHub style asynchronous approach. What I mean is that I don't think we want to be implementing "content staging" functionality, at least not yet. That starts to feel a lot like BPM, getting into human approvals and other business process type checks. That could be a slippery slope, so I would prefer it if we started out small.

Perhaps the right thing to do is expand the /rules endpoints to support additional rule types that can be configured by the admin. In that way perhaps the /rules endpoints can support more than just Validity and Compatibility? So for example someone could register a new webhook-based rule called Linting and then that rule could be enabled/disabled just like the two existing rules.

Thoughts?

[EricWittmann]

Also sorry for the delay - I was on PTO for two weeks and am in the process of digging through my backlog of emails. :)