GitOps to automat CI - Github and Apicurio Registry

[ozarkblue]

I'm looking for CI automation with Github and Apicurio Registry using a GitOps process.

Use case : Engineers can keep one yaml file with schema metadata and content together OR 2 files, one yaml with metadata and another '.sr' file with content in Github. Once PR is raised in Github then it should run the validation (syntax check/content rule etc) pulling content form Git repo. Once PR is merged it should be able to perform Create/Update new version of the schema into Registry based on metadata and content.

Keeping Manuel schema management process can cause version mismatch issues while a strong CI/CD automated process being followed already within Org.
Also it'll help to address current limitation of a team's access to upload their artifact/update version. As this way we can use a service ID for this automated CI having upload access and restrict all engineer's IDs to Registry UI with readonly access for search and find purpose.

Also engineers can maintain schema along with event producer/consumer code in Github and it can perform the registry maintenance job automated way.

Is there any solution in place to make this happen?
Any schema validation or schema push API in place that can be used as Github webhook for this GitOps process?

I'v seen Operator sync CRDs in Gitbub project , yet to test functionality though.
But reading document in that Github I'm not getting much yet how it'll enable GitOps.

@EricWittmann appreciate for any solution/approach to this pls

[jsenko]

Thank you for bringing up this use case. We've recently discussed that we would like to improve our story in this area, since we do not have a specific solution at the moment. Current options are:

• Using the sync operator. Tools like ArgoCD can synchronize custom k8s resources from a git repository to a k8s cluster, from where the operator then performs the sync. This should work, however, it would restrict the format of the files in the repo and the sync operator is not very mature yet.
• Using one of the apicurio-registry-client-sdk-* projects to create a custom tool that would do the sync. We have client SDKs for JS, Go and Python. There is an API endpoint for testing whether an artifact would pass the configured rules /groups/{groupId}/artifacts/{artifactId}/test

Maybe other devs can suggest more options.

[EricWittmann]

Yes we were just talking about the GitOps use-case earlier this week. And the more we discuss it the more we like it. :) But right now Jakub's answer is the right one. I suspect we'll be exploring GitOps + Apicurio Registry more in the future...

[ozarkblue]

I'm able to get some success ,very recently, implementing a basic version of this automation . It worked to take content from Git repo and create artifact on gitpush.

Here is what I did.

Created a CI pipeline look based on gitpush event on put company CI pipeline tool.
In that pipeline provided Git repo URL which is containing an YML file to control the pipeline flow
In that YML configured code to trigger a flow based on gitpush event
In that flow read the schema.json in same git repo and made PPOST to Registry artifact creation endpoint.
It worked. I'm yet to harden this to create version and some validation check etc.

YML file in my Git repo to control the CI flow:

To enhance this I'm looking for additional service endpoints - For validation (syntax, compatibility rule check etc)
Planning to call that validation for Git-PR review event so that it performs validation and return any error to fix.

Then during merge this gitpush will be called to create a new artifact or create a new version if exists.