release: optionnaly skip pushing incomplete OCI manifests

[jbtrystram]

We historically support publishing incomplete releases (not including all architectures).

Recently we started to publish OCI images to the ART repos to allow the build-node-image job to derive the OCP container image. A corner case here is that publishing an incomplete OCI manifest triggers the build-node-image which will fail because the manifest is incomplete,
until the next release job successfully push a complete manifest.

This is problematic because ART need to be able to reliably rebuild the OCP images if the openshift bits change, so they need a reliabe base manifest.

I suggest to skip publishing the manifest if it's incomplete. Let's add a pipecfg knob e.g. push-partial-manifest: false under registry_repos

[dustymabe]

I see two options here:

1. Modify the build-node-image job to also just build for the architectures that have been built
2. Modify the calls to the release job to stop setting ALLOW_MISSING_ARCHES to true

[prestist]

Which one would be preferred? @dustymabe

I was thinking we could add the arg mentioned above per registry (defined in the config.yaml) defaulting to true except on whichever one we don't want partial manifests to, and pivoting in the push step for release.jenkins; or is that not suitable?

[dustymabe]

Which one would be preferred? @dustymabe

It probably depends on what ART requires. Do they absolutely need every time the node-image container gets updated in the registry for it to have all architectures? If so then it probably requires 2. so the release job just fails if there are missing architectures.

I was thinking we could add the arg mentioned above per registry (defined in the config.yaml) defaulting to true except on whichever one we don't want partial manifests to, and pivoting in the push step for release.jenkins; or is that not suitable?

ehh. I know we push to multiple registries, but I'm not sure having them be out of sync would be ideal. It would probably be more confusing than anything else.

[dustymabe]

basically the tradeoff between 1. and 2. is that with 2. if one architecture is failing for an extended period of time OpenShift gets no updates to be testing in the meantime for all the other successful architectures.

[jlebon]

AIUI, ART will just not use builds which are incomplete. So we might as well do option 2.

New e.g. openshift-kubelet RPM builds at least would not be blocked from going into the nightly/CI payloads.

[dustymabe]

In that case I think we just need to change the logic slightly and also the comment in

fedora-coreos-pipeline/jobs/build.Jenkinsfile

Lines 571 to 586 in f9e562e

	def run_release_job(buildID) {
	stage('Publish') {
	// Since we are only running this stage for non-production (i.e.
	// mechanical and development) builds we'll default to allowing failures
	// for additional architectures.
	build job: 'release', wait: params.WAIT_FOR_RELEASE_JOB, parameters: [
	string(name: 'STREAM', value: params.STREAM),
	string(name: 'ADDITIONAL_ARCHES', value: params.ADDITIONAL_ARCHES),
	string(name: 'VERSION', value: buildID),
	booleanParam(name: 'ALLOW_MISSING_ARCHES', value: true),
	booleanParam(name: 'CLOUD_REPLICATION', value: params.CLOUD_REPLICATION),
	string(name: 'PIPECFG_HOTFIX_REPO', value: params.PIPECFG_HOTFIX_REPO),
	string(name: 'PIPECFG_HOTFIX_REF', value: params.PIPECFG_HOTFIX_REF)
	]
	}
	}

I think we do still want ALLOW_MISSING_ARCHES=true for FCOS development/mechanical streams, but for RHCOS we can just always set it to false.

[jbtrystram]

basically the tradeoff between 1. and 2. is that with 2. if one architecture is failing for an extended period of time OpenShift gets no updates to be testing in the meantime for all the other successful architectures.

Yeah, I think ART have some tooling that blocks (or at least mark as errors) payloads that have version inconsistencies between arches .

[jbtrystram]

Re - registries being out of sync : one thing I just thought of is that it would lead us to not notice/realize that no new image have been pushed to ART repos for a while when monitoring, possibly delaying response.

So yeah, toggling ALLOW_MISSING_ARCHES for RHCOS is both simpler and the more logical choice here.