Filter builds by labels to get lower traffic

[vpavlin]

Jenkins Idler in OSIO production generates (consumes) quite a lot of nework traffic

We could lower that traffic by filtering builds by labels the same way we do for DC events (

fabric8-jenkins-idler/internal/openshift/client/openshift_client.go

Line 368 in f47714b

v.Add("labelSelector", "app=jenkins")

)

This would require a change to Jenkins Sync plugin (https://github.com/fabric8io/jenkins-sync-plugin/) (adding a label similar to https://github.com/jenkinsci/openshift-sync-plugin/blob/d3b43c767085d1224859345907d41d1e4f917ead/src/main/java/io/fabric8/jenkins/openshiftsync/Constants.java#L33, but "hardcoding" value like "type=jenkins-pipeline")

That way we could filter on that label and not get all the s2i builds happening in the cluster

[hrishin]

@vpavlin @hferentschik
This may be happening because of Jenkins sync plugin polling behavior. It would be better if we can catch HTTP queries for a particular resource rather than just labels.
For instance, I have fixed the fabric8io/jenkins-sync-plugin#16 Build resource polling. But this change is not present in Jenkins Image yet.

[vpavlin]

No, this is strictly about Idler calling OpenShift GET /builds API with watch parameter, which means OpenShift sends all build events to Idler (JenkinsPipeline, Docker, S2I...). We only care about

strategy:
    jenkinsPipelineStrategy:

which we filter for when we receive the event, but I'd prefer if the filtering happened on OpenShift API server (i.e. only event for builds with specific labels would be sent)

Hope it makes more sense now.

[hferentschik]

Sounds reasonable. So right now we don't have any way to filter what we are interested in using watch? The less we have to process the better. We should create a matching issue in the sync plugin and reference it here. Once was have something to limit our watch with, it should be easy to use.

[vpavlin]

The only filtering I know about is based on labels - as we do for DCs. If you know about any other way, that would be great

[aslakknutsen]

The traffic provided in the example above has little to do with the Build actually. That's primarily caused by the DC Watch and evidence of the Idle loop that happened last week.

Before/After loop:

More "normal" traffic:

Labeling to only get pipelines would help some, but even then it seems like the majority of Build events are simple 'timestamp' updates or similar. The last 24h there has been 1.6Mil Build Events on roughly only a handful of actual builds.

My guess is, that as long as Jenkins is running, the Jenkins Sync plugin will update the Openshift Build model about every second to set some 'ran for time' or updating it with status Running or whatever, this change is in effect triggering the OS Build Events that are consumed by Idler and generates about 99.99% of the traffic we're seeing. (@hrishin)

On top of my head, I would assume Idler really only cares about BuildStart and BuildStopped. Everything else is noise.

[hrishin]

My guess is, that as long as Jenkins is running, the Jenkins Sync plugin will update the Openshift Build model about every second to set some 'ran for time' or updating it with status Running or whatever, this change is in effect triggering the OS Build Events that are consumed by Idler and generates about 99.99% of the traffic we're seeing.

There might be a good possibility that Jenkins sync plugin updating the Build resource timestamp.

https://github.com/fabric8io/jenkins-sync-plugin/blob/job-to-bc/src/main/java/io/fabric8/jenkins/openshiftsync/BuildSyncRunListener.java#L373

https://github.com/fabric8io/jenkins-sync-plugin/blob/job-to-bc/src/main/java/io/fabric8/jenkins/openshiftsync/BuildSyncRunListener.java#L324

What behavior are we expecting if the plugin is doing so? or along with a timestamp can we consider the build number to evaluate idling condition? maybe that can get added from sync plugin as a label to build metadata if it's not exist.

[aslakknutsen]

It's easy to see what OSO Is spewing out via the Build events API if you watch the network traffic via the Pipeline view on OSiO. It essentially ends up in what seems to be an endless loop of reporting the same thing over and over and over... I'm just suspecting it's the Sync plugin.

There are a few different actual versions of the BuildEvents as Jenkins progress in the stages etc, but at minimum 10 identical Events with no changes in the data being sent beyond the Resource Version in OSO (One can question why OS don't 'invalidate an identical Object' value wise as a new events, but.. ). Since the revision id is different it would imply 'something' is updating it over and over.

This is not just a problem for the Idler mind you, it's an issue for the pipeline view, the oso proxy, the user etc as endless amounts of pointless traffic is sent over the wire.

Based on how the idler works atm, it will be listening to the same build events about the build being 'in approval stage' about every second until it gets sick of it all and idles it after 45min. :)

[aslakknutsen]

What behavior are we expecting if the plugin is doing so?

@hrishin A good first step would be; Don't update Openshift if nothing changed.

[hrishin]

A good first step would be; Don't update Openshift if nothing changed.

Update: it has been fixed on Jenkins sync plugin side.
cc: @aslakknutsen @piyush1594

[piyush-garg]

Yaa this has been fixed in openshift sync plugin by openshiftio/openshift.io#3266