Jenkins idler is retaining stale data on active builds

[ldimaggi]

Related to issue: openshiftio/openshift.io#2418

It appears that the issue reported in #2418 is caused by stale data relating to a completed build that is marked as active. The idler sees this build as active:

    "Name": "ldimaggi-osiotest2",
    "ID": "61ddf7b3-d141-402e-ae6a-30ee65f1879b",
    "ActiveBuild": {
        "metadata": {
            "name": "march1test-2",
            "\nnamespace": "ldimaggi-osiotest2",
            "annotations": {
                "openshift.io/build.number": "2",
                "openshift.io/jenkins-namespace": "ldimaggi-o\nsiotest2-jenkins"
            },
            "Generation": 0
        },
        "status": {
            "phase": "Running",
            "startTimestamp": "2018-03-01T18:18:49Z",
            "completionTimestamp\n": "2018-03-01T18:18:51.416135334Z"
        },
        "spec": {
            "replicas": 0,
            "Strategy": {
                "Type": "JenkinsPipeline"
            }
        }
    },
    "DoneBuild": {
        "metadata": {
            "\nname": "march1test-1",
            "namespace": "ldimaggi-osiotest2",
            "annotations": {
                "openshift.io/build.number": "1",
                "openshift.io/jenkins-\nnamespace": "ldimaggi-osiotest2-jenkins"
            },
            "Generation": 0
        },
        "status": {
            "phase": "Complete",
            "startTimestamp": "2018-03-01T17:44:17\nZ",
            "completionTimestamp": "2018-03-01T18:09:19Z"
        },
        "spec": {
            "replicas": 0,
            "Strategy": {
                "Type": "JenkinsPipeline"
            }
        }
    },
    "JenkinsState\nList": null,
    "JenkinsLastUpdate": "2018-03-02T13:26:41Z"
}

And - an active build prevents the idler from running.

But - no builds are active:

oc get builds -n ldimaggi-osiotest2
No resources found.

[hferentschik]

@ldimaggi So here is the thing, where what happened with your builds in general? There is not even a completed build. Where are the march1test-1 and march1test-2 builds? They don't even seem to show up as completed builds.

I think you reset the environment, right? Does this also reset pipeline builds? The logic in the Idler is expecting a specific flow a Build goes through and transitions the internal state accordingly. I am wondering whether the resetting of the environment basically screws up this state transitions so that the state we keep in memory is getting out of sync.

@vpavlin what do you think?

@ldimaggi @aslakknutsen I am not familiar with how the resetting of the environment works. What does it in terms of OpenShift actions? What happens with existing builds? What type of events (if any) would be oberservable?

[hferentschik]

I think this relates to issue #120 and #141. We should consider timestamps and we should change how we model the data.

[ldimaggi]

It's my understanding that the env reset removes the build configs and deploy configs - not sure what in addition to that. Wouldn't deleting the bc's and dc's remove all running builds?

How can I clean up this situation today? I cannot see anything in OS O via oc.

[hferentschik]

It's my understanding that the env reset removes the build configs and deploy configs - not sure what in addition to that.

Sure

Wouldn't deleting the bc's and dc's remove all running builds?

I think the problem is really on the Idler side. Not sure whether there is much you can do from your end right now. Restarting the Idler might help. As part of this issue, I am planning to add some sort of reset call which would allow to reset the state for a single namespace. This way if a namespace gets into a inconsistent state (in terms of the model the Idler build of it), there is an easy way to reset just this namespace. But all this required changes on the Idler code first.