Turn down janitor jobs / boskos deployments ahead of default cluster turndown

[BenTheElder]

We should plan to turn these down a bit before actually deactivating the default cluster

1. stop jobs from accessing boskos resources
2. allow cleanup to run
3. turn down entirely
4. turn down default culster

for 1) I think we can do this ~August 1st, but it will take a little planning, as what we really want to do is just stop renting clean projects and allow everything to get cleaned ...

cc @dims @upodroid @ameukam

Need to think about how we can best accomplish that. Ideally even as we cut off the legacy resources we let the janitor jobs / boskos do their jobs one last time.

We could manually run cleanup, but ... we haven't done that in a long time and I'd rather not have to figure that out with auth for vsphere, azure, AWS, GCP ...

[BenTheElder]

/sig testing k8s-infra

[ameukam]

For (1), could be:

• slowly remove projects in the resources pool until it's empty (in 3-4 days). (According the https://monitoring.prow.k8s.io/d/wSrfvNxWz/boskos-resource-usage?orgId=1, There are not used)
• Set boskos deployment replicas to 0
• wait a week (or less) and delete the boskos k8s service

Alternatively, we could allow projects renting but disable GCP services in each services.

[BenTheElder]

We have at least AWS and VSphere in addition to GCP janitors.

Ramping down the pool doesn't let the cleanup run?

otherwise if we depend only on janitor jobs and not the reaper we could just disable boskos service endpoint

[ameukam]

AWS janitor currently runs against a CNCF account used by kOps. It should be fine to copy over those creds to the k8s-infra-prow-build-trusted cluster until we fix kubernetes/k8s.io#5127

[BenTheElder]

After sleeping on this ... I think we should just drop all of the non-janitor jobs and leave boskos/janitor deployed for another 24h.

So the list becomes:

1. delete all non-janitor default-cluster jobs
2. wait some period (24h? 12?)
3. turn down janitor, boskos, default cluster (or revert if there's some good reason)
4. ready for prow migration

[BenTheElder]

Though until kubernetes/k8s.io#5127 we probably do need to migrate that specific janitor then ...

[ameukam]

+1 on moving AWS janitor and dropping (disabling is a better word?) the non-janitor jobs.

[BenTheElder]

#33229 removes some of the last GCE jobs, will remove the GCP janitors a bit after we're done removing GCP jobs, following #33226 / https://groups.google.com/a/kubernetes.io/g/dev/c/qzNYpcN5la4

[BenTheElder]

https://monitoring.prow.k8s.io/d/wSrfvNxWz/boskos-resource-usage?orgId=1

We have a wedged ingress project, but otherwise we're close to being able to turn down the GCP janitors and cluster: default boskos instance.

[BenTheElder]

#33234 removes the legacy GCP janitors.

Need to follow up even on our current projects re: why are some of them in "other" state in boskos ... https://kubernetes.slack.com/archives/CCK68P2Q2/p1722900397667809

[BenTheElder]

#33241 will unblock dropping, or outright drop the vsphere janitors.

that leaves the AWS janitors to consider @dims, I think the issue is we still have some old CNCF AWS usage and not just the k8s infra stuff covered by the migrated janitors? not confident on this.

maintenance-ci-aws-janitor is still running in cluster: default.

[BenTheElder]

AFAICT, the legacy boskos only ever had GCP, so I can start looking at turning that down.

https://github.com/kubernetes/test-infra/blob/d72c3fcc5617fc61d02555658bc954dd24d6e6a6/config/prow/cluster/build/boskos-resources/boskos-resources.yaml

[BenTheElder]

Running vsphere janitor after #33241

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-provider-vsphere-janitor/1820933712594866176

[BenTheElder]

#33246 removed the vsphere janitor.
#33254 migrated the AWS janitor, so now it's on to turning down boskos.

[BenTheElder]

#33260 to turn down boskos

[BenTheElder]

This is complete.