Migrate away from google.com gcp project kubernetes-jenkins

[spiffxp]

Part of umbrella issue to migrate away from google.com gcp projects: #1469

Umbrella issue for migrating or removing dependence on all of the kubernetes project infra that lives under kubernetes-jenkins:

GCS buckets:
gsutil ls -p kubernetes-jenkins | grep -v kubernetes-staging- | sed -e 's/.*/- [ ] `&`/'

• gs://artifacts-upload-test/: TODO: ???
• gs://artifacts.kubernetes-jenkins.appspot.com/: TODO: sincerely doubt we need to migrate GCR buckets
• gs://gcf-sources-661044217466-us-central1/: TODO: going to assume we can ignore this
• gs://k8s-bazel-cache/: May be used by kubernetes/kubernetes bazel, may be defunct? Last updated files 2019-03-11
• gs://k8s-kops-gce/: TODO: ???
• gs://kubernetes-e2e-soak-configs/: TODO: ???
• gs://kubernetes-federation-release/: TODO: ??? for this and the other federation buckets, surely these aren't used
• gs://kubernetes-federation-release-1-4/
• gs://kubernetes-federation-release-1-5/
• gs://kubernetes-federation-release-1-6/
• gs://kubernetes-federation-release-1-7/
• gs://kubernetes-federation-release-1-8/
• gs://kubernetes-federation-release-jenkins/: TODO: ???
• gs://kubernetes-federation-release-serial/: TODO: ???
• gs://kubernetes-jenkins/: hosts all logs/artifacts from prow.k8s.io, will need a plan/proposal
• gs://kubernetes-jenkins-gcslogs/: definitely do not migrate
• gs://kubernetes-jenkins-old/: TODO: : we probably don't care if this is migrated
• gs://kubernetes-release-dev-jenkins/: TODO: this probably shouldn't be migrated
• gs://kubernetes-test-history/: we can ignore this
• gs://sig-scalability-logs/: should happen as part of Migrate away from google.com gcp project kubernetes-scale #2241
• gs://us.artifacts.kubernetes-jenkins.appspot.com/: TODO: sincerely doubt we need to migrate GCR buckets

Service Accounts:
I'm not sure of everything I am (not) allowed to list here. So this isn't an exhaustive list. But we should make sure none of the service accounts in this project are used in any IAM bindings in kubernetes.io. Googlers will need to help identify this.

• kubekins@kubernetes-jenkins.iam.gserviceaccount.com - This is the big one, identify when/where this has access to services/resources that should be migrated or have equivalents stood up in kubernetes.io
• test-owners@kubernetes-jenkins.iam.gserviceaccount.com
• queue-health@kubernetes-jenkins.iam.gserviceaccount.com

[spiffxp]

At the moment it's not clear to me whether we can update prow.k8s.io to support different jobs writing to different buckets, or if we're going to have to make the change in lockstep. This is why migrating gs://kubernetes-jenkins needs a proposal

[spiffxp]

/lifecycle frozen

[tosi3k]

On the occasion of moving things out of the kubernetes-jenkins GCP project, gs://sig-scalability-logs bucket should also be migrated.

SIG scalability uses this bucket to store master and node logs separately from the gs://kubernetes-jenkins bucket for two reasons:

1. it was needed for us in order to finish migration of our Prow jobs to pod-utils (Add GCS_ARTIFACTS_DIR env var handling in pod-utils test-infra#17215)
2. speeding up loading the Prow GCS viewer of Prow jobs where many artifacts were stored in gs://kubernetes-jenkins bucket (we produce lots of them in our tests given the test cluster' sizes)

[ameukam]

On the occasion of moving things out of the kubernetes-jenkins GCP project, gs://sig-scalability-logs bucket should also be migrated.

SIG scalability uses this bucket to store master and node logs separately from the gs://kubernetes-jenkins bucket for two reasons:

1. it was needed for us in order to finish migration of our Prow jobs to pod-utils (Add GCS_ARTIFACTS_DIR env var handling in pod-utils test-infra#17215)
2. speeding up loading the Prow GCS viewer of Prow jobs where many artifacts were stored in gs://kubernetes-jenkins bucket (we produce lots of them in our tests given the test cluster' sizes)

@tosi3k Do you want the contents of this bucket to be preserved? if yes, what is the estimated size of sig-scalability-logs ?

[tosi3k]

@tosi3k Do you want the contents of this bucket to be preserved?

We just need the last 90 days of the contents therein.

I wonder - would applying some retention policy to gs://sig-scalability-logs bucket make sense? I'm not a big GCS / test-infra expert but I think that some kind of mechanism like this is already done for gs://kubernetes-jenkins bucket where we store the logs for jobs that have finished in the last 90 days IIUC.

After we migrate the scalability job configs to use gs://k8s-infra-scalability-tests-logs instead and 90 days pass, we could remove the old bucket (gs://sig-scalability-logs) completely as there would be no need for it anymore. Would that make sense here?

if yes, what is the estimated size of sig-scalability-logs?

I don't know but if we were to introduce a 90d retention policy mechanism we would probably comfortably fit in 5 TBs. This is the size of the bucket after 97 days of existence: