Reduce Infrastructure Spending

[afrittoli]

We need to monitor infrastructure spending and make sure we reduce costs where possible.
Currently, our spending breakdown looks like this:

[afrittoli]

The cloud storage spending is associated mostly with the container registry image download bandwidth. We will have to migrate to the artifact registry soon, but that will not reduce cost.

[afrittoli]

Areas where we could reduce spending:

• Cloud Storage / Container Registry: move released container images to ghcr.io. This will shift over time the traffic from gcr.io to ghcr.io, as Tekton users move to the latest releases.
  • Knowing the source of the egress traffic would help optimise this work, unfortunately, it is currently not possible to find that out from Google cloud monitoring tools.
  • It is possible that some or most of the traffic may be generated by CI systems of Tekton or other projects that use Tekton. If this is true, even moving new releases might alleviate the cost.
• Compute Engine: this is made of a combination of running services and jobs (CI/CD pipelines). The data needs further analysis; some initial considerations:
  • Prow: this is used by most CI jobs today, and it cannot be removed easily. The CPU/memory consumption of the control plane seems contained. The test jobs name space is where most of the CPU requests and consumptions are spent. We can review existing CI jobs to see if there is any optimisation work that can be done. We have done work in the past to optimise these, using kind for e2e tests, with a nodepool that scales down to zero when not used.
    • An option to reduce cost here is to run kind tests as GitHub actions. This requires a few steps:
      • Setup the workflow as a matrix test that covers the various e2e tests (in each project)
        • Pipeline: Run e2e tests in GHA pipeline#8411
        • Triggers
        • Chains
        • Operator
        • Results
        • Catalog
        • Dashboard: Add GitHub Actions workflow for E2E tests dashboard#3834
        • Cli?
      • Mark the new tests as mandatory in prow/tide config
      • Give users a way to re-run failed tests
        • Re-usable action in plumbing Reusable actions to restest GitHub actions #2330
        • Adopt in all repos
  • Tekton (dogfooding): this is used by all CD pipelines, so it cannot be removed easily. The CPU/memory consumption of the control plane seems contained.
  • Messaging (knative + kafka): this is used to transport events from Tekton pipelines to Tekton Event Listeners. Since we only have one consumer (the tekton-events event listener) we could remove knative + kafka and rely on 1:1 delivery of events. Kafka is one of the top consumers of memory among the various services deployed, so it may be worth removing it even if it means losing some reliability in the delivery of events
    • Implemented in Send events from pipeline to the listener #2245
  • Nightly builds: these use decent amounts of CPU and memory, even if only for 30m each
    • Reduce the frequency of nightly builds
    • Reduce the amount of testing we do there since the code is already tested in each PR.
      • The runTests flag has been added to nightly releases, and it defaults to false. Add runTests to the release cronjob base #2208, Run build and tests conditionally pipeline#8252
      • The runTests flag is supported by most release pipelines, the complete list is tracked in Reduce Cloud Storage / Container Registry Spending #2157
    • Move nightly builds to GHA and reduce frequency #2670
• Container Registry vulnerability scanning: we could disable this. We use minimal base images which are already scanned elsewhere, and we perform various types of security scanning on the code in each PR, which may be sufficient
  • This is disabled now
• Robocat cluster: today, this is used mainly to deploy services to dogfooding via Tekton pipelines. We should find an alternative deployment solution and remove the robocat cluster: Remove the robocat cluster #2255

Other services are negligible and not worth looking into for now.

[afrittoli]

This PR enables removing tests from nightly builds: tektoncd/pipeline#8252