Control costs of mybinder.org build registries

[minrk]

Problem Statement

As a mybinder.org federation operator, I want to control the costs of my deployment. One of the costs that starts small, but grows over time is the storage cost for the build cache (an OCI registry). Because this is a cache, it is fine to delete 'unused' images, since they can be rebuilt if they are ever launched again. However, most registries do not preserve the necessary information to determine which images are 'unused' and therefore safe to delete.

Proposed Solution

Deploy a registry implementation that has sufficient rules to limit size for cost purposes. The registry should be configurable to allow deleting the least recently used (LRU) images to keep the size in check. How much to delete and how often can be tuned per deployment according to cost requirements.

Proposed Implementation

Deploy Harbor registries backed by local Object Storage. Harbor has both last pull time metadata and sufficient garbage collection rules to use it as a regular cache.

How will this fit in the ecosystem?

costs to deploy new members to mybinder.org are lowered, and an example is set for other BinderHub instances that may want to follow suit.

Endorsements

• @minrk
• @yuvipanda

[minrk]

Linking lots of issues:

• harbor issues
• harbor PRs

This is basically done, and we now have full replication of images across the federation, and have disabled rendezvous routing, meaning more even balancing of load across the federation when a small number of repos represent a large fraction of traffic.

Replication has complicated one angle of cost minimization, because now the majority of images in each registry have 0 pulls.

Looking at metrics and pull rates, I am highly suspicious that a large fraction of our traffic (at least in terms of number of images, if not number of launches) is from bots and crawlers that don't actually run a session.

When I analyze the expected number of builds expected from our launch data, I get around 8,000, but we have around 18,000 images in the registry. If I've measured correctly (big if), that would mean that over 50% of the images we build never get launched, which is a strong indication of abuse / automated use of Binder by crawlers and bots (or unused CI).

We don't currently have any direct measurement of the number of builds (other than what's in harbor itself), so we can't actually directly compare the number of builds to the number of launches without a lot of reconstruction.