Add support for gitlab as build registry

[rochaporto]

We're trying to use a gitlab registry as backend for build images.

The setup is done using a helm chart, and passing in the configuration:

config:
  DockerRegistry:
    token_url: "https://gitlab.cern.ch/jwt/auth"
    username: <username>
    password: <password>
  BinderHub:
    image_prefix: gitlab-registry.cern.ch/binder/images/build-
registry:
  url: "gitlab-registry.cern.ch"

After debugging the multiple authz errors, it seems to boil down to:

1. missing "service": "container_registry" when fetching a valid token using the token_url (this is required by gitlab)
2. the build image name logic assumes a dockerhub like structure (up to two levels in the path). The gitlab registry allows more and we rely on this (see above with binder/images/actual-image-name)

I'll push a PR soon, just figuring out the best way to make 2. generic.

[betatim]

That would be great! There are a few parts of the code where we have taken shortcuts because we made assumptions about the structure of the URL. Probably a lot of them would be cleaner if we use more of urllib.parse() instead of blah.split("/")[-2].

[bdrian]

Since @betatim asked for it I prepared #986 as an alternative approach to the second problem (more than two levels in image names, cf. GitLab docs). There are some educated guesses introduced in that approach, but it's almost not breaking any existing setups or code.

After way too much time spent on thinking this through, I believe the viable options are:

1. require users to specify registry-1.docker.io/ before their desired prefix when using Docker Hub, so we can always cut the first part off. This may require configuration changes on some BinderHub installations.
2. check if the image name looks like it may come from Docker Hub (two components, some name restrictions beyond what Docker alone could handle). If not, we cut the first component, otherwise we keep the full name. This is what Add support for 3+-part image names (like on GitLab) #986 does.
3. separate the image_prefix configuration into two parts, registry and image name (similar to what's assumed by Add support for storing build images in gitlab registries #966). Changes to code, docs and configuration would be necessary, but it cuts the guesswork.

Personally I'd slightly prefer 1), but that should reach some form of consensus first, because of said configuration/documentation changes.

Using urllib for this task is equal to simply cutting the first part off, as that must be the (registry) host when interpreting the image name as URL. This would still break Docker Hub short names (see above).

The token acquiring issue is not touched by this and still open for solutions.

[dichaelen]

Hi @rochaporto,

As of primo February 2020 is there any solution to this? Or are anybody working on it to your knowledge? We are facing the same challenge and would like to have as much input as possible, A shortterm solution is to use our legacy registry outside GitLab, but it is not sustainable.

thanks in advance!

[betatim]

I don't think anyone has worked on this.

[rochaporto]

Sorry, i left this here and never pushed the PR. Will try to do it asap.

[ctr26]

This works for me:

https://github.com/bioimagearchive/k8s-jupyterhub/blob/master/binderhub/ebi_dockerhub.yaml

[ctr26]

Update, it doesn't work for finding pre built images it just successfully pushes

[darwyncook]

We are interested in using Gitlab for a build registry. We need to have our registry on-prem, and have gitlab available. He there been any progress on this?

[manics]

There's an open PR:
#1628

[ctr26]

Best solution I found was to self host a dockerhub, our docker registry was throttling