[Bug]: fetches from gitlab.arm.com are not reliable

[alexeagle]

What happened?

Some modules point their source to GitLab, such as https://github.com/bazelbuild/bazel-central-registry/blob/main/modules/ape/1.0.1/source.json#L2

I have some customers reporting their developers complain about the availability, getting Read timed out errors (below)

(15:06:36) WARNING: Download from https://gitlab.arm.com/bazel/ape/-/releases/v1.0.1/downloads/src.tar.gz failed: class java.io.IOException Read timed out02:23
(15:06:36) INFO: Repository ape~ instantiated at:02:23
  <builtin>: in <toplevel>02:23
Repository rule http_archive defined at:02:23
  /home/semaphore/.cache/bazel/_bazel_semaphore/40affbd12709619c4a92f431bf6ac55a/external/bazel_tools/tools/build_defs/repo/http.bzl:387:31: in <toplevel>02:23
(15:06:36) ERROR: /home/semaphore/.cache/bazel/_bazel_semaphore/40affbd12709619c4a92f431bf6ac55a/external/bazel_tools/tools/build_defs/repo/http.bzl:136:45: An error occurred during the fetch of repository 'ape~':02:23
   Traceback (most recent call last):02:23
	File "/home/semaphore/.cache/bazel/_bazel_semaphore/40affbd12709619c4a92f431bf6ac55a/external/bazel_tools/tools/build_defs/repo/http.bzl", line 136, column 45, in _http_archive_impl02:23
		download_info = ctx.download_and_extract(02:23
Error in download_and_extract: java.io.IOException: Error downloading [https://gitlab.arm.com/bazel/ape/-/releases/v1.0.1/downloads/src.tar.gz] to /home/semaphore/.cache/bazel/_bazel_semaphore/40affbd12709619c4a92f431bf6ac55a/external/ape~/temp11647312340057587529/src.tar.gz: Read timed out02:23
(15:06:36) ERROR: Analysis of target '//:format.check' failed; build aborted: java.io.IOException: Error downloading [https://gitlab.arm.com/bazel/ape/-/releases/v1.0.1/downloads/src.tar.gz] to /home/semaphore/.cache/bazel/_bazel_semaphore/40affbd12709619c4a92f431bf6ac55a/external/ape~/temp11647312340057587529/src.tar.gz: Read timed out

Version

HEAD

How to reproduce

depend on the ape module and try to build during a GitLab CDN outage

Any other information?

No response

[fmeum]

@mattyclarkson

[mattyclarkson]

cc: @jonkirkham-arm

[fzakaria]

sources.json I think maps to the http_archive object in Bazel

Can we make it urls then and just fork it in GitHub also ?

[fzakaria]

Maybe we can edit: https://github.com/bazelbuild/bazel/blob/5d53bf66d6083e701ad183772d728c6b10931af3/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/IndexRegistry.java#L450 to support url or urls
(the integrity would be the same? if we fork it ... I would hope but maybe not if it's a ZIP.)

Separate issue?

[alexeagle]

Maybe I misreported the issue, I can't tell if the CDN in question is operated by GitLab or ARM.

In either case, there should be a fallback URL. @Wyverald can probably recall why the source.json has a single url while the API it wraps allows multiple.

[Wyverald]

We originally wanted the "mirrors" field of the bazel_registry.json file to take care of needing multiple URLs, although that ended up falling through for the BCR. Maybe we can add back the default of https://mirror.bazel.build/, without offering to automatically mirror artifacts. I'd prefer that over adding a urls field at this point. @meteorcloudy WDYT?

[fzakaria]

That would necessitate someone mirroring the BCR (Registry) vs. an individual module code URL.
Aren't these two different problems?
(I may be misunderstanding)

[Wyverald]

No, I'm talking about the field that controls mirroring the source archives (see https://bazel.build/external/registry#bazel-registry-json)

[mattyclarkson]

Maybe I misreported the issue, I can't tell if the CDN in question is operated by GitLab or ARM.

gitlab.arm.com is operated by Arm.

[meteorcloudy]

Maybe we can add back the default of https://mirror.bazel.build/, without offering to automatically mirror artifacts.

Wouldn't that generate too many warning messages for modules that are not mirrored? And it still requires us to mirror the artifact to mirror.bazel.build manually, right?

Unfortunately, we cannot do automatic mirroring due to potential copy-right issues.

[meteorcloudy]

I'm actually fine with adding "urls" to source.json, then the module maintainer can provide a backup URL.

[fzakaria]

@meteorcloudy should I open a Bazel issue for this and try to tackle it ?
I added the overlay option; was a bit slow for me but I can try

[meteorcloudy]

I think we still need to consider how to better introduce it in a backwards compatible way.

I can imagine a few issues if we just replace url with urls:

• new modules won't work with older Bazel versions
• we also need to update bcr_validation.py to check the new urls
• the BCR attestations check might also need to be updated.

How about we just add a mirror_urls attribute in source.json? Then newer Bazel versions would recognise it and add them as backup, while older Bazel versions and workflows won't be broken.

@Wyverald WDYT?