Bazel 8.0.0 compatibility

[gergelyfabian]

When testing Bazel 8.0.0 RC versions I discovered rules_scala may be incompatible with the latest rules_java (that you in turn my need for Bazel 8.0.0).

This came out when testing a Bazel bug's solution: bazelbuild/bazel#24235.

After upgrading rules_java to 8.5.0+ rules_scala reported errors:

# Using https://github.com/gergelyfabian/bazel-scala-example/tree/bazel-8.0.0rc1
$ USE_BAZEL_VERSION=8.0.0rc3 bazel build //...
INFO: Invocation ID: b8abf68e-49c2-48ee-a5e4-f0d8710f759d
ERROR: Traceback (most recent call last):
	File "/home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala/scala/private/rules/scala_doc.bzl", line 52, column 37, in <toplevel>
		_scaladoc_transitive_aspect = aspect(
Error in aspect: Illegal argument: element in 'required_aspect_providers' is of unexpected type. Either all elements should be providers, or all elements should be lists of providers, but got list with an element of type NoneType.
ERROR: Error computing the main repository mapping: at /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala/scala/scala.bzl:20:5: initialization of module 'scala/private/rules/scala_doc.bzl' failed

# Or another one:

$ USE_BAZEL_VERSION=8.0.0rc3 bazel build //...
INFO: Invocation ID: 117a90d7-da17-400c-9f83-908c7ddd7317
WARNING: WORKSPACE support will be removed in Bazel 9 (late 2025), please migrate to Bzlmod, see https://bazel.build/external/migration.
ERROR: Traceback (most recent call last):
	File "/home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala/scala/private/common_attributes.bzl", line 18, column 28, in <toplevel>
		"deps": attr.label_list(
Error in label_list: Illegal argument: element in 'providers' is of unexpected type. Either all elements should be providers, or all elements should be lists of providers, but got list with an element of type NoneType.
ERROR: Error computing the main repository mapping: at /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala/scala/scala.bzl:37:5: at /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala/scala/private/rules/scala_repl.bzl:5:5: initialization of module 'scala/private/common_attributes.bzl' failed

Please note, that rules_cc version is also too "old" for Bazel 8.0.0, so I upgraded it on my branch too explicitly. It may need to be updated in rules_scala.

[mbland]

I noted in #1625, based on copious notes from cd22d88 (under Bump to rules_java 7.9.0 for Bazel 7 compatibility), why we're stuck at rules_java 7.9.0 for now.

But, the good news is, Bzlmod builds aren't affected by that particular problem, and Bzlmodification is in the home stretch. Once that lands, we can bump up the rules_java version.

FWIW, I've been using rules_java 7.11.1 in my bzlmod working branch. I'll try to bump it to the latest 8.5.1 and see what happens.

[mbland]

Ah. We have a continuation of the protobuf + ScalaPB saga from #1647.

If I bump to rules_java 8.5.1 (and rules_cc 0.1.0), and stick with protobuf 21.7 (the current rules_scala version, via single_version_override) or protobuf 24.4 (the latest Bazel Central Registry protobuf module compatible with ScalaPB 0.11.17), I get:

$ bazel build //src/... //test/... //third_party/... //scala_proto/...

ERROR: Skipping '//src/...': error loading package under directory 'src':
  error loading package 'src/java/io/bazel/rulesscala/io_utils':
  at .../external/rules_java~/java/defs.bzl:16:6:

Label '@@protobuf~//bazel:java_lite_proto_library.bzl' is invalid because
  'bazel' is not a package; perhaps you meant to put the colon here:
  '@@protobuf~//:bazel/java_lite_proto_library.bzl'?

ERROR: error loading package under directory 'src':
  error loading package 'src/java/io/bazel/rulesscala/io_utils':
  at .../external/rules_java~/java/defs.bzl:16:6:
  
Label '@@protobuf~//bazel:java_lite_proto_library.bzl' is invalid because 
  'bazel' is not a package; perhaps you meant to put the colon here:
  '@@protobuf~//:bazel/java_lite_proto_library.bzl'?

The //bazel package doesn't appear until Protobuf v25.0, and //bazel:java_lite_proto_library.bzl doesn't appear until Protobuf v27.0. Per #1647, there's no version of ScalaPB that works with Protobuf v27.

However, bumping ScalaPB to 1.0.0-alpha.1 and Protobuf v28.3 did work. But as noted in #1647, committing the ScalaPB 1.0.0-alpha.1 bump would mean effectively forcing rules_scala users to upgrade to at least Protobuf v28.2 immediately.

Maybe that's what we ought to do anyway, because clearly we don't want to block anyone from upgrading to Bazel 8. But we'd have to shout it from the rooftops—at the same time we try to persuade folks to switch from depending on rules_scala via WORKSPACE to Bzlmod.

Even so, rules_cc 0.0.9 and rules_java 7.9.0 still work with Protobuf v28.3 and ScalaPB 1.0.0-alpha.1. That means we wouldn't leave Bazel 6 or 7 WORKSPACE users behind entirely, so long as Protobuf v28.3 works for them, if they insist on holding onto WORKSPACE for now.

[mbland]

Ah, more info. The results from my last comment were from building with Bazel 7.4.1. Building with 8.0.0rc4 is not yet possible.

First, I had to update to the highest versions of all dependencies in the Bazel module graph:

bazel_dep(name = "rules_cc", version = "0.1.0")
bazel_dep(name = "rules_java", version = "8.5.1")
bazel_dep(name = "rules_proto", version = "7.0.2")
bazel_dep(name = "protobuf", version = "29.0-rc3", repo_name = "com_google_protobuf")
bazel_dep(name = "rules_python", version = "0.40.0", dev_dependency = True)

I also bumped the following in scripts/create_repository.py and ran it:

PROTOBUF_JAVA_VERSION = "4.29.0-RC3"

I had to add one, the other, or both of these lines to the *.bzl files that needed them:

load("@rules_java//java/common:java_common.bzl", "java_common")
load("@rules_java//java/common:java_info.bzl", "JavaInfo")

But sadly, ScalaPB 1.0.0-alpha.1 and protobuf-java:4.29.0-RC3 aren't compatible:

$ bazel build //src/... //test/... //third_party/... //scala_proto/...

ERROR: .../test/proto/BUILD:184:14:
  ProtoScalaPBRule test/proto/nested_proto_jvm_extra_protobuf_generator_scalapb.srcjar
  failed: (Exit 1): scalapb_worker failed:
  error executing ProtoScalaPBRule command
  (from target //test/proto:nested_proto)
  bazel-out/.../bin/src/scala/scripts/scalapb_worker ...
    (remaining 2 arguments skipped)

--jvm_extra_protobuf_generator_out: java.lang.ExceptionInInitializerError
  at scalapb.options.compiler.Scalapb$.registerAllExtensions(Scalapb.scala:8)
  at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.run(ExtraProtobufGenerator.scala:53)
  at protocbridge.frontend.PluginFrontend$.$anonfun$runWithBytes$1(PluginFrontend.scala:51)
  at scala.util.Try$.apply(Try.scala:217)
  at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:51)
  at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:121)
  at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$2(PosixPluginFrontend.scala:40)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
  at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:60)
  at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118)
  at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:71)
  at scala.concurrent.package$.blocking(package.scala:124)
  at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$1(PosixPluginFrontend.scala:38)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
  at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:687)
  at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
  at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
  at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
  at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
  at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
  at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
  at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

Caused by: com.google.protobuf.RuntimeVersion$ProtobufRuntimeVersionException:
  Detected mismatched Protobuf Gencode/Runtime version suffixes when loading
  scalapb.options.Scalapb: gencode 4.28.2, runtime 4.29.0-rc3.
  Version suffixes must be the same.

  at com.google.protobuf.RuntimeVersion.validateProtobufGencodeVersionImpl(RuntimeVersion.java:126)
  at com.google.protobuf.RuntimeVersion.validateProtobufGencodeVersion(RuntimeVersion.java:71)
  at scalapb.options.Scalapb.<clinit>(Scalapb.java:11)
  ... 22 more

java.lang.RuntimeException: Exit with code 1
  at scala.sys.package$.error(package.scala:27)
  at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44)
  at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96)
  at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49)
  at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39)
  at scripts.ScalaPBWorker.main(ScalaPBWorker.scala)

ERROR: .../test/proto/BUILD:184:14
  Building source jar test/proto/nested_proto_scalapb-src.jar failed:
  (Exit 1): scalapb_worker failed:
  error executing ProtoScalaPBRule command
  (from target //test/proto:nested_proto)
  bazel-out/.../bin/src/scala/scripts/scalapb_worker ...
    (remaining 2 arguments skipped)

Pinning Protobuf to v28.3 using single_version_override didn't work:

$ bazel build //src/... //test/... //third_party/... //scala_proto/...

ERROR: Skipping '//scala_proto/...':
  error loading package under directory 'scala_proto':
  error loading package 'scala_proto/private':
  Internal error while loading Starlark builtins:
  Failed to autoload external symbols:
  cannot load '@@protobuf+//bazel/private:bazel_cc_proto_library.bzl':
  no such file

ERROR: error loading package under directory 'scala_proto':
  error loading package 'scala_proto/private':
  Internal error while loading Starlark builtins:
  Failed to autoload external symbols:
  cannot load '@@protobuf+//bazel/private:bazel_cc_proto_library.bzl':
  no such file

@protobuf//bazel/private:bazel_cc_proto_library.bzl doesn't appear until v29-dev.

These aren't insurmountable issues; perhaps once Protobuf v29.0 lands, I can try again, and ScalaPB v1.0.0-alpha.1 will be happy with plain protobuf-java:4.29.0. Building with Bazel 8 will be blocked until then, even after Bzlmodification.

However, building with Bazel 8 will require those new load("@rules_java//") statements, and that will break WORKSPACE users for sure.

[mbland]

I spoke too soon: I can get a WORKSPACE build to use rules_java 8.5.1 and the other updated repo versions, and get to the point where it reaches the Detected mismatched Protobuf Gencode/Runtime version suffixes error.

So it seems we could bump to Bazel 8 without abandoning WORKSPACE, but we will abandon Bazel 6 and 7 at that point.

Update: To confirm, I went back to Bazel 7.4.1 to build using the new WORKSPACE config with rules_java 8.5.1, etc., and it died on building the remote_java_tools repo:

$ bazel build //src/... //test/... //third_party/... //scala_proto/...

ERROR: Traceback (most recent call last):
  File ".../external/remote_java_tools/BUILD", line 7, column 60, in <toplevel>
    load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library", "cc_proto_library")
Error: file '@rules_cc//cc:defs.bzl' does not contain symbol 'cc_proto_library'

ERROR: .../external/remote_java_tools/BUILD:
  no such target '@@remote_java_tools//:TurbineDirect':
  target 'TurbineDirect' not declared in package ''
  defined by .../external/remote_java_tools/BUILD

ERROR: .../test/toolchains/BUILD.bazel:10:23:
  no such target '@@remote_java_tools//:TurbineDirect':
  target 'TurbineDirect' not declared in package ''
  defined by .../external/remote_java_tools/BUILD
  and referenced by '//test/toolchains:java21_toolchain'

ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed;
  build aborted: Analysis failed

[mbland]

I've created the bazel-8-compatibility working branch in my rules_scala fork with the commit message for mbland@ebe714d describing the current build breakages, which are the same under WORKSPACE and Bzlmod.

Note that this branch/commit is based on my bzlmod working branch, not the current upstream master branch.

[hvadehra]

I believe the original issue in the description (#1652 (comment)) is because the scala/scala.bzl file exports helper macros for WORKSPACE setup (such as scala_repositories.bzl) but also transitively loads bzl files not intended for WORKSPACE evaluation - for eg scala_library.bzl. The latter refer to BUILD/bzl symbols such as JavaInfo, and this breaks with Bazel 8 / Bazel@HEAD as these symbols have been removed.

Historically, this has only worked by accident, as the Starlark environment for WORKSPACE evaluation also (incorrectly) included these symbols. See bazelbuild/bazel#17713 for when this first occurred while moving some of these symbols to Starlark (but still embedded in Bazel).

The right fix would to partition the rules_scala bzl load graph into non-overlapping WORKSPACE & BUILD/bzl subsets.

(cc @comius, @meteorcloudy)
However, it may also be possible to make Bazel 8 fully backwards-compatibile by also injecting the relocated symbols in the WORKSPACE environment here . Another alternative could be to fix this by replacing the symbols with non-functional but type-compatible ones (i.e. a dummy Provider instead of Starlark.NONE for JavaInfo).

[meteorcloudy]

I think it's better to do the right thing here instead of implementing more hacks that would be remove anyway in future.

[hvadehra]

However, it may also be possible to make Bazel 8 fully backwards-compatibile by also injecting the relocated symbols in the WORKSPACE environment here.

As it turns out, this can't actually work (at least not easily), as WORKSPACE evaluation is itself needed before these symbols are available for injection.

Another alternative could be to fix this by replacing the symbols with non-functional but type-compatible ones (i.e. a dummy Provider instead of Starlark.NONE for JavaInfo).

This seems doable and I might end up doing this for Bazel@HEAD, if only to turn our downstream CI green. However, (if it does happen) cherry-picking the fix to Bazel 8 will only occur if we end up needing a new RC for other reasons.