Network issue with new version 2.5.6 behind corporate firewall

[christianknoepfle]

Hi,

we are using sbt and sbt-scalfmt for quite some time on our internal cicd-runner, which is located behind a corporate firewall. Internet access is limited but we can request to open https access to various sites.

Until last week sbt-scalafmt worked fine (we were using 2.5.5 and calling scalafmtCheckAll). Since our upgrade to 2.5.6 the sbt job get stuck and after 30min it errors out with

org.scalafmt.sbt.ScalafmtSbtReporter$ScalafmtSbtError: scalafmt: Missing config [/builds/enya-dap-core/source2landing/database-ingest/.scalafmt.conf]

The message is confusing because there is a .scalafmt.conf

The change was:

reverting back to 2.5.5 solves the issue.

Due to this long waiting period of 30min and the fact that it works on my local machine (sbt 1.9.7) where I have full internet access, I assume that some download is intended to happen from a location we have no access to.

Any ideas on what has changed here?

The log for 2.5.6 looks like this:

++ sbt --timings -Denv=dev -DbuildTimestamp=20251106111349 clean update scalafmtCheckAll test publish assembly doc

downloading sbt launcher 1.10.11
copying runtime jar...
[info] Updated file /builds/enya-dap-core/source2landing/database-ingest/project/build.properties: set sbt.version to 1.10.11
[info] welcome to sbt 1.10.11 (Amazon.com Inc. Java 17.0.17)
[info] loading settings for project database-ingest-build from plugins.sbt...
[info] loading project definition from /builds/enya-dap-core/source2landing/database-ingest/project
[info] compiling 1 Scala source to /builds/enya-dap-core/source2landing/database-ingest/project/target/scala-2.12/sbt-1.0/classes ...
[info] done compiling
[info] Total time: 15005 ms
...
[info] scalafmt: Checking 11 Scala sources (/builds/enya-dap-core/source2landing/database-ingest)...
[info] Total time: 1938827 ms
[info]   database-ingest / Compile / scalafmtCheck                                : 1938776 ms
[info]   database-ingest / Test / scalafmtCheck                                   :  600010 ms
[info]   database-ingest / Test / unmanagedSources / allInputPathsAndAttributes   :      48 ms
[info]   database-ingest / Compile / unmanagedSources / allInputPathsAndAttributes:      24 ms
[info]   database-ingest / Compile / unmanagedSources / inputFileStamps           :      11 ms
[info]   database-ingest / IntegrationTest / scalafmtCheck                        :       3 ms
[info]   database-ingest / Test / unmanagedSources / inputFileStamps              :       2 ms
[error] org.scalafmt.sbt.ScalafmtSbtReporter$ScalafmtSbtError: scalafmt: Missing config [/builds/enya-dap-core/source2landing/database-ingest/.scalafmt.conf]
[error] (Test / scalafmtCheck) org.scalafmt.sbt.ScalafmtSbtReporter$ScalafmtSbtError: scalafmt: Missing config [/builds/enya-dap-core/source2landing/database-ingest/.scalafmt.conf]
[error] Total time: 1939 s (0:32:19.0), completed Nov 6, 2025, 11:46:43 AM
Total time: 1964699 ms

With 2.5.5 scalafmt finishes in 6secs

...
[info] scalafmt: Checking 11 Scala sources (/builds/enya-dap-core/source2landing/database-ingest)...
[info] scalafmt: Checking 5 Scala sources (/builds/enya-dap-core/source2landing/database-ingest)...
[info] Total time: 5636 ms
[info]   database-ingest / Compile / scalafmtCheck                                : 5570 ms
[info]   database-ingest / Test / scalafmtCheck                                   : 5369 ms
[info]   database-ingest / Test / unmanagedSources / allInputPathsAndAttributes   :   31 ms
[info]   database-ingest / Compile / unmanagedSources / allInputPathsAndAttributes:   31 ms
[info]   database-ingest / Test / unmanagedSources / inputFileStamps              :    7 ms
[info]   database-ingest / Compile / unmanagedSources / inputFileStamps           :    3 ms
[info]   database-ingest / IntegrationTest / scalafmtCheck                        :    3 ms
[success] Total time: 6 s, completed Nov 6, 2025, 10:58:29 AM

Thanks a lot, Christian

[kitbellew]

please provide a small project that is fully reproducible within your environment. it needs to include build.sbt, .scalafmt.conf and any SBT or Coursier environment variables you use.

[kitbellew]

more specifically: how do you tell your build where to get dependencies?

[christianknoepfle]

I can put a project together but specifically to the dependencies we do the following:

in the ci pipeline we always create sbt-cache/repositories to redirect everything to our enterprise artifactory (which mirrors maven etc)

# see https://www.scala-sbt.org/1.x/docs/Proxy-Repositories.html for details, especially chapter "sbt Configuration"
mkdir -p $CI_PROJECT_DIR/sbt-cache
echo "
[repositories]
  local
  artifactory-sbt-plugins: https://my.artifactory.de:443/artifactory/sbt-plugin-releases-remote/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  artifactory-maven: https://my.artifactory.de:443/artifactory/Maven-Central-remote/
  enya-prod: https://my.artifactory.de/artifactory/enya-sbt-prod-local
  enya-stage: https://my.artifactory.de/artifactory/enya-sbt-stage-$TARGET_ENV-local" > $CI_PROJECT_DIR/sbt-cache/repositories
echo "creating repositories file in $CI_PROJECT_DIR/sbt-cache/"

then we set SBT_OPTS

export SBT_OPTS="-Dsbt.global.base=$CI_PROJECT_DIR/sbt-cache/ 
    -Dsbt.boot.directory=$CI_PROJECT_DIR/sbt-cache/boot 
    -Dsbt.ivy.home=$CI_PROJECT_DIR/sbt-cache/ivy 
    -Dsbt.coursier.home=$CI_PROJECT_DIR/sbt-cache/coursier 
    -Dsbt.override.build.repos=true"

then we call sbt

in build.sbt the resolvers are like this

ThisBuild / resolvers ++= Seq(
  "Artifactory releases".at("https://my.artifactory.de/artifactory/enya-sbt-prod-local"),
  "Artifactory dev".at("https://my.artifactory.de/artifactory/enya-sbt-stage-dev-local")
)

Does that help or should I still put a sample repo together.

Thanks a lot, Christian

[kitbellew]

... it needs to include ... .scalafmt.conf and ...

[mino5]

@christianknoepfle I’m not sure if it’s the same issue in your case, but I had it stuck for quite a while too, and upgrading Java from 22 to 25 resolved it for me. Of course, that upgrade might not be an option on your end, but I thought I’d share just in case it helps.

[kitbellew]

org.scalafmt.sbt.ScalafmtSbtReporter$ScalafmtSbtError: scalafmt: Missing config [/builds/enya-dap-core/source2landing/database-ingest/.scalafmt.conf]

The message is confusing because there is a .scalafmt.conf

Let me repeat. What does that file contain?

As it stands, I don't know what has changed to behave like this. Anything else special to your environment? Credentials for these custom repositories? Are they fully up-to-date with external repos?

The only thing I can think of as a culprit is this change: scalameta/scalafmt#4962. However, to prove this, you'd need to use the scalafmt binary v3.9.10 or later (NOT the sbt plugin, https://scalameta.org/scalafmt/docs/installation.html#cli) and provide these custom repositories the way coursier expects you to (https://get-coursier.io/docs/other-repositories, but don't ask me how to use it, I don't know.)

[sageserpent-open]

The only thing I can think of as a culprit is this change: scalameta/scalafmt#4962.

Hmm, I have observed a build problem due to binary incompatibility in Kinetic Merge that may be related to this ticket.

Upgrading sbt-scalafmt from 2.5.5 to 2.5.6 (sageserpent-open/kineticMerge@f4ae3cb) provoked this build error when the project's build.sbt attempts to invoke Coursier to package up the application:

Packaging executable with version: 1.10.1-SNAPSHOT
[error] java.lang.NoSuchMethodError: 'java.lang.String scala.cli.config.Key$StringListEntry$.$lessinit$greater$default$3()'
[error] 	at coursier.cli.config.Config$.<init>(Config.scala:17)
[error] 	at coursier.cli.config.Config$.<clinit>(Config.scala)
[error] 	at coursier.cli.Coursier$.<init>(Coursier.scala:35)
[error] 	at coursier.cli.Coursier$.<clinit>(Coursier.scala)
[error] 	at $09b33f8636d8dc3ad626$.$anonfun$root$14(build.sbt:74)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error] 	at sbt.Execute.work(Execute.scala:292)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error] 	at java.base/java.lang.Thread.run(Thread.java:1583)
[error] (packageExecutable) java.lang.NoSuchMethodError: 'java.lang.String scala.cli.config.Key$StringListEntry$.$lessinit$greater$default$3()'

The SBT build depends on Coursier via a meta-definition in project/build.sbt:

ThisBuild / libraryDependencies += "io.get-coursier" %% "coursier-cli" % "2.1.14"

I presume I'm seeing a classic dependency hell of incompatible transitive dependencies, in my case the one in the project meta-build and the one in the ScalaFmt plugin.

Possibly something similar is happening for @christianknoepfle , causing the build to hang (ie. not a network issue at all)?

FYI: invocation of Coursier from Kinetic Merge's build.sbt:

      coursier.cli.Coursier.main(
        s"bootstrap --verbose --bat=true --scala-version ${scalaBinaryVersion.value} -f $localArtifactCoordinates -o $executablePath"
          .split("\\s+")

[kitbellew]

Hmm, I have observed a build problem due to binary incompatibility in Kinetic Merge that may be related to this ticket.

[error] java.lang.NoSuchMethodError: 'java.lang.String scala.cli.config.Key$StringListEntry$.$lessinit$greater$default$3()'

you should report this problem in the repository which maintains scala-cli, not here.

[sageserpent-open]

I presume I'm seeing a classic dependency hell of incompatible transitive dependencies, in my case the one in the project meta-build and the one in the ScalaFmt plugin.

I don't think Scala CLI is the problem here, rather the dependencies that lead to it. I could well be wrong, though.

Anyway, I'm alright with reverting the upgrade; good luck with your problem solving!

[kitbellew]

ThisBuild / libraryDependencies += "io.get-coursier" %% "coursier-cli" % "2.1.14"

I presume I'm seeing a classic dependency hell of incompatible transitive dependencies, in my case the one in the project meta-build and the one in the ScalaFmt plugin.

perhaps you are correct, @sageserpent-open . scalafmt-dynamic (which sbt-scalafmt uses) is built with coursier 2.1.24 which has been stable for the past 11 months (while 2.1.14 became 2.1.24 in only 3 months). have you tried upgrading your version of cousier?

[sageserpent-open]

have you tried upgrading your version of cousier?

@kitbellew: yes, I had tried that prior to commenting on this ticket. The reason for choosing 2.1.14 was that SBT invokes Coursier explicitly in the project’s build.sbt, and SBT requires Scala 2.12 for code that it executes directly - more recent versions of Coursier are only produced for 2.13 and 3.

I’m waiting for SBT to move on to 2.13, at which point the upgrade can take place. I may be wrong about the enforced dependency on Scala 2.12 - happy to be contradicted there.