mutually incompatible python modules installed on debian 12

[cg2v]

My site is building a cluster with a3mega nodes using cluster toolkit. The a3mega samples build a custom cluster image based on debian 12 rather than using an existing slurm image.
I discovered that the google api python modules installed in /usr/local are not fully compatible with each other (secret manager needs a newer version of google-auth)

I traced the installation of those python modules to this repo. The problematic line in the requirements seems to be:
google-cloud-bigquery~=2.0
On Debian 11 and many other systems, this resolves to google-cloud-bigquery-2.34.4 and google-auth-2.37 is used, which is fine

Debian 12 has python 3.11. Many versions of google-cloud-bigquery 2.x declare themselves incompatible with python 3.10 and 3.11.
The dependency resolver chooses google-cloud-bigquery-2.6.1 to satisfy google-cloud-bigquery~=2.0 which presumably is the last version that didn't specify python version dependencies. This in turn causes google_auth-1.35.0 to be used, which is NOT fine. secret-manager expects credentials to have a universe_domain, and 1.35 doesn't include that.

Debian 12 at least needs to be able to use google-cloud-bigquery 3.x

[mr0re1]

Thank you @cg2v , this is great write up!

We are working on general solution for this issue (uniform Python+deps), but it will not be resolved soon.
Meanwhile, to unblock you, I would recommend to try using "vanilla" SlurmGCP image for controller.
E.g. add following to the controller module settings:

  instance_image:
    project: schedmd-slurm-public
    family: slurm-gcp-6-8-hpc-rocky-linux-8