Nexus repository database gets corrupted

[cptanalatriste]

✅ Checklist

• I have searched open and closed issues for duplicates.
• This is a problem observed when deploying a Data Safe Haven.
• I can reproduce this with the latest version.
• I have read through the documentation.
• This isn't an open-ended question (open a discussion if it is).

💻 System information

• Operating System: MacOS
• Data Safe Haven version: v5.5.1

📦 Packages

List of packages

Paste list of packages here

🚫 Describe the problem

End users report the impossibility of installing packages with pip:

When checking the logs of the Nexus container instances, it appears the Nexus database repository is corrupted:

com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: File corrupted while reading record: "/nexus-data/db/nexus.mv.db". Possible solution: use the recovery tool [90030-232]

Caused by: org.h2.mvstore.MVStoreException: File is corrupted - unable to recover a valid set of chunks [2.3.232/6]

It appears DSH is using an Embedded H2 database, which is prone to corruption. According to the documentation:

There is a heightened risk of data corruption, particularly if the server is not shut down properly, a power failure occurs, or for larger datasets.

Container instance tend to restart at random, which can explain why corruption tend to happen in multiple production SREs.

As a workaround, you can delete the Nexus container instance as well as the file shares related to it.

🌳 Log messages

See above.

Relevant log messages

Your log details here

♻️ To reproduce

It's very hard to reproduce. Corruption happens at random.

[cptanalatriste]

cc.ing @helendduncan and @llewelld , as they found the issue in a DSG SRE. I have also seeing it in CVD-Net

[llewelld]

After performing the following steps:

1. delete nexus, nexus-allowlists and caddy file shares;
2. delete the container-group-nexus container instances;
3. redeploy;

The problem doesn't seem to be solved (restoring the pip.config and attempting to install pandas fails).

There are similar errors ("Failed to initialize pool") appearing in the logs around the time of the redeploy.

[cptanalatriste]

@llewelld can you check if the DNS records are correct?

[llewelld]

How would I check this? The addresses that the /etc/pip.conf config file points to resolve, but return 404 errors:

$ curl -s -o /dev/null -w "%{http_code}" https://<domain_name>/repository/pypi-proxy
404

(where <domain_name> is the domain name of the DSH).

[cptanalatriste]

FYI, on setting up Nexus with PostgreSQL: https://help.sonatype.com/en/install-nexus-repository-with-postgresql.html