Lock acs_snapshot_data while writing

[rautenrieth-da]

Fixes https://github.com/DACH-NY/canton-network-internal/issues/1782

[meiersi-da]

Thanks. Looks good to me, but I'm not deep enough in the code to judge whether it is really safe.

[ray-roestenburg-da]

I don't like locks so you need to convince me there is no other option :-) (looks like I misread 'lock' vs 'advisory lock')

[OriolMunoz-da]

will it though? We've seen queries stuck for longer than 15s after a pod kill/restart

[meiersi-da]

@nmarton-da: I remember that there was some trickery around keep alives that you figured out as part of the HA config for the IndexDB. Something around clients and servers having to set the right params? Do you remember?

[meiersi-da]

Actually, this ChatGPT convo has quite a bit of interesting info: https://chatgpt.com/share/68dbb7bd-48c8-8007-85e9-ae43fa3a5b54

Seems like it's worth checking our keep-alive configs for server and client -- independently of this PR.

[rautenrieth-da]

Added #2488 for double checking disconnect behavior, and leaving the advisory lock. We'd rather have stray locks than a corrupted database.

[ray-roestenburg-da]

Thanks!

[rautenrieth-da]

Actual behavior on CILR:

2025-10-02 07:11:52.069Z: trigger runs (never finishing)
2025-10-02T07:13:18.317Z: scan shuts down
2025-10-02 07:13:51.000Z: scan restarts
2025-10-02 07:15:14.951Z: trigger runs again on the same snapshot record time
2025-10-02 07:15:14.973Z: trigger fails with java.lang.Exception: Failed to acquire exclusive lock (not retrying the error)
2025-10-02 07:15:46.134Z: trigger runs again on the same snapshot record time
2025-10-02 07:19:12.841Z: trigger succeeds

I think that's enough to conclude that it takes a few minutes to release the lock, potentially leading to log warnings, but the trigger does not deadlock itself.