[Bug]: I keep disconnected on the UI with OIDC and 2+ nodes

[quentin9696]

zot version

digestID: 227a95be907ee20a322cb241f43f2c515affa53b1be71909d7f09afa5ff5b6a4

Describe the bug

I try to run zot in cluster mode. I try both cluster and scale out cluster and having the same issue.

I configure zot, with the UI, and OIDC/Oauth2 authentication. I use S3 as remote storage and dynamoDB as cache.
Everything is running fine without clustering.

When I start running zot in a clustering mode, I randomly get disconnected from the UI. Sometimes, I can browse the UI for 30 secondes without issue, sometimes, it loop refresh the page and I'm redirect to the zot login page.

I use AWS Network Load balancer. I also try with and without sticky cookie session, with no luck.

I think zot is missing a session share between nodes.

To reproduce

Configuration used on both node:

{
  "distSpecVersion": "1.1.1",
  "storage": {
      "rootDirectory": "/tmp/zot",
      "dedupe": true,
      "remoteCache": true,
      "storageDriver": {
          "name": "s3",
          "rootdirectory": "/",
          "region": "us-east-1",
          "bucket": "REDACTED",
          "secure": true,
          "skipverify": false
      },
      "cacheDriver": {
          "name": "dynamodb",
          "endpoint": "https://dynamodb.us-east-1.amazonaws.com",
          "region": "us-east-1",
          "cacheTablename": "ZotCacheTablet",
          "repoMetaTablename": "ZotRepoMetadataTable",
          "imageMetaTablename": "ZotImageMetaTable",
          "repoBlobsInfoTablename": "ZotRepoBlobsInfoTable",
          "userDataTablename": "ZotUserDataTable",
          "apiKeyTablename": "ZotApiKeyTable",
          "versionTablename": "ZotVersion"
      }
  },
  "http": {
      "address": "0.0.0.0",
      "port": "5000",
      "compat": ["docker2s2"],
      "externalUrl": "https://REDACTED",
    "realm": "zot",
    "auth": {
      "openid": {
        "providers": {
          "oidc": {
            "name": "Corporate SSO",
            "issuer": "https://REDACTED",
            "clientid": "REDACTED",
            "clientsecret": "REDACTED",
            "scopes": ["openid", "profile", "email", "groups"]
          }
        }
      },
      "failDelay": 5
    },
    "accessControl": {
      "repositories": {
        "foo/**": {
          "policies": [
            {
              "groups": ["zot-admin"],
              "actions": ["read", "create"]
            }
          ]
        }
      },
      "adminPolicy": {
        "groups": ["admin"],
        "actions": ["read", "create", "update", "delete"]
      }
    }
  },
  "log": {
      "level": "debug"
  },
  "extensions": {
    "ui": {
      "enable": true
    },
    "search": {
      "enable": true
    }
  }
}

2. Run this config (replace REDACTED field by your value) on 2 nodes
3. Try to login on zot UI. Login will work fine, and novigate through the UI. Sometimes, it will disconnect you and redirect to the login page. You can also multiple F5 in the web browser to trigger the issue

Expected behavior

No deconnection should happened until the max session time is over

Screenshots

No response

Additional context

No response

[eusebiu-constantin-petu-dbk]

It happens because we are using in memory session storage when storage is remote, @rchincha do you remember why went this way?

https://github.com/project-zot/zot/blob/main/pkg/api/cookiestore.go#L63

[quentin9696]

Hi !

Thanks for you reply.
As I understand, the cookie is store under storage.rootDirectory. In my case, this folder is set to /tmp/zot. However, when I try to login, I don't see any _session folder created.

I made a test by removing both remote cache and remote storage and I'm able to see my _session cookies in /tmp/zot. My question is: where the the cookies are stored in case of remote storage, since my /tmp/zot folder remain empty.

Any idea?