Virtual node pods perpetually restart despite Running state with no errors

[m-cappi]

Describe the Issue
Hi, I am trying to serve my app on virtual nodes, but all my pods persist on a Running state that keeps restarting every minute.
I've tried running them on agent pools and found no issue.
The current logs do not provide any helpful information, and I've been going through the list of known issues but found no match so far.

my-app pod logs

6/27/2023, 1:05:40 PM | my-app-68885f969f-56vdf | b9fea0a1eaad726a9d4bc1d7040eace7c93540a5f2ae1e2571ab0eb42623fc0e | INFO: Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)
12/31/1969, 9:00:00 PM | my-app-68885f969f-56vdf | b9fea0a1eaad726a9d4bc1d7040eace7c93540a5f2ae1e2571ab0eb42623fc0e | ===========PLEASE NOTE THAT THERE MAY BE MISSING DATA HERE.===========
6/27/2023, 1:05:49 PM | my-app-68885f969f-56vdf | b9fea0a1eaad726a9d4bc1d7040eace7c93540a5f2ae1e2571ab0eb42623fc0e | [INFO] 2023-06-27 16:05:49,732 - APP_ROOT_PATH: /app
6/27/2023, 1:05:49 PM | my-app-68885f969f-56vdf | b9fea0a1eaad726a9d4bc1d7040eace7c93540a5f2ae1e2571ab0eb42623fc0e | [INFO] 2023-06-27 16:05:49,733 - API Hydrate mode enabled: true
6/27/2023, 1:05:49 PM | my-app-68885f969f-56vdf | b9fea0a1eaad726a9d4bc1d7040eace7c93540a5f2ae1e2571ab0eb42623fc0e | INFO: Started server process [3715]
6/27/2023, 1:05:49 PM | my-app-68885f969f-56vdf | b9fea0a1eaad726a9d4bc1d7040eace7c93540a5f2ae1e2571ab0eb42623fc0e | INFO: Waiting for application startup.
6/27/2023, 1:06:04 PM | my-app-68885f969f-56vdf | b9fea0a1eaad726a9d4bc1d7040eace7c93540a5f2ae1e2571ab0eb42623fc0e | INFO: Application startup complete.
6/27/2023, 1:06:04 PM | my-app-68885f969f-56vdf | b9fea0a1eaad726a9d4bc1d7040eace7c93540a5f2ae1e2571ab0eb42623fc0e | INFO: Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)
6/27/2023, 1:05:44 PM | my-app-68885f969f-56vdf | b9fea0a1eaad726a9d4bc1d7040eace7c93540a5f2ae1e2571ab0eb42623fc0e | INFO: 10.1.0.57:41152 - "GET /metrics HTTP/1.1" 200 OK
12/31/1969, 9:00:00 PM | my-app-68885f969f-56vdf | b9fea0a1eaad726a9d4bc1d7040eace7c93540a5f2ae1e2571ab0eb42623fc0e | ===========PLEASE NOTE THAT THERE MAY BE MISSING DATA HERE.===========
6/27/2023, 1:06:13 PM | my-app-68885f969f-56vdf | b9fea0a1eaad726a9d4bc1d7040eace7c93540a5f2ae1e2571ab0eb42623fc0e | [INFO] 2023-06-27 16:06:13,226 - APP_ROOT_PATH: /app

The aci-connector-linux is the only thing logging some sort of error, but again it's not helpful:

6/27/2023, 12:57:21 PM | aci-connector-linux-578d774657-8mkdg | 5a3a8c3140df4619261ed58079e2b6c01bfb7394f5f544a40ea16ffc64013822 | time="2023-06-27T15:57:21Z" level=error msg="failed to retrieve pod my-app-68885f969f-56vdf status from provider" error="container my-app properties CurrentState StartTime cannot be nil" method=PodsTracker.processPodUpdates node=virtual-node-aci-linux operatingSystem=Linux provider=azure watchedNamespace=
6/27/2023, 12:58:09 PM | aci-connector-linux-578d774657-8mkdg | 5a3a8c3140df4619261ed58079e2b6c01bfb7394f5f544a40ea16ffc64013822 | time="2023-06-27T15:58:09Z" level=error msg="failed to retrieve pod my-app-68885f969f-56vdf status from provider" error="container my-app properties CurrentState StartTime cannot be nil" method=PodsTracker.processPodUpdates node=virtual-node-aci-linux operatingSystem=Linux provider=azure watchedNamespace=

Do you have any tips on how to solve this or where I should be looking for the actual error?
Thanks!

[helayoty]

@m-cappi, thanks for contacting us. Please provide more information about the issue.

• What is the virtual node version?
• Full logs for the aci-connector-linux pod
• Your deployment template.