[BUG]: Site Agent gRPC client is `nil` in workflows when it fails to connect to Core gRPC server

[thossain-nv]

Version

1.0.5

Which installation method(s) does this occur on?

No response

Describe the bug.

Currently Site Agent pods don't continue to retry connecting to Core gRPC if the initial attempt fails. However, it still continues to register workflows with Temporal and receives them. When executing those workflows, the code encounters a nil client and throws exception.

Minimum reproducible example

Bring Site Agent up locally without the mock Core server and attempt to create a VPC.

Relevant log output

Failed to execute sync workflow to create Instance on Site: workflow execution error (type: CreateInstanceV2, workflowID: instance-create-99328833-74c9-41cd-bcf7-745a387f4133, runID: a66ba286-8478-49b9-8f94-d1bceefbc278): activity error (type: CreateInstanceOnSite, scheduledEventID: 5, startedEventID: 6, identity: 1@elektra-site-agent-2@): interface conversion: interface {} is nil, not *client.CarbideClient

Other/Misc.

No response

Code of Conduct

• I agree to follow Carbide's Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report