Operator unable to deploy Spinnaker when istio-sidecar injection is enabled

[ghq2340]

Spinnaker deployed with this operator is unable to start when istio-sidecar injection is enabled (kubectl label namespace spinnaker istio-injection=enabled) our Kubernetes cluster is istio-mesh enabled so this will be a default configuration. If spinnaker is deployed with Halyard (without operator) all pods come up successfully. Since Halyard is able to deploy the cluster while istio-injection is enabled, is there anything that needs to be configured to be able to use Istio sidecars?

Tested versions:
Istio = 1.11.1-1.11.3
Kubernetes= 1.20
Spinnaker= 1.26.6

spinnaker            spin-fiat-656c594998-6wtbz                1/2     CrashLoopBackOff   22         107m
spinnaker            spin-clouddriver-74c5564645-lvhlr         1/2     Running            0          107m
spinnaker            spin-deck-c664db795-q2t2v                 2/2     Running            0          107m
spinnaker            spin-rosco-5d7cc8df77-nx2cj               1/2     Running            0          107m
spinnaker            spin-redis-59fb75c8f8-4gjz7               2/2     Running            0          107m
spinnaker            spin-front50-d4859897f-jt4vn              2/2     Running            0          107m
spinnaker            spin-echo-6dffd76ccb-c4pcz                2/2     Running            0          107m
spinnaker            spin-gate-69fcdbbdb8-z26b7                1/2     Running            0          107m
spinnaker            spin-orca-84768c79bf-x9fzf                1/2     CrashLoopBackOff   21         107m
spinnaker-operator   spinnaker-operator-69544556d-55tpw        3/3     Running            1          115m

NAME        VERSION   LASTCONFIGURED   STATUS    SERVICES   URL
spinnaker   1.26.6    98m              Failure   9

Logs:
clouddriver.log
fiat.log
orca.log
rosco.log
gate.log

[jaypz]

Try turning off sidecar injection on spin-redis, redis seems to not play well with a sidecar and prevents other services from connecting

[celiawa]

Hi @jaypz, I'm encountering the same issure when deployed via Operator and isito enabled. But it's strange that the pods are running welI when deployed via pure Halyard, also with istio injection and redis pod. Do you have any idea.

[kushwahagHub]

Hi @jaypz - I am getting the exact same issue when deploying via operator. Please can you help us finding a resolution for this issue?

[jaypz]

Try annotating spin-redis if you use it to turn off sidecar injection, you may also want to try using PERMISSIVE mTLS peerauth policy for spinnaker services

[kushwahagHub]

Hi @jaypz - thanks for your suggestions, tried both turned off sidecar on redis pod and also enabled PERMISSIVE mTLS on spinnaker namespace but that didn't make any difference. The error I am getting in orca pod is something like this:

Failed to instantiate [com.netflix.spinnaker.q.redis.RedisQueue]: Factory method 'queue' threw exception; nested exception is redis.clients.jedis.exception.JedisConnectionException: java.net.SocketException: Connection reset

[sandstheman]

We've been hitting the same problem and figured out a workaround. The spin-redis service is deployed with the name value of the port set to http, switching it to tcp and creating the relevant virtualservice and detinationrule and we have spinnaker running with the out of the box redis pod with istio sidecar:

the service we are applying to correct this is as below

apiVersion: v1
kind: Service
metadata:
  labels:
    app: spin
    cluster: spin-redis
  name: spin-redis
  namespace: spinnaker
spec:
  ports:
  - name: tcp
    port: 6379
    protocol: TCP
    targetPort: 6379
  selector:
    app: spin
    cluster: spin-redis
  sessionAffinity: None
  type: ClusterIP

The virtual service is thus:

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: spin-redis
  namespace: spinnaker
spec:
  hosts:
    - spin-redis
  tcp:
  - match:
    - port: 6379
    route:
    - destination:
        host: spin-redis
        port:
          number: 6379

[jaypz]

@sandstheman good find, it seems this might be issue with halyard then & the manifests it applies?