More aggressive readiness probes

[stas-sbi]

While having a comprehensive monitoring is a good thing I believe that obvious problems should manifest themselves in obvious ways. More sophisticated tooling comes especially useful when the problem is already detected and needs further investigation. This approach does not only help during initial setup but also during upgrades, experiments, when an extensive monitoring is not setup yet or when it (extensive monitoring) just fails for whatever reason.

Some of SV components (such as sequencer) become ready and get exposed on a service endpoints before the port which is supposed to serve the traffic opens. In case of sequencer this is done to make sure that another port becomes accessible before sequencer is initialized which is used to initialize sequencer.

What if services are split into two, "init" and "serve", where "init" would be exposed as soon as container live and "serve" would be exposed only when serving port is ready? This both ports would be accessible when they are needed and the app would only become ready when it's initialized and ready to serve traffic?

[moritzkiefer-da]

What if services are split into two, "init" and "serve", where "init" would be exposed as soon as container live and "serve" would be exposed only when serving port is ready? This both ports would be accessible when they are needed and the app would only become ready when it's initialized and ready to serve traffic?

They already are. The "init" API is on port 5009, the "serve" API is on port 5008. But a readiness probe is per pod not per service so this doesn't help. You could overwrite the k8s behavior that it doesn't route to non-ready ports but imho that is more sketchy than useful. Or we could try to split it into two separate pods but that is a very large undertaking and isn't going to happen anytime soon.

[stas-sbi]

They already are. The "init" API is on port 5009, the "serve" API is on port 5008

They are on different ports but the same kubernetes service

You could overwrite the k8s behavior that it doesn't route to non-ready ports but imho that is more sketchy than useful. Or we could try to split it into two separate pods but that is a very large undertaking and isn't going to happen anytime soon.

There is a simpler way - splitting the service into two:

• global-domain-sequencer-init (ignores readiness probe)
• global-domain-sequencer-serve (normal service which doesn't ignore readiness probe)

[moritzkiefer-da]

Right, I'm still not sold on ignoring readiness probes in services though.

[stas-sbi]

I'm closing this issue for now for now as the original issues which made me create this request don't seem to manifest themselves as well as the suggested by me solution not being without trade-offs. We'll come back and re-think the approach if this becomes an issue again.