Expose management http endpoint for K8S startup probe

[ppawlowski]

Description

The goal of this task is to implement an HTTP endpoint that will return a HTTP 200 status once installation of all node packages is completed successfully, 400 otherwise.
Preferably, it should be exposed on the management port.
Once implemented, endpoint might be used to configure Kubernetes probes.

Having a deployment's startup probe configured to this endpoint will decrease the unavailability of a Node-RED application during instance restart.
This should be considered as a part of FlowFuse/flowfuse#5554 implementation.

Requested By

• Customer name and/or link to HubSpot contact

[joepavitt]

@ppawlowski can you add some detail to this issue for the why please. Not clear to me what we/customer gets out of this item being delivered.

Once implemented, endpoint might be used to configure Kubernetes probes.

Alludes to what it unlocks, but I'm naive as to what value this brings?

[ppawlowski]

@joepavitt description extended to include a customer value

[joepavitt]

Having a deployment's startup probe configured to this endpoint will decrease the unavailability of a Node-RED application during instance restart.

Can you clarify this please, I'm not sure I understand still. Are you saying that by adding a single HTTP endpoint, the time it takes to restart reduces? How does this compare against how we do things now?

[ppawlowski]

Can you clarify this please, I'm not sure I understand still.

Currently, Kubernetes does not have insight into whether the Node-RED instance has fully installed required packages during startup. This causes Kubernetes to mark the container as "ready" too early, leading to extended downtime.
By configuring Kubernetes startup probes to use this new endpoint, Kubernetes will be able to detect when the application is (almost) ready to serve traffic.

Are you saying that by adding a single HTTP endpoint, the time it takes to restart reduces?

While the total restart time of the Node-RED instance is not itself shortened by the endpoint, its accurate readiness signalling reduces unavailability by ensuring traffic is routed only when the app is ready.

How does this compare against how we do things now?

Compared to the current state, this means improved user experience with less downtime during restarts. Currently, when NR instsance restart is initiated, it will be offline for the time, when packages are installed and flows are started. With startup probe pointing to the endpoint that indicates completion of dependencies installation, unavailability of NR flows will decrease (significantly for some apps with a lot of dependencies and limited CPU resources). Additionally, in case of a any failure during dependencies installation phase, user's application will remain online (from the old replica) instead remain in the restart limbo.

[joepavitt]

Just spoke with Piotr - the value here is mostly constrained to reducing downtime for FlowFuse/flowfuse#5554 - not a general restart.

This is because this would spin up a second Node-RED instance when a restart is scheduled. Once that second instance is then available, the first is torn down, and we switch over. So, whilst the actual restart time is the same, by splitting this across two instances, we disguise the installation period (the longest part of a restart) from the user.

We can also use this when doing Node-RED version upgrades, and when manually suspending/restarting, however, it's worth noting UX is important as the current Instance would still be available for x minutes whilst the second one spins up, which may not be clear to the user having just clicked "Restart" or "Update".

The key value added here too is that by minimising downtime, the flows themselves are operational for longer, e.g subscribing/publishing MQTT into a UNS, you would not be seeing minutes of data missing anymore.

@ppawlowski has agreed also to do some benchmarking so that we can confirm the time reductions for several different insatnces with different levels of dependencies.

[ppawlowski]

Instance details

Size: Medium
Node-RED version: 4.1.0

Install times

Blueprint name	Dependencies install time (s)	Instance available after (s)
"Blank workspace"	0	15
"Getting Started with Dashboard"	71	86
"OEE Dashboard"	89	104

[joepavitt]

Thanks @ppawlowski - so in terms of downtime, we're looking at 15, 86 and 104 respectively. With this work, downtime would be down to 15, 15 and 15 respectively?