Start-up Issues, Readiness check failing

[dibstern-work]

Start-up Issues, Readiness check failing in prod-like

We've had issues with starting our Seq containers in Kubernetes. For some reason, in one of our GCP environments (in prodlike only, not prod) the seq pod shuts down when starting up, after not responding to its health check.

• Environment: GCP K8s Deployment, prod-like instance of Seq only (not our prod instance); both use the same deployment scripts and variables, the only difference is the actual environment to which it's deployed. This environment is working perfectly well for our many other services.
• Problem timeline:
  • Pod starts up, event store opened, ingestion enabled, metastore opened, storage subsystem available, seq is listening
  • seqcli 'started'
  • Either
    1. Failure Mode: 5m 42s later, we see a Termination signal received, pod is recreated and restarts; OR
       • Usually this happens a few times
       • Readiness check fails; you can actually exec into the pod, and even locally seq doesn't accept connections for its health check (it times out on curl).
    2. Success Mode: Less than 5m 42s later, we have some log in GCS, either e.g. Free space check or Metrics sampled or user logged in successfully, etc., and the pod is now alive and doesn't shut down.
       • Usually only after a few failures

The working, production deploy logs

Failing Deploy on 2025-02-18, eventually working

Failing Deploy on Feb 5, eventually working

Please let me know if I can get any more helpful data to help you understand what's happening.

Thank you so much.

[nblumhardt]

Hi @davidsr2r, thanks for getting in touch.

My first guess is that the container's memory or CPU allocation is too low. In Kubernetes it's fairly easy to inadvertently leave the container starved by specifying low request values, despite higher limits, since placement may lead to the lower of these values being used.

The very slow free space check is also slightly suspicious - without knowing anything about the volume of data involved it's hard to say, but this could be a sign that the disk class is inadequate, or insufficient IOPS are available.

Checking over these things, as well as checking the amount of stored data in your Seq instance's Data > Storage page might offer some clues. If not, sending a diagnostic report from the Seq instance through to our support address would help us dig in further.

Hope this helps,
Nick

[dibstern-work]

Thanks Nick! I did try increasing the memory (prod had more) but it made no difference.

I then came across this message you posted in 2021:

This will happen if the filesystem takes too long to return free disk space information. If your /data volume is mounted to a networked filesystem, for example, free space checking will sometimes require enumerating every file in every directory, which has a severe performance impact.

Our /data volume is mounted to a networked file system for backup storage, and free space checking is doing either whatever you referred to here, or something else which makes it take forever and prevents it from starting up before the timeout. I disabled the check using the env variable, and it's now working just like prod.

Thanks for your help, your very fast response pointed me in the right direction :)

[nblumhardt]

Fantastic - thanks for sharing your conclusions! 👍