Server Failing to Start After Latest

[Mike-E-angelo]

Hi Team Seq,

After many many months of flawless uptime I have encountered an issue with the latest image on my Azure Container Instance.

I noticed that my ACI had a failed instance after numerous restarts about an hour ago (the restarts occurred earlier this morning).

After stopping the container group and restarting, the instance is up but I am getting output in the logs line the following:

[16:49:04 INF] Check alert alert-44 (64.507 ms)
[16:49:04 INF] Check alert alert-45 (11.598 ms)
[16:49:05 INF] HTTP GET /api/diagnostics/status (172309.065 ms)
[16:49:05 INF] HTTP GET /api/settings/setting-checkforupdates (0.436 ms)
[16:49:05 INF] HTTP GET /api/signals/ (1.652 ms)
[16:49:05 INF] HTTP GET /api/sqlqueries/ (0.352 ms)
[16:49:05 INF] HTTP GET /api/sqlqueries/ (0.399 ms)
[16:49:05 INF] HTTP GET /api/signals/ (2.01 ms)
[16:49:11 INF] HTTP GET /api/diagnostics/status (177953.222 ms)
[16:49:11 INF] HTTP GET /api/workspaces/ (3.258 ms)
[16:49:11 INF] HTTP GET /api/sqlqueries/ (0.58 ms)
[16:49:12 INF] HTTP GET /api/signals/ (3.73 ms)
[16:49:18 INF] HTTP GET /api/diagnostics/status (184178.391 ms)
[16:49:24 INF] HTTP GET /api/licenses/license-server (187489.498 ms)
[16:49:24 INF] HTTP GET /api/diagnostics/status (190433.198 ms)
[16:49:24 INF] HTTP GET /api/sqlqueries/ (0.539 ms)
[16:49:24 INF] HTTP GET /api/workspaces/ (0.264 ms)
[16:49:24 INF] HTTP GET /api/workspaces/ (1.026 ms)
[16:49:24 INF] HTTP GET /api/expressionindexes (0.284 ms)
[16:49:24 INF] HTTP GET /api/sqlqueries/ (0.464 ms)
[16:49:24 INF] HTTP GET /api/signals/ (3.062 ms)
[16:49:30 INF] HTTP GET /api/diagnostics/status (190830.451 ms)
[16:49:30 INF] HTTP GET /api/workspaces/ (0.545 ms)
[16:49:30 INF] HTTP GET /api/settings/setting-checkforupdates (0.465 ms)
[16:49:34 INF] Check alert alert-44 (73.011 ms)
[16:49:34 INF] Check alert alert-45 (12.923 ms)
[16:49:37 INF] HTTP GET /api/diagnostics/status (192814.283 ms)

Notice the large amount of time in the /api/diagnostics/status calls.

Note that HTTPS calls to load the console from a browser simply spins (but does not timeout).

I would greatly appreciate any pointers on how to diagnose/address this.

For further context, the following was the image pulled:

Successfully pulled image "index.docker.io/datalust/seq@sha256:671cd1b032ff724911e4784054fc9c9c07b07d282465999e882b0a3511ae2867"

Thank you for any assistance you can provide.

[Mike-E-angelo]

Well the HTTPS console loads now with the glowing pretty colors so that's an improvement, but nothing else loads.

Seeing this in logs:

[17:06:59 INF] HTTP GET /api/diagnostics/status (343471.949 ms)
[17:07:06 INF] HTTP GET /api/diagnostics/status (343315.579 ms)
[17:07:07 INF] Check alert alert-44 (26.878 ms)
[17:07:07 INF] Check alert alert-45 (10.95 ms)
[17:07:13 INF] HTTP GET /api/diagnostics/status (342255.552 ms)
[17:07:19 INF] HTTP GET /api/licenses/license-server (626586.746 ms)
[17:07:19 INF] HTTP GET /api/licenses/license-server (620113.224 ms)
[17:07:19 INF] HTTP GET /api/licenses/license-server (335181.854 ms)
[17:07:19 INF] HTTP GET /api/diagnostics/status (342049.869 ms)

[Mike-E-angelo]

Woohoo! After stopping and restarting a few times, it now works 🙏🙏🙏 I think it was an update to 2025.2, and that threw it for a loop for some reason. Everything is fine now, but if anyone wants to further look into the above wait times I am happy to help where I can. 👍

Thank you for all your efforts out there. Outside of this Seq works so very very well, and I never have a problem.

[liammclennan]

It depends on many things, but you can see where indexing is up to on the Data > Storage screen. Note that retention policies require data to be re-indexed.

[Mike-E-angelo]

Thank you for that @liammclennan. This is what I see:

When visiting Indexes, this is what I see:

If I understand correctly, it appears there has been work done, but I do not see where it says how much left there is to do. 🤔 Any further guidance would be appreciated so that I know when I should start to worry. :)

[nblumhardt]

Thanks for digging in, @Mike-E-angelo!

That storage/those indexes all look fine to us, so we might need to dig in deeper. There's no way that query should take 35 seconds over 7 days - definitely something odd going on.

Since we're not sure where the issue lies at this point, perhaps switching to a support@datalust.co thread would be a good next step - we'll need to confirm more information about the container (any metrics available from the Azure portal?), the query (how many hits do you end up with, and how far back in time are they?), and the distribution of timings (is every run of the query slow, and what's the spread of timings over a few attempts?). A diagnostic report (now under the Support menu) would also help flesh out the picture.

If we can narrow it down to a general area we'll be able to post a more useful issue description from there.

[nblumhardt]

Just one more quick note, we're pushing out a new 2025.2 patch today, https://github.com/datalust/seq-tickets/milestone/181?closed=1, which includes a fix for an indexing throughput regression. It's not especially likely to be the root cause of the problem here, but it's probably also worth grabbing the fix just to make sure there's no secondary effect of that bug slowing things down for you.

[Mike-E-angelo]

Thank you for that, @nblumhardt ... now the query completes in 15 seconds. 😁 I can confirm the behavior is the same each time. Before with the previous version it took about 10 seconds for the first hit to display (from today), and it would slowly walk through the others until the 35 seconds was up.

Here it seems now that the first few show after a few seconds and then the rest all complete shortly thereafter at the 15 second mark.

I will send an email and we can take it from there. 👍