High CPU Usage - limit threads per SILO instance ?

[gordonkoehn]

For complex queries, i.e., looking for co-occurrences, I have noticed some longer load times, and gaps in the LAPIS actuator monitoring, and extraordinarily high CPU usage. See below.

I know that currently we have set LAPIS-SILO to use available threads. Perhaps we should adjust this to ensure responsiveness for the other services.

General Machine Stats

LAPIS-SILO (Covid)

[gordonkoehn]

@Taepper, what do you think? This is expected behaviour, right? Will send the queries in an hour.

[Taepper]

Yes, this seems expected. Your solution sounds sensible. This is the flag needed for SILO:

  --api-threads-for-http-connections i32
    Env var : SILO_API_THREADS_FOR_HTTP_CONNECTIONS
    YAML key: api.threadsForHttpConnections

    The number of worker threads. If set to 0 it will be set to the number of processors.

    Default: 0

which is currently explicitly set to 96:

WisePulse/roles/srsilo/templates/docker-compose.yml.j2

Line 27 in e47a2d7

command: api --api-threads-for-http-connections 96 --api-max-queued-http-connections 1024

[gordonkoehn]

OK, perfect - then I will adjust this to something like 86, so reserve 10 cores for other services.

[gordonkoehn]

But we also set it in the database config:

WisePulse/roles/srsilo/files/database_config.yaml

Line 28 in 8a0b21e

siloClientThreadCount: 96

Is one of these redundant?

[Taepper]

siloClientThreadCount: 96

Uh, this might be a LAPIS option @fhennig

I guess this can be misleading

[fhennig]

Ah yes that's used for LAPIS, basically how many connections are made to SILO, so it would be good to set this to the same number as the CLI flag, I think.

[gordonkoehn]

Handover note: As lookback duration increases (#156), CPU usage will grow. Consider limiting SILO threads to maintain responsiveness for other services.