Scheduled no cleanup_query_results

[Armadik]

I noticed that periodically cleaning of unused query results does not start. Restarting Scheduled helps

At the moment of the problem, I stop receiving metrics redash_db_users_update, redash_db_queries_update redash_db_query_results_delete, maybe some others.

There is almost no information in the scheduledworker logs, maybe you have some ideas?

[2024-10-18 11:48:40,495][PID:7][INFO][redash.tasks.schedule] Scheduling 305c0cae0a196ae96915fd2b6f81001c435aad65 (refresh_queries) with interval 30.
[2024-10-18 11:48:40,497][PID:7][INFO][redash.tasks.schedule] Scheduling 18281a865ed3d1a60f366aeb8596fe2283aa421f (sync_user_details) with interval 60.
[2024-10-18 11:48:40,499][PID:7][INFO][redash.tasks.schedule] Scheduling e27209059575fcc17c527c47d0957cb21756e551 (cleanup_query_results) with interval 300.
[2024-10-18 11:48:40,501][PID:7][INFO][rq_scheduler.scheduler] Registering birth`

[Armadik]

@justinclift @arikfr Hi, maybe you have some ideas? Maybe it's worth setting the container to restart once a day?

[Armadik]

I noticed that the problem occurs at 00:20 at night, I think this is related to daily dashboard updates, I think it is possible to add resources or somehow distribute the launches over time?

[arikfr]

It is possible that if no workers are available to run the job it might get delayed. Can you share some info about your setup: # of scheduled queries, # of workers, runtime of queries, etc?

[Armadik]

arikfr At 00:15 I have about 2400 requests for update, I see it in the database. At the same time in my installation I have 5 workers in kubernetes. I think the problem is in this, but how to fix the problem correctly, so that the scheduler does not hang. Distribute requests by time or increase the number of workers and resources?