Scheduler PID is "inactive"?

[ccurvey]

Not sure what I broke here...or what to look for. I'm running on Heroku.

Over the weekend, I upgraded the plan of my OpenRedis server to deal with an expected unusually large load. All went well

This morning at 9:08am, I downgraded that server back to where it had been

At 9:09, I see Scheduler with PID 18 has stopped in my logs. That's not unusual

Since then, if I look in the admin interface for Django-RQ, I see that "Scheduler PID" is inactive. And sure enough, jobs that have been scheduled for the future are not getting re-queued.

I've tried restarting everything -- even going as far as pushing a new release to production, but that Scheduler PID stays "Inactive". I can manually enqueue the jobs and they work.

I can't find anything in the logs that looks interesting.

Help?

[ccurvey]

After a fair bit of back-and-forth with a chatbot (because I don't really know Redis at all), here's what I think happened. If any of the big brains around here can confirm or deny (or clarify) my analysis, I would be eternally grateulf.

It appears that there are two different bits of information about the queues:

1. There is a queues registry. The registry has a key of "rq:queues" and the value is a set like {"rq:queue:default", "rq:queue:high", "rq:queue:low"}
2. There are also the queues themselves, which have keys like "rq:queue:default" (and the value is a list of jobs), "rq:queue:high" (and the value is a list of jobs), etc.

So we think what happened during the upgrade was that the registry of queues got cleared, but the actual queues remained. The workers were able to find the actual queues and process them.

However, the scheduler process (which is separate from the workers) uses the registry to see which queues need to be monitored for scheduled jobs. And since the registry had been deleted/cleared, the scheduler didn't look anywhere.

[selwin]

Did you try restarting the workers?