Provisioning Requests Stuck in “Queued” in the morning only, until ~12:00 daily

[bryanfrancois]

Describe the issue you are having and what you expected to happen.

When launching a server build in one of our environment, provisioning requests remain stuck in “Queued” for several hours every morning.
The request only starts processing around 12:00, after which the VM is provisioned normally.

We expect VM builds to start immediately or within 5–10 minutes, regardless of the time of day.

Note:

• The issue only affects our pre-production environment, we are not encountering this in our production environement (same deployment type (VMware OVA Appliance) and same versions (Radjabov)
• The issue occurs whether provisioning is triggered through a Service Catalog or via the API.

Describe the steps to reproduce, including any log snippets and stack traces that will help diagnose.

Steps to reproduce:

1. Between 09:00 and 12:00, initiate a VM build

• via a Service Catalog item
• or via the REST API

2. Navigate to:

• Automation → Requests → Provision Requests

3. Observe the Request State = “Queued” for several hours with no progress.

4. After approximately 12:00, the same request is automatically processed and completes normally.

Additional notes:

• The problem does not occur in the afternoon.
• It is consistently reproducible only in the morning window.
• No queue buildup is visible afterward — all runs after noon complete immediately.

Describe your environment, including

• ManageIQ Version: radjabov-1.20250408024349_e386fb4
• Deployment type: VMware OVA appliance

[Fryguy]

That is incredibly strange. I don't recall anything in the code that would cause this. Some questions...do you see any differences across time zones? Do you see anything in the logs around these stuck requests (for example, anything like queue messages with due_on or deliver_on dates)? Is there any time-based component of your custom automate code?

[bryanfrancois]

Hi Fryguy,

Update on your questions and additional findings:

• Queue messages:

I checked journalctl -u evmserverd (since no evm.log or automation.log is present on the appliance). I see regular MiqQueue activity with messages being delivered and processed, but I do not see any due_on or deliver_on fields in the log output.

There’s no indication of deferred or scheduled delivery.

• Time zones:

Both pre-prod and prod are set to Europe/Paris (CET, +0100). NTP is active, but timedatectl reports System clock synchronized: no on both. Only pre-prod is affected by the morning queue issue.

• Automate code:

No time-based logic is present in our custom Automate code. I’ve double-checked for any accidental time-based conditions and found none.

• Workers:

All provisioning workers and roles appear active in the UI during the affected period.

Additional notes:

• The issue is 100% reproducible in pre-prod between 09:00 and 12:00, regardless of how provisioning is triggered.

• After 12:00, all queued requests are processed immediately, and new requests are provisioned without delay.

• No backlog or queue buildup is visible after noon.

• No scheduled tasks or maintenance jobs are configured that could explain this behavior.

Questions:

• Is there another log or database table where I could check for deferred queue messages or provisioning tasks?

• Are there any known issues with queue polling or worker scheduling that could be time-dependent, even indirectly (e.g., via cron, systemd timers, or Ruby time calculations)?

Thanks again for your help!