[Security Solution] Randomize task schedules when bulk scheduling (#269991)

sdesalas · cursoragent · web-flow · commit 6669b2ad084d · 2026-05-21T11:19:55.000+02:00
**Resolves: #195136** **Related to: #264893** **Related to: #269340** ## Summary `TaskScheduling.bulkSchedule` previously sent every task to the store with no `runAt`, which caused the store to default them all to "now". When a caller bulk-scheduled many recurring tasks at once, the polling queue was flooded with [simultaneous claims](https://en.wikipedia.org/wiki/Thundering_herd_problem). This PR brings `bulkSchedule` in line with `bulkEnable` (see #172742): the first task in the batch still runs immediately, but subsequent enabled recurring tasks are scheduled with a randomized `runAt`, evenly distributed up to 5 minutes in the future. Ad-hoc tasks (no `schedule.interval`) and disabled tasks are left untouched and run immediately as before. This also helps unblock upcoming work on `RulesClient.bulkCreate` (#264893), where a single API call may schedule a large number of detection-rule tasks at once. ## Whats included - The existing `randomlyOffsetRunTimestamp()` helper is replaced by a smaller pure helper `addJitter()` that returns `{ runAt, scheduledAt }` - or `undefined` when no interval is supplied - so callers control the spread. - `bulkSchedule` map callback now receives the index `i` and uses `addJitter()` when `enabled && i > 0`. - `bulkEnable`'s behavior continues exactly the same with the `i > 0` branch using the shared `addJitter()` helper. ## How to test > [!IMPORTANT] > There is no easy way to test TM `bulkSchedule()` randomizing without `v2` alerting. Because of this, the test below only covers task randomizing under `bulkEnable()`. If you apply debugging here, you will notice that enabling detection rules in bulk uses `bulkSchedule()` under the hood, but it does so in `enabled: false` state. In other words, no jitter will get applied until the following pass. This is expected, what the test below does primarily is to verify that existing behavior is unaffected. The changes to behavior in `bulkSchedule(enabled:true)` will become more meaningful with upcoming work on alerting `v2` and `RulesClient.bulkCreate`. 1. Start ES + Kibana from this branch. Make sure you have a clean ES with no rules. 2. In Kibana, navigate to **Security → Rules → Detection rules (SIEM)** and click **Add Elastic Rules** to install the prebuilt detection rule set (~1850 rules). Leave them disabled. > Note: This is a good time to place some breakpoints if you're debugging locally. 3. Go back to the rules management screen. Under "Installed Rules" click the checkbox to select first 20 rules then `Bulk actions` > `Enable`. You should see a message saying "Successfully enabled 20 rules" 4. Verify the `runAt` / `scheduledAt` distribution using [`check-task-runtime.sh`](https://github.com/sdesalas/kibana-knowledge/blob/main/scripts/check-task-runtime.sh): ```bash $ ./check-task-runtime.sh ``` Or if you are using ports different to the standard `5601` and `9200` ```bash $ KIBANA_DEV_PORT=5606 ES_DEV_PORT=9205 ./check-task-runtime.sh ``` 5. Expected output: counts match, and the first-20 task timestamps are **spread across several minutes** rather than all stamped with the same "now": ``` starting.. KIBANA_URL=http://localhost:5601/kbn ES_URL=http://localhost:9200 1. 2. 3. 4. 5. 6. rules: 1850 rules_enabled: 20 tasks: 20 tasks_enabled: 20 api_key_owner: 20 apiKey present: 20 first 20 tasks: taskType status enabled runAt scheduledAt alerting:siem.queryRule idle true 2026-05-19T16:20:08.323Z 2026-05-19T16:20:08.323Z alerting:siem.queryRule idle true 2026-05-19T16:21:28.689Z 2026-05-19T16:21:28.689Z alerting:siem.eqlRule idle true 2026-05-19T16:21:05.927Z 2026-05-19T16:21:05.927Z alerting:siem.queryRule idle true 2026-05-19T16:20:53.163Z 2026-05-19T16:20:53.163Z alerting:siem.queryRule idle true 2026-05-19T16:23:30.562Z 2026-05-19T16:23:30.562Z alerting:siem.esqlRule idle true 2026-05-19T16:23:45.295Z 2026-05-19T16:23:45.295Z ... ``` For every task, `runAt` should equal its matching `scheduledAt`. The timestamps should be distributed across the configured jitter window (`min(rule interval, 5m)`) - confirming jitter is applied per-task. On `main` without this PR, every task's `runAt` collapses to the same value. ## Callers of `bulkSchedule` For reference, the production callers of `TaskScheduling.bulkSchedule` and whether they exercise the new jitter: | Caller | What it schedules | Hits the new jitter? | |---|---|---| | [`alerting_v2/.../rules_client.bulkEnableRules`](https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/alerting_v2/server/lib/rules_client/rules_client.ts) | enabled, recurring (`schedule.interval`, `enabled: true`) | **Yes** — primary path exercised by the manual test above | | [`alerting/.../bulk_enable_rules.ts`](https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/alerting/server/application/rule/methods/bulk_enable/bulk_enable_rules.ts) (legacy) | recurring but `enabled: false` — the comment in that file explicitly says "we create the task as disabled, taskManager.bulkEnable will enable them by randomising their schedule datetime" | No (jitter applied later by `bulkEnable`) | | [`workflows_execution_engine/server/plugin.ts`](https://github.com/elastic/kibana/blob/main/src/platform/plugins/shared/workflows_execution_engine/server/plugin.ts) | enabled but ad-hoc (no `schedule`) | No (ad-hoc — runs immediately, correct) | | [`actions/create_execute_function.ts`](https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/actions/server/create_execute_function.ts) | ad-hoc action tasks | No | | [`actions/create_unsecured_execute_function.ts`](https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/actions/server/create_unsecured_execute_function.ts) | ad-hoc action tasks | No | | [`alerting/.../backfill_client.ts`](https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/alerting/server/backfill_client/backfill_client.ts) | variable is literally `adHocTasksToSchedule` | No | Of these, only the alerting_v2 `bulkEnableRules` path schedules enabled recurring tasks in bulk, so it is the only caller whose runtime behavior changes with this PR. ## Release note skip ## Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Identify risks - Behavior change is scoped to recurring tasks at `i > 0`; single-task `bulkSchedule` calls and ad-hoc tasks retain the existing "run now" semantics. - The `bulkEnable` path is unchanged in semantics; only the helper signature changed. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
diff --git a/x-pack/platform/plugins/shared/task_manager/server/task_scheduling.test.ts b/x-pack/platform/plugins/shared/task_manager/server/task_scheduling.test.ts
@@ -1337,6 +1337,8 @@ describe('TaskScheduling', () => {
             schedule: undefined,
             traceparent: 'parent',
             enabled: true,
+            runAt: new Date(),
+            scheduledAt: new Date(),
           },
         ],
         undefined
@@ -1366,6 +1368,8 @@ describe('TaskScheduling', () => {
             schedule: undefined,
             traceparent: 'parent',
             enabled: true,
+            runAt: new Date(),
+            scheduledAt: new Date(),
           },
           {
             ...task2,
@@ -1379,6 +1383,126 @@ describe('TaskScheduling', () => {
       );
     });
 
+    test('leaves the first task untouched and jitters subsequent recurring tasks within min(interval, 5m)', async () => {
+      const taskScheduling = new TaskScheduling(taskSchedulingOpts);
+      const task0 = {
+        taskType: 'foo',
+        params: {},
+        state: {},
+        schedule: { interval: '1m' },
+      };
+      const task1 = {
+        taskType: 'foo',
+        params: {},
+        state: {},
+        schedule: { interval: '1m' },
+      };
+      const task2 = {
+        taskType: 'foo',
+        params: {},
+        state: {},
+        schedule: { interval: '1h' },
+      };
+      await taskScheduling.bulkSchedule([task0, task1, task2]);
+
+      const bulkSchedulePayload = mockTaskStore.bulkSchedule.mock.calls[0][0];
+
+      expect(bulkSchedulePayload.length).toBe(3);
+
+      expect(bulkSchedulePayload[0]).toEqual({
+        ...task0,
+        id: undefined,
+        traceparent: 'parent',
+        enabled: true,
+        runAt: new Date(),
+        scheduledAt: new Date(),
+      });
+
+      expect(omit(bulkSchedulePayload[1], 'runAt', 'scheduledAt')).toEqual({
+        ...task1,
+        id: undefined,
+        traceparent: 'parent',
+        enabled: true,
+      });
+      expect(omit(bulkSchedulePayload[2], 'runAt', 'scheduledAt')).toEqual({
+        ...task2,
+        id: undefined,
+        traceparent: 'parent',
+        enabled: true,
+      });
+
+      const t1RunAt = bulkSchedulePayload[1].runAt!.getTime();
+      expect(bulkSchedulePayload[1].scheduledAt!.getTime()).toBe(t1RunAt);
+      expect(t1RunAt).toBeGreaterThanOrEqual(1);
+      expect(t1RunAt).toBeLessThanOrEqual(60 * 1000);
+
+      const t2RunAt = bulkSchedulePayload[2].runAt!.getTime();
+      expect(bulkSchedulePayload[2].scheduledAt!.getTime()).toBe(t2RunAt);
+      expect(t2RunAt).toBeGreaterThanOrEqual(1);
+      expect(t2RunAt).toBeLessThanOrEqual(5 * 60 * 1000);
+    });
+
+    test('runs ad-hoc tasks immediately without jitter, even at i > 0', async () => {
+      const taskScheduling = new TaskScheduling(taskSchedulingOpts);
+      const recurringTask = {
+        taskType: 'foo',
+        params: {},
+        state: {},
+        schedule: { interval: '1m' },
+      };
+      const adHocTask = {
+        taskType: 'foo',
+        params: {},
+        state: {},
+      };
+      await taskScheduling.bulkSchedule([recurringTask, adHocTask]);
+
+      const bulkSchedulePayload = mockTaskStore.bulkSchedule.mock.calls[0][0];
+
+      expect(bulkSchedulePayload).toEqual([
+        {
+          ...recurringTask,
+          id: undefined,
+          traceparent: 'parent',
+          enabled: true,
+          runAt: new Date(),
+          scheduledAt: new Date(),
+        },
+        {
+          ...adHocTask,
+          id: undefined,
+          schedule: undefined,
+          traceparent: 'parent',
+          enabled: true,
+          runAt: new Date(),
+          scheduledAt: new Date(),
+        },
+      ]);
+    });
+
+    test('does not jitter disabled tasks even if they have an interval schedule', async () => {
+      const taskScheduling = new TaskScheduling(taskSchedulingOpts);
+      const task = {
+        taskType: 'foo',
+        params: {},
+        state: {},
+        schedule: { interval: '1h' },
+        enabled: false,
+      };
+      await taskScheduling.bulkSchedule([task]);
+
+      const bulkSchedulePayload = mockTaskStore.bulkSchedule.mock.calls[0][0];
+
+      expect(bulkSchedulePayload).toEqual([
+        {
+          ...task,
+          id: undefined,
+          traceparent: 'parent',
+          enabled: false,
+        },
+      ]);
+    });
+
     test('doesnt allow naively rescheduling existing tasks that have already been scheduled', async () => {
       const taskScheduling = new TaskScheduling(taskSchedulingOpts);
       mockTaskStore.bulkSchedule.mockRejectedValueOnce({
diff --git a/x-pack/platform/plugins/shared/task_manager/server/task_scheduling.ts b/x-pack/platform/plugins/shared/task_manager/server/task_scheduling.ts
@@ -148,15 +148,26 @@ export class TaskScheduling {
         ? agent.currentTraceparent
         : '';
     const modifiedTasks = await Promise.all(
-      taskInstances.map(async (taskInstance) => {
+      taskInstances.map(async (taskInstance, i) => {
         const { taskInstance: modifiedTask } = await this.middleware.beforeSave({
           ...omit(options, 'apiKey', 'request'),
           taskInstance: ensureDeprecatedFieldsAreCorrected(taskInstance, this.logger),
         });
+        const enabled = modifiedTask.enabled ?? true;
+        let scheduling: Partial<{ runAt: Date; scheduledAt: Date }> = {};
+        if (enabled) {
+          // Run the first task now. Run all other tasks a random number of ms in the future,
+          // with a maximum of 5 minutes or the task interval, whichever is smaller.
+          scheduling =
+            i === 0
+              ? { runAt: new Date(), scheduledAt: new Date() }
+              : addJitter(modifiedTask.schedule?.interval) ?? {};
+        }
         return {
           ...modifiedTask,
           traceparent: traceparent || '',
-          enabled: modifiedTask.enabled ?? true,
+          enabled,
+          ...scheduling,
         };
       })
     );
@@ -200,11 +211,9 @@ export class TaskScheduling {
         if (runSoon) {
           // Run the first task now. Run all other tasks a random number of ms in the future,
           // with a maximum of 5 minutes or the task interval, whichever is smaller.
-          const taskToRun =
-            i === 0
-              ? { ...task, runAt: new Date(), scheduledAt: new Date() }
-              : randomlyOffsetRunTimestamp(task);
-          return { ...taskToRun, enabled: true };
+          return i === 0
+            ? { ...task, enabled: true, runAt: new Date(), scheduledAt: new Date() }
+            : { ...task, enabled: true, ...addJitter(task.schedule?.interval ?? '0s') };
         }
         return { ...task, enabled: true };
       },
@@ -375,17 +384,16 @@ export class TaskScheduling {
   }
 }
 
-const randomlyOffsetRunTimestamp: (task: ConcreteTaskInstance) => ConcreteTaskInstance = (task) => {
+const addJitter = (interval?: string): { runAt: Date; scheduledAt: Date } | undefined => {
+  // Adhoc tasks run immediately.
+  if (!interval) return { runAt: new Date(), scheduledAt: new Date() };
+
   const now = Date.now();
   const maximumOffsetTimestamp = now + 1000 * 60 * 5; // now + 5 minutes
-  const taskIntervalInMs = parseIntervalAsMillisecond(task.schedule?.interval ?? '0s');
+  const taskIntervalInMs = parseIntervalAsMillisecond(interval);
   const maximumRunAt = Math.min(now + taskIntervalInMs, maximumOffsetTimestamp);
 
   // Offset between 1 and maximumRunAt ms
   const runAt = new Date(now + Math.floor(Math.random() * (maximumRunAt - now) + 1));
-  return {
-    ...task,
-    runAt,
-    scheduledAt: runAt,
-  };
+  return { runAt, scheduledAt: runAt };
 };
