fix: prevent duplicate scheduled runs via deployment schedule lock + ID preservation

[devin-ai-integration]

What this PR fixes

This fixes duplicate auto-scheduled flow runs caused by deployment schedule races.

Root cause: when schedules were replaced, schedule IDs changed. Since scheduler idempotency keys include deployment_schedule_id, races between scheduling and deployment updates could produce duplicates.

Approach (short)

1. Preserve schedule IDs in no-slug schedule updates for PATCH /deployments/{id}.
2. Preserve schedule IDs in POST /deployments upsert path as well (important for flow.serve / deployment.apply flows).
3. Add deployment schedule locking for create/update and align lock key usage so POST and PATCH contend on the same identity key.
4. Fail closed when Redis locking is configured but unavailable (503 instead of silent in-memory fallback).

Locking behavior

• Lock key for create/upsert and update is aligned to deployment identity: {flow_id}:{deployment_name}.
• Redis configured (PREFECT_MESSAGING_BROKER contains prefect_redis): use Redis lock with blocking=False, 30s timeout.
• Non-Redis broker: use per-key in-memory asyncio.Lock with fail-fast 409 behavior.
• Redis broker configured but Redis client unavailable: return 503 to avoid unsafe process-local fallback in distributed setups.

No-slug schedule reconciliation

For both PATCH updates and POST upserts:

• Read existing schedules
• Sort by ID for stable positional matching
• Update existing rows in place for overlapping positions (preserve IDs)
• Create rows only for additional incoming schedules
• Delete extra old rows only when incoming list is shorter

This stabilizes schedule IDs across redeploys and keeps scheduler idempotency keys stable.

Test coverage added/updated

API tests (tests/server/orchestration/api/test_deployments.py):

• Updated no-slug PATCH multi-schedule test to assert ID preservation.
• Added test_create_deployment_upsert_preserves_schedule_ids_without_slugs for POST upsert path.
• Added test_post_and_patch_contend_on_same_schedule_lock_key (cross-route lock contention).
• Added test_deployment_schedule_lock_fails_closed_when_redis_unavailable (503 behavior).

Integration test (integration-tests/test_schedule_statefulness.py):

• Added test_schedule_id_stability_for_no_slug_redeploy.
• Uses ephemeral local API settings to ensure integration coverage runs against branch code and does not depend on external profile/server configuration.

Known limitations / follow-ups

• Positional matching is heuristic for no-slug updates; explicit reordering can remap which existing ID gets which schedule config.
• No full real-Redis integration contention test in this PR (current lock tests cover API behavior and error semantics).

Link to Devin Session

Requested by: @zzstoatzz

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[codspeed-hq]

Merging this PR will not alter performance

✅ 2 untouched benchmarks

---

_{Comparing devin/1772469892-fix-schedule-duplication-on-redeploy (6616f29) with main (f8874f4)}