feat: Allow task processor to have its own database

[emyller]

Fixes #5136.

---

• I have added information to docs/ if required so people know about the feature!
• I have filled in the "Changes" section below?
• I have filled in the "How did you test this code" section below?
• I have used a Conventional Commit title for this Pull Request

Changes

Based on the original issue (#5136) description.

Configures support to multi-database task processor as implemented in Flagsmith/flagsmith-common#68.

Defaults to consuming from both databases to ensure remaining tasks from 'default' are processed. Introduced configuration + documentation [intend to] make it easy to disable/fine-tune this behavior in favor of performance.

How did you test this code?

• Unit tests (flagsmith-common)
• Manual testing (new deployments).
• Manual testing (existing deployments).

Test logs

TEST 1: No task processor database

Current functionality is kept intact.

unset TASK_PROCESSOR_DATABASES
unset TASK_PROCESSOR_DATABASE_URL
DATABASE_URL=postgres://postgres@database:5432/postgres
/app/scripts/run-docker.sh run-task-processor

-- 'default'
insert into task_processor_task (uuid, created_at, scheduled_for, task_identifier, num_failures, completed, is_locked) values (gen_random_uuid(), now(), now(), 'tasks.update_organisation_subscription_information_influx_cache', 0, false, false);

task_processor.processor DEBUG    Running 1 task(s) from database 'default'
task_processor.processor DEBUG    Running task tasks.update_organisation_subscription_information_influx_cache id=9 args=[] kwargs={}
task_processor.processor DEBUG    Task tasks.update_organisation_subscription_information_influx_cache id=9 completed
task_processor.processor DEBUG    Finished running 1 task(s) from database 'default'

-- 'default'
postgres=# select * from task_processor_taskrun;
 id |          started_at           |          finished_at          | result  | error_details | task_id 
----+-------------------------------+-------------------------------+---------+---------------+---------
  4 | 2025-05-13 22:08:30.662564+00 | 2025-05-13 22:08:30.671458+00 | SUCCESS |               |       9

TEST 2: Task processor database introduced

Task processor now consumes from both default and task_processor databases, in that order.

-- 'default'
insert into task_processor_task (uuid, created_at, scheduled_for, task_identifier, num_failures, completed, is_locked) values (gen_random_uuid(), now(), now(), 'tasks.update_organisation_subscription_information_influx_cache', 0, false, false);

-- 'task_processor'
insert into task_processor_task (uuid, created_at, scheduled_for, task_identifier, num_failures, completed, is_locked) values (gen_random_uuid(), now(), now(), 'tasks.update_organisation_subscription_information_influx_cache', 0, false, false);

unset TASK_PROCESSOR_DATABASES
DATABASE_URL=postgres://postgres@database:5432/postgres
TASK_PROCESSOR_DATABASE_URL=postgres://postgres@task-processor-database:5432/postgres
/app/scripts/run-docker.sh run-task-processor

task_processor.processor DEBUG    Running 1 task(s) from database 'default'
task_processor.processor DEBUG    Running task tasks.update_organisation_subscription_information_influx_cache id=10 args=[] kwargs={}
task_processor.processor DEBUG    Running 1 task(s) from database 'task_processor'
task_processor.processor DEBUG    Running task tasks.update_organisation_subscription_information_influx_cache id=1 args=[] kwargs={}
task_processor.processor DEBUG    Finished running 1 task(s) from database 'default'
task_processor.processor DEBUG    Finished running 1 task(s) from database 'task_processor'

-- 'default'
postgres=# select * from task_processor_taskrun;
 id |          started_at           |          finished_at          | result  | error_details | task_id 
----+-------------------------------+-------------------------------+---------+---------------+---------
  5 | 2025-05-13 22:34:28.095687+00 | 2025-05-13 22:34:28.10459+00  | SUCCESS |               |      10

-- 'task_processor'
postgres=# select * from task_processor_taskrun;
 id |          started_at          |          finished_at          | result  | error_details | task_id 
----+------------------------------+-------------------------------+---------+---------------+---------
  1 | 2025-05-13 22:34:28.09814+00 | 2025-05-13 22:34:28.104634+00 | SUCCESS |               |       1

TEST 3: Task processor database only

Task processor now consumes from task_processor database only.

-- 'default'
insert into task_processor_task (uuid, created_at, scheduled_for, task_identifier, num_failures, completed, is_locked) values (gen_random_uuid(), now(), now(), 'tasks.update_organisation_subscription_information_influx_cache', 0, false, false);

-- 'task_processor'
insert into task_processor_task (uuid, created_at, scheduled_for, task_identifier, num_failures, completed, is_locked) values (gen_random_uuid(), now(), now(), 'tasks.update_organisation_subscription_information_influx_cache', 0, false, false);

TASK_PROCESSOR_DATABASES=task_processor
DATABASE_URL=postgres://postgres@database:5432/postgres
TASK_PROCESSOR_DATABASE_URL=postgres://postgres@task-processor-database:5432/postgres
/app/scripts/run-docker.sh run-task-processor

task_processor.processor DEBUG    Running 1 task(s) from database 'task_processor'
task_processor.processor DEBUG    Running task tasks.update_organisation_subscription_information_influx_cache id=2 args=[] kwargs={}
task_processor.processor DEBUG    Task tasks.update_organisation_subscription_information_influx_cache id=2 completed
task_processor.processor DEBUG    Finished running 1 task(s) from database 'task_processor'

-- 'default'
postgres=# select task_identifier, scheduled_for, completed, is_locked from task_processor_task;
                         task_identifier                         |         scheduled_for         | completed | is_locked 
-----------------------------------------------------------------+-------------------------------+-----------+-----------
 tasks.update_organisation_subscription_information_influx_cache | 2025-05-13 22:41:12.385212+00 | f         | f

-- 'task_processor'
postgres=# select task_identifier, scheduled_for, completed, is_locked from task_processor_task;
                         task_identifier                         |         scheduled_for         | completed | is_locked 
-----------------------------------------------------------------+-------------------------------+-----------+-----------
 tasks.update_organisation_subscription_information_influx_cache | 2025-05-13 22:41:13.711131+00 | t         | f

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

3 Skipped Deployments

Name	Status	Preview	Comments	Updated (UTC)
docs	⬜️ Ignored (Inspect)	Visit Preview		May 19, 2025 9:26pm
flagsmith-frontend-preview	⬜️ Ignored (Inspect)	Visit Preview		May 19, 2025 9:26pm
flagsmith-frontend-staging	⬜️ Ignored (Inspect)	Visit Preview		May 19, 2025 9:26pm

[github-actions]

Uffizzi Preview deployment-63316 was deleted.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.65%. Comparing base (dc2d513) to head (abf5efa).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5406   +/-   ##
=======================================
  Coverage   97.65%   97.65%           
=======================================
  Files        1235     1235           
  Lines       43326    43334    +8     
=======================================
+ Hits        42309    42317    +8     
  Misses       1017     1017           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[matthewelwell]

Looks good to me, but there are a couple of general comments:

1. I'd love to see in the description confirmation that this has been tested manually
2. We should add these new settings to the docs

[github-actions]

Docker builds report

Image	Build Status	Security report
ghcr.io/flagsmith/flagsmith-e2e:pr-5406	Finished ✅	Skipped
ghcr.io/flagsmith/flagsmith-api-test:pr-5406	Finished ✅	Skipped
ghcr.io/flagsmith/flagsmith-api:pr-5406	Finished ✅	Results ✅
ghcr.io/flagsmith/flagsmith-frontend:pr-5406	Finished ✅	Results ✅
ghcr.io/flagsmith/flagsmith:pr-5406	Finished ✅	Results ✅
ghcr.io/flagsmith/flagsmith-private-cloud:pr-5406	Finished ✅	Results ✅

[khvn26]

This looks good to me now 👍