-
Notifications
You must be signed in to change notification settings - Fork 739
Description
Describe the bug
When asyncIndexingEnabled=true
, tasks from multi-task workflows randomly fail to index to the Search DB (here OpenSearch) due to unhandled CompletableFuture
exceptions, causing data inconsistency between the primary database and search index.
Details
Conductor version: Latest (main branch)
Persistence implementation: Postgres
Queue implementation: Redis standalone
Lock: Redis
Workflow definition: Any workflow with 2+ tasks
Task definition: Any SIMPLE tasks
Event handler definition: N/A
To Reproduce
Steps to reproduce the behavior:
- Set
conductor.app.asyncIndexingEnabled=true
in configuration - Create a workflow with 2 or more tasks
- Start and complete the workflow (all tasks complete successfully)
- Check the search index (OpenSearch/Elasticsearch) - only some tasks are indexed
- Check PostgreSQL - all tasks exist correctly
- Repeat with different workflows - random tasks are missing from search index
Expected behavior
All completed tasks should be indexed to the Search DB and be searchable via the API and UI.
Screenshots


Additional context
- Root cause: In
ExecutionDAOFacade.java
,asyncIndexTask()
CompletableFuture returns are ignored, causing silent failures - Affects all deployments with async indexing enabled
- Workaround: Set
asyncIndexingEnabled=false
(impacts performance)
Proposed Solution
Replace the current individual async indexing approach with bulk task indexing, following the same pattern already used for task execution logs. This approach would:
- Add bulk indexing methods to the
IndexDAO
interface similar to the existingaddTaskExecutionLogs
methods - Implement these methods in all
IndexDAO
implementations using their existing bulk request infrastructure - Update
ExecutionDAOFacade
to collect all tasks and submit them as a single bulk operation instead of individual async calls - Leverage existing error handling, retry logic, and monitoring that's already proven to work with bulk operations