[bug] Workflow task list API loads full task documents and takes ~55s for page_size=50

[FunnyIceCube]

What happened?

The workflow task history API becomes extremely slow as page_size increases:

GET /api/aslan/workflow/v4/workflowtask?workflow_name=<workflow>&page_num=1&page_size=50&projectName=<project>

The same latency is reproduced by calling Aslan directly inside the pod, bypassing the frontend, NodePort, Gloo, and external auth routing. The API returns only about 100 KB, but takes about 55 seconds before the first byte.

A goroutine profile captured while the request was running showed the handler blocked in MongoDB I/O:

WorkflowTaskv4Coll.ListByFilter
  -> mongo.Collection.Find
  -> mongo driver readWireMessage
  -> IO wait

In Zadig v4.2.1, ListByFilter performs CountDocuments, then Find without a projection, and finally cursor.All into full WorkflowTask objects. Each task contains WorkflowArgs, OriginWorkflowArgs, and runtime Stages, including job specs, build/deploy options, variables, and YAML/Values. The list response only needs preview fields, so a large amount of MongoDB data is transferred and decoded but then discarded.

The repository method also uses context.TODO(). If a browser request times out or the user refreshes, the MongoDB query can continue after the HTTP client disconnects, allowing repeated requests to accumulate.

Relevant source:

• https://github.com/koderover/zadig/blob/v4.2.1/pkg/microservice/aslan/core/common/repository/mongodb/workflow_task_v4.go
• https://github.com/koderover/zadig/blob/v4.2.1/pkg/microservice/aslan/core/common/repository/models/wokflow_task_v4.go

Measured from inside the Aslan pod:

page_size	Time to first byte	API response size
1	2.97 s	1.8 KB
5	6.33 s	9.1 KB
10	8.56 s	18.4 KB
50	54.87 s	100 KB

Aslan CPU and memory were not saturated during the tests.

What did you expect to happen?

The workflow task history list should return summary records quickly and should not load full workflow snapshots and runtime job specifications for every list item.

Ideally, the MongoDB query should project directly into the fields required by WorkflowTaskPreview, and database operations should use the HTTP request context so cancelled requests stop consuming database connections and bandwidth.

How To Reproduce it(as minimally and precisely as possible)

1. Create a custom workflow containing several build and deployment jobs.
2. Include multiple services/modules and deployment variables or YAML/Values in the workflow.
3. Run the workflow enough times to create at least 50 task records.
4. Open the workflow task history page.
5. Compare the API latency with page_size=1, 10, and 50.
6. Capture a goroutine profile during the request; the handler waits in WorkflowTaskv4Coll.ListByFilter while reading MongoDB results.

Possible improvements

• Add a MongoDB projection for list/preview fields instead of decoding full WorkflowTask documents.
• Avoid loading workflow_args, origin_workflow_args, full job specs, and large YAML fields in the history list.
• Pass the request context into CountDocuments, Find, and cursor.All.
• Consider avoiding or caching a full CountDocuments on every page request.
• Verify the query plan and consider an index including project_name, workflow_name, is_archived, is_deleted, and create_time.

Versions Used

zadig: v4.2.1

kubernetes: v1.30.10

Environment

Cloud Provider: Self-hosted Kubernetes with an external MongoDB-compatible managed database

Resources: Cluster and Aslan CPU/memory were not saturated

OS: Ubuntu 24.04 LTS

Services Status

All Zadig core pods were Running and Ready during reproduction. The issue is reproducible by connecting directly to the Aslan container, so it is not caused by the frontend or gateway.