Adding WorkflowSpec Arguments and WorkflowStatus Nodes to archivedWorkflowMetadata

[ranasinghem]

Currently, the Workflow struct stores WorkflowSpec and WorkflowStatus. When retrieving live workflow runs via the WorkflowService API, all fields within the WorkflowSpec and WorkflowStatus are returned.

However, when retrieving workflow runs that have been archived, only a subset of fields are returned for WorkflowSpec and WorkflowStatus (see code snippet here).

Is this intentional? (I had been searching through previous issues, and from what I understand was that this was a performance improvement, but I'm not positive). Is there a manner to support storing WorkflowSpec.Arguments as well as WorkflowStatus.Nodes. It would be useful to have these fields returned back for archived workflow runs (essentially similarly how its done for live workflow runs).

[ranasinghem]

Hi @agilgur5! Hope things are going well! Just liked to tag you in here, since you have great familiarity with this. Thanks for any time or suggestions in advance, really do appreciate it!

[tczhao]

I believe everything is stored.
The code snippet here you provide is the listing function, we filter necessary fields for listing workflow.

The storing function and retrieval function use all fields from workflow object

[ranasinghem]

Hi @tczhao Thanks for your response (I listed a TLDR comment at the bottom)

Yes, your correct. The storing function stores into the database all metadata for workflow runs which is great. However, the retrieval function you listed will return all fields as part of a specific workflow run.

I was curious to see if the same behavior can be made when listing all workflow runs for a specific workflow object. As of now, the archivedWorkflowServer.ListArchivedWorkflows will only get back a subset of metadata for every workflow run (see this line here). It seems to stem down to the following call wfArchive.ListWorkflows - when this function is called it will end up storing each workflow as defined in the initial code snippet I sent here

---

TLDR:

• When requesting a specific workflow run (GetWorkflows), we get back all metadata for the given workflow object
• When requesting all workflow runs (ListWorkflows), we only get back a subset of metadata for each workflow object
• Potential Solution: Update the Spec and Status fields for each workflow object to include all properties of the workflow object. This will allow ListWorkflows to return all metadata fields for each workflow object

---

[ranasinghem]

Hi @tczhao What I ended up doing as a work around is the following:

1. Invoke ListArchivedWorkflows to retrieve all archived workflows
2. For each archived workflow, invoke the GetArchivedWorkflows to retrieve the full WorkflowSpec
3. Update the associated archived workflow with the full WorkflowSpec

This definitely takes a performance hit, as I am calling the GetArchivedWorkflows endpoint for as many number of archived workflows in the database.

As mentioned in the reply, there is a better potential solution (which I'd be happy to open an issue/contribute too).

Thanks again for your time in helping in this matter! Really do appreciate it