While, Do-while loop implementation for fetching paginated data from an API

[nyamathshaik]

What would you like to be added?

Support for while or do-while loop constructs to simplify workflows that need to fetch paginated data from APIs until a condition is met (e.g., no more pages to fetch).

Proposal(s):

Currently, implementing workflows that require repetitive API calls until a condition is met (e.g., while nextPageToken is present) is cumbersome and verbose using the existing specification. There is no native support for loop constructs that check a condition after each iteration (i.e., a do-while pattern).

Proposal:

• Introduce a native while or do-while loop construct within the Serverless Workflow specification.
• Support a conditional expression (JQ) that evaluates whether the loop should continue.
• Allow a loopBody or array of steps to be executed within the loop.
• Optional: support max iterations to prevent infinite loops.

Example syntax (pseudo-DSL):

document:
  dsl: '1.0.0'
  namespace: test
  name: paginated-fetch-example
  version: '0.1.0'

do:
  - fetchPaginatedData:
      while: .hasNextPage == true
      postConditionCheck: false // Enables do-while 
      maxIterations: 100
      do:
        - fetchPage:
            call: getPageData
            input:
              pageToken: .nextPageToken
            output: .fetchedData
        - accumulateData:
            run: mergeResults
            input:
              newData: .fetchedData
              existingData: .accumulatedResults
            output: .accumulatedResults

Alternative(s):

Currently, the same can be achieved by:

• Manually chaining states with transition and using a switch to re-enter a state based on a condition.
• Using recursion with workflow calls, which increases complexity and reduces readability.
• Implementing a custom function or orchestrator outside of the workflow itself.

These approaches are harder to read, maintain, and error-prone when retry logic and timeouts are introduced.

Additional info:

This feature would greatly improve:

• Workflows that deal with pagination (API data fetching, batch processing).
• Polling use cases where the data becomes available over time.
• Any retry-until-success or loop-until-condition patterns.

Let me know if you'd like me to open a PR to help explore this idea! Or do you think this can be achieved with current For loop? @ricardozanini @cdavernas

Community Notes

• Please vote by adding a 👍 reaction to the feature to help us prioritize.
• If you are interested to work on this feature, please leave a comment.

[nyamathshaik]

However, It appears that the current for task can be utilized as a loop, allowing a For task to operate as a traditional while loop in Serverless Workflow.

By setting the for.in property of a For task to an empty collection, the while property takes over as the primary control mechanism for the loop, facilitating the use of both while and do-while loop patterns within Serverless Workflows.

Example:

 - name: WhileLoopExample
    type: while
    for: 
      in: ${ [] }
    while: ${ .counter < 5 }
    maxIterations: 100
    do:
      - name: IncrementCounter
        type: set
        data:
          counter: ${ .counter + 1 }
      - name: LogMessage
        type: run
        run:
          script:
            lang: javascript
            code: |
              console.log("Counter is: " + context.counter);

Example:

- name: DoWhileLoopExample
   type: do-while
   for: 
     in: ${ [] }
   while: ${ .counter < 5 } # false initially
   maxIterations: 100
   do:
     - name: IncrementCounter
       type: set
       data:
         counter: ${ .counter + 1 }
     - name: LogMessage
       type: run
       run:
         script:
           lang: javascript
           code: |
             console.log("This runs even though .counter < 5 is false!");

[nyamathshaik]

May I know if for is mandatory or can we get it of

for: 
     in: ${ [] }

According to typescript SDK it's optional, but its no-where that's mentioned in the specs, can you clarify please?

https://github.com/serverlessworkflow/sdk-typescript/blob/main/src/lib/generated/definitions/specification.ts#L252

[cdavernas]

@nyamathshaik Yes, for is mandatory, and also serves as a discriminator. In addition, your suggestion would not work because the enumeration would stop before evaluating the while condition, as the array is empty. while in that context is an exit condition, not a continuation one, in which case it would create many unpleasant side effects (one being a null ref for enumerated item).

Therefore, you initial proposal looks to me to be the most adequate, and would also restore a construct we had in previous versions.

[nyamathshaik]

@cdavernas Thank you for the feedback. I was considering something along the lines of the following, but I am still open to suggestions. Please review it and let me know if it looks good.

export type LoopTask = TaskBase &{
    /**
     * The type of loop to use.
     */
    loop?: 'for' | 'while' | 'do-while';
    /**
     * The configuration for the for loop.
     */
    for?: ForTaskConfiguration;
    /**
     * The configuration or conditions for the while loop.
     */
    while?: WhileTaskConfiguration | string;
    /**
     * The tasks to execute if the loop is a for, while, do-while loop.
     */
    do?: TaskList;
   
    
    [k: string]: unknown;
};

export interface WhileTaskConfiguration {
    /**
     * The name of the variable used to store the current item being enumerated.
     */
    condition?: string;
    /**
     * A runtime expression used to get the collection to enumerate.
     */
    maxIterations?: number;
    /**
     * The name of the variable used to store the index of the current item being enumerated.
     */
    at?: string;
}

Benefits of Proposed Approach

• Backward Compatibility: Existing ForTask implementation remains unchanged.
• Enhanced Clarity: Clear differentiation between for, while, and do-while loops.
• Safety Mechanism: The addition of maxIterations prevents infinite loops.
• Extensibility: Future improvements can be made without affecting existing task definitions.

This proposed solution provides a structured approach to implementing loop tasks while maintaining compatibility with Serverless Workflow specifications.

[cdavernas]

@nyamathshaik First of all, thanks for your awesome work!

Second, I have a couple of remarks regarding the proposal you made in related PR:

do:
  - fetchPaginatedData:
      while: .hasNextPage == true
      postConditionCheck: false # Enables do-while behavior
      maxIterations: 100
      do:
        - fetchPage:
            call: getPageData
            input:
              pageToken: .nextPageToken
            output: .fetchedData
        - accumulateData:
            run: mergeResults
            input:
              newData: .fetchedData
              existingData: .accumulatedResults
            output: .accumulatedResults

• The while keyword will need to change, either in the task you propose to add, or in the for task, or discrimination will no longer be (easily) possible (i.e. check for presence of while keyword, which as a primary identifier must not be use by any other task)
• I'm personnaly not a fan of the postConditionCheck and maxIterations terminology, which goes against the design guidelines. I'm convinced we can come up with imperative, actionable, single words names instead!

[nyamathshaik]

@cdavernas Thanks a lot for the feedback and kind words! 🙌 Really appreciate you taking the time to review the proposal.

Totally hear you on the concerns around the while keyword clash and the naming of postConditionCheck and maxIterations.

For the keyword conflict: I’m happy to revise the structure to avoid any ambiguity with for tasks. Would something like below look good?

do:
  - fetchPaginatedData:
      loop: while | do-while
      condition: .hasNextPage == true
      limit: 100
      do:
        - fetchPage:
            call: getPageData
            input:
              pageToken: .nextPageToken
            output: .fetchedData
        - accumulateData:
            run: mergeResults
            input:
              newData: .fetchedData
              existingData: .accumulatedResults
            output: .accumulatedResults

Open to suggestions! I'm happy to iterate further on the proposal based on whatever naming conventions or structural guidance the maintainers feel would fit best.

Looking forward to your thoughts!

[nyamathshaik]

or how about just simply renaming the initial proposal to below :

do:
  - fetchPaginatedData:
      condition: .hasNextPage == true
      isDoWhile: true # Enables do-while behavior
      limit: 100
      do:
        - fetchPage:
            call: getPageData
            input:
              pageToken: .nextPageToken
            output: .fetchedData
        - accumulateData:
            run: mergeResults
            input:
              newData: .fetchedData
              existingData: .accumulatedResults
            output: .accumulatedResults

[ricardozanini]

Why not:

 - name: WhileLoopExample
   while: ${ .counter < 5 }
   do:
      - name: IncrementCounter
        type: set
        data:
          counter: ${ .counter + 1 }
      - name: LogMessage
        type: run
        run:
          script:
            lang: javascript
            code: |
              console.log("Counter is: " + context.counter);

And

 - name: WhileLoopExample
   do:
      - name: IncrementCounter
        type: set
        data:
          counter: ${ .counter + 1 }
      - name: LogMessage
        type: run
        run:
          script:
            lang: javascript
            code: |
              console.log("Counter is: " + context.counter);
    while: ${ .counter < 5 }

The challenge is to interpret this in the JSON Schema.

Essentially, you want a for without in, correct?

[ricardozanini]

We can add the limit keyword, no problem. But yes, we must avoid camel-cased and composition words.

[ricardozanini]

The first is a do task with an optional while parameter. The other is a while task. I think we can accommodate this in the JSON schema with backwards comp for 1.1.0.

[nyamathshaik]

Thanks for reviewing @ricardozanini JSON Schema doesn’t support positional logic. So I don't think comment actually works. (i.e., if while appears before, interpret it one way; if after, interpret differently).

Also, this creates more Ambiguity in Semantics (while appearing before or after do: changes the behavior).