DTask: a new awaitable type for distributed asynchronous workflows

[gianvitodifilippo]

Over the past year, I’ve been working on an experimental library, and now it's ready to start gathering some early feedback: DTasks.

DTasks simplifies writing long-running distributed async operations. It does so by introducing two new awaitable types: DTask and DTask<TResult>. Just as Task can represent a locally asynchronous operation, DTask is its distributed counterpart: it represents asynchronous operations that can span multiple machines.

How it works

DTasks is built on top of the async/await pattern we're all familiar with, so you can use DTask as the return type of an async method and await other DTasks inside it. The API mirrors Task, including:

• DTask.Delay, DTask.Yield,
• DTask.WhenAll, DTask.WhenAny,
• DTask.CompletedTask, DTask.FromResult.

DTasks are durable, meaning the state of the async method (its local variables) is automatically persisted, and distributed, as execution may resume on a different machine (just like a Task can resume on a different thread).

If you have used Microsoft's DTFx, all this may sound familiar. So, how is DTasks different?

1. It uses a dedicated type (DTask) for representing distributed/durable operations.
2. It does not necessarily require a replay model: in theory, it supports it, but the current default implementation is snapshot-based.
3. This implies that many of the constraints of the replay model no longer apply: code can be non-deterministic, have side effects, etc.
4. You can await any Task and DTask without needing an orchestration context object (IDurableOrchestrationContext).

Check out the GitHub repository, and the provided samples with step-by-step guides to run them. I also included a roadmap with the project's current status and planned features.

Feedback

I'd like to know what you think about it! Is this approach useful? Could it be a valid alternative to existing solutions? Is it worth investing even more time into it?

A quick disclaimer: this library is currently pre-alpha. I know practical concerns like scalability and reliability are crucial, but at this stage I’ve focused on proving the concept. That said, feel free to ask anything and I’ll do my best to answer.

Runtime async (async2)

I'm not that sure how this would play with the new runtime async model. DTasks' dehydration/hydration strategy require the exsistence of a state machine and I don't know if developers can opt in/out any of either async model. What are your thoughts?

[Seabizkit]

Hi, i would like to better understand the usage.

It would be helpful to explain what problem your solving and also how it works. Uses Redis to record state, what state?

perhaps give a better fuller example: including requirements Redis for the distributed backplane to maintain some state.... incase something drops?

"DTask is its distributed counterpart: it represents asynchronous operations that can span multiple machines."

? can you show this with the example... make it easier to understand what problem your solving, with an example which should the core advantage.

asynchronous operations, are one where it does not need the current thread...and can be reallicated while it wait.. for non cpu required operation to complete ...mutiple task can each be awaited..so "asynchronous operations that can span multiple machines." is confusing

you gave the following as an example;
it unclear what is the key take way....
could you asisst.

using DTasks;

public class AsyncEndpoints(
    ApproverRepository repository, // Dependency injection is supported
    ApprovalService service)
{
    [HttpPost("approvals")] // Map your async endpoint using ASP.NET Core attributes
    public async DTask<IResult> NewApproval(NewApprovalRequest request) // Returning DTask allows you to write async endpoints
    {
        // Look up approver's email in the database
        string? email = await repository.GetEmailByIdAsync(request.ApproverId); // Await any "normal" Tasks, including those that are non-deterministic or have side effects
        if (email is null)
            return Results.BadRequest("Invalid approver id");

        // Send approval/rejection links to the approver
        DTask<ApprovalResult> approvalTask = service.SendApprovalRequestDAsync(request.Details, email); // This DTask will complete when the approver clicks on either link
        DTask timeout = DTask.Delay(TimeSpan.FromDays(7)); // Give them 7 days to review the request

        // Wait until either the approver reviews the request or the timeout expires
        DTask winner = await DTask.WhenAny(timeout, approvalTask); // DTasks has an API similar to Task, including DTask.WhenAny, DTask.WhenAll, etc.
        ApprovalResult result = winner == timeout
            ? ApprovalResult.Reject
            : approvalTask.Result; // "DTask.Result" can be accessed only if the DTask was awaited, otherwise it throws

        return AsyncResults.Success(result); // If you need to return IResult, use AsyncResults.Success to terminate the workflow
    }
}

[gianvitodifilippo]

Hi, thanks for your question! I'll try to make the readme clearer.
The state that's saved into Redis in the provided samples corresponds to the local variables of the methods. In the piece of code you pasted, they would be request, email, approvalTask, and so on.
I claim a DTask is distributed because it may start on one machine/process, it may resume on another one after awaiting another DTask. This can happen because the original process is down or for load-balancing reasons (even though this logic is not yet implemented).
Have a look at the README of this sample if you haven't already, it provides a step-by-step guide that should hopefully make things clearer. There is an optional step that instructs to restart the service while awaiting DTask.WhenAny(timeout, approvalTask) and it shows that the state of the method is not lost and it resumes just like nothing happened.