Allow the runtime to time travel

[edumserrano]

Idea

I'd like the runtime to provide a mechanism for the user to control the passage of time when using operations like Task.Dealy, PeriodicTimer and any other related timing APIs.

Has the runtime team ever talked about something like this?

Problem statement

When testing code that depends on time the test results will hardly be deterministic. They will vary for instance due to hardware. A test suite might always run fine locally on a powerful developer machine but then be flaky when running on CI because the agents have much lower hardware specification. In my experience, even running tests in parallel can contribute to increased flakiness in this type of tests, probably due to pushing CPU more when compared to running the tests sequentially.

Recently I faced an issue in one of my personal projects that I wished the runtime had better support for. The problem was some of my tests were flaky because they depended on time, more specifically in checking that X amount of actions happened in Y amount of time.

The code being tested sometimes used a Task.Delay and sometimes a PeriodicTimer. In the end I was able to significantly reduce the flakiness of the tests by replacing PeriodicTimer or Task.Delay with the use of RX. I created the following:

internal sealed class RxPeriodicTimer
{
    private readonly TimeSpan _period;
    private readonly IScheduler _scheduler;

    public RxPeriodicTimer(TimeSpan period, IScheduler scheduler)
    {
        _period = period;
        _scheduler = scheduler;
    }

    public Task WaitForNextTickAsync(CancellationToken cancellationToken = default)
    {
        return Observable
            .Timer(_period, _scheduler)
            .ToTask(cancellationToken);
    }
}

The idea of the above is that in tests I can pass in a TestScheduler as the IScheduler to control the passage of time. So during tests I could have things like this:

var runUntilTimeout = TimeSpan.FromMilliseconds(100);
var runUntilTimeoutTask = hostedServicesWebAppFactory.RunUntilTimeoutAsync(runUntilTimeout);
callCount.ShouldBe(0);
testScheduler.AdvanceBy(MyBackgroundService.Period.Ticks);
callCount.ShouldBe(1);
testScheduler.AdvanceBy(MyBackgroundService.Period.Ticks);
callCount.ShouldBe(2);
testScheduler.AdvanceBy(MyBackgroundService.Period.Ticks);
callCount.ShouldBe(3);
await runUntilTimeoutTask;

Regardless of the details of what the above code sample is doing, what I'm trying to call out is that with RX and the TestScheduler I can then control the passage of time as I see fit and assert on what should have happened after X time. Note that the TestScheduler even supports more than just the AdvanceBy as a way to control time and schedule operations.

Existing solutions

Abstract time dependency

As a dotnet runtime user I can abstract away the parts of my code that interact with time related APIs. Then on tests I can pass in a a different implementation of my abstraction in order to test my app.

Or perhaps I can avoid using the built in dotnet runtime APIs such as Task.Delay and PeriodicTImer and use RX, or similar libraries, to achieve equivalent outcomes. Like the RxPeriodicTimer class shown above. Then on tests I would have to be able to provide the TestScheduler to my app whilst by default the system would use the real scheduler.

Patching the runtime during tests

As a dotnet runtime user I could use a library like Harmony to provide my own implementation of the dotnet runtime time APIs that I use on my app. This patching would only be applied on tests and the implementation I provide with the patching would have to provide a way to control time.

This is something that the fake-async library does. As far as I understand the fundamental building block for this library is the replacement of dotnet APIs by using Harmony. For instance, see how fake-async replaces Task.Delay with something else.

Other

There probably are other variations of the above but is there anything else significantly different that would solve this problem ?

Notes

• The main use case for this would be for test purposes.
• At the moment I don't think I have enough to try and do an API proposal. Right now it's just an idea and I felt it was better to start with a discussion to understand if this is even feasible in the near future or in which scope it could be done. Meaning, perhaps a time control mechanism would not be "added" to all APIs but only a few.
• There have been conversations on at least providing a way to control the value given by the current system time. See API to provide the current system time #36617. I believe what I'm asking here goes further than that because I'd like to be able to control time when using other APIs such as Task.Delay or PeriodicTimer. Or perhaps it's related and there's a way to coordinate both efforts.
• It might help show how other languages provide similar concepts. See Angular fake-async.

[stephentoub]

#36617 (see the latest iterations on what's being proposed... it's not just current time).

[edumserrano]

Well now I feel silly, I took the time write up my idea as best as I could and did research to make sure I could find out related issues and it turns out one of the issues I mentioned is a proposal that solves the problem.

@stephentoub my apologies, you are correct. I read #36617 properly and the proposal as it is will work for my described scenario given it has for instance the TimeProvider.CreatePeriodicTimer and TimeProvider.DelayAsync methods.

At some point perhaps you could also consider providing us with a TestTimeProvider similar to how the package Microsoft.AspNetCore.Mvc.Testing provides a WebApplicationFactory to facilitate testing. I'm thinking it would be great to have something ready to use out of the box in testing scenarios that would allow us to start writing tests straight away without having to override some of the virtual methods or provide a custom implementation for interfaces like IPeriodicTimer. I guess the open source community will eventualy come up with something like that even if it's not available as part of doing the work on #36617.

It would also be nice that a TestTimeProvider or a TestPeriodicTimer could have some methods that probably don't make sense to be on the TimeProvider class or the IPeriodicTimer interface but would facilitate the manipulation of time in tests. For instance, an TestPeriodicTimer.AdvanceBy(TimeSpan interval) method that would adjust the timer by the given time interval. Or perhaps these extra test methods wouldn't be needed and I'm just not properly understanding how easily it would be to use the APIs proposed in #36617 to solve the problem I described.

I do think however that adding some default test implementation classes for the APIs in #36617 would be a nice bonus. (hopefully this wasn't something you also already discussed in that API proposal, I've read the initial and ending comments but not all of the 127 comments in between).

P.S. - thank you @stephentoub for replying. Fan of your blogs posts 😄

[stephentoub]

Well now I feel silly

No need to feel silly :) That proposal has evolved significantly over time.

At some point perhaps you could also consider providing us with a TestTimeProvider

Yes, we plan to make something along those lines available, one where you can manually set and advance the time.

thank you @stephentoub for replying. Fan of your blogs posts 😄

Thanks for saying so. Much appreciated.

[Xyncgas]

Have you considered compiling your code to wasm and running it in the browser which you can throttle the performance