Improve automated browser testing with real server

[danroth27]

Feedback from @shanselman:

We should do better with testing. There's issues with moving from the inside out:

LEVELS OF TESTING

• GOOD - Unit Testing - Make a PageModel and call On Get
• GOOD - Functional Testing - Make a WebApplicationFactory and make in-memory HTTP calls
• BAD - Automated Browser Testing with Real Server - can't easily use Selenium or call a real server. I shouldn't have to do this. We should decouple the WebApplicationFactory from the concrete TestServer implementation. @davidfowl

public class RealServerFactory<TStartup> : WebApplicationFactory<Startup> where TStartup : class
{
    IWebHost _host;
    public string RootUri { get; set; }
    public RealServerFactory()
    {
        ClientOptions.BaseAddress = new Uri("https://localhost");
    }

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.UseEnvironment("Development"); //will be default in RC1
    }

    protected override TestServer CreateServer(IWebHostBuilder builder)
    {
        //Real TCP port
        _host = builder.Build();
        _host.Start();
        RootUri = _host.ServerFeatures.Get<IServerAddressesFeature>().Addresses.FirstOrDefault();

        //Fake Server we won't use...sad!
        return new TestServer(new WebHostBuilder().UseStartup<TStartup>());
    }

    protected override void Dispose(bool disposing) 
    {
         base.Dispose(disposing);
         if (disposing) 
        {
                _host.Dispose();
            }
        }
}

/cc @javiercn

[mkArtakMSFT]

As this is not a priority for 2.2 for now, moving to the backlog.

[davidfowl]

@mkArtakMSFT just to clarify the concrete change we should make in 2.2:

We should make it so that it's possible to boot up the WebApplicationFactory (or another derived type) without a test server. It makes functional testing of your application absolutely trivial and the changes required to do this should be small.

/cc @javiercn

[steveoh]

I'm not sure I follow. How would these changes make it easier to use something like puppeteer and chrome headless/selenium for automated browser testing. What is a real server?

[shanselman]

@steveoh the current set up allows for very convenient Unit Testing by spinning up the App/WebHost and talking to it 'in memory." So no HTTP, no security issue, you're basically talking HTTP without actually putting bytes on the wire (or localhost). Super useful.

But if you want to use Automated Browser Testing and have it driven by a Unit Test Framework, you are hampered by the concrete TestServer class and have to do a little dance (above) to fake it out and start up the WebHost yourself. I'd propose no regular person could/would figure that out.

David is saying that if WebApplicationFactory could be started without the fake TestServer we could easily and cleanly do Unit Testing AND Browser Automation Testing with the piece we have here.

[rgamage]

This is why we love Scott! Always the advocate for the mainstream developers out there wanting to use MS tools, but stymied by various obscure limitations. Thank you sir!

[giggio]

The provided workaround has problems.

The problems start with the fact that now we have 2 hosts. One from the TestServer, and another built to work with http. And WebApplicationFactory.Server references the first one, which we are not testing against. And to make things worse, calls to methods that configure the builder, such as WebApplicationFactory.WithWebHostBuilder will not work with the dummy TestServer.

Because of these problems we cannot easily interact with the real host, the one being tested. It is very common that I change some backend service and configure it before a test is run. Suppose I need to access some service that is not callable during development, only production. I replace that service when I configure the services collection with a fake, and then configure it to respond the way I want it to respond. I can't do that through WebApplicationFactory.Server.Host.Services.

The resulting code I have works, but it is ugly as hell, it is an ugly ugly hack.

I hope we can move this forward and do not require TestServer, maybe an IServer. I thought about forking the whole TestServer and WebApplicationFactory infrastructure, but as this is somewhat planning I'll wait. I hope it gets fixed soon. I am just commenting to complement that the provided workaround is not enough and to really work around you have to avoid WebApplicationFactory.Server and create a terrible work around it.

[bchavez]

One way I solved this is to stop using WebApplicationFactory<T> all together.

Refactored Program.Main() to:

public static Task<int> Main(string[] args)
{
   return RunServer(args);
}
public static async Task<int> RunServer(string[] args,
                                        CancellationToken cancellationToken = default)
{
    ...
    CreateWebHostBuilder()
          .Build()
          .RunAsync(cancellationToken)
}

So, my unit test fixtures new up a var cts = new CancellationTokenSource(), then pass the cancellation token by calling Program.RunServer(new string[0], cts.Token). The server starts up as normal without having to create a separate process.

Make real HTTP calls like normal. When your done and your unit test completes, call cts.Cancel() to clean up and shutdown the HTTP server.

One down side is you need to copy appsettings.json to an output directory from your test project; potentially managing two appsettings.json files (one in your server project, and one in your test project). Maybe a linked project file could help eliminate the issue.

YMMV.

🤔 ✨ "Do you ponder the manner of things... yeah yeah... like glitter and gold..."

[giggio]

Hello everyone, this issue is still unresolved and seems to keep being postponed. Just so we know the planning, are you considering resolving this for ASP.NET Core 3.0? If not, do you have a workaround that does not incur on the problems I mentioned earlier (#4892 (comment))?

[BennyM]

This would be very welcome. The workaround mentioned here is great, except when you have html files and what not that you would also need to copy over.

[Tratcher]

WebApplicationFactory is designed for a very specific in-memory scenario. Having it start Kestrel instead and wire up HttpClient to match is a fairly different feature set. We actually do have components that do this in Microsoft.AspNetCore.Server.IntegrationTesting but we've never cleaned them up for broader usage. It might make more sense to leave WebApplicationFactory for in-memory and improve the IntegrationTesting components for this other scenario.