Tests don't pass on Windows

[simonpasquier]

Similar to prometheus/prometheus#4622

I tried to run the tests on a Windows machine and some of the tests failed. It probably means that AlertManager doesn't run properly on Windows though we release the Windows binaries.

I may try to fix the tests soon but anybody pick it up if desired. Eventually we need somehow to think about which level of support we expect for Windows.

[brian-brazil]

#6457 added running tests on Windows, but only for TSDB. We should fix the rest and then run them too.

[roidelapluie]

@brian-brazil wrong repository I guess

[brian-brazil]

Ah yes, so it is.

[goran-rumin]

I will try to tackle this in the next couple of days. From the test results I have a rough idea what are the issues.

[goran-rumin]

I fixed windows specific stuff, but as it turns out, making all tests pass is a bit more work :D

More specifically, I had issues with acceptance tests, and after some investigation, I pinpointed the issue to setting the test start absolute time at the start of acceptance test, then defining assertions in relative time (which use the mentioned start time for calculations) and then starting alertmanager instances. On my 2012 machine, the tolerance of 150ms wasn't enough, so the current testing logic seems flaky.
Explanation on the example of CLI acceptance test is below:

- amtool executed at 2020-01-09 20:24:18.3542521 +0100 CET 
  with params --alertmanager.url=http://127.0.0.1:60306/,alert,add,alertname=test1,--start=2020-01-09T20:24:17.905+01:00,--end=2020-01-09T20:24:18.905+01:00
- Actual POST request:
  2020-01-09 20:24:18.455937 POST /api/v2/alerts
- AM logs:
  level=debug ts=2020-01-09T19:24:18.457Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=test1[5ae90ff][active]
  level=debug ts=2020-01-09T19:24:19.457Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"test1\"}" msg=flushing alerts=[test1[5ae90ff][resolved]]

I was trying to change tests so that the evaluation begins once all dependencies are ready, but it would be great if I could hear your input and ideas in this.