[HttpListener] Hang of System.Net.HttpListener.Tests in CI

[danmoseley]

https://ci.dot.net/job/dotnet_corefx/job/master/job/ubuntu14.04_debug_prtest/8257/consoleFull#130661499679494335-f7bd-47d0-8771-8661e00c2db2

22:05:43   Finished running tests. End time=05:05:43. Return value was 0
23:44:38 
Build timed out (after 120 minutes). Marking the build as aborted.
23:44:38 
Build was aborted
23:44:38 [xUnit] [INFO] - Starting to record.
23:44:38 [xUnit] [INFO] - Processing xUnit.Net-v2 (default)
23:44:38 
MSBUILD : error MSB4166: Child node "2" exited prematurely. Shutting down. Diagnostic information may be found in files in the temporary files directory named MSBuild_*.failure.txt.
23:44:38 [xUnit] [INFO] - [xUnit.Net-v2 (default)] - 155 test report file(s) were found with the pattern 'bin/**/testResults.xml' relative to '/mnt/j/workspace/dotnet_corefx/master/ubuntu14.04_debug_prtest' for the testing framework 'xUnit.Net-v2 (default)'.
23:44:38 /mnt/j/workspace/dotnet_corefx/master/ubuntu14.04_debug_prtest/Tools/tests.targets(345,5): warning MSB5021: "sh" and its child processes are being terminated in order to cancel the build. [/mnt/j/workspace/dotnet_corefx/master/ubuntu14.04_debug_prtest/src/System.Net.HttpListener/tests/System.Net.HttpListener.Tests.csproj]
23:44:39 Command execution failed with exit code 143.

Not actionable based on this information, but starting an issue.

[danmoseley]

fyi @hughbe @stephentoub note these are in master.

[danmoseley]

Debian & Ubuntu:
https://mc.dot.net/#/user/mmitche/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fportable~2Fcli~2F/6c6eb50345329c470db608fef70710e52270c507/workItem/System.Net.HttpListener.Tests

[karelz]

Questions:

• Does it happen only on Linux/Mac (product issue), or also on Windows (test issue)

[mellinoe]

If we see this actively hung in any CI jobs, please let me know and I will try to debug it. Bonus points to anyone who logs into a machine and attaches a debugger before the process terminates 😄

[danmoseley]

All the time now.
https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fportable~2Fcli~2F/build/20170525.01/workItem/System.Net.HttpListener.Tests/wilogs

[stephentoub]

Managed to catch one in CI. At least in the one I saw, it appears to be hanging here:
https://github.com/dotnet/corefx/blob/bbb63de0aac837895542e1b126ef71b3ba1a51aa/src/System.Net.HttpListener/tests/HttpRequestStreamTests.cs#L68

@geoffkizer, looks like this might be the same hang you're investigating locally...

[geoffkizer]

Yep, that's exactly what I see locally.

[danmoseley]

https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fportable~2Fcli~2F/build/20170525.02/workItem/System.Net.HttpListener.Tests/wilogs

[danmoseley]

@stephentoub @geoffkizer does this mean if we disable Read_FullLengthAsynchronous_Success it will avoid this hang? If so could one of you please disable it?
There is no need to go through CI for this, as it's so impactful I suggest hitting merge immediately.

[stephentoub]

does this mean if we disable Read_FullLengthAsynchronous_Success it will avoid this hang?

I don't think so; I think there are several (not sure how many) tests that hit the same underlying issue or issues. I don't believe it's a problem with a specific test, but rather with something in the implementation (or something systemic to many tests via code that's shared between them). It's possible we could find and disable enough tests to avoid the hangs.

@geoffkizer has been investigating...

[stephentoub]

But I'll see if I can temporarily disable enough to minimize the impact...

[stephentoub]

A bunch of tests have been temporarily disabled. Just search for 20246 to find them. Let's try to get to the bottom of this asap. Hopefully I disabled enough to prevent the hangs in CI.