xUnit console runner fails to execute some (random) tests on Team City

[gmanvel]

I can't reproduce this if I run tests via Visual Studio 2017 (xunit.runner.visualstudio version="2.1.0").
Issue only appears on Team City (TeamCity 9.1.6, build 37459), randomly failing on different tests within a test dll.

I tried to execute following command on the Team City itself

C:\BuildAgent\tools\xunit-runner\bin\2.1.0\xunit.console.exe C:\BuildAgent\work\ba34c42088193a39\tests\bin\Release\SharedServices.Tests.dll -teamcity

It keeps failing with the following error message:

System.Runtime.Serialization.SerializationException: Unable to find assembly 'Microsoft.WindowsAzure.Storage, Version=6.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. [Testing SharedServices.Tests.dll] at System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssembly() [Testing SharedServices.Tests.dll] at System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType(BinaryAssemblyInfo assemblyInfo, String name) [Testing SharedServices.Tests.dll] at System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor(String objectName, String[] memberNames, BinaryTypeEnum[] binaryTypeEnumA, Object[] typeInformationA, Int32[] memberAssemIds, ObjectReader objectReader, Int32 objectId, BinaryAssemblyInfo assemblyInfo, SizedArray assemIdToAssemblyTable) [Testing SharedServices.Tests.dll] at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryObjectWithMapTyped record) [Testing SharedServices.Tests.dll] at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run() [Testing SharedServices.Tests.dll] at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage) [Testing SharedServices.Tests.dll] at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage) [Testing SharedServices.Tests.dll] at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.DeserializeObject(MemoryStream stm) [Testing SharedServices.Tests.dll] at System.AppDomain.Deserialize(Byte[] blob) [Testing SharedServices.Tests.dll] at System.AppDomain.UnmarshalObject(Byte[] blob)

I can confirm that 'Microsoft.WindowsAzure.Storage, Version=6.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35 exists under C:\BuildAgent\work\ba34c42088193a39\tests\bin\Release path.

Trying other versions of runners (v. 2.2.0 & v. 2.4.0) shows the same behavior.
Tried to switch between x86 & x64 runners, unfortunately, same output.

Closest possible solution I could find is https://stackoverflow.com/a/21169644/997668, however, it doesn't seem to help as I'm passing test dll to command explicitly.

[carlpett]

Hey @gmanvel,
Sorry to hear that. Does "randomly failing" mean that the entire test assembly sometimes pass, or that it is random which test is the first to fail?
Assembly loading issues are typically quite hard to debug remotely, but here are some things to check:
Could you open SharedServices.Tests.dll with dotPeek for example and have a look at exactly how this references the Azure assembly? Also, is there a SharedServices.Tests.dll.config file?

[gmanvel]

Thank you for prompt reply @carlpett & I appreciate your help on this.
By "randomly fails" I mean it is random which test is the first to fail. Not all tests fail, quite the opposite, a lot of them pass and then randomly it fails on some test and stops execution.
There is SharedServices.Tests.dll.config, attaching file here
SharedServices.dll.zip

As for how Azure assembly is referenced - I don't see any direct dependency from dotPeeks output.
Attaching screenshot from dotPeek, please let me know if I can provide more information.