Inconsistent macOS unit tests failures in CI

[mokagio]

You may have noticed some builds failing the macOS tests step. Emphasis on some, because this doesn't seem to be related to the code but to the agent running the builds.

The failure

[16:45:44]: ▸ Build Succeeded
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksiOS_AutomatticTracksModelObjC' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticRemoteLogging' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksConstantsObjC' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksModelObjC' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksiOS_AutomatticTracksTests' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticEncryptedLogs' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksEventsForSwift' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticCrashLoggingUI' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksModel' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracks' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksEvents' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksTestsObjC' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksTests' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ ❌  error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticExperiments' from project 'AutomatticTracksiOS')
  | [16:45:44]: ▸ Testing failed:
  | [16:45:44]: ▸ 	Could not compute signing task inputs - operation was cancelled.
  | [16:45:44]: ▸ 	Testing cancelled because the build failed.
  | [16:45:44]: ▸ ** TEST FAILED **

A quick Google search for this brings up a few results (e.g. this, this, and this) but all related to running tests via Fastlane on a Swift Package.

The inconsistency

It works on my machine... But not only that, the same commit can pass on one agent and fail on another. For example 10f02d

and ef5cc5a

I wonder if this has something to do with the agents? Although I don't recall there being any partial update of our fleet. Update: This turned out not to be the case, see details below.

Details

To verify whether we're facing an agent-based issue, I thought we could list failed builds and their agents. If the certain agents consistently pass or fail, that would be an indicator to further investigate the theory.

Agent	Failed Build(s)	Passed Build(s)
001		https://buildkite.com/automattic/tracks-ios/builds/179
002	https://buildkite.com/automattic/tracks-ios/builds/179	https://buildkite.com/automattic/tracks-ios/builds/177 (notice it was before all other failures, though, 2022/08/26 vs 08/29+ 🤔 )
004	https://buildkite.com/automattic/tracks-ios/builds/167	https://buildkite.com/automattic/tracks-ios/builds/198
007	https://buildkite.com/automattic/tracks-ios/builds/180
013	https://buildkite.com/automattic/tracks-ios/builds/167
014		https://buildkite.com/automattic/tracks-ios/builds/167, https://buildkite.com/automattic/tracks-ios/builds/191

Notice that the table is quite empty right now. This repository doesn't see much action.

Update: As you can see, the MV-MKE-X64-004 agent has a failed and a successful build, so clearly the agent-based theory was wrong.