[Failing Test]: :sdks:go:test:ulrValidatesRunner appears to be very flaky at master

[tvalentyn]

What happened?

Sample failure runs: https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/2996

Also failed on some in-flight PRs.

13:19:29 --- PASS: TestEmitParDoAfterGBK (4.50s)
13:19:29 PASS
13:19:29 ok  	github.com/apache/beam/sdks/v2/go/test/regression	124.686s
13:19:29 FAIL
13:19:29 
13:19:29 > Task :sdks:go:test:ulrValidatesRunner FAILED
13:19:29 
13:19:29 FAILURE: Build failed with an exception.
13:19:29 
13:19:29 * Where:
13:19:29 Build file '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_GoPortable_Cron/src/sdks/go/test/build.gradle' line: 152
13:19:29 
13:19:29 * What went wrong:
13:19:29 Execution failed for task ':sdks:go:test:ulrValidatesRunner'.
13:19:29 > Process 'command 'sh'' finished with non-zero exit value 1
13:19:29 
13:19:29 * Try:
13:19:29 > Run with --stacktrace option to get the stack trace.
13:19:29 > Run with --info or --debug option to get more log output.

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

cc: @lostluck

[lostluck]

Note that this is happening on the Python Portable Runner, but not on any of the other validates runner suites, so it doesn't seem like it's Go side.

I was chasing this down from a different series. It's failing on artifact upload, which doesn't have any recent work on the Go side (https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/artifact).

I'd be concerned it was the datalayer rewrite (#25982), but that wasn't merged until the 28th, and this started happening on the 25th. (The datalayer rewrite does have a flake, but it's in a unit test #26057, not a runner test).

But python's portable runner doesn't seem to have anything new there either.

Very confusing. But then I'd expect to see non-infra flakes in the other runner suite tests.

[lostluck]

2023/03/31 20:19:00 Prepared job with id: go-testxlang_multi-841-7e5f7f7d-e29d-41a5-85b4-c701695d54d4 and staging token: go-testxlang_multi-841-7e5f7f7d-e29d-41a5-85b4-c701695d54d4
    ptest.go:108: Failed to execute job: 	staging artifacts
        failed to stage /tmp/worker-7-1680293938316973953 in 3 attempts: failed to send chunks for /tmp/worker-7-1680293938316973953
        	caused by:
        chunk send failed
        	caused by:
        EOF; failed to send chunks for /tmp/worker-7-1680293938316973953
        	caused by:
        chunk send failed
        	caused by:
        EOF; failed to send chunks for /tmp/worker-7-1680293938316973953
        	caused by:
        chunk send failed
        	caused by:
        EOF; failed to send chunks for /tmp/worker-7-1680293938316973953
        	caused by:
        chunk send failed
        	caused by:
        EOF
--- FAIL: TestXLang_Multi (12.99s)

Same thing for some of the others. But it's very odd that it's happening to the Python runner and not Flink/Spark/Samza.

Feels sort of like a grpc thing, but again, not sure why it's only started recently.

[lostluck]

That path isn't doing the correct thing WRT the error on Send. The EOF means close the stream and see what the server is returning. Typically the EOF means that the server side closed for some reason.
The upload is retrying 3 times and still getting failures like that, so lets try following the proper protocol WRT errors on sends...