Force killing Bazel leaves invocation in progress

[darkrift]

$ bazel test --config re --runs_per_test 100 -t- --build_event_json_file build.json //...
Starting local Bazel server (8.4.2) and connecting to it...
INFO: Invocation ID: 13f33d20-07da-4aaa-9091-0c8a871b6a53
INFO: Streaming build results to: https://<snipped>bazel-invocations/13f33d20-07da-4aaa-9091-0c8a871b6a53
WARNING: The highest API version Bazel support 2.2 is deprecated by the server. Please upgrade to server's recommended version: 2.3 to 2.11.
Analyzing: 451 targets (306 packages loaded, 11139 targets configured)
    currently loading: @@rules_java+//java/runfiles ... (4 packages)

Analyzing: 451 targets (325 packages loaded, 13581 targets configured)
    Fetching module extension @@rules_jvm_external+//:extensions.bzl%maven; starting
^C
Bazel caught interrupt signal; cancelling pending invocation.

^C
Bazel caught third interrupt signal; server killed. (This may be expensive, see https://bazel.build/advanced/performance/iteration-speed#avoid-ctrl-c.)

^C

[arozenfe]

If you stop bazel, it doesn't send the messages to complete the invocation.

It seems there's a new option in the configuration (in one of the last releases) to face this problem.

bb-portal/pkg/proto/configuration/bb_portal/bb_portal.proto

Lines 84 to 89 in 267496b

	// If an invocation that has not yet received the last message in the BES
	// stream does not receive any new messages within this duration, the
	// invocation is considered abandoned, and will be locked to prevent further
	// modifications.
	// Recommended value: '3600s' (i.e., 1 hour)
	google.protobuf.Duration invocation_message_timeout = 3;

[meroton-benjamin]

The recommended timeout is fairly conservative as sometimes bazel might take a really long time in between sending messages. As we do save data about when the last connection was and when the last received event was, we should be able to visualize a suspected disconnection state in the ui before locking (i.e. invocations has no ongoing connection and has not received a message for a certain period of time, it might be disconnected).