FIX: rustc (<1.98) segfault in LatencyMetric

[thorfour]

We discovered that using a closure in the LatencyMetric:new call exposes a rustc bug where the second call to the closure ends up using the same memory buffer as the previous call to builder. This eventually causes a segfault when it goes to drop the second metrics and discovers it's already been dropped. This incorrect memory aliasing is fixed in rustc 1.98 and newer.

To recreate the panic in the test add

[profile.release]
panic = "abort"

to Cargo.toml and run

RUSTC_BOOTSTRAP=1 \
RUSTC="$(rustup which --toolchain stable rustc)" \
"$(rustup which --toolchain nightly cargo)" -Z panic-abort-tests \
  test -p datafusion-distributed --release --lib \
  coordinator_metrics_have_distinct

Need to run 1.96 rustc or lower, but need the nightly build of cargo to include the -Z panic-abort-tests flag.

running 1 test
test coordinator::query_coordinator::tests::coordinator_metrics_have_distinct_label_buffers ... FAILED

failures:

---- coordinator::query_coordinator::tests::coordinator_metrics_have_distinct_label_buffers stdout ----
---- coordinator::query_coordinator::tests::coordinator_metrics_have_distinct_label_buffers stderr ----

thread 'main' (315798) panicked at src/coordinator/query_coordinator.rs:522:17:
assertion `left == right` failed: iteration 0: labeled metric at 0x55d2077ae070 carries 2 labels, expected 1 — label-buffer aliasing miscompilation
  left: 2
 right: 1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    coordinator::query_coordinator::tests::coordinator_metrics_have_distinct_label_buffers

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 269 filtered out; finished in 0.12s

In our production we confirmed the panic no longer happens with the removal of the closure.

[gabotechs]

This is pretty interesting. Could this be the issue you are facing?

rust-lang/rust#155241

[gabotechs]

I confirm that I can reproduce this bug in main. Thanks @thorfour!

[thorfour]

This is pretty interesting. Could this be the issue you are facing?

rust-lang/rust#155241

That sounds like exactly it! Good find