PrometheusExporter::Client still sending data after shutdown

[fabn]

During our rails app shutdown we got lot of errors like this:

Prometheus Exporter, failed to send message Connection refused - connect(2) for \"localhost\" port 9394

This is happening because the metrics sidecar is faster than our up in shutdown, so it finishes before us.

I added a preStop hook in our app deployment that performs (more or less) the following ruby code:

    logger.info "Stopping PrometheusExporter and relevant instrumentation"
    # Stop the periodic stats, stop is a noop if instrumentation didn't start thus is safe to
    # call it for every instrumentation we're using
    PrometheusExporter::Instrumentation::Puma.stop
    PrometheusExporter::Instrumentation::ActiveRecord.stop
    PrometheusExporter::Instrumentation::Process.stop
    PrometheusExporter::Instrumentation::SidekiqProcess.stop
    PrometheusExporter::Instrumentation::SidekiqQueue.stop
    PrometheusExporter::Instrumentation::SidekiqStats.stop
    # Finally also stop the client, just to be safe
    PrometheusExporter::Client.default.stop(wait_timeout_seconds: 10)
    logger.info "Stopped all Prometheus instrumentations"

But I still get 2 or 3 messages after Stopped all Prometheus instrumentations log message and that sounds very weird to me. Am I doing something wrong? Is this the proper way to stop the exporter?

Here's a snippet from our logfiles.

timestamp;level;message;name
2024-10-18T14:18:25.608302Z;error;"Prometheus Exporter, failed to send message Connection refused - connect(2) for ""localhost"" port 9394";PrometheusExporter::Client
2024-10-18T14:18:25.107240Z;error;"Prometheus Exporter, failed to send message Connection refused - connect(2) for ""localhost"" port 9394";PrometheusExporter::Client
2024-10-18T14:18:24.606151Z;error;"Prometheus Exporter, failed to send message Connection refused - connect(2) for ""localhost"" port 9394";PrometheusExporter::Client
2024-10-18T14:18:24.603777Z;info;Completed #shutdown;Api::Internal::ShutdownController
2024-10-18T14:18:24.530111Z;info;Stopped all Prometheus instrumentations;MetricsExporter
2024-10-18T14:18:24.528236Z;info;Requested shutdown by internal API;Api::Internal::ShutdownController