GenServer Shared.Infrastructure.Telemetry.PromEx.ETSCronFlusher terminating

[krisleech]

Describe the bug
We have been getting about 1.6k of these errors per day.

stacktrace:

    lib/task.ex:892: Elixir.Task.Task.await_receive/3
    lib/prom_ex/ets_cron_flusher.ex:61: Elixir.PromEx.ETSCronFlusher.PromEx.ETSCronFlusher.handle_info/2
    gen_server.erl:1095: gen_server.:gen_server.try_handle_info/3
    gen_server.erl:1183: gen_server.:gen_server.handle_msg/6
    proc_lib.erl:241: proc_lib.:proc_lib.init_p_do_apply/3

message:

GenServer Shared.Infrastructure.Telemetry.PromEx.ETSCronFlusher terminating
** (stop) exited in: Task.await(%Task{mfa: {:erlang, :apply, 2}, owner: #PID<0.5266.0>, pid: #PID<0.55400.0>, ref: #Reference<0.0.674051.2997190859.2677866497.240445>}, 10000)
    ** (EXIT) time out
    (elixir 1.19.3) lib/task.ex:892: Task.await_receive/3
    (prom_ex 1.11.0) lib/prom_ex/ets_cron_flusher.ex:61: PromEx.ETSCronFlusher.handle_info/2
    (stdlib 5.2.3.3) gen_server.erl:1095: :gen_server.try_handle_info/3
    (stdlib 5.2.3.3) gen_server.erl:1183: :gen_server.handle_msg/6
    (stdlib 5.2.3.3) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
Last message: :flush_ets
State: %{timer_ref: #Reference<0.2997190859.2677800961.230034>, prom_ex_module: Shared.Infrastructure.Telemetry.PromEx, ets_flush_interval: 7500}

Maybe this is expected and I prevent this error being reported.

Environment

• PromEx: 1.11.0
• Elixir version: 1.19.3
• Erlang/OTP version: 26
• Grafana version: n/a
• Prometheus version: n/a

Additional context
Add any other context about the problem here.

Many thanks for any advice! ❤️

[btribouillet]

We hit the same issue in production (~500 crashes/12h). The root cause is Task.await/2 in ETSCronFlusher.handle_info/2 which raises an exit on timeout, terminating the GenServer.

Opened a fix in #281 which:

1. Replaces Task.await with Task.yield + Task.shutdown so the GenServer logs a warning and continues instead of crashing.
2. Adds a configurable ets_flush_timeout option (default 10_000ms) so users on busy/shared-cpu machines can increase it:

config :my_app, MyApp.PromEx,
  ets_flush_timeout: 30_000

[akoutmos]

Out of curiosity are you guys using the Peep storage engine, or the default one (that being telemetry_metrics_prometheus_core)?

[btribouillet]

We're using the default storage engine (telemetry_metrics_prometheus_core), not Peep.