feat: Ruby Logger instrumentation

[kaylareopelle]

Description

This an OpenTelemetry logs bridge for Ruby's standard Logger library.

It also includes patches to ActiveSupport::Logger.broadcast and the ActiveSupport::BroadcastLogger to emit only one log record for a broadcast.

@khushijain21 is a co-author of this PR and contributed functionality as part of her LFX mentorship with OpenTelemetry in 2024.

Closes #668

[kaylareopelle]

New Relic uses formatted_message here. This includes both the progname and the message in the body, whereas msg does not include the progname. Which seems more appropriate?

[kaylareopelle]

I'm still trying to figure out why one of the tests is failing on the CI, but passes locally. However, everything except the ActiveSupportLogger code should be ready for review. Please take a look and I'll push the update as soon as I can!

[tomash]

I am very interested in getting this to work with Opentelemetry. Any way I could help with this work?

[kaylareopelle]

Hi @tomash! Thanks for reaching out. It's great to hear there's interest in this PR. I'd love your help!

The features for the bridge should be complete. I just updated the branch to align with some of the changes in main. Before this is merged, we need to agree on the approach and get more feedback overall.

Specifically, I could use help with:

1. Reviewing the code: What are your thoughts on how OpenTelemetry logs are bridged from the Ruby Logger? Do you have suggestions on how we could improve the approach? An alternative approach has been presented in Rails <> OpenTelemetry Logs opentelemetry-ruby#1789, would you rather see something like this?

2. Testing the code in an app: Install the gem from this branch and provide feedback on how it works in your application. This is a Gemfile from a sample application I wrote to add the OpenTelemetry bridge for the Ruby logger to a Rails application. Give it a try and let us know how it goes.

[tomash]

@kaylareopelle

For now I have this monstrosity in Gemfile:

gem "opentelemetry-instrumentation-logger",
  git: "https://github.com/kaylareopelle/opentelemetry-ruby-contrib.git",
  branch: "logger-instrumentation",
  glob: "instrumentation/logger/opentelemetry-instrumentation-logger.gemspec"

And I can confirm it works well for our app (the logs appear in Honeycomb just as events). The only problem is that Rails logs are noisy by default and it makes "top messages" unusable (top message being "rendered profiles/_carousel.html.erb"), so I've also dropped in Lograge to produce fewer logs.

I haven't seen open-telemetry/opentelemetry-ruby#1789 , will evaluate and compare! I do like how lightweight and easy to read it is.

[tomash]

So I'm after a few weeks of running this PR in production for a medium-sized app with a lot of traffic, together with Lograge. Works beautifully.

I've also evaluated open-telemetry/opentelemetry-ruby#1789 but we have a specific setup -- using both Newrelic and Honeycomb -- and that one-screener did not produce logs to both outputs (so maybe it was intercepting too hard). It also does not use OTEL_EXPORTER_OTLP_HEADERS so this PR is our choice.

The only flaw I found in this one is a very specific case.

1. Due to our traffic we do head-based sampling with OTEL_TRACES_SAMPLER=traceidratio and OTEL_TRACES_SAMPLER_ARG=0.01 which works great for sampling our traces.
2. Of course we do not want log sampling and it's all good, 100% of logs are sent out with this PR.
3. BUT because of that sampling, to have proper counts and reflect trace representation, we have OTEL_RESOURCE_ATTRIBUTES="SampleRate=100" (a reverse of 0.01 sampling fraction).
4. This SampleRate attribute seems to be also attached to log events because for each log entry Honeycomb multiplies its occurence count by 100.

[fguillen]

Hello @tomash I also would like to test this in my Rails project. Can you share your integration? Especially interested in the lograge integration. Thanks!

UPDATE: forget it, I see it just work out of the box after installing the gem with your gem call piece of art and requiring the gem.

I am sharing here my configurations:

# Gemfile
gem "opentelemetry-exporter-otlp-logs"
gem "opentelemetry-exporter-otlp-metrics"
gem "opentelemetry-exporter-otlp"
gem "opentelemetry-instrumentation-all"
gem "opentelemetry-sdk"
gem "opentelemetry-instrumentation-logger",
  git: "https://github.com/kaylareopelle/opentelemetry-ruby-contrib.git",
  branch: "logger-instrumentation",
  glob: "instrumentation/logger/opentelemetry-instrumentation-logger.gemspec"

# config/initializers/lograge.rb
Rails.application.configure do
  config.lograge.enabled = true
  config.lograge.formatter = Lograge::Formatters::Json.new

  # User id and host (this is very personal for my project)
  config.lograge.custom_payload do |controller|
    user_id = nil
    user_id = controller.send(:current_front_user).try(:uuid) if controller.respond_to?(:current_front_user, true)
    user_id ||= controller.send(:current_tester_user).try(:uuid) if controller.respond_to?(:current_tester_user, true)
    user_id ||= controller.send(:current_admin_user).try(:uuid) if controller.respond_to?(:current_admin_user, true)
    {
      host: controller.request.host,
      user_id:
    }
  end

  config.lograge.custom_options = lambda do |event|
    context = OpenTelemetry::Trace.current_span.context
    {
      trace_id: context.valid? ? context.hex_trace_id : nil,
      span_id: context.valid? ? context.hex_span_id : nil,
      user_id: event.payload[:user_id], # if available
      host: event.payload[:host],
      remote_ip: event.payload[:remote_ip],
      params: event.payload[:params].except("controller", "action")
    }
  end
end

# config/initializers/opentelemetry.rb
ENV["OTEL_TRACES_EXPORTER"] ||= "otlp"
ENV["OTEL_METRICS_EXPORTER"] ||= "otlp"
ENV["OTEL_LOGS_EXPORTER"] ||= "otlp"
ENV["OTEL_EXPORTER_OTLP_ENDPOINT"] ||= "https://myendpoint:4318"
ENV["OTEL_LOG_LEVEL"] ||= "info"

# Basic auth (I haven't test this yet)
username = "MyUser"
password = "MyPass"
basic_auth_hash = Base64.strict_encode64("#{username}:#{password}")
basic_auth_header = "Authorization=Basic #{basic_auth_hash}"s
ENV["OTEL_EXPORTER_OTLP_HEADERS"] = basic_auth_header

require "opentelemetry-metrics-sdk"
require "opentelemetry-logs-sdk"
require "opentelemetry/sdk"
require "opentelemetry/exporter/otlp"
require "opentelemetry/instrumentation/all"
require "opentelemetry-instrumentation-logger"

OpenTelemetry::SDK.configure do |c|
  c.service_name = "app.playcocola.com"
  c.use_all() # enables all instrumentation!
end

UPDATE 2: I see the author or the PR has a proper Rails demo published

[fguillen]

Funny behaviour. If I have activated ENV["OTEL_LOG_LEVEL"] ="debug". When a log is exported another DEBUG log is generated like:

D, [2025-05-01T22:42:17.912579 #95172] DEBUG -- : Successfully exported 1 log records

And it gets into an infinite loop. It may be expected. But just for you to take into consideration :)

[tomash]

@fguillen I was about to write an answer but you figured it all out in the meantime!

Sharing my slightly-different lograge.rb initializer:

Rails.application.configure do
  config.lograge.enabled = true

  # the hotwire connect-disconnect logs are just noise
  config.lograge.ignore_actions = [
    "Turbo::StreamsChannel#subscribe",
    "Turbo::StreamsChannel#unsubscribe",
    "Hotwire::Livereload::ReloadChannel#subscribe",
    "Hotwire::Livereload::ReloadChannel#unsubscribe",
    "ApplicationCable::Connection#connect",
    "ApplicationCable::Connection#disconnect",
    "ApplicationCable::Connection#reconnect",
    "ActionCable::Connection::Base#connect",
    "ActionCable::Connection::Base#disconnect",
  ]
  config.lograge.custom_payload do |controller|
    {
      params: controller.request.filtered_parameters,
    }
  end
end

In opentelemetry.rb initializer we also have this line, as I saw it in this PR's example code:

  at_exit do
    OpenTelemetry.logger_provider.shutdown
  end

Why do you use OTEL_LOG_LEVEL instead of application-global log level?