Sanitize encoding at data-fill points instead of reactive deep-encode retries

Per review feedback: drop the reactive 'catch JSON.generate, deep-encode,
and retry' pattern from serialization exit points (Envelope::Item,
Transport#serialize_envelope headers, Event#to_json_compatible,
TelemetryEventBuffer, DebugTransport, DebugStructuredLogger). Those
are reverted to plain JSON.generate/JSON.dump.

Instead, sanitize encoding proactively wherever the affected data
actually gets filled in:

- Breadcrumb#data= now force-encodes/scrubs strings in the arbitrary
  user-supplied data hash to valid UTF-8 (both on construction and on
  later assignment), matching the existing pattern used for
  Breadcrumb#message=, SingleException#value, and RequestInterface's
  body/headers.
- Utils::TelemetryAttributes#attribute_hash now sanitizes String
  attribute values directly, and sanitizes non-scalar (Hash/Array)
  attribute values before JSON.generate rather than reacting to a
  raised error.

Transport#send_envelope keeps its rescue for EncodingError/
JSON::GeneratorError as a last-resort safety net (log + record lost
event) for any case not already covered by the above, without
attempting to fix and retry.

Updated specs accordingly.

Author: sentry-junior[bot]

CHANGELOG.md

@@ -66,7 +66,7 @@ We recommend migrating to `data_collection` to match the behavior you want event
 
 ### Bug Fixes 🐛
 
-- Sanitize payload encodings and rescue `EncodingError`/`JSON::GeneratorError` during envelope/event JSON serialization to support the `json` gem 3.0, which now raises instead of warning when generating JSON from a String tagged with a non-UTF-8 encoding. Fixes #2462
+- Sanitize breadcrumb data and structured-log attribute values to valid UTF-8 where they're filled in, and add a last-resort rescue for `EncodingError`/`JSON::GeneratorError` in `Transport#send_envelope`, to support the `json` gem 3.0, which now raises instead of warning when generating JSON from a String tagged with a non-UTF-8 encoding. Fixes #2462
 
 ## 6.7.0
 

sentry-ruby/lib/sentry/breadcrumb.rb

@@ -8,7 +8,7 @@ class Breadcrumb
     # @return [String, nil]
     attr_accessor :category
     # @return [Hash, nil]
-    attr_accessor :data
+    attr_reader :data
     # @return [String, nil]
     attr_reader :level
     # @return [Time, Integer, nil]
@@ -26,7 +26,7 @@ class Breadcrumb
     # @param type [String, nil]
     def initialize(category: nil, data: nil, message: nil, timestamp: nil, level: nil, type: nil)
       @category = category
-      @data = data || {}
+      self.data = data
       @timestamp = timestamp || Sentry.utc_now.to_i
       @type = type
       self.message = message
@@ -51,6 +51,14 @@ def message=(message)
       @message = message && Utils::EncodingHelper.valid_utf_8?(message) ? message.byteslice(0..Event::MAX_MESSAGE_SIZE_IN_BYTES) : ""
     end
 
+    # Sanitizes the breadcrumb's arbitrary, user-supplied data so it doesn't
+    # carry a String with an invalid/non-UTF-8 encoding into JSON generation.
+    # @param data [Hash, nil]
+    # @return [void]
+    def data=(data)
+      @data = Utils::EncodingHelper.deep_encode_utf_8(data || {})
+    end
+
     # @param level [String]
     # @return [void]
     def level=(level) # needed to meet the Sentry spec
@@ -62,11 +70,6 @@ def level=(level) # needed to meet the Sentry spec
     def serialized_data
       begin
         ::JSON.parse(::JSON.generate(@data, max_nesting: MAX_NESTING))
-      rescue EncodingError, ::JSON::GeneratorError
-        # As of json 3.0, `JSON.generate` raises instead of warning when it
-        # encounters a String with an invalid/non-UTF-8 encoding (e.g. a
-        # BINARY-tagged String). Sanitize and retry once before giving up.
-        ::JSON.parse(::JSON.generate(Utils::EncodingHelper.deep_encode_utf_8(@data), max_nesting: MAX_NESTING))
       rescue Exception => e
         Sentry.sdk_logger.debug(LOGGER_PROGNAME) do
           <<~MSG

sentry-ruby/lib/sentry/debug_structured_logger.rb

@@ -47,7 +47,7 @@ def capture_log_event(level, message, parameters, **attributes)
         attributes: attributes
       }
 
-      File.open(log_file, "a") { |file| file << safe_json_dump(log_event_json) << "\n" }
+      File.open(log_file, "a") { |file| file << JSON.dump(log_event_json) << "\n" }
       log_event_json
     end
 
@@ -66,15 +66,6 @@ def clear
 
     private
 
-    # Sanitizes `value` before dumping to JSON so a String tagged with an
-    # invalid/non-UTF-8 encoding (which raises with json 3.0+) doesn't
-    # crash debug logging.
-    def safe_json_dump(value)
-      JSON.dump(value)
-    rescue EncodingError, JSON::GeneratorError
-      JSON.dump(Sentry::Utils::EncodingHelper.deep_encode_utf_8(value))
-    end
-
     def initialize_backend(configuration)
       StructuredLogger.new(configuration)
     end

sentry-ruby/lib/sentry/envelope/item.rb

@@ -51,14 +51,11 @@ def item_count
     def lost_event_byte_size
       return unless byte_data_category
 
-      (payload.is_a?(String) ? payload : safe_json_generate(payload)).bytesize
+      (payload.is_a?(String) ? payload : JSON.generate(payload)).bytesize
     end
 
     def to_s
-      [
-        safe_json_generate(@headers),
-        @payload.is_a?(String) ? @payload : safe_json_generate(@payload)
-      ].join("\n")
+      [JSON.generate(@headers), @payload.is_a?(String) ? @payload : JSON.generate(@payload)].join("\n")
     end
 
     def serialize
@@ -79,24 +76,12 @@ def serialize
 
     def size_breakdown
       payload.map do |key, value|
-        "#{key}: #{safe_json_generate(value).bytesize}"
+        "#{key}: #{JSON.generate(value).bytesize}"
       end.join(", ")
     end
 
     private
 
-    # Sanitizes `value` (removing invalid UTF-8 byte sequences from any
-    # tagged strings) before generating JSON, and retries once if
-    # `JSON.generate` still raises an encoding related error.
-    #
-    # See `Utils::EncodingHelper.deep_encode_utf_8` for details on why this
-    # is necessary as of json 3.0.
-    def safe_json_generate(value)
-      JSON.generate(value)
-    rescue EncodingError, JSON::GeneratorError
-      JSON.generate(Utils::EncodingHelper.deep_encode_utf_8(value))
-    end
-
     def remove_breadcrumbs!
       if payload.key?(:breadcrumbs)
         payload.delete(:breadcrumbs)

sentry-ruby/lib/sentry/event.rb

@@ -121,11 +121,7 @@ def to_h
 
     # @return [Hash]
     def to_json_compatible
-      hash = to_h
-
-      JSON.parse(JSON.generate(hash))
-    rescue EncodingError, JSON::GeneratorError
-      JSON.parse(JSON.generate(Utils::EncodingHelper.deep_encode_utf_8(hash)))
+      JSON.parse(JSON.generate(to_h))
     end
 
     private

sentry-ruby/lib/sentry/telemetry_event_buffer.rb

@@ -62,7 +62,7 @@ def add_item(item)
           @client.transport.record_lost_event(
             :queue_overflow,
             @data_category,
-            num_bytes: safe_json_bytesize(item.to_h)
+            num_bytes: JSON.generate(item.to_h).bytesize
           )
         else
           @pending_items << item
@@ -103,7 +103,7 @@ def send_items
             envelope_items << processed_item.to_h
           else
             discarded_count += 1
-            discarded_bytes += safe_json_bytesize(item.to_h)
+            discarded_bytes += JSON.generate(item.to_h).bytesize
           end
         end
       else
@@ -131,14 +131,5 @@ def send_items
     ensure
       clear!
     end
-
-    # Sanitizes `value` before generating JSON so that a String tagged with
-    # an invalid/non-UTF-8 encoding (which raises with json 3.0+) doesn't
-    # crash telemetry buffering.
-    def safe_json_bytesize(value)
-      JSON.generate(value).bytesize
-    rescue EncodingError, JSON::GeneratorError
-      JSON.generate(Utils::EncodingHelper.deep_encode_utf_8(value)).bytesize
-    end
   end
 end

sentry-ruby/lib/sentry/transport.rb

@@ -70,9 +70,9 @@ def send_envelope(envelope)
     rescue EncodingError, JSON::GeneratorError => e
       # As of json 3.0, `JSON.generate` raises instead of warning when it
       # encounters a String with an invalid/non-UTF-8 encoding (e.g. a
-      # BINARY-tagged String). Individual envelope items already sanitize
-      # their own payloads, but this is a last resort so a single bad
-      # event can't crash the background worker.
+      # BINARY-tagged String). We sanitize known data-entry points (e.g.
+      # breadcrumb data, log attributes), but this is a last resort so an
+      # unexpected case can't crash the background worker.
       log_error("[Transport] Failed to serialize envelope", e, debug: @debug)
 
       # `serialized_items` may still be nil here if the error was raised
@@ -101,13 +101,7 @@ def serialize_envelope(envelope)
       end
 
       unless serialized_results.empty?
-        headers = begin
-          JSON.generate(envelope.headers)
-        rescue EncodingError, JSON::GeneratorError
-          JSON.generate(Utils::EncodingHelper.deep_encode_utf_8(envelope.headers))
-        end
-
-        data = [headers, *serialized_results].join("\n")
+        data = [JSON.generate(envelope.headers), *serialized_results].join("\n")
       end
 
       [data, serialized_items]

sentry-ruby/lib/sentry/transport/debug_transport.rb

@@ -36,7 +36,7 @@ def log_envelope(envelope)
         end
       }
 
-      File.open(log_file, "a") { |file| file << safe_json_dump(envelope_json) << "\n" }
+      File.open(log_file, "a") { |file| file << JSON.dump(envelope_json) << "\n" }
     end
 
     def logged_envelopes
@@ -54,15 +54,6 @@ def clear
 
     private
 
-    # Sanitizes `value` before dumping to JSON so a String tagged with an
-    # invalid/non-UTF-8 encoding (which raises with json 3.0+) doesn't
-    # crash debug logging.
-    def safe_json_dump(value)
-      JSON.dump(value)
-    rescue EncodingError, JSON::GeneratorError
-      JSON.dump(Utils::EncodingHelper.deep_encode_utf_8(value))
-    end
-
     def initialize_backend(configuration)
       backend = configuration.dsn.local? ? DummyTransport : HTTPTransport
       backend.new(configuration)

sentry-ruby/lib/sentry/utils/telemetry_attributes.rb

@@ -19,26 +19,20 @@ def attribute_hash(raw_value)
         result =
           case value
           when String
-            { value: value, type: "string" }
+            { value: Utils::EncodingHelper.encode_to_utf_8(value), type: "string" }
           when TrueClass, FalseClass
             { value: value, type: "boolean" }
           when Integer
             { value: value, type: "integer" }
           when Float
             { value: value, type: "double" }
           else
+            # `value` may be an arbitrary object (e.g. a Hash/Array) that
+            # contains a String with an invalid/non-UTF-8 encoding, which
+            # `JSON.generate` raises on as of json 3.0+. Sanitize it before
+            # generating rather than reacting to the error.
             begin
-              { value: JSON.generate(value), type: "string" }
-            rescue EncodingError, JSON::GeneratorError
-              # As of json 3.0, `JSON.generate` raises instead of warning
-              # when it encounters a String with an invalid/non-UTF-8
-              # encoding (e.g. a BINARY-tagged String). Sanitize and retry
-              # once before giving up.
-              begin
-                { value: JSON.generate(Sentry::Utils::EncodingHelper.deep_encode_utf_8(value)), type: "string" }
-              rescue
-                { value: value, type: "string" }
-              end
+              { value: JSON.generate(Utils::EncodingHelper.deep_encode_utf_8(value)), type: "string" }
             rescue
               { value: value, type: "string" }
             end

sentry-ruby/spec/sentry/breadcrumb_spec.rb

@@ -115,22 +115,17 @@
       expect(stringio.string).to match(/can't serialize breadcrumb data because of error: nesting of 10 is too deep/)
     end
 
-    it "sanitizes and retries when JSON.generate raises an encoding error (json 3.0+ behavior)" do
-      # json 3.0+ raises Encoding::UndefinedConversionError (a subclass of
-      # EncodingError) instead of just warning when JSON.generate encounters
-      # a String tagged with a non-UTF-8 encoding that contains bytes
-      # invalid for the target encoding. Simulate that here regardless of
-      # the json gem version actually loaded.
+    it "sanitizes non-UTF-8 encoded strings in data at assignment time (json 3.0+ behavior)" do
+      # json 3.0+ raises Encoding::UndefinedConversionError instead of just
+      # warning when JSON.generate encounters a String tagged with a
+      # non-UTF-8 encoding that contains bytes invalid for the target
+      # encoding. Breadcrumb#data= sanitizes proactively so this never
+      # reaches JSON.generate in the first place.
       invalid_string = "\xFF\xFEinvalid".dup.force_encoding(Encoding::BINARY)
       crumb = Sentry::Breadcrumb.new(category: "foo", message: "crumb", data: { note: invalid_string })
 
-      call_count = 0
-      allow(JSON).to receive(:generate).and_wrap_original do |original, *args|
-        call_count += 1
-        raise EncodingError, "simulated json 3.0 encoding error" if call_count == 1
-
-        original.call(*args)
-      end
+      expect(crumb.data[:note].encoding).to eq(Encoding::UTF_8)
+      expect(crumb.data[:note].valid_encoding?).to eq(true)
 
       result = crumb.to_h
       expect(result[:data]).not_to eq({ error: Sentry::Breadcrumb::DATA_SERIALIZATION_ERROR_MESSAGE })