Preserve and sanitise imported heartbeat strings

[skyfallwastaken]

Summary of the problem

The deprecated Oj SAJ parser silently truncated imported strings at embedded NUL characters. Import normalisation also lacked the sanitation used by direct ingestion.

Describe your changes

Use the streaming ScHandler parser and strip NUL characters before normalisation and identity hashing, preserving the remaining string content and replay deduplication.

Screenshots / Media

Not applicable: backend-only changes.

[greptile-apps]

Greptile Summary

This PR replaces the deprecated SAJ parser with Oj's streaming ScHandler and sanitises imported heartbeat strings before normalisation and identity hashing.

• Streams heartbeat objects without retaining the surrounding export structure.
• Removes embedded NUL bytes from heartbeat values and user-agent metadata.
• Canonicalises user-agent lookup keys so sanitised heartbeat IDs still resolve metadata.
• Adds regression coverage for preserved string suffixes, metadata lookup and replay deduplication.

Confidence Score: 5/5

The PR appears safe to merge, with the previous metadata-lookup issue addressed and regression tests covering the corrected behaviour.

The current implementation sanitises both imported user-agent identifiers and metadata-map keys before lookup, fully addressing the previous finding. No new actionable correctness, security or repository-rule violations remain.

Important Files Changed

Filename	Overview
app/services/heartbeat_import_service.rb	Replaces SAJ parsing with a streaming ScHandler that preserves complete heartbeat strings.
app/services/heartbeat_ingest.rb	Sanitises imported heartbeat data, user-agent metadata and lookup identifiers consistently.
test/services/heartbeat_import_service_test.rb	Covers NUL removal, preservation of valid suffix content and deduplication across corrected replays.
test/services/heartbeat_ingest_test.rb	Covers sanitised user-agent identifier resolution, metadata sanitation and replay deduplication.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Imported JSON] --> B[Oj ScHandler]
  B --> C[Stream heartbeat objects]
  C --> D[Strip embedded NUL bytes]
  E[User-agent metadata map] --> F[Sanitise identifier keys]
  F --> G[Resolve metadata]
  D --> G
  G --> H[Normalise heartbeat]
  H --> I[Compute canonical identity]
  I --> J[Persist or classify as duplicate]

Loading

_{Reviews (2): Last reviewed commit: "Sanitise imported user agent lookup iden..." | Re-trigger Greptile}