Skip to content

Rebuild ZCS realtime data unmarshaler#78

Merged
zmoog merged 4 commits intomainfrom
zmoog/feat/zcs-unmarshaler
Jun 13, 2025
Merged

Rebuild ZCS realtime data unmarshaler#78
zmoog merged 4 commits intomainfrom
zmoog/feat/zcs-unmarshaler

Conversation

@zmoog
Copy link
Copy Markdown
Owner

@zmoog zmoog commented Jun 12, 2025

Summary

Complete overhaul of the ZCS Azzurro receiver with enhanced data processing,
improved marshaling logic, and optimized state management using LRU cache.

Key Changes

🔄 Enhanced Data Processing Pipeline:

  • Multi-device support: Now processes multiple devices from single API response
  • Duplicate detection: Added timestamp-based deduplication to prevent reprocessing
    identical data
  • Robust error handling: Improved error handling with detailed logging and graceful
    recovery
  • Proper unmarshaling: Completely reworked data unmarshaling with device-specific
    processing

⚡ State Management Optimization:

  • Replaced manual tracking: Eliminated custom map + mutex pattern
  • Implemented LRU cache: Added github.com/elastic/go-freelru with 1000-device
    capacity
  • Simplified architecture: Direct timestamp storage instead of wrapper structs
  • Thread-safe operations: Native concurrency support without manual synchronization

🛠️ Technical Improvements:

  • Dependency upgrade: Updated to github.com:zmoog/ws@v0.1.1
  • Better logging: Added debug logging for processing decisions and metrics
  • Memory efficiency: Bounded memory usage with automatic LRU eviction
  • Code reduction: Removed ~40 lines of complex state management code

Before vs After

Previous Implementation:

  // Basic single-response processing
  realtimeDataResponse, err := z.scraper.Scrape(...)
  metrics, err := z.marshaler.UnmarshalMetrics(realtimeDataResponse)
  z.consumer.ConsumeMetrics(ctx, metrics)

New Implementation:

  // Multi-device processing with deduplication
  for _, v := range response.RealtimeData.Params.Value {
      for thingKey, metrics := range v {
          if !z.shouldProcessThing(thingKey, metrics.LastUpdate) {
              continue // Skip duplicates
          }
          processedMetrics, err := z.marshaler.UnmarshalMetrics(thingKey, metrics)
          z.consumer.ConsumeMetrics(ctx, processedMetrics)
          z.updateThingState(thingKey, metrics.LastUpdate)
      }
  }

Benefits

  • Performance: Thread-safe LRU cache vs manual mutex operations
  • Scalability: Handles multiple devices efficiently with bounded memory
  • Reliability: Duplicate detection prevents redundant data processing
  • Maintainability: Cleaner, more focused code with better separation of concerns

zmoog added 2 commits June 10, 2025 23:32
v0.1.1 stops writing non-JSON log messages to stderr
@zmoog zmoog self-assigned this Jun 12, 2025
@zmoog zmoog requested a review from Copilot June 12, 2025 21:01

This comment was marked as outdated.

Consume logs only if the last update timestamp is new.
@zmoog zmoog requested a review from Copilot June 12, 2025 22:37

This comment was marked as outdated.

@zmoog zmoog marked this pull request as ready for review June 13, 2025 22:21
@zmoog zmoog requested a review from Copilot June 13, 2025 22:21
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR overhauls the ZCS realtime data unmarshaler and receiver to support multi-device processing with deduplication while optimizing state management through an LRU cache and refined metric construction.

  • Enhanced data processing for multiple devices with timestamp-based deduplication
  • Refactored metric unmarshaling by introducing helper functions and updating API conventions
  • Upgraded dependencies and simplified the state management mechanism

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.

Show a summary per file
File Description
receiver/zcsazzurroreceiver/testdata/response.json Added sample JSON response reflecting multi-device support
receiver/zcsazzurroreceiver/receiver.go Introduced LRU cache and multi-device processing with deduplication logic
receiver/zcsazzurroreceiver/marshaler.go Refactored metric creation using helper functions and corrected scopeVersion
receiver/zcsazzurroreceiver/marshaler_test.go Extended tests for comprehensive verification of metric unmarshaling
receiver/zcsazzurroreceiver/factory.go Updated initialization to use the new marshaler constructor and LRU cache
Various go.mod and generated package test files Updated dependency versions and adjusted import order
Comments suppressed due to low confidence (2)

receiver/zcsazzurroreceiver/factory.go:29

  • [nitpick] Consider adding a brief comment explaining the choice of maphash as the hash function, outlining its performance benefits and suitability for the LRU cache.
func hashString(s string) uint32 {

receiver/zcsazzurroreceiver/marshaler.go:108

  • Ensure that truncating LastUpdate to derive the start of the day aligns with the intended timezone; consider converting metrics.LastUpdate to UTC if needed to avoid subtle discrepancies.
startOfTodayTimestamp := pcommon.Timestamp(metrics.LastUpdate.Truncate(24 * time.Hour).UnixNano())

@zmoog zmoog merged commit e5fce2c into main Jun 13, 2025
5 checks passed
@zmoog zmoog deleted the zmoog/feat/zcs-unmarshaler branch June 13, 2025 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants