New component: `azureencodingextension`

[Fiery-Fenix]

The purpose and use-cases of the new component

Goal

Create a configurable, maintainable and reusable Encoding Extension for all telemetry data types available via Azure Diagnostic Settings export

Description

Azure Monitor at the moment support export of Logs, Metrics and Traces using Azure Diagnostic Settings to different destination that can be consumed by OpenTelemetry Collector.

Sources of telemetry data exposed via Azure Diagnostic Settings with corresponding OTEL data type

• Platform metrics - Metrics
• Activity log - Logs
• Resource logs - Logs and Traces (Azure expose traces as logs within microsoft.insights/components Resource Type)

Supported Azure Diagnostic Settings destination with OpenTelemetry Collector receiver components

• Azure Event Hubs - can be consumed by azureeventhubreceiver (AMQP-based) or kafkareceiver (Kafka-proto based)
• Azure Storage - can be consument by azureblobreceiver, but this component does not have support for Azure-specific data unmarshaling

Current state of support of Azure-specific telemetry data types

In general exported Azure telemetry data is just a JSON with specific structure for each type of data (and log category as well), so in general it can be consumed as-is, but, in this case no normalization and aligning with OTEL SemConv will happened
There is already some packages and code that is helping with translation of Azure-specific telemetry data to correct OTEL data type with proper Resource and Record Attributes:

• pkg/translator/azure - is used for Logs and Traces, but Logs Attribute names are not translated to corresponding OTEL SemConv naming (for example httpMethod is to translated to http.request.method)
• pkg/translator/azurelogs - is used for Logs only and has Logs Attribute name translation into OTEL SemConv (this is WIP now)
• azureeventhubreceiver - this component has built-in translator for Metrics, at the same time using pkg/translator/azure and pkg/translator/azurelogs for Traces and Logs
• kafkareceiver - is using pkg/translator/azure for Logs, but missing for any support for Azure Traces or Metrics translation
• azureblobreceiver - has no Azure-specific telemetry data support at all, only plain JSONs

Problem statement

From the descriptions above I can identify following problem statements with consuming Azure-specific telemetry data:

• Not all OpenTelemetry receivers, that can actually be used, are supporting consuming Azure-specific telemetry data: azureeventhubreceiver - has all (but not all with proper semconv translation), kafkareceiver - only Logs (also without proper semconv translation), azureblobreceiver - has no support at all
• Azure-specific translation code is scattered between multiple components (pkg/translator/azure, pkg/translator/azurelogs and azureeventhubreceiver) has duplicate functionality (Logs for example), sometimes lack of tests and has different approaches of translating data into OTEL format
• Azure-specific translation code is hard to configure, for example support for multiple time formats was actually copy-pasted to multiple components to be enabled (see here and here), while kafkareceiver was missed and doesn't have this functionality at all
• Some portions of code are duplicated, for example asTimestamp function that parses timestamp has 3 copies: here, here and here
• Last, but not the least - adding support for Azure-specific telemetry data, for example into azureblobreceiver became quite hard task, because of all points described above

Proposed solution

From my perspective, the most appropriate solution to mitigate mentioned issues - is to create Azure Encoding Extension, that will:

• Consolidate all relevant code in single component, instead of 3 different. This will greatly increase maintainability of this code
• Defines single approach in translating Azure-specific telemetry data into OTEL telemetry data. This already happens with Logs in pkg/translator/azurelogs, but I believe it should be relevant for other data types as well - resulting OTEL telemetry data should conform SemConv as much as possible
• Greatly decrease amount of required breaking changes for other involved components. New component could be developed independently from other components and only after stabilizing introduce single breaking change for migration
• Make Azure-specific translation code more flexible by providing additional configuration options, that doesn't impact receivers that is actually using this code
• Allows to use Azure-specific translation code in components that has/will support encoding extensions, which covers more specific use-cases for advanced users

Proposed implementation plan

Important: Most code for proposed Azure Encoding Extension is already present in this repository, but stretched across multiple components. So, actually this is not development from scratch, it's rather smart refactoring of the existing code

1. Introduce skeleton for the new Azure Encoding Extension, discuss and align with proposed configuration options
2. Implement Traces and Logs translation as simple usage of existing Unmarshallers in pkg/translator/azure and pkg/translator/azurelogs. Unfortunately Metrics unmarshaller in azureeventhubreceiver is not exported and can't be used in this way, see next step
3. Copy (to maintain compatibility) Metrics Unmarshaller from azureeventhubreceiver to the new Extension, add missing unit tests and validate that produced result is aligned with OTEL SemConv
4. Copy (to maintain compatibility) Traces Unmarshaller from pkg/translator/azure, to the new Extension, add missing unit tests and validate that produced result is aligned with OTEL SemConv
5. Copy (to maintain compatibility) Logs Unmarshaller from pkg/translator/azurelogs to the new Extension
6. Add missing Azure Resource Logs translator code for specific categories of Azure Logs, which is not translated yet (this is current WIP in pkg/translator/azurelogs)
7. Release component, at this point it will be available as an option for at least kafkareceiver component
8. Implement optional support for Encoding Extensions to receiver/azureeventhubreceiver to provide ability to utilize new encoding extension (still no breaking changes BTW)
9. Deprecate usage of logs.encoding=azure_resource_logs in kafkareceiver and format=azure in azureeventhubreceiver, in favor of usage of azureencodingextension
10. After deprecation period - remove pkg/translator/azure, pkg/translator/azurelogs and respective unmarshalling code from azureeventhubreceiver (this is the only one breaking change that is required)

Example configuration for the component

extensions:
  azure_encoding:
    logs:
      time_formats: ["01/02/2006 15:04:05", "2006-01-02T15:04:05Z"]
      # other logs-specific settings
    metrics:
      time_formats: ["01/02/2006 15:04:05", "2006-01-02T15:04:05Z"]
      # other logs-specific settings
    traces:
      time_formats: ["01/02/2006 15:04:05", "2006-01-02T15:04:05Z"]
      # other logs-specific settings

receivers:
  kafka/azure:
    encoding: azure_encoding

Telemetry data types supported

traces, metrics and logs

Code Owner(s)

@constanca-m, @zmoog (eventually, pending membership), @Fiery-Fenix (eventually, pending membership), others welcome

Sponsor (optional)

@axw

Additional context

There are existing issues that somehow will be fulfilled by the current issue:

• New component: Azure logs encoding extension #39095 - proposal for the same Azure Encoding Extension, but focused only on Logs
• Proposal: merge pkg/translator/azure and pkg/translator/azurelogs #39969 - proposal for merging pkg/translator/azure and pkg/translator/azurelogs

I would like to contribute some time to help create this Azure Encoding Extension, as a start of discussion point I have also created initial PR #41708

Pinging current maintainers of pkg/translator/azure, pkg/translator/azurelogs and azureeventhubreceiver as your code is planned to be used for this extension: @atoulme, @cparkins, @MikeGoldsmith, @constanca-m

As for Sponsorship for this new extension:
Taking in account that code is already in place and will be simple refactored into new component - I would love to see someone of maintainers of existing code (pkg/translator/azure, pkg/translator/azurelogs and azureeventhubreceiver) as a Sponsor, if it required of course

Open for your thoughts

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[constanca-m]

Hi @Fiery-Fenix , thanks for the ping.

I still prefer #39095, as it seems you want to move the pkg code around for this encoding. I don't see a reason for this.

Why duplicate the issues?

It seems to me the only difference is that you want to receive metrics and traces and #39095 is only for logs. I would contribute to logs, but I have no interest in metrics and traces, and the scope would get too big. My idea is that #39095 is similar to AWS logs encoding and GCP log entry encoding, which I am familiar with.

[constanca-m]

Regarding this part:

Azure-specific translation code is scattered between multiple components (pkg/translator/azure, pkg/translator/azurelogs and azureeventhubreceiver) has duplicate functionality (Logs for example), sometimes lack of tests and has different approaches of translating data into OTEL format

translator/azurelogs is the only one that follows semantic conventions. I would expect new users would not use pkg/translator/azure, so in that sense, all new code should be placed in translator/azurelogs. I believe azureeventhubreceiver as the same issue: it supports semantic conventions (the translator translator/azurelogs), and then has its own implementation. azureeventhubreceiver should deprecate its own implementation.

Add missing Azure Resource Logs translator code for specific categories of Azure Logs, which is not translated yet (this is current WIP in pkg/translator/azurelogs)

There are so, so many missing categories, this is a big WIP Again, I don't see a reason why this would move out of translator, there is an issue on going there: #39186, that I have been contributing to

Release component, at this point it will be available as an option for at least kafkareceiver component

We will use azure logs encoding (#39095) internally at Elastic, so we have our own components using it

[constanca-m]

And there is one last thing: Azure logs encoding requires a source, so we know the struct of the log (see #39095 again).

The translator/azurelogs has the issue #39623, so the logs encoding can have support for logs coming from blob storage. The issue has been created for a while, but it is not forgotten, I just didn't have the capacity to work on it yet, but plan to in the upcoming months

[Fiery-Fenix]

Hi @constanca-m, many thanks for your response!

I still prefer #39095, as it seems you want to move the pkg code around for this encoding. I don't see a reason for this.

The reason for moving the code instead of making in-place changes - avoid many breaking changes, like this or this, because both pkg/translator/azure and pkg/translator/azurelogs are actively in use and it's hard to make changes to them without breaking user experience.
As you mentioned in your comment - your great work to make Azure Resource Logs aligned with SemConv is a big WIP, so it will introduce many more breaking changes, as it impossible to be done at once.
Using copy-paste approach looks "dirty", but really helpful in case of avoiding massive breaking changes, you can continue your titanic work and introduce the breaking change only once - when your work will be done.

Why duplicate the issues?

I believe that's only partially the duplicate. I would say that my issue is more generalized point of view on consuming Azure Telemetry Data, rather than only focusing on Azure Resource Logs only.
For example, as a active end-user of Azure Telemetry Data - I'm interested in receiving functionality that covers full scope (Logs+Traces+Metrics) rather than just focusing on the biggest part of this scope (Resource Logs)

It seems to me the only difference is that you want to receive metrics and traces and #39095 is only for logs. I would contribute to logs, but I have no interest in metrics and traces, and the scope would get too big. My idea is that #39095 is similar to AWS logs encoding and GCP log entry encoding, which I am familiar with.

That's why I'm proposing my help in this case, because I'm also interested in Metrics and Traces as well ;)
For example, I would like to have capability of consuming Azure Platform Metrics using kafkareceiver, which is impossible right now due absence of Azure Metrics translator (it's hidden in azureeventhubreceiver) and no support for it inside kafkareceiver. Using new proposed extension - it will be super easy task via simple configuration, without code change

translator/azurelogs is the only one that follows semantic conventions. I would expect new users would not use pkg/translator/azure, so in that sense, all new code should be placed in translator/azurelogs. I believe azureeventhubreceiver as the same issue: it supports semantic conventions (the translator translator/azurelogs), and then has its own implementation. azureeventhubreceiver should deprecate its own implementation.

As I mentioned in the description of issue - it's not focused on Logs only. Simply, Azure Metrics translator is available in azureeventhubreceiver only, and couldn't be reused by other components. This proposal is also intended to fix this issue and provided more or less standardized to full scope of Azure-related translators

There are so, so many missing categories, this is a big WIP Again, I don't see a reason why this would move out of translator, there is an issue on going there: #39186, that I have been contributing to

As I mention above, "dirty" copy paste will be beneficial for you in this work by given you more freedom in changing code without introducing another one breaking change on actively in use code

We will use azure logs encoding (#39095) internally at Elastic, so we have our own components using it

I believe that there is no issues with using either my or your approach, both are supporting Azure Logs translation, both are re-usable Encoding Extensions. Advantage of my proposal that it can be used for other Telemetry Data Types (Metrics and Logs), which might be beneficial for other users that are using other receivers that supports Encoding Extensions

And there is one last thing: Azure logs encoding requires a source, so we know the struct of the log (see #39095 again).

The translator/azurelogs has the issue #39623, so the logs encoding can have support for logs coming from blob storage. The issue has been created for a while, but it is not forgotten, I just didn't have the capacity to work on it yet, but plan to in the upcoming months

Make sense, I wasn't aware of this difference in export formats for Azure Blob Storage
But it can be easily mitigated by extension configuration, for example:

extensions:
  azure_encoding:
    logs:
      format: "resource_logs"
      # or
      format: "blob_storage"

Moreover, by developing this extension as isolated (not using pkg directly) gives us more flexibility to extend and modify extension configuration until it will be published

[constanca-m]

As you mentioned in your comment - your great work to make Azure Resource Logs aligned with SemConv is a big WIP, so it will introduce many more breaking changes, as it impossible to be done at once.

Do you expect any things to break apart from what has already been reported in the issues? I don't expect it, and you can actively contribute to translator/azurelogs as well, and become a codeowner of it. I think translator/azurelogs is the only one currently being worked on, correct?

I believe that's only partially the duplicate. I would say that my issue is more generalized point of view on consuming Azure Telemetry Data, rather than only focusing on Azure Resource Logs only.

I fully understand this. But let's say I only want to contribute to logs. Then I become a codeowner of it. How many codeowners do you expect this component to have? At what point will it become unsustainable to have one big component that has both logs and metrics?

For example, I would like to have capability of consuming Azure Platform Metrics using kafkareceiver, which is impossible right now due absence of Azure Metrics translator (it's hidden in azureeventhubreceiver) and no support for it inside kafkareceiver

I believe the team would also like to use metrics, but the work internally hasn't started for it, so I am not sure how it would go.

As I mention above, "dirty" copy paste will be beneficial for you in this work by given you more freedom in changing code without introducing another one breaking change on actively in use code

I am not sure if we want more freedom, to be honest. I think it is best to work with the comunity. But I can see where you are coming from, since I faced the same issues for both the translator and later adding breaking changes for GCP encoding

I believe that there is no issues with using either my or your approach

Correct, in my configuration, I mentioned azurefunctionsreceiver, but this component hasn't been donated upstream yet

But it can be easily mitigated by extension configuration, for example:

Yes, that's right. We also use format in AWS logs encoding.

---

Overall, I would say my main issue is to create such a big component, and being tagged as a codeowner for parts of the code I don't have the capacity to contribute to. I feel it will require a lot of people, and a few codeowners, to make all of this work. Our expectation is to never use any, and use the right structures (performance is really important for our components), and that's where all the effort comes from.

But if people prefer an encoding that includes everything, instead of just the encoding for logs, I would be happy to contribute as well

[axw]

As long as the configuration doesn't become too convoluted, I like the idea of having a single encoding extension for all the signals since it may be used as a forcing function for keeping the SemConv translations consistent.

As far as having a giant component goes, we could have internal components with their own codeowners if this becomes a problem. For example, see how the hostmetrics receiver does it with scrapers: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver/internal/scraper/cpuscraper

[Fiery-Fenix]

Do you expect any things to break apart from what has already been reported in the issues? I don't expect it, and you can actively contribute to translator/azurelogs as well, and become a codeowner of it. I think translator/azurelogs is the only one currently being worked on, correct?

Unfortunately yes, I'm expecting a new breaking change every time adding new Azure Logs Category translator to translator/azurelogs, because attributes will be taken out of Body and placed to Attributes with changed names. So end users will have broken Dashboard/Alerts after those changes.
As it extremely hard to add to existing components translator/azurelogs all available Azure Logs Categories at once (because of amount) - it will definitely introduce several breaking changes in future.
On the other hand, as I'm proposing, we can take some time to stabilize new component, add most of Azure Logs Categories in new component and introduce single breaking change for end users in the end of component development.
We have already set of Azure Logs Categories processed in our local fork, so we would contribute them 100%, the only stopper for now - breaking changes for other users.

I fully understand this. But let's say I only want to contribute to logs. Then I become a codeowner of it. How many codeowners do you expect this component to have? At what point will it become unsustainable to have one big component that has both logs and metrics?

As I mentioned in Issue description - most code is already written, present in this repository and has it's codeowners, we just need to combine that code from different locations (Metrics, Traces, Logs), make small refactoring, if required, and add missing unit tests.
So I would expect that codeowners of existing code will be naturally transitioned to a new component, alongside with the code. All of them are already mentioned in this issue.
Moreover, I would expect much better maintainability after full migration to new Encoding Extension, as amount of maintained component will decrease for all mentioned codeowners, because of natural merge of translator/azure and translator/azurelogs.

Overall, I would say my main issue is to create such a big component, and being tagged as a codeowner for parts of the code I don't have the capacity to contribute to. I feel it will require a lot of people, and a few codeowners, to make all of this work. Our expectation is to never use any, and use the right structures (performance is really important for our components), and that's where all the effort comes from.

After going through existing code - I'm expecting that up to 80% of code in new component will be occupied by Azure Resource Logs processing, as current code for Metrics and Traces is relatively small and doesn't require big changes.
So I would expect the size of the new Encoding Extension will be only slightly bigger than existing translator/azurelogs, at least at the starting point.
Also I'm agree with @axw point that codeowners could be split between internal sub-components, as I'm expecting to have 1 unmarshaller sub-component per each type of telemetry data, like it's for example done in awslogsencodingextension/internal/unmarshaler

[constanca-m]

I'm ok with it then @Fiery-Fenix . I think the option of having different codeowners per signal fixes the issue. I look forward to contribute!

So I would expect that codeowners of existing code will be naturally transitioned to a new component, alongside with the code. All of them are already mentioned in this issue.

I don't expect this, if you look at the past contributions, there hasn't been much. But the more people we have contributing, the better

Unfortunately yes, I'm expecting a new breaking change every time adding new Azure Logs Category translator to translator/azurelogs

This is mentioned in the issue #39186, so all this is known already