[Azure App Insights] add support for Client Secret (#48880)

* [Azure App Insights] add support for ouath2

* check that at least one auth is configured

* add changelog fragemnt and update docs

* small fixes

* fix yml files

* add auth_type to config

* remove sensitive data from debug log

* remove redundant log msg

* add required roles

* remove logger and ctx nil check

* add deprecation message and remove active_directory_endpoint

* fix doc

* add applies_to to docs

Author: jakubgalecki0

changelog/fragments/1771338300-app_insights_oauth2.yaml

@@ -0,0 +1,45 @@
+# REQUIRED
+# Kind can be one of:
+# - breaking-change: a change to previously-documented behavior
+# - deprecation: functionality that is being removed in a later release
+# - bug-fix: fixes a problem in a previous version
+# - enhancement: extends functionality but does not break or fix existing behavior
+# - feature: new functionality
+# - known-issue: problems that we are aware of in a given version
+# - security: impacts on the security of a product or a user’s deployment.
+# - upgrade: important information for someone upgrading from a prior version
+# - other: does not fit into any of the other categories
+kind: feature
+
+# REQUIRED for all kinds
+# Change summary; a 80ish characters long description of the change.
+summary: Add client secret authentication support to Azure App Insights module
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# Long description; in case the summary is not enough to describe the change
+# this field accommodate a description without length limits.
+# description:
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# impact:
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# action:
+
+# REQUIRED for all kinds
+# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
+component: metricbeat
+
+# AUTOMATED
+# OPTIONAL to manually add other PR URLs
+# PR URL: A link the PR that added the changeset.
+# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
+# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
+# Please provide it if you are adding a fragment for a different PR.
+# pr: https://github.com/owner/repo/1234
+
+# AUTOMATED
+# OPTIONAL to manually add other issue URLs
+# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
+# If not present is automatically filled by the tooling with the issue linked to the PR number.
+# issue: https://github.com/owner/repo/1234

docs/reference/metricbeat/metricbeat-metricset-azure-app_insights.md

@@ -23,10 +23,47 @@ This metricset allows users to retrieve application insights metrics from specif
 ### Config options to identify resources [_config_options_to_identify_resources]
 
 `application_id`
-:   (*[]string*) ID of the application. This is Application ID from the API Access settings blade in the Azure portal.
+:   (*string*) ID of the application. This is Application ID from the API Access settings blade in the Azure portal.
+
+
+### Authentication [_authentication]
+
+Two authentication methods are supported: **Client secret (Microsoft Entra ID)** and **API key**. The method is selected using the `auth_type` option.
+
+`auth_type`
+:   (*string*) The authentication method to use. Valid values: `api_key` (default), `client_secret`.
+
+#### Client secret authentication
+
+{applies_to}`stack: ga 8.19.13` {applies_to}`stack: ga 9.2.7` {applies_to}`stack: ga 9.3.2`
+
+Set `auth_type: "client_secret"` and provide the following options:
+
+`tenant_id`
+:   (*string*) The tenant ID of the Microsoft Entra ID (Azure Active Directory) instance. More on service principal authentication can be found here [https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal).
+
+`client_id`
+:   (*string*) The client/application ID of the service principal registered in Microsoft Entra ID.
+
+`client_secret`
+:   (*string*) The client secret associated with the service principal.
+
+All three of `tenant_id`, `client_id`, and `client_secret` are required when `auth_type` is `client_secret`.
+
+**Required permissions:** The service principal must be assigned a role that grants read access to Application Insights data. The minimum built-in role is **Monitoring Reader**, assigned at the Application Insights resource scope. Other roles that include the required permissions are **Monitoring Contributor**, **Contributor**, and **Owner**. For more details, see [Azure built-in roles for Monitor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/monitor).
+
+#### API key authentication
+
+::::{warning}
+Microsoft is retiring API key authentication for Application Insights on **March 31, 2026**. After this date, API key authentication will no longer work. It is recommended to migrate to [client secret authentication](#_authentication) before this deadline. For more details, see [Transition to Microsoft Entra ID authentication](https://azure.microsoft.com/en-us/updates?id=transition-to-azure-ad-to-query-data-from-azure-monitor-application-insights-by-31-march-2026).
+::::
+
+Set `auth_type: "api_key"` (or omit `auth_type`, as it defaults to `api_key`) and provide:
 
 `api_key`
-:   (*[]string*) The API key which will be generated, more on the steps here [https://dev.applicationinsights.io/documentation/Authorization/API-key-and-App-ID](https://dev.applicationinsights.io/documentation/Authorization/API-key-and-App-ID).
+:   (*string*) The API key which will be generated, more on the steps here [https://dev.applicationinsights.io/documentation/Authorization/API-key-and-App-ID](https://dev.applicationinsights.io/documentation/Authorization/API-key-and-App-ID).
+
+**Required permissions:** The API key must be created with the **Read telemetry** permission enabled in the Azure portal (under the API Access blade of the Application Insights resource).
 
 
 ### App insights metric configurations [_app_insights_metric_configurations]

docs/reference/metricbeat/metricbeat-module-azure.md

@@ -265,6 +265,11 @@ metricbeat.modules:
   enabled: true
   period: 300s
   application_id: ''
+  # auth_type: "api_key" (default) or "client_secret"
+  #auth_type: "api_key"
+  #client_id: '${AZURE_CLIENT_ID:""}'
+  #client_secret: '${AZURE_CLIENT_SECRET:""}'
+  #tenant_id: '${AZURE_TENANT_ID:""}'
   api_key: ''
   metrics:
     - id: ["requests/count", "requests/duration"]

x-pack/metricbeat/metricbeat.reference.yml

@@ -407,6 +407,11 @@ metricbeat.modules:
   enabled: true
   period: 300s
   application_id: ''
+  # auth_type: "api_key" (default) or "client_secret"
+  #auth_type: "api_key"
+  #client_id: '${AZURE_CLIENT_ID:""}'
+  #client_secret: '${AZURE_CLIENT_SECRET:""}'
+  #tenant_id: '${AZURE_TENANT_ID:""}'
   api_key: ''
   metrics:
     - id: ["requests/count", "requests/duration"]

x-pack/metricbeat/module/azure/_meta/config.reference.yml

@@ -99,6 +99,11 @@
   enabled: true
   period: 300s
   application_id: ''
+  # auth_type: "api_key" (default) or "client_secret"
+  #auth_type: "api_key"
+  #client_id: '${AZURE_CLIENT_ID:""}'
+  #client_secret: '${AZURE_CLIENT_SECRET:""}'
+  #tenant_id: '${AZURE_TENANT_ID:""}'
   api_key: ''
   metrics:
     - id: ["requests/count", "requests/duration"]

x-pack/metricbeat/module/azure/_meta/config.yml

@@ -108,7 +108,12 @@
 #  enabled: true
 #  period: 300s
 #  application_id: ''
-#  api_key: ''
+#  # auth_type: "api_key" (default) or "client_secret"
+#  auth_type: "client_secret"
+#  client_id: '${AZURE_CLIENT_ID:""}'
+#  client_secret: '${AZURE_CLIENT_SECRET:""}'
+#  tenant_id: '${AZURE_TENANT_ID:""}'
+#  #api_key: ''
 #  metrics:
 #   - id: ["requests/count", "requests/duration"]
 

x-pack/metricbeat/module/azure/app_insights/_meta/docs.md

@@ -11,10 +11,47 @@ This metricset allows users to retrieve application insights metrics from specif
 ### Config options to identify resources [_config_options_to_identify_resources]
 
 `application_id`
-:   (*[]string*) ID of the application. This is Application ID from the API Access settings blade in the Azure portal.
+:   (*string*) ID of the application. This is Application ID from the API Access settings blade in the Azure portal.
+
+
+### Authentication [_authentication]
+
+Two authentication methods are supported: **Client secret (Microsoft Entra ID)** and **API key**. The method is selected using the `auth_type` option.
+
+`auth_type`
+:   (*string*) The authentication method to use. Valid values: `api_key` (default), `client_secret`.
+
+#### Client secret authentication
+
+{applies_to}`stack: ga 8.19.13` {applies_to}`stack: ga 9.2.7` {applies_to}`stack: ga 9.3.2`
+
+Set `auth_type: "client_secret"` and provide the following options:
+
+`tenant_id`
+:   (*string*) The tenant ID of the Microsoft Entra ID (Azure Active Directory) instance. More on service principal authentication can be found here [https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal).
+
+`client_id`
+:   (*string*) The client/application ID of the service principal registered in Microsoft Entra ID.
+
+`client_secret`
+:   (*string*) The client secret associated with the service principal.
+
+All three of `tenant_id`, `client_id`, and `client_secret` are required when `auth_type` is `client_secret`.
+
+**Required permissions:** The service principal must be assigned a role that grants read access to Application Insights data. The minimum built-in role is **Monitoring Reader**, assigned at the Application Insights resource scope. Other roles that include the required permissions are **Monitoring Contributor**, **Contributor**, and **Owner**. For more details, see [Azure built-in roles for Monitor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/monitor).
+
+#### API key authentication
+
+::::{warning}
+Microsoft is retiring API key authentication for Application Insights on **March 31, 2026**. After this date, API key authentication will no longer work. It is recommended to migrate to [client secret authentication](#_authentication) before this deadline. For more details, see [Transition to Microsoft Entra ID authentication](https://azure.microsoft.com/en-us/updates?id=transition-to-azure-ad-to-query-data-from-azure-monitor-application-insights-by-31-march-2026).
+::::
+
+Set `auth_type: "api_key"` (or omit `auth_type`, as it defaults to `api_key`) and provide:
 
 `api_key`
-:   (*[]string*) The API key which will be generated, more on the steps here [https://dev.applicationinsights.io/documentation/Authorization/API-key-and-App-ID](https://dev.applicationinsights.io/documentation/Authorization/API-key-and-App-ID).
+:   (*string*) The API key which will be generated, more on the steps here [https://dev.applicationinsights.io/documentation/Authorization/API-key-and-App-ID](https://dev.applicationinsights.io/documentation/Authorization/API-key-and-App-ID).
+
+**Required permissions:** The API key must be created with the **Read telemetry** permission enabled in the Azure portal (under the API Access blade of the Application Insights resource).
 
 
 ### App insights metric configurations [_app_insights_metric_configurations]

x-pack/metricbeat/module/azure/app_insights/app_insights.go

@@ -16,15 +16,69 @@ import (
 	"github.com/elastic/elastic-agent-libs/logp"
 )
 
-const metricsetName = "app_insights"
+const (
+	metricsetName = "app_insights"
+
+	// AuthTypeAPIKey uses API key authentication (default for backwards compatibility).
+	AuthTypeAPIKey string = "api_key"
+	// AuthTypeClientSecret uses client secret credentials (Microsoft Entra ID).
+	AuthTypeClientSecret string = "client_secret"
+)
 
 // Config options
 type Config struct {
-	ApplicationId string        `config:"application_id"    validate:"required"`
-	ApiKey        string        `config:"api_key" validate:"required"`
+	ApplicationId string        `config:"application_id" validate:"required"`
 	Period        time.Duration `config:"period" validate:"nonzero,required"`
 	Metrics       []Metric      `config:"metrics" validate:"required"`
 	Namespace     string        `config:"namespace"`
+
+	// AuthType specifies the authentication method.
+	// Valid values: api_key (default), client_secret.
+	AuthType string `config:"auth_type"`
+
+	// API key authentication
+	ApiKey string `config:"api_key"`
+
+	// Client secret authentication (Microsoft Entra ID)
+	TenantId     string `config:"tenant_id"`
+	ClientId     string `config:"client_id"`
+	ClientSecret string `config:"client_secret"`
+}
+
+// Validate checks that the authentication configuration is complete.
+func (c *Config) Validate() error {
+	if c.AuthType == "" {
+		c.AuthType = AuthTypeAPIKey
+	}
+
+	switch c.AuthType {
+	case AuthTypeAPIKey:
+		return c.validateAPIKeyAuth()
+	case AuthTypeClientSecret:
+		return c.validateClientSecretAuth()
+	default:
+		return fmt.Errorf("unknown auth_type: %s (valid values: %s, %s)", c.AuthType, AuthTypeAPIKey, AuthTypeClientSecret)
+	}
+}
+
+func (c *Config) validateAPIKeyAuth() error {
+	if c.ApiKey == "" {
+		return fmt.Errorf("api_key is required when auth_type is %s", AuthTypeAPIKey)
+	}
+	return nil
+}
+
+func (c *Config) validateClientSecretAuth() error {
+	if c.TenantId == "" {
+		return fmt.Errorf("tenant_id is required when auth_type is %s", AuthTypeClientSecret)
+	}
+	if c.ClientId == "" {
+		return fmt.Errorf("client_id is required when auth_type is %s", AuthTypeClientSecret)
+	}
+	if c.ClientSecret == "" {
+		return fmt.Errorf("client_secret is required when auth_type is %s", AuthTypeClientSecret)
+	}
+	return nil
 }
 
 // Metric struct used for configuration options

x-pack/metricbeat/module/azure/app_insights/client_test.go

@@ -13,14 +13,15 @@ import (
 	"github.com/Azure/azure-sdk-for-go/services/preview/appinsights/v1/insights"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/mock"
+	"github.com/stretchr/testify/require"
 
 	"github.com/elastic/elastic-agent-libs/logp/logptest"
 )
 
 var (
 	config = Config{
 		ApplicationId: "",
-		ApiKey:        "",
+		ApiKey:        "test-api-key",
 		Metrics: []Metric{
 			{
 				ID: []string{"requests/count"},
@@ -51,7 +52,109 @@ func TestClient(t *testing.T) {
 		client.Service = m
 		results, err := client.GetMetricValues()
 		assert.NoError(t, err)
-		assert.Equal(t, len(*results.Value), 2)
+		assert.Len(t, *results.Value, 2)
 		m.AssertExpectations(t)
 	})
 }
+
+func TestConfigValidate(t *testing.T) {
+	tests := []struct {
+		name    string
+		config  Config
+		wantErr string
+	}{
+		{
+			name: "valid config with API key (explicit auth_type)",
+			config: Config{
+				ApplicationId: "app-id",
+				AuthType:      "api_key",
+				ApiKey:        "test-api-key",
+			},
+			wantErr: "",
+		},
+		{
+			name: "valid config with API key (default auth_type)",
+			config: Config{
+				ApplicationId: "app-id",
+				ApiKey:        "test-api-key",
+			},
+			wantErr: "",
+		},
+		{
+			name: "valid config with client secret",
+			config: Config{
+				ApplicationId: "app-id",
+				AuthType:      "client_secret",
+				TenantId:      "tenant-id",
+				ClientId:      "client-id",
+				ClientSecret:  "client-secret",
+			},
+			wantErr: "",
+		},
+		{
+			name: "invalid config - api_key auth_type without api_key",
+			config: Config{
+				ApplicationId: "app-id",
+				AuthType:      "api_key",
+			},
+			wantErr: "api_key is required when auth_type is api_key",
+		},
+		{
+			name: "invalid config - default auth_type without api_key",
+			config: Config{
+				ApplicationId: "app-id",
+			},
+			wantErr: "api_key is required when auth_type is api_key",
+		},
+		{
+			name: "invalid config - client_secret missing tenant_id",
+			config: Config{
+				ApplicationId: "app-id",
+				AuthType:      "client_secret",
+				ClientId:      "client-id",
+				ClientSecret:  "client-secret",
+			},
+			wantErr: "tenant_id is required when auth_type is client_secret",
+		},
+		{
+			name: "invalid config - client_secret missing client_id",
+			config: Config{
+				ApplicationId: "app-id",
+				AuthType:      "client_secret",
+				TenantId:      "tenant-id",
+				ClientSecret:  "client-secret",
+			},
+			wantErr: "client_id is required when auth_type is client_secret",
+		},
+		{
+			name: "invalid config - client_secret missing client_secret",
+			config: Config{
+				ApplicationId: "app-id",
+				AuthType:      "client_secret",
+				TenantId:      "tenant-id",
+				ClientId:      "client-id",
+			},
+			wantErr: "client_secret is required when auth_type is client_secret",
+		},
+		{
+			name: "invalid config - unknown auth_type",
+			config: Config{
+				ApplicationId: "app-id",
+				AuthType:      "invalid",
+			},
+			wantErr: "unknown auth_type: invalid",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			err := tt.config.Validate()
+			if tt.wantErr == "" {
+				require.NoError(t, err)
+			} else {
+				require.Error(t, err)
+				assert.Contains(t, err.Error(), tt.wantErr)
+			}
+		})
+	}
+}