[Azure] Add client secret (Oauth2) support for eventhub filebeat input

changelog/fragments/1761678237-feat-add-client-secret-auth-for-azure-eventhub.yaml

@@ -0,0 +1,45 @@
+# REQUIRED
+# Kind can be one of:
+# - breaking-change: a change to previously-documented behavior
+# - deprecation: functionality that is being removed in a later release
+# - bug-fix: fixes a problem in a previous version
+# - enhancement: extends functionality but does not break or fix existing behavior
+# - feature: new functionality
+# - known-issue: problems that we are aware of in a given version
+# - security: impacts on the security of a product or a user’s deployment.
+# - upgrade: important information for someone upgrading from a prior version
+# - other: does not fit into any of the other categories
+kind: feature
+
+# REQUIRED for all kinds
+# Change summary; a 80ish characters long description of the change.
+summary: Add client secret authentication method for Azure Event Hub and storage in Filebeat.
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# Long description; in case the summary is not enough to describe the change
+# this field accommodate a description without length limits.
+# description:
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# impact:
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# action:
+
+# REQUIRED for all kinds
+# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
+component: "filebeat"
+
+# AUTOMATED
+# OPTIONAL to manually add other PR URLs
+# PR URL: A link the PR that added the changeset.
+# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
+# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
+# Please provide it if you are adding a fragment for a different PR.
+# pr: https://github.com/owner/repo/1234
+
+# AUTOMATED
+# OPTIONAL to manually add other issue URLs
+# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
+# If not present is automatically filled by the tooling with the issue linked to the PR number.
+# issue: https://github.com/owner/repo/1234

docs/reference/filebeat/filebeat-input-azure-eventhub.md

@@ -8,66 +8,196 @@ applies_to:
 
 # Azure eventhub input [filebeat-input-azure-eventhub]
 
-
 Users can make use of the `azure-eventhub` input in order to read messages from an azure eventhub. The azure-eventhub input implementation is based on the the event processor host (EPH is intended to be run across multiple processes and machines while load balancing message consumers more on this here [https://github.com/Azure/azure-event-hubs-go#event-processor-host](https://github.com/Azure/azure-event-hubs-go#event-processor-host), [https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-event-processor-host](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-event-processor-host)). State such as leases on partitions and checkpoints in the event stream are shared between receivers using an Azure Storage container. For this reason, as a prerequisite to using this input, users will have to create or use an existing storage account.
 
 Users can enable internal logs tracing for this input by setting the environment variable `BEATS_AZURE_EVENTHUB_INPUT_TRACING_ENABLED: true`. When enabled, this input will log additional information to the logs. Additional information includes partition ownership, blob lease information, and other internal state.
 
-Example configuration:
+## Example configurations
+
+### Connection string authentication (processor v1)
+
+Example configuration using connection string authentication with processor v1:
 
 ```yaml
 filebeat.inputs:
 - type: azure-eventhub
   eventhub: "insights-operational-logs"
-  consumer_group: "test"
-  connection_string: "Endpoint=sb://....."
-  storage_account: "azureeph"
-  storage_account_key: "....."
-  storage_account_container: ""
-  resource_manager_endpoint: ""
+  consumer_group: "$Default"
+  connection_string: "Endpoint=sb://your-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=your-shared-access-key"
+  storage_account: "your-storage-account"
+  storage_account_key: "your-storage-account-key"
+  storage_account_container: "your-storage-container"
+  processor_version: "v1"
 ```
 
-## Configuration options [_configuration_options]
+### Connection string authentication (processor v2)
 
-The `azure-eventhub` input supports the following configuration:
+Example configuration using connection string authentication with processor v2:
 
+```yaml
+filebeat.inputs:
+- type: azure-eventhub
+  eventhub: "insights-operational-logs"
+  consumer_group: "$Default"
+  auth_type: "connection_string"
+  connection_string: "Endpoint=sb://your-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=your-shared-access-key"
+  storage_account: "your-storage-account"
+  storage_account_connection_string: "DefaultEndpointsProtocol=https;AccountName=your-storage-account;AccountKey=your-storage-account-key;EndpointSuffix=core.windows.net"
+  storage_account_container: "your-storage-container"
+```
 
-## `eventhub` [_eventhub]
+{applies_to}`stack: ga 9.3.0` ### Client secret authentication (processor v2)
 
-The name of the eventhub users would like to read from, field required.
+Example configuration using Azure Active Directory service principal authentication with processor v2:
+
+```yaml
+filebeat.inputs:
+- type: azure-eventhub
+  eventhub: "insights-operational-logs"
+  consumer_group: "$Default"
+  auth_type: "client_secret"
+  eventhub_namespace: "your-namespace.servicebus.windows.net"
+  tenant_id: "your-tenant-id"
+  client_id: "your-client-id"
+  client_secret: "your-client-secret"
+  storage_account: "your-storage-account"
+  storage_account_container: "your-storage-container"
+```
+
+**Note:** When using `client_secret` authentication, the service principal must have the appropriate Azure RBAC permissions. See [Required permissions](#_required_permissions) for details.
 
+## Authentication [_authentication]
+
+The azure-eventhub input supports multiple authentication methods. The [`auth_type` configuration option](#_auth_type) controls the authentication method used for both Event Hub and Storage Account.
+
+### Authentication types
+
+The following authentication types are supported:
+
+- **`connection_string`** (default if `auth_type` is not specified): Uses Azure Event Hubs and Storage Account connection strings.
+- {applies_to}`stack: ga 9.3.0` **`client_secret`**: Uses Azure Active Directory service principal with client secret credentials.
+
+### Required permissions [_required_permissions]
+
+When using `client_secret` authentication, the service principal needs the following Azure RBAC permissions:
+
+**For Azure Event Hubs:**
+- `Azure Event Hubs Data Receiver` role on the Event Hubs namespace or Event Hub
+- Alternatively, a custom role with the following permissions:
+  - `Microsoft.EventHub/namespaces/eventhubs/read`
+  - `Microsoft.EventHub/namespaces/eventhubs/consumergroups/read`
+
+**For Azure Storage Account:**
+- `Storage Blob Data Contributor` role on the Storage Account or container
+- Alternatively, a custom role with the following permissions:
+  - `Microsoft.Storage/storageAccounts/blobServices/containers/read`
+  - `Microsoft.Storage/storageAccounts/blobServices/containers/write`
+  - `Microsoft.Storage/storageAccounts/blobServices/containers/delete`
+  - `Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action`
+
+For detailed instructions on how to set up an Azure AD service principal and configure permissions, refer to the official Microsoft documentation:
+
+- [Create an Azure service principal with Azure CLI](https://learn.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli)
+- [Create an Azure AD app registration using the Azure portal](https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal)
+- [Assign Azure roles using Azure CLI](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-cli)
+- [Azure Event Hubs authentication and authorization](https://learn.microsoft.com/en-us/azure/event-hubs/authorize-access-azure-active-directory)
+- [Authorize access to blobs using Azure Active Directory](https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory)
+
+## Configuration options [_configuration_options]
+
+The `azure-eventhub` input supports the following configuration options:
+
+### `eventhub` [_eventhub]
+
+The name of the eventhub users would like to read from, field required.
 
-## `consumer_group` [_consumer_group]
+### `consumer_group` [_consumer_group]
 
 Optional, we recommend using a dedicated consumer group for the azure input. Reusing consumer groups among non-related consumers can cause unexpected behavior and possibly lost events.
 
+### `auth_type` [_auth_type]
 
-## `connection_string` [_connection_string]
+```{applies_to}
+stack: ga 9.3.0
+```
 
-The connection string required to communicate with Event Hubs, steps here [https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string).
+Specifies the authentication method to use for both Event Hub and Storage Account. If not specified, defaults to `connection_string` for backwards compatibility.
 
-A Blob Storage account is required in order to store/retrieve/update the offset or state of the eventhub messages. This means that after stopping filebeat it can start back up at the spot that it stopped processing messages.
+Valid values include:
+- `connection_string` (default): Uses connection string authentication. You _must_ provide a [`connection_string`](#_connection_string).
+- `client_secret`: Uses Azure Active Directory service principal with client secret credentials.
 
+### `connection_string` [_connection_string]
 
-## `storage_account` [_storage_account]
+The connection string required to communicate with Event Hubs when using `connection_string` authentication. For more information, refer to [Get an Azure Event Hubs connection string](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string).
 
-The name of the storage account. Required.
+This option is required if:
 
+* `auth_type` is set to `connection_string`
+* `auth_type` is not specified (in which case it defaults to `connection_string` for backwards compatibility)
 
-## `storage_account_key` [_storage_account_key]
+A Blob Storage account is required to store, retrieve, or update the offset or state of the Event Hub messages. This means that after stopping Filebeat it can resume from where it stopped processing messages.
 
-The storage account key, this key will be used to authorize access to data in your storage account, option is required.
+### `eventhub_namespace` [_eventhub_namespace]
+
+```{applies_to}
+stack: ga 9.3.0
+```
 
+The fully qualified namespace for the Event Hub. Required when using `client_secret` authentication (`auth_type` is set to `client_secret`). Format: `your-eventhub-namespace.servicebus.windows.net`
 
-## `storage_account_container` [_storage_account_container]
+### `tenant_id` [_tenant_id]
 
-Optional, the name of the storage account container you would like to store the offset information in.
+```{applies_to}
+stack: ga 9.3.0
+```
 
+The Azure Active Directory tenant ID. Required when using `client_secret` authentication for Event Hub or Storage Account.
 
-## `resource_manager_endpoint` [_resource_manager_endpoint]
+### `client_id` [_client_id]
 
-Optional, by default we are using the azure public environment, to override, users can provide a specific resource manager endpoint in order to use a different azure environment. Ex: [https://management.chinacloudapi.cn/](https://management.chinacloudapi.cn/) for azure ChinaCloud [https://management.microsoftazure.de/](https://management.microsoftazure.de/) for azure GermanCloud [https://management.azure.com/](https://management.azure.com/) for azure PublicCloud [https://management.usgovcloudapi.net/](https://management.usgovcloudapi.net/) for azure USGovernmentCloud Users can also use this in case of a Hybrid Cloud model, where one may define their own endpoints.
+```{applies_to}
+stack: ga 9.3.0
+```
+
+The Azure Active Directory application (client) ID. Required when using `client_secret` authentication for Event Hub or Storage Account.
+
+### `client_secret` [_client_secret]
+
+```{applies_to}
+stack: ga 9.3.0
+```
+
+The Azure Active Directory application client secret. Required when using `client_secret` authentication for Event Hub or Storage Account.
+
+### `authority_host` [_authority_host]
+
+```{applies_to}
+stack: ga 9.3.0
+```
+
+The Azure Active Directory authority host. Optional when using `client_secret` authentication. Defaults to Azure Public Cloud (`https://login.microsoftonline.com`).
+
+Supported values:
+- `https://login.microsoftonline.com` (Azure Public Cloud - default)
+- `https://login.microsoftonline.us` (Azure Government)
+- `https://login.chinacloudapi.cn` (Azure China)
+
+### `storage_account` [_storage_account]
+
+The name of the storage account. Required.
+
+### `storage_account_key` [_storage_account_key]
+
+The storage account key, this key will be used to authorize access to data in your storage account, option is required.
+
+### `storage_account_container` [_storage_account_container]
+
+Optional, the name of the storage account container you would like to store the offset information in.
 
+### `resource_manager_endpoint` [_resource_manager_endpoint]
+
+Optional, by default we are using the azure public environment, to override, users can provide a specific resource manager endpoint in order to use a different azure environment. Ex: [https://management.chinacloudapi.cn/](https://management.chinacloudapi.cn/) for azure ChinaCloud [https://management.microsoftazure.de/](https://management.microsoftazure.de/) for azure GermanCloud [https://management.azure.com/](https://management.azure.com/) for azure PublicCloud [https://management.usgovcloudapi.net/](https://management.usgovcloudapi.net/) for azure USGovernmentCloud Users can also use this in case of a Hybrid Cloud model, where one may define their own endpoints.
 
 ## Metrics [_metrics_3]
 
@@ -83,5 +213,3 @@ This input exposes metrics under the [HTTP monitoring endpoint](/reference/fileb
 | `sent_events_total` | Number of events that were sent successfully. |
 | `processing_time` | Histogram of the elapsed processing times in nanoseconds. |
 | `decode_errors_total` | Number of errors that occurred while decoding a message. |
-
-

x-pack/filebeat/input/azureeventhub/README.md

@@ -282,7 +282,7 @@ Using the following configuration:
       storage_account_key: "<redacted>"
       storage_account_connection_string: "<redacted>"
       processor_version: "v1"
-      migrate_checkpoint: yes
+      migrate_checkpoint: true
       start_position: "earliest"
 ```
 
@@ -377,7 +377,7 @@ Stop Filebeat and update the config with the following changes:
       storage_account_key: "<redacted>"
       storage_account_connection_string: "<redacted>" # NOTE: make sure this is set
       processor_version: "v2" # CHANGE: v1 > v2
-      migrate_checkpoint: yes
+      migrate_checkpoint: true
       start_position: "earliest"
 ```
 
@@ -438,7 +438,7 @@ Using the following configuration:
       storage_account_key: "<redacted>"
       storage_account_connection_string: "<redacted>"
       processor_version: "v2"
-      migrate_checkpoint: yes
+      migrate_checkpoint: true
       start_position: "earliest"
 ```
 
@@ -564,7 +564,7 @@ Using the following configuration for all inputs:
       storage_account_key: "<redacted>"
       storage_account_connection_string: "<redacted>"
       processor_version: "v2"
-      migrate_checkpoint: yes
+      migrate_checkpoint: true
       start_position: "earliest"
 ```
 
@@ -783,7 +783,7 @@ Using the following configuration:
       storage_account_key: "<redacted>"
       storage_account_connection_string: "<redacted>"
       processor_version: "v2"
-      migrate_checkpoint: yes
+      migrate_checkpoint: true
       start_position: "earliest"
 ```