feat(sqlserverreceiver): Port v0.150.0 with passfile auth, gap metrics, and top applications

receiver/sqlserverreceiver/README.md

@@ -1,27 +1,73 @@
+<!-- status autogenerated section -->
 # Microsoft SQL Server Receiver
 
-<!-- status autogenerated section -->
+The `sqlserver` receiver grabs metrics/logs about a Microsoft SQL Server instance. The receiver works by either using the
+Windows Performance Counters, or by directly connecting to the instance and querying it. Windows Performance Counters
+are only available when running on Windows.
+
+
 | Status        |           |
 | ------------- |-----------|
 | Stability     | [development]: logs   |
 |               | [beta]: metrics   |
 | Distributions | [contrib] |
 | Issues        | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Areceiver%2Fsqlserver%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fsqlserver) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Areceiver%2Fsqlserver%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fsqlserver) |
-| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner)    | [@StefanKurek](https://www.github.com/StefanKurek), [@sincejune](https://www.github.com/sincejune), [@crobert-1](https://www.github.com/crobert-1) \| Seeking more code owners! |
+| Code coverage | [![codecov](https://codecov.io/github/open-telemetry/opentelemetry-collector-contrib/graph/main/badge.svg?component=receiver_sqlserver)](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=receiver_sqlserver&displayType=list) |
+| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner)    | [@sincejune](https://www.github.com/sincejune), [@crobert-1](https://www.github.com/crobert-1) \| Seeking more code owners! |
+| Emeritus      | [@StefanKurek](https://www.github.com/StefanKurek) |
 
 [development]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#development
 [beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
 [contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
 <!-- end autogenerated section -->
 
-The `sqlserver` receiver grabs metrics/logs about a Microsoft SQL Server instance. The receiver works by either using the
-Windows Performance Counters, or by directly connecting to the instance and querying it. Windows Performance Counters
-are only available when running on Windows.
+## Required Permissions
+
+### Windows Performance Counters
+
+Make sure to run the collector as administrator in order to collect all performance counters for metrics.
+
+### Direct Connection
+
+When configured to directly connect to the SQL Server instance, the user must have the following permissions:
+
+1. At least one of the following permissions:
+- `CREATE DATABASE`
+- `ALTER ANY DATABASE`
+- `VIEW ANY DATABASE`
 
-Make sure to run the collector as administrator in order to collect all performance counters for metrics. 
+2. Permission to view server state:
+   - SQL Server pre-2022: `VIEW SERVER STATE`
+   - SQL Server 2022 and later: `VIEW SERVER PERFORMANCE STATE`
 
 ## Configuration
 
+The following is a generic configuration that can be used for the default logs and metrics scraped
+by the SQL Server receiver. A basic explanation on some of the fields has also been provided. For more
+information, please reference the following section.
+
+```yaml
+sqlserver:
+  collection_interval: 10s                     # interval for overall collection
+  instance_name: CustomInstance
+  username: myusername
+  password: mypassword
+  server: sqlserver.address
+  port: 1433
+  events:
+    db.server.query_sample:
+      enabled: true
+    db.server.top_query:
+      enabled: true
+  top_query_collection:                        # this collection exports the most expensive queries as logs
+    lookback_time: 60s                         # which time window should we look for the top queries
+    max_query_sample_count: 1000               # maximum number query we store in cache for top queries.
+    top_query_count: 250                       # The maximum number of active queries to report in a single run.
+    collection_interval: 60s                   # collection interval for top query collection specifically
+  query_sample_collection:                     # this collection exports the currently (relate to the query time) executing queries as logs
+    max_rows_per_query: 100                    # the maximum number of samples to return for one single query.
+```
+
 The following settings are optional:
 - `collection_interval` (default = `10s`): The interval at which metrics should be emitted by this receiver.
 - `instance_name` (optional): The instance name identifies the specific SQL Server instance being monitored.
@@ -42,16 +88,20 @@ Windows-specific options:
   If specified, `instance_name` is also required to be defined. This option is ignored in non-Windows environments.
 
 Top-Query collection specific options (only useful when top-query collection are enabled):
-- `lookback_time` (optional, example = `60`, default = `2 * collection_interval`): The time window (in second) in which to query for top queries.
+- `lookback_time` (optional, example = `60s`, default = `2 * collection_interval`): The time window (in second) in which to query for top queries.
   - Queries that were finished execution outside the lookback window are not included in the collection. Increasing the lookback window (in seconds) will be useful for capturing long-running queries.
 - `max_query_sample_count` (optional, example = `5000`, default = `1000`): The maximum number of records to fetch in a single run.
-- `top_query_count`: (optional, example = `100`, default = `200`): The maximum number of active queries to report (to the next consumer) in a single run.
-- `enabled`: (optional, default = `false`): Enable collection of top queries.
-  - e.g. `sqlserver` receiver will fetch 1000 (value: `max_query_sample_count`) queries from database and report the top 200 (value: `top_query_count`) which used the most CPU time.
+- `top_query_count`: (optional, example = `100`, default = `250`): The maximum number of active queries to report (to the next consumer) in a single run.
+- `collection_interval`: (optional, default = `60s`): The interval at which top queries should be emitted by this receiver.
+  - This value can only guarantee that the top queries are collected at most once in this interval.
+    - For instance, you have global `collection_interval` as `10s` and `top_query_collection.collection_interval` as `60s`.
+      - In this case, the default receiver scraper will still try to run in every 10 seconds.
+      - However, the top queries collection will only run after 60 seconds have passed since the last collection.
+    - For instance, you have global `collection_interval` as `10s` and `top_query_collection.collection_interval` as `5s`.
+      - In this case, `top_query_collection.collection_internal` will make no effects to the collection
 
 Query sample collection related options (only useful when query sample is enabled)
 - `max_rows_per_query`: (optional, default = `100`) use this to limit rows returned by the sampling query.
-- `enabled`: (optional, default = `false`): Enable collection of sample queries.
 Example:
 
 ```yaml
@@ -94,15 +144,23 @@ Top query collection enabled:
         server: 0.0.0.0
         port: 1433
         top_query_collection:
-          enabled: true
-          lookback_time: 60
+          lookback_time: 60s
           max_query_sample_count: 1000
           top_query_count: 200
         query_sample_collection:
-          enabled: true
           max_rows_per_query: 1450
-
 ```
+
+## Feature Gate
+
+A new feature gate was added in `v0.129.0` for removing the `server.address` and `server.port` 
+resource attributes, as they are not identified as resources attributes in the semantic conventions.
+To enable it, pass the following argument to the Collector:
+
+```
+--feature-gates=receiver.sqlserver.RemoveServerResourceAttribute
+```
+
 ## Metrics
 
 Details about the metrics produced by this receiver can be found in [documentation.md](./documentation.md)
@@ -119,3 +177,11 @@ SQL Server docker users may run into an issue that the collector fails to parse
 references:
 1. https://pkg.go.dev/crypto/x509#ParseCertificate
 2. https://github.com/microsoft/mssql-docker/issues/895
+
+## Troubleshooting
+
+### `service.instance.id` is `unknown:1433`
+
+In a rare case, the `service.instance.id` resource attribute is set to `unknown:1433`. This is because the receiver is unable to parse and compute the `service.instance.id` resource attribute.
+
+You can file an issue that includes your configuration to help us investigate the issue.

receiver/sqlserverreceiver/config.go

@@ -5,6 +5,7 @@ package sqlserverreceiver // import "github.com/open-telemetry/opentelemetry-col
 
 import (
 	"errors"
+	"time"
 
 	"go.opentelemetry.io/collector/config/configopaque"
 	"go.opentelemetry.io/collector/scraper/scraperhelper"
@@ -13,8 +14,10 @@ import (
 )
 
 type QuerySample struct {
-	Enabled         bool   `mapstructure:"enabled"`
 	MaxRowsPerQuery uint64 `mapstructure:"max_rows_per_query"`
+
+	// prevent unkeyed literal initialization
+	_ struct{}
 }
 
 type TopQueryCollection struct {
@@ -23,16 +26,32 @@ type TopQueryCollection struct {
 	// The query statement will also be reported, hence, it is not ideal to send it as a metric. Hence
 	// we are reporting them as logs.
 	// The `N` is configured via `TopQueryCount`
-	Enabled             bool `mapstructure:"enabled"`
-	LookbackTime        uint `mapstructure:"lookback_time"`
-	MaxQuerySampleCount uint `mapstructure:"max_query_sample_count"`
-	TopQueryCount       uint `mapstructure:"top_query_count"`
+	LookbackTime        time.Duration `mapstructure:"lookback_time"`
+	MaxQuerySampleCount uint          `mapstructure:"max_query_sample_count"`
+	TopQueryCount       uint          `mapstructure:"top_query_count"`
+	CollectionInterval  time.Duration `mapstructure:"collection_interval"`
+
+	// QueryPlanCacheSize is the maximum number of query plans to cache in memory.
+	// Caching reduces database load by avoiding repeated fetches of the same plan.
+	// Set to 0 to disable caching.
+	QueryPlanCacheSize int `mapstructure:"query_plan_cache_size"`
+
+	// QueryPlanCacheTTL is how long to keep query plans in the cache.
+	// Plans older than this will be evicted and refetched on next collection.
+	QueryPlanCacheTTL time.Duration `mapstructure:"query_plan_cache_ttl"`
+
+	// MaxQueryPlanSize is the maximum size in bytes for a query plan after compression.
+	// Plans exceeding this size will be truncated. Set to 0 for no limit.
+	// This is a safety net for extremely large plans.
+	// Note: Query plans are always compressed using gzip to fit CloudWatch Logs 1MB limit.
+	MaxQueryPlanSize int `mapstructure:"max_query_plan_size"`
 }
 
 // Config defines configuration for a sqlserver receiver.
 type Config struct {
 	scraperhelper.ControllerConfig `mapstructure:",squash"`
 	metadata.MetricsBuilderConfig  `mapstructure:",squash"`
+	metadata.LogsBuilderConfig     `mapstructure:",squash"`
 	// EnableTopQueryCollection enables the collection of the top queries by the execution time.
 	// It will collect the top N queries based on totalElapsedTimeDiffs during the last collection interval.
 	// The query statement will also be reported, hence, it is not ideal to send it as a metric. Hence
@@ -48,6 +67,7 @@ type Config struct {
 	DataSource string `mapstructure:"datasource"`
 
 	Password configopaque.String `mapstructure:"password"`
+	Passfile string              `mapstructure:"passfile"`
 	Port     uint                `mapstructure:"port"`
 	Server   string              `mapstructure:"server"`
 	Username string              `mapstructure:"username"`
@@ -63,6 +83,10 @@ func (cfg *Config) Validate() error {
 		return err
 	}
 
+	if cfg.LookbackTime < 0 {
+		return errors.New("lookback_time cannot have negative values")
+	}
+
 	if cfg.MaxQuerySampleCount > 10000 {
 		return errors.New("`max_query_sample_count` must be between 0 and 10000")
 	}
@@ -71,27 +95,51 @@ func (cfg *Config) Validate() error {
 		return errors.New("`top_query_count` must be less than or equal to `max_query_sample_count`")
 	}
 
+	if cfg.TopQueryCollection.CollectionInterval < 0 {
+		return errors.New("`top_query_collection.collection_interval` must not be less than 0")
+	}
+
 	cfg.isDirectDBConnectionEnabled, err = directDBConnectionEnabled(cfg)
+	if err != nil {
+		return err
+	}
+
+	// When a password is set it takes priority over the passfile, so only the
+	// passfile permissions need to be validated when no inline password is set.
+	if cfg.isDirectDBConnectionEnabled && string(cfg.Password) == "" && cfg.Passfile != "" {
+		if err := cfg.validatePassfilePermissions(); err != nil {
+			return err
+		}
+	}
 
-	return err
+	return nil
 }
 
 func directDBConnectionEnabled(config *Config) (bool, error) {
-	noneOfServerUserPasswordPortSet := config.Server == "" && config.Username == "" && string(config.Password) == "" && config.Port == 0
+	credentialPresent := string(config.Password) != "" || config.Passfile != ""
+
+	noneOfServerUserPasswordPortSet := config.Server == "" && config.Username == "" && !credentialPresent && config.Port == 0
 	if config.DataSource == "" && noneOfServerUserPasswordPortSet {
 		// If no connection information is provided, we can't connect directly and this is a valid config.
 		return false, nil
 	}
 
-	anyOfServerUserPasswordPortSet := config.Server != "" || config.Username != "" || string(config.Password) != "" || config.Port != 0
+	anyOfServerUserPasswordPortSet := config.Server != "" || config.Username != "" || credentialPresent || config.Port != 0
 	if config.DataSource != "" && anyOfServerUserPasswordPortSet {
-		return false, errors.New("wrong config: when specifying 'datasource' no other connection parameters ('server', 'username', 'password', or 'port') should be set")
+		return false, errors.New("wrong config: when specifying 'datasource' no other connection parameters ('server', 'username', 'password', 'passfile', or 'port') should be set")
 	}
 
-	if config.DataSource == "" && (config.Server == "" || config.Username == "" || string(config.Password) == "" || config.Port == 0) {
-		return false, errors.New("wrong config: when specifying either 'server', 'username', 'password', or 'port' all of them need to be specified")
+	if config.DataSource == "" && (config.Server == "" || config.Username == "" || !credentialPresent || config.Port == 0) {
+		return false, errors.New("wrong config: when specifying either 'server', 'username', 'password' (or 'passfile'), or 'port' all of them need to be specified")
 	}
 
 	// It is a valid direct connection configuration
 	return true, nil
 }
+
+func (cfg *Config) EffectiveLookbackTime() time.Duration {
+	if cfg.LookbackTime == 0 {
+		return 2 * cfg.TopQueryCollection.CollectionInterval
+	}
+	return cfg.LookbackTime
+}

receiver/sqlserverreceiver/config.schema.yaml

@@ -0,0 +1,50 @@
+$defs:
+  query_sample:
+    type: object
+    properties:
+      max_rows_per_query:
+        type: integer
+        x-customType: uint64
+  top_query_collection:
+    type: object
+    properties:
+      collection_interval:
+        type: string
+        format: duration
+      lookback_time:
+        description: Enabled enables the collection of the top queries by the execution time. It will collect the top N queries based on totalElapsedTimeDiffs during the last collection interval. The query statement will also be reported, hence, it is not ideal to send it as a metric. Hence we are reporting them as logs. The `N` is configured via `TopQueryCount`
+        type: string
+        format: duration
+      max_query_sample_count:
+        type: integer
+        x-customType: uint
+      top_query_count:
+        type: integer
+        x-customType: uint
+description: Config defines configuration for a sqlserver receiver.
+type: object
+properties:
+  computer_name:
+    type: string
+  datasource:
+    type: string
+  instance_name:
+    type: string
+  password:
+    $ref: go.opentelemetry.io/collector/config/configopaque.string
+  passfile:
+    description: Path to an ADO-connection-string password file; used instead of password. On Linux must be 0600 or 0400.
+    type: string
+  port:
+    type: integer
+    x-customType: uint
+  server:
+    type: string
+  username:
+    type: string
+allOf:
+  - $ref: go.opentelemetry.io/collector/scraper/scraperhelper.controller_config
+  - $ref: ./internal/metadata.metrics_builder_config
+  - $ref: ./internal/metadata.logs_builder_config
+  - $ref: top_query_collection
+  - $ref: query_sample

receiver/sqlserverreceiver/config_others.go

@@ -5,6 +5,31 @@
 
 package sqlserverreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/sqlserverreceiver"
 
-func (cfg *Config) validateInstanceAndComputerName() error {
+import (
+	"fmt"
+	"os"
+	"runtime"
+)
+
+func (*Config) validateInstanceAndComputerName() error {
+	return nil
+}
+
+// validatePassfilePermissions checks that the passfile is accessible on
+// non-Windows platforms. On Linux, it additionally enforces strict permissions.
+func (cfg *Config) validatePassfilePermissions() error {
+	info, err := os.Stat(cfg.Passfile)
+	if err != nil {
+		return fmt.Errorf("`passfile` is inaccessible: %w", err)
+	}
+
+	// On Linux, enforce strict permissions (0600 or 0400 only)
+	if runtime.GOOS == "linux" {
+		perm := info.Mode().Perm()
+		if perm != 0o600 && perm != 0o400 {
+			return fmt.Errorf("`passfile` permissions must be 0600 or 0400, got %#o", perm)
+		}
+	}
+
 	return nil
 }