Skip to content

Some Azure SQL fall "flat" after upgrade to 2.14.1 #2718

@SurveySky

Description

@SurveySky

Report

After upgrading scarper and discovery to 2.14.1 and 0.14.1 versions I have noticed that some Azure SQL metrics fall flat like this:

Image

From the screenshot you can see that from 01/22, when I upgraded, metrics were flat until 01/26, when I reverted back to scraper version 2.9.0 and discovery version 0.8.1. Then from 01/28 I upgraded again and they fell flat again. I tried version 2.13.0 and it did the same thing.
I did not see any errors in logs.

Affected metrics that I have noticed:
azure_sql_database_cpu_percent
azure_sql_database_connection_successful
azure_sql_database_dtu_consumption_percent
azure_sql_database_dtu_used
azure_sql_database_deadlock

Expected Behavior

Show actual values for those metrics.

Actual Behavior

It seems like it just shows one value.

Steps to Reproduce the Problem

  1. Upgrade Promitor scarper and resource discovery to 2.14.1 and 0.14.1 versions
  2. Query metrics in Grafana or Prometheus

Component

Scraper

Version

2.14.1

Configuration

Configuration:

---
image:
  repository: __promitor-repo-url__/tomkerkhove/promitor-agent-scraper
  pullPolicy: IfNotPresent
  pullSecrets:
    - name: __artifactory-secret-name__
azureAuthentication:
  identity:
    id: __promitor_sp_app_id__
    key: __promitor_sp_app_key__
# prometheus
metricSinks:
  prometheusScrapingEndpoint:
    enabled: true
    baseUriPath: /metrics
    enableMetricTimestamps: false
    serviceMonitor:
      enabled: true
      namespace: promitor

azureMetadata:
  tenantId: __promitor_sp_tenant_id__
  subscriptionId: __subscription-id__
  resourceGroupName: promitor
  cloud: "Global"
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping:
    schedule: "*/5 * * * *"

resources:
  requests:
    cpu: 1500m
    memory: 128Mi

resourceDiscovery:
  enabled: true
  host: promitor-agent-resource-discovery
  port: 8889

tolerations:
  - key: dedicated_pool
    operator: Equal
    value: monitoring
    effect: NoSchedule
nodeSelector:
  purpose: monitoring

metrics:
  # SQL Database
  - name: azure_sql_database_allocated_data_storage
    azureMetricConfiguration:
      metricName: allocated_data_storage
      aggregation:
        type: Average
    description: Allocated data storage. Not applicable to data warehouses.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_app_cpu_billed
    azureMetricConfiguration:
      metricName: app_cpu_billed
      aggregation:
        type: Total
    description: App CPU billed. Applies to serverless databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_app_cpu_percent
    azureMetricConfiguration:
      metricName: app_cpu_percent
      aggregation:
        type: Average
    description: App CPU percentage. Applies to serverless databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_app_memory_percent
    azureMetricConfiguration:
      metricName: app_memory_percent
      aggregation:
        type: Average
    description: App memory percentage. Applies to serverless databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_blocked_by_firewall
    azureMetricConfiguration:
      metricName: blocked_by_firewall
      aggregation:
        type: Total
    description: Blocked by Firewall
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_connection_failed
    azureMetricConfiguration:
      metricName: connection_failed
      aggregation:
        type: Total
    description: Failed Connections
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_connection_successful
    azureMetricConfiguration:
      metricName: connection_successful
      aggregation:
        type: Total
    description: Successful Connections
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_cpu_limit
    azureMetricConfiguration:
      metricName: cpu_limit
      aggregation:
        type: Average
    description: CPU limit. Applies to vCore-based databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_cpu_percent
    azureMetricConfiguration:
      metricName: cpu_percent
      aggregation:
        type: Average
    description: CPU percentage
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_cpu_used
    azureMetricConfiguration:
      metricName: cpu_used
      aggregation:
        type: Average
    description: CPU used. Applies to vCore-based databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_deadlock
    azureMetricConfiguration:
      metricName: deadlock
      aggregation:
        type: Total
    description: Deadlocks. Not applicable to data warehouses.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_diff_backup_size_bytes
    azureMetricConfiguration:
      metricName: diff_backup_size_bytes
      aggregation:
        type: Maximum
    description: Cumulative differential backup storage size. Applies to vCore-based databases. Not applicable to Hyperscale databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_dtu_consumption_percent
    azureMetricConfiguration:
      metricName: dtu_consumption_percent
      aggregation:
        type: Average
    description: DTU Percentage. Applies to DTU-based databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_dtu_limit
    azureMetricConfiguration:
      metricName: dtu_limit
      aggregation:
        type: Average
    description: DTU Limit. Applies to DTU-based databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_dtu_used
    azureMetricConfiguration:
      metricName: dtu_used
      aggregation:
        type: Average
    description: DTU used. Applies to DTU-based databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_full_backup_size_bytes
    azureMetricConfiguration:
      metricName: full_backup_size_bytes
      aggregation:
        type: Maximum
    description: Cumulative full backup storage size. Applies to vCore-based databases. Not applicable to Hyperscale databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_log_backup_size_bytes
    azureMetricConfiguration:
      metricName: log_backup_size_bytes
      aggregation:
        type: Maximum
    description: Cumulative log backup storage size. Applies to vCore-based and Hyperscale databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_log_write_percent
    azureMetricConfiguration:
      metricName: log_write_percent
      aggregation:
        type: Average
    description: Log IO percentage. Not applicable to data warehouses.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_physical_data_read_percent
    azureMetricConfiguration:
      metricName: physical_data_read_percent
      aggregation:
        type: Average
    description: Data IO percentage
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_sessions_percent
    azureMetricConfiguration:
      metricName: sessions_percent
      aggregation:
        type: Average
    description: Sessions percentage. Not applicable to data warehouses.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_sqlserver_process_core_percent
    azureMetricConfiguration:
      metricName: sqlserver_process_core_percent
      aggregation:
        type: Maximum
    description: CPU usage as a percentage of the SQL DB process. Not applicable to data warehouses.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_sqlserver_process_memory_percent
    azureMetricConfiguration:
      metricName: sqlserver_process_memory_percent
      aggregation:
        type: Maximum
    description: Memory usage as a percentage of the SQL DB process. Not applicable to data warehouses.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_storage
    azureMetricConfiguration:
      metricName: storage
      aggregation:
        type: Maximum
    description: Data space used. Not applicable to data warehouses.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_storage_percent
    azureMetricConfiguration:
      metricName: storage_percent
      aggregation:
        type: Maximum
    description: Data space used percent. Not applicable to data warehouses or hyperscale databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_tempdb_data_size
    azureMetricConfiguration:
      metricName: tempdb_data_size
      aggregation:
        type: Maximum
    description: Space used in tempdb data files in kilobytes. Not applicable to data warehouses.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_tempdb_log_size
    azureMetricConfiguration:
      metricName: tempdb_log_size
      aggregation:
        type: Maximum
    description: Space used in tempdb transaction log file in kilobytes. Not applicable to data warehouses.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_tempdb_log_used_percent
    azureMetricConfiguration:
      metricName: tempdb_log_used_percent
      aggregation:
        type: Maximum
    description: Space used percentage in tempdb transaction log file. Not applicable to data warehouses.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_workers_percent
    azureMetricConfiguration:
      metricName: workers_percent
      aggregation:
        type: Average
    description: Workers percentage. Not applicable to data warehouses.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_xtp_storage_percent
    azureMetricConfiguration:
      metricName: xtp_storage_percent
      aggregation:
        type: Average
    description: In-Memory OLTP storage percent. Not applicable to data warehouses.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_availability
    azureMetricConfiguration:
      metricName: availability
      aggregation:
        type: Average
    description: SLA-compliant availability percentage for the SQL database.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_connection_failed_user_error
    azureMetricConfiguration:
      metricName: connection_failed_user_error
      aggregation:
        type: Total
    description: Count of failed connections due to user errors.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_sessions_count
    azureMetricConfiguration:
      metricName: sessions_count
      aggregation:
        type: Average
    description: Active session count.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_replication_lag_seconds
    azureMetricConfiguration:
      metricName: replication_lag_seconds
      aggregation:
        type: Maximum
    description: Replication lag in seconds between primary and replica databases.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_sql_instance_cpu_percent
    azureMetricConfiguration:
      metricName: sql_instance_cpu_percent
      aggregation:
        type: Average
    description: CPU usage by all user and system workloads on the SQL instance.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

  - name: azure_sql_database_sql_instance_memory_percent
    azureMetricConfiguration:
      metricName: sql_instance_memory_percent
      aggregation:
        type: Average
    description: Memory usage by the database engine instance.
    resourceType: SqlDatabase
    resourceDiscoveryGroups:
      - name: sql-database

Logs

No response

Platform

Microsoft Azure

Contact Details

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions