Docker core check "invalid image name" log spam on kubernetes cluster

[jalaziz]

Describe what happened:

The agent seems to be producing log spam of the nature:

2021-09-28 07:35:03 UTC | CORE | ERROR | (pkg/collector/corechecks/containers/docker/docker.go:64 in updateContainerRunningCount) | Cannot split the image name sha256:41adeff1b17af5dd87ce40fadd5f702460acd9b78fdc5ec5c3db14c1377aba40: invalid image name (is a sha256)

Many such lines are printed approximately every minute. The issue is related to the docker core check.

Describe what you expected:

Given that this does not seem to be an actual erroneous condition, I would expect that the logger would not produce hundreds of these ERROR log lines and gracefully handle the condition.

Steps to reproduce the issue:

Deployed the datadog helm chart onto a k8s cluster with other workloads.

Additional environment details (Operating System, Cloud provider, etc):

AWS EKS / k8s version 1.19

Output of the info page (if this is a bug)

===============
Agent (v7.31.0)
===============

  Status date: 2021-09-28 07:31:04.509 UTC (1632814264509)
  Agent start: 2021-09-28 07:01:40.207 UTC (1632812500207)
  Pid: 1
  Go Version: go1.15.13
  Python Version: 3.8.11
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: INFO

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: -67µs
    System time: 2021-09-28 07:31:04.509 UTC (1632814264509)

  Host Info
  =========
    bootTime: 2021-05-24 12:47:41 UTC (1621860461000)
    kernelArch: x86_64
    kernelVersion: 5.4.117-58.216.amzn2.x86_64
    os: linux
    platform: ubuntu
    platformFamily: debian
    platformVersion: 21.04
    procs: 674
    uptime: 3042h14m7s

  Hostnames
  =========
    ec2-hostname: ip-10-128-1-212.us-west-2.compute.internal
    host_aliases: [ip-10-128-1-212.us-west-2.compute.internal-eks-dev]
    hostname: i-007e0e8de9091aa01
    instance-id: i-007e0e8de9091aa01
    socket-fqdn: datadog-vw78n
    socket-hostname: datadog-vw78n
    host tags:
      cluster_name:eks-dev
      kube_cluster_name:eks-dev
    hostname provider: aws
    unused hostname providers:
      azure: azure_hostname_style is set to 'os'
      configuration/environment: hostname is empty
      gce: unable to retrieve hostname from GCE: status code 404 trying to GET http://169.254.169.254/computeMetadata/v1/instance/hostname

  Metadata
  ========
    cloud_provider: AWS
    hostname_source: aws

=========
Collector
=========

  Running Checks
  ==============

    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
      Total Runs: 118
      Metric Samples: Last Run: 9, Total: 1,055
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-09-28 07:31:03 UTC (1632814263000)
      Last Successful Execution Date : 2021-09-28 07:31:03 UTC (1632814263000)


    disk (4.4.0)
    ------------
      Instance ID: disk:e5dffb8bef24336f [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 208, Total: 24,336
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 158ms
      Last Execution Date : 2021-09-28 07:30:56 UTC (1632814256000)
      Last Successful Execution Date : 2021-09-28 07:30:56 UTC (1632814256000)


    docker
    ------
      Instance ID: docker [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 1,700, Total: 197,742
      Events: Last Run: 1, Total: 3
      Service Checks: Last Run: 1, Total: 117
      Average Execution Time : 932ms
      Last Execution Date : 2021-09-28 07:31:04 UTC (1632814264000)
      Last Successful Execution Date : 2021-09-28 07:31:04 UTC (1632814264000)


    file_handle
    -----------
      Instance ID: file_handle [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 5, Total: 585
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-09-28 07:30:54 UTC (1632814254000)
      Last Successful Execution Date : 2021-09-28 07:30:54 UTC (1632814254000)


    io
    --
      Instance ID: io [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 52, Total: 6,048
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-09-28 07:31:01 UTC (1632814261000)
      Last Successful Execution Date : 2021-09-28 07:31:01 UTC (1632814261000)


    istio (3.14.0)
    --------------
      Instance ID: istio:21edaea6997f7fd0 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 117
      Metric Samples: Last Run: 8,448, Total: 988,416
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 3.695s
      Last Execution Date : 2021-09-28 07:30:56 UTC (1632814256000)
      Last Successful Execution Date : 2021-09-28 07:30:56 UTC (1632814256000)

      Instance ID: istio:4b6a6420b74cf3d6 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 117
      Metric Samples: Last Run: 8, Total: 936
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 30ms
      Last Execution Date : 2021-09-28 07:30:58 UTC (1632814258000)
      Last Successful Execution Date : 2021-09-28 07:30:58 UTC (1632814258000)


    kubelet (7.0.0)
    ---------------
      Instance ID: kubelet:5bbc63f3938c02f4 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/kubelet.d/conf.yaml.default
      Total Runs: 88
      Metric Samples: Last Run: 1,882, Total: 165,486
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 4, Total: 352
      Average Execution Time : 4.205s
      Last Execution Date : 2021-09-28 07:30:54 UTC (1632814254000)
      Last Successful Execution Date : 2021-09-28 07:30:54 UTC (1632814254000)


    load
    ----
      Instance ID: load [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 6, Total: 702
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-09-28 07:30:53 UTC (1632814253000)
      Last Successful Execution Date : 2021-09-28 07:30:53 UTC (1632814253000)


    memory
    ------
      Instance ID: memory [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 18, Total: 2,106
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-09-28 07:31:00 UTC (1632814260000)
      Last Successful Execution Date : 2021-09-28 07:31:00 UTC (1632814260000)


    network (2.3.0)
    ---------------
      Instance ID: network:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 307, Total: 35,721
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 102ms
      Last Execution Date : 2021-09-28 07:30:52 UTC (1632814252000)
      Last Successful Execution Date : 2021-09-28 07:30:52 UTC (1632814252000)


    ntp
    ---
      Instance ID: ntp:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/ntp.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 1, Total: 2
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 2
      Average Execution Time : 0s
      Last Execution Date : 2021-09-28 07:16:48 UTC (1632813408000)
      Last Successful Execution Date : 2021-09-28 07:16:48 UTC (1632813408000)


    redisdb (4.1.0)
    ---------------
      Instance ID: redisdb:1808b16535a5a372 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/redisdb.d/auto_conf.yaml
      Total Runs: 117
      Metric Samples: Last Run: 51, Total: 6,099
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 117
      Average Execution Time : 23ms
      Last Execution Date : 2021-09-28 07:30:50 UTC (1632814250000)
      Last Successful Execution Date : 2021-09-28 07:30:50 UTC (1632814250000)
      metadata:
        version.major: 5
        version.minor: 0
        version.patch: 13
        version.raw: 5.0.13
        version.scheme: semver


    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 1, Total: 117
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-09-28 07:30:59 UTC (1632814259000)
      Last Successful Execution Date : 2021-09-28 07:30:59 UTC (1632814259000)

========
JMXFetch
========

  Information
  ==================
  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    Cluster: 0
    CronJob: 0
    DaemonSet: 0
    Deployment: 0
    Dropped: 0
    DroppedOnInput: 0
    Job: 0
    Node: 0
    PersistentVolume: 0
    PersistentVolumeClaim: 0
    Pod: 0
    ReplicaSet: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Service: 0
    StatefulSet: 0

  Transaction Successes
  =====================
    Total number: 598
    Successes By Endpoint:
      check_run_v1: 117
      intake: 15
      series_v1: 466

  API Keys status
  ===============
    API key ending with 57c25: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - 57c25

==========
Logs Agent
==========


  Logs Agent is not running

=========
APM Agent
=========
  Status: Running
  Pid: 1
  Uptime: 1764 seconds
  Mem alloc: 16,090,176 bytes
  Hostname: i-007e0e8de9091aa01
  Receiver: 0.0.0.0:8126
  Endpoints:
    https://trace.agent.datadoghq.com

  Receiver (previous minute)
  ==========================
    From python 3.9.7 (CPython), client 0.53.0
      Traces received: 730 (1,392,592 bytes)
      Spans received: 4327

    From python 3.9.7 (CPython), client 0.53.2
      Traces received: 66 (48,208 bytes)
      Spans received: 66

    Default priority sampling rate: 10.3%

  Writer (previous minute)
  ========================
    Traces: 0 payloads, 0 traces, 0 events, 0 bytes
    Stats: 0 payloads, 0 stats buckets, 0 bytes

=========
Aggregator
=========
  Checks Metric Sample: 1,432,341
  Dogstatsd Metric Sample: 33,708
  Event: 4
  Events Flushed: 4
  Number Of Flushes: 117
  Series Flushed: 1,423,673
  Service Check: 2,083
  Service Checks Flushed: 2,193
=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 33,707
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 3,800,612
  Udp Packet Reading Errors: 0
  Udp Packets: 17,399
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 1
  Unterminated Metric Errors: 0

=====================
Datadog Cluster Agent
=====================

  - Datadog Cluster Agent endpoint detected: https://172.20.169.146:5005
  Successfully connected to the Datadog Cluster Agent.
  - Running: 1.15.0+commit.6781e85

[drusellers]

Having this same problem. My logs are full of this. For me its while running the dd agent in a docker compose local set up

[gavinbains]

Having the same issue while running the Datadog Agent. Unable to increment custom metrics, could it be a cause?

[jalaziz]

Just upgraded to the latest datadog agent and it's still happening.

[wontae913]

Having the same issue v7.32.4 Datadog Agent...

[v-kumar]

Having similar issues and these messages are polluting the logs and unnecessarily growing their size.

Example:

| CORE | ERROR | (pkg/collector/corechecks/containers/docker/docker.go:64 in updateContainerRunningCount) | Cannot split the image name sha256:c8c23118d9ba3dcc65deacf098dc15a2d1dc10cc60ebd80b91732a3d4a3eb1cc: invalid image name (is a sha256)

[jBouyoud]

Having similar issues

2022-03-01 08:11:42 UTC | PROCESS | WARN | (pkg/workloadmeta/collectors/kubelet/kubelet.go:167 in parsePodContainers) | cannot split image name "sha256:dd5a13fea6fd412ce447742a6c74400527e06afe10a278fccf52e7eb2de43bc8": invalid image name (is a sha256)
2022-03-01 08:11:41 UTC | CORE | WARN | (pkg/workloadmeta/collectors/kubelet/kubelet.go:167 in parsePodContainers) | cannot split image name "sha256:dd5a13fea6fd412ce447742a6c74400527e06afe10a278fccf52e7eb2de43bc8": invalid image name (is a sha256)

[mazay]

seeing the same errors related to ingress-controller pods, we're on v7.34.0

[dd-octo-sts]

This issue has been automatically marked as stale because it has not had activity in the past 15 days.

It will be closed in 30 days if no further activity occurs. If this issue is still relevant, adding a comment will keep it open. Also, you can always reopen the issue if you missed the window.

Thank you for your contributions!

[jacjtito]

Hello, I'm closing this stale issue with no activity. Feel free to reopen it if you have any question. Thanks!