Skip to content

Logging multi-line klog output on cluster-autoscaler when scaling failed #7702

Open
@krrrr38

Description

Which component are you using?:

/area cluster-autoscaler

What version of the component are you using?:

Component version: v1.31.1

What k8s version are you using (kubectl version)?:

kubectl version Output
$ kubectl version
Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.31.4-eks-2d5f260

What environment is this in?:

AWS EKS

What did you expect to happen?:

cluster-autoscaler always logs single-line klog output.

What happened instead?:

auto_scaling_groups.go:551 log contains multi line logging in klog.

I0116 07:10:42.719737       1 auto_scaling_groups.go:508] Instance group eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx has only 0 instances created while requested count is 1. Creating placeholder instances.
W0116 07:10:42.815402       1 auto_scaling_groups.go:551] ASG eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx scaling failed with {
  ActivityId: "4b6651b7-c98a-a0c3-5fe1-xxxxxxxxxxxx",
  AutoScalingGroupARN: "arn:aws:autoscaling:ap-northeast-1:xxxxxxxxxxxx:autoScalingGroup:15be2ddc-7564-4bf2-bb35-xxxxxxxxxxxx:autoScalingGroupName/eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx",
  AutoScalingGroupName: "eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx",
  Cause: "At 2025-01-16T07:09:48Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1.",
  Description: "Launching a new EC2 instance.  Status Reason: Could not launch Spot Instances. UnfulfillableCapacity - Unable to fulfill capacity due to your request configuration. Please adjust your request and try again. Launching EC2 instance failed.",
  Details: "{\"Subnet ID\":\"subnet-xxxxxxxxxxxx\",\"Availability Zone\":\"ap-northeast-1a\"}",
  EndTime: 2025-01-16 07:09:49 +0000 UTC,
  Progress: 100,
  StartTime: 2025-01-16 07:09:49.096 +0000 UTC,
  StatusCode: "Failed",
  StatusMessage: "Could not launch Spot Instances. UnfulfillableCapacity - Unable to fulfill capacity due to your request configuration. Please adjust your request and try again. Launching EC2 instance failed."
}
W0116 07:10:42.815520       1 auto_scaling_groups.go:516] Instance group eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx cannot provision any more nodes!
I0116 07:10:43.011817       1 auto_scaling_groups.go:161] Updated ASG cache for eks-daemon-spot-202407091400-ap-northeast-1a-56c85d24-0d05-8da9-1e78-xxxxxxxxxxxx. min/max/current is 1/450/2

Then some services cannot parse this log as correct klog such as datadog and so on.

How to reproduce it (as minimally and precisely as possible):

just logging on

klog.Warningf("ASG %s scaling failed with %s", asgRef.Name, *activity)

Anything else we need to know?:

N/A

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions