Logging multi-line klog output on cluster-autoscaler when scaling failed #7702
Open
Description
Which component are you using?:
/area cluster-autoscaler
What version of the component are you using?:
Component version: v1.31.1
What k8s version are you using (kubectl version)?:
kubectl version Output
$ kubectl version Client Version: v1.30.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.31.4-eks-2d5f260
What environment is this in?:
AWS EKS
What did you expect to happen?:
cluster-autoscaler always logs single-line klog output.
What happened instead?:
auto_scaling_groups.go:551 log contains multi line logging in klog.
I0116 07:10:42.719737 1 auto_scaling_groups.go:508] Instance group eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx has only 0 instances created while requested count is 1. Creating placeholder instances.
W0116 07:10:42.815402 1 auto_scaling_groups.go:551] ASG eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx scaling failed with {
ActivityId: "4b6651b7-c98a-a0c3-5fe1-xxxxxxxxxxxx",
AutoScalingGroupARN: "arn:aws:autoscaling:ap-northeast-1:xxxxxxxxxxxx:autoScalingGroup:15be2ddc-7564-4bf2-bb35-xxxxxxxxxxxx:autoScalingGroupName/eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx",
AutoScalingGroupName: "eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx",
Cause: "At 2025-01-16T07:09:48Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1.",
Description: "Launching a new EC2 instance. Status Reason: Could not launch Spot Instances. UnfulfillableCapacity - Unable to fulfill capacity due to your request configuration. Please adjust your request and try again. Launching EC2 instance failed.",
Details: "{\"Subnet ID\":\"subnet-xxxxxxxxxxxx\",\"Availability Zone\":\"ap-northeast-1a\"}",
EndTime: 2025-01-16 07:09:49 +0000 UTC,
Progress: 100,
StartTime: 2025-01-16 07:09:49.096 +0000 UTC,
StatusCode: "Failed",
StatusMessage: "Could not launch Spot Instances. UnfulfillableCapacity - Unable to fulfill capacity due to your request configuration. Please adjust your request and try again. Launching EC2 instance failed."
}
W0116 07:10:42.815520 1 auto_scaling_groups.go:516] Instance group eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx cannot provision any more nodes!
I0116 07:10:43.011817 1 auto_scaling_groups.go:161] Updated ASG cache for eks-daemon-spot-202407091400-ap-northeast-1a-56c85d24-0d05-8da9-1e78-xxxxxxxxxxxx. min/max/current is 1/450/2Then some services cannot parse this log as correct klog such as datadog and so on.
How to reproduce it (as minimally and precisely as possible):
just logging on
Anything else we need to know?:
N/A