Strimzi
KafkaRebalance stuck in PendingProposal state #8248
prateekkohli21
started this conversation in
General
Replies: 1 comment 12 replies
-
|
How much did you wait? Cruise Control takes some time for getting enough samples in the time window in order to provide you a proposal. It could even take several minutes the first time. It will be faster later. |
Beta Was this translation helpful? Give feedback.
12 replies
-
|
Still facing this random issue. Pasting Cruise Control logs below, if they can help. 2023-03-23 10:20:21 INFO operationLogger:740 - Task [cba39f40-4771-4986-9f2b-017ad2a7addf] calculation finishes, result:
MonitorState: {state: RUNNING(20.000% trained), NumValidWindows: (0/1) (0.000%), NumValidPartitions: 36/65 (55.385%), flawedPartitions: 0}
ExecutorState: {state: NO_TASK_IN_PROGRESS}
AnalyzerState: {isProposalReady: false, readyGoals: [ReplicaDistributionGoal, PreferredLeaderElectionGoal, RackAwareGoal, MinTopicLeadersPerBrokerGoal, TopicReplicaDistributionGoal, LeaderReplicaDistributionGoal, ReplicaCapacityGoal]}
AnomalyDetectorState: {selfHealingEnabled:[], selfHealingDisabled:[BROKER_FAILURE, DISK_FAILURE, GOAL_VIOLATION, METRIC_ANOMALY, TOPIC_ANOMALY, MAINTENANCE_EVENT], selfHealingEnabledRatio:{BROKER_FAILURE=0.0, DISK_FAILURE=0.0, GOAL_VIOLATION=0.0, METRIC_ANOMALY=0.0, TOPIC_ANOMALY=0.0, MAINTENANCE_EVENT=0.0}, recentGoalViolations:[], recentBrokerFailures:[{anomalyId=888aeba3-4bf9-4e47-a0e4-d347c5ae78ed, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:48:03Z, statusUpdateDate=2023-03-23T09:48:03Z, status=IGNORED}, {anomalyId=05dd102e-72b2-4fb9-80ab-0cca6a6314c0, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:38:03Z, statusUpdateDate=2023-03-23T09:38:03Z, status=IGNORED}, {anomalyId=cd829b45-bfba-4278-81d3-766988329682, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T10:03:03Z, statusUpdateDate=2023-03-23T10:03:03Z, status=IGNORED}, {anomalyId=425dfbc4-b332-4e63-8f16-3ce0d1ecb6da, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T10:13:03Z, statusUpdateDate=2023-03-23T10:13:03Z, status=IGNORED}, {anomalyId=692f30eb-5228-4aed-a42b-c475adea6b8f, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T10:18:03Z, statusUpdateDate=2023-03-23T10:18:03Z, status=IGNORED}, {anomalyId=80cb0541-ff78-4ee8-a8a5-c697b390714f, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:53:03Z, statusUpdateDate=2023-03-23T09:53:03Z, status=IGNORED}, {anomalyId=0595981c-8f0a-4a9d-a17b-82d62305e784, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:58:03Z, statusUpdateDate=2023-03-23T09:58:03Z, status=IGNORED}, {anomalyId=cd1362d8-f482-45ca-98df-8cb48b3663d7, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:43:03Z, statusUpdateDate=2023-03-23T09:43:03Z, status=IGNORED}, {anomalyId=39b1502a-0fa1-47d4-bce7-075f152a197b, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T10:08:03Z, statusUpdateDate=2023-03-23T10:08:03Z, status=IGNORED}], recentMetricAnomalies:[], recentDiskFailures:[], recentTopicAnomalies:[], recentMaintenanceEvents:[], metrics:{meanTimeBetweenAnomalies:{GOAL_VIOLATION:0.00 milliseconds, BROKER_FAILURE:3.34 milliseconds, METRIC_ANOMALY:0.00 milliseconds, DISK_FAILURE:0.00 milliseconds, TOPIC_ANOMALY:0.00 milliseconds}, meanTimeToStartFix:0.00 milliseconds, numSelfHealingStarted:0, numSelfHealingFailedToStart:0, ongoingAnomalyDuration=42.24 minutes}, ongoingSelfHealingAnomaly:None, balancednessScore:-1.000}
2023-03-23 10:20:25 INFO NetworkClient:935 - [AdminClient clientId=adminclient-2] Node 2 disconnected.
2023-03-23 10:20:28 INFO AbstractAsyncRequest:50 - Processing async request CruiseControlStateRequest.
2023-03-23 10:20:28 INFO UserTaskManager:263 - Create a new UserTask f23a48a6-d2a2-4fb3-ad2e-86c9c5e80e06 with SessionKey SessionKey{_httpSession=Session@d54367f{id=node0m290lqrs36ee136o364b1dkfr539,x=node0m290lqrs36ee136o364b1dkfr539.node0,req=1,res=true},_requestUrl=GET /kafkacruisecontrol/state,_queryParams={}}
2023-03-23 10:20:28 INFO AbstractAsyncRequest:58 - Computation is completed for async request: /state.
2023-03-23 10:20:28 INFO CruiseControlPublicAccessLogger:62 - 127.0.0.1 - user [23/Mar/2023:10:20:28 +0000] "GET /kafkacruisecontrol/state HTTP/1.1" 200 3064
2023-03-23 10:20:28 INFO AbstractAsyncRequest:50 - Processing async request CruiseControlStateRequest.
2023-03-23 10:20:28 INFO UserTaskManager:263 - Create a new UserTask 5f9d8fb2-c257-4fc3-9874-8779031680bd with SessionKey SessionKey{_httpSession=Session@58b74271{id=node04exdlah06pxv11j67ecnuglvn540,x=node04exdlah06pxv11j67ecnuglvn540.node0,req=1,res=true},_requestUrl=GET /kafkacruisecontrol/state,_queryParams={}}
2023-03-23 10:20:28 INFO AbstractAsyncRequest:58 - Computation is completed for async request: /state.
2023-03-23 10:20:28 INFO CruiseControlPublicAccessLogger:62 - 127.0.0.1 - user [23/Mar/2023:10:20:28 +0000] "GET /kafkacruisecontrol/state HTTP/1.1" 200 3064
2023-03-23 10:20:30 INFO NetworkClient:935 - [AdminClient clientId=adminclient-3] Node 1 disconnected.
2023-03-23 10:20:31 INFO UserTaskManager:305 - Expiring the session associated with SessionKey{_httpSession=Session@49e56076{id=node017da1bajp6gkk1hio457um9vr0527,x=node017da1bajp6gkk1hio457um9vr0527.node0,req=0,res=true},_requestUrl=GET /kafkacruisecontrol/state,_queryParams={}}.
2023-03-23 10:20:31 INFO UserTaskManager:305 - Expiring the session associated with SessionKey{_httpSession=Session@619c59f6{id=node010u2ol3wibbid19hip40z0ghzp528,x=node010u2ol3wibbid19hip40z0ghzp528.node0,req=0,res=true},_requestUrl=GET /kafkacruisecontrol/state,_queryParams={}}.
2023-03-23 10:20:31 INFO UserTaskManager:353 - UserTask f23a48a6-d2a2-4fb3-ad2e-86c9c5e80e06 is completed and removed from active tasks list
2023-03-23 10:20:31 INFO UserTaskManager:353 - UserTask 5f9d8fb2-c257-4fc3-9874-8779031680bd is completed and removed from active tasks list
2023-03-23 10:20:31 INFO operationLogger:740 - Task [f23a48a6-d2a2-4fb3-ad2e-86c9c5e80e06] calculation finishes, result:
MonitorState: {state: RUNNING(20.000% trained), NumValidWindows: (0/1) (0.000%), NumValidPartitions: 36/65 (55.385%), flawedPartitions: 0}
ExecutorState: {state: NO_TASK_IN_PROGRESS}
AnalyzerState: {isProposalReady: false, readyGoals: [ReplicaDistributionGoal, PreferredLeaderElectionGoal, RackAwareGoal, MinTopicLeadersPerBrokerGoal, TopicReplicaDistributionGoal, LeaderReplicaDistributionGoal, ReplicaCapacityGoal]}
AnomalyDetectorState: {selfHealingEnabled:[], selfHealingDisabled:[BROKER_FAILURE, DISK_FAILURE, GOAL_VIOLATION, METRIC_ANOMALY, TOPIC_ANOMALY, MAINTENANCE_EVENT], selfHealingEnabledRatio:{BROKER_FAILURE=0.0, DISK_FAILURE=0.0, GOAL_VIOLATION=0.0, METRIC_ANOMALY=0.0, TOPIC_ANOMALY=0.0, MAINTENANCE_EVENT=0.0}, recentGoalViolations:[], recentBrokerFailures:[{anomalyId=888aeba3-4bf9-4e47-a0e4-d347c5ae78ed, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:48:03Z, statusUpdateDate=2023-03-23T09:48:03Z, status=IGNORED}, {anomalyId=05dd102e-72b2-4fb9-80ab-0cca6a6314c0, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:38:03Z, statusUpdateDate=2023-03-23T09:38:03Z, status=IGNORED}, {anomalyId=cd829b45-bfba-4278-81d3-766988329682, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T10:03:03Z, statusUpdateDate=2023-03-23T10:03:03Z, status=IGNORED}, {anomalyId=425dfbc4-b332-4e63-8f16-3ce0d1ecb6da, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T10:13:03Z, statusUpdateDate=2023-03-23T10:13:03Z, status=IGNORED}, {anomalyId=692f30eb-5228-4aed-a42b-c475adea6b8f, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T10:18:03Z, statusUpdateDate=2023-03-23T10:18:03Z, status=IGNORED}, {anomalyId=80cb0541-ff78-4ee8-a8a5-c697b390714f, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:53:03Z, statusUpdateDate=2023-03-23T09:53:03Z, status=IGNORED}, {anomalyId=0595981c-8f0a-4a9d-a17b-82d62305e784, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:58:03Z, statusUpdateDate=2023-03-23T09:58:03Z, status=IGNORED}, {anomalyId=cd1362d8-f482-45ca-98df-8cb48b3663d7, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:43:03Z, statusUpdateDate=2023-03-23T09:43:03Z, status=IGNORED}, {anomalyId=39b1502a-0fa1-47d4-bce7-075f152a197b, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T10:08:03Z, statusUpdateDate=2023-03-23T10:08:03Z, status=IGNORED}], recentMetricAnomalies:[], recentDiskFailures:[], recentTopicAnomalies:[], recentMaintenanceEvents:[], metrics:{meanTimeBetweenAnomalies:{GOAL_VIOLATION:0.00 milliseconds, BROKER_FAILURE:3.33 milliseconds, METRIC_ANOMALY:0.00 milliseconds, DISK_FAILURE:0.00 milliseconds, TOPIC_ANOMALY:0.00 milliseconds}, meanTimeToStartFix:0.00 milliseconds, numSelfHealingStarted:0, numSelfHealingFailedToStart:0, ongoingAnomalyDuration=42.41 minutes}, ongoingSelfHealingAnomaly:None, balancednessScore:-1.000}
2023-03-23 10:20:31 INFO operationLogger:740 - Task [5f9d8fb2-c257-4fc3-9874-8779031680bd] calculation finishes, result:
MonitorState: {state: RUNNING(20.000% trained), NumValidWindows: (0/1) (0.000%), NumValidPartitions: 36/65 (55.385%), flawedPartitions: 0}
ExecutorState: {state: NO_TASK_IN_PROGRESS}
AnalyzerState: {isProposalReady: false, readyGoals: [ReplicaDistributionGoal, PreferredLeaderElectionGoal, RackAwareGoal, MinTopicLeadersPerBrokerGoal, TopicReplicaDistributionGoal, LeaderReplicaDistributionGoal, ReplicaCapacityGoal]}
AnomalyDetectorState: {selfHealingEnabled:[], selfHealingDisabled:[BROKER_FAILURE, DISK_FAILURE, GOAL_VIOLATION, METRIC_ANOMALY, TOPIC_ANOMALY, MAINTENANCE_EVENT], selfHealingEnabledRatio:{BROKER_FAILURE=0.0, DISK_FAILURE=0.0, GOAL_VIOLATION=0.0, METRIC_ANOMALY=0.0, TOPIC_ANOMALY=0.0, MAINTENANCE_EVENT=0.0}, recentGoalViolations:[], recentBrokerFailures:[{anomalyId=888aeba3-4bf9-4e47-a0e4-d347c5ae78ed, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:48:03Z, statusUpdateDate=2023-03-23T09:48:03Z, status=IGNORED}, {anomalyId=05dd102e-72b2-4fb9-80ab-0cca6a6314c0, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:38:03Z, statusUpdateDate=2023-03-23T09:38:03Z, status=IGNORED}, {anomalyId=cd829b45-bfba-4278-81d3-766988329682, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T10:03:03Z, statusUpdateDate=2023-03-23T10:03:03Z, status=IGNORED}, {anomalyId=425dfbc4-b332-4e63-8f16-3ce0d1ecb6da, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T10:13:03Z, statusUpdateDate=2023-03-23T10:13:03Z, status=IGNORED}, {anomalyId=692f30eb-5228-4aed-a42b-c475adea6b8f, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T10:18:03Z, statusUpdateDate=2023-03-23T10:18:03Z, status=IGNORED}, {anomalyId=80cb0541-ff78-4ee8-a8a5-c697b390714f, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:53:03Z, statusUpdateDate=2023-03-23T09:53:03Z, status=IGNORED}, {anomalyId=0595981c-8f0a-4a9d-a17b-82d62305e784, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:58:03Z, statusUpdateDate=2023-03-23T09:58:03Z, status=IGNORED}, {anomalyId=cd1362d8-f482-45ca-98df-8cb48b3663d7, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T09:43:03Z, statusUpdateDate=2023-03-23T09:43:03Z, status=IGNORED}, {anomalyId=39b1502a-0fa1-47d4-bce7-075f152a197b, failedBrokersByTimeMs={4=1679564283839, 3=1679564283839}, detectionDate=2023-03-23T10:08:03Z, statusUpdateDate=2023-03-23T10:08:03Z, status=IGNORED}], recentMetricAnomalies:[], recentDiskFailures:[], recentTopicAnomalies:[], recentMaintenanceEvents:[], metrics:{meanTimeBetweenAnomalies:{GOAL_VIOLATION:0.00 milliseconds, BROKER_FAILURE:3.33 milliseconds, METRIC_ANOMALY:0.00 milliseconds, DISK_FAILURE:0.00 milliseconds, TOPIC_ANOMALY:0.00 milliseconds}, meanTimeToStartFix:0.00 milliseconds, numSelfHealingStarted:0, numSelfHealingFailedToStart:0, ongoingAnomalyDuration=42.41 minutes}, ongoingSelfHealingAnomaly:None, balancednessScore:-1.000}
2023-03-23 10:20:31 INFO MetricFetcherManager:172 - Kicking off metric sampling for time range [1679566771396, 1679566831396], duration 60000 ms with timeout 60000 ms.
2023-03-23 10:20:31 INFO KafkaConsumer:1585 - [Consumer clientId=CruiseControlMetricsReporterSampler-consumer--2327907188206490066, groupId=null] Seeking to offset 51606 for partition strimzi.cruisecontrol.metrics-0
2023-03-23 10:20:31 INFO CruiseControlMetricsReporterSampler:116 - Finished sampling from topic strimzi.cruisecontrol.metrics for partitions [0] in time range [1679566771396,1679566831396]. Collected 273 metrics.
2023-03-23 10:20:31 INFO CruiseControlMetricsProcessor:150 - Generated 36(29 skipped by broker {-1=29}) partition metric samples and 3 broker metric samples for timestamp 1679566814384.
2023-03-23 10:20:31 INFO MetricSampleAggregator:496 - PARTITION Aggregator rolled out 1 new windows, reset 1 windows, current window range [1679566800000, 1679567100000], abandon 180 samples.
2023-03-23 10:20:31 INFO SamplingFetcher:94 - Collected 36 partition metric samples for 36 partitions. Total partition assigned: 65.
2023-03-23 10:20:31 INFO MetricSampleAggregator:496 - BROKER Aggregator rolled out 1 new windows, reset 1 windows, current window range [1679561100000, 1679567100000], abandon 0 samples.
2023-03-23 10:20:31 INFO SamplingFetcher:121 - Collected 3 broker metric samples for 3 brokers.
2023-03-23 10:20:31 INFO MetricFetcherManager:242 - Finished sampling in 27 ms.
2023-03-23 10:20:31 INFO GoalOptimizer:150 - Skipping proposal precomputing because load monitor does not have enough snapshots. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @prateekkohli21 thanks for posting the new Cruise Control logs
This implies there is a problem with broker -1. Until it is configured properly, CC will not be able to collect samples from all the brokers and won't be able to generate optimization proposals. What is strange is that the broker mentioned has a negative integer for a broker id, |
Beta Was this translation helpful? Give feedback.
-
|
ID But no idea if it is in any way related to this. |
Beta Was this translation helpful? Give feedback.
-
That is interesting, definitely would be good to see those Kafka broker logs @prateekkohli21 For getting a better idea of what is happening here, could you share the complete logs of:
They will likely be long and messy so if you could attach them as files here instead of posting the text that would be great! |
Beta Was this translation helpful? Give feedback.
-
|
Interesting that in the above Cruise Control log, I see lines where the consumer is consuming metrics produced by the metrics reporter running on the Kafka brokers. It also print |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I deployed Cruise Control and created a KafkaRebalance resource in FULL mode, but my Rebalace resource is stuck in PendingProposal state.
Getting below error in Cruise Control logs continuously:
2023-03-17 12:36:58 INFO UserTaskManager:305 - Expiring the session associated with SessionKey{_httpSession=Session@347da2c9{id=node0vugo9sa6nqwhzymvs3mryfiu1593,x=node0vugo9sa6nqwhzymvs3mryfiu1593.node0,req=0,res=true},_requestUrl=POST /kafkacruisecontrol/rebalance,_queryParams={dryrun=[true], skip_hard_goal_check=[false], rebalance_disk=[false], json=[true], verbose=[true]}}. 2023-03-17 12:36:58 WARN UserTaskManager:347 - UserTask c58ee597-67fe-466f-84e1-de82d6302db7 is completed with Exception and removed from active tasks list 2023-03-17 12:36:58 INFO operationLogger:742 - Task [c58ee597-67fe-466f-84e1-de82d6302db7] calculation fails, exception: java.util.concurrent.ExecutionException: Operation 'Rebalance' received exception. com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There are only 0 valid windows when aggregating in range [-1, 1679056613911] for aggregation options (minValidEntityRatio=1.00, minValidEntityGroupRatio=0.00, minValidWindows=1, numEntitiesToInclude=65, granularity=ENTITY) 2023-03-17 12:36:58 INFO AbstractAsyncRequest:50 - Processing async request RebalanceRequest. 2023-03-17 12:36:58 INFO UserTaskManager:263 - Create a new UserTask 3def3af2-4e99-40b0-80b6-2e96c8c64b42 with SessionKey SessionKey{_httpSession=Session@10ca3fa6{id=node01kh5zcmg3e22bhzm26pv6abis1618,x=node01kh5zcmg3e22bhzm26pv6abis1618.node0,req=1,res=true},_requestUrl=POST /kafkacruisecontrol/rebalance,_queryParams={dryrun=[true], skip_hard_goal_check=[false], rebalance_disk=[false], json=[true], verbose=[true]}} 2023-03-17 12:36:58 WARN OperationRunnable:50 - Received exception when trying to execute runnable for "Rebalance" com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There are only 0 valid windows when aggregating in range [-1, 1679056618909] for aggregation options (minValidEntityRatio=1.00, minValidEntityGroupRatio=0.00, minValidWindows=1, numEntitiesToInclude=65, granularity=ENTITY) at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:167) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.workWithoutClusterModel(RebalanceRunnable.java:118) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:173) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.getResult(RebalanceRunnable.java:90) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.getResult(RebalanceRunnable.java:30) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.OperationRunnable.run(OperationRunnable.java:45) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.run(GoalBasedOperationRunnable.java:36) ~[cruise-control-2.5.100.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] Caused by: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There are only 0 valid windows when aggregating in range [-1, 1679056618909] for aggregation options (minValidEntityRatio=1.00, minValidEntityGroupRatio=0.00, minValidWindows=1, numEntitiesToInclude=65, granularity=ENTITY) at com.linkedin.cruisecontrol.monitor.sampling.aggregator.MetricSampleAggregator.validateCompleteness(MetricSampleAggregator.java:542) ~[cruise-control-core-2.5.100.jar:?] at com.linkedin.cruisecontrol.monitor.sampling.aggregator.MetricSampleAggregator.aggregate(MetricSampleAggregator.java:213) ~[cruise-control-core-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.monitor.sampling.aggregator.KafkaPartitionMetricSampleAggregator.aggregate(KafkaPartitionMetricSampleAggregator.java:151) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.monitor.LoadMonitor.clusterModel(LoadMonitor.java:496) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.clusterModel(KafkaCruiseControl.java:369) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.ProposalsRunnable.workWithClusterModel(ProposalsRunnable.java:88) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:161) ~[cruise-control-2.5.100.jar:?] ... 11 more 2023-03-17 12:36:58 ERROR KafkaCruiseControlServlet:141 - Error processing POST request '/rebalance' due to: 'com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There are only 0 valid windows when aggregating in range [-1, 1679056618909] for aggregation options (minValidEntityRatio=1.00, minValidEntityGroupRatio=0.00, minValidWindows=1, numEntitiesToInclude=65, granularity=ENTITY)'. java.util.concurrent.ExecutionException: com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There are only 0 valid windows when aggregating in range [-1, 1679056618909] for aggregation options (minValidEntityRatio=1.00, minValidEntityGroupRatio=0.00, minValidWindows=1, numEntitiesToInclude=65, granularity=ENTITY) at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022) ~[?:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.AbstractAsyncRequest.getResponse(AbstractAsyncRequest.java:57) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.AbstractRequest.handle(AbstractRequest.java:41) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.handlePost(KafkaCruiseControlServlet.java:229) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.doGetOrPost(KafkaCruiseControlServlet.java:127) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.doPost(KafkaCruiseControlServlet.java:106) ~[cruise-control-2.5.100.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[javax.servlet-api-3.1.0.jar:3.1.0] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) ~[jetty-servlet-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560) ~[jetty-security-9.4.47.v20220610.jar:9.4.47.v20220610] at com.linkedin.kafka.cruisecontrol.servlet.security.CruiseControlSecurityHandler.handle(CruiseControlSecurityHandler.java:22) ~[cruise-control-2.5.100.jar:?] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) ~[jetty-io-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) ~[jetty-io-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) ~[jetty-io-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610] at java.lang.Thread.run(Thread.java:829) ~[?:?] Caused by: com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There are only 0 valid windows when aggregating in range [-1, 1679056618909] for aggregation options (minValidEntityRatio=1.00, minValidEntityGroupRatio=0.00, minValidWindows=1, numEntitiesToInclude=65, granularity=ENTITY) at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:167) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.workWithoutClusterModel(RebalanceRunnable.java:118) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:173) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.getResult(RebalanceRunnable.java:90) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.getResult(RebalanceRunnable.java:30) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.OperationRunnable.run(OperationRunnable.java:45) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.run(GoalBasedOperationRunnable.java:36) ~[cruise-control-2.5.100.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] ... 1 more Caused by: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There are only 0 valid windows when aggregating in range [-1, 1679056618909] for aggregation options (minValidEntityRatio=1.00, minValidEntityGroupRatio=0.00, minValidWindows=1, numEntitiesToInclude=65, granularity=ENTITY) at com.linkedin.cruisecontrol.monitor.sampling.aggregator.MetricSampleAggregator.validateCompleteness(MetricSampleAggregator.java:542) ~[cruise-control-core-2.5.100.jar:?] at com.linkedin.cruisecontrol.monitor.sampling.aggregator.MetricSampleAggregator.aggregate(MetricSampleAggregator.java:213) ~[cruise-control-core-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.monitor.sampling.aggregator.KafkaPartitionMetricSampleAggregator.aggregate(KafkaPartitionMetricSampleAggregator.java:151) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.monitor.LoadMonitor.clusterModel(LoadMonitor.java:496) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.clusterModel(KafkaCruiseControl.java:369) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.ProposalsRunnable.workWithClusterModel(ProposalsRunnable.java:88) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:161) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.workWithoutClusterModel(RebalanceRunnable.java:118) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:173) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.getResult(RebalanceRunnable.java:90) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.getResult(RebalanceRunnable.java:30) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.OperationRunnable.run(OperationRunnable.java:45) ~[cruise-control-2.5.100.jar:?] at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.run(GoalBasedOperationRunnable.java:36) ~[cruise-control-2.5.100.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] ... 1 more 2023-03-17 12:36:58 INFO CruiseControlPublicAccessLogger:62 - 192.168.152.222 - admin [17/Mar/2023:12:36:58 +0000] "POST /kafkacruisecontrol/rebalance?json=true&dryrun=true&verbose=true&skip_hard_goal_check=false&rebalance_disk=false HTTP/1.1" 500 7955I tried recreating KafkaRebalance resource, but the behavior is same in the new resource as well. Any suggestion why this error is coming?
Beta Was this translation helpful? Give feedback.
All reactions