[FLINK-33634] Add Conditions to Flink CRD's Status field #957

lajith2006 · 2025-03-20T05:15:17Z

What is the purpose of the change

This PR address the FLIP to add conditions to flink deployment status related to issue https://issues.apache.org/jira/browse/FLINK-33634

Brief change log

Added list of conditions to FlinkDeploymentStatus
A new class ConditionUtils introduced as utility class to build condition.

Verifying this change

Deployed and validated session job and application custer deployment.
Validated the status transitions.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changes to the CustomResourceDescriptors: yes
Core observer or reconciler logic that is regularly executed: (yes / no)

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

davidradl · 2025-03-21T15:51:36Z

@lajith2006 please could you fill in a descriptive title next to the Jira number

davidradl

removed comment

davidradl · 2025-03-21T15:56:23Z

@lajith2006 this implementation does not match the Flip words and has not been discussed in the slack thread, please can you update the discussion thread to ensure that you get consensus, and confirm that the Flip vote still is valid after this change.

davidradl · 2025-03-21T16:01:34Z

...api/src/main/java/org/apache/flink/kubernetes/operator/api/status/FlinkDeploymentStatus.java

+        } else if (getJobStatus() != null && getJobStatus().getState() != null) {
+            switch (getJobStatus().getState()) {
+                case RECONCILING:
+                    phase = "Pending";


why is this one not RECONCILING like the pattern the others follow others? I suggest a comment, also a constant is better then a an inline literal.

Right, as mentioned here , phase which is intended to keep the current status of FlinkDeployment especially useful in Openshift environment, as Openshift UI can render the value from status.phase and populate the current status of deployment.

As when FlinkDeployment applied in Application Mode, in the initial phase, as the JM brining up , job state would be in RECONCILING state, so status.phase is kept as "Pending".

davidradl · 2025-03-21T16:04:20Z

...api/src/main/java/org/apache/flink/kubernetes/operator/api/status/FlinkDeploymentStatus.java

+                    updateCondition(
+                            conditions,
+                            ConditionUtils.runningTrue(
+                                    "JobManager is running and ready to receive REST API call",


nit: call -> calls

davidradl · 2025-03-21T16:04:31Z

...api/src/main/java/org/apache/flink/kubernetes/operator/api/status/FlinkDeploymentStatus.java

+        if (reconciliationStatus != null
+                && reconciliationStatus.deserializeLastReconciledSpec() != null
+                && reconciliationStatus.deserializeLastReconciledSpec().getJob() == null) {
+            switch (jobManagerDeploymentStatus) {


can we have a datastructure, keyed off the jobManagerDeploymentStatus, that we interrogate to get the inserts for the updateConditions. Maybe a map with the key of the status and the value of an object RunningCondition, that has 2 fields the boolean and the description. Something like :

Map<JobManagerDeploymentStatus,String> jobmanagerDeploymentStatusMap = new HashMap<String,String>() {{ put(READY, new RunningCondition(true, "JobManager is running and ready to receive REST API calls")), .... }};

then the code just loops through the map updating conditions using the map values. Similar for jobstatus

Wondering will that add any value?. Any way to build the Map , we have to call ConditionUtils to build the Condition, so rather than have a Prebuild Map , we can directly call ConditionUtils to build them right. Your thoughts?.

I think you have pre built the map now - with map.of - looks good - thanks.

davidradl · 2025-03-21T16:11:59Z

...api/src/main/java/org/apache/flink/kubernetes/operator/api/status/FlinkDeploymentStatus.java

+                            conditions,
+                            ConditionUtils.runningFalse(
+                                    JobStatus.RESTARTING.name(),
+                                    "The job is currently undergoing a restarting"));


nit: "The job is currently undergoing a restarting" -> "The job is currently restarting"

davidradl · 2025-03-21T16:13:27Z

...api/src/main/java/org/apache/flink/kubernetes/operator/api/status/FlinkDeploymentStatus.java

+    private String phase;
+
+    public List<Condition> getConditions() {
+        if (reconciliationStatus != null


I am curious which parts of the code refer to application mode and which are session cluster. It would be good to have some comments to detail this and maybe use the mode name in method name or variable names to make this more intuitive to read.

I will add comments in the respective code.

davidradl · 2025-03-21T16:24:12Z

...api/src/main/java/org/apache/flink/kubernetes/operator/api/status/FlinkDeploymentStatus.java

+                            conditions,
+                            ConditionUtils.runningFalse(
+                                    "JobManager process is starting up",
+                                    "JobManager process is starting up"));


I am curious why the message and reaon is the same . I would expect them to be different. In this case I would expect the reason to be "A New JVM deployment exists and is being created" - i.e. the word on the arrow in the UML.

As per description here which says Reason is intended to be a one-word, CamelCase representation of the category of cause of the current status, and Message is intended to be a human-readable phrase or sentence, I would use the existing enum https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/status/JobManagerDeploymentStatus.java, to use as reason and respective explanation as message

I see https://github.com/apache/flink-kubernetes-operator/pull/961/files, gives examples of camelcase single words reasons. We should follow that style for the cases that PR does not cover.

Done a refactoring for the conditions.

davidradl · 2025-03-21T16:26:48Z

...test/java/org/apache/flink/kubernetes/operator/controller/FlinkDeploymentControllerTest.java

@@ -448,6 +465,15 @@ public void verifyUpgradeFromSavepointLegacyMode(FlinkVersion flinkVersion) thro
        assertEquals(
                "savepoint_1", appCluster.getStatus().getJobStatus().getUpgradeSavepointPath());

+        // Validate status conditions


can we use a java parameterized test (or more than one as appropriate) to cover all the permutations of the tests.

davidradl · 2025-03-21T16:30:57Z

...perator-api/src/main/java/org/apache/flink/kubernetes/operator/api/utils/ConditionUtils.java

+import java.util.Date;
+
+/**
+ * Creates a condition object with the specified parameters.


nit: with the specified parameters -> whether this is running , the message and the reason.
then for return just say @return A condition

davidradl · 2025-03-21T16:32:00Z

...perator-api/src/main/java/org/apache/flink/kubernetes/operator/api/utils/ConditionUtils.java

+ */
+public class ConditionUtils {
+    public static Condition runningTrue(final String message, final String reason) {
+        return crCondition("Running", "True", message, reason);


no need to pass "Running" as a parameter, we can hard code that in the method,

nit: rename crCondition -> crRunningCondition

I would like to keep method name as crCondition rather crRunningCondition , just because , crCondition will keep it as more generic , so that if in case, if we need to address any other condition with type other than Running , we just need to pass parameter to build respective condition.

I suggest doing that refactor when we need to, in the spirit of keeping the code lean.

davidradl · 2025-03-21T16:32:39Z

...perator-api/src/main/java/org/apache/flink/kubernetes/operator/api/utils/ConditionUtils.java

+    }
+
+    public static Condition runningFalse(final String message, final String reason) {
+        return crCondition("Running", "False", message, reason);


are there constants / enums for "False" and "True" we can use?
Os there an existing constant for "Running" we can use from OCP or the like. If not we should define this as a constant. We use this literal a few times in this change.

I haven't see any existing enum/constants for "False" / "True , instead looks like used as string itself in wherever required in existing code.

lajith2006 · 2025-04-01T07:38:53Z

As mentioned in the FLIP , we will have status.conditions, which holds the list of conditions that contains the status of Job transitions in each stage.

When the FlinkOperator is installed in Openshift and manages FlinkDeployment , with status.conditions in the CR , FlinkDeployment status in the Openshift UI will provide some misleading information.

Openshift UI works in such a way that, if there are multiple conditions , and at any point one of the condition had a flag of status with true and type as Running , irrespective of latest condition status has a flag as of false and type Running , In Openshift UI , status says that condition of deployment/job is Running. Which is misleading information, if you consider the scenario of Job was running and then later point job was suspended.

For example as below when a running Job is stopped at some point , we will have a condition as below.

conditions:
  - lastTransitionTime: '2025-04-01T06:17:08Z'
    message: Reconciling
    reason: Job is currently reconciling
    status: 'False'
    type: Running
  - lastTransitionTime: '2025-04-01T07:07:23Z'
    message: JobRunning
    reason: Job is running
    status: 'True'
    type: Running
  - lastTransitionTime: '2025-04-01T07:28:37Z'
    message: JobFinished
    reason: Job's tasks have successfully finished
    status: 'False'
    type: Running

and Openshift UI will have status as

So to overcome those problems, we will have a status.phase(same as other k8s resources like pod) , which will hold the current status of deployment/job. Openshift UI will take precedence of status.phase over status.condition to display the status of FlinkDeployment resource.

csviri · 2025-04-01T12:32:19Z

Would it be useful also to set also observedGeneration on the conditions, I believe it is in general a good practice.

csviri · 2025-04-01T12:51:16Z

...api/src/main/java/org/apache/flink/kubernetes/operator/api/status/FlinkDeploymentStatus.java

+                && existingCondition.getMessage().equals(condition.getMessage())) {
+            return;
+        }
+        conditions.add(condition);


It's maybe just me, just don't get how this list is truncated eventually, in other words, how is it prevented from growing indefinitely?

Thanks for review @csviri . Right now it's not preventing from growing indefinitely. As conditions reflects the different transitions state of Job/Deployment , thinking what could be the use case where can grow indefinitely?.

I think there should only ever be a single condition of type Running in the list that we return. Since we only have a single condition type right now, then the list should only have a single element. The latestTransition timestamp needs to represent when running changed from true->false or false->true. We can however keep updating the message if we want.

Thanks @gyfora for review , If I read correctly , you meant that we need to have only condition in the list at right now as we have only one type Running instead of multiple conditions of same type Running in the list as currently this PR having. And the lastTransitionTime in the condition must represent when the Running type changed its status from true>false or false > true. Is that correct?.

@lajith2006 you say the lastTransitionTime in the condition must represent when the Running type changed its status from true>false or false > true. Is that correct?. So I am curious what happens in the history if we change the reason Text? Can you check that if we change the reason text and not the running flag, we see all of the entries in the history if all the historical conditions have the same lastTransitionTime with changing reasons.

davidradl · 2025-04-10T10:41:44Z

...api/src/main/java/org/apache/flink/kubernetes/operator/api/status/FlinkDeploymentStatus.java

+                && reconciliationStatus.deserializeLastReconciledSpec().getJob() == null) {
+            // Populate conditions for SessionMode deployment
+            switch (jobManagerDeploymentStatus) {
+                case READY:


instead of this switch can we just issue

updateCondition( conditions, ConditionUtils.crCondition( ConditionUtils.SESSION_MODE_CONDITION.get( jobManagerDeploymentStatus.name())));

davidradl · 2025-04-10T10:43:03Z

...api/src/main/java/org/apache/flink/kubernetes/operator/api/status/FlinkDeploymentStatus.java

+            }
+        } else if (getJobStatus() != null && getJobStatus().getState() != null) {
+            // Populate conditions for ApplicationMode deployment
+            switch (getJobStatus().getState()) {


can this switch as-is be replaced with

updateCondition( conditions, ConditionUtils.crCondition( ConditionUtils.APPLICATION_MODE_CONDITION.get( getJobStatus().getState().name())));

davidradl · 2025-04-10T10:45:35Z

...perator-api/src/main/java/org/apache/flink/kubernetes/operator/api/utils/ConditionUtils.java

+                            .withType("Running")
+                            .withStatus("True")
+                            .withMessage("Ready")
+                            .withReason("JobManager is running and ready to receive REST API calls")


isn't the reason - how it got to this state, not a description of the state itself?

lajith2006 added 4 commits March 12, 2025 14:49

[FLINK-33634] Add Conditions to Flink CRD's Status field

afdf40b

[FLINK-33634] Add Conditions to Flink CRD's Status field

1d84164

[FLINK-33634] Add Conditions to Flink CRD's Status field

727bb4b

[FLINK-33634] Add Conditions to Flink CRD's Status field

c66fd1f

davidradl suggested changes Mar 21, 2025

View reviewed changes

davidradl reviewed Mar 21, 2025

View reviewed changes

gyfora changed the title ~~FLINK-33634~~ [FLINK-33634] Add Conditions to Flink CRD's Status field Mar 23, 2025

lajith2006 added 3 commits March 26, 2025 17:09

[FLINK-33634] Add Conditions to Flink CRD's Status field

c623f33

[FLINK-33634] Add Conditions to Flink CRD's Status field

196c494

[FLINK-33634] Add Conditions to Flink CRD's Status field

32294f4

csviri reviewed Apr 1, 2025

View reviewed changes

davidradl reviewed Apr 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-33634] Add Conditions to Flink CRD's Status field #957

[FLINK-33634] Add Conditions to Flink CRD's Status field #957

lajith2006 commented Mar 20, 2025 •

edited

Loading

davidradl commented Mar 21, 2025

davidradl left a comment •

edited

Loading

davidradl commented Mar 21, 2025

davidradl Mar 21, 2025

lajith2006 Mar 26, 2025

davidradl Mar 21, 2025

davidradl Mar 21, 2025 •

edited

Loading

lajith2006 Mar 26, 2025

davidradl Apr 10, 2025

davidradl Mar 21, 2025

davidradl Mar 21, 2025

lajith2006 Mar 26, 2025

davidradl Mar 21, 2025

lajith2006 Mar 26, 2025 •

edited

Loading

davidradl Mar 26, 2025

lajith2006 Mar 28, 2025

davidradl Mar 21, 2025

davidradl Mar 21, 2025

davidradl Mar 21, 2025

lajith2006 Mar 26, 2025 •

edited

Loading

davidradl Mar 26, 2025

davidradl Mar 21, 2025 •

edited

Loading

lajith2006 Mar 26, 2025

lajith2006 commented Apr 1, 2025

csviri commented Apr 1, 2025

csviri Apr 1, 2025

lajith2006 Apr 3, 2025

gyfora Apr 5, 2025

lajith2006 Apr 10, 2025 •

edited

Loading

davidradl Apr 10, 2025

davidradl Apr 10, 2025 •

edited

Loading

davidradl Apr 10, 2025 •

edited

Loading

davidradl Apr 10, 2025

[FLINK-33634] Add Conditions to Flink CRD's Status field #957

Are you sure you want to change the base?

[FLINK-33634] Add Conditions to Flink CRD's Status field #957

Conversation

lajith2006 commented Mar 20, 2025 • edited Loading

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

davidradl commented Mar 21, 2025

davidradl left a comment • edited Loading

Choose a reason for hiding this comment

davidradl commented Mar 21, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidradl Mar 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lajith2006 Mar 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lajith2006 Mar 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidradl Mar 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lajith2006 commented Apr 1, 2025

csviri commented Apr 1, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lajith2006 Apr 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidradl Apr 10, 2025 • edited Loading

Choose a reason for hiding this comment

davidradl Apr 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lajith2006 commented Mar 20, 2025 •

edited

Loading

davidradl left a comment •

edited

Loading

davidradl Mar 21, 2025 •

edited

Loading

lajith2006 Mar 26, 2025 •

edited

Loading

lajith2006 Mar 26, 2025 •

edited

Loading

davidradl Mar 21, 2025 •

edited

Loading

lajith2006 Apr 10, 2025 •

edited

Loading

davidradl Apr 10, 2025 •

edited

Loading

davidradl Apr 10, 2025 •

edited

Loading