Skip to content

UPSTREAM: <carry>: reconcile task states with run state on cancellation/failure - #397

Open
jira-autofix[bot] wants to merge 1 commit into
masterfrom
autofix/rhoaieng-74717
Open

UPSTREAM: <carry>: reconcile task states with run state on cancellation/failure#397
jira-autofix[bot] wants to merge 1 commit into
masterfrom
autofix/rhoaieng-74717

Conversation

@jira-autofix

@jira-autofix jira-autofix Bot commented Jul 6, 2026

Copy link
Copy Markdown

Description of your changes:

When a pipeline run is canceled or fails, the RHOAI UI was showing individual tasks as still "progressing" even though backend pods had errored out. This fix reconciles individual task display states with the run-level state.

Changes:

  • Added isRunTerminated() and reconcileExecutionState() helpers in DynamicFlow.ts that override RUNNING/NEW execution states to CANCELED/FAILED when the run itself is terminated
  • Applied reconciliation to all 3 node-update paths: parallel-for iterations, execution nodes, and sub-dag nodes
  • Passed run.state from RunDetailsV2.tsx into updateFlowElementsState() and added it to the memo dependency array
  • Added 4 unit tests covering RUNNING->CANCELED, RUNNING->FAILED, NEW->CANCELED, and the no-override path when the run is still active

Fixes: RHOAIENG-74717

Checklist:

  • You have signed off your commits
  • The title for your pull request (PR) follows the title convention
  • Make sure the changes work with DSPO

@openshift-ci
openshift-ci Bot requested review from VaniHaripriya and gmfrasca July 6, 2026 09:46
@openshift-ci

openshift-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign dsp-developers for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@dsp-developers

Copy link
Copy Markdown

Commit Checker results:

**NOTE**: These are the results of the commit checker scans. 
If these are not commits from upstream kfp, then please ensure
you adhere to the commit checker formatting
commitchecker verson unknown
Validating 1 commits between fdfb9ac4c79b107ad35b63783f384a1a3f2da31f...dda95204c39640037de7563846c1c138d59d0ac8

UPSTREAM commit dda9520 has invalid summary fix(frontend): reconcile task states with run state on cancellation/failure.

UPSTREAM commits are validated against the following regular expression:
  ^UPSTREAM: (revert: )?(([\w.-]+/[\w-.-]+)?: )?(\d+:|<carry>:|<drop>:)

UPSTREAM commit summaries should look like:

  UPSTREAM: <PR number|carry|drop>: description

UPSTREAM commits which revert previous UPSTREAM commits should look like:

  UPSTREAM: revert: <normal upstream format>

Examples of valid summaries:

  UPSTREAM: 12345: A kube fix
  UPSTREAM: <carry>: A carried kube change
  UPSTREAM: <drop>: A dropped kube change
  UPSTREAM: revert: 12345: A kube revert


@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

Security note (CWE-693): This change alters client-side state derivation from run status; no dependency, CI/CD, IDE config, or .gitattributes files are changed.

updateFlowElementsState now accepts an optional runState and reconciles RUNNING/NEW element states to FAILED/CANCELED when the run is terminated. RunDetailsV2 passes run.state into that call and includes it in the memo dependencies. The test suite adds cases for terminated-run overrides and the no-override path.

Estimated code review effort: 2 (Simple) | ~15 minutes

Changes

File Change
frontend/src/lib/v2/DynamicFlow.ts Adds runState, termination helpers, and state reconciliation in three node-update paths
frontend/src/pages/RunDetailsV2.tsx Passes run.state to updateFlowElementsState and updates memo dependencies
frontend/src/lib/v2/DynamicFlow.test.ts Adds tests for override and no-override behavior

Related issues: None provided.

Related PRs: None provided.

Suggested labels: frontend, bug-fix

Suggested reviewers: None provided.

Poem:
Run state arrived, the nodes complied,
RUNNING shifted where the rules applied.
CWE-693 kept the checks in view,
No hidden path, just state made true.

🚥 Pre-merge checks | ✅ 10
✅ Passed checks (10 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Contribution Quality And Spam Detection ✅ Passed Only one mild templated-content signal; the PR has a Jira ref and 4 unit tests covering the state override, with no second-category spam/security signal.
No Hardcoded Secrets ✅ Passed No hardcoded secrets found in the touched TS/TSX files: no api_key/secret/token/password literals, private keys, embedded creds, or long base64 config strings.
No Weak Cryptography ✅ Passed No banned primitives, roll-your-own crypto, or timing-unsafe secret comparisons in the touched frontend files (CWE-327, CWE-208).
No Injection Vectors ✅ Passed No CWE-89/78/94/502/79 sink in the touched code; changes only reconcile MLMD state and pass run.state, and tests use yaml.safeLoad.
No Privileged Containers ✅ Passed Diff touches only frontend TS/TSX; no Kubernetes/OpenShift manifests, Helm templates, or Dockerfiles were added with privileged settings (CWE-250/CWE-284).
No Sensitive Data In Logs ✅ Passed No sensitive-data logging added; touched code only adds run-state reconciliation/tests, and existing console.warns expose only non-secret IDs/messages (CWE-532 not triggered).
Title check ✅ Passed The title is specific and directly matches the main change: reconciling task states with run state on cancellation or failure.
Description check ✅ Passed The description covers the change summary, issue link, and checklist; only the DSPO testing item remains unchecked.

Comment @coderabbitai help to get the list of available commands.

@dsp-developers

Copy link
Copy Markdown

A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-397
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-397
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-397
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-397
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-397
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-397
TESTS: quay.io/opendatahub/ds-pipelines-tests:pr-397

@dsp-developers

Copy link
Copy Markdown

An OCP cluster where you are logged in as cluster admin is required.

The Data Science Pipelines team recommends testing this using the Data Science Pipelines Operator. Check here for more information on using the DSPO.

To use and deploy a DSP stack with these images (assuming the DSPO is deployed), first save the following YAML to a file named dspa.pr-397.yaml:

apiVersion: datasciencepipelinesapplications.opendatahub.io/v1
kind: DataSciencePipelinesApplication
metadata:
  name: pr-397
spec:
  dspVersion: v2
  apiServer:
    image: "quay.io/opendatahub/ds-pipelines-api-server:pr-397"
    argoDriverImage: "quay.io/opendatahub/ds-pipelines-driver:pr-397"
    argoLauncherImage: "quay.io/opendatahub/ds-pipelines-launcher:pr-397"
  persistenceAgent:
    image: "quay.io/opendatahub/ds-pipelines-persistenceagent:pr-397"
  scheduledWorkflow:
    image: "quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-397"
  mlmd:  
    deploy: true  # Optional component
    grpc:
      image: "quay.io/opendatahub/mlmd-grpc-server:latest"
    envoy:
      image: "registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2"
  mlpipelineUI:
    deploy: true  # Optional component 
    image: "quay.io/opendatahub/ds-pipelines-frontend:pr-397"
  objectStorage:
    minio:
      deploy: true
      image: 'quay.io/opendatahub/minio:RELEASE.2019-08-14T20-37-41Z-license-compliance'

Then run the following:

cd $(mktemp -d)
git clone git@github.com:opendatahub-io/data-science-pipelines.git
cd data-science-pipelines/
git fetch origin pull/397/head
git checkout -b pullrequest dda95204c39640037de7563846c1c138d59d0ac8
oc apply -f dspa.pr-397.yaml

More instructions here on how to deploy and test a Data Science Pipelines Application.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
frontend/src/lib/v2/DynamicFlow.test.ts (1)

28-28: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Test coverage gap: parallel-for and sub-DAG branches untested.

All four new tests exercise only the NodeTypeNames.EXECUTION branch (DynamicFlow.ts lines 298-301). Neither the parallel-for sub-DAG branch (lines 280-283) nor the NodeTypeNames.SUB_DAG branch (lines 332-335) — both of which also had getLastKnownState() replaced with reconcileExecutionState(...) — are covered by a run-state test case.

Also applies to: 147-264

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@frontend/src/lib/v2/DynamicFlow.test.ts` at line 28, Add run-state test
coverage in DynamicFlow.test for the branches in DynamicFlow that handle
parallel-for sub-DAG and NodeTypeNames.SUB_DAG, not just
NodeTypeNames.EXECUTION. Update or add cases around the DynamicFlow logic that
now calls reconcileExecutionState(...) in those branches, and assert the
resulting state behavior from getRunState()/run-state handling using the
existing V2beta1RuntimeState test setup.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@frontend/src/lib/v2/DynamicFlow.test.ts`:
- Line 28: Add run-state test coverage in DynamicFlow.test for the branches in
DynamicFlow that handle parallel-for sub-DAG and NodeTypeNames.SUB_DAG, not just
NodeTypeNames.EXECUTION. Update or add cases around the DynamicFlow logic that
now calls reconcileExecutionState(...) in those branches, and assert the
resulting state behavior from getRunState()/run-state handling using the
existing V2beta1RuntimeState test setup.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 856c5cbe-9965-4d45-859a-b7056ed9c0d2

📥 Commits

Reviewing files that changed from the base of the PR and between fdfb9ac and dda9520.

📒 Files selected for processing (3)
  • frontend/src/lib/v2/DynamicFlow.test.ts
  • frontend/src/lib/v2/DynamicFlow.ts
  • frontend/src/pages/RunDetailsV2.tsx

…on/failure

When a pipeline run is cancelled or fails, individual tasks that are still
in RUNNING or NEW state in MLMD are now overridden to show CANCELED or
FAILED in the UI. Previously, updateFlowElementsState only read MLMD
execution states without considering the run-level state, so cancelled runs
would still display in-progress tasks as running.

**Description of your changes:**

- Add runState parameter to updateFlowElementsState() in DynamicFlow.ts
- Add reconcileExecutionState() helper that overrides RUNNING/NEW
  execution states to CANCELED or FAILED when the run is terminated
- Pass run.state from RunDetailsV2.tsx into the flow state update
- Add tests for RUNNING→CANCELED, RUNNING→FAILED, NEW→CANCELED overrides,
  and verify no override when run is still RUNNING

**Checklist:**
- [x] The title for your pull request (PR) follows the title convention
- [x] Make sure the changes work with DSPO

Ref: RHOAIENG-74717

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
@jira-autofix jira-autofix Bot changed the title fix(frontend): reconcile task states with run state on cancellation/failure UPSTREAM: <carry>: reconcile task states with run state on cancellation/failure Jul 6, 2026
@jira-autofix
jira-autofix Bot force-pushed the autofix/rhoaieng-74717 branch from dda9520 to 4649ac1 Compare July 6, 2026 12:18
@dsp-developers

Copy link
Copy Markdown

Commit Checker results:

**NOTE**: These are the results of the commit checker scans. 
If these are not commits from upstream kfp, then please ensure
you adhere to the commit checker formatting
commitchecker verson unknown
Validating 1 commits between fdfb9ac4c79b107ad35b63783f384a1a3f2da31f...4649ac184546aef94111e46b26937d61270aed8b

@dsp-developers

Copy link
Copy Markdown

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-397
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-397
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-397
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-397
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-397
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-397
TESTS: quay.io/opendatahub/ds-pipelines-tests:pr-397

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant