Skip to content

fixes #7743: stop overwriting workflow parentId with Spark variable name - #7746

Merged
mattcasters merged 1 commit into
apache:mainfrom
mattcasters:issue-7743
Aug 2, 2026
Merged

fixes #7743: stop overwriting workflow parentId with Spark variable name#7746
mattcasters merged 1 commit into
apache:mainfrom
mattcasters:issue-7743

Conversation

@mattcasters

Copy link
Copy Markdown
Contributor

Summary

Fixes #7743 (thanks @nadment for the report and repro).

With Caching Database execution info location, workflow rows were stored with:

parent_id = Internal.Spark.TransformOwnerId

instead of:

  • null for a top-level parent workflow
  • the parent workflow’s log-channel id for a nested child workflow

Root cause

This is not a bug in the Caching Database location. The denormalized parent_id column only made a registration bug obvious.

In LocalWorkflowEngine (from the Native Spark engine work), parent rebind used:

resolve("Internal.Spark.TransformOwnerId")

IVariables.resolve() only expands ${...} / %%...%% tokens. A bare name is returned unchanged, so every local workflow execution overwrote the correct parent id with that literal string.

Spark is supposed to set the real owner id via HopMapPartitionsFn when a workflow runs nested under a mapPartitions Workflow Executor. Rebind should only run when that variable is actually set.

Fix

  • Look up with getVariable(...) (returns null when unset) instead of resolve(...)
  • Centralize the variable name as VAR_SPARK_TRANSFORM_OWNER_ID (kept in sync with SparkConst in the spark plugin; engine cannot depend on that plugin)
  • Unit tests for unset / set / empty variable, and documenting the resolve vs getVariable distinction

Expected after fix

Scenario Before After
Top-level workflow parent_id = Internal.Spark.TransformOwnerId null
Child workflow (local) same literal parent log-channel id
Nested under Spark Workflow Executor same literal synthetic pipeId|name|copy

Test plan

  • LocalWorkflowEngineSparkParentRebindTest (7 tests)
  • Repro from [Bug]: Bad PARENT_ID when using DatabaseExecutionInfoLocation with Hop-Gui #7743 (workflows.zip): parent → child local workflows, Caching Database execution info location, hop-gui
    • Parent row: parent_id null/empty
    • Child row: parent_id = parent’s id
  • Optional: nested workflow under Spark Workflow Executor still parents to synthetic owner id

…able name

LocalWorkflowEngine rebind used resolve("Internal.Spark.TransformOwnerId"),
which returns the bare name unchanged when no ${} tokens are present, so every
local workflow got that literal as PARENT_ID. Use getVariable() so rebind only
applies when Spark actually set the owner id.
@nadment

nadment commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

I haven't encountered “Internal.Spark.TransformOwnerId” since that patch.

@mattcasters
mattcasters merged commit 3e53c72 into apache:main Aug 2, 2026
3 checks passed
jefersonminatozaki pushed a commit to jefersonminatozaki/hop that referenced this pull request Aug 14, 2026
…able name (apache#7746)

LocalWorkflowEngine rebind used resolve("Internal.Spark.TransformOwnerId"),
which returns the bare name unchanged when no ${} tokens are present, so every
local workflow got that literal as PARENT_ID. Use getVariable() so rebind only
applies when Spark actually set the owner id.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Bad PARENT_ID when using DatabaseExecutionInfoLocation with Hop-Gui

2 participants