fixes #7743: stop overwriting workflow parentId with Spark variable name - #7746
Merged
Conversation
…able name
LocalWorkflowEngine rebind used resolve("Internal.Spark.TransformOwnerId"),
which returns the bare name unchanged when no ${} tokens are present, so every
local workflow got that literal as PARENT_ID. Use getVariable() so rebind only
applies when Spark actually set the owner id.
Contributor
|
I haven't encountered “Internal.Spark.TransformOwnerId” since that patch. |
jefersonminatozaki
pushed a commit
to jefersonminatozaki/hop
that referenced
this pull request
Aug 14, 2026
…able name (apache#7746) LocalWorkflowEngine rebind used resolve("Internal.Spark.TransformOwnerId"), which returns the bare name unchanged when no ${} tokens are present, so every local workflow got that literal as PARENT_ID. Use getVariable() so rebind only applies when Spark actually set the owner id.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #7743 (thanks @nadment for the report and repro).
With Caching Database execution info location, workflow rows were stored with:
parent_id = Internal.Spark.TransformOwnerIdinstead of:
nullfor a top-level parent workflowRoot cause
This is not a bug in the Caching Database location. The denormalized
parent_idcolumn only made a registration bug obvious.In
LocalWorkflowEngine(from the Native Spark engine work), parent rebind used:IVariables.resolve()only expands${...}/%%...%%tokens. A bare name is returned unchanged, so every local workflow execution overwrote the correct parent id with that literal string.Spark is supposed to set the real owner id via
HopMapPartitionsFnwhen a workflow runs nested under a mapPartitions Workflow Executor. Rebind should only run when that variable is actually set.Fix
getVariable(...)(returnsnullwhen unset) instead ofresolve(...)VAR_SPARK_TRANSFORM_OWNER_ID(kept in sync withSparkConstin the spark plugin; engine cannot depend on that plugin)resolvevsgetVariabledistinctionExpected after fix
parent_id = Internal.Spark.TransformOwnerIdnullpipeId|name|copyTest plan
LocalWorkflowEngineSparkParentRebindTest(7 tests)workflows.zip): parent → child local workflows, Caching Database execution info location, hop-guiparent_idnull/emptyparent_id= parent’sid