Use Parsl HTEX submit_payload #1965
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
khk-globus
requested review from
LeiGlobus,
chris-janidlo,
rjmello and
yadudoc
khk-globus
added
the
no-news-is-good-news
Parsl has recently learned how to dynamically import a task-specific `execute_task()` function, which Compute will soon utilize. [sc-35486]
Parsl has learned the ability to dynamically import an `execute_task()`
function. Use this capability to tell Parsl to use the installed endpoint's
`execute_task()`. This enables specification of the allowed serializers, per
task. They are currently dynamically specified per task based on the engine
configuration.
The core change is "small" but with big impact in the tests. The substantive
changes are:
- `base.py` - `submit()` via `task_deserializers`, `args`, and `kwargs`
- `helper.py` - update `execute_task()` to instantiate a `ComputeSerializer()`,
rather than attempting to marshal an instance via the arguments. Relates to
`task_deserializers` in `base.py`
- Update logging to include a task_id prefix to each log record -- hopefully
helpful to users (or us!) when tracking down issues.
- `globus_compute.py` - `_submit()` with `context` and `submit_payload`;
requirements to use Parsl's dynamic import functionality
- `globus_mpi.py` - no need to duplicate parent's `_submit()`
[sc-35486]
khk-globus
force-pushed
the
sc-35486/custom_execute_task
branch
from
f3a140c to
7eada59
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Parsl has learned the ability to dynamically import an
execute_task()function. Use this capability to tell Parsl to use the installed endpoint'sexecute_task(). This enables specification of the allowed serializers, per task. They are currently dynamically specified per task based on the engine configuration.The core change is "small" but with big impact in the tests. The substantive changes are:
base.py-submit()viatask_deserializers,args, andkwargshelper.py- updateexecute_task()to instantiate aComputeSerializer(),rather than attempting to marshal an instance via the arguments. Relates to
task_deserializersinbase.pyhelpful to users (or us!) when tracking down issues.
globus_compute.py-_submit()withcontextandsubmit_payload;requirements to use Parsl's dynamic import functionality
globus_mpi.py- no need to duplicate parent's_submit()[sc-35486]
Type of change