File transfer and sublattice enhancements #1981
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 tasks
cjao
force-pushed
the
file-transfer-enhancements
branch
6 times, most recently
from
19905ce to
73e9ef5
Compare
*** File Transfers ***
Improve automatic file transfer strategy selection by automatically
handling the following cases:
- Local-Remote
- Remote-Local
- Local-Local
Allow registering new file transfer strategies
Enhance HTTP strategy to support uploads
Centralize and abstract file transfers in SDK. Dispatch submission and
retrieval now use FileTransfers to upload and download assets
Compute Result.error client side to avoid needless server-side writes
to object store
*** Sublattices ***
Add endpoint to link sublattice with parent. Link sublattice dispatch
with parent electron in `_build_sublattice_graph`. The dedicated
sublattice registration endpoint `POST
/dispatches/{dispatch_id}/sublattices` is no longer used.
Adjust server-side handling of sublattice dispatches in `update_node_result`
Introduce new sublattice tarball importer:
- Replace json sublattice importer. Future improvements will allow the
server to avoid loading all (serialized) sublattice assets in memory
- Remove obsolete endpoint `POST /dispatches/{dispatch_id}/sublattices`
** Misc improvements **
Sublattices: Only copy db references when copying assets
Sublattice result -> parent electron output
Final electron output -> Dispatch result
Original electron attributes -> redispatched electron attributes
Sublattice: Back up manifest asset record before overwriting it
Cleanup: remember to hold refs to background tasks
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
These changes pave the way for more general object storage backends (e.g. s3)
To upload the output of an electron:
1. Send POST `/dispatches/{dispatch_id}/electrons/{node_id}/assets/output`
2. PUT contents to the returned `remote_uri`. In the future, one can
use a file transfer to handle more general destination URLs.
To download the output of an electron:
1. Send GET `/dispatches/{dispatch_id}/electrons/{node_id}/assets/output`
2. Send GET request `remote_uri`. In the future, one can
use a file transfer to handle more general source URLs.
Cleanup: remove unused asset endpoints
Also demo file transfers in local executor
cjao
force-pushed
the
file-transfer-enhancements
branch
from
73e9ef5 to
821e942
Compare
|
Gentle reminder @wjcunningham7 @FyzHsn : ) |
3 tasks
cjao
force-pushed
the
file-transfer-enhancements
branch
from
821e942 to
6e1144f
Compare
cjao
force-pushed
the
file-transfer-enhancements
branch
from
6e1144f to
ea02034
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce file transfer and sublattice enhancements to decentralize asset handling
File Transfers
Improve automatic file transfer strategy selection by automatically
handling the following cases:
Allow registering new file transfer strategies to handle custom URL schemes
Enhance HTTP strategy to support uploads
Centralize and abstract file transfers in SDK. Dispatch submission and
retrieval now use FileTransfers to upload and download assets
Compute
Result.errorclient side to avoid needless server-side writesto object store
For asset uploads/downloads, decouple the steps of
Sublattices
Add endpoint to link sublattice with parent. Link sublattice dispatch with parent electron in
_build_sublattice_graph. The dedicated sublattice registration endpointPOST /dispatches/{dispatch_id}/sublatticesis no longer used.Adjust server-side handling of sublattice dispatches in
update_node_result. Since the sublatticedispatch_idis now linked to the parent electron during_build_sublattice_graph, the server no longer needs to read the sublattice manifest from object storage if it can see the link in the DB.Improve handling of sublattices built using executors that can't reach the control plane (e.g. AWS Batch -|-> Covalent running on local workstation).
_build_sublattice_graphis unable to submit the sublattice directly, it tars up the staging directory returns the tarball to the control plane.POST /dispatches/{dispatch_id}/sublatticesMisc improvements
Sublattices: Only copy db references when copying assets
Sublattice result -> parent electron output
Final electron output -> Dispatch result
Original electron attributes -> redispatched electron attributes
Sublattice: Back up manifest asset record before overwriting it
Cleanup: remember to hold refs to background tasks
Cleanup: refactor URI filtering
Cleanup: remove unused APIs