Refactor deploying image logic #346
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ing and function registration - Changed `_copy_image_to_region` to `_copy_image_if_not_exists` and modified logic - Added logic to skip image copying if it already exists in the current region. - Updated Docker image naming to use `workflow_instance_id`, so every lambda function uses the same docker image. - Introduced `set_wrapped_function` method in `CaribouFunction` to register wrapped functions. - Enhanced `CaribouWorkflow` to pass `successor_function_name` in various methods for better tracking. - Added a new `generic_handler.py` to handle dynamic function routing in Lambda. - Updated deployment packager to include the new generic handler. - Adjusted tests to reflect changes in image copying logic.
|
@Haivilo for the sake of documentation, can you provide details on performance gains as a result of this change in a comment? @Danidite and @vGsteiger can you engage with Steve to fix rough edges and bring the new capability to the project? Thanks |
|
Hi @engshahrad, the build time improves as the number of functions increase, which reaches ~50% reduction when you have 5 workflow functions. This was tested locally. Here is the improvement of example build time: For the performance related to migrating speed, I will discuss with @Danidite and get back to you. I was unable to retrieve the results efficiently. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TODO
Fix integration tests
Summary of Changes
Image Mapping Changed from Per-Function to Per-Workflow
workflow_instance_idinstead of per function.Introduced Generic Lambda Handler
generic_handler.py, a dynamic entrypoint that routes invocations based on thetargetfield in the event payload.Payload Format Enhancement
"target"field indicating the function name to invoke, to support the generic Lambda handler.Minor Enhancements
workflow_instance_idinstead of full function name._copy_image_to_regionwas renamed to_copy_image_if_not_existsand now avoids redundant ECR operations when the image exists in the same region.