[NOMERGE] Production concatenate calls squashed #6
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…cer#585) * refactor deduplicate_data_wrappers to avoid dependence on erroneous super().rec usage in CachedMapAndCopyMapper Here is a sketch of what happens with super().rec vs Mapper.rec for the previous implementation of deduplicate_data_wrappers. Suppose we have 2 data wrappers a and b with the same data pointer. With super().rec: 1) map_fn maps a to itself, then mapper copies a to a'; mapper caches a -> a' (twice, once in super().rec and then again in rec), 2) map_fn maps b to a, then mapper maps (via cache in super().rec call) a to a'; mapper caches b -> a'. => Only a' in output DAG. With Mapper.rec: 1) map_fn maps a to itself, then mapper copies a to a'; caches a -> a', 2) map_fn maps b to a, then mapper copies a to a''; caches b -> a''. => Both a' and a'' in output DAG. * call Mapper.rec instead of super().rec to avoid double caching * call Mapper.rec from CachedMapper too just to avoid copy/paste errors * add assertion to check for double caching * add comment explaining use of Mapper.rec
…nition_cache_key for extra args case ambiguous due to the fact that any arg can be specified with/without keyword
…n_definition_cache_key are not defined for general extra args/kwargs
…sformMapperWithExtraArgs
apparently TypeVar(..., <type>) doesn't include subclasses of <type>
now inherits from CachedMapper
|
FYI: I haven't been able tor run prediction driver past 128 ranks. I keep getting errors like this one: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summarize concat/outlining changes only.