[GLUTEN-11406][VL] Update global off-heap memory to reuse the execution memory allocation code path#11456
Draft
zhztheplayer wants to merge 6 commits intoapache:mainfrom
Draft
[GLUTEN-11406][VL] Update global off-heap memory to reuse the execution memory allocation code path#11456zhztheplayer wants to merge 6 commits intoapache:mainfrom
zhztheplayer wants to merge 6 commits intoapache:mainfrom
Conversation
…cation code path For allocating global off-heap memory in Gluten, we have been using the storage-memory-based approach provided by utility GlobalOffHeapMemory (which is renamed and deprecated in this PR). But we noticed that the approach didn't bring us much benefit than directly allocating from off-heap execution memory via a dummy task memory manager. The PR refactors the code to implement the later solution instead. The main reason we do this is for more conveniently reusing the existing tree consumer code in Gluten, which helps memory allocations to account themselves in an account with a specific name and category, which helps user and developer track these allocations, both in production and test.
|
Run Gluten Clickhouse CI on x86 |
zhztheplayer
changed the title
…cation code path For allocating global off-heap memory in Gluten, we have been using the storage-memory-based approach provided by utility GlobalOffHeapMemory (which is renamed and deprecated in this PR). But we noticed that the approach didn't bring us much benefit than directly allocating from off-heap execution memory via a dummy task memory manager. The PR refactors the code to implement the later solution instead. fixup
|
Run Gluten Clickhouse CI on x86 |
zhztheplayer
changed the title
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks. |
|
Run Gluten Clickhouse CI on x86 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For allocating global off-heap memory in Gluten, we have been using the storage-memory-based approach provided by utility
GlobalOffHeapMemory(which is being renamed and deprecated in this PR). However, so far we noticed that the approach didn't bring us much benefit compared to directly allocating from off-heap execution memory via a process-wise dummy task memory manager. The PR refactors the code to rollback to the execution memory solution instead.The main reason we do this is for more conveniently reusing the existing tree consumer code in Gluten, which helps memory allocations to account themselves to Spark with a specific name and category, which helps user and developer track these allocations, both in production and test.
Related issue: #11169
Related issue: #11406