[FSU] Inference FSU with Shared memory #2969
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DonghakPark
requested review from
EunjuYang,
SeoHyungjun,
again4you,
anyj0527,
baek2sm,
djeong20,
dkjung,
gichan-jang,
haehun,
jaeyun-jung,
jihochu,
jijoongmoon,
leemgs,
lhs8928,
myungjoo,
skykongkong8,
songgot and
wooksong
as code owners
This was referenced Feb 25, 2025
DonghakPark
force-pushed
the
FSU_with_shared_Mem
branch
3 times, most recently
from
ec044b7 to
d328785
Compare
DonghakPark
force-pushed
the
FSU_with_shared_Mem
branch
3 times, most recently
from
e41af8c to
fb045da
Compare
SeoHyungjun
reviewed
Mar 4, 2025
haehun
reviewed
Mar 4, 2025
| model_graph.LoadTensors(f); | ||
| model_graph.checkLoadComplete(f); | ||
| node->forwarding(training); | ||
| model_graph.UnloadTensors(f); |
There was a problem hiding this comment.
Yes, no longer need unload --> at unload logic there are 2 work : 1. dealloc, 2. write to file
- we use shared memory that allocated fixed amount of mem and reuse, so the allocated memory will dealloc at end of inference
- inference workload : no need to write at file
Has the formatting style changed? --> No! Sorry for confuse there some conflicts on my local machine |
DonghakPark
force-pushed
the
FSU_with_shared_Mem
branch
from
fb045da to
ab147da
Compare
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
Update FSU forwarding logic - FSU will handle look ahead tensor inside of pool - so we don't need to call Loadtensor for f + i **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
Add memory ptr for allocate shared mem - add mem_ptr - add unmap - array for manage unmapped ptr **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
I have changed the method from using dynamic memory allocation to using static memory allocation. In order to prevent multiple frees, I added a map to check whether the mem_address has already been processed. Previously, memory was allocated through buf, but now it is being allocated directly. **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Co-authored-by: jijoong.moon <[email protected]> Signed-off-by: Donghak PARK <[email protected]>
make neuralnet can pass path to the swap_device & weight offset (file offset) it can make calculate weight file's offset **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Co-authored-by: hyeonseok <[email protected]> Signed-off-by: Donghak PARK <[email protected]>
Apply Shared mem & FSU - when inference mode : read from weight bin ( weight offset ) - when train mode : same logic with swap **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
Fix Unittest Fail bug at Training Case Swap - There are some issue on PutBuffer that can not free ptr **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
Apply clang format at changed File **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
DonghakPark
force-pushed
the
FSU_with_shared_Mem
branch
from
ab147da to
ef465db
Compare
Update FSU Unitter - For now, we should set our weight & input size as pagesize * N - For later i will add Page Align Algorithm **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
DonghakPark
force-pushed
the
FSU_with_shared_Mem
branch
from
ef465db to
69b8783
Compare
|
This PR include at #3004. --> close |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[FSU] Inference FSU with Shared memory
To reduce memory usage during inference by utilizing FSU, and to minimize speed degradation by performing loading during forwarding, changed to use shared memory. and ensure the existing swap in training mode is also performed normally.
Commit 1 : [FSU] Update FSU Forwarding (Load) Logis
Commit 2 : [FSU] Update swap device & cache element
Commit 3 : [FSU] Update FSU mem allocate Logic
Commit 4 : [FSU] add FSU file offset info
Commit 5 : [FSU] Apply Shared Mem & FSU
This PR was include #2957 #2927 #2949, So i will close previous PRs