fix: various bugs to eliminate crashes when SGLang overlap scheduling is enabled - #19

Merged

ivanium merged 15 commits into

ovg-project:mainfrom

ivanium:fix-overlap-scheduling

Jul 17, 2025

Commits on Jul 17, 2025

fix [csrc]: make FTensorAllocator thread safe for overlap scheduling
ivanium
committed
3163078
fix [slab_allocator]: clean up Page.destroy() method
ivanium
committed
b5e9e0b
feat [KVCacheManager]: make class funcs synchronous
ivanium
committed
97d375e
fix [csrc]: fix the vaddr start addr
ivanium
committed
ea00d7e
fix [sglang-patch]: pass in python list rather than numpy array when freeing kv cache; suppress memory leak warnings when kvcached is enabled; add linter hints for in-func imports
ivanium
committed
a6c9ac9
fix [page-allocator]: synchrnoize cuda before unmap pages
ivanium
committed
54b8fb5
fix [interfaces]: update interfaces to pass in `async_sched` as a configurable arg. It is enabled for sglang by default for now
ivanium
committed
ee958be
perf [KVCacheManager]: use noop lock if async scheduling is not enabled
ivanium
committed
211d983
fix [KVCacheManager]: reserve the 0-th token slot for SGLang as padded token
ivanium
committed
0e2795c
fix [lock]: more locks to be conservative
ivanium
committed
2393be3
feat [FTensorAllocator]: add kv_tensors_created method to check if KV tensors are created; ensure python PageAllocator prealloc thread starts after KV tensors are created
ivanium
committed
f6f588b
chore [start_server.sh]: remove redundant PYTHONPATH exports
ivanium
committed
03d243a
feat [PageAllocator]: implement no-op locking and condition classes; refactor preallocation thread management to use these no-op implementations when preallocation is disabled
ivanium
committed
068ba2b
fix [KVCacheManager]: update alloc method to return None instead of an empty list
ivanium
committed
682c145
docs [FTensorAllocator]: add thread-safety note to create_ftensor_ and free_ftensor_; remove lock from free_tensor_ to keep code consistency
ivanium
committed
c72354f

Navigation Menu

Uh oh!

fix: various bugs to eliminate crashes when SGLang overlap scheduling is enabled - #19

fix: various bugs to eliminate crashes when SGLang overlap scheduling is enabled#19
ivanium merged 15 commits into
ovg-project:mainfrom
ivanium:fix-overlap-scheduling

Commits on Jul 17, 2025

fix [csrc]: make FTensorAllocator thread safe for overlap scheduling

fix [slab_allocator]: clean up Page.destroy() method

feat [KVCacheManager]: make class funcs synchronous

fix [csrc]: fix the vaddr start addr

fix [sglang-patch]: pass in python list rather than numpy array when freeing kv cache; suppress memory leak warnings when kvcached is enabled; add linter hints for in-func imports

fix [page-allocator]: synchrnoize cuda before unmap pages

fix [interfaces]: update interfaces to pass in `async_sched` as a configurable arg. It is enabled for sglang by default for now

perf [KVCacheManager]: use noop lock if async scheduling is not enabled

fix [KVCacheManager]: reserve the 0-th token slot for SGLang as padded token

fix [lock]: more locks to be conservative

feat [FTensorAllocator]: add kv_tensors_created method to check if KV tensors are created; ensure python PageAllocator prealloc thread starts after KV tensors are created

chore [start_server.sh]: remove redundant PYTHONPATH exports

feat [PageAllocator]: implement no-op locking and condition classes; refactor preallocation thread management to use these no-op implementations when preallocation is disabled

fix [KVCacheManager]: update alloc method to return None instead of an empty list

docs [FTensorAllocator]: add thread-safety note to create_ftensor_ and free_ftensor_; remove lock from free_tensor_ to keep code consistency