Fix GPU Resource Reservation Preventing Non-GPU Frame Dispatch #2095
+19
−2
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Hosts with GPU capabilities (ALLOW_GPU=true) incorrectly reject frames that have zero GPU memory requirements. This prevents efficient resource utilization where GPU-enabled hosts should accept both GPU and CPU-only workloads.
Root Cause
The getGpuJobs() method in CoreUnitDispatcher calls removeGpu() when no GPU-specific jobs are found. This method reduces all host resources (CPU cores, memory, GPU resources) to "reserve space for future GPU frames." When normal frames are subsequently evaluated, they fail resource checks due to these artificially reduced limits.
The problematic flow:
Solution
Add a configuration property dispatcher.gpu.skip_resource_reservation that disables GPU resource reservation when set to true. Default is false to maintain backward compatibility.
Configuration
The new behavior is controlled by environment variable CUEBOT_DISPATCHER_SKIP_GPU_RESERVATION:
Recap
Backward compatibility is preserved.
This resolves the counter-intuitive behavior where GPU-capable hosts reject valid CPU-only frames.