- 
                Notifications
    
You must be signed in to change notification settings  - Fork 2.4k
 
Description
I'm training a YOLOX model for object detection using a dataset that includes:
- Annotated images (positive samples)
 - Unannotated images (negative samples, i.e., no bounding boxes)
 
When I include negative images in the training dataset (images with no annotations), I consistently encounter the following CUDA error:
Assertion input_val >= zero && input_val <= one failed
From this trace:
../aten/src/ATen/native/cuda/Loss.cu:94: operator(): block: [0,0,0], thread: [0,0,0]
Assertion input_val >= zero && input_val <= one failed.
...
terminate called after throwing an instance of 'c10::DistBackendError'
what():  Process group watchdog thread terminated with exception: CUDA error: device-side assert triggered
The unannotated images in the .json file are only in the images section; they do not appear in the annotations section.
Is happening in the epoch 3 or later.
How can I safely include negative samples (images without annotations) in YOLOX training without triggering the CUDA assert?