Skip to content

Assertion input_val >= zero && input_val <= one failed #1850

@llongbif

Description

@llongbif

I'm training a YOLOX model for object detection using a dataset that includes:

  • Annotated images (positive samples)
  • Unannotated images (negative samples, i.e., no bounding boxes)

When I include negative images in the training dataset (images with no annotations), I consistently encounter the following CUDA error:

Assertion input_val >= zero && input_val <= one failed

From this trace:

../aten/src/ATen/native/cuda/Loss.cu:94: operator(): block: [0,0,0], thread: [0,0,0]
Assertion input_val >= zero && input_val <= one failed.
...
terminate called after throwing an instance of 'c10::DistBackendError'
what(): Process group watchdog thread terminated with exception: CUDA error: device-side assert triggered

The unannotated images in the .json file are only in the images section; they do not appear in the annotations section.

Is happening in the epoch 3 or later.

How can I safely include negative samples (images without annotations) in YOLOX training without triggering the CUDA assert?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions