Skip to content

numpy._core._exceptions._ArrayMemoryError: Unable to allocate 11.9 MiB for an array with shape (6, 1080, 1920) and data type bool #336

Closed
autodistill/autodistill
#180
@Jarradmorden

Description

@Jarradmorden

Search before asking

  • I have searched the Roboflow Notebooks issues and found no similar bug report.

Notebook name

https://github.com/roboflow/notebooks/blob/main/notebooks/how-to-auto-train-yolov8-model-with-autodistill.ipynb

Bug

Hello,

I am following some of the tutorials that roboflow offers, I am doing a custom dataset I have over 9000 pictures and I am using the ontology, it would seem when I reach the end of my training I run into memory issues "numpy._core._exceptions._ArrayMemoryError: Unable to allocate 11.9 MiB for an array with shape (6, 1080, 1920) and data type bool" and it only works if I train with a much smaller sample, I am guessing this is because they are all being processed in one go, I followed the tutorial for making a custom dataset but I think this would happen to anyone with a much larger size. How can I get around this issue.

I understand lowering the resolution would help but still by having so many issues I face the same issue, how would i go around this?

I put this as a bug because I am not sure if this notebook accounts for very large datasets so it would be good to do that if it's something that can happen to anyone, thank you

#####ISSUE########

left_9_2024-10-18 15-14-24.770364.png: 100%|██████████████████████████████████| 12498/12498 [6:07:58<00:00, 1.77s/it]
Passing a Dict[str, np.ndarray] into DetectionDataset is deprecated and will be removed in supervision-0.26.0. Use a list of paths List[str] instead.
Found dataset\train\images\left_9339_2024-10-18 15-23-39.366113.jpg as already present, not moving anything to dataset\train\images
Found dataset\train\labels\left_9339_2024-10-18 15-23-39.366113.txt as already present, not moving anything to dataset\train\labels
Found dataset\train\images\left_4584_2024-10-18 15-18-59.357296.jpg as already present, not moving anything to dataset\train\images
Found dataset\train\labels\left_4584_2024-10-18 15-18-59.357296.txt as already present, not moving anything to dataset\train\labels
Found dataset\train\images\left_4583_2024-10-18 15-18-59.323971.jpg as already present, not moving anything to dataset\train\images
Found dataset\train\labels\left_4583_2024-10-18 15-18-59.323971.txt as already present, not moving anything to dataset\train\labels
Found dataset\train\images\left_3070_2024-10-18 15-17-30.481540.jpg as already present, not moving anything to dataset\train\images
Found dataset\train\labels\left_3070_2024-10-18 15-17-30.481540.txt as already present, not moving anything to dataset\train\labels
Found dataset\train\images\left_9338_2024-10-18 15-23-39.299582.jpg as already present, not moving anything to dataset\train\images
Found dataset\train\labels\left_9338_2024-10-18 15-23-39.299582.txt as already present, not moving anything to dataset\train\labels
Found dataset\train\images\left_9335_2024-10-18 15-23-39.131904.jpg as already present, not moving anything to dataset\train\images
Found dataset\train\labels\left_9335_2024-10-18 15-23-39.131904.txt as already present, not moving anything to dataset\train\labels
Found dataset\train\images\left_9336_2024-10-18 15-23-39.198825.jpg as already present, not moving anything to dataset\train\images
Found dataset\train\labels\left_9336_2024-10-18 15-23-39.198825.txt as already present, not moving anything to dataset\train\labels
Found dataset\train\images\left_9337_2024-10-18 15-23-39.265686.jpg as already present, not moving anything to dataset\train\images
Found dataset\train\labels\left_9337_2024-10-18 15-23-39.265686.txt as already present, not moving anything to dataset\train\labels
Labeled dataset created - ready for distillation.
Traceback (most recent call last):
File "c:\Users\xxxxxx\DATA\model.py", line 45, in
dataset = sv.DetectionDataset.from_yolo(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Usersxxxx\AppData\Local\anaconda3\envs\visper_environment\Lib\site-packages\supervision\dataset\core.py", line 497, in from_yolo
classes, image_paths, annotations = load_yolo_annotations(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "cxxxxxx\AppData\Local\anaconda3\envs\xxxx_environment\Lib\site-packages\supervision\dataset\formats\yolo.py", line 120, in yolo_annotations_to_detections
mask = _polygons_to_masks(polygons=polygons, resolution_wh=resolution_wh)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\xxxxxx\AppData\Local\anaconda3\envs\xxx_environment\Lib\site-packages\supervision\dataset\formats\yolo.py", line 50, in _polygons_to_masks
return np.array(
^^^^^^^^^
numpy._core._exceptions._ArrayMemoryError: Unable to allocate 11.9 MiB for an array with shape (6, 1080, 1920) and data type bool

####MY CODE#####

import os
import torch  # Ensure torch is available for CUDA checks
from autodistill_grounded_sam import GroundedSAM
from autodistill.detection import CaptionOntology
from autodistill.utils import plot
import cv2
import supervision as sv
from autodistill_yolov8 import YOLOv8

IMAGE_DIR_PATH = f"2024-10-18_15-13-59/left_images"
DATASET_DIR_PATH = f"dataset"
ANNOTATIONS_DIRECTORY_PATH = f"dataset/train/labels"
IMAGES_DIRECTORY_PATH = f"dataset/train/images"
DATA_YAML_PATH = f"dataset/data.yaml"

image_paths = sv.list_files_with_extensions(
    directory=IMAGE_DIR_PATH,
    extensions=["png", "jpg", "jpg"]
)
print('image count:', len(image_paths))

ontology = CaptionOntology({
    "rear end of car": "Rear Car",
    "back view of vehicle": "Rear Car",
    "trunk of car": "Rear Car",
    "number plate": "licence plate"
})

device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f"Training will use: {device.upper()}")

base_model = GroundedSAM(ontology=ontology)
dataset = base_model.label(
    input_folder=IMAGE_DIR_PATH,
    extension=".png",
    output_folder=DATASET_DIR_PATH
)

dataset = sv.DetectionDataset.from_yolo(
    images_directory_path=IMAGES_DIRECTORY_PATH,
    annotations_directory_path=ANNOTATIONS_DIRECTORY_PATH,
    data_yaml_path=DATA_YAML_PATH
)

SAMPLE_SIZE = 16
SAMPLE_GRID_SIZE = (4, 4)
SAMPLE_PLOT_SIZE = (16, 16)

mask_annotator = sv.MaskAnnotator()
box_annotator = sv.BoxAnnotator()
image = cv2.imread(f"xxxxxxx/DATA/2024-10-18_15-13-59/sample/left_4583_2024-10-18 15-18-59.323971.png")


images = []
image_names = list(dataset.images.keys())[:SAMPLE_SIZE]
for image_name in image_names:
    image = dataset.images[image_name]
    annotations = dataset.annotations[image_name]
    labels = [dataset.classes[class_id] for class_id in annotations.class_id]
    
    annotated_image = mask_annotator.annotate(scene=image.copy(), detections=annotations)
    annotated_image = box_annotator.annotate(scene=annotated_image, detections=annotations)
    images.append(annotated_image)

sv.plot_images_grid(
    images=images,
    titles=image_names,
    grid_size=SAMPLE_GRID_SIZE,
    size=SAMPLE_PLOT_SIZE
)
sv.plot_image(image=annotated_image, size=(8, 8))

target_model = YOLOv8("yolov8s-seg.pt")
target_model.train(DATA_YAML_PATH, epochs=2, device=device)

Windows 11

I have quite good specs too
specs
python 3.11
and NVIDEA RTX A5000 Graphics card

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions