How to use OpenCV objects in Flutter?

[redcurry]

I think I'm having object memory issues. I'm using opencv_dart to find contours and store them in an object as cv.Contours. I then pass it to a CustomPainter to draw them on an image. I verified that the contours contain points, but at a later time they are empty, so nothing is drawn. Also, the application crashes randomly. I think I'm handling cv.Contours incorrectly. Am I not supposed to use them directly when drawing? Should I be copying any opencv_dart objects into Dart/Flutter objects? It's not clear to me how I should be managing the memory for opencv_dart objects.

[rainyl]

cv.Contours is actually cv.VecVecPoint, a dart wrapper of std::vector<std::vector<cv::Point>>, the native memory is managed by dart VM since it is attached to the dart instance, therefore, theoretically the native memory should have the same life time as the dart instance.

I am not sure how you process the opencv_dart objects, you can provide a demo so I can help locating the problem.

[redcurry]

It's difficult to provide a demo because there's a lot going on in my app. But do you see a problem with grabbing each cv.VecPoint from a cv.VecVecPoint and storing it in an object? Something like this:

Future<List<ContourAndHoles>> findAllAsync(cv.Mat binary) async {
  final List<ContourAndHoles> contoursAndHoles = [];

  final result = await cv.findContoursAsync(
    binary,
    cv.RETR_TREE,
    cv.CHAIN_APPROX_SIMPLE,
  );

  final cv.Contours contours = result.$1;
  final cv.VecVec4i hierarchy = result.$2;

  final List<int> contourIndexes = externalIndexes(hierarchy);
  for (final contourIndex in contourIndexes) {
    final cv.Contour contour = contours[contourIndex];

    // Skip small contours
    final contourArea = await cv.contourAreaAsync(contour);
    if (contourArea < minContourArea) {
      continue;
    }

    final contourAndPixelArea = ContourAndPixelArea(contour: contour, pixelArea: contourArea);

    final holes = await getHolesAsync(contours, hierarchy, contourIndex);

    contoursAndHoles.add(ContourAndHoles(contour: contourAndPixelArea, holes: holes));
  }

  return contoursAndHoles;
}

The ContourAndPixelArea object I create is where I store the individual cv.VecPoint and area, then add it to a list.

[rainyl]

@redcurry aha, I got it.

TL;DR, copy the contour passed to ContourAndPixelArea explicitly.

More details refer to #311 (comment)

Edit:

Maybe it should be mentioned in the doc, I will add it.

[redcurry]

Ah, so if I understand correctly, at some point cv.VecVecPoint is garbage-collected because I don't keep a reference to it. But the way that accessing its individual cv.VecPoint items works (via operator[]), those objects now point to invalid memory after the garbage-collection.

Thank you for your workaround and link to the details!

[rainyl]

:) U R welcome.

Ah, so if I understand correctly, at some point cv.VecVecPoint is garbage-collected because I don't keep a reference to it. But the way that accessing its individual cv.VecPoint items works (via operator[]), those objects now point to invalid memory after the garbage-collection.

Yes, the VecPoint returned from operator [] is a reference and the native memory belongs to VecVecPoint, when returned from the function the VecVecPoint is planned to be GC-ed. You can call VecPoint.clone(), i.e., final contourAndPixelArea = ContourAndPixelArea(contour: contour.clone(), pixelArea: contourArea); to copy it explicitly and try whether it works.

Maybe we can try to attach the child element of VecVecPoint to its parent but this will introduce extra memory consumption, so for now I will add a document to mind the users.

[redcurry]

Yes, I can confirm that calling .clone() fixes the problem. Thanks again! And, yes, something about this topic in the documentation would be very helpful.