flutter_d2go

Flutter Plugin inferring using d2go, the mobile model of detectron2.

Features

• Get class and boundary box by object detection (Android and iOS)
• Get keypoints by keypoint estimation (Android and iOS)
• Get mask data by instance segmentation (Android only)
• Live inference for camera stream images (Android only)

Preview

• Live inference for camera stream images

  

• Object detection and instance segmentation

  

• Keypoints estimation

  

Installation

Add flutter_d2go to your pubspec.yaml.

Put the d2go model and class file in the assets directory.

assets:
  - assets/models/d2go.pt
  - assets/models/classes.txt

Usage

1. Load model and classes

The model is in Pytorch format.
The format of classes file is here.

await FlutterD2go.loadModel(
    modelPath: 'assets/models/d2go.pt',     // required
    labelPath: 'assets/models/classes.txt', // required
);

2. Get static image predictions

List<Map<String, dynamic>> output = await FlutterD2go.getImagePrediction(
    image: image,           // required File(dart:io) image
    width: 320,             // defaults to 320
    height: 320,            // defaults to 320
    mean: [0.0, 0.0, 0.0],  // defaults to [0.0, 0.0, 0.0]
    std: [1.0, 1.0, 1.0],   // defaults to [1.0, 1.0, 1.0]
    minScore: 0.7,          // defaults to 0.5
);

3. Get stream images predictions

List<Map<String, dynamic>> output = await FlutterD2go.getStreamImagePrediction(
    imageBytesList: cameraImage.planes.map((plane) => plane.bytes).toList(),             // required List<Uint8List> image byte array
    imageBytesPerPixel: cameraImage.planes.map((plane) => plane.bytesPerPixel).toList(), // default to [1, 2, 2]
    width: cameraImage.width,               // default to 720
    height: cameraImage.height,             // default to 1280
    inputWidth: 320,                        // defaults to 320
    inputHeight: 320,                       // defaults to 320
    mean: [0.0, 0.0, 0.0],                  // defaults to [0.0, 0.0, 0.0]
    std: [1.0, 1.0, 1.0],                   // defaults to [1.0, 1.0, 1.0]
    minScore: 0.7,                          // default to 0.5
    rotation: 90,                           // default to 0
);

Predictions output format

rect is the scale of the original image.
mask and keypoints depend on whether the d2go model has mask and keypoints.

mask will be a Uint8List of bitmap images bytes. keypoints will be a list of 17 (x, y).

[
  {
    "rect": {
      "left": 74.65713500976562,
      "top": 76.94147491455078,
      "right": 350.64324951171875,
      "bottom": 323.0279846191406
    },
    "mask": [66, 77, 122, 0, 0, 0, 0, 0, 0, 0, 122, ...],
    "keypoints": [[117.14504, 77.277405], [122.74037, 73.53044], [105.95437, 73.53044], ...],
    "confidenceInClass": 0.985002338886261,
    "detectedClass": "bicycle"
  }, // For each instance
...
]

Issues

Please contact here if you find any bugs or features you would like to add.