Active Learning Training System

[john-]

When training my own models I was slowed down by gathering new images for me to annotate. As an active Viseron user I was initially looking at changes to Viseron to support this process. From the very helpful discussion in that thread I stepped back and revisited my approach.

The outcome of that is the creation of ALTS. At a high level, it is an MQTT client that receives events from Viseron and uses those as a basis for feeding Label Studio images and metadata to be used by an operator to annotate the images. These would then be used by model training process to (hopefully) improve the object detection model.

It is completely separate from Viseron. Although this causes some challenges, it seems like the right decision due to all the complexity in automating the triage process.

I have only been using it for a few days now but so far it has been extremely helpful. I have ideas for enhancement including generating links to the point in the Viseron stream that generated the snapshot.

Thanks for your assistance @roflcoopter and @kaburagisec!.

[kaburagisec]

great, will take a look

[roflcoopter]

Nice work, would be great if we could work together in the future to try and incorporate some of this in Viseron directly.
Could you explain to me in a step by step manner how you use this together with your other project, ODTP?
Having a streamlined transfer learning process inside Viseron would be a quite unique capability, sort of like Frigate+ but locally

[john-]

High level workflow

1. Configure Viseron with an object detector component

• ALTS works best when low confidence detections are in the mix along with high confidence
• It has only been tested with YOLO

Object detector snippet:

yolo:
  object_detector:
    model_path: /detectors/models/yolo/my_model.pt
    cameras:
      camera_1:
        fps: 1
        scan_on_motion_only: false
        labels:
        - label: person
          confidence: 0.3
          trigger_event_recording: true
        - label: cat
          confidence: 0.3
          trigger_event_recording: true
        - label: dog
          confidence: 0.3

MQTT snippet:

mqtt:
  broker: !secret mqtt_broker
  port: 1883

2. Run ALTS service to extract targeted images/bounding boxes from Viseron MQTT interface and add them to Label Studio (LS)

• There are configuration options in ALTS to throttle outout to LS (e.g. limit to 35% uncertain detections, 25% high confidence and 40% potential false negatives)
• Point ALTS to the same MQTT broker as Viseron

3. The user goes into Label Studio to validate the added images/bounding boxes

• This is a manual process where images can be discard, bounding boxes adjusted, and class reassigned)
• Only the self-hosted Label Studio has been tested (e.g. local Docker container)
• Configure Label Studio (specifically labels and their associated index)

4. The user runs ODTP pipeline to train a model

• In my experience training is iterative and with my limited knowledge of the science it is not always intuitive
  • Increasing volume of annotated training images improves the model (assuming appropriate balance between classes)
  • The merge of the COCO dataset into a combined dataset (LS+COCO) has multiple parameters that can be adjusted to impact the quality of the model
• Practically speaking training an object detection model requires a GPU (takes about 10-12 hours for ~7K annotated images at 1024 pixels wide on a 2080Ti with ~12GB VRAM)

Compared to Frigate+

It has been a couple years since I used Frigate and I never fully used Frigate+ (e.g. use the hosted service to generate a model). I did do some image annotation and upload to Frigate+ infratructure but things could have changed in that time.

1. Configure Frigate to upload images to their annotation infrastructure
2. User uses the Frigate hosted tool to annotate the images

• I don't recall the details but at a high level it is functionaly similar to what is done in Label Studio

3. Frigate infrastructure generates a model based on what is supported
4. User deploys the model

Again, I do not know much about the details of the process and things might have changed over the last couple of years.

Tighter integration with Viseron

When I first started this I envisioned very tight coupling with Viserion. With the current approach that is not the case. The only interface is MQTT and generic Viseron configuration.

Drawbacks:

• Possibly unwanted events in Viseron UI: ALTS logic seems to benifet from taking in images/detections with a wide range of confidence scores
  • The downside is that if threshold is set to 25% for ALTS they will also appear in the Viseron UI
• Detecting possible false negatives: Current implementation in ALTS considers a possible false negative one in which motion was detected but no object was detected
  • This logic is awkward over the MQTT link and there may be benifet to tigher coupling into Viseron

I don't see anything in the ODTP (LS extraction/dataset consolidation with COCO/training pipeline) that would benefit from tigher Viseron coupling.