Support custom user-trained models in AI Text-to-Annotation

[wkentaro]

This was generated by AI during triage.

Summary

Allow AI Text-to-Annotation to run a user's own custom-trained detection model, not only the built-in sam3 and yoloworld models.

Use case / motivation

Reported by a daily LabelMe user who builds YOLO training datasets:

• Their target objects are domain-specific and not covered by the built-in models.
• Their workflow today: annotate in LabelMe -> train a YOLO model -> run detection.
• If LabelMe's AI Text-to-Annotation could use their own trained model, they could pre-annotate the objects their model already knows and focus manual effort on the under-trained ones, i.e. model-assisted labeling on their own domain (a productivity flywheel).

Current behavior

• The model list is hardcoded in labelme/widgets/_ai_text_to_annotation_widget.py (sam3:latest, yoloworld:latest); there is no way to register a user model.
• Models are served through osam, whose model/cache location is hardcoded with no config override.

Considerations

• Likely requires upstream osam support for registering / loading user-provided weights, and possibly a configurable model directory (see Suggest customizing the model directory #1428).
• UX questions: how the user points to a model file/dir, how predicted class indices map to label names, validation/error reporting.
• Distinct from but related to Automatic AI annotation function, it is recommended to add ultralytics models to the entire series #1658 (add the official ultralytics model zoo). This request is specifically bring-your-own custom-trained weights.

Related

• Automatic AI annotation function, it is recommended to add ultralytics models to the entire series #1658 (add ultralytics models)
• Suggest customizing the model directory #1428 (configurable model directory)

[wkentaro]

This was generated by AI during triage.

Additional use case (from support feedback)

A user working on power-line replacement construction confirmed this need with a concrete example. Their domain objects (e.g. 金車 / sheave-pulleys mounted on transmission-tower arms) are construction-specific and not in any built-in model, so they train their own YOLO weights for them.

Open-vocabulary doesn't cover it: they tried YOLO-World via AI Text-to-Annotation with the term "pully" and it returned a box around the entire transmission tower rather than the pulley — expected behavior for open-vocab detection on niche jargon, and exactly why bring-your-own-weights matters here.

New angle worth noting: beyond reducing annotation effort, they want to use their own model inside LabelMe to verify/validate its detections on new images — i.e. LabelMe as a lightweight model-verification tool, not only an annotation generator. Worth keeping in mind for the UX (showing predictions, confidence, letting the user accept/reject).