Skip to content

digital-ink-recognition

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

digital-ink-recognition

README.md

Capacitor ML Kit Digital Ink Recognition Plugin

Unofficial Capacitor plugin for ML Kit Digital Ink Recognition.1

Use Cases

The Digital Ink Recognition plugin recognizes handwritten text and hand-drawn shapes from stroke coordinates directly on the device, for example:

  • Handwriting input: Let users write text with their finger or a stylus instead of typing.
  • Note taking: Convert handwritten notes into digital text.
  • Form filling: Recognize handwritten input in signature or annotation fields.
  • Sketch recognition: Classify hand-drawn shapes and emojis with the special gesture models.

Compatibility

Plugin Version Capacitor Version Status
8.x.x >=8.x.x Active support

Installation

You can use our AI-Assisted Setup to install the plugin. Add the Capawesome Skills to your AI tool using the following command:

npx skills add capawesome-team/skills --skill capacitor-plugins

Then use the following prompt:

Use the `capacitor-plugins` skill from `capawesome-team/skills` to install the `@capacitor-mlkit/digital-ink-recognition` plugin in my project.

If you prefer Manual Setup, install the plugin by running the following commands and follow the platform-specific instructions below:

npm install @capacitor-mlkit/digital-ink-recognition
npx cap sync

Attention: This plugin only supports CocoaPods for iOS dependency management. Swift Package Manager (SPM) is not supported for the ML Kit SDK, see this comment.

Android

Variables

If needed, you can define the following project variable in your app’s variables.gradle file to change the default version of the dependency:

  • $mlkitDigitalInkRecognitionVersion version of com.google.mlkit:digital-ink-recognition (default: 19.0.0)

This can be useful if you encounter dependency conflicts with other plugins in your project.

iOS

Minimum Deployment Target

Make sure to set the deployment target in your ios/App/Podfile to at least 15.5:

platform :ios, '15.5'

Configuration

No configuration required for this plugin.

Demo

A working example can be found here: robingenz/capacitor-mlkit-plugin-demo

Usage

Before any ink can be recognized, the model for the desired language must be downloaded:

import { DigitalInkRecognition } from '@capacitor-mlkit/digital-ink-recognition';

const downloadModel = async () => {
  await DigitalInkRecognition.downloadModel({
    languageTag: 'en-US',
  });
};

The plugin does not render anything. Capture the strokes with your own drawing surface, for example a canvas with pointer events:

import type { Stroke, StrokePoint } from '@capacitor-mlkit/digital-ink-recognition';

const canvas = document.querySelector('canvas');

const strokes: Stroke[] = [];
let currentPoints: StrokePoint[] | undefined;

canvas.addEventListener('pointerdown', event => {
  currentPoints = [{ x: event.offsetX, y: event.offsetY, t: Date.now() }];
});
canvas.addEventListener('pointermove', event => {
  currentPoints?.push({ x: event.offsetX, y: event.offsetY, t: Date.now() });
});
canvas.addEventListener('pointerup', event => {
  if (currentPoints) {
    currentPoints.push({ x: event.offsetX, y: event.offsetY, t: Date.now() });
    strokes.push({ points: currentPoints });
    currentPoints = undefined;
  }
});

Finally, pass the captured strokes to the recognize(...) method:

const recognize = async () => {
  const { candidates } = await DigitalInkRecognition.recognize({
    languageTag: 'en-US',
    strokes,
    writingArea: {
      width: canvas.width,
      height: canvas.height,
    },
  });
  return candidates;
};

Besides the language models, there are also special models for recognizing hand-drawn shapes (zxx-Zsym-x-autodraw) and emojis (zxx-Zsym-x-emoji).

API

deleteDownloadedModel(...)

deleteDownloadedModel(options: DeleteDownloadedModelOptions) => Promise<void>

Delete the downloaded model for the given language.

Only available on Android and iOS.

Param Type
options DeleteDownloadedModelOptions

Since: 8.2.0

downloadModel(...)

downloadModel(options: DownloadModelOptions) => Promise<void>

Download a model for the given language.

Only available on Android and iOS.

Param Type
options DownloadModelOptions

Since: 8.2.0

getDownloadedModels()

getDownloadedModels() => Promise<GetDownloadedModelsResult>

Get the languages for which a model has been downloaded.

Only available on Android and iOS.

Returns: Promise<GetDownloadedModelsResult>

Since: 8.2.0

recognize(...)

recognize(options: RecognizeOptions) => Promise<RecognizeResult>

Recognize the handwritten text or hand-drawn shapes in the given strokes.

The model for the given language must be downloaded before calling this method, otherwise the call is rejected with the error code MODEL_NOT_DOWNLOADED.

Only available on Android and iOS.

Param Type
options RecognizeOptions

Returns: Promise<RecognizeResult>

Since: 8.2.0

Interfaces

DeleteDownloadedModelOptions

Prop Type Description Since
languageTag string The BCP-47 language tag of the model to delete. 8.2.0

DownloadModelOptions

Prop Type Description Since
languageTag string The BCP-47 language tag of the model to download. See the supported languages for a list of available models. The special tag zxx-Zsym-x-autodraw identifies the model for recognizing hand-drawn shapes. 8.2.0

GetDownloadedModelsResult

Prop Type Description Since
languageTags string[] The BCP-47 language tags of the downloaded models. 8.2.0

RecognizeResult

Prop Type Description Since
candidates RecognitionCandidate[] The recognition candidates ordered by descending quality. 8.2.0

RecognitionCandidate

Prop Type Description Since
text string The recognized text. 8.2.0
score number The score of the candidate. Lower scores indicate better results. The score is not set for all models. 8.2.0

RecognizeOptions

Prop Type Description Default Since
languageTag string The BCP-47 language tag of the model to use for recognition. 8.2.0
strokes Stroke[] The strokes to recognize. 8.2.0
maxResultCount number The maximum number of recognition candidates to return. 5 8.2.0
preContext string The characters immediately preceding the strokes to improve recognition accuracy. A good rule of thumb is to provide as many characters as possible, up to around 20 characters (including spaces). 8.2.0
writingArea WritingArea The size of the writing area to improve recognition accuracy. 8.2.0

Stroke

Prop Type Description Since
points StrokePoint[] The points of the stroke in the order in which they were captured. 8.2.0

StrokePoint

Prop Type Description Since
x number The horizontal coordinate of the point. Increases to the right. 8.2.0
y number The vertical coordinate of the point. Increases going downward. 8.2.0
t number The timestamp of the point in milliseconds. Providing a timestamp improves recognition accuracy. 8.2.0

WritingArea

Prop Type Description Since
width number The width of the writing area. The unit must be the same as the one used for the stroke coordinates. 8.2.0
height number The height of the writing area. The unit must be the same as the one used for the stroke coordinates. 8.2.0

FAQ

Which platforms are supported by this plugin?

The plugin is available on Android and iOS. The recognition and model management methods are only available on Android and iOS, so there is no web implementation.

Which languages are supported?

Models are available for over 300 languages identified by BCP-47 language tags. There are also special models for recognizing hand-drawn shapes (zxx-Zsym-x-autodraw) and emojis (zxx-Zsym-x-emoji).

Do I need to download a model?

Yes. A model must be downloaded for each language before it can be used with the recognize(...) method. If the model has not been downloaded yet, the call is rejected with the error code MODEL_NOT_DOWNLOADED.

How do I capture the strokes?

The plugin does not provide a drawing surface. Capture the pointer events on your own canvas and pass the collected strokes to the recognize(...) method (see Usage).

Can I install this plugin using Swift Package Manager?

No, this plugin only supports CocoaPods for iOS dependency management because the ML Kit SDK itself does not support Swift Package Manager. Also make sure to set the deployment target in your ios/App/Podfile to at least 15.5 (see Installation).

Related Plugins

Terms & Privacy

This plugin uses the Google ML Kit:

Newsletter

Stay up to date with the latest news and updates about the Capawesome, Capacitor, and Ionic ecosystem by subscribing to our Capawesome Newsletter.

Changelog

See CHANGELOG.md.

License

See LICENSE.

Footnotes

  1. This project is not affiliated with, endorsed by, sponsored by, or approved by Google LLC or any of their affiliates or subsidiaries. ↩