I want to allow a user to select caption/subtitle annotations to display over top of a video item

[elynema]

Description

I have a video file that has either captions or subtitles. I want the IIIF viewer to be able to identify the annotations that contain captions or subtitles so that they can be presented to the user via a closed caption / subtitle gadget to the user, and then displayed on top of the video file as it is played. Other types of supplementing annotations should not be presented to the user in this manner.

Variation(s)

(do you know of, or can you imagine, similar use cases?)

Proposed Solutions

Introduce annotation motivations of captioning and subtitling to declare the nature of the annotation(s).

Proposed snippet that utilizes a captioning motivation and refers out to an external webvtt file. In this example, the second annotation is a supplementing annotation that links out to a text file that contains a transcription of spoken audio, but is not timed text so could not be displayed as a caption.

annotations": [
{
"type": "AnnotationPage",
"id": "https://avalon-dev.dlib.indiana.edu/media_objects/gf06g267d/manifest/canvas/sj139194f/annotation_page/4cb485a0-c9e8-42ac-8a31-06aea8358251",
"items": [
{
"type": "Annotation",
"motivation": "captioning",
"body": {
"id": "https://avalon-dev.dlib.indiana.edu/master_files/sj139194f/supplemental_files/87",
"type": "Text",
"format": "text/vtt",
"label": {
"none": [
"Leonardo Da Vinci (1452-1519).vtt"
]
},
"language": "eng"
},
"id": "https://avalon-dev.dlib.indiana.edu/media_objects/gf06g267d/manifest/canvas/sj139194f/annotation_page/4cb485a0-c9e8-42ac-8a31-06aea8358251/supplementing/29d050e3-ca40-49c5-85f9-268da0b1cc74",
"target": "https://avalon-dev.dlib.indiana.edu/media_objects/gf06g267d/manifest/canvas/sj139194f"
},

{
"type": "Annotation",
"motivation": "supplementing",
"body": {
"id": "https://avalon-dev.dlib.indiana.edu/master_files/ng451h49f/supplemental_files/147/transcripts",
"type": "Text",
"format": "text/plain",
"label": {
"none": [
"lunchroom_manners.txt"
]
},
"language": "eng"
},
"id": "https://avalon-dev.dlib.indiana.edu/media_objects/fj236208t/manifest/canvas/ng451h49f/annotation_page/8c46422d-0f7d-4f13-95ab-d2ab074a4d3b/supplementing/25ff9231-9276-4036-bba7-76632f666d6d",
"target": "https://avalon-dev.dlib.indiana.edu/media_objects/fj236208t/manifest/canvas/ng451h49f"
}
]

Additional Background

(more about your perspective, existing work, etc. goes here.)

[jcoyne]

What if we have a file that contains captions or subtitles, but we don't know the motivation for the file's existence. Is there a more generic term that we could use that would encompass both cases (and would hint to the viewer to display it as closed captions).

[kevinglickavp]

The Aviary platform has been using the subtitling motivation for this purpose for a couple of years. Here these example manifests: https://weareavp.aviaryplatform.com/iiif/mc8rb6wr0j/manifest and https://weareavp.aviaryplatform.com/iiif/z892805d83/manifest. See the attached screenshot displaying how the subtitling motivation displays in the Aviary IIIF Player.

[elynema]

Ramp (being used within Avalon) is displaying the closed caption widget (the player would want to present any caption or subtitle available as a choice to a user within this widget) and the text of the selected caption over top of the video. This represents an annotation we ned to be able to identify as a subtitle / caption.

On the right you can see some textual transcription content that would not be appropriate to play over top of the video content since it is not timed text. So the caption / subtitle annotation needs to be distinguishable from the transcription annotation.

Currently, Ramp is labeling both these annotations as supplementing motivation and using a workaround with Avalon to identify content that should be presented on top of video by adding a term at the end of the ID in the annotation body. This obviously doesn't conform to IIIF spec:

"id": "https://avalon-dev.dlib.indiana.edu/master_files/ng451h49f/captions"

Full manifest is available here: https://raw.githubusercontent.com/elynema/sample-manifests/main/lunchroom-manners.json

[nfreire]

Europeana is currently using the motivations "subtitling" and "captioning".
One example is the video EXCELLENT EXTRA
in which the video has captions in Dutch (the language of the audio) and subtitles in English.
The Europeana IIIF viewer has been customized to support these motivations, showing the user the possibility to select the captions/subtitles in a menu, as shown in this screenshot: