Feature request: play videos in archive

[bendichter]

For .avi and other similar files, it would be great if there were a button to stream the video in browser without having to download it. The video player should have standard video controls: play, pause, scan, etc.

[CodyCBakerPhD]

I looked into how to add this to the widgets several months ago, which could be an option here unless there's a more readily available web solution

OpenCV video capture class accepts S3 URLs as inputs and did the streaming without needing fsspec or anything else so I was able to get that far

I know Plotly also has some functions for essentially playing videos given an array of image data, which was going to be the next thing I was going to try but then I got distracted by more important things

Only thing about the widgets is it might have to be a stand-alone thing run on an S3 URL (as opposed to being in the main accordion), since the relative paths from NWB files on the archive don't know anything about each other

[bendichter]

@CodyCBakerPhD yes, we should support this in the NWB Widgets (the I agree it's not highest priority). It also might be nice to have an implementation working for stand-alone video files. I know for myself, when I see an .avi file in an archive, my first impulse is to try to watch it directly. Having this implemented outside of NWB Widgets could have a number of advantages:

• The DANDI Archive may have video files that are not associated with NWB files.
• The user may only be interested in the video and not the rest of the NWB file.
• We might be able to implement in a serverless fashion where the data is streamed to the browsers and the browser plays the video. This could save us some overhead on spinning up servers for the widgets.

[waxlamp]

This is a regression: the eye icon is supposed to do exactly what you're looking for, @bendichter. I'll look into it.

[yarikoptic]

I beat you for a feature request @bendichter in #1389 . browser does play e.g https://api.dandiarchive.org/api/assets/bceab4dc-f921-444c-9a85-d26fa5e9b1c6/download/?content_disposition=inline from https://dandiarchive.org/dandiset/000231/0.220904.1554/files?location=sub-219CR%2Fsub-219CR_ses-20190403T123013_behavior%2Bimage for me. But indeed doesn't play https://api.dandiarchive.org/api/assets/942b0806-2c8b-4289-a072-9e965884fcb6/download?content_disposition=inline you mention. I guess those two have different encodingFormat.

• works:

❯ curl --silent https://api.dandiarchive.org/api/assets/bceab4dc-f921-444c-9a85-d26fa5e9b1c6/ | jq .encodingFormat
"video/x-matroska"

• not works:

❯ curl --silent https://api.dandiarchive.org/api/assets/942b0806-2c8b-4289-a072-9e965884fcb6/ | jq .encodingFormat
"video/x-msvideo"

not a regression. I will propose a quick fix shortly.

[waxlamp]

Unfortunately, @yarikoptic, I don't think your PR #1627 is a fix for Ben's feature request. The fundamental problem is that some video formats just won't play in the browser natively (even if the browser recognizes them by mime type; e.g., video/x-msvideo for .wmv files, etc.). With #1627, I believe .mp4, .mkv, and other "web native" formats like .webm will play in the browser when a user clicks the eye icon, but other formats won't (.avi, .wmv, etc.).

I think there are two main approaches to solving this: one is to find a javascript framework that can play a wider range of videos and create frontend components to use such a thing (no longer relying on the server to send HTML with a <video> tag and the browser to play it natively). I am not sure if there is a full-featured video player for Javascript (will need to consult some experts here and do some research). If there is, this seems like the best approach.

The other approach is to transcode all video files to a web-native format as they are uploaded, ensuring that all such videos can be played in standard browsers (we take this approach in a different project at Kitware, but there it is of top importance that we have full control over precise frame timing, etc.) This brings its own difficulties: for example, the file that the user uploads is no longer guaranteed to be the same file that is stored in the archive; presumably, this would have serious implications for reproducible science and NIH's data storage policies. Furthermore, we would need to add yet more task infrastructure to handle the transcoding, which will complicate dandiset upload and validation.

To summarize: we cannot rely on the browser to play all video files in the archive. We can mitigate this by controlling what video files are stored in the archive (transcoding approach) or by finding a frontend library that can play non web-native video files (or by some third approach that eludes me at the moment).

[bendichter]

Thanks for investigating, @waxlamp. It's good to know that at least some video formats will play. I'm not surprised that browsers don't have the same level of support as e.g. VLC, but I am a bit surprised that .avi isn't supported. If we can get ahold of a javascript library that supports avi, that seems like the best option.

If we go with transcoding the data, I don't think this would cause any problems for NIH's data storage policies per se, as they don't even necessarily require that the raw data be shared at all, but changing the data in subtle ways could cause all sorts of unexpected consequences for tool compatibility and reproducibility I'd like to avoid if possible. I am also unfamiliar with the compression options of mp4. How close could a transcoder get to the original .avi file frames?

A third option might be to transcode the video and save it as a copy, so that is the version that is played when requested in the browser, while the other one is downloaded when requested. That may be a lot of work though, I don't know, and could potentially lead to a lot of unnecessary extra data to store.

[yarikoptic]

I wonder if there is some sample collection of videos/formats to target here. https://en.wikipedia.org/wiki/HTML5_video#Browser_support gives a list of formats support in browsers... Having a nice player (better than browser, e.g. https://github.com/videojs/video.js) would be nice indeed... may be there is even some more crazy toolkit which would also allow to annotate them and we could integrate with so it would upload annotations? ;) well, if we were a beh data archive maybe