MediaStream.addTrack does not accept CanvasCaptureMediaStreamTrack

[abardik]

Expected behavior

MediaStream.addTrack should allow to use as a first argument an instance of CanvasCaptureMediaStreamTrack captured from <canvas> by captureStream method.

Observerd behavior

MediaStream.addTrack throws an exception "argument must be an instance of MediaStreamTrack" if the first argument is an instance of CanvasCaptureMediaStreamTrack.

In Chrome/Firefox/Safari an instance of CanvasCaptureMediaStreamTrack inherits from MediaStreamTrack, so it can be used in MediaStream.addTrack method. In cordova app using iosrtc plugin CanvasCaptureMediaStreamTrack is not an instance of MediaStreamTrack, so we have an aforementioned exception in MediaStream.addTrack:

cordova-plugin-iosrtc/js/MediaStream.js

Lines 258 to 263 in b1bb4cb

	MediaStream.prototype.addTrack = function (track) {
	debug('addTrack() [track:%o]', track);
	if (!(track instanceof MediaStreamTrack)) {
	throw new Error('argument must be an instance of MediaStreamTrack');
	}

Steps to reproduce the problem

// capture local camera and microphone
navigator.mediaDevices.getUserMedia({ audio: true, video: true }).then(function(stream) {
  // get canvas stream
  var canvas = document.createElement('canvas');
  var canvasStream = canvas.captureStream();
  var track = canvasStream.getVideoTracks()[0];

  // add canvas video track to the stream
  try {
    stream.addTrack(track);
  }
  catch(err) {
    console.error(err); // output: Error: argument must be an instance of MediaStreamTrack
    console.log(track instanceof MediaStreamTrack); // output: false
  }
});

Platform information

• Cordova version: cli-9.0.0 (iOS 5.0.1)
• Plugin version: 6.0.10
• iOS version: 13.3.1, 13.4.1
• Xcode version: 10.2

[hthetiot]

@abardik

In Chrome/Firefox/Safari an instance of CanvasCaptureMediaStreamTrack inherits from MediaStreamTrack, so it can be used in MediaStream.addTrack method. In cordova app using iosrtc plugin CanvasCaptureMediaStreamTrack is not an instance of MediaStreamTrack, so we have an aforementioned exception in MediaStream.addTrack:

First, This plugin is not suposed to be used ourside of WKWebview and will not support any other enviroment.

Second, You have captureStream on Canvas working on iOS WKWebview ?
That would surpise me.

[hthetiot]

In any case, even if WKWebview was allowing Canvas.captureStream that I'm pretty sure it does not, It will require enormous work to enable converting a "Native" MediaStream to an iosRTC WebRTC MediaStream. But again I already think this is irrelevant since CaptureStream is not even working in Safari therefor it will not work in WKWebview.

• CaptureStream is not working in safari  webrtc/samples#1164

This is a free software, I will not implement such features for your blue eyes ;)

[abardik]

@hthetiot

First, This plugin is not suposed to be used ourside of WKWebview and will not support any other enviroment.

I mentioned other browsers only to show that CanvasCaptureMediaStreamTrack is an instance of MediaStreamTrack in those browsers. Of course, I'm not using iosrtc outside WKWebView, because it's impossible and doesn't make sence.

Second, You have captureStream on Canvas working on iOS WKWebview ?
That would surpise me.

It works in Safari and WKWebView at least on iOS 13.4.1 and 13.3.1 (iPhone XS), I don't know about earlier versions. I can send CanvasCaptureMediaStreamTrack to MediaStream.addTrack in Safari, but not in WKWebView.

[abardik]

@hthetiot

This is a free software, I will not implement such features for your blue eyes ;)

I'm not asking you to implement any features for me. I've already managed this without intermediate local stream. I'm just reporting an incompatibility issue in MediaStream.addTrack method.

You did a lot of work to make this plugin compatible with W3C WebRTC standard. I thought reporting this issue would help in this. Also, CanvasCaptureMediaStreamTrack is a good way to implement collaboration features like whiteboard, bringing new user experience in cordova apps, which is also good.

[hthetiot]

Thank you @abardik

I can address the fact that it support an instance of MediaStreamTrack for pure MediaStream the problem is that it will not work when you add that stream to the iOSRTC PeerConnection do you understand that ?

Note: until an issue is close any status like wonfix can change. I update label to reflect the last status, not the final.

[hthetiot]

@abardik

It works in Safari and WKWebView at least on iOS 13.4.1 and 13.3.1 (iPhone XS), I don't know about earlier versions. I can send CanvasCaptureMediaStreamTrack to MediaStream.addTrack in Safari, but not in WKWebView.

"It works in Safari and WKWebView" "but not in WKWebView." Sorry I'm confused by your answer, it does work in WKWebView or it does not, OR it does not because you use iosRTC ?

[hthetiot]

Let say it does in WKWebView, let say I allow it to work on native MediaStream (should actually already works with new MediaStream BTW).

what should be the behavior when you add this stream to the RTCPeerConnection in term of failure?

[hthetiot]

Ok I confirm captureStream landed on iOS WKWebView.

/ capture local camera and microphone
navigator.mediaDevices.getUserMedia({ audio: true, video: true }).then(function(stream) {
  // get canvas stream
  var canvas = document.createElement('canvas');
  var canvasStream = canvas.captureStream();
  var track = canvasStream.getVideoTracks()[0];

  // add canvas video track to the stream
  try {
    stream.addTrack(track);
  }
  catch(err) {
    console.error(err); // output: Error: argument must be an instance of MediaStreamTrack
    console.log(track instanceof MediaStreamTrack); // output: false
  }
});

This example is not even logic, you ask for video and audio from iosRTC and add a new video Native MediaStreamTrack on the same iosRTC MediaStream (so you have 2 video tracks on MediaStream).

Let say that you want to capture the VideoStream to draw on the canvas, If so you will need iOSRTC Renderer.save https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/extra/renderer-and-libwebrtc-tests.js#L385, but even with that you will only be able to display the MediaStream on a video tag, not sending it to RTCPeerConnection anyway.

Why use navigator.mediaDevices.getUserMedia in your example that will provide an iosRTC MediaStream that will not support your CanvasCaptureMediaStreamTrack as explained because this is not a WebRTC bridged MediaStreamTrack ?

You expect to be able to send a MediaStream from iosRTC getUserMedia to the Canvas then create an iosRTC MediaStream from it ? That is a features that will require lot of development, mainly converting the video of CanvasCaptureMediaStreamTrack to a WebRTC MediaStreamTrack inside the plugin.

Finally, if what you want is to add CanvasCaptureMediaStreamTrack, to a MediaStream, you can already, you need to use new MediaStream([canvasTrack]) or new MediaStream(canvasStream) instead that handle NativeMediaStreamTrack and result into a NativeMediaStream.

https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/js/MediaStream.js#L47

This work today on iosrtc and result in a "NativeMediaStreamTrack"

var canvas = document.createElement('canvas');
var canvasStream = canvas.captureStream();

var stream = new MediaStream(canvasStream);

// OR
var canvasTrack = canvasStream.getVideoTracks()[0];
var stream = new MediaStream([canvasTrack]);

But the stream or the track will not be compatible with WebRTC, that would require to convert NativeMediaStreamTrack or NativeMediaStream to iosRTC MediaStreamTrack or MediaStream and I will not implement that features for the moment due that this is a free software I maintain on my spare time.

[hthetiot]

"I'm not asking you to implement any features for me. "
Yes you do, you just dont understand that If I let people use NativeMediaStreamTrack on iosRTC generated MediaStream without making distinction they will expect the track to be streamed via WebRTC when it's not possible that why it does result in argument must be an instance of MediaStreamTrack to prevent this.

You create a iosRTC MediaStream with getUserMedia you can only add iosRTC MediaStreamTrack so you know it can be added to iosRTC PeerConnection.

If you want a "native" Stream with "native" MediaTrack use new MediaStream([canvasTrack]); or new MediaStream(canvasStream); it works today. But I alrady suspect that if you draw on canvas then add on Stream that for sending the stream back via WebRTC, otherwise you can simply display the canvas....

[hthetiot]

Also you have to realize that not all WKWebView are equals for example on iOS 10 "Native" MediaStream does not even exist and iOS 11 and 12 and less recent 13 will not have CaptureStream implemented.