Support for Custum Video Effects - Win10 UWP

[kairohmer]

Hey there,
as I'm not able to find a way to use SharpDX an grab an live image from a webcam for access shaders (without mapping and copying every frame from CPU side), I'm suggesting a way to support it in more general way. Therefore, creating a custom video effects using SharpDX seems to be promising, as it can be used for all kinds of video effects, too.

An example how such an effect is created can be found here:
https://docs.microsoft.com/en-us/windows/uwp/audio-video-camera/custom-video-effects

The hardware supported version provides an IDirect3DDevice and an IDirect3DSurface, which are both com based (see https://docs.microsoft.com/en-us/uwp/api/windows.graphics.directx.direct3d11.idirect3dsurface)

Unfortunately, I not really familiar with com and was not able to get any valid native points to create a shared resource for instance.

If this is already support please give me a hint ;)
Cheers,

[kairohmer]

I got it working.. I'll post some details after testing and maybe I find some time to create a small sample
thanks anyway

[amerkoleci]

@kairohmer Do you plan to release example?

[kairohmer]

Unfortunately, I don't have time to write a complete example but I uploaded a file that contains the effect which allows to access the data: https://gist.github.com/kairohmer/af65c35a735879538082b04894238999

The idea is to provide a shared texture that can be consumed by you application. Note that the effect has to be part of Run-time Component Library (RTC) in order to work. A draft of the application side code can be found in the comments. Basically, you register two events. The first is fired every time a new (not yet seen) input texture arrives. Use the provided handle to open a shared Texture2D and store it in a hashmap with the handle as key. The second event is invoked for every incoming frame and provides the handle to the current input texture. Use the handle to access the dictionary and thereby the data in your shared texture. The events can occur anytime, so you probably want to store a flag when new images arrive and after processing an image to be able to use the image when it fits into your pipeline and to avoid redundant work.

I hope this will help some of you! In case of questions, feel free to ask ;)

Cheers,
Kai

[kairohmer]

@JimSEOW I'm not quite sure what you are trying to say. As far as I can see, there are no big changes concerning interrop since the "Metro" times, at least not on the windows platform. I recently ported a project from Windows 8.1 Store Apps to UWP -- the reason for the question above -- without ANY adjustments. I'm quite fine with the SwapChainPanels for SharpDX+XAML.

It's a bit different when it comes to Windows Desktop and interrop between WPF and DirectX11+, but unfortunately, the reason is the same.. nothing has changed ;)

[Gavin-Williams]

There is no mystery to using SharpDX with UWP XAML. You simply use the ImageSource, SurfaceImageSource, VirtualSurfaceImageSource or SwapChainPanel as per the Microsoft documentation .. https://docs.microsoft.com/en-us/windows/uwp/gaming/directx-and-xaml-interop.

It's not really interop, as SharpDX already hides the interop aspect. You are just drawing to these XAML controls with SharpDX as normal. This is basic SharpDX usage, there is really no need for extra samples as there are already samples and documentation available. If you need specific help you should post to MSDN or gamedev.net

[xoofx]

@JimSEOW ok, you gonna have to calm down otherwise I'm gonna lock your account to comment on any issues here.

Again, show a DirectX C++ example that is working on W10 Fall Creator, ported as-is to SharpDX, and is not working there.
If there is such a case, then yes, there is a bug in SharpDX. Otherwise, it is a bug in a yet unsupported OS/hardware (Same goes for 950XL and W10M)

[JimSEOW]

@xoofx I tested the example https://github.com/minhcly/UWP3DTest in desktop (Slow Ring) insider builds which will become official by Oct 15th using VS2017 (15.4 preview 2)

It would be great that this example becomes one of the "official" Store samples so there is a community effort to figure out why this XAML-interop UWP sample sometimes works and sometime not with the various (soon to be official) insider builds.

The VVVV team has tested this sample to work

Could be worth adding this to sample (to be honest I'd even remove rotating cube and such, as those part are exactly the same as a desktop app, so at least we only have one project with the exact minimal uwp setup required).

[xoofx]

You need to take an official DirectX C++ sample, and port it to C#. That's what I did in the past for the XAML SurfaceImageSource DirectX interop sample. It is laborious but very easy to do. I won't take for granted a random code on internet that doesn't give me quite some assurance that they have almost exactly the same C++ code in C#. So if you care so much about DirectX C# on UWP, you could do this. (In the first place, I would be very surprised that the original StoreApp sample is not working anymore, unless they have changed something in the original sample since then...)

[JimSEOW]

@xoofx according to this doc.

Two ways to do UWP XAML-DirectX Interop
(a) Shared Surface: SurfaceImageSource DirectX interop sample
(b) Swap chains : UWP3DTest

I have ported the Shared Surface example before to W10. Not a problem
The challenges I face is the Swap Chians with various slow ring insider builds.

[xoofx]

Great, so port an official DirectX sample that is using a Swapchain W10 so that we can have a look at the problem

[JimSEOW]

I just tested UWP3DTest (Swap Chain)
For both VS2015 and VS2017, it works fine with Creative Update (15063) for the latest SharpDX 4.01

The problem I face is VS2017 15.4 preview 2 when targeting higher SDK (16278). still investigating, why this is the case.

The latest SharpDX is FINE. The UWP3DTest is a suitable SwapChain sample.