win-mediafoundation: Add Windows Media Foundation Capture Device and Intel NPU AI features#10471
win-mediafoundation: Add Windows Media Foundation Capture Device and Intel NPU AI features#10471thyintel wants to merge 2 commits into
Conversation
fc46b24 to
0584cda
Compare
RytoEX
left a comment
There was a problem hiding this comment.
Please use a more descriptive commit message.
In C++ for variables/functions/identifiers that remain in C++, we prefer camelCase over snake_case.
Is it possible to, at minimum, split this into two commits, or perhaps two PRs:
- Add Media Foundation
- Add the NPU features
c8280fd to
dcea1bb
Compare
|
|
|
The deps updated with obsproject/obs-deps#242 was merged, so this is no longer a draft, but it will need to be rebased. |
|
So I was able to get this building with current deps, however it does not work on Windows 10. This is due to usage of IMFCameraControlMonitor. I assume this is expected, but I'm noting it for posterity. |
|
I launched this PR on a Windows 11 Insider build, and got a crash when adding the source. Snippets: Full logs (session and crash): |
It looks like the Crash logs are from the Windows 10 build, are there any from your attempt at adding the source on Windows 11? |
|
This builds and runs on Windows 11 23H2 with the Windows 11 SDK (10.0.22621.0). The "Video Capture Device 2" is available and correctly loads my webcam. However, in Debug configuration, it spams the log with the following: log spamGiven the log spam, I wonder if something is not appropriately gated. Appropriately, since the hardware I'm testing on at the moment does not have an NPU, I do not see the NPU features (background removal, eye gaze correction, etc.). I will re-check on capable hardware at a later time. |
|
The current implementation of this PR will also need to bump the minimum Windows SDK version to 22621 due to the usage of IMFCameraControlNotify. obs-studio/cmake/windows/compilerconfig.cmake Lines 45 to 47 in 8b975c4 |
e2ca0f8 to
3609748
Compare
| add_library(win-mf MODULE) | ||
| add_library(OBS::mf ALIAS win-mf) | ||
|
|
||
| target_sources(win-mf PUBLIC mf-plugin.cpp win-mf.cpp) |
There was a problem hiding this comment.
This will need to be renamed as it will clobber the plugin name used by the MediaFoundation encoder plugin in development as well.
Will reply back once I have guidance for what the correct plugin name/ID and subdirectory should be to get a proper, final, name.
There was a problem hiding this comment.
Please use win-mediafoundation-capture for the plugin ID/name.
There was a problem hiding this comment.
Should be changed now. Please let me know if any of the file names need to be changed as well.
|
|
||
| target_sources(win-mf PUBLIC mf-plugin.cpp win-mf.cpp) | ||
|
|
||
| add_library(libmfcapture INTERFACE) |
There was a problem hiding this comment.
Is libmfcapture a bona-fide third party library? We do not want to carry around 3rd party libraries as vendored-in source code in our own source code, as this has lead to issues over the years (particularly with the vendored-in code becoming stale, unmaintained, and severely outdated).
If it is a "proper" library, it should be added to obs-deps so it's shipped with our other dependencies and then just pulled in by CMake just like WIL here.
There was a problem hiding this comment.
The code here is written by Intel.
There was a problem hiding this comment.
I think the point here was, "Is libmfcapture available as a third-party library? Can we pull it in from an outside source?" If so, we should pull it in via obs-deps, as we prefer to avoid vendoring dependencies. If it is not available that way, then that might be something to investigate such that the library is not a "bespoke OBS vendored dependency" but rather a "third party library available for other implementations".
RytoEX
left a comment
There was a problem hiding this comment.
Place the plugin files into plugins/win-mediafoundation/capture.
3609748 to
5aa187c
Compare
RytoEX
left a comment
There was a problem hiding this comment.
A nit while I notice it: update the commit message prefix to win-mediafoundation to match the paths.
|
As a note, I was testing this last week, and it looks like strings are failing to load for this, at least from the PR artifacts. |
I will try and recreate this issue! |
7eb6560 to
d8c5fe3
Compare
|
Had a chance to test this as well and found 2 issues with it: I have an Elgato Facecam Pro, and also their Camera Hub app installed, which in turn installs a virtual camera. This is on the device default resolution option. The device I actually wanted to test it with is my Capture Card from Elgato (the Elgato 4K X) - It's been suggested to me to give it a try with this to see if that eliminates the random stuttering / frame pacing issues that are happening with the current DirectShow Video Capture Device source in OBS. A lack of options compared to that source aside (whether intentional or not, such as Hardware Acceleration - not sure how Media Foundation handles that for example), I've noticed color differences. On the left is the Media Foundation source (from this PR), on the right the DirectShow source (normal video capture device in OBS): Doesn't seem to matter which video format / codec I select (XRGB, I420, NV12, YUY2 or MJPEG - P010 looks much more different, but that's obviously because of it being an HDR format) (Someone in the OBS Discord brought up the idea that this could be Full vs. Limited color space stuff perhaps?) Oh and yeah, the localization doesn't work on my end either - I grabbed the Windows x64 artifact from the Checks tab on this PR Yes, so after a quick test with the DirectShow Video Capture source set to "Color Range: Full" instead of the Default (which for this device it picks Limited) the colors are the exact same. So it seems like the Media Foundation Video Capture source doesn't automatically detect which Color Range to use and just defaults to Full range (lack of a select dropdown means I can't force it to Limited either) |
@Lordmau5 |
|
As a note here, this does not appear to do any kind of logging of device state, which will be critical for support. Example of a current DirectShow Device: Current Media Foundation device logs nothing of device state. This is just an example log output, and is not exhaustive of options/configurations that would need to be logged, and some of these are hidden if they are not set (i.e. the audio settings at the bottom are not shown if device audio is being used). We should try to match the paradigms on the DirectShow source as much as possible. |
| } else if (IsEqualCLSID(controlSet, PROPSETID_VIDCAP_CAMERACONTROL)) { | ||
| printf("PROPSETID_VIDCAP_CAMERACONTROL\n"); | ||
|
|
||
| } else if (IsEqualCLSID(controlSet, KSPROPERTYSETID_ExtendedCameraControl)) { |
There was a problem hiding this comment.
| } else if (IsEqualCLSID(controlSet, PROPSETID_VIDCAP_CAMERACONTROL)) { | |
| printf("PROPSETID_VIDCAP_CAMERACONTROL\n"); | |
| } else if (IsEqualCLSID(controlSet, KSPROPERTYSETID_ExtendedCameraControl)) { | |
| } else if (IsEqualCLSID(controlSet, PROPSETID_VIDCAP_CAMERACONTROL)) { | |
| printf("PROPSETID_VIDCAP_CAMERACONTROL\n"); | |
| } else if (IsEqualCLSID(controlSet, KSPROPERTYSETID_ExtendedCameraControl)) { |
| #include "mfcapture.hpp" | ||
| #include "win-mf.hpp" |
There was a problem hiding this comment.
Move these to the top of the includes?
| VALUE "InternalName", "win-mf" | ||
| VALUE "OriginalFilename", "win-mf" |
There was a problem hiding this comment.
This will need to be re-generated.
@RytoEX Grabbed the newest artifact, made a portable instance with it, added the source to it and when it popped up with the settings it had the Elgato Virtual Camera auto selected. New crash log (looks very similar, just different offsets): As before, if I'm fast enough I can switch it to either the Elgato Facecam Pro (not Virtual Camera) or my capture card and it'll work fine. But as soon as I switch back to the Virtual Camera and wait about 2 seconds it crashes again |
I can confirm that installing Elgato Camera Hub (currently version 2.2.2) and then selecting the Elgato Virtual Camera in the new "Video Capture Device (Media Foundation)" source, OBS will crash. You do not need to be running Elgato Camera Hub and you do not need to have an Elgato device connected to the system. Loading the Elgato Virtual Camera in the DShow source (Video Capture Device) does not crash.
Call stack: When loading a built-in webcam that works, I get: When loading the Elgato Virtual Camera, I get: I've passed this information on to Elgato in case this is an issue with their Virtual Camera rather than an issue with this PR. |
| MIT License | ||
|
|
||
| Copyright (c) 2025 Microsoft Corporation. |
There was a problem hiding this comment.
Why is the MIT License from Microsoft instead of Intel?
There was a problem hiding this comment.
This file is derived from Microsoft's MIT-licensed Media Foundation capture sample, with Intel's additions layered on top. MIT requires retaining Microsoft's copyright notice, so it stays. Intel's contributions are GPLv2-or-later.
| This is provided under a dual MIT/GPLv2 license. When using or | ||
| redistributing this, you may do so under either license. | ||
|
|
||
| GPL LICENSE SUMMARY | ||
|
|
||
| Copyright(c) 2025 Intel Corporation. | ||
|
|
||
| This program is free software; you can redistribute it and/or modify | ||
| it under the terms of version 2 of the GNU General Public License as |
There was a problem hiding this comment.
If Intel is the sole author of this file, GPLv2+ would be preferable to GPLv2-only.
There was a problem hiding this comment.
Updated Intel's GPL grant to "version 2 or, at your option, any later version." Intel isn't the sole author, but the only non-Intel code is Microsoft's MIT sample, which is compatible with GPLv2+.
| /* | ||
|
|
||
| This is provided under a dual MIT/BSD/GPLv2 license. When using or | ||
| redistributing this, you may do so under either license. |
There was a problem hiding this comment.
If licensed under three licenses, probably should not say "dual".
There was a problem hiding this comment.
You're right — three blocks shouldn't be called "dual." The BSD option isn't needed for this plugin, so I've removed the BSD block entirely. This should be fixed now.
| This is provided under a dual MIT/BSD/GPLv2 license. When using or | ||
| redistributing this, you may do so under either license. | ||
|
|
||
| GPL LICENSE SUMMARY | ||
|
|
||
| Copyright(c) 2025 Intel Corporation. | ||
|
|
||
| This program is free software; you can redistribute it and/or modify | ||
| it under the terms of version 2 of the GNU General Public License as | ||
| published by the Free Software Foundation. |
There was a problem hiding this comment.
If Intel is the sole author of this file, GPLv2+ would be preferable to GPLv2-only.
|
|
||
| target_sources(win-mf PUBLIC mf-plugin.cpp win-mf.cpp) | ||
|
|
||
| add_library(libmfcapture INTERFACE) |
There was a problem hiding this comment.
I think the point here was, "Is libmfcapture available as a third-party library? Can we pull it in from an outside source?" If so, we should pull it in via obs-deps, as we prefer to avoid vendoring dependencies. If it is not available that way, then that might be something to investigate such that the library is not a "bespoke OBS vendored dependency" but rather a "third party library available for other implementations".
|
Pushed a rebase which will fail. Will fix build failures after. |
|
Rebased. Logging still needs to be addressed to be equivalent to the existing DirectShow implementation. |
| This program is free software; you can redistribute it and/or modify | ||
| it under the terms of version 2 of the GNU General Public License as | ||
| published by the Free Software Foundation; either version 2 of the License, or (at your opinion) any later version. |
There was a problem hiding this comment.
Wrap text at 80 characters. There should be an example license available for the correct text wrapping.
| it under the terms of version 2 of the GNU General Public License as | ||
| published by the Free Software Foundation; either version 2 of the License, or (at your opinion) any later version. |
There was a problem hiding this comment.
Same here. Wrap at 80 characters (verify against the example license).



Description
PR is linked with: obsproject/obs-deps#242 (Requires wil headers to build)
Windows 11 is also required for this PR to build and Windows 11 SDK (10.0.22000.0).
This is a draft implementation of Intel AI features using the NPU(=”Intel AI Boost”) on Intel Core Ultra(a.k.a Meteor Lake).
It enables Background Blur(Standard and Portrait), Background Removal, Auto Framing and Eye Contact correction over new Microsoft Effect Package and Media Foundation.
Because this capture device is based on Media Foundation, we created separate “Video Capture Device 2”. We are thinking that it’s better to integrate it into the existing DShow based Video Capture Device in order to not to make end-users confused.
Thus, we need help for (1) overall design guidance how we can integrate - how we can kick off MF based device instead of DShow base, (2) Any missing features to get merged? Because we focus on prototyping new AI features, wondering if we’re missing something on the MF device.
BTW, this feature only works on OEM integrated web-cam, external USB camera is planned to be supported for the future, but not yet supported as of March 2024, no ETA announced.
Screen shots:

GPU 99%, NPU 30% over 3D Mark, w/ AI features are enabled
New UI for AI – Background Blur, Removal, Auto Framing, Eye Contact by NPU(=Intel AI Boost)

they only appear on MTL, otherwise this section would be skipped.
Motivation and Context
Utilize Intel NPU for OBS and be able to add AI capabilities in OBS for Intel devices.
How Has This Been Tested?
This has been tested on:
Types of changes
Checklist: