Skip to content

win-mf: Add Windows Media Foundation Capture Device and Intel NPU AI features #10471

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

thyintel
Copy link
Contributor

@thyintel thyintel commented Apr 2, 2024

Description

PR is linked with: obsproject/obs-deps#242 (Requires wil headers to build)
Windows 11 is also required for this PR to build and Windows 11 SDK (10.0.22000.0).

This is a draft implementation of Intel AI features using the NPU(=”Intel AI Boost”) on Intel Core Ultra(a.k.a Meteor Lake).
It enables Background Blur(Standard and Portrait), Background Removal, Auto Framing and Eye Contact correction over new Microsoft Effect Package and Media Foundation.
Because this capture device is based on Media Foundation, we created separate “Video Capture Device 2”. We are thinking that it’s better to integrate it into the existing DShow based Video Capture Device in order to not to make end-users confused.
Thus, we need help for (1) overall design guidance how we can integrate - how we can kick off MF based device instead of DShow base, (2) Any missing features to get merged? Because we focus on prototyping new AI features, wondering if we’re missing something on the MF device.
BTW, this feature only works on OEM integrated web-cam, external USB camera is planned to be supported for the future, but not yet supported as of March 2024, no ETA announced.

Screen shots:
GPU 99%, NPU 30% over 3D Mark, w/ AI features are enabled
image

New UI for AI – Background Blur, Removal, Auto Framing, Eye Contact by NPU(=Intel AI Boost)
they only appear on MTL, otherwise this section would be skipped.
image

Motivation and Context

Utilize Intel NPU for OBS and be able to add AI capabilities in OBS for Intel devices.

How Has This Been Tested?

This has been tested on:

  • MTL system containing the Intel NPU
  • ADL system without NPU

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist:

  • [x ] My code has been run through clang-format.
  • [x ] I have read the contributing document.
  • [x ] My code is not on the master branch.
  • [x ] The code has been tested.
  • [x ] All commit messages are properly formatted and commits squashed where appropriate.
  • [x ] I have included updates to all appropriate documentation.

@thyintel thyintel marked this pull request as draft April 2, 2024 21:29
@WizardCM WizardCM added the New Feature New feature or plugin label Apr 2, 2024
@RytoEX RytoEX self-assigned this Apr 3, 2024
@thyintel thyintel force-pushed the obs_intel_ai branch 3 times, most recently from fc46b24 to 0584cda Compare April 5, 2024 20:49
Copy link
Member

@RytoEX RytoEX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use a more descriptive commit message.

In C++ for variables/functions/identifiers that remain in C++, we prefer camelCase over snake_case.

Is it possible to, at minimum, split this into two commits, or perhaps two PRs:

  1. Add Media Foundation
  2. Add the NPU features

@thyintel thyintel changed the title win-mf: Add win-mf project win-mf: Add Windows Media Foundation Capture Device and Intel NPU AI features Apr 15, 2024
@thyintel thyintel force-pushed the obs_intel_ai branch 6 times, most recently from c8280fd to dcea1bb Compare April 23, 2024 03:06
@RytoEX
Copy link
Member

RytoEX commented Apr 24, 2024

plugins/win-mf/data/placeholder.png is only used for the DirectShow-based Virtual Camera. It doesn't seem to have a purpose here?

@RytoEX RytoEX marked this pull request as ready for review May 16, 2024 17:55
@RytoEX
Copy link
Member

RytoEX commented May 16, 2024

The deps updated with obsproject/obs-deps#242 was merged, so this is no longer a draft, but it will need to be rebased.

@WizardCM
Copy link
Member

So I was able to get this building with current deps, however it does not work on Windows 10. This is due to usage of IMFCameraControlMonitor. I assume this is expected, but I'm noting it for posterity.

10:31:38.979: Module '../../obs-plugins/64bit/win-mf.dll' not loaded

@WizardCM
Copy link
Member

I launched this PR on a Windows 11 Insider build, and got a crash when adding the source.

Snippets:

11:04:53.494: Video Capture Device 2 - MF: DecodeDeviceId failed
11:04:53.494: Video Capture Device 2 - MF: Video configuration failed
11:04:53.501: User added source 'Video Capture Device 2 - MF' (MediaFoundationsource_input) to scene 'Scene'
win-mf.dll!PhysicalCamera::GetBlur+0x55
win-mf.dll!PhysicalCamera::GetMepSetting+0x29
win-mf.dll!PhysicalCamera::SaveSettingsToDefault+0x11c
win-mf.dll!PhysicalCamera::Stop+0x2d
win-mf.dll!PhysicalCamera::Uninitialize+0x2d
win-mf.dll!PhysicalCamera::~PhysicalCamera+0x2e
win-mf.dll!PhysicalCamera::Release+0x22
win-mf.dll!MF_Create+0x4f
win-mf.dll!DeviceEnumerator::EnumerateCameraCallback+0xc2
win-mf.dll!MF_EnumerateCameras+0x93
win-mf.dll!DeviceEnumerator::Enumerate+0x7c
win-mf.dll!GetMediaFoundationSourceProperties+0xb2
obs.dll!obs_source_properties+0x8d
obs64.exe!OBSPropertiesView::ReloadProperties+0x81

Full logs (session and crash):

2024-06-16 11-04-39.txt

Crash 2024-06-16 11-04-54.txt

@thyintel
Copy link
Contributor Author

I launched this PR on a Windows 11 Insider build, and got a crash when adding the source.

Snippets:

11:04:53.494: Video Capture Device 2 - MF: DecodeDeviceId failed
11:04:53.494: Video Capture Device 2 - MF: Video configuration failed
11:04:53.501: User added source 'Video Capture Device 2 - MF' (MediaFoundationsource_input) to scene 'Scene'
win-mf.dll!PhysicalCamera::GetBlur+0x55
win-mf.dll!PhysicalCamera::GetMepSetting+0x29
win-mf.dll!PhysicalCamera::SaveSettingsToDefault+0x11c
win-mf.dll!PhysicalCamera::Stop+0x2d
win-mf.dll!PhysicalCamera::Uninitialize+0x2d
win-mf.dll!PhysicalCamera::~PhysicalCamera+0x2e
win-mf.dll!PhysicalCamera::Release+0x22
win-mf.dll!MF_Create+0x4f
win-mf.dll!DeviceEnumerator::EnumerateCameraCallback+0xc2
win-mf.dll!MF_EnumerateCameras+0x93
win-mf.dll!DeviceEnumerator::Enumerate+0x7c
win-mf.dll!GetMediaFoundationSourceProperties+0xb2
obs.dll!obs_source_properties+0x8d
obs64.exe!OBSPropertiesView::ReloadProperties+0x81

Full logs (session and crash):

2024-06-16 11-04-39.txt

Crash 2024-06-16 11-04-54.txt

It looks like the Crash logs are from the Windows 10 build, are there any from your attempt at adding the source on Windows 11?

@WizardCM
Copy link
Member

@thyintel the crash log is definitely from Windows 11, per the build number 26236 - not sure why it says 10. As I mentioned here, the MediaFoundation source is entirely hidden/disabled on on Windows 10 due to using Windows 11-only APIs.

@RytoEX
Copy link
Member

RytoEX commented Jul 10, 2024

This builds and runs on Windows 11 23H2 with the Windows 11 SDK (10.0.22621.0). The "Video Capture Device 2" is available and correctly loads my webcam. However, in Debug configuration, it spams the log with the following:

log spam
source 'Video Capture Device 2' (MediaFoundationsource_input) created
Video Capture Device 2: DecodeDeviceId failed
Video Capture Device 2: Video configuration failed
User added source 'Video Capture Device 2' (MediaFoundationsource_input) to scene 'Scene'
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(1363)\win-mf.dll!00007FFA13A03EB7: (caller: 00007FFA13A04389) ReturnHr(3003) tid(4df8) 80070490 Element not found.
    [PhysicalCamera::GetMepSetting(GetBlur(blur, shallowfocus, mask))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(1401)\win-mf.dll!00007FFA13A043E9: (caller: 00007FFA13A06055) ReturnHr(3004) tid(4df8) 80070490 Element not found.
    [PhysicalCamera::SaveSettingsToDefault(GetMepSetting(setting))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(1363)\win-mf.dll!00007FFA13A03EB7: (caller: 00007FFA13A04389) ReturnHr(3005) tid(3978) 80070490 Element not found.
    [PhysicalCamera::GetMepSetting(GetBlur(blur, shallowfocus, mask))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(1401)\win-mf.dll!00007FFA13A043E9: (caller: 00007FFA13A06055) ReturnHr(3006) tid(3978) 80070490 Element not found.
    [PhysicalCamera::SaveSettingsToDefault(GetMepSetting(setting))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(393)\win-mf.dll!00007FFA13A04567: (caller: 00007FFA13A04CDB) ReturnHr(3007) tid(5490) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(pSample->GetUnknown(MFSampleExtension_CaptureMetadata, IID_PPV_ARGS(&spMetadataAttri)))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(393)\win-mf.dll!00007FFA13A04567: (caller: 00007FFA13A04CDB) ReturnHr(3008) tid(3fcc) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(pSample->GetUnknown(MFSampleExtension_CaptureMetadata, IID_PPV_ARGS(&spMetadataAttri)))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(397)\win-mf.dll!00007FFA13A0462E: (caller: 00007FFA13A04CDB) ReturnHr(3009) tid(5490) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(spMetadataAttri->GetBlobSize( MF_CAPTURE_METADATA_FRAME_BACKGROUND_MASK, &cbBlobSize))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(397)\win-mf.dll!00007FFA13A0462E: (caller: 00007FFA13A04CDB) ReturnHr(3010) tid(3fcc) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(spMetadataAttri->GetBlobSize( MF_CAPTURE_METADATA_FRAME_BACKGROUND_MASK, &cbBlobSize))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(397)\win-mf.dll!00007FFA13A0462E: (caller: 00007FFA13A04CDB) ReturnHr(3011) tid(5490) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(spMetadataAttri->GetBlobSize( MF_CAPTURE_METADATA_FRAME_BACKGROUND_MASK, &cbBlobSize))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(397)\win-mf.dll!00007FFA13A0462E: (caller: 00007FFA13A04CDB) ReturnHr(3012) tid(5994) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(spMetadataAttri->GetBlobSize( MF_CAPTURE_METADATA_FRAME_BACKGROUND_MASK, &cbBlobSize))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(397)\win-mf.dll!00007FFA13A0462E: (caller: 00007FFA13A04CDB) ReturnHr(3013) tid(5490) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(spMetadataAttri->GetBlobSize( MF_CAPTURE_METADATA_FRAME_BACKGROUND_MASK, &cbBlobSize))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(397)\win-mf.dll!00007FFA13A0462E: (caller: 00007FFA13A04CDB) ReturnHr(3014) tid(5994) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(spMetadataAttri->GetBlobSize( MF_CAPTURE_METADATA_FRAME_BACKGROUND_MASK, &cbBlobSize))]

Given the log spam, I wonder if something is not appropriately gated.

Appropriately, since the hardware I'm testing on at the moment does not have an NPU, I do not see the NPU features (background removal, eye gaze correction, etc.). I will re-check on capable hardware at a later time.

@WizardCM
Copy link
Member

The current implementation of this PR will also need to bump the minimum Windows SDK version to 22621 due to the usage of IMFCameraControlNotify.

if(CMAKE_VS_WINDOWS_TARGET_PLATFORM_VERSION VERSION_LESS 10.0.20348)
message(FATAL_ERROR "OBS requires Windows SDK version 10.0.20348.0 or more recent.\n"
"Please download and install the most recent Windows SDK.")

@WizardCM
Copy link
Member

Ran the build on a debugger on Windows and confirmed the crash was not due to building on Windows:

image

@RytoEX RytoEX added the Seeking Testers Build artifacts on CI label Sep 13, 2024
@RytoEX RytoEX mentioned this pull request Sep 13, 2024
6 tasks
@thyintel thyintel force-pushed the obs_intel_ai branch 2 times, most recently from 6598aae to eca2e43 Compare December 5, 2024 21:06
Copy link
Member

@RytoEX RytoEX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this PR has some files generated using an older version of obs-plugintemplate. Those files should be replaced using a newer version of the obs-plugintemplate.

New code files should have appropriate license headers (GPLv2+).

@thyintel thyintel force-pushed the obs_intel_ai branch 2 times, most recently from 2c6ef1a to bbe146a Compare February 4, 2025 20:39
@@ -87,4 +87,5 @@ add_obs_plugin(text-freetype2)
add_obs_plugin(vlc-video WITH_MESSAGE)
add_obs_plugin(win-capture PLATFORMS WINDOWS)
add_obs_plugin(win-dshow PLATFORMS WINDOWS)
add_obs_plugin(win-mf PLATFORMS WINDOWS)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Todo for OBS: are we satisfied using win-mf for the plugin name?

Copy link
Member

@PatTheMav PatTheMav Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer if the name were spelt-out - the abbreviation is ambiguous from the outside and doesn't provide any good hint at what's inside the can.

win-mediafoundation would be acceptable, personally I don't like the platform prefixes and rather let the technology speak for itself (i.e., a plugin called "obs-mediafoundation" contains support for "Media Foundation", which just happens to be only available on Windows).

Copy link
Member

@RytoEX RytoEX Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incidentally, we used to have a module named win-mf that provided Media Foundation encoders on Windows, so it might not be a good idea to use the same name for a capture plugin.

Comment on lines +104 to +125
VS_VERSION_INFO VERSIONINFO
FILEVERSION 30,0,1,0
BEGIN
BLOCK "StringFileInfo"
BEGIN
BLOCK "040904B0"
BEGIN
VALUE "CompanyName", "Intel Corporation"
VALUE "FileDescription", "OBS Video Capture module"
VALUE "FileVersion", "30.0.1"
VALUE "ProductName", "OBS Studio"
VALUE "ProductVersion", "30.0.1"
VALUE "LegalCopyright", "Copyright (C) 2023, Intel Corporation"
VALUE "InternalName", "win-mf"
VALUE "OriginalFilename", "win-mf"
END
END
BLOCK "VarFileInfo"
BEGIN
VALUE "Translation", 0x409,1200
END
END
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than hard-coding this, we tend to use an .rc.in file and have CMake process that into an .rc file, with few exceptions. @PatTheMav ?

I'm not sure if the rest of the file contents here are used.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, almost all values are defined by CMake project definition, so the file should be generated by CMake as well. Particularly version numbers will be fully dynamic.

@@ -0,0 +1,23 @@
// {{NO_DEPENDENCIES}}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to the .rc file, I'm not sure what these are used for.

@@ -0,0 +1,20 @@
/* mf-plugin.cpp */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add an appropriate license header with copyright info.

Comment on lines 17 to 70
class VideoBufferLock {

public:
VideoBufferLock(IMFMediaBuffer *pBuffer)
{
m_spBuffer = pBuffer;
m_spBuffer->QueryInterface(IID_PPV_ARGS(&m_sp2DBuffer));
}

~VideoBufferLock()
{
UnlockBuffer();
m_spBuffer = nullptr;
m_sp2DBuffer = nullptr;
}

HRESULT LockBuffer(LONG lDefaultStride, DWORD dwHeightInPixels, BYTE **ppbScanLine0, LONG *plStride)
{
HRESULT hr = S_OK;
if (m_sp2DBuffer) {
hr = m_sp2DBuffer->Lock2D(ppbScanLine0, plStride);
} else {
BYTE *pData = NULL;
hr = m_spBuffer->Lock(&pData, NULL, NULL);
if (SUCCEEDED(hr)) {
*plStride = lDefaultStride;
if (lDefaultStride < 0) {
*ppbScanLine0 = pData + abs(lDefaultStride) * (dwHeightInPixels - 1);
} else {
*ppbScanLine0 = pData;
}
}
}

m_bLocked = (SUCCEEDED(hr));
return hr;
}

void UnlockBuffer()
{
if (m_bLocked) {
if (m_sp2DBuffer) {
m_sp2DBuffer->Unlock2D();
} else {
m_spBuffer->Unlock();
}
m_bLocked = FALSE;
}
}

private:
wil::com_ptr_nothrow<IMFMediaBuffer> m_spBuffer;
wil::com_ptr_nothrow<IMF2DBuffer> m_sp2DBuffer;

BOOL m_bLocked = FALSE;
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we should define this in a separate file. @PatTheMav ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, any class definition should get its own separate source file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have moved the code into a separate file, but should it also have a header file?

Copy link
Member

@RytoEX RytoEX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For new files, use appropriate license headers (GPLv2+).

@thyintel thyintel force-pushed the obs_intel_ai branch 3 times, most recently from d1d4308 to c014990 Compare March 26, 2025 19:29
@thyintel thyintel force-pushed the obs_intel_ai branch 5 times, most recently from 9ff3014 to d9c667d Compare April 2, 2025 21:10
@thyintel thyintel force-pushed the obs_intel_ai branch 4 times, most recently from 3b22ef5 to e925aa4 Compare April 17, 2025 15:27
Copy link
Member

@RytoEX RytoEX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Newly added files should have appropriate and compatible license headers (GPLv2+).

@@ -0,0 +1,21 @@
class CriticalSection {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add appropriate license headers (GPLv2+).

@@ -0,0 +1,58 @@
#include <mfidl.h>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add appropriate license headers (GPLv2+).

Comment on lines +19 to +23
libmfcapture/source/DeviceControlChangeListener.cpp
libmfcapture/source/mfcapture.cpp
libmfcapture/source/PhysicalCamera.cpp
libmfcapture/source/VideoBufferLock.cpp
libmfcapture/source/CriticalSection.cpp
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alphabetize these.

Comment on lines +287 to +291
bool _deactivateWhenNotShowing = false;
bool _flip = false;
bool _active = false;
bool _autorotation = true;
bool _firstframe = true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid prefixing variables with _. While technically permitted by the spec in these cases, it's better to avoid altogether.


void MediaFoundationSourceLoop();

void static __stdcall _OnVideoData(void *pData, int Size, long long llTimestamp, void *pUserData);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The C++ standard reserves identifiers that begin with an underscore and a capital letter.

static DWORD CALLBACK MediaFoundationSourceThread(LPVOID ptr)
{
MediaFoundationSourceInput *input = (MediaFoundationSourceInput *)ptr;
os_set_thread_name("win-MediaFoundation: MediaFoundationSourceThread");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
os_set_thread_name("win-MediaFoundation: MediaFoundationSourceThread");
os_set_thread_name("win-mediafoundation: MediaFoundationSourceThread");

Comment on lines +836 to +837
if (!properties.empty()) {
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the intent of this block?


inline bool MediaFoundationSourceInput::Activate(obs_data_t *settings)
{

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove unnecessary blank line.

Comment on lines +15 to +16
m_spBuffer = nullptr;
m_sp2DBuffer = nullptr;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these need to be manually set to nullptr in the destructor? I thought wil::com_ptr_nothrow and wil::com_ptr were smart pointers that should clean up after themselves.

VALUE "ProductName", "${OBS_PRODUCT_NAME}"
VALUE "ProductVersion", "${OBS_VERSION_CANONICAL}"
VALUE "Comments", "${OBS_COMMENTS}"
VALUE "LegalCopyright", "${OBS_LEGAL_COPYRIGHT}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may need to hardcode this line to use a specific copyright attribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
New Feature New feature or plugin Seeking Testers Build artifacts on CI
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

5 participants