Windows camera driver fixes

[sirzooro]

This patch set fixes multiple Windows-only issues in github.com/pion/mediadevices camera capture:

• Prevents a hard crash (access violation) during camera enumeration/open.
• Fixes capability enumeration/selection so GetUserMedia can find a usable camera mode.
• Prevents a panic during device shutdown caused by late DirectShow callbacks.

Root cause (crash in listResolution / openCamera)

• selectCamera() uses an inverted/ambiguous return convention:

  • It returns 1 when a camera is found (success).
  • It returns 0 when the camera is not found.
  • Bug: On the internal fail: path (e.g. COM/DirectShow enumeration failure), it also returned 1.

• Callers interpret selectCamera() as a boolean success predicate:

  • listResolution() does if (!selectCamera(...)) goto fail;.
  • When selectCamera() hit the fail: label, it returned 1, so listResolution() proceeded as if successful, even though moniker was still nullptr.
  • listResolution() then executed moniker->BindToObject(...), causing an access violation (0xc0000005).

This is particularly easy to trigger when the COM environment is not initialized on the current OS thread (COM init is per-thread on Windows), but the crash itself is due to the incorrect return value and missing defensive checks.

Changes

• Correct selectCamera() failure behavior: return 0 on the fail: path so callers correctly take the failure path.
• Add defensive checks in listResolution() and openCamera():
  • Verify moniker != nullptr before use.
  • Check BindToObject(...) return value; on failure, set an error string and bail out.
• Fix freeCameraList(): return 0 for success (the function’s comment says it frees resources; returning 1 incorrectly indicates failure).

Root cause (GetUserMedia fails with “failed to find the best driver that fits the constraints”)

mediadevices selects devices based on the driver’s reported Properties(). On Windows, the camera driver could report zero usable properties, causing selection to fail even when a camera exists.

Contributing issues:

• Some devices report stream caps as FORMAT_VideoInfo2 (not just FORMAT_VideoInfo), so the current capability parsing misses them.
• The Windows driver filtered properties to only those with native YUY2 FOURCC, which can be empty for cameras that primarily expose MJPG/etc through IAMStreamConfig even though DirectShow can still deliver YUY2 to the SampleGrabber via decoders/converters.

Changes (capability enumeration / reporting)

• Parse FORMAT_VideoInfo2 caps in listResolution() and normalize negative dimensions.
• Include the required headers so VIDEOINFOHEADER2 is available across Windows SDK toolchains (amvideo.h + dvdmedia.h).
• Relax Properties() filtering to advertise width/height modes regardless of the native FOURCC, and report YUY2 as the output format since the SampleGrabber requests YUY2 (DirectShow may insert converters/decoders).

Root cause (panic during shutdown)

During shutdown, DirectShow may deliver a final frame concurrently with Close(). The exported Go callback (imageCallback) attempted to send into c.ch after Close() had already closed it, causing:

panic: send on closed channel

Changes (shutdown race)

• Add a closed flag to the Windows camera driver.
• Make imageCallback return early if closed and use a non-blocking send.
• Guard the send with a small recover() to handle the narrow race where close happens between check and send.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 42.13%. Comparing base (932e23a) to head (85060bc).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #686   +/-   ##
=======================================
  Coverage   42.13%   42.13%           
=======================================
  Files          86       86           
  Lines        5186     5186           
=======================================
  Hits         2185     2185           
  Misses       2839     2839           
  Partials      162      162           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.