QSV Hardware Encoding - HardwareFramesContext.Initialize() Error

[Maizer]

Hi @asticode ,

Sorry to bother you again, but I've run into another issue that I can't seem to figure out.
I'm currently testing QSV hardware encoding/decoding with astiav (specifically transcoding MP4 to WebM), and I'm consistently encountering an error at the HardwareFramesContext.Initialize() step.
error:The hardware pixel format 'nv12' is not supported by the device type 'QSV'

I've also tried the hardware_encoding example provided, and it fails with the same error.
I've spent a considerable amount of time looking into this but haven't been able to pinpoint the cause.

1.OS : Windows11
2.FFMPEG : 7.0.2

Here's my logs:

qsv support : nv12
qsv support : p010le
2025/05/14 00:58:17 ffmpeg log: All info found - class: mov,mp4,m4a,3gp,3g2,mj2 [AVFormatContext] @ 0x18a6eb48980 - level: 48
2025/05/14 00:58:17 ffmpeg log: After avformat_find_stream_info() pos: 684019 bytes read:390998 seeks:4 frames:49 - class: mov,mp4,m4a,3gp,3g2,mj2 [AVFormatContext] @ 0x18a6eb48980 - level: 48
2025/05/14 00:58:17 ffmpeg log: Defaulting child_device_type to AV_HWDEVICE_TYPE_D3D11VA for oneVPL.Please explicitly set child device type via "-init_hw_device" option if needed. - class: AVHWDeviceContext [AVHWDeviceContext] @ 0x18a7059ee80 - level: 40
2025/05/14 00:58:18 ffmpeg log: Selecting d3d11va adapter 1 - class: AVHWDeviceContext [AVHWDeviceContext] @ 0x18a6eb73400 - level: 40
2025/05/14 00:58:18 ffmpeg log: Using device 8086:9a60 (Intel(R) UHD Graphics). - class: AVHWDeviceContext [AVHWDeviceContext] @ 0x18a6eb73400 - level: 32
2025/05/14 00:58:18 ffmpeg log: Use Intel(R) oneVPL to create MFX session, API version is 2.12, the required implementation version is 1.3 - class: AVHWDeviceContext [AVHWDeviceContext] @ 0x18a7059ee80 - level: 40
2025/05/14 00:58:22 ffmpeg log: Initialize MFX session: implementation version is 2.14 - class: AVHWDeviceContext [AVHWDeviceContext] @ 0x18a7059ee80 - level: 40
2025/05/14 00:58:22 ffmpeg log: Setting default whitelist 'file,crypto,data' - class: file [URLContext] @ 0x18a42fc9cc0 - level: 48
2025/05/14 00:58:22 ffmpeg log: Format nv12 chosen by get_format(). - class: hevc_qsv [AVCodecContext] @ 0x18a6eb51540 - level: 48
2025/05/14 00:58:22 ffmpeg log: Decoder: output is system memory surface - class: hevc_qsv [AVCodecContext] @ 0x18a6eb51540 - level: 40
2025/05/14 00:58:22 ffmpeg log: Use Intel(R) oneVPL to create MFX session with the specified MFX loader - class: hevc_qsv [AVCodecContext] @ 0x18a6eb51540 - level: 40
2025/05/14 00:58:22 ffmpeg log: Format nv12 chosen by get_format(). - class: hevc_qsv [AVCodecContext] @ 0x18a6eb51540 - level: 48
2025/05/14 00:58:22 ffmpeg log: Decoder: output is system memory surface - class: hevc_qsv [AVCodecContext] @ 0x18a6eb51540 - level: 40
2025/05/14 00:58:22 ffmpeg log: Use Intel(R) oneVPL to create MFX session with the specified MFX loader - class: hevc_qsv [AVCodecContext] @ 0x18a6eb51540 - level: 40
q.encCodecContext.HardwareFramesContext() <nil>
main: hardware frames context is nil
2025/05/14 00:58:22 ffmpeg log: The hardware pixel format 'nv12' is not supported by the device type 'QSV' - class: AVHWFramesContext [AVHWFramesContext] @ 0x18a426dc840 - level: 16
2025/05/14 00:58:22 **main: initializing hardware frame context failed: Function not implemented**
Process 10872 has exited with status 1
Detaching

And here are the relevant code:

package test1

import (
	"errors"
	"fmt"
	"log"
	"maizer_nasapp/core/ffmpeg/astiav"
	"strings"

	"github.com/asticode/go-astikit"
)

type QsvTranscode struct {
	c               *astikit.Closer
	decCodecContext *astiav.CodecContext
	encCodecContext *astiav.CodecContext

	inputFormatContext  *astiav.FormatContext
	outputFormatContext *astiav.FormatContext

	supported_hardware_pixel_format astiav.PixelFormat
	supported_hardware_type         astiav.HardwareDeviceType

	video_stream int
}

func FindCodec(codecName string, supported_hardware_type astiav.HardwareDeviceType) (pf astiav.PixelFormat, encCodec *astiav.Codec) {

	if encCodec = astiav.FindEncoderByName(codecName); encCodec == nil {
		return astiav.PixelFormatNone, nil
	}

	for _, p := range encCodec.HardwareConfigs() {
		// Valid hardware config
		if p.MethodFlags().Has(astiav.CodecHardwareConfigMethodFlagHwDeviceCtx) && p.HardwareDeviceType() == supported_hardware_type {
			pf = p.PixelFormat()
			break
		}
	}

	for _, p := range encCodec.HardwareConfigs() {
		// Valid hardware config
		if p.MethodFlags().Has(astiav.CodecHardwareConfigMethodFlagHwDeviceCtx) && p.HardwareDeviceType() == supported_hardware_type {
			fmt.Printf("%s support : %s \n", supported_hardware_type.Name(), p.PixelFormat())
		}
	}

	return
}

func (q *QsvTranscode) getPixelFormatCallback(pifs []astiav.PixelFormat) astiav.PixelFormat {

	for _, pif := range pifs {
		if pif == q.supported_hardware_pixel_format {
			return pif
		}
	}

	return astiav.PixelFormatNone
}

func (q *QsvTranscode) openInputFile(input string) (err error) {

	// Allocate input format context
	if q.inputFormatContext = astiav.AllocFormatContext(); q.inputFormatContext == nil {
		err = errors.New("main: input format context is nil")
		return
	}
	q.c.Add(q.inputFormatContext.Free)

	// Open input
	if err = q.inputFormatContext.OpenInput(input, nil, nil); err != nil {
		err = fmt.Errorf("main: opening input failed: %w", err)
		return
	}
	q.c.Add(q.inputFormatContext.CloseInput)

	// Find stream info
	if err = q.inputFormatContext.FindStreamInfo(nil); err != nil {
		err = fmt.Errorf("main: finding stream info failed: %w", err)
		return
	}

	// Loop through streams
	for _, is := range q.inputFormatContext.Streams() {

		// Only process audio or video
		if is.CodecParameters().MediaType() != astiav.MediaTypeVideo {
			continue
		}

		q.video_stream = is.Index()

		isCodec := is.CodecParameters()

		var decoderName = ""

		switch isCodec.CodecID() {
		case astiav.CodecIDH264:
			decoderName = "h264_qsv"
		case astiav.CodecIDHevc:
			decoderName = "hevc_qsv"
		case astiav.CodecIDVp9:
			decoderName = "vp9_qsv"
		case astiav.CodecIDVp8:
			decoderName = "vp8_qsv"
		case astiav.CodecIDAv1:
			decoderName = "av1_qsv"
		case astiav.CodecIDMpeg2Video:
			decoderName = "mpeg2_qsv"
		case astiav.CodecIDMjpeg:
			decoderName = "mjpeg_qsv"
		default:
			err = errors.New("main: codec is not supportted by qsv")
			return
		}

		// Create stream

		var decCodec *astiav.Codec = nil

		// Find decoder
		if decCodec = astiav.FindDecoderByName(decoderName); decCodec == nil {
			err = errors.New("main: codec is nil")
			return
		}

		// Allocate codec context
		if q.decCodecContext = astiav.AllocCodecContext(decCodec); q.decCodecContext == nil {
			err = errors.New("main: codec context is nil")
			return
		}
		q.c.Add(q.decCodecContext.Free)

		// Update codec context
		if err = is.CodecParameters().ToCodecContext(q.decCodecContext); err != nil {
			err = fmt.Errorf("main: updating codec context failed: %w", err)
			return
		}

		// Set framerate
		if is.CodecParameters().MediaType() == astiav.MediaTypeVideo {
			q.decCodecContext.SetFramerate(q.inputFormatContext.GuessFrameRate(is, nil))
		}

		supported_hardware_type := q.supported_hardware_type

		// supported_hardware_type = astiav.HardwareDeviceTypeD3D11VA

		var hardwareDeviceContext *astiav.HardwareDeviceContext = nil
		if hardwareDeviceContext, err = astiav.CreateHardwareDeviceContext(supported_hardware_type, "auto", nil, 0); hardwareDeviceContext == nil || err != nil {
			if err == nil {
				err = errors.New("main: hardware device context is nil")
			}
			return
		}
		q.c.Add(hardwareDeviceContext.Free)

		q.decCodecContext.SetHardwareDeviceContext(hardwareDeviceContext)
		q.decCodecContext.SetPixelFormatCallback(q.getPixelFormatCallback)
		q.decCodecContext.SetPkTimeBase(is.TimeBase())

		// Open codec context
		if err = q.decCodecContext.Open(decCodec, nil); err != nil {
			err = fmt.Errorf("main: opening codec context failed: %w", err)
			return
		}
	}

	return nil
}

func (q *QsvTranscode) openOutputFile(encCodec *astiav.Codec, output string) (err error) {

	// Allocate output format context
	if q.outputFormatContext, err = astiav.AllocOutputFormatContext(nil, "", output); q.outputFormatContext == nil || err != nil {
		err = errors.New("main: output format context is nil")
		return
	}
	q.c.Add(q.outputFormatContext.Free)

	var outputStream *astiav.Stream = nil

	// Create output stream
	if outputStream = q.outputFormatContext.NewStream(encCodec); outputStream == nil {
		err = errors.New("main: output stream is nil")
		return
	}

	// Allocate codec context
	if q.encCodecContext = astiav.AllocCodecContext(encCodec); q.encCodecContext == nil {
		err = errors.New("main: codec context is nil")
		return
	}
	q.c.Add(q.encCodecContext.Free)

	// If this is a file, we need to use an io context
	if !q.outputFormatContext.OutputFormat().Flags().Has(astiav.IOFormatFlagNofile) {
		// Open io context
		var ioContext *astiav.IOContext
		if ioContext, err = astiav.OpenIOContext(output, astiav.NewIOContextFlags(astiav.IOContextFlagWrite), nil, nil); err != nil {
			err = fmt.Errorf("main: opening io context failed: %w", err)
			return
		}
		q.c.AddWithError(ioContext.Close)

		// Update output format context
		q.outputFormatContext.SetPb(ioContext)
	}

	return nil
}

func MainTest6(i string, o string) {

	// Handle ffmpeg logs
	astiav.SetLogLevel(astiav.LogLevelDebug)
	astiav.SetLogCallback(func(c astiav.Classer, l astiav.LogLevel, fmt, msg string) {
		var cs string
		if c != nil {
			if cl := c.Class(); cl != nil {
				cs = " - class: " + cl.String()
			}
		}
		log.Printf("ffmpeg log: %s%s - level: %d\n", strings.TrimSpace(msg), cs, l)
	})

	q := &QsvTranscode{
		c:                       astikit.NewCloser(),
		supported_hardware_type: astiav.HardwareDeviceTypeQSV,
		// supported_hardware_type: astiav.HardwareDeviceTypeD3D11VA,
	}

	// ffmpeg -init_hw_device qsv=hw -filter_hw_device hw -hwaccel qsv -hwaccel_output_format qsv -i "input.mkv"
	// -c:v vp9_qsv -preset fast -b:v 2M -c:a libopus -b:a 128k -y "output.webm"

	//ffmpeg -init_hw_device qsv=hw -filter_hw_device hw -hwaccel qsv -hwaccel_output_format qsv -i "input.mkv" -c:v vp9_qsv -preset fast -b:v 2M -c:a libopus -b:a 128k -y "output.webm"

	pf, encodec := FindCodec("vp9_qsv", q.supported_hardware_type)

	q.supported_hardware_pixel_format = pf

	if err := q.openInputFile(i); err != nil {
		fmt.Println(err)
		return
	}

	if err := q.openOutputFile(encodec, o); err != nil {
		fmt.Println(err)
		return
	}

	dec_pkt := astiav.AllocPacket()

	for {
		// We use a closure to ease unreferencing the packet
		if stop := func() bool {
			// Read frame
			if err := q.inputFormatContext.ReadFrame(dec_pkt); err != nil {
				if errors.Is(err, astiav.ErrEof) {
					return true
				}
				log.Fatal(fmt.Errorf("main: reading frame failed: %w", err))
			}

			// Make sure to unreference the packet
			defer dec_pkt.Unref()

			if dec_pkt.StreamIndex() != q.video_stream {
				return false
			}

			err := q.decCodecContext.SendPacket(dec_pkt)
			if err != nil {
				fmt.Println("decCodecContext.SendPacket:", err)
				return false
			}

			var frame *astiav.Frame = nil

			for {

				if stop := func() bool {
					if frame == nil {
						frame = astiav.AllocFrame()
					}

					err = q.decCodecContext.ReceiveFrame(frame)
					if err != nil {
						if errors.Is(err, astiav.ErrEof) || errors.Is(err, astiav.ErrEagain) {
							return true
						}
						fmt.Println("decCodecContext.ReceiveFrame:", err)
						return false
					}

					defer frame.Unref()

					if q.encCodecContext.HardwareFramesContext() == nil {

						fmt.Println("q.encCodecContext.HardwareFramesContext()", frame.HardwareFramesContext())

						if q.decCodecContext.HardwareFramesContext() == nil {
							fmt.Println("main: hardware frames context is nil")
							// return true
						}

						// hardwareFramesContext := q.decCodecContext.HardwareFramesContext()

						hardwareDeviceContext := q.decCodecContext.HardwareDeviceContext()
						if hardwareDeviceContext == nil {
							fmt.Println("main: hardware device context is nil")
							return true
						}
						sup_type := q.supported_hardware_pixel_format
						hardwareFramesContext := astiav.AllocHardwareFramesContext(hardwareDeviceContext)
						if hardwareFramesContext == nil {
							log.Fatal("main: hardware frames context is nil")
						}
						defer hardwareFramesContext.Free()

						// Set hardware frame content
						hardwareFramesContext.SetHardwarePixelFormat(sup_type)
						hardwareFramesContext.SetSoftwarePixelFormat(astiav.PixelFormatYuv420P)
						hardwareFramesContext.SetWidth(q.decCodecContext.Width())
						hardwareFramesContext.SetHeight(q.decCodecContext.Height())
						hardwareFramesContext.SetInitialPoolSize(20)

						// Initialize hardware frame context
						if err := hardwareFramesContext.Initialize(); err != nil {
							log.Fatal(fmt.Errorf("main: initializing hardware frame context failed: %w", err))
						}

						q.encCodecContext.SetWidth(q.decCodecContext.Width())
						q.encCodecContext.SetHeight(q.decCodecContext.Height())
						q.encCodecContext.SetPixelFormat(q.supported_hardware_pixel_format)
						q.encCodecContext.SetTimeBase(q.decCodecContext.Framerate())
						q.encCodecContext.SetHardwareFramesContext(hardwareFramesContext)

						enc_stream := q.outputFormatContext.NewStream(encodec)

						if enc_stream == nil {
							fmt.Println("main: output stream is nil")
							return true
						}

						enc_stream.SetTimeBase(q.encCodecContext.TimeBase())
						err = enc_stream.CodecParameters().FromCodecContext(q.encCodecContext)
						if err != nil {
							fmt.Println("main: from codec context failed:", err)
							return true
						}

						err = q.outputFormatContext.WriteHeader(nil)
						if err != nil {
							fmt.Println("main: write header failed:", err)
							return true
						}
					}

					return false
				}(); stop {
					break
				}
			}

			frame.SetPts(astiav.RescaleQ(frame.Pts(), q.decCodecContext.PktTimeBase(), q.encCodecContext.TimeBase()))

			return false
		}(); stop {
			break
		}
	}

}
```

Could you please take a look when you have a moment?
Thank you very much for your time and help.

[Maizer]

Hey, just a quick update: I managed to figure out the issue I mentioned earlier.
It turns out the encCodec.HardwareConfigs method，doesn't actually retrieve the valid pixel formats supported by a specific hardware instance. These need to be fetched using the av_hwdevice_get_hwframe_constraints function.
I've pushed a PR with a fix for this. Please take a look when you get a chance.

[asticode]

👍