ProresNV

CUDA-accelerated Apple ProRes encoder integrated into FFmpeg as prores_nv. Produces standard icpf bitstreams decodable by ffmpeg, QuickTime, and any ProRes-compatible player. Full GPU pipeline (DCT + quantization + VLC entropy coding) for frames ≥ 1920×1080, with a multi-threaded CPU VLC fallback for small frames. Competitive bitrates vs prores_ks with higher PSNR.

License

ProresNV inherits FFmpeg's license: the encoder is derived from FFmpeg's prores_ks implementation — encode_vlc_codeword (Rice/Golomb entropy coding), the ff_jpeg_fdct_islow_10 port, the QUANT_MAT_* quantization matrices, and the ProRes slice/block layout — and the prores_nv plugin builds inside FFmpeg's libavcodec. The combined work is therefore covered by the GNU General Public License version 2.0 or later (GPL-2.0-or-later).

• The full license text is in LICENSE, byte-identical to FFmpeg's COPYING.GPLv2.
• This matches FFmpeg's default build (GPL-2.0-or-later). FFmpeg may also be configured for LGPL-2.1-or-later; when building ProresNV into such a configuration, the licensing of the combined work must be reviewed accordingly (the plugin is part of libavcodec and inherits the configuration's license).

What is implemented

• Full ProRes profile range: 422 Proxy, 422 LT, 422, 422 HQ, 4444, 4444 XQ
• 10-bit and 12-bit input support
• Chroma formats: 4:2:2, 4:4:4, and 4:4:4:4 (with alpha channel)
• GPU DCT + quantization + zigzag for 8×8 blocks when CUDA is available
• All profiles run the same async multi-stream GPU pipeline (Y/U/V/A on independent CUDA streams, pinned-memory transfers)
• Automatic CPU fallback for environments without CUDA
• Per-profile quantization matrices matching Apple ProRes specification (ffmpeg QUANT_MAT_*)
• Fused GPU VLC: entropy coding (Rice/Golomb hybrid, encode_vlc_codeword compatible) runs on the GPU for frames ≥ 1920×1080 — a single fused kernel (estimate → atomic reserve → encode, one thread per 2-MB slice), byte-identical to the CPU path; small frames use a persistent multi-threaded CPU pool
• Proper Apple ProRes icpf frame format, standard MOV container
• 60+ dB PSNR (luma and chroma; 65+ dB at 4444 XQ q=4; 64+ dB 12-bit 4444), with the GPU path byte-identical to the CPU VLC path at the same quantizer

Performance

Measured with the ffmpeg plugin (prores_nv) on testsrc2-derived sources (q=4), Windows 11 / CUDA 13.3:

Case	2026-07 baseline	GPU VLC (08-01)	Async pipeline (08-01)	Kernel rewrite (08-01)
1080p 422 HQ, 30 frames	0.49 s	0.36 s	0.293 s	0.19 s
1080p 4444 XQ, 30 frames	0.55 s	0.46 s	0.337 s	0.21 s
4K 422 HQ, 10 frames	0.51 s	0.237 s	0.245 s	0.22 s

Steady state, ffmpeg plugin (prores_nv vs prores_ks, same command):

Source	prores_nv	prores_ks	Speedup
1080p 422 HQ, 300 frames	3.1 ms/f (9.7×RT)	5.85 ms/f (5.1×RT)	1.9×
4K 422 HQ, 40 frames	11.5 ms/f (2.6×RT)	24.1 ms/f (1.2×RT)	2.1×

Library (encmov, no ffmpeg overhead): 1080p 2.2 ms/f (460 fps), 4K 7.9 ms/f (127 fps). Host-side per-frame budget (1080p): submit 0.09 ms

• D2H 0.19 ms + assembly 0.12 ms ≈ 0.4 ms host cost/frame; the GPU is fully hidden behind the host. At 4K: D2H 0.67 ms + assembly 0.55 ms ≈ 1.3 ms host cost (the GPU's 4K kernels run concurrently).

• Pipeline exhaust (2026-08-01, session 4c): submit is split into a stage/finish pair (pnv_encode_stage copies the planes into pinned staging — pure host work; pnv_encode_submit_finish queues the H2D + kernels and advances the pipeline state). The ffmpeg plugin runs the stage on a persistent worker thread (Windows _beginthreadex + events) while the main thread collects the previous frame, then finishes the submit after the copy completes — the plane copies now overlap the collect instead of sitting on the critical path. Frame assembly is split across a small persistent pool (AssembleProResFrameDirectParallel, chunked slice claiming: a per-slice atomic counter was measured at ~0.7 ms of pure contention at 4K, so slices are claimed 128 at a time) — 4K assembly 0.93 → 0.55 ms. The VLC-total downloads in the collect are now async (one cudaMemcpyAsync per plane stream + a single sync instead of four serialized blocking copies) — D2H dropped from 0.98 to 0.19 ms at 1080p and 1.56 to 0.67 ms at 4K. encmov allocates its input planes pinned and double-buffered; the stage detects the pinned source (cudaPointerGetAttributes) and skips the staging copy entirely, so H2D runs straight from the caller's buffer (safe: frame N's H2D completes at collect(N), before fread(N+2) overwrites buffer N%2). A CUDA init warm-up (cudaFree(0) + a 1-block DCT launch + symbol upload) runs on the first submit, moving the ~90 ms context/JIT cost out of the first frame. Stage copies ≥ 2 MB use a 6-thread copy pool (~4× faster than a single memcpy at 4K). All output remains byte-identical (bench/ab_test.ps1 6/6, plugin == encmov).
• Kernel rewrite (2026-08-01, session 4): the DCT kernel went from one thread per 8×8 block (133 registers → 12.5% occupancy) to one warp per block (21 registers; lanes 0-7 run the row pass, then the column pass, via shared memory; vectorized 32-bit loads) — 4K DCT dropped from ~9 ms to ~0.5 ms. The VLC bit writer is now register-staged (bits accumulate in a 64-bit register, whole bytes flushed) instead of a dependent byte read-modify-write per bit, and the slice geometry uses 2-MB-wide slices (kMaxSliceMbWidth = 2) — 4× more threads for the per-slice entropy coding (12,240 threads at 1080p vs 3,060). GPU time per 1080p frame: 0.59 ms (DCT 0.11 + VLC ~0.5). The collect now syncs only its own set's streams instead of the whole device. Cost: ~2.6% larger files at 1080p, ~8% at 4K (extra slice headers + per-slice raw DC codewords — still valid ProRes; decoders read the slice width from the picture header). As of 2026-08-13 the slice width is adaptive: 4-MB slices on ≥4K-class frames (mb_width × mb_height ≥ 30000) halve the per-slice overhead at 4K (−4.9% size, −2.4% wall), 2-MB elsewhere.
• Fused GPU VLC + async pipeline (2026-08-01, session 3): the entropy coding is a single fused kernel (estimate → atomic slice reservation → encode, one thread per slice) and the encoder is double-buffered: SubmitFrame(N+1) launches the GPU work without waiting, so the GPU encodes frame N+1 while the host assembles frame N's packet. The icpf frame is assembled directly from the GPU's VLC blobs (no ~16K per-slice vector allocations per 4K frame). The ffmpeg plugin runs this pipeline with AV_CODEC_CAP_DELAY; encmov and the C API expose the same submit/collect interface. Output is byte-identical to the CPU VLC path in all profiles (bench/ab_test.ps1, SHA-256 verified), and the plugin's output is byte-identical to encmov's including the flush-drained last frame.
• GPU VLC (2026-08-01, session 2): the 4K time is 0.237 s vs 0.456 s for the CPU-VLC build — the entropy-coding stage (was ~17 ms/frame at 4K) now runs on the GPU.
• Small frames (< 1.5 Mpx, e.g. 720p) use the CPU VLC pool: the GPU path's fixed per-frame transfer costs exceed the VLC time it saves below 1080p.
• 12-bit encoding fixed (2026-08-01): 12-bit 4444 was ~7 dB garbage (wrong DC centering/coefficient scale for the decoder's 12-bit IDCT) and is now 64.1 dB at q=4 (66.3 dB at 4444XQ q=4). 10-bit output is unchanged.
• Steady-state 4K 422 HQ host budget (s4c): submit ~0.2 ms + D2H 0.67 ms + assembly 0.55 ms ≈ 1.3 ms/frame — the GPU's 4K kernels run concurrently and are fully hidden behind the host work.
• Perf batch (2026-08-13): four byte-exact measures — dead per-slice memsets removed (−10% on the fused VLC kernel), __clz codeword lengths, adaptive 4-MB slices at 4K, and a CollectSet rework (counter+meta+blob in one contiguous region, one D2H copy per plane, 12→8 stream syncs). Plugin steady state vs prores_ks: 1080p 2.47 vs 6.27 ms/f (2.54×), 4K 10.34 vs 24.10 ms/f (2.33×). All output byte-identical (bench/ab_test.ps1 6/6 after each measure; details in docs/status.md).

Profiles

Profile	Enum	Chroma	Max Bit Depth	Alpha	Typical Use
422 Proxy	k422Proxy	4:2:2	12	No	Offline proxy
422 LT	k422LT	4:2:2	12	No	Light mastering
422	k422	4:2:2	12	No	Standard mastering
422 HQ	k422HQ	4:2:2	12	No	High-quality mastering
4444	k4444	4:4:4	12	Optional	VFX / compositing
4444 XQ	k4444XQ	4:4:4	12	Optional	Highest quality (visually lossless)

Build

cmake -S . -B build
cmake --build build

Full end-to-end guide (library → FFmpeg plugin → validation → benchmarks, Windows/CUDA specifics, and the known gotchas): BUILDING.md.

Test

ctest --test-dir build --output-on-failure

---

Architecture

High-Level Design

flowchart TD
    IN["Input: Y/U/V(/A) planes<br/>10-bit or 12-bit uint16"] --> DECIDE{"Frames ≥ 1.5 Mpx<br/>and CUDA available?"}

    DECIDE -->|"Yes"| STAGE["Stage: copy planes into pinned staging<br/>(plugin: on a worker thread, overlapped with<br/>the previous frame's collect; pinned input<br/>skips the copy entirely)"]
    DECIDE -->|"No / small frames"| CDCT["JpegFdctIslow10 CPU DCT<br/>+ quant + zigzag (or GPU DCT)"]

    STAGE --> FINISH["Finish: queue H2D + kernels<br/>(main thread, advances the double-buffer state)"]
    FINISH --> DCT["JpegIslow10DctQuantizeKernel<br/>one warp per 8×8 block<br/>DCT + quant, writes ProRes scan order"]
    DCT --> VLC["ProResVlcFusedKernelGeom<br/>one thread per 2-MB slice:<br/>estimate bits → atomic reserve (offset, bytes)<br/>→ register-staged VLC encode"]

    CDCT --> CD2H["D2H coefficients"]
    CD2H --> CVLC["Persistent CPU VLC thread pool<br/>row stealing, direct slice writes"]

    VLC --> COLLECT["Collect: sync only this set's streams<br/>double-buffered: GPU encodes frame N+1<br/>while the host assembles frame N"]
    COLLECT --> D2H["Async D2H: VLC totals + slice meta + blob<br/>(per-plane streams, one sync each)"]
    D2H --> ASM["AssembleProResFrameDirectParallel<br/>icpf header + slice table + slice data<br/>(8-thread pool, chunked claiming)"]
    ASM --> MOV["MOV muxer"]
    MOV --> OUT["Output: .mov file"]

    CVLC --> ASM

Loading

Component Map

Component	File	Role
Public API	include/proresnv_encoder.h	ProresEncoder class, EncoderConfig, EncodeStatus, persistent VLC pool handle
CPU encoding + frame assembly	src/proresnv_encoder.cpp	Async multi-plane orchestration (EncodePlanes), slice layout, MB-major VLC packing, persistent thread pool, serial + parallel (AssembleProResFrameDirectParallel) frame assembly
FDCT implementations	src/proresnv_fdct.cpp	JpegFdctIslow10 (CPU fallback), FdctIfast (unused)
GPU encoding kernel	src/proresnv_encoder_cuda.cu	JpegIslow10DctQuantizeKernel (warp-per-block DCT + quant + zigzag — 21 regs, 256 B smem), 4 CUDA streams (Y/U/V/A), pinned host staging, fused GPU VLC kernel (ProResVlcFusedKernelGeom — estimate + atomic reservation + register-staged encode, one thread per 2-MB slice), two-phase submit (EncodeFrameGpuVlcStage / EncodeFrameGpuVlcSubmitAfterStage), 6-thread staging-copy pool, CUDA init warm-up
MOV container writer	src/mov_muxer.cpp	Standard QuickTime/MOV container with 64-bit co64 atoms
C API wrapper	src/proresnv_c_api.cpp	C-callable wrappers for FFmpeg integration (pnv_encode_*, two-phase pnv_encode_stage / pnv_encode_submit_finish)
CLI encoder tool	tools/encmov.cpp	proresnv_encmov: reads raw YUV, encodes, writes .mov
Unit tests	tests/proresnv_encoder_test.cpp	Frame integrity, idempotency, header format verification

Data Flow per Frame

1. Input: 16-bit unsigned plane samples (10-bit or 12-bit, uint16_t)
2. Submit (EncodePlanes): all planes are uploaded through pinned staging buffers (cudaHostAlloc), so cudaMemcpyAsync is truly asynchronous on Windows; each plane gets its own CUDA stream — Y, U, V, A DCTs run concurrently
3. GPU DCT (JpegIslow10DctQuantizeKernel):
   • JpegFdctIslow10 on GPU — exact port of ffmpeg's ff_jpeg_fdct_islow_10
   • Uses int (32-bit) intermediates to prevent 4K overflow
   • Sign-aware rounding quantization: (val >= 0) ? (val + d/2)/d : (val - d/2)/d
   • Writes coefficients already in ProRes progressive-scan (zigzag) order via a constant-memory LUT (kProResScanInvDevice) — the CPU reorder pass is eliminated
4. GPU VLC (frames ≥ 1920×1080): coefficients stay on the device — a single fused kernel per plane (one thread per 2-MB slice) computes the exact VLC bit count, atomically reserves a byte region in the output blob (nondeterministic order, recorded in per-slice (offset, bytes) meta), zeroes it, and writes the Rice/Golomb bitstream. The host downloads only the meta and the blob (no per-slice vectors, no prefix sums). Small frames go through the CPU path instead (steps 5–8 below) because the GPU path's fixed transfer costs exceed the VLC time it saves below 1080p.
5. Download: pinned D2H copy per plane, then a fast host copy into the plane's coefficient vector (zigzag order preserved)
6. Slice split + MB-major reorder: EncodeSliceVlcFull gathers each slice's blocks from the plane (raster) and reorders them to MB-major (column-major for 444 chroma)
7. Multi-threaded VLC (CPU path): persistent per-encoder pool (≤24 threads, spawned once) — workers claim macroblock rows via atomic fetch_add and write VLC bytes directly into the per-slice output vectors (EncodeDcsProRes + EncodeAcsProRes in parallel across rows)
8. Frame assembly: standard Apple ProRes icpf atom with 148-byte frame header, 8-byte picture header, per-slice 6-byte headers (422/444) or 8-byte headers (4444 with alpha)

Async Submit/Collect Pipeline (frames ≥ 1920×1080)

The encoder is double-buffered: SubmitFrame(N+1) uploads the planes and launches the DCT + fused VLC kernels without synchronizing, so the GPU works on frame N+1 while the host assembles frame N's packet (CollectFrame = settle + D2H meta/blob + AssembleProResFrameDirectParallel, which copies slice data straight from the blobs). End of stream uses DrainFrame for the last frame. The ffmpeg plugin drives this with AV_CODEC_CAP_DELAY (packets come out one call late; the flush callback drains the final frame).

Submit is split into stage (copy the planes into pinned staging — pure host work, no CUDA calls on the hot path) and finish (queue H2D + kernels + advance the double-buffer state). The plugin runs the stage on a persistent worker thread while the main thread collects the previous frame; when the copy is done the main thread calls pnv_encode_submit_finish and always waits for the worker before returning (so ffmpeg's frame buffers are never touched after the callback returns). If the source planes are already pinned (encmov's double-buffered cudaHostAlloc input), the stage skips the copy and the H2D sources straight from the caller's buffer (Set::h2d_src). The collect's per-plane VLC-total downloads are async on the plane streams (one sync each) so the four totals run in parallel — four serialized blocking copies cost ~0.8 ms of pure latency at 1080p. AssembleProResFrameDirectParallel splits the per-slice copies across a small persistent pool (chunked claiming: 128 slices per atomic).

MB-Major Block Ordering

This is critical and non-obvious. Within each slice, ProRes expects blocks in MB-major order:

For a slice with 4 MBs (luma, 422):
  ffmpeg order: [MB0_TL, MB0_TR, MB0_BL, MB0_BR, MB1_TL, MB1_TR, MB1_BL, MB1_BR, ...]
  raster order: [MB0_TL, MB0_TR, MB1_TL, MB1_TR, ..., MB0_BL, MB0_BR, MB1_BL, ...]

The raster → MB-major mapping happens inside EncodeSliceVlcFull in proresnv_encoder.cpp. For 422 chroma, blocks_per_mb=2; for luma/alpha and 444 chroma, blocks_per_mb=4.

444 chroma is column-major within each MB — a subtle trap: ffmpeg's get_slice_data gathers chroma blocks as [TL, BL, TR, BR] (not [TL, TR, BL, BR] like luma), and the decoder places them the same way. Sending luma order for 444 chroma transposes the two bottom blocks of every macroblock, costing ~45 dB of chroma PSNR. EncodeSliceVlcFull takes a column_major flag that the 444 chroma call sites set (uv_bpm == 4).

API

#include "proresnv_encoder.h"

proresnv::EncoderConfig config;
config.width     = 1920;
config.height    = 1080;
config.profile   = proresnv::ProResProfile::k422HQ;
config.bit_depth = 10;
config.quality   = 0;   // 0 = default for profile (typically 4)

proresnv::ProresEncoder encoder(config);

std::vector<uint8_t> packet;
encoder.EncodeFrame422P10(
    y_plane, y_stride,
    u_plane, u_stride,
    v_plane, v_stride,
    &packet);  // packet is a standard ProRes icpf frame

Entry Points

Method	Chroma	Bit Depth	Planes
EncodeFrame422P10	4:2:2	10	Y, U, V
EncodeFrame422P12	4:2:2	12	Y, U, V
EncodeFrame444P10	4:4:4	10	Y, U, V
EncodeFrame444P12	4:4:4	12	Y, U, V
EncodeFrame4444P10	4:4:4:4	10	Y, U, V, A
EncodeFrame4444P12	4:4:4:4	12	Y, U, V, A

All methods produce a self-contained ProRes icpf frame in output_packet. Pass this directly to the MOV muxer or any ProRes-capable container writer.

Output Format

The output std::vector<uint8_t> from each EncodeFrameXXX call is a standard Apple ProRes icpf atom:

Bytes  0– 3:  frame_size (uint32 BE) — total size including this field
Bytes  4– 7:  "icpf"  — ProRes frame magic
Bytes  8– 9:  frame_hdr_size (uint16 BE) = 148
Bytes 10–11:  version = 0
Bytes 12–15:  vendor = "appl"
Bytes 16–17:  width (uint16 BE)
Bytes 18–19:  height (uint16 BE)
Byte  20:     chroma_flags (0x80=422, 0xC0=444)
Byte  21:     reserved
Byte  22:     color_primaries
Byte  23:     color_trc
Byte  24:     colorspace
Byte  25:     alpha_bits >> 3
Byte  26:     reserved
Byte  27:     quant_flags = 0x03 (both luma and chroma matrices present)
Bytes 28–91:  luma quantization matrix (64 bytes, row-major)
Bytes 92–155: chroma quantization matrix (64 bytes, row-major)
Byte  156:    pic_hdr_bits = 0x40 (8-byte picture header)
Bytes 157–160: pic_data_size (uint32 BE) — excludes the pic_hdr_bits byte
Bytes 161–162: slice_count (uint16 BE)
Byte  163:    log2(mbs_per_slice) << 4
Bytes 164+:   slice size table (uint16 BE × slice_count)
              slice data (variable)

Each slice header is 2 × num_planes bytes:

Byte 0:   (slice_hdr_size_bytes << 3)  — e.g. 0x30 for 422 (6 bytes)
Byte 1:   quantizer (1–224)
Bytes 2–3: Y plane size (uint16 BE)
Bytes 4–5: U plane size (uint16 BE)
[Bytes 6–7: V plane size (uint16 BE)]  — 4444 only; V is inferred for 422/444

---

Key Design Decisions

Why JpegFdctIslow10 not FdctIfast? The islow DCT (ff_jpeg_fdct_islow_10) is what ffmpeg's prores_ks encoder actually uses for 10-bit encoding. FdctIfast (which ffmpeg uses for 8-bit JPEG) has different scaling and produces different coefficient ranges. Using the wrong one causes ~10 dB PSNR instead of 60+ dB.

Why MB-major block ordering? ProRes's encode_dcs/encode_acs functions, and their decoder counterparts, process blocks in the order they appear in get_slice_data — which iterates macroblock by macroblock, then fills all 4 (or 2) blocks within each MB before moving to the next MB. Raster order causes different DC delta chains, breaking DC prediction and scrambling AC interleaving. 444 chroma uses column-major within the MB (see above).

Why slice_hdr_size = 2 × num_planes? The ProRes slice header stores the sizes of all planes except the last, which is inferred from the slice's total size in the seek table. For 422 (3 planes), this is 6 bytes (1B hdr_bits + 1B quant + 2B Y_sz + 2B U_sz); V_sz is implicit. Our original 8-byte header shifted all VLC data and caused completely wrong decoded values.

Why pic_data_size excludes the first byte? The pic_data_size field in the ProRes picture header counts bytes starting after the pic_hdr_bits byte, matching ffmpeg's convention. The total frame_size therefore includes icpf(8) + frame_hdr(148) + 1 + pic_data_size.

Why GPU VLC? The VLC stage was the CPU bottleneck (~17 ms/frame at 4K). The fused GPU path runs DCT → estimate → encode entirely on the device and is byte-identical to the CPU VLC (validated by bench/ab_test.ps1 across all profiles). Two subtle correctness requirements: the AC run counter carries across scan positions (ffmpeg encode_acs declares run at function scope — resetting it per position diverges the bitstream), and 444 chroma blocks are indexed column-major within each MB. For 12-bit input the coefficients are scaled by 1/4 (the decoder's 12-bit IDCT path has 4× the 10-bit gain) with the DC centered at 2048 — which lands on the same 0x4000 FDCT bias after scaling.

---

Postmortem: CUDA Build on Windows

Environment: Windows 11, CMake 4.4.0-rc3, Visual Studio 2022 BuildTools (MSVC 19.44), CUDA 13.3

Issue 1 — CMake Visual Studio generator cannot find CUDA toolset

CMake Error: No CUDA toolset found.

Root cause: CMake 4.4's Visual Studio generator relies on CUDA MSBuild integration files (.props / .targets / .xml) being registered in the VS BuildCustomizations directory. These files shipped with CUDA 13.3 but were not copied to the VS 2022 BuildTools MSBuild tree — CUDA's installer detected only the full VS IDE, not the standalone BuildTools, as a valid integration target.

Resolution: Switched to the Ninja generator. Ninja does not depend on MSBuild toolset discovery; it invokes nvcc directly. Steps:

1. Downloaded ninja.exe (v1.12.1)
2. Ran vcvars64.bat to prime the MSVC environment
3. Configured with -G Ninja -DCMAKE_CUDA_COMPILER=<path to nvcc>

Issue 2 — Unresolved external EncodePlaneCuda at link time

error LNK2019: unresolved external symbol
  "proresnv::`anonymous namespace'::EncodePlaneCuda(...)"

Root cause: Anonymous namespaces produce internal linkage. The forward declaration in proresnv_encoder.cpp was inside namespace { ... }, while the definition was in a different anonymous namespace in proresnv_encoder_cuda.cu. The linker treated them as two separate, unreachable symbols.

Resolution: Moved EncodePlaneCuda and friends to proresnv:: namespace scope in both files — only the CUDA kernel helpers remain anonymous.

Issue 3 — goto fail crosses variable initialization

Root cause: CUDA error-handling pattern used goto fail to jump to cleanup. Variables declared between a goto source and its target triggered warnings about skipped constructors.

Resolution: Moved variables to function top or wrapped them in {} block scopes at their point of use.