Memory Leak: Player Destructor Does Not Free Allocated Memory on Linux

[vakiller]

Memory Leak: Player Destructor Does Not Free Allocated Memory on Linux

Describe the bug

The mdk-sdk Player class does not properly release allocated memory when destroyed on Linux. After creating and destroying Player instances, 67-83% of the allocated memory remains leaked and is never returned to the operating system. This occurs even when following the complete cleanup sequence: stopping playback, waiting for state change, unloading media, deleting the player, and destroying OpenGL contexts.

The memory leak appears to originate from:

1. Global/static video frame buffer pools that are never released
2. FFmpeg decoder contexts (AVCodecContext/AVFormatContext) not being properly freed in the Player destructor
3. Frame allocators retaining ~12 MB per buffered frame across all players

This issue has been reproduced in both C++ standalone applications and Flutter desktop applications on Ubuntu 22.04, indicating it is a core library issue rather than language-binding specific.

To Reproduce

Test Setup

• Number of streams: 10 simultaneous RTSP H.265 streams
• Resolution: 2304×1296, 20fps
• Decoders tested: FFmpeg (software), VDPAU (NVIDIA hardware)

Steps to reproduce the behavior:

1. Create 10 mdk Player instances, each with its own OpenGL context
2. Load RTSP streams using setMedia(url)
3. Set decoder priority: setDecoders(MediaType::Video, {"VDPAU", "VAAPI", "FFmpeg"})
4. Start playback: set(State::Playing)
5. Wait for streams to play (~10-15 seconds, memory usage climbs to peak)
6. Perform complete cleanup sequence:
   • set(State::Stopped) - Stop playback
   • waitFor(State::Stopped, 5000) - Wait for async stop to complete
   • setMedia(nullptr) - Unload media
   • delete player - Destroy player instance
   • Destroy SDL_GLContext for each player
   • Destroy SDL_Window for each player
7. Measure memory at each cleanup step using /proc/[pid]/status VmRSS

Reproduction Code

Complete test application available at: test_10_players.cpp

Key code snippet:

// Create players
for (int i = 0; i < NUM_PLAYERS; i++) {
    players[i] = new Player();
    players[i]->setMedia(rtsp_url);
    players[i]->setDecoders(MediaType::Video, {"VDPAU", "VAAPI", "FFmpeg"});
    players[i]->setVideoSurfaceSize(WINDOW_WIDTH, WINDOW_HEIGHT);
    players[i]->set(State::Playing);
}

// Complete cleanup sequence
for (int i = 0; i < NUM_PLAYERS; i++) {
    // Step 1: Stop playback
    players[i]->set(State::Stopped);
    players[i]->waitFor(State::Stopped, 5000);

    // Step 2: Unload media
    players[i]->setMedia(nullptr);

    // Step 3: Delete player
    delete players[i];
    players[i] = nullptr;
}

// Step 4: Destroy GL contexts
for (int i = 0; i < NUM_PLAYERS; i++) {
    SDL_GL_DeleteContext(glContexts[i]);
}

// Step 5: Destroy windows
for (int i = 0; i < NUM_PLAYERS; i++) {
    SDL_DestroyWindow(windows[i]);
}

Expected behavior

After executing the complete cleanup sequence (stop, wait, unload, delete, destroy contexts/windows), the application's memory usage should return to near-baseline levels (within 5-10% of initial memory usage). The Player destructor should:

1. Release all FFmpeg decoder contexts (AVCodecContext, AVFormatContext)
2. Free all video frame buffers from internal pools
3. Return allocated memory to the operating system

Expected memory profile (for 10 streams):

• Initial: ~57 MB
• Peak (playing): ~350-1400 MB (depending on decoder)
• After cleanup: ~60-70 MB (baseline + small overhead)

Actual behavior

Memory is NOT freed after Player destruction. The majority of allocated memory remains in use indefinitely.

FFmpeg Software Decoder Results (10 streams):

Stage	Memory (MB)	Change	% of Used Memory Freed
Initial (baseline)	57	-	-
Playing (peak)	1,450	+1,393	-
After set(State::Stopped) + waitFor()	1,281	-169	12%
After setMedia(nullptr)	1,281	-0	0%
After delete player	1,203	-78	6%
After destroy GL contexts	1,200	-3	0%
After destroy windows	1,196	-4	0%
TOTAL FREED	-	-242 MB	17%
LEAKED	1,196	+1,151 MB	83% LEAKED

VDPAU Hardware Decoder Results (10 streams):

Stage	Memory (MB)	Change	% of Used Memory Freed
Initial (baseline)	57	-	-
Playing (peak)	407	+350	-
After set(State::Stopped) + waitFor()	308	-99	28%
After setMedia(nullptr)	308	-0	0%
After delete player	293	-15	4%
After destroy GL contexts	290	-3	1%
After destroy windows	286	-4	1%
TOTAL FREED	-	-121 MB	33%
LEAKED	286	+236 MB	67% LEAKED

Key Findings:

1. setMedia(nullptr) frees 0 MB - Unloading media has no effect on memory
2. delete player frees only 4-6% of used memory - Destructor is not cleaning up properly
3. Leak persists after destroying GL contexts and windows - Memory is held inside mdk-sdk
4. Leak accumulates with multiple create/destroy cycles - Each cycle adds more leaked memory

Memory Calculation Analysis:

For H.265 2304×1296 @ 20fps with 10 streams:

• Frame size (YUV420): 2304 × 1296 × 1.5 = ~4.5 MB per frame
• Frames buffered per stream: ~10-15 frames (estimated)
• Total for 10 streams: 4.5 MB × 10-15 frames × 10 streams = 450-675 MB

With FFmpeg decoder overhead and additional allocations, the ~1.2 GB leak matches the expected size of frame buffers that are never released.

Flutter Desktop Confirmation

This issue has also been observed in Flutter desktop applications using fvp (Flutter Video Player with mdk backend) on the same system. The memory leak behavior is identical:

• Memory usage climbs during video playback
• Disposing video player widgets does not free memory
• Memory accumulates across multiple player lifecycle operations
• Application must be restarted to reclaim memory

This confirms the leak is in the mdk-sdk core library, not specific to C++ bindings or usage patterns.

Environment

• OS: Ubuntu 22.04.5 LTS (Linux 6.8.0-87-generic)
• Architecture: x86_64 (amd64)
• GPU: NVIDIA (VDPAU available)
• mdk-sdk version: 0.35.0 (mdk-sdk-linux.tar.xz from SourceForge)
• Compiler: GCC 11.4.0
• FFmpeg: Built-in with mdk-sdk
• Testing frameworks:
  • C++: SDL2 2.0.20 with OpenGL 3.3
  • Flutter: Desktop (Linux) with fvp plugin

Log

Memory monitoring was performed using /proc/[pid]/status VmRSS (Resident Set Size) measurements.

Console output during cleanup (VDPAU decoder):

=== MEMORY BEFORE OPERATIONS ===
Current Memory: 57 MB

[... Players created and playing ...]

Player 0 - first frame!
Player 1 - first frame!
[... all 10 players rendering frames ...]

Current Memory: 407 MB (+350 MB from baseline)

=== DESTROYING EVERYTHING ===

Step 1: Stopping players...
All players stopped. Memory after stop: 308 MB (freed 99 MB)

Step 2: Unloading media...
Media unloaded. Memory after unload: 308 MB (freed 0 MB)

Step 3: Deleting players...
Players deleted. Memory after delete: 293 MB (freed 15 MB)

Step 4: Destroying GL contexts...
GL contexts destroyed. Memory: 290 MB (freed 3 MB)

Step 5: Destroying windows...
Windows destroyed. Memory: 286 MB (freed 4 MB)

=== FINAL MEMORY AFTER COMPLETE DESTRUCTION ===
Final Memory: 286 MB
Memory leaked: 236 MB (67% of used memory never freed)

Critical observation: Even after delete player, the Player destructor only freed 15 MB out of 350 MB used. The remaining 236 MB is permanently leaked.

Additional logging with mdk log handler:

setLogHandler([](LogLevel level, const char* msg) {
    auto now = std::chrono::system_clock::now();
    auto time = std::chrono::system_clock::to_time_t(now);
    std::cout << std::put_time(std::localtime(&time), "%Y-%m-%d %H:%M:%S")
              << " [" << level << "] " << msg << std::endl;
});

Output shows normal playback operation with no errors or warnings during cleanup, yet memory remains allocated.

Suspected Root Cause

Based on the memory leak pattern, the issue likely resides in:

1. Global frame pool allocator - Frame buffers allocated during playback are added to a global pool that is never freed, even when all Player instances are destroyed
2. FFmpeg context cleanup - avcodec_free_context() and avformat_close_input() may not be called in Player destructor
3. Static/singleton cleanup - If mdk uses static storage for decoders or frame pools, they persist until process termination

Impact

• High memory usage in applications with multiple Player instances
• Memory accumulation in applications that create/destroy Players repeatedly
• Affects all language bindings (C++, Flutter, etc.) since it's a core library issue

Request

Please investigate the Player destructor and frame pool management to ensure all allocated memory is properly released when Player instances are destroyed. This is critical for production use in multi-stream video applications.

[wang-bin]

I've notice the leak a few months ago when using VDPAU decoder but still don't have time to fix it. Can you try other decoders? VAAPI is the de-facto standard hardware acceleration api and well supported by all gpu vendors.

[vakiller]

I've notice the leak a few months ago when using VDPAU decoder but still don't have time to fix it. Can you try other decoders? VAAPI is the de-facto standard hardware acceleration api and well supported by all gpu vendors.

Thanks for the reply, i've tested with the VAAPI and unfortunerly, it have the same problem. I've tested it with my flutter desktop app and still no luck. But i have to say the VAAPI have very good performance with no gpu devices ( only intel cpu ). That's great, but sadly the memory leak issue still there, please consider take time and fix the leak issue.

[wang-bin]

I've tested with this example, a prebuilt binary can be download from https://nightly.link/wang-bin/mdk-examples/workflows/main/master/example-linux-stable-any.zip. vaapi is used by default, VmRSS does not keep increasing.

[wang-bin]

from your sdl code(also my sdl example code), i see gl resources are not released. You have to ensure current gl context is valid and call player[i]->setVideoSurfaceSize(-1, -1); before SDL_GL_DeleteContext, or Player::foreignGLContextDestroyed() if player is already destroyed when destroying gl context. this is a qt example