Skip to content

Memory leak in fvp_playground with both hardware and software decoders on eLinux ARM64 #333

New issue
New issue
Open
Open
Memory leak in fvp_playground with both hardware and software decoders on eLinux ARM64#333
@josephnarai

Description

@josephnarai
josephnarai
opened on Oct 20, 2025

MDK Memory Leak Bug Report

Describe the Bug

Memory leak in the fvp_playground sample on Flutter eLinux affecting BOTH hardware and software decoders. RSS grows continuously with each video load/dispose cycle (~8 MiB per cycle), eventually leading to system crash from memory exhaustion.

Test Setup:
We added a Memory Test tab to the official fvp_playground sample that repeatedly creates and disposes VideoPlayerController instances. The test loads a video file, plays it for 3 seconds, pauses, disposes the controller, then loads a different video and repeats. This mimics real-world scenarios where videos are loaded, played, and disposed (e.g., sequential video playback, playlist functionality).

Severity:

  • Impact: Sample app RSS climbs by ~160 MiB every 20 load/dispose cycles; prolonged operation exhausts the 2 GB device and triggers OOM killer
  • Platform: RK3566 ARM64 embedded device with 2 GB RAM
  • Test Scenario: 20 video load/dispose cycles (alternating between 2 different video files)
  • Workaround: None identified

Root Cause Assessment:
This indicates a core MDK memory management issue affecting multiple decoder backends, NOT hardware-specific interop problems. The leak appears related to VideoFrame pool/queue management during player disposal and recreation cycles.

Expected Behavior

Memory usage should remain stable during repeated video player create/dispose cycles. After disposing a VideoPlayerController and its associated MDK player instance, native memory should be freed. RSS should stabilize or fluctuate within a small range rather than continuously growing.

Environment

OS:

  • Embedded Linux based on Buildroot
  • Kernel: Linux 6.1.x
  • Display Server: Wayland 1.20 with Weston 10.0 compositor

GPU:

  • Mali-G52 MP2 (2 cores @ 800 MHz)
  • Driver: EGL 1.4, OpenGL ES 3.2

Hardware:

  • SoC: RockChip RK3566 (ARM64, quad-core Cortex-A55 @ 1.8 GHz)
  • RAM: 2 GB DDR4
  • Video Decoder: RockChip MPP (hardware H.264/H.265 decoder)

Software:

  • Flutter: 3.27.1 (eLinux SDK for ARM64)
  • FVP (Flutter Video Player): 0.35.0
  • MDK: 0.35.0 (bundled with FVP 0.35.0)
  • Video Codec: H.264/AVC (MP4 container, 1920x1080, 15-60 seconds)

Reproduction Steps

  1. Clone the fvp_playground sample from the FVP repository
  2. Add a Memory Test tab to lib/main.dart (code provided below)
  3. Build using scripts/package_for_device.sh --release (or your platform's build process)
  4. Deploy to RK3566 device at /usr/lib/fvp_playground/ (or run on your platform)
  5. Launch the app and navigate to the Memory Test tab
  6. Click "Start Test" to run 20 video load/dispose cycles
  7. Monitor memory growth via /proc/self/status (VmRSS, RssAnon, RssFile, VmData)

Memory Test Tab Code:
The test creates a simple widget that repeatedly:

  • Loads a video file via VideoPlayerController.networkUrl(Uri.file(...))
  • Initializes and plays for 3 seconds
  • Pauses and disposes the controller
  • Repeats with alternating video files

See complete implementation in the test widget that reads memory from /proc/self/status and counts DMA-BUF file descriptors in /proc/self/fd.

Test Configuration:

fvp.registerWith(
  options: {
    'video.decoders': ['rockchip', 'rkmpp', 'FFmpeg', 'dav1d'],
    'global': {'gl.ubo': 0},
  },
);

Test Results

Hardware Decoder Test (RockChip MPP + DRM-EGL)

Configuration:

  • Decoder: RockChip MPP hardware decoder
  • Rendering: DRM-EGL interop via DMA-BUF (zero-copy GPU texture import)
  • GPU: Mali-G52 EGL/OpenGL ES

Memory Growth (20 iterations, ~6 minutes):

Iteration VmRSS (MiB) RssAnon (MiB) RssFile (MiB) VmData (MiB) DMA-BUF FDs
1 147.2 42.2 104.9 185.6 29
5 187.4 49.8 137.5 202.5 38
10 226.1 48.9 177.1 202.7 50
15 266.3 49.8 216.3 203.6 65
20 307.4 51.6 255.6 205.2 80

Total Growth: +160.2 MiB RSS, +51 DMA-BUF FDs
Growth Rate: ~8.0 MiB per iteration

Log Evidence:

[MemoryTest] Iteration 1: rss=147.2 MiB rssAnon=42.2 MiB rssFile=104.9 MiB vmData=185.6 MiB dmabufFDs=29
[MemoryTest] Iteration 10: rss=226.1 MiB rssAnon=48.9 MiB rssFile=177.1 MiB vmData=202.7 MiB dmabufFDs=50
[MemoryTest] Iteration 20: rss=307.4 MiB rssAnon=51.6 MiB rssFile=255.6 MiB vmData=205.2 MiB dmabufFDs=80

MDK Logs Show:

  • reuse EGLImage: 0 consistently (EGLImages not being reused)
  • VideoFrame...to be destroyed is not rendered messages (frames disposed without rendering)
  • DMA-BUF file descriptors accumulate

Software Decoder Test (FFmpeg CPU-only)

Configuration:

  • Decoder: FFmpeg software decoder (CPU-based, no hardware acceleration)
  • Rendering: CPU memory to GPU texture upload (no DMA-BUF)
  • GPU: Minimal (texture upload only)

Test Setup:

fvp.registerWith(
  options: {
    'video.decoders': ['FFmpeg'],  // Force software decoding
    'global': {'gl.ubo': 0},
  },
);

Memory Growth (20 iterations, ~6 minutes):

Iteration VmRSS (MiB) RssAnon (MiB) RssFile (MiB) VmData (MiB) DMA-BUF FDs
1 181.2 72.7 108.4 237.3 0
5 214.3 71.1 143.1 241.6 0
10 248.8 73.9 174.8 249.4 0
15 297.1 82.3 214.6 261.4 0
20 339.4 85.7 253.7 260.9 0

Total Growth: +158.2 MiB RSS, 0 DMA-BUF FDs (as expected)
Growth Rate: ~7.9 MiB per iteration

Log Evidence:

[MemoryTest] Iteration 1: rss=181.2 MiB rssAnon=72.7 MiB rssFile=108.4 MiB vmData=237.3 MiB dmabufFDs=0
[MemoryTest] Iteration 10: rss=248.8 MiB rssAnon=73.9 MiB rssFile=174.8 MiB vmData=249.4 MiB dmabufFDs=0
[MemoryTest] Iteration 20: rss=339.4 MiB rssAnon=85.7 MiB rssFile=253.7 MiB vmData=260.9 MiB dmabufFDs=0

MDK Logs Show:

  • No DMA-BUF or DRM-EGL messages (as expected for software path)
  • Same VideoFrame...to be destroyed is not rendered pattern
  • Pure CPU decoding still leaks at similar rate

Key Findings

1. Both Decoders Leak

  • Hardware (RockChip MPP): ~8.0 MiB/iteration
  • Software (FFmpeg): ~7.9 MiB/iteration
  • Nearly identical leak rates rule out hardware-specific driver issues

2. No DMA-BUF Involvement in Software Path

  • FFmpeg test shows 0 DMA-BUF file descriptors throughout
  • Leak occurs entirely in MDK's native heap
  • No GPU interop, no zero-copy paths, pure software still leaks

3. Common VideoFrame Pattern

Both decoders show identical disposal patterns:

FINEST: [mdk] VideoFrame...to be destroyed is not rendered by 0x...

Suggests frame pool/queue management issue in MDK core.

4. Memory Growth is Native

  • VmData (native heap) and VmRSS grow continuously
  • Dart garbage collection has no effect
  • Growth is in MDK's C++ library, not Flutter layer

5. Hardware-Specific Issues

Hardware decoder has additional symptoms:

  • reuse EGLImage: 0 (never reuses EGLImages)
  • DMA-BUF FD accumulation (+51 FDs over 20 iterations)
  • Slightly faster leak rate may be due to additional EGL resource leaks

Log

Log Configuration:
The FVP plugin enables MDK logging via:

fvp.registerWith(options: {'video.decoders': ['rockchip', 'FFmpeg'], 'global': {'gl.ubo': 0}});

MDK logs are captured through Flutter's debug print and written to /home/root/flux/logs/fvp_playground.log.

Key Log Patterns Observed:

Hardware decoder (RockChip MPP):

FINE: [mdk] DRM-EGL interop via dma buf composed layers import w/ modifiers as an external rgba texture, reuse EGLImage: 0
FINEST: [mdk] [email protected] to be destroyed is not rendered by 0x557f38c4c0
FINEST: [mdk] [email protected] to be destroyed is not rendered by 0x557f3ca8f0

Software decoder (FFmpeg):

FINE: [mdk] video decoders: "FFmpeg"
FINE: [mdk] AVCodec.Video[h264.Main] decoder: 5 Frame threads
FINEST: [mdk] [email protected] to be destroyed is not rendered by 0x559f4d0730
FINEST: [mdk] [email protected] to be destroyed is not rendered by 0x559f4d0730

Both show the same "VideoFrame...to be destroyed is not rendered" pattern, suggesting unreleased frame pool resources.

Full logs with increased verbosity available upon request.

Crash

No immediate crash occurs, but extended operation (30-50+ load/dispose cycles) leads to:

  • OOM killer activation
  • System becomes unresponsive
  • Process terminated by kernel

Example from dmesg during extended software decoder test:

Out of memory: Killed process <pid> (fvp_playground) total-vm:XXXXkB, anon-rss:XXXXkB, file-rss:XXXXkB

No crash stack trace available as the issue is memory exhaustion rather than segmentation fault. Memory profiling with Valgrind or AddressSanitizer would be helpful to identify exact allocation sites.

Comparison with Issue #304

Issue #304 (VAAPI + i915 memory leak, June 2025) reported:

  • Hardware decoder (VAAPI) leaked
  • Software decoder (FFmpeg) did NOT leak
  • Fixed in June 2025 builds

Our case is more severe:

  • Hardware decoder (RockChip MPP) leaks
  • Software decoder (FFmpeg) ALSO leaks at similar rate
  • Suggests either:
    1. Issue Possible memory leak with VAAPI + i915 #304 fix didn't fully address core VideoFrame management, OR
    2. ARM64/Mali/eLinux platform has additional issues, OR
    3. Flutter/eLinux usage pattern triggers different leak path

Root Cause Hypothesis

Primary: VideoFrame Pool Management

  1. MDK maintains internal VideoFrame pools/queues for decoded frames
  2. Frames allocated but marked "not rendered" during disposal
  3. VideoFrames not properly freed when player is disposed
  4. Memory accumulates with each player create/dispose cycle
  5. Affects ALL decoder backends (not hardware-specific)

Secondary: Hardware EGLImage Reuse

For hardware decoder only:

  1. reuse EGLImage: 0 indicates no reuse between players
  2. New EGLImages/DMA-BUFs created for each player instance
  3. Old resources not released on disposal
  4. This may contribute to hardware path leaking slightly faster

Questions for MDK Maintainer

  1. Why does FFmpeg software decoder leak when Issue Possible memory leak with VAAPI + i915 #304 reported no leak?

    • Is this Flutter/eLinux platform-specific?
    • Is this ARM64/Mali-specific vs Intel x86?

  2. Are VideoFrames being properly freed during player disposal?

    • Why do both decoders show "VideoFrame...not rendered" messages?
    • Is there a frame pool leak in MDK core?

  3. Is EGLImage reuse working correctly for RockChip MPP?

    • Why is reuse EGLImage: 0 always (never 1)?
    • Does this contribute to the leak?

  4. Does MDK 0.35.0 include the Issue Possible memory leak with VAAPI + i915 #304 fix?

    • Was the fix applied to RockChip MPP decoder?
    • Or only VAAPI/VDPAU?

  5. Is this specific to Flutter/eLinux usage pattern?

    • Rapid player create/dispose cycles (loading different videos sequentially)
    • Does native Qt/C++ usage show the same leak?
    • Should we reuse player instances instead of dispose/recreate?

Additional Diagnostics Available

If needed, we can provide:

  • Full MDK logs with increased verbosity
  • Complete /proc/PID/smaps memory maps during leak progression
  • DRM/KMS driver logs from dmesg
  • Longer test runs with more detailed memory snapshots
  • Testing with different video resolutions/codecs/bitrates

References

Date: October 20, 2025
MDK Version: 0.35.0 (via FVP 0.35.0)
Platform: RK3566 ARM64, Mali-G52, Buildroot Linux, Wayland/Weston
Status: CRITICAL - Unresolved

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions