[BUG] More CMA memory issues on RPi 3A+

[mmoollllee]

On a Raspberry Pi 3A+ with an IMX477 on current software versions I'm running into CMA memory issues more often, as mentioned in #1322 . Before the process got killed and I was able to do a programmatic retry. Now the process gets stuck in uninterruptible sleep. Just want to share this... Maybe theres something to fix? :)

The logs from the moment it happens:

Sep 22 10:55:10 Papagai kernel: vc_sm_cma_import_dmabuf: imported vc_sm_cma_get_buffer failed -12
Sep 22 10:55:10 Papagai kernel: bcm2835_mmal_vchiq: vchiq_mmal_submit_buffer: vc_sm_import_dmabuf_fd failed, ret -12
Sep 22 10:57:51 Papagai kernel: INFO: task python:3555 blocked for more than 122 seconds.
Sep 22 10:57:51 Papagai kernel:       Tainted: G         C         6.12.34+rpt-rpi-v7 #1 Raspbian 1:6.12.34-1+rpt1~bookworm
Sep 22 10:57:51 Papagai kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 22 10:57:51 Papagai kernel: task:python          state:D stack:0     pid:3555  tgid:3543  ppid:1033   flags:0x00000004
Sep 22 10:57:51 Papagai kernel: Call trace: 
Sep 22 10:57:51 Papagai kernel:  __schedule from schedule+0x28/0x134
Sep 22 10:57:51 Papagai kernel:  schedule from vb2_wait_for_all_buffers+0x98/0x100 [videobuf2_common]
Sep 22 10:57:51 Papagai kernel:  vb2_wait_for_all_buffers [videobuf2_common] from __vb2_queue_cancel+0x30/0x324 [videobuf2_common]
Sep 22 10:57:51 Papagai kernel:  __vb2_queue_cancel [videobuf2_common] from vb2_core_streamoff+0x20/0xa8 [videobuf2_common]
Sep 22 10:57:51 Papagai kernel:  vb2_core_streamoff [videobuf2_common] from __video_do_ioctl+0x23c/0x470 [videodev]
Sep 22 10:57:51 Papagai kernel:  __video_do_ioctl [videodev] from video_usercopy+0x24c/0x640 [videodev]
Sep 22 10:57:51 Papagai kernel:  video_usercopy [videodev] from sys_ioctl+0x134/0xc54
Sep 22 10:57:51 Papagai kernel:  sys_ioctl from ret_fast_syscall+0x0/0x54
Sep 22 10:57:51 Papagai kernel: Exception stack(0x9c851fa8 to 0x9c851ff0)
Sep 22 10:57:51 Papagai kernel: 1fa0:                   4ee24870 4ee2487c 00000012 40045613 4ee69f08 4f7de6c8
Sep 22 10:57:51 Papagai kernel: 1fc0: 4ee24870 4ee2487c 4ee14e78 00000036 4ee26ea8 4aa0101c 4ee20db8 4f7de848
Sep 22 10:57:51 Papagai kernel: 1fe0: 00000000 4f7de6b0 76e4d000 76dd0174
Sep 22 10:59:54 Papagai kernel: INFO: task python:3555 blocked for more than 245 seconds.
Sep 22 10:59:54 Papagai kernel:       Tainted: G         C         6.12.34+rpt-rpi-v7 #1 Raspbian 1:6.12.34-1+rpt1~bookworm
Sep 22 10:59:54 Papagai kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 22 10:59:54 Papagai kernel: task:python          state:D stack:0     pid:3555  tgid:3543  ppid:1033   flags:0x00000004
Sep 22 10:59:54 Papagai kernel: Call trace: 
Sep 22 10:59:54 Papagai kernel:  __schedule from schedule+0x28/0x134
Sep 22 10:59:54 Papagai kernel:  schedule from vb2_wait_for_all_buffers+0x98/0x100 [videobuf2_common]
Sep 22 10:59:54 Papagai kernel:  vb2_wait_for_all_buffers [videobuf2_common] from __vb2_queue_cancel+0x30/0x324 [videobuf2_common]
Sep 22 10:59:54 Papagai kernel:  __vb2_queue_cancel [videobuf2_common] from vb2_core_streamoff+0x20/0xa8 [videobuf2_common]
Sep 22 10:59:54 Papagai kernel:  vb2_core_streamoff [videobuf2_common] from __video_do_ioctl+0x23c/0x470 [videodev]
Sep 22 10:59:54 Papagai kernel:  __video_do_ioctl [videodev] from video_usercopy+0x24c/0x640 [videodev]
Sep 22 10:59:54 Papagai kernel:  video_usercopy [videodev] from sys_ioctl+0x134/0xc54
Sep 22 10:59:54 Papagai kernel:  sys_ioctl from ret_fast_syscall+0x0/0x54
Sep 22 10:59:54 Papagai kernel: Exception stack(0x9c851fa8 to 0x9c851ff0)
Sep 22 10:59:54 Papagai kernel: 1fa0:                   4ee24870 4ee2487c 00000012 40045613 4ee69f08 4f7de6c8
Sep 22 10:59:54 Papagai kernel: 1fc0: 4ee24870 4ee2487c 4ee14e78 00000036 4ee26ea8 4aa0101c 4ee20db8 4f7de848
Sep 22 10:59:54 Papagai kernel: 1fe0: 00000000 4f7de6b0 76e4d000 76dd0174
Sep 22 11:01:57 Papagai kernel: INFO: task python:3555 blocked for more than 368 seconds.
Sep 22 11:01:57 Papagai kernel:       Tainted: G         C         6.12.34+rpt-rpi-v7 #1 Raspbian 1:6.12.34-1+rpt1~bookworm
Sep 22 11:01:57 Papagai kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 22 11:01:57 Papagai kernel: task:python          state:D stack:0     pid:3555  tgid:3543  ppid:1033   flags:0x00000004
Sep 22 11:01:57 Papagai kernel: Call trace: 
Sep 22 11:01:57 Papagai kernel:  __schedule from schedule+0x28/0x134
Sep 22 11:01:57 Papagai kernel:  schedule from vb2_wait_for_all_buffers+0x98/0x100 [videobuf2_common]
Sep 22 11:01:57 Papagai kernel:  vb2_wait_for_all_buffers [videobuf2_common] from __vb2_queue_cancel+0x30/0x324 [videobuf2_common]
Sep 22 11:01:57 Papagai kernel:  __vb2_queue_cancel [videobuf2_common] from vb2_core_streamoff+0x20/0xa8 [videobuf2_common]
Sep 22 11:01:57 Papagai kernel:  vb2_core_streamoff [videobuf2_common] from __video_do_ioctl+0x23c/0x470 [videodev]
Sep 22 11:01:57 Papagai kernel:  __video_do_ioctl [videodev] from video_usercopy+0x24c/0x640 [videodev]
Sep 22 11:01:57 Papagai kernel:  video_usercopy [videodev] from sys_ioctl+0x134/0xc54
Sep 22 11:01:57 Papagai kernel:  sys_ioctl from ret_fast_syscall+0x0/0x54
Sep 22 11:01:57 Papagai kernel: Exception stack(0x9c851fa8 to 0x9c851ff0)
Sep 22 11:01:57 Papagai kernel: 1fa0:                   4ee24870 4ee2487c 00000012 40045613 4ee69f08 4f7de6c8
Sep 22 11:01:57 Papagai kernel: 1fc0: 4ee24870 4ee2487c 4ee14e78 00000036 4ee26ea8 4aa0101c 4ee20db8 4f7de848
Sep 22 11:01:57 Papagai kernel: 1fe0: 00000000 4f7de6b0 76e4d000 76dd0174
Sep 22 11:03:46 Papagai sshd[3572]: Accepted publickey for kuckadm1n from 192.168.178.55 port 62682 ssh2: ED25519 SHA256:RtBA9Iv18/Lg0E69RFtzw+FE9ickYgRMtIci3KOOE/g
Sep 22 11:03:46 Papagai sshd[3572]: pam_unix(sshd:session): session opened for user kuckadm1n(uid=1000) by (uid=0)
Sep 22 11:03:46 Papagai systemd-logind[478]: New session 9 of user kuckadm1n.
Sep 22 11:03:46 Papagai systemd[1]: Started session-9.scope - Session 9 of User kuckadm1n.
Sep 22 11:03:46 Papagai sshd[3572]: pam_env(sshd:session): deprecated reading of user environment enabled
Sep 22 11:04:00 Papagai kernel: INFO: task python:3555 blocked for more than 491 seconds.
Sep 22 11:04:00 Papagai kernel:       Tainted: G         C         6.12.34+rpt-rpi-v7 #1 Raspbian 1:6.12.34-1+rpt1~bookworm
Sep 22 11:04:00 Papagai kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 22 11:04:00 Papagai kernel: task:python          state:D stack:0     pid:3555  tgid:3543  ppid:1033   flags:0x00000004
Sep 22 11:04:00 Papagai kernel: Call trace: 
Sep 22 11:04:00 Papagai kernel:  __schedule from schedule+0x28/0x134
Sep 22 11:04:00 Papagai kernel:  schedule from vb2_wait_for_all_buffers+0x98/0x100 [videobuf2_common]
Sep 22 11:04:00 Papagai kernel:  vb2_wait_for_all_buffers [videobuf2_common] from __vb2_queue_cancel+0x30/0x324 [videobuf2_common]
Sep 22 11:04:00 Papagai kernel:  __vb2_queue_cancel [videobuf2_common] from vb2_core_streamoff+0x20/0xa8 [videobuf2_common]
Sep 22 11:04:00 Papagai kernel:  vb2_core_streamoff [videobuf2_common] from __video_do_ioctl+0x23c/0x470 [videodev]
Sep 22 11:04:00 Papagai kernel:  __video_do_ioctl [videodev] from video_usercopy+0x24c/0x640 [videodev]
Sep 22 11:04:00 Papagai kernel:  video_usercopy [videodev] from sys_ioctl+0x134/0xc54
Sep 22 11:04:00 Papagai kernel:  sys_ioctl from ret_fast_syscall+0x0/0x54
Sep 22 11:04:00 Papagai kernel: Exception stack(0x9c851fa8 to 0x9c851ff0)
Sep 22 11:04:00 Papagai kernel: 1fa0:                   4ee24870 4ee2487c 00000012 40045613 4ee69f08 4f7de6c8
Sep 22 11:04:00 Papagai kernel: 1fc0: 4ee24870 4ee2487c 4ee14e78 00000036 4ee26ea8 4aa0101c 4ee20db8 4f7de848
Sep 22 11:04:00 Papagai kernel: 1fe0: 00000000 4f7de6b0 76e4d000 76dd0174

[davidplowman]

Are you able to provide a minimal script - i.e. as short and simple as possible - that provokes this error? The easier it is for us to reproduce this kind of thing, the more chance we can do something about it.

Please also confirm the total memory on your Pi 3A+ (is that 1GB?), and whether you've changed the CMA heap size in your config.txt. Thanks.

[mmoollllee]

✅ Pi 3A+ with 512MB and 1GB SWAP!
✅ CMA heap size is changed to 320 in config.txt.

The following get's called from some app.py via subprocess.check_output('python capture.py, stderr=subprocess.STDOUT). Usually this raised an exception if something fails. Now it gets stuck.

This is the capture method within a Camera class. Nothing special happening around

def capture(self):
   tuning = Picamera2.load_tuning_file("imx477.json")
   # ... get the tuning file and add custom metering and constraint behavior

   self.picam2 = Picamera2(tuning=tuning)

   self.picam2.start(show_preview=False)

   # default controls
   camera_ctrls = {
      "AeConstraintMode": controls.AeConstraintModeEnum.Custom,
      "AeMeteringMode": controls.AeMeteringModeEnum.Custom,
      "AeExposureMode": controls.AeExposureModeEnum.Short,
      "FrameDurationLimits": (100, 1000000),
   }

   # ... some controls might be changed for night time

   self.picam2.set_controls(camera_ctrls)

   capture_config = self.picam2.create_still_configuration(controls=camera_ctrls, transform=transform)
   self.picam2.switch_mode(capture_config)

   # some camera cycles to settle exposure
   meta = None
   for _ in range(3):
      meta = self.picam2.capture_metadata()
      time.sleep(0.25)

   self.picam2.capture_file(self.filepath)
   with open(self.filepath.replace(".jpg", ".json"), "w") as f:
         json.dump(meta, f, indent=2)

[davidplowman]

Ah, 512MB certainly makes things trickier. The usual problem is that you start to get fragmentation in the CMA heap because stuff doesn't get cleared up instantly. I'll try and have a look but can't promise as to how fixable that is.

A couple of things to try:

1. Configure both the preview mode (you don't actually set up a preview config, but it will default to one when started without anything else) and the capture mode to use "format": "YUV420". Recent versions of Picamera2 can save YUV files directly, and this will use way less memory (18MB per full res image rather than 48MB for the default XRGB format).

2. Mode switching can be a cause of fragmentation. You could start the camera straight in the full res capture mode. Add buffer_count=2 to the configuration, otherwise it will drop every other frame. Use YUV420 format as well, as above. This might use more CMA memory in total, but removing the mode switch might (can't be sure) make it behave more predictably. The full resolution framerate is slower, of course, but without the mode switch you might find there's little difference.

[mmoollllee]

Dropping frames is okay! I only need a single image. So buffer_count=1 should be fine, right?

Okay great! You mean by starting with a still configuration right away I won't start a separate preview stream?

In which cases is the set_controls() call needed after start()?

def capture(self):
   tuning = Picamera2.load_tuning_file("imx477.json")
   # ... get the tuning file and add custom metering and constraint behavior

   transform = Transform(hflip=True, vflip=True) if self.rotate == 180 else Transform()

   self.picam2 = Picamera2(tuning=tuning)

   # default controls
   camera_ctrls = {
      "AeConstraintMode": controls.AeConstraintModeEnum.Custom,
      "AeMeteringMode": controls.AeMeteringModeEnum.Custom,
      "AeExposureMode": controls.AeExposureModeEnum.Short,
      "FrameDurationLimits": (100, 1000000),
   }

   # ... some controls might be changed for night time

   capture_config = self.picam2.create_still_configuration(
      {
         'size': self.picam2.sensor_resolution,
         'format': 'YUV420'
      }, 
      buffer_count=1, # It's okay to drop frames as I'm only capturing a single image
      controls=camera_ctrls,
      transform=transform
   )
   self.picam2.configure(capture_config)
   self.picam2.start(show_preview=False)

   # is this needed in some cases?
   self.picam2.set_controls(camera_ctrls)

   # some camera cycles to settle exposure
   meta = None
   for _ in range(3):
      meta = self.picam2.capture_metadata()
      time.sleep(0.25)

   self.picam2.capture_file(self.filepath)
   with open(self.filepath.replace(".jpg", ".json"), "w") as f:
      json.dump(meta, f, indent=2)

[mmoollllee]

Update: With this added on first try I got the following in journalctl

Sep 22 16:57:32 Papagai kernel: vc_sm_cma_import_dmabuf: imported vc_sm_cma_get_buffer failed -12
Sep 22 16:57:33 Papagai kernel: bcm2835_mmal_vchiq: vchiq_mmal_submit_buffer: vc_sm_import_dmabuf_fd failed, ret -12

and this from python output

[0:04:16.041448343] [1174]  INFO Camera camera_manager.cpp:330 libcamera v0.5.2+125-69c6546d-dirty (2025-09-12T07:42:08BST)
[0:04:16.146322844] [1182]  INFO IPAProxy ipa_proxy.cpp:180 Using tuning file /tmp/tmpv11641bu
[0:04:16.168845950] [1182]  WARN RPI vc4.cpp:394 Mismatch between Unicam and CamHelper for embedded data usage!
[0:04:16.170615693] [1182]  INFO Camera camera_manager.cpp:220 Adding camera '/base/soc/i2c0mux/i2c@1/imx477@1a' for pipeline handler rpi/vc4
[0:04:16.170754511] [1182]  INFO RPI vc4.cpp:448 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media0 and ISP device /dev/media1
[0:04:16.170906144] [1182]  INFO RPI pipeline_base.cpp:1146 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
[0:04:16.196607286] [1174]  INFO Camera camera.cpp:1215 configuring streams: (0) 4056x3040-YUV420/sYCC (1) 4056x3040-SRGGB12_CSI2P/RAW
[0:04:16.197551669] [1182]  INFO RPI vc4.cpp:623 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 4056x3040-SRGGB12_1X12/RAW - Selected unicam format: 4056x3040-pRCC/RAW

after Ctrl+C

  File "/home/pi/app/cameras/RPiCamera.py", line 137, in capture
    meta = self.picam2.capture_metadata()  # advances one frame each time
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/.local/lib/python3.11/site-packages/picamera2/picamera2.py", line 1917, in capture_metadata
    return self.dispatch_functions(functions, wait, signal_function)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/.local/lib/python3.11/site-packages/picamera2/picamera2.py", line 1466, in dispatch_functions
    return job.get_result(timeout=timeout) if wait else job
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/.local/lib/python3.11/site-packages/picamera2/job.py", line 87, in get_result
    return self._future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 451, in result
    self._condition.wait(timeout)
  File "/usr/lib/python3.11/threading.py", line 320, in wait
    waiter.acquire()

where RPiCamera.py:137 belongs to the following lines of the script in the previous post

   for _ in range(3):
      meta = self.picam2.capture_metadata()
      time.sleep(0.25)

After reboot, with the second try, it worked!

rpicam-hello --version
rpicam-apps build: v1.9.0 a819af48889f-dirty 12-09-2025 (07:56:45)
rpicam-apps capabilites: egl:1 qt:1 drm:1 libav:1
libcamera build: v0.5.2+125-69c6546d-dirty (2025-09-12T07:42:08BST)

And for an older libcamera version it worked (at least for the first try)

libcamera-hello --version
rpicam-apps build: v1.6.0 9825703b669a-dirty 05-02-2025 (06:33:05)
rpicam-apps capabilites: egl:1 qt:1 drm:1 libav:1
libcamera build: v0.4.0+78-818d45eb-dirty (2025-02-14T02:13:36GMT)

[davidplowman]

So buffer_count=1 may be fine. When the camera system starts it normally swallows 6 or 7 frames for the AGC/AWB algorithms to settle so that might take longer - but the memory saving in having just one buffer might be more important.

Actually, it looks like you've put the camera_ctrls directly into the capture_config which feels to me like the right thing to do. You don't need to set_controls() them again as they should take effect on the very first camera frame after start(). (Setting controls after start(), and not before, means there can be quite a few frames of delay before they take effect.)

Note that older versions of Picamera2 won't be able to save a YUV420 image directly. Though even in this case, setting the format to "RGB888" (36MB for 12MP) would be better than the default. And it may be that you could still use YUV420 if you do a software conversion (using OpenCV's cv2.cvtColor()) to go from YUV420 to RGB888. Though it will be a bit slower, it may alleviate the problem because the really big buffers will be in user space, not in the CMA heap.