Skip to content

Commit 4b94ad0

Browse files
jburo1jburo1
committed
feat(vision pipeline): producer sidestable, v4l2 capture -> rga preprocess -> image buffer pool stages verified via hw integration tests, begin working on orchestration/profiling.
1 parent 552989e commit 4b94ad0

23 files changed

Lines changed: 1480 additions & 43 deletions

‎docs/software/vision_pipeline.md‎

Lines changed: 349 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,349 @@
1+
“Frames come from V4L2. We allocate a ring of DMA-capable buffers and run the capture device in streaming mode with enqueue/dequeue semantics. Each dequeued frame is handed off downstream without memcpy by sharing the same underlying allocation (DMA-BUF) into the hardware preprocessor (RGA) and then into the NPU runtime. A lock-free/SPSC queue carries buffer handles across threads, and backpressure is enforced by queue depth so we don’t drop frames or blow latency. We measure end-to-end latency per frame with timestamps at capture, post-preprocess, and post-inference.”
2+
3+
# Vision Pipeline
4+
5+
This document describes a multi-stage, low-latency, zero copy-ish, real-time computer vision pipeline for the Rockchip RK3588 SoC. The pipeline transforms raw pixel data acquired by a camera into a format that is compatible with downstream neural network inference.
6+
7+
#TODO when online, write some measurements here, e.g. inference achieves xfps at Y format here, maybe link to profiler JSONL dump etcs
8+
9+
## Major Design Considerations
10+
11+
- Latency-first: pipeline is "latest-wins": consumer always takes freshest processed frame, older frames are dropped.
12+
13+
- Accelerator-first: all suitable ops are offloaded to ISP/RGA/NPU instead of clogging CPU: ISP/V4L2 delivers 1280x720 NV12 formatted images at 60 FPS, RGA does image resizing/letterboxing/colorspace manipulatoin to ready model input, NPU does inference.
14+
15+
- Zero-copy-ish: direct memory access buffers (DMA-BUF) connect pipeline stages to avoid CPU memcpys of full frames.
16+
17+
- Multithreaded: two threads operate the pipeline, a producer owns image capture and preprocess, and a consumer owns inference + postprocess.
18+
19+
- Observable: performance information emitted at every stage.
20+
21+
## Stages
22+
23+
| Stage | Thread | Input | Output | Primary Code |
24+
|---|---|---|---|---|
25+
| Capture (V4L2) | Producer | `/dev/video*` | `FrameDescriptor` (borrowed V4L2 slot) | `v4l2_capture.hpp/cpp` |
26+
| Preprocess (RGA) | Producer | `FrameDescriptor` (NV12 DMA-BUF) | `ImageBuffer` (RGB DMA-BUF) | `rga_preprocess.hpp/cpp` |
27+
| Buffering/Drop policy | Cross-thread boundary | Pool indices | “latest wins” ready buffer index | `image_buffer_pool.hpp/cpp` |
28+
| Inference (NPU) | Consumer | `ImageBuffer` (RGB DMA-BUF) | model outputs | `vision/include/omniseer/vision/rknn_runner.hpp` (TODO), `vision/src/rknn_runner.cpp` (TODO) |
29+
| Postprocess/Publish | Consumer | model outputs | ROS msgs, telemetry | TODO |
30+
31+
## Diagram
32+
33+
```
34+
photons
35+
|
36+
v
37+
+-------+
38+
|radxa 4k camera |
39+
| |
40+
+-------+
41+
|
42+
v
43+
44+
+----------------------------------------------+
45+
| Camera / ISP -> V4L2 N slot capture ring |
46+
| exported as DMA-BUF fd per slot via EXPBUF |
47+
+--------------------------+-------------------+
48+
|
49+
v VIDIOC_DQBUF (nonblocking)
50+
+----------+-----------+
51+
| V4l2Capture |
52+
| owns: slot DMA-BUF fds|
53+
+----------+-----------+
54+
|
55+
| FrameDescriptor (borrow slot i)
56+
v
57+
+----------+-------------------------+
58+
| RgaPreprocess (librga / im2d) |
59+
| NV12 (DMA-BUF) -> RGB888 (DMA-BUF) |
60+
| mode: Letterbox |
61+
+----------+-------------------------+
62+
|
63+
| publish_ready(pool_idx)
64+
v
65+
+----------+------------------+
66+
| ImageBufferPool (SPSC) |
67+
| free_ring + ready_idx |
68+
| policy: latest wins |
69+
+----------+------------------+
70+
|
71+
v acquire_read(pool_idx)
72+
+----------+-----------+
73+
| RKNN Runner |
74+
| reads RGB888 DMA-BUF |
75+
+----------+-----------+
76+
|
77+
v
78+
model outputs / detections
79+
|
80+
v
81+
ROS publish / logging
82+
83+
TODO: create a nicer visual than this simple ascii
84+
```
85+
86+
## Interfaces
87+
88+
### Core data types
89+
90+
Defined in `types.hpp`:
91+
92+
- `FrameDescriptor`: content description of a V4L2 ring buffer slot (owned by the kernel driver). Handle for ISP output/RGA input.
93+
- `ImageBuffer`: DMA-BUF fd-backed buffer (owned by the application). Handle for RGA output/RKNN(NPU) input.
94+
95+
### Capture: `V4l2Capture`
96+
97+
Manages the V4L2 (Video for Linux 2) streaming lifecycle and defines a borrow-token style API for accessing ISP output from downstram devices in a zero-copy fashion.
98+
99+
Defined/implemented in `v4l2_capture.hpp/cpp`.
100+
101+
- `start()`:
102+
- Opens device path (e.g. `/dev/video12`), negotiates image format, allocates a driver-managed ring buffer,
103+
exports each slot as a DMA-BUF fd, queues all slots, and starts streaming.
104+
105+
- `dequeue(FrameDescriptor& out)`:
106+
- Dequeues the most recently filled V4L2 ring slot, populates `out` with a borrow-token (v4l2_index, DMA-BUF fd, layout, metadata). Thse caller must, after performing its work, `requeue(out.v4l2_index)` to return the slot to the driver so it can refill it with a fresh frame.
107+
108+
- `requeue(uint32_t index)`:
109+
- Return the specified V4L2 ring buffer slot at `index` to the driver so it can be filled with a fresh frame.
110+
111+
### Preprocess: `RgaPreprocess`
112+
113+
Manages the RGA (2D blitter) transformations from ISP output to correct model input.
114+
115+
Defined/implemented in `rga_preprocess.hpp/cpp`.
116+
117+
- `run(const FrameDescriptor& src_nv12, ImageBuffer& dst_rgb, ...)`:
118+
- Runs the RGA hardware pipeline to convert a captured NV12 DMA-BUF frame into a RGB888 destination buffer. Synchronous.
119+
120+
- `prefill(ImageBuffer& dst_rgb)`:
121+
- The RK3588's RGA device does not support colorfilling a RGB888 buffer, so callers must do it themselves. This must only be called once at buffer init time.
122+
123+
124+
### Buffering: `ImageBufferPool`
125+
126+
This is the boundary between the producer and consumer threads. It facilitates the data handoff between the RGA output and the NPU input in a lock-free fashion. It implements a "freshest-first" policy, where the consumer only has access to the latest processed image.
127+
128+
Defined/implemented in `image_buffer_pool.hpp/cpp`
129+
130+
The usage is as follows:
131+
132+
- Producer: `acquire_write()` -> RGA writes -> `publish_ready()`
133+
- Consumer: `acquire_read()` -> RKNN reads -> `publish_release()`
134+
135+
- `acquire_write(int& idx)`
136+
- Obtains a free buffer index into `idx` for the producer to write into.
137+
138+
- `publish_ready(int idx)`
139+
- Publishes `idx` as the newest ready buffer.
140+
- Should be called after performing the write.
141+
142+
- `acquire_read(int& idx)`
143+
- Atomically grabs the currently ready buffer index into `idx`.
144+
145+
- `publish_release(int idx)`
146+
- Returns the consumed buffer index `idx` back to the pool.
147+
- Should be called after performing the read + consumption.
148+
149+
- `buffer_at(int idx)`
150+
- Accessor function for buffer at `pool[idx]`
151+
- Required to access data once ownership established
152+
- Comes in non-const/const flavours for producer/consumer
153+
154+
### Buffer Allocation: `DmaHeapAllocator` + `DmaHeapAllocation`
155+
156+
"Video malloc" allocator + allocation classes that create shareable RGB image buffers for zero-copy-ish data movement between RGA and RKNN. Resource-safe bridge between kernel memory and accelerators.
157+
158+
Defined and implemented in `dma_heap_alloc.hpp/cpp`.
159+
160+
`DmaHeapAllocator`:
161+
- `DmaHeapAllocator()`
162+
- Create factory
163+
164+
- `allocate(int width, int height, PixelFormat fmt)`
165+
- Allocate a DMA-BUF suitable for RGA write / RKNN read and return an ImageBuffer
166+
and descriptor that points at it.
167+
168+
169+
170+
## Ownership & Lifetime Rules for Buffers
171+
172+
### V4L2 ring slots (`FrameDescriptor`)
173+
174+
- Owned by: kernel driver.
175+
- Userspace handle lifetime:
176+
- The exported DMA-BUF fds are owned by `V4l2Capture` for the duration of streaming.
177+
- Each `dequeue()` borrows one ring slot at index `v4l2_index`.
178+
- A `requeue(v4l2_index)` must occur for every successful `dequeue()` to allow slot to be refilled. This should happen after RGA is finished using the buffer.
179+
180+
### Model input buffers (`ImageBufferPool`)
181+
182+
- Owned by: `ImageBufferPool` (backing `DmabufAllocation`s are RAII).
183+
- Cross-thread rule:
184+
- Producer may only write to a buffer index after `acquire_write(idx)` returns true.
185+
- Consumer may only read from a buffer index after `acquire_read(idx)` returns true.
186+
- Consumer must call `publish_release(idx)` once it is done reading.
187+
188+
## Overview of Producer Responsibilities
189+
190+
Own the upstream clock: drive the loop cadence (dequeue frames) and decide when to drop work to maintain “latest-wins” latency.
191+
192+
Dequeue from V4L2: call DQBUF, receive the newest captured slot, and package it into a FrameDescriptor (fd(s), strides, w/h, format, timestamp, slot index).
193+
194+
Respect V4L2 slot lifetime: treat the dequeued slot as borrowed from the driver; do not hold it longer than necessary.
195+
196+
Acquire an output buffer: get a writable ImageBuffer slot from ImageBufferPool::acquire_write(idx) (or decide to skip processing if none are available).
197+
198+
Run preprocess on accelerators: invoke RGA to transform NV12 DMA-BUF → RGB/BGR DMA-BUF, including resize + letterbox/stretch policy, and produce LetterboxMeta if needed.
199+
200+
Write output metadata: fill ImageBuffer fields (fd, stride, dims, pixel format, timestamp, letterbox params, sequence number).
201+
202+
Publish the newest buffer: call publish_ready(idx) with release semantics so the consumer sees a fully-written frame.
203+
204+
Recycle old ready frames: if publish_ready “steals” the previous ready buffer (because latest-wins), ensure it goes back into the producer/free path so buffers don’t leak.
205+
206+
Return camera buffers promptly: QBUF the V4L2 slot back to the driver as soon as RGA is done with it (or immediately if you drop the frame).
207+
208+
Maintain steady-state buffer hygiene: one-time prefill/padding initialization for destination buffers (your RGB888 imfill limitation means you may do a CPU prefill fallback).
209+
210+
Instrumentation: emit per-stage timings (DQBUF wait, RGA submit/complete, publish cost), drop counters, and queue depths.
211+
212+
Error containment: handle transient failures (EINTR/EAGAIN, occasional RGA errors) without wedging the pipeline; perform clean shutdown (stop streaming, close fds, free allocations).
213+
214+
## Consumer Responsibilities
215+
216+
- Acquire the newest frame (latest-wins)
217+
218+
- Run RKNN inference (NPU)
219+
220+
Initialize RKNN once at startup (load .rknn, init runtime).
221+
222+
Per frame:
223+
224+
Feed the input tensor (usually uint8 NHWC or NCHW depending on how you exported/configured).
225+
226+
Call inference.
227+
228+
Read output tensors.
229+
230+
Why this structure matters: your inference FPS will be lower than camera FPS, so consumer naturally drops frames and always processes “most recent state,” which is exactly what you want for robotics.
231+
232+
- Postprocess (CPU, usually)
233+
234+
Decode YOLO head outputs → candidate boxes + scores + class ids
235+
236+
Apply thresholding + NMS (non-max suppression)
237+
238+
Undo letterbox/resize to map boxes back to 1280×720 (or whatever your original frame is)
239+
240+
- Publish results downstream
241+
242+
Provide a simple struct like:
243+
244+
timestamp, list of {class_id, score, x1,y1,x2,y2} in original image coordinates
245+
246+
Feed tracking / “seek-and-capture” logic.
247+
248+
- Release buffer
249+
250+
pool.release(idx) so RGA can reuse it.
251+
252+
A minimal consumer loop looks like:
253+
254+
acquire → infer → decode → publish → release
255+
(no queue buildup, no waiting on stale frames)
256+
257+
## Threading Model (Current Intended)
258+
259+
Two threads:
260+
261+
1) Capture/Preprocess thread (producer):
262+
- `cap.dequeue(frame)` (nonblocking loop/poll)
263+
- `pool.acquire_write(idx)`; if false, immediately `cap.requeue(frame.v4l2_index)` and continue
264+
- `rga.run(frame, pool.buffer_at(idx), &meta)`
265+
- `pool.publish_ready(idx)`
266+
- `cap.requeue(frame.v4l2_index)`
267+
268+
2) Inference thread (consumer):
269+
- `pool.acquire_read(idx)` (nonblocking loop/condition variable)
270+
- `rknn.infer(pool.buffer_at(idx), ...)` (TODO)
271+
- `pool.publish_release(idx)`
272+
273+
Notes:
274+
- The pool policy intentionally drops frames if inference cannot keep up (“latest wins”).
275+
- `V4l2Capture::dequeue()` is currently nonblocking; production code should prefer `poll()`/`select()`
276+
over spin-sleep loops.
277+
278+
## Failure Modes & Handling
279+
280+
### Capture
281+
282+
- `start()` throws:
283+
- bad device path / permissions
284+
- missing V4L2 streaming or MPLANE caps
285+
- driver rejects requested size or format
286+
- ioctl failures (REQBUFS/QUERYBUF/EXPBUF/QBUF/STREAMON)
287+
288+
- `dequeue()`:
289+
- returns `false` on `EAGAIN` (no frame ready)
290+
- throws on other errors
291+
292+
Recommended handling:
293+
- Treat `start()` failures as fatal at node startup (log and exit or retry with backoff).
294+
- Treat repeated dequeue `EAGAIN` as “no data”; use `poll()` for readiness.
295+
- Any early exit after a successful dequeue must still `requeue(v4l2_index)`.
296+
297+
### Preprocess (RGA)
298+
299+
`run()` returns `false` if:
300+
- config is invalid (dst_w/dst_h <= 0)
301+
- input descriptor is not NV12 or does not match the contiguous NV12 layout assumptions
302+
- destination buffer is invalid (fd/stride mismatch)
303+
- librga rejects parameters (`imcheck` failure) or processing fails (`improcess` < 0)
304+
305+
Recommended handling:
306+
- On preprocess failure, log once per N frames, drop the frame, and continue:
307+
- `cap.requeue(v4l2_index)` must still happen.
308+
- If the destination pool index was acquired, it should be returned to the pool (either via
309+
a dedicated “abort” path or by publishing/releasing consistently).
310+
311+
### Buffer Pool
312+
313+
- Producer `acquire_write()` can fail if the consumer has not released any buffers and the
314+
producer stash is empty.
315+
- “Latest wins” means drops are expected under load; this is not an error.
316+
317+
Recommended handling:
318+
- If `acquire_write()` fails, drop the current frame (requeue immediately) to protect latency.
319+
320+
### Inference (RKNN) — TODO
321+
322+
Expected failure classes:
323+
- model load failures (missing `.rknn`, incompatible runtime)
324+
- tensor layout mismatch (RGB vs BGR, NCHW vs NHWC, quantization)
325+
- device/runtime errors during inference
326+
327+
Recommended handling:
328+
- Fail fast at init if the model cannot be loaded.
329+
- If inference fails mid-run, drop that frame and continue (don’t stall capture).
330+
331+
## Hardware/Platform Assumptions
332+
333+
- V4L2 node exposes NV12 in a single exported DMA-BUF allocation with UV data located at
334+
`stride_bytes * height`. This is validated in `RgaPreprocess` and in the V4L2/RGA tests.
335+
- librga (im2d) is available and functional (`/dev/rga` present on target).
336+
- libdrm is available and accessible for dumb-buffer DMA-BUF allocation.
337+
338+
## Current Repo Status (as of 2026-01-27)
339+
340+
- Implemented and tested (hardware-dependent tests may skip at runtime):
341+
- `V4l2Capture`
342+
- `DrmDmabufAllocator`
343+
- `ImageBufferPool` + `SpscRing`
344+
- `RgaPreprocess` (NV12 -> RGB888, letterbox)
345+
- Not implemented yet:
346+
- Pipeline orchestration (`vision/src/pipeline.cpp`)
347+
- Profiling implementation (`vision/src/profiler.cpp`)
348+
- RKNN runner (`vision/src/rknn_runner.cpp`)
349+
- A real `vision_harness` executable (currently a stub placeholder)

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ nav:
2424
- Build pipeline: software/build_pipeline.md
2525
- ROS packages: software/ros_packages.md
2626
- Code documentation: software/code_documentation.md
27+
- Vision pipeline: software/vision_pipeline.md
2728
- Hardware:
2829
- Circuit: hardware/circuit.md
2930
- Wiring guide: hardware/wiring.md

0 commit comments

Comments
 (0)