Merge branch 'feature/add_esp_capture_app' into 'main'

esp_capture: Add initial version of esp_capture for media capture

See merge request adf/multimedia/esp-gmf!75

Author: jason-mao

.codespellrc

@@ -1,4 +1,4 @@
 [codespell]
 skip = build,*.yuv,alice.txt,*.rgb
-ignore-words-list = ser,dout,rsource,fram,inout,shs,ans,aci,unstall,unstalling,hart,wheight,wel,ot,afe
+ignore-words-list = ser,dout,rsource,fram,inout,shs,ans,aci,unstall,unstalling,hart,wheight,wel,ot,afe,muxer
 write-changes = true

.github/workflows/upload_component.yml

@@ -22,5 +22,7 @@ jobs:
             gmf_video:./elements/gmf_video
             gmf_loader:./packages/gmf_loader
             esp_audio_simple_player:./packages/esp_audio_simple_player
+            esp_capture: ./packages/esp_capture
+            gmf_app_utils: ./packages/gmf_app_utils
           namespace: "espressif"
           api_token: ${{ secrets.IDF_COMPONENT_API_TOKEN }}

README.md

@@ -43,6 +43,8 @@ Advanced components are encapsulation modules in ESP-GMF that target specific ap
 | [esp_audio_simple_player](./packages/esp_audio_simple_player) | Simple audio player | - `gmf_audio`<br>- `gmf_io` |
 | [gmf_loader](./packages/gmf_loader) | Set up the given GMF pool using the <br> configuration selected via `Kconfig` | - `gmf_core`<br>- `gmf_io`<br>- `gmf_audio`<br>- `gmf_misc`<br>- `gmf_video`<br>- `gmf_ai_audio`<br>- `esp_codec_dev`<br>- `esp_audio_codec`<br>- `esp_video_codec` |
 | [gmf_app_utils](./packages/gmf_app_utils) | Common peripheral configuration, unit <br>testing tools Memory leak detection tools | - `gmf_core`<br>- `protocol_examples_common`<br>- `codec_board`|
+| [esp_capture](./packages/esp_capture) | High-level multimedia capture module | - `gmf_core`<br>- `gmf-audio`<br>- `gmf-video`<br>- `esp_muxer`<br>- `esp_codec_dev`<br>- `esp-sr`<br>- `esp_video`<br>- `esp32-camera`|
+
 
 # ESP-GMF Usage Guide
 

README_CN.md

@@ -43,6 +43,7 @@ ESP-GMF 各个模块以组件的形式存在，组件又按功能分为**原子
 |  [esp_audio_simple_player](./packages/esp_audio_simple_player) | 简单的音频播放器 | - `gmf_audio`<br>- `gmf_io` |
 |  [gmf_loader](./packages/gmf_loader) | 使用 `Kconfig` 选择的配置<br>设置给定的 GMF Pool | - `gmf_core`<br>- `gmf_io`<br>- `gmf_audio`<br>- `gmf_misc`<br>- `gmf_video`<br>- `gmf_ai_audio`<br>- `esp_codec_dev`<br>- `esp_audio_codec`<br>- `esp_video_codec` |
 |  [gmf_app_utils](./packages/gmf_app_utils) | 常用外设配置，单元测试工具<br>内存泄漏检测工具 | - `gmf_core`<br>- `protocol_examples_common`<br>- `codec_board` |
+|  [esp_capture](./packages/esp_capture) | 易用的音视频采集器 | - `gmf_core`<br>- `gmf-audio`<br>- `gmf-video`<br>- `esp_muxer`<br>- `esp_codec_dev`<br>- `esp-sr`<br>- `esp_video`<br>- `esp32-camera`|
 
 # ESP-GMF 使用说明
 

packages/esp_capture/CHANGELOG.md

@@ -0,0 +1,6 @@
+
+## v0.7.0
+
+### Features
+
+- Initial version of `esp_capture`

packages/esp_capture/CMakeLists.txt

@@ -0,0 +1,43 @@
+# Gather common sources
+set(COMPONENT_SRC_DIRS
+    "src"
+    "src/utils"
+    "impl/capture_gmf_path/src"
+    "impl/capture_gmf_path/src/elements"
+    "impl/capture_file_src"
+)
+
+set(COMPONENT_INCLUDE
+  "./include" ./include/impl "./interface"
+  "impl/capture_gmf_path/include"
+)
+
+set(COMPONENT_PRIV_INCLUDE
+    "private_inc"
+    "impl/capture_gmf_path/include/elements"
+)
+
+if(CONFIG_ESP_CAPTURE_ENABLE_AUDIO)
+    list(APPEND COMPONENT_SRC_DIRS "impl/capture_audio_src")
+endif()
+
+if(CONFIG_ESP_CAPTURE_ENABLE_VIDEO)
+    list(APPEND COMPONENT_SRC_DIRS
+        "impl/capture_video_src"
+        "impl/capture_text_overlay"
+        "impl/capture_text_overlay/font"
+    )
+    list(APPEND COMPONENT_PRIV_INCLUDE
+        "impl/capture_text_overlay"
+    )
+endif()
+
+idf_component_register(
+    SRC_DIRS ${COMPONENT_SRC_DIRS}
+    INCLUDE_DIRS ${COMPONENT_INCLUDE}
+    PRIV_INCLUDE_DIRS ${COMPONENT_PRIV_INCLUDE}
+    REQUIRES esp_timer
+)
+
+include(package_manager)
+cu_pkg_define_version(${CMAKE_CURRENT_LIST_DIR})

packages/esp_capture/Kconfig

@@ -0,0 +1,20 @@
+menu "Espressif Capture Configuration"
+    config ESP_CAPTURE_ENABLE_AUDIO
+        bool "Enable Audio Capture"
+        default y
+
+    config ESP_CAPTURE_ENABLE_VIDEO
+        bool "Enable Video Capture"
+        default y
+
+    config ESP_CAPTURE_ENABLE_VIDEO_OVERLAY
+        bool "Enable Video Overlay for Capture"
+        default y
+        depends on ESP_CAPTURE_ENABLE_VIDEO
+
+    config ESP_CAPTURE_ENABLE_PERF_MON
+        bool "Enable Performance Monitor for Capture"
+        default n
+
+    rsource "impl/capture_text_overlay/Kconfig"
+endmenu

packages/esp_capture/README.md

@@ -0,0 +1,234 @@
+# Espressif Multimedia Capture
+
+- [![Component Registry](https://components.espressif.com/components/espressif/esp_capture/badge.svg)](https://components.espressif.com/components/espressif/esp_capture)
+
+- [中文版](./README_CN.md)
+
+Espressif Multimedia Capture (**esp_capture**) is a lightweight multimedia capture component developed by Espressif, based on the [ESP-GMF](https://github.com/espressif/esp-gmf/blob/main/README.md) architecture. It features low memory footprint, high flexibility, and a modular design. The component integrates functions such as audio/video encoding, image rotation and scaling, echo cancellation, and text overlay. It is widely applicable to scenarios including audio/video recording, AI large model input, WebRTC, RTMP/RTSP streaming, local storage, and remote monitoring.
+
+## 🔑 Key Features
+
+- 📦 **Low memory overhead** with modular pipeline structure
+- 🎚️ **Tight integration with ESP-GMF** for advanced audio/video processing
+- 🎥 **Support for multiple input devices**: V4L2, DVP cameras, audio codecs
+- 🔁 **Parallel streaming and storage** options
+- ⚙️ **Automatic source-sink negotiation** for simplified configuration
+- ✨ **Customizable processing pipelines** for professional use cases
+
+## ⚙️ Architecture Overview
+
+A capture system connects sources (input devices) to sinks (output targets) through an intermediate processing path.
+
+```mermaid
+graph LR
+    Capture_Source --> Capture_Path --> Capture_Sink
+```
+
+| Component          | Description                                                        |
+|-------------------|--------------------------------------------------------------------|
+| **Capture Source** | Interfaces for physical input devices (camera, mic, etc.)          |
+| **Capture Path**   | Processing pipeline (audio/video filters, encoders, overlays)      |
+| **Capture Sink**   | Output targets (e.g., streaming, storage, muxers)                  |
+
+### 🧠 AV Synchronization and Muxing
+
+To enable synchronized audio-video muxing, a dedicated sync module aligns timestamps across streams.
+
+```mermaid
+graph LR
+    capture_audio_src --> capture_audio_path --> capture_audio_sink
+    capture_audio_src --> capture_sync
+    capture_video_src --> capture_sync
+    capture_video_src --> capture_video_path --> capture_video_sink
+    capture_audio_sink --> capture_muxer
+    capture_video_sink --> capture_muxer
+    capture_muxer --> capture_muxer_sink
+```
+
+## 🔊 Audio Sources
+
+Audio sources are used to acquire audio data from audio input devices connected via various buses (like I2S, USB, etc.).
+
+**Interface**: `esp_capture_audio_src_if_t`
+
+Built-in sources:
+
+- `esp_capture_new_audio_dev_src`: Codec-based audio capture
+- `esp_capture_new_audio_aec_src`: Codec-based audio capture with Acoustic Echo Cancellation (AEC)
+
+## 🎥 Video Sources
+
+Video sources are used to capture video data from video input devices connected via various buses (like SPI, MIPI, USB, etc.).
+
+**Interface**: `esp_capture_video_src_if_t`
+
+Built-in sources:
+
+- `esp_capture_new_video_v4l2_src`: V4L2 camera input (via `esp_video`)
+- `esp_capture_new_video_dvp_src`: DVP camera input
+
+## 🕓 Stream Synchronization
+
+Stream synchronization is achieved by the `capture_sync` module. `capture_sync` aligns audio and video frame timestamps for synchronized playback or muxing. It is automatically configured through `esp_capture_open`.
+
+## 🔧 Audio/Video Processing Paths
+
+**Interface**: `esp_capture_path_mngr_if_t`
+
+### 🎚️ Audio Path
+
+Built-in:
+
+- `esp_capture_new_gmf_audio_mngr`: Creates audio processing path using `ESP-GMF` with elements like:
+  - `aud_rate_cvt` – Sample rate conversion
+  - `aud_ch_cvt` – Channel conversion (mono ↔ stereo)
+  - `aud_bit_cvt` – Bit depth conversion`
+  - `aud_enc` – Audio encoder
+
+**Pipeline Builders** (`esp_capture_pipeline_builder_if_t`):
+
+- `esp_capture_create_auto_audio_pipeline`: Auto-generated audio pipeline based on negotiation
+- `esp_capture_create_audio_pipeline`: Prebuilt audio template pipeline
+
+### 🎛️ Video Path
+
+Built-in:
+
+- `esp_capture_new_gmf_video_mngr`: Creates video processing path using `ESP-GMF` with elements like:
+  - `vid_ppa` – Resize, crop, color conversion
+  - `vid_overlay` – Text/graphic overlays
+  - `vid_fps_cvt` – Framerate conversion
+  - `vid_enc` – Video encoder
+
+**Pipeline Builders**:
+
+- `esp_capture_create_auto_video_pipeline`: Auto-generated video pipeline based on negotiation
+- `esp_capture_create_video_pipeline`: Prebuilt video template pipeline
+
+## 🎞️ Muxing
+
+Mux audio/video into containers for storage or streaming:
+
+- MP4: File-based only
+- TS: Supports streaming and file-based
+
+### Data Flow Control for Muxers
+
+The module provides flexible data flow control options for muxers:
+
+- **Muxer-only mode**: All data is consumed by the muxer, preventing access to raw audio/video streams
+- **Streaming while storage**: Simultaneous storage and streaming when supported by the muxer
+- **Unified API**: Use `esp_capture_sink_acquire_frame` for both muxer output and direct stream access
+
+## 🖋️ Overlays
+
+Overlays are used to mix text or images into original video frames.
+Typical use cases include: Adding real-time timestamps or statistical data onto video frames.
+
+**Interface**: `esp_capture_overlay_if_t`
+
+- Built-in: `esp_capture_new_text_overlay`
+- Automatically handled if overlay is present in the video path
+
+## ⚡ Auto Capture Mode
+
+Simplified configuration by automatically connecting sources, paths, and sinks.
+Typical call sequence for auto capture is shown below (using audio capture as an example):
+
+```mermaid
+sequenceDiagram
+    participant App as Application
+    participant AudioSrc as Audio Source
+    participant Capture as ESP Capture
+    participant Sink as Capture Sink
+
+    App->>AudioSrc: esp_capture_new_audio_dev_src(...)
+    AudioSrc-->>App: audio_src handle
+
+    App->>Capture: esp_capture_open(&cfg, &capture)
+    Note over App,Capture: cfg.audio_src = audio_src
+
+    App->>Capture: esp_capture_sink_setup(capture, 0, &sink_cfg, &sink)
+
+    App->>Sink: esp_capture_sink_enable(sink, ESP_CAPTURE_RUN_MODE_ALWAYS)
+
+    App->>Capture: esp_capture_start(capture)
+
+    loop Frame Processing
+        App->>Sink: esp_capture_sink_acquire_frame(sink, &frame, false)
+        App->>Sink: esp_capture_sink_release_frame(sink, &frame)
+    end
+
+    App->>Capture: esp_capture_stop(capture)
+```
+
+For detailed examples, see [audio_capture](examples/audio_capture/README.md) and [video_capture](examples/video_capture/README.md)
+
+## 🧩 Customizing Auto Pipelines
+
+1. Register Custom Elements
+
+```c
+esp_capture_register_element(capture, ESP_CAPTURE_STREAM_TYPE_AUDIO, proc_element);
+```
+
+2. Customize Pipeline Before Start
+
+```c
+const char *elems[] = { "aud_ch_cvt", "aud_rate_cvt", "aud_enc" };
+esp_capture_sink_build_pipeline(sink, ESP_CAPTURE_STREAM_TYPE_AUDIO, elems, 3);
+```
+
+## 🤝 Auto-Negotiation
+
+### Audio
+
+- Automatically inserts elements like `aud_rate_cvt`, `aud_ch_cvt` on demand
+- Negotiates format based on encoder requirements
+- Elements are configured based on negotiation results
+
+Built-in:
+
+- `esp_capture_audio_pipeline_auto_negotiate` – Auto negotiate from audio source to multiple audio sinks
+
+### Video
+
+- Automatically inserts `vid_ppa`, `vid_fps_cvt` on demand
+- Prioritizes high-quality format
+- Negotiates source format based on encoder capabilities
+
+Built-in:
+
+- `esp_capture_video_pipeline_auto_negotiate` – Auto negotiate from video source to multiple video sinks
+
+### Fixed Negotiation for Sources
+
+In some cases, auto-negotiation for source format and information may not meet requirements.
+Audio sources and video sources support `set_fixed_caps` to fix source format settings and avoid negotiation failure cases.
+
+## ❌ When Auto-Negotiation Fails
+
+In complex pipelines, auto-negotiation may fail (e.g., redundant sample rate converter in one pipeline). Manual configuration is recommended.
+
+## 📦 Binary Size Optimization
+
+Unused elements are excluded unless registered.
+
+### Menuconfig Options
+
+Enable features only when needed:
+- `CONFIG_ESP_CAPTURE_ENABLE_AUDIO`: Enable audio support
+- `CONFIG_ESP_CAPTURE_ENABLE_VIDEO`: Enable video support
+
+### Optional Registrations
+
+- `mp4_muxer_register()` / `ts_muxer_register()` – on-demand muxers
+- `esp_audio_enc_register_default()` / `esp_video_enc_register_default()` – customize encoder usage via menuconfig
+
+## 🔧 Extending esp_capture
+
+You can extend `esp_capture` by:
+
+1. Adding a custom capture source
+2. Implementing a new muxer using `esp_muxer`
+3. Creating new encoders via `esp_audio_codec` / `esp_video_codec`