Merge branch 'feature/add_wn_element' into 'main'

gmf_ai_audio: Add wn element

See merge request adf/multimedia/esp-gmf!67

Author: jason-mao

elements/gmf_ai_audio/CHANGELOG.md

@@ -4,8 +4,14 @@
 
 ### Features
 
-- Add API `esp_gmf_afe_set_event_cb` to set AFE element event callback
-- Replaced the interface for decoding reconfig
+- Added `esp_gmf_afe_set_event_cb` API to register AFE element event callbacks
+- Replaced the interface for decoder reconfiguration
+- Introduced `esp_gmf_wn` element to support wake word detection
+
+### Bug Fixes
+
+- Fixed an issue where the output attribute of the `esp_gmf_afe` element was not set correctly
+- Fixed a build error occurring when SPIRAM is disabled
 
 ## v0.6.2
 

elements/gmf_ai_audio/CMakeLists.txt

@@ -1,5 +1,12 @@
+set(srcs "./src/esp_gmf_wn.c")
+
+if((${IDF_TARGET} STREQUAL "esp32s3") OR (${IDF_TARGET} STREQUAL "esp32p4") OR (${IDF_TARGET} STREQUAL "esp32"))
+    list(APPEND srcs "./src/esp_gmf_afe_manager.c" "./src/esp_gmf_afe.c" "./src/esp_gmf_aec.c")
+endif()
+
 idf_component_register(
-    SRC_DIRS "./src"
+    SRCS "${srcs}"
     INCLUDE_DIRS "./include"
+    PRIV_INCLUDE_DIRS "./private_include"
     REQUIRES esp-sr esp_timer
 )

elements/gmf_ai_audio/README.md

@@ -4,9 +4,10 @@
 
 `GMF AI Audio` is an artificial intelligence audio processing module that provides users with convenient and easy-to-use intelligent audio processing algorithms at the [GMF](https://github.com/espressif/esp-gmf) framework, such as voice wake-up, command word recognition, and echo cancellation. Currently, it offers the following modules based on `esp-sr`:
 
-* [esp_gmf_afe_manager](./src/esp_gmf_afe_manager.c): `audio front end(afe)` manager
-* [esp_gmf_aec](./src/esp_gmf_aec.c): Echo Cancellation
-* [esp_gmf_afe](./src/esp_gmf_afe.c): An easy-to-use interface based on the `audio front end (afe)` from `esp-sr`, providing functionalities such as voice wake-up, command word recognition, and speech detection
+- [esp_gmf_afe_manager](./src/esp_gmf_afe_manager.c): `audio front end(afe)` manager
+- [esp_gmf_aec](./src/esp_gmf_aec.c): Echo Cancellation
+- [esp_gmf_wn](./src/esp_gmf_wn.c): A standalone wake word detection module that can be used independently
+- [esp_gmf_afe](./src/esp_gmf_afe.c): An easy-to-use interface based on the `audio front end (afe)` from `esp-sr`, providing functionalities such as voice wake-up, command word recognition, and speech detection
 
 ## AFE Manager `esp_gmf_afe_manager`
 
@@ -53,7 +54,27 @@
 
 ### Technical Specifications
 
-- Supported Hardware: `ESP32`, `ESP32-S3`, `ESP32-C5`, and `ESP32-P4`
+- Supported Hardware: `ESP32`, `ESP32-S3`, and `ESP32-P4`
+- Input Format
+  - Sampling Rate: 8 kHz and 16 kHz
+  - Bit Width: 16-bit PCM
+- Channel Configuration: String identifiers (e.g., `MMNR`)
+  - `M`: Microphone Channel
+  - `R`: Reference Signal Channel
+  - `N`: Invalid Signal
+- Output Format: 16-bit single-channel PCM
+
+## Wake Word Detection `esp_gmf_wn`
+
+### Features
+
+- Runs independently without AFE dependency, low resource usage
+- Supports multi-channel input and single-channel output
+- Notifies wake word detection results via callback function
+
+### Technical Specifications
+
+- Supported Hardware: `ESP32`, `ESP32-S3`, `ESP32-C3`, `ESP32-C5` and `ESP32-P4`
 - Input Format
   - Sampling Rate: 16 kHz
   - Bit Width: 16-bit PCM
@@ -165,7 +186,7 @@ The following illustrates state transitions when features are enabled. The `/` c
 
 - **Enable Both Wake Word Detection and VAD**
 
-  This scenario combines command word detection with voice activity detection to avoid frequent voice activity events outside the wake word interval
+  This scenario combines wake word detection with voice activity detection to avoid frequent voice activity events outside the wake word interval
 
   Modify the configuration in the example [wwe](./examples/wwe/main/main.c) to use this scenario
 
@@ -193,8 +214,8 @@ The following illustrates state transitions when features are enabled. The `/` c
 
 Users need to decide when to start command word detection. A typical use case is to enable detection after the state machine pushes a (WAKEUP_START) event and determine the next operation based on the detected command word index in the callback function
 
-* Command word detection is independent of the wake word state machine
-* Command word detection supports continuous detection until timeout
+- Command word detection is independent of the wake word state machine
+- Command word detection supports continuous detection until timeout
 
 ## Usage
 

elements/gmf_ai_audio/README_CN.md

@@ -4,9 +4,10 @@
 
 `GMF AI Audio` 是一个人工智能语音处理模块，它在 [GMF](https://github.com/espressif/esp-gmf) 层面为用户提供方便、易用的语音唤醒、命令词识别和回声消除等常用智能语音处理算法。目前基于 `esp-sr` 提供以下模块：
 
-* [esp_gmf_afe_manager](./src/esp_gmf_afe_manager.c): `audio front end(afe)` 管理器
-* [esp_gmf_aec](./src/esp_gmf_aec.c): 回声消除
-* [esp_gmf_afe](./src/esp_gmf_afe.c): 基于 `esp-sr` 中 `audio front end (afe)` 实现的易用接口，提供了语音唤醒，命令词识别，人声检测等语音识别的功能
+- [esp_gmf_afe_manager](./src/esp_gmf_afe_manager.c): `audio front end(afe)` 管理器
+- [esp_gmf_aec](./src/esp_gmf_aec.c): 回声消除
+- [esp_gmf_wn](./src/esp_gmf_wn.c): 独立的唤醒词检测模块，不依赖AFE
+- [esp_gmf_afe](./src/esp_gmf_afe.c): 基于 `esp-sr` 中 `audio front end (afe)` 实现的易用接口，提供了语音唤醒，命令词识别，人声检测等语音识别的功能
 
 ## AFE 管理器 `esp_gmf_afe_manager`
 
@@ -53,7 +54,7 @@
 
 ### 技术规格
 
-- 支持硬件：`ESP32`、`ESP32-S3`、`ESP32-C5` 和 `ESP32-P4`
+- 支持硬件：`ESP32`、`ESP32-S3` 和 `ESP32-P4`
 - 输入格式
   - 采样率：16 kHz
   - 位宽：16-bit PCM
@@ -63,6 +64,26 @@
   - `N`：无效信号
 - 输出格式：16-bit 单通道 PCM
 
+## 唤醒词检测 `esp_gmf_wn`
+
+### 功能
+
+- 独立运行，不依赖AFE，资源占用小
+- 支持多通道输入，单通道输出
+- 通过回调函数通知唤醒词检测结果
+
+### 技术规格
+
+- 支持硬件：`ESP32`、`ESP32-S3`、`ESP32-C3`、`ESP32-C5` 和 `ESP32-P4`
+- 输入格式
+  - 采样率：8 kHz, 16 kHz
+  - 位宽：16-bit PCM
+- 通道配置：字符串标识（如 `MMNR`）
+  - `M`：麦克风通道
+  - `R`：回采信号通道
+  - `N`：无效信号
+- 输出格式：16-bit 单通道 PCM
+
 ## 语音识别 `esp_gmf_afe`
 
 ### 功能
@@ -165,7 +186,7 @@
 
 - **使能唤醒和VAD**
 
-  该场景结合命令词与人声检测，避免在唤醒间隔之外频繁触发人声检测事件
+  该场景结合唤醒词与人声检测，避免在唤醒间隔之外频繁触发人声检测事件
 
   修改示例 [wwe](./examples/wwe/main/main.c) 中的配置，从而使用该场景
 
@@ -193,8 +214,8 @@
 
 用户需决定何时开始检测命令词，如典型的使用场景：在状态机推送（WAKEUP_START）事件之后，开启检测，并在回调函数中根据检测到的命令词序号决定下一步操作
 
-* 命令词检测与唤醒状态机独立
-* 命令词支持连续检测，直到超时
+- 命令词检测与唤醒状态机独立
+- 命令词支持连续检测，直到超时
 
 ## 示例
 

elements/gmf_ai_audio/examples/common/gmf_setup/esp_gmf_gpio_config.h

@@ -48,7 +48,7 @@ extern "C" {
 #define ESP_GMF_AMP_IO_NUM (GPIO_NUM_21)
 
 #elif CONFIG_IDF_TARGET_ESP32C3
-// ESP32c3-Lyra
+// XD_AIOT_C3
 // Sdmmc
 #define ESP_GMF_SD_CLK_IO_NUM       (GPIO_NUM_NC)
 #define ESP_GMF_SD_CMD_IO_NUM       (GPIO_NUM_NC)
@@ -65,25 +65,61 @@ extern "C" {
 #define ESP_GMF_SD_WIDTH            (1)
 
 // I2C
-#define ESP_GMF_I2C_SDA_IO_NUM      (GPIO_NUM_9)
-#define ESP_GMF_I2C_SCL_IO_NUM      (GPIO_NUM_8)
+#define ESP_GMF_I2C_SDA_IO_NUM      (GPIO_NUM_0)
+#define ESP_GMF_I2C_SCL_IO_NUM      (GPIO_NUM_1)
+
+// I2S
+#define ESP_GMF_I2S_DAC_MCLK_IO_NUM (GPIO_NUM_10)
+#define ESP_GMF_I2S_DAC_BCLK_IO_NUM (GPIO_NUM_8)
+#define ESP_GMF_I2S_DAC_WS_IO_NUM   (GPIO_NUM_12)
+#define ESP_GMF_I2S_DAC_DO_IO_NUM   (GPIO_NUM_11)
+#define ESP_GMF_I2S_DAC_DI_IO_NUM   (GPIO_NUM_7)
+
+#define ESP_GMF_I2S_ADC_MCLK_IO_NUM (GPIO_NUM_10)
+#define ESP_GMF_I2S_ADC_BCLK_IO_NUM (GPIO_NUM_8)
+#define ESP_GMF_I2S_ADC_WS_IO_NUM   (GPIO_NUM_12)
+#define ESP_GMF_I2S_ADC_DO_IO_NUM   (GPIO_NUM_11)
+#define ESP_GMF_I2S_ADC_DI_IO_NUM   (GPIO_NUM_7)
+
+// PA
+#define ESP_GMF_AMP_IO_NUM          (GPIO_NUM_13)
+#elif CONFIG_IDF_TARGET_ESP32C5
+// ESP-SPOT
+// Sdmmc
+#define ESP_GMF_SD_CLK_IO_NUM       (GPIO_NUM_NC)
+#define ESP_GMF_SD_CMD_IO_NUM       (GPIO_NUM_NC)
+#define ESP_GMF_SD_D0_IO_NUM        (GPIO_NUM_NC)
+#define ESP_GMF_SD_D1_IO_NUM        (GPIO_NUM_NC)
+#define ESP_GMF_SD_D2_IO_NUM        (GPIO_NUM_NC)
+#define ESP_GMF_SD_D3_IO_NUM        (GPIO_NUM_NC)
+#define ESP_GMF_SD_D4_IO_NUM        (GPIO_NUM_NC)
+#define ESP_GMF_SD_D5_IO_NUM        (GPIO_NUM_NC)
+#define ESP_GMF_SD_D6_IO_NUM        (GPIO_NUM_NC)
+#define ESP_GMF_SD_D7_IO_NUM        (GPIO_NUM_NC)
+#define ESP_GMF_SD_CD_IO_NUM        (GPIO_NUM_NC)
+#define ESP_GMF_SD_WP_IO_NUM        (GPIO_NUM_NC)
+#define ESP_GMF_SD_WIDTH            (1)
+
+// I2C
+#define ESP_GMF_I2C_SDA_IO_NUM      (GPIO_NUM_25)
+#define ESP_GMF_I2C_SCL_IO_NUM      (GPIO_NUM_26)
+#define ESP_SPOT_ES8311_I2C_ADDR    (0x18)
 
 // I2S
 #define ESP_GMF_I2S_DAC_MCLK_IO_NUM (GPIO_NUM_NC)
-#define ESP_GMF_I2S_DAC_BCLK_IO_NUM (GPIO_NUM_NC)
-#define ESP_GMF_I2S_DAC_WS_IO_NUM   (GPIO_NUM_NC)
-#define ESP_GMF_I2S_DAC_DO_IO_NUM   (GPIO_NUM_3)
-#define ESP_GMF_I2S_DAC_DI_IO_NUM   (GPIO_NUM_NC)
+#define ESP_GMF_I2S_DAC_BCLK_IO_NUM (GPIO_NUM_7)
+#define ESP_GMF_I2S_DAC_WS_IO_NUM   (GPIO_NUM_8)
+#define ESP_GMF_I2S_DAC_DO_IO_NUM   (GPIO_NUM_9)
+#define ESP_GMF_I2S_DAC_DI_IO_NUM   (GPIO_NUM_6)
 
 #define ESP_GMF_I2S_ADC_MCLK_IO_NUM (GPIO_NUM_NC)
-#define ESP_GMF_I2S_ADC_BCLK_IO_NUM (GPIO_NUM_NC)
-#define ESP_GMF_I2S_ADC_WS_IO_NUM   (GPIO_NUM_NC)
-#define ESP_GMF_I2S_ADC_DO_IO_NUM   (GPIO_NUM_3)
-#define ESP_GMF_I2S_ADC_DI_IO_NUM   (GPIO_NUM_NC)
+#define ESP_GMF_I2S_ADC_BCLK_IO_NUM (GPIO_NUM_7)
+#define ESP_GMF_I2S_ADC_WS_IO_NUM   (GPIO_NUM_8)
+#define ESP_GMF_I2S_ADC_DO_IO_NUM   (GPIO_NUM_9)
+#define ESP_GMF_I2S_ADC_DI_IO_NUM   (GPIO_NUM_6)
 
 // PA
-#define ESP_GMF_AMP_IO_NUM          (GPIO_NUM_1)
-
+#define ESP_GMF_AMP_IO_NUM          (GPIO_NUM_23)
 #elif CONFIG_IDF_TARGET_ESP32S3
 // ESP32s3_Korvo_2
 // Sdmmc

elements/gmf_ai_audio/examples/common/gmf_setup/esp_gmf_setup_peripheral.c

@@ -14,7 +14,9 @@
 #include "driver/i2c.h"
 #include "esp_gmf_gpio_config.h"
 #include "esp_gmf_setup_peripheral.h"
+#if SOC_SDMMC_IO_POWER_EXTERNAL
 #include "sd_pwr_ctrl_by_on_chip_ldo.h"
+#endif  /* SOC_SDMMC_IO_POWER_EXTERNAL */
 #ifdef USE_ESP_GMF_ESP_CODEC_DEV_IO
 #include "esp_gmf_io_codec_dev.h"
 #include "esp_gmf_io_i2s_pdm.h"
@@ -136,11 +138,6 @@ static const audio_codec_data_if_t *setup_periph_new_i2s_data(void *tx_hd, void
 
 static void setup_periph_new_play_codec()
 {
-#ifdef CONFIG_IDF_TARGET_ESP32C3
-    gpio_if = audio_codec_new_gpio();
-    gpio_if->setup(ESP_GMF_AMP_IO_NUM, AUDIO_GPIO_DIR_OUT, AUDIO_GPIO_MODE_PULL_DOWN);
-    gpio_if->set(ESP_GMF_AMP_IO_NUM, 1);
-#else
     audio_codec_i2c_cfg_t i2c_ctrl_cfg = {.addr = ES8311_CODEC_DEFAULT_ADDR, .port = 0, .bus_handle = i2c_handle};
     out_ctrl_if = audio_codec_new_i2c_ctrl(&i2c_ctrl_cfg);
     gpio_if = audio_codec_new_gpio();
@@ -153,13 +150,15 @@ static void setup_periph_new_play_codec()
         .use_mclk = true,
     };
     out_codec_if = es8311_codec_new(&es8311_cfg);
-#endif  /* CONFIG_IDF_TARGET_ESP32C3 */
 }
 
 static void setup_periph_new_record_codec(void)
 {
-#if defined CONFIG_IDF_TARGET_ESP32P4
+#if defined(CONFIG_IDF_TARGET_ESP32P4) || defined(CONFIG_IDF_TARGET_ESP32C3) || defined(CONFIG_IDF_TARGET_ESP32C5)
     audio_codec_i2c_cfg_t i2c_ctrl_cfg = {.addr = ES8311_CODEC_DEFAULT_ADDR, .port = 0, .bus_handle = i2c_handle};
+#if defined(CONFIG_IDF_TARGET_ESP32C5)
+    i2c_ctrl_cfg.addr = ESP_SPOT_ES8311_I2C_ADDR;
+#endif
     in_ctrl_if = audio_codec_new_i2c_ctrl(&i2c_ctrl_cfg);
     gpio_if = audio_codec_new_gpio();
     // New output codec interface
@@ -266,12 +265,13 @@ void teardown_periph_record_codec(void *record_dev)
 }
 #endif  /* USE_ESP_GMF_ESP_CODEC_DEV_IO */
 
+#if SOC_SDMMC_HOST_SUPPORTED
 void esp_gmf_setup_periph_sdmmc(void **out_card)
 {
     sdmmc_card_t *card = NULL;
     sdmmc_host_t host = SDMMC_HOST_DEFAULT();
     sdmmc_slot_config_t slot_config = SDMMC_SLOT_CONFIG_DEFAULT();
-#if defined CONFIG_IDF_TARGET_ESP32P4
+#if SOC_SDMMC_IO_POWER_EXTERNAL
     host.slot = SDMMC_HOST_SLOT_0;
     host.max_freq_khz = SDMMC_FREQ_HIGHSPEED;
     sd_pwr_ctrl_ldo_config_t ldo_config = {
@@ -284,7 +284,7 @@ void esp_gmf_setup_periph_sdmmc(void **out_card)
         return;
     }
     host.pwr_ctrl_handle = pwr_ctrl_handle;
-#endif  /* defined CONFIG_IDF_TARGET_ESP32P4 */
+#endif  /* SOC_SDMMC_IO_POWER_EXTERNAL */
 
 #if defined CONFIG_IDF_TARGET_ESP32
     gpio_config_t sdcard_pwr_pin_cfg = {
@@ -326,8 +326,26 @@ void esp_gmf_setup_periph_sdmmc(void **out_card)
 
 void esp_gmf_teardown_periph_sdmmc(void *card)
 {
-    esp_vfs_fat_sdcard_unmount("/sdcard", card);
+    sdmmc_card_t *sd_card = card;
+    esp_vfs_fat_sdcard_unmount("/sdcard", sd_card);
+#if SOC_SDMMC_IO_POWER_EXTERNAL
+    int ret = sd_pwr_ctrl_del_on_chip_ldo(sd_card->host.pwr_ctrl_handle);
+    if (ret != ESP_OK) {
+        ESP_LOGE(TAG, "Failed to delete the on-chip LDO power control driver");
+    }
+#endif  /* SOC_SDMMC_IO_POWER_EXTERNAL */
+}
+#else
+void esp_gmf_setup_periph_sdmmc(void **out_card)
+{
+    // TODO nothing
+}
+
+void esp_gmf_teardown_periph_sdmmc(void *card)
+{
+    // TODO nothing
 }
+#endif  // SOC_SDMMC_HOST_SUPPORTED
 
 void esp_gmf_setup_periph_i2c(int port)
 {