API function documentation added (using Doxygen)

Author: ReinhardKeil

.github/workflows/pack.yml

@@ -28,15 +28,16 @@ jobs:
 
       - name: Setup Python
         uses: actions/setup-python@v5
-      
+
       - name: Install MkDocs
         run: |
           pip install mkdocs
           pip install mkdocs-mermaid2-plugin
+          pip install mkdoxy
 
       - uses: Open-CMSIS-Pack/gen-pack-action@main
         with:
-          doxygen-version: none
+          doxygen-version: 1.13.2 
           packchk-version: 1.3.95
           gen-pack-script: ./gen_pack.sh
           gen-pack-output: ./output

README.md

@@ -2,43 +2,37 @@
 
 The **Synchronous Data Stream (SDS) Framework** implements a data stream management, provides methods and helper tools for developing and optimizing embedded applications that integrate DSP and ML algorithms. This framework may be used stand-alone, but also in combination with [**CMSIS-Stream**](https://github.com/ARM-software/CMSIS-Stream) that allows to combine algorithms using a compute graph.
 
-## Overview 
+## Features
 
-- Implements a flexible data stream management for sensor and audio data interfaces
-   - Supports data streams from multiple interfaces including provisions for time drifts. 
+- Implements a flexible data stream management for sensor, audio, and video interfaces that process data streams.
+- Supports data streams from multiple interfaces (i.e. for sensor fusion) including provisions for time drifts.
+- **Record real-world data** with original data sources of the target hardware for analysis and development.
+- **Playback real-world data** for algorithm validation using target hardware or [Arm Virtual Hardware - FVP](https://github.com/arm-software/avh).
 
-- Provides methods to **record real-world data** for analysis and development
-  - Input to Digital Signal Processing (DSP) development tools such as filter designers
-  - Input to Machine Learning (ML) model classification, training, and performance optimization
-  - Verify execution of DSP algorithm on Cortex-M targets with off-line tools
+The captured data streams are stored in SDS data files. A [YAML metadata file](./schema/README.md) can be used to describe the content. The SDS data files have several use cases such as:
 
-- **Playback real-world data** for algorithm validation using Arm Virtual Hardware
-  - Input to Digital Signal Processing (DSP) development tools such as filter designers
-  - Input to Machine Learning (ML) model classification, training, and performance optimization 
-  - Verify execution of DSP algorithm on Cortex-M targets with off-line tools
+- Input to Digital Signal Processing (DSP) development tools such as filter designers
+- Input to Machine Learning (ML) model classification, training, and performance optimization 
+- Verify execution of DSP algorithm on Cortex-M targets with off-line tools
 
-- Defines binary data format with [YAML metadata file](./schema/README.md).
+[Python-based utilities](./utilities/README.md) are provided for recording, playback, visualization, and data conversion
 
-- [Python-based utilities](./utilities/README.md) for recording, playback, visualization, and data conversion
-
-Refer to ["ML Developers Guide for Cortex-M Processors and Ethos-U NPU" - "Tool Support" - "SDS Framework Usage"](https://developer.arm.com/documentation/109267/0100/Tool-Support-for-the-Arm-Ethos-U-NPU/SDS-Framework-Usage) for more information.
-
-## Interfaces
-
-The [SDS Framework interfaces](./sds/README.md) implement the SDS streaming interfaces via different methods. The functions of the **sds.h Circular Buffer Handling** may be used at any level of a Compute Graph to provide inputs or capture outputs of an DSP or ML algorithm.
-
-![Interfaces](./documentation/images/SDS-Interfaces.png "Interfaces")
-
-## Repository top-level structure
+## Repository structure
 
 Directory                         | Description
 ----------------------------------|-------------------------------
 [examples](./examples)            | Example implementations for various evaluation boards
-[documentation](./documentation/) | User documentation of the SDS Framework
+[documentation](./documentation/) | [User documentation](https://arm-software.github.io/SDS-Framework/main/index.html) of the SDS Framework
 [schema](./schema)                | Schema for SDS File Format
 [sds](./sds)                      | Interfaces of the SDS Framework for Cortex-M devices
 [utilities](./utilities)          | Python scripts for processing of SDS binary data files
 
+## Related
+
+- [SDS Examples](https://github.com/Arm-Examples/sds-examples)
+- [ML Developers Guide for Cortex-M Processors and Ethos-U NPU](https://developer.arm.com/documentation/109267)
+- [Arm Virtual Hardware - FVP](https://github.com/arm-software/avh)
+
 ## License
 
 The SDS Framework is licensed under [![License](https://img.shields.io/github/license/arm-software/sds-framework?label)](https://github.com/ARM-software/sds-framework/blob/main/LICENSE).
@@ -47,6 +41,11 @@ The SDS Framework is licensed under [![License](https://img.shields.io/github/li
 
 Contributions are accepted under [![License](https://img.shields.io/github/license/arm-software/CMSIS_6?label)](https://github.com/ARM-software/CMSIS_6/blob/main/LICENSE). Only submit contributions where you have authored all of the code.
 
+The documentation is generated using [MKDocs](https://www.mkdocs.org/) with the following additional plugins:
+
+- [mermaid2](https://mkdocs-mermaid2.readthedocs.io/en/latest/) for sequence diagrams.
+- [mkdoxy](https://pypi.org/project/mkdoxy) for API documentation.
+
 ### Issues and Labels
 
 Please feel free to raise an [issue on GitHub](https://github.com/ARM-software/sds-framework/issues)

documentation/index.md

@@ -13,6 +13,7 @@ This user's guide assumes basic knowledge about Cortex-M software development. I
 - [**SDS Interface**](sds_interface.md) describes the various interfaces (USB, Socket, File System) for connecting to the target.
 - [**Examples**](examples.md) lists the examples that are available.
 - [**Utilities**](utilities.md) explains Python based utilities that operate with SDS data files for converting, viewing, recording, and playback.
+- [**API Modules**](SDS_API\modules.md) describes the C interface of the SDS functions that may be used in the target system.
 
 ## Revision History
 

documentation/theory.md

@@ -7,7 +7,8 @@ The SDS Framework enables to record and playback one or more data streams to an
 
 The DSP or ML algorithms that are tested operate on blocks and are executed periodically. This documentation uses these terms:
 
-- **Block size**: is the number of bytes processed by a DSP or ML compute node.
+- **Data Block**: is a set of input or output data which is processed in one step by a DSP or ML compute note.
+- **Block size**: is the number of bytes of a data block. 
 - **Interval**: is the periodic time interval that the compute node executes.
 
 ![SDSIO Interface for Player and Recorder](images/SDS-InOut.png)
@@ -121,7 +122,7 @@ In most cases the granularity of an RTOS tick (typically 1ms) is a good choice f
 ## SDS File Format
 
 The SDS file format is described [here](https://github.com/ARM-software/SDS-Framework/tree/main/schema).  Each call to 
-`sdsRecWrite` creates one record.
+`sdsRecWrite` writes one data block.
 
 ## SDSIO Server Protocol
 
@@ -294,3 +295,89 @@ but sds_rec.c does not really use this information
 
 https://github.com/Arm-Examples/SDS-Examples/blob/main/Hardware/DataTest/rec_management.c
 - do we really need `recdone`?
+
+## Guidelines for Stream Buffer sizing and Threshold settings
+
+### Overview
+
+The **SDS Recorder/Player** uses memory buffers to manage data recording and playback efficiently.
+Proper buffer and threshold configurations optimize performance by balancing data production, consumption, and system resource utilization.
+
+### Stream Buffer Size
+
+The size of memory buffers affects the balance between data production and consumption.
+
+The **absolute minimum stream buffer size** should be large enough to store **one maximum record along with its header (8 bytes)**.
+
+The **recommended stream buffer size** should be large enough to store **at least two maximum records along with headers (8 bytes per record)** and
+can be rounded up for convenience.
+
+The record size generally corresponds to the data size used by the underlying technology.
+For example, in typical **ML applications**, the record size should ideally match the **data slice required by the DSP process**.
+
+If sufficient RAM is available, increasing the buffer size can improve performance.
+
+### Threshold Settings for SDS Recorder/Player
+
+#### **Function of Thresholds**
+
+- **For the SDS Recorder:** Determines when data writing to the **I/O** begins.
+- **For the SDS Player:** Determines when data reading from the **I/O** begins.
+
+The **recommended threshold setting** is **half of the stream buffer size**, enabling a **double-buffering technique** where one half of the buffer is transferred while the other half continues to fill or be consumed.
+
+A well-optimized system ensures timely data transfer over the I/O:
+
+- For the **Recorder**: Previously acquired data must be transferred before the remaining of the buffer fills with new data.
+- For the **Player**: New playback data should be transferred before the previously transferred playback data is consumed.
+
+#### **Impact on System Performance**
+
+The threshold setting directly affects system performance, as **I/O transfers temporarily occupy the CPU**.
+During these operations, other system processes may experience limited CPU availability.
+
+If the system requires additional CPU resources for other tasks, adjustments can be made by:
+
+1. **Increasing the priority of critical threads**.
+2. **Reducing the threshold setting**, which shortens the duration of each transfer but increases the frequency of transfers.
+
+> **Note:** Thresholds operate based on discrete record sizes. A threshold is only triggered when a write or read operation
+> surpasses (for write) or falls below (for read) the set limit.
+> When handling large records, breaking them into smaller chunks may be necessary to optimize system performance.
+
+### Additional Settings Affecting I/O Bandwidth
+
+Several additional factors influence I/O bandwidth, including:
+
+1. **Temporary Recorder/Player buffer size** (configured in the `sds_rec_config.h` / `sds_play_config.h` files).
+2. **I/O low-level buffering**.
+
+#### **Optimizing I/O Buffering**
+
+For **optimal performance**, the **temporary Recorder/Player buffer size should match the I/O low-level buffer size**.
+
+For some interfaces, the I/O low-level buffer size can be configurable, for others, it is fixed.
+
+Example: The **USB Virtual COM interface** allows I/O low-level buffer size configuration.
+It is recommended to set this buffer size as a multiple of the **bulk endpoint maximum packet size** (512 bytes for USB high-speed connections).
+
+### Example configurations for typical ML use cases
+
+| **ML Use Case**      | **DSP slice**                       | **Calculation**                                                                      | **Buffer Size**  | **Threshold** |
+| -------------------- | ----------------------------------- | ------------------------------------------------------------------------------------ | ---------------- | ------------- |
+| **Motion detection** | 125 accelerometer samples           | `2 × [(125 samples × 3 axes × 4 bytes per axis) + 8] = 3016 -> rounded to 3072`      | **3072 bytes**   | **1536**      |
+| **Keyword spotting** | 250 ms of audio data samples (16kHz)| `2 × [(4,000 audio samples × 4 bytes per sample) + 8] = 32,016 -> rounded to 32,768` | **32768 bytes**  | **16384**     |
+| **Object detection** | 1 picture (320x320)                 | `2 × [(320 × 320 pixels × 4 bytes per pixel) + 8] = 819,216`                         | **819216 bytes** | **409608**    |
+
+### **Best Practices Summary**
+
+- **Ensure buffer sizes align with DSP ata slice sizes** for efficient ML processing.
+
+- **Use double-buffering** to enhance I/O efficiency.
+
+- **Adjust threshold settings** to balance performance and CPU usage.
+
+- **Match temporary buffer sizes to I/O low-level buffer sizes** where possible.
+
+By following these guidelines, the **SDS Recorder/Player** can be configured efficiently to balance **performance, latency, and CPU utilization**,
+ensuring smooth data processing and system stability.

mkdocs.yml

@@ -8,10 +8,20 @@ nav:
     - SDS Interface: sds_interface.md
     - Examples: examples.md
     - Utilities: utilities.md
+    - API Modules: SDS_API\modules.md
 
 plugins:
     - search
     - mermaid2    # pip install mkdocs-mermaid2-plugin
+    - mkdoxy:     # pip install mkdoxy  (may require PYTHONPATH to be set)
+        projects:
+          SDS_API:
+            src-dirs: ./sds/include
+            full-doc: true
+            doxy-cfg:
+              FILE_PATTERNS: "*.h *.txt"
+
+
 theme:
   name: readthedocs
 markdown_extensions:

sds/include/sds.h

@@ -29,7 +29,7 @@ extern "C"
 // ==== Synchronous Data Stream (SDS) ====
 
 /// Identifier
-typedef void *sdsId_t;
+typedef void *sdsId_t;                      ///< handle to SDS stream
 
 /// Function return codes
 #define SDS_OK                  (0)         ///< Operation completed successfully
@@ -39,7 +39,12 @@ typedef void *sdsId_t;
 #define SDS_EVENT_DATA_LOW      (1UL << 0)  ///< Data bellow or equal to threshold
 #define SDS_EVENT_DATA_HIGH     (1UL << 1)  ///< Data above or equal to threshold
 
-/// Event callback function
+/**
+  \typedef void (*sdsPlayEvent_t) (sdsPlayId_t id, uint32_t event)
+  \brief       Call back function for SDS circular buffer handling
+  \param[in]   id             handle to SDS file for playback
+  \param[in]   event          event code
+*/
 typedef void (*sdsEvent_t) (sdsId_t id, uint32_t event, void *arg);
 
 /**
@@ -49,22 +54,22 @@ typedef void (*sdsEvent_t) (sdsId_t id, uint32_t event, void *arg);
   \param[in]   buf_size       buffer size in bytes
   \param[in]   threshold_low  data low threshold in bytes
   \param[in]   threshold_high data high threshold in bytes
-  \return      \ref sdsId_t
+  \return      \ref sdsId_t   Handle to SDS stream
 */
 sdsId_t sdsOpen (void *buf, uint32_t buf_size, uint32_t threshold_low, uint32_t threshold_high);
 
 /**
   \fn          int32_t sdsClose (sdsId_t id)
   \brief       Close stream.
-  \param[in]   id             \ref sdsId_t
+  \param[in]   id             \ref sdsId_t handle to SDS stream
   \return      return code
 */
 int32_t sdsClose (sdsId_t id);
 
 /**
   \fn          int32_t  sdsRegisterEvents (sdsId_t id, sdsEvent_t event_cb, uint32_t event_mask, void *event_arg)
   \brief       Register stream events.
-  \param[in]   id             \ref sdsId_t
+  \param[in]   id             \ref sdsId_t handle to SDS stream
   \param[in]   event_cb       pointer to \ref sdsEvent_t
   \param[in]   event_mask     event mask
   \param[in]   event_arg      event argument
@@ -75,7 +80,7 @@ int32_t sdsRegisterEvents (sdsId_t id, sdsEvent_t event_cb, uint32_t event_mask,
 /**
   \fn          uint32_t sdsWrite (sdsId_t id, const void *buf, uint32_t buf_size)
   \brief       Write data to stream.
-  \param[in]   id             \ref sdsId_t
+  \param[in]   id             \ref sdsId_t handle to SDS stream
   \param[in]   buf            pointer to buffer with data to write
   \param[in]   buf_size       buffer size in bytes
   \return      number of bytes written
@@ -85,7 +90,7 @@ uint32_t sdsWrite (sdsId_t id, const void *buf, uint32_t buf_size);
 /**
   \fn          uint32_t sdsRead (sdsId_t id, void *buf, uint32_t buf_size)
   \brief       Read data from stream.
-  \param[in]   id             \ref sdsId_t
+  \param[in]   id             \ref sdsId_t handle to SDS stream
   \param[out]  buf            pointer to buffer for data to read
   \param[in]   buf_size       buffer size in bytes
   \return      number of bytes read
@@ -95,15 +100,15 @@ uint32_t sdsRead (sdsId_t id, void *buf, uint32_t buf_size);
 /**
   \fn          int32_t sdsClear (sdsId_t id)
   \brief       Clear stream data.
-  \param[in]   id             \ref sdsId_t
+  \param[in]   id             \ref sdsId_t handle to SDS stream
   \return      return code
 */
 int32_t sdsClear (sdsId_t id);
 
 /**
   \fn          uint32_t sdsGetCount (sdsId_t id)
   \brief       Get data count in stream.
-  \param[in]   id             \ref sdsId_t
+  \param[in]   id             \ref sdsId_t handle to SDS stream
   \return      number of bytes in stream
 */
 uint32_t sdsGetCount (sdsId_t id);