Skip to content

Commit 3dbd0d6

Browse files
VladimirUmekVladimirUmek
committed
Doc: add documentation for implementation of vStream via VSI
1 parent 682f4b2 commit 3dbd0d6

4 files changed

Lines changed: 178 additions & 2 deletions

File tree

‎DoxyGen/simulation/simulation.dxy‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1009,6 +1009,7 @@ INPUT = ./src/main.md \
10091009
./src/Ref_audio.txt \
10101010
./src/Ref_video.txt \
10111011
./src/Ref_sensor.txt \
1012+
./src/Ref_vstream.txt \
10121013
../../interface/audio/include/audio_drv.h \
10131014
../../interface/video/include/video_drv.h \
10141015
../../interface/sensor/include/sensor_drv.h

‎DoxyGen/simulation/src/Ref_audio.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313

1414
\ref arm_vsi_video implements VSI-based video data streaming with local video or image files. Alternatively, PC's camera can be used as video input and PC's display as video output.
1515

16-
Other examples for VSI peripherals will be added in future as well.
16+
\ref arm_vsi_vstream implements VSI-based audio and video streaming using the standardized CMSIS-Driver vStream interface. It supports both file-based and live device-based streaming for audio input/output and video input/output.
1717

1818
*/
1919

‎DoxyGen/simulation/src/Ref_vsi.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ the \ref arm_vsi_py "VSI Python scripts" that implement peripheral behavior in t
2222
The path to the VSI python scripts is provided to the FVP using the configuration parameter `v_path`. Section \ref Execution gives an example.
2323

2424
Section \ref arm_vsi_cases explains in detail the reference implementations of VSI-based peripherals.
25-
Specifically \ref arm_vsi_audio implementing audio streaming input from a local audio wave file and \ref arm_vsi_video implementing video streaming from a local video or image file.
25+
Specifically \ref arm_vsi_audio implementing audio streaming input from a local audio wave file, \ref arm_vsi_video implementing video streaming from a local video or image file, and \ref arm_vsi_vstream implementing audio and video streaming using the CMSIS-Driver vStream interface.
2626

2727
Access to the VSI peripherals is defined in the following files:
2828

‎DoxyGen/simulation/src/Ref_vstream.txt‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
/**
2+
\addtogroup arm_vsi_vstream vStream via VSI
3+
\ingroup arm_vsi_cases
4+
\brief Fixed-size blocks streaming interface.
5+
\details
6+
7+
The vStream via VSI use case is implemented for AVH FVPs based on the general-purpose \ref arm_vsi.
8+
9+
It provides a [CMSIS-Driver vStream](https://arm-software.github.io/CMSIS_6/latest/Driver/group__vstream__interface__gr.html)
10+
implementation for audio and video data streaming, offering a standardized interface to interact with streaming devices.
11+
This allows seamless development on the FVP simulation environment and direct deployment on physical hardware without changing the application code.
12+
13+
The vStream implementation consists of:
14+
- \ref vstream_c_implementation — CMSIS-Driver vStream C implementation for the FVP firmware.
15+
- \ref vstream_py_implementation — Python VSI scripts that bridge the firmware to host audio and video devices.
16+
17+
18+
\defgroup vstream_c_implementation vStream C Implementation
19+
\ingroup arm_vsi_vstream
20+
\brief vStream C Implementation.
21+
\details
22+
23+
The vStream C implementation provides the [CMSIS-Driver vStream](https://arm-software.github.io/CMSIS-Driver/main/group__vStream__interface__gr.html)
24+
interface for the AVH FVPs. The C driver communicates with the FVP simulation layer by reading and writing VSI
25+
peripheral registers, which the corresponding Python scripts interpret and forward to the Audio or Video Server.
26+
27+
The following files implement the vStream C driver:
28+
29+
Item | Description
30+
:----------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------
31+
[./interface/vstream/source/vstream_audio_in.c](https://github.com/ARM-software/AVH/blob/main/interface/vstream/source/vstream_audio_in.c) | vStream AudioIn driver implementation for AVH FVPs. Exposes `Driver_vStreamAudioIn` on VSI instance 0.
32+
[./interface/vstream/source/vstream_audio_in.h](https://github.com/ARM-software/AVH/blob/main/interface/vstream/source/vstream_audio_in.h) | vStream AudioIn driver header file.
33+
[./interface/vstream/config/vstream_audio_in_config.h](https://github.com/ARM-software/AVH/blob/main/interface/vstream/config/vstream_audio_in_config.h) | Configuration header for the AudioIn driver.
34+
[./interface/vstream/source/vstream_audio_out.c](https://github.com/ARM-software/AVH/blob/main/interface/vstream/source/vstream_audio_out.c) | vStream AudioOut driver implementation for AVH FVPs. Exposes `Driver_vStreamAudioOut` on VSI instance 1.
35+
[./interface/vstream/source/vstream_audio_out.h](https://github.com/ARM-software/AVH/blob/main/interface/vstream/source/vstream_audio_out.h) | vStream AudioOut driver header file.
36+
[./interface/vstream/config/vstream_audio_out_config.h](https://github.com/ARM-software/AVH/blob/main/interface/vstream/config/vstream_audio_out_config.h) | Configuration header for the AudioOut driver.
37+
[./interface/vstream/source/vstream_video_in.c](https://github.com/ARM-software/AVH/blob/main/interface/vstream/source/vstream_video_in.c) | vStream VideoIn driver implementation for AVH FVPs. Exposes `Driver_vStreamVideoIn` on VSI instance 4.
38+
[./interface/vstream/source/vstream_video_in.h](https://github.com/ARM-software/AVH/blob/main/interface/vstream/source/vstream_video_in.h) | vStream VideoIn driver header file.
39+
[./interface/vstream/config/vstream_video_in_config.h](https://github.com/ARM-software/AVH/blob/main/interface/vstream/config/vstream_video_in_config.h) | Configuration header for the VideoIn driver.
40+
[./interface/vstream/source/vstream_video_out.c](https://github.com/ARM-software/AVH/blob/main/interface/vstream/source/vstream_video_out.c) | vStream VideoOut driver implementation for AVH FVPs. Exposes `Driver_vStreamVideoOut` on VSI instance 6.
41+
[./interface/vstream/source/vstream_video_out.h](https://github.com/ARM-software/AVH/blob/main/interface/vstream/source/vstream_video_out.h) | vStream VideoOut driver header file.
42+
[./interface/vstream/config/vstream_video_out_config.h](https://github.com/ARM-software/AVH/blob/main/interface/vstream/config/vstream_video_out_config.h) | Configuration header for the VideoOut driver.
43+
44+
### Driver configuration
45+
46+
The vStream driver behavior is controlled at compile-time through configuration header files located in `./interface/vstream/config/`.
47+
These files are copied to the project and can be modified to adapt the stream parameters to the application requirements.
48+
49+
<b>Audio stream configuration</b>
50+
51+
Configuration options for the audio vStream drivers are:
52+
53+
Parameter | Description | Available Options | Configuration Macro
54+
:-------------------------|:--------------------------------------------------------|:-------------------------------------|:--------------------------------
55+
Number of channels | Number of audio channels in the stream | Mono (1), Stereo (2) | `AUDIO_[IN\|OUT]_CHANNELS`
56+
Bits per sample | Number of bits of information in each sample | 8, 16, 24, or 32 | `AUDIO_[IN\|OUT]_SAMPLE_BITS`
57+
Sample rate | Number of audio samples captured or played per second | 8000, 16000, 44100, or 48000 Hz | `AUDIO_[IN\|OUT]_SAMPLE_RATE`
58+
Streaming device index | System index of the audio device | -1 for system default, 0, 1, ... | `AUDIO_[IN\|OUT]_DEVICE`
59+
Audio file name | Name of the WAV file to read from or write to | Path string, or empty to use device | `AUDIO_[IN\|OUT]_FILENAME`
60+
61+
When `AUDIO_IN_FILENAME` / `AUDIO_OUT_FILENAME` is set to a non-empty string, the driver streams audio data from or
62+
to the specified WAV file. When left empty, the streaming device index (`AUDIO_IN_DEVICE` / `AUDIO_OUT_DEVICE`)
63+
is used to select an audio hardware device, where `-1` selects the system default device.
64+
65+
<b>Video stream configuration</b>
66+
67+
Configuration options for the video vStream drivers are:
68+
69+
Parameter | Description | Available Options | Configuration Macro
70+
:-------------|:------------------------------------------------------------|:------------------------------------------|:--------------------------------
71+
Frame width | Video stream frame width in pixels | User-defined (e.g. 320, 640, 1280) | `VIDEO_[IN\|OUT]_FRAME_WIDTH`
72+
Frame height | Video stream frame height in pixels | User-defined (e.g. 240, 480, 720) | `VIDEO_[IN\|OUT]_FRAME_HEIGHT`
73+
Frame rate | Video stream frame rate in frames per second | User-defined (e.g. 15, 25, 30, 60) | `VIDEO_[IN\|OUT]_FRAME_RATE`
74+
Color format | Video frame pixel color space | Grayscale8, RGB888, BGR565, YUV420, NV12, NV21 | `VIDEO_[IN\|OUT]_FRAME_COLOR`
75+
Device index | System index of the video capture device (input only) | -1 for system default, 0, 1, ... | `VIDEO_[IN\|OUT]_DEVICE`
76+
File name | Name of the video or image file to read from or write to | Path string, or empty to use device | `VIDEO_[IN\|OUT]_FILENAME`
77+
78+
When `VIDEO_IN_FILENAME` is set to a non-empty string, the VideoIn driver reads frames from the specified video or
79+
image file. When left empty, the device index (`VIDEO_IN_DEVICE`) selects the camera to use, where `-1` selects
80+
the system default device.
81+
82+
For VideoOut, when `VIDEO_OUT_FILENAME` is set to a non-empty string, frames are written to the specified video file.
83+
When left empty, frames are displayed in an OpenCV window on the host.
84+
85+
### Audio streaming operation
86+
87+
The audio streaming flow for input and output follows a common pattern based on the CMSIS-Driver vStream API:
88+
89+
1. At FVP model start or reset, the VSI peripheral calls `init()` in the relevant Python scripts.
90+
For AudioIn this is `arm_vsi0.py`, for AudioOut `arm_vsi1.py`.
91+
Each script starts the Audio Server (`vsi_audio_server.py`) in the background and establishes a TCP connection to it.
92+
2. The application calls `Driver_vStreamAudioIn.Initialize()` or `Driver_vStreamAudioOut.Initialize()`, registering
93+
an optional callback function for stream events. The driver writes the audio configuration (channels, sample bits,
94+
sample rate, device or filename) to the VSI peripheral registers, which the Python script forwards to the Audio Server.
95+
3. The application calls `SetBuf()` to assign a circular DMA buffer divided into fixed-size blocks. The VSI DMA
96+
is configured to transfer one block at a time, synchronized by the VSI Timer.
97+
4. The application calls `Start()` with `VSTREAM_MODE_CONTINUOUS` or `VSTREAM_MODE_SINGLE` to begin streaming.
98+
The VSI Timer is programmed with an interval derived from the configured sample rate and block size to deliver
99+
one audio block per timer period.
100+
5. While streaming, the `VSTREAM_EVENT_DATA` callback event signals that a completed block is available.
101+
The application calls `GetBlock()` to obtain a pointer to the block and processes the data, then calls
102+
`ReleaseBlock()` to return the block to the driver for reuse.
103+
If the application does not release blocks fast enough, a `VSTREAM_EVENT_OVERFLOW` event is generated.
104+
6. When the end of the audio file is reached, the `VSTREAM_EVENT_EOS` event is generated.
105+
7. The application calls `Stop()` to halt streaming and `Uninitialize()` to release resources.
106+
107+
### Video streaming operation
108+
109+
The video streaming flow for input and output follows the same pattern as audio streaming, adapted for frame-based data:
110+
111+
1. At FVP model start or reset, the VSI peripheral calls `init()` in the relevant Python scripts.
112+
For VideoIn channel 0 this is `arm_vsi4.py`, for VideoOut channel 0 `arm_vsi5.py`.
113+
Each script starts the Video Server (`vsi_video_server.py`) in the background and establishes a TCP connection to it.
114+
2. The application calls `Driver_vStreamVideoIn.Initialize()` or `Driver_vStreamVideoOut.Initialize()`, registering
115+
an optional callback function for stream events. The driver writes the video configuration (frame width, height,
116+
frame rate, color format, device or filename) to the VSI peripheral registers, which the Python script forwards
117+
to the Video Server. The Video Server performs any required color space conversion and frame resizing.
118+
3. The application calls `SetBuf()` to assign a circular DMA buffer sized to hold one or more video frames.
119+
4. The application calls `Start()` with `VSTREAM_MODE_CONTINUOUS` or `VSTREAM_MODE_SINGLE` to begin streaming.
120+
The VSI Timer is programmed with an interval derived from the configured frame rate.
121+
5. While streaming, the `VSTREAM_EVENT_DATA` callback event signals that a complete frame is available.
122+
The application calls `GetBlock()` to obtain a pointer to the frame buffer and passes it to the processing
123+
algorithm, then calls `ReleaseBlock()` to return the buffer to the driver for reuse.
124+
If the application does not release frames fast enough, a `VSTREAM_EVENT_OVERFLOW` event is generated.
125+
6. When the end of the video file or image sequence is reached, the `VSTREAM_EVENT_EOS` event is generated.
126+
7. The application calls `Stop()` to halt streaming and `Uninitialize()` to release resources.
127+
128+
@{
129+
@}
130+
131+
132+
\defgroup vstream_py_implementation VSI Python for vStream
133+
\ingroup arm_vsi_vstream
134+
\brief VSI Python for vStream.
135+
\details
136+
137+
The VSI Python scripts bridge the C driver running on the FVP firmware to the Audio or Video Server running on the
138+
host. Each script corresponds to one VSI peripheral instance and is loaded by the FVP at model start. The scripts
139+
use a client-server architecture: the `arm_vsi*.py` scripts act as thin clients that relay register writes from the
140+
firmware to a background server process, which interfaces with the host audio or video hardware.
141+
142+
The following files implement the VSI Python layer for vStream:
143+
144+
Item | Description
145+
:-------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------
146+
[./interface/vstream/python/arm_vsi0.py](https://github.com/ARM-software/AVH/blob/main/interface/vstream/python/arm_vsi0.py) | VSI Python script for AudioIn (VSI instance 0). Bridges the FVP to the Audio Server.
147+
[./interface/vstream/python/arm_vsi1.py](https://github.com/ARM-software/AVH/blob/main/interface/vstream/python/arm_vsi1.py) | VSI Python script for AudioOut (VSI instance 1). Bridges the FVP to the Audio Server.
148+
[./interface/vstream/python/arm_vsi4.py](https://github.com/ARM-software/AVH/blob/main/interface/vstream/python/arm_vsi4.py) | VSI Python script for VideoIn channel 0 (VSI instance 4). Bridges the FVP to the Video Server.
149+
[./interface/vstream/python/arm_vsi5.py](https://github.com/ARM-software/AVH/blob/main/interface/vstream/python/arm_vsi5.py) | VSI Python script for VideoOut channel 0 (VSI instance 5). Bridges the FVP to the Video Server.
150+
[./interface/vstream/python/arm_vsi6.py](https://github.com/ARM-software/AVH/blob/main/interface/vstream/python/arm_vsi6.py) | VSI Python script for VideoIn channel 1 (VSI instance 6). Bridges the FVP to the Video Server.
151+
[./interface/vstream/python/arm_vsi7.py](https://github.com/ARM-software/AVH/blob/main/interface/vstream/python/arm_vsi7.py) | VSI Python script for VideoOut channel 1 (VSI instance 7). Bridges the FVP to the Video Server.
152+
[./interface/vstream/python/vsi_audio.py](https://github.com/ARM-software/AVH/blob/main/interface/vstream/python/vsi_audio.py) | VSI Audio client module. Handles communication between the VSI Python scripts and the Audio Server.
153+
[./interface/vstream/python/vsi_audio_server.py](https://github.com/ARM-software/AVH/blob/main/interface/vstream/python/vsi_audio_server.py) | Audio Server module. Implements a TCP server for audio streaming using PyAudio.
154+
[./interface/vstream/python/vsi_video.py](https://github.com/ARM-software/AVH/blob/main/interface/vstream/python/vsi_video.py) | VSI Video client module. Handles communication between the VSI Python scripts and the Video Server.
155+
[./interface/vstream/python/vsi_video_server.py](https://github.com/ARM-software/AVH/blob/main/interface/vstream/python/vsi_video_server.py) | Video Server module. Implements a TCP server for video streaming using OpenCV.
156+
157+
### Python requirements
158+
159+
In addition to the base Python environment, the VSI vStream use case requires the following Python packages:
160+
161+
- **Audio streaming**: PyAudio package for audio device access. Run the following command to install it with pip:
162+
\code{.sh}
163+
pip install pyaudio
164+
\endcode
165+
166+
- **Video streaming**: OpenCV Python package for camera and video file access. Run the following command to install it with pip:
167+
\code{.sh}
168+
pip install opencv-python
169+
\endcode
170+
171+
@{
172+
@}
173+
174+
*/
175+
// End vStream via VSI

0 commit comments

Comments
 (0)