This monorepo contains the following packages:
@siteed/expo-audio-studio(formerly@siteed/expo-audio-stream)@siteed/expo-audio-ui@siteed/react-native-essentia@siteed/sherpa-onnx.rn(in development)
Give it a GitHub star 🌟, if you found this repo useful.
Love this repo? Support my journey in audio, AI, & blockchain, and help fund new tools! Sponsor me.
The monorepo includes several example applications in the /apps directory that demonstrate different use cases and integration patterns:
A fully-featured showcase app that demonstrates the capabilities of all the libraries. The audio playground app is available on app stores and serves as a comprehensive showcase of audio processing features, UI components, and integrations.
Key features:
- Real-world implementation of all monorepo libraries
- Audio recording, processing, and visualization
- Speech recognition and audio analysis examples
- Available on iOS and Android app stores
A focused demo showcasing the integration of the Sherpa-ONNX speech recognition and text-to-speech capabilities.
Try it at https://deeeed.github.io/expo-audio-stream/sherpa-onnx-demo/
A specialized app demonstrating the advanced audio analysis features of the Essentia library integration.
A stripped-down implementation showing the bare minimum required to integrate the audio libraries, perfect for developers who want to understand the core concepts without additional complexity.
@siteed/expo-audio-studio (formerly @siteed/expo-audio-stream) is a comprehensive library designed to facilitate real-time audio processing and streaming across iOS, Android, and web platforms.
For more details, please refer to the README of the package.
@siteed/expo-audio-ui provides UI components to visualize audio data processed by @siteed/expo-audio-studio.
For more details, please refer to the README of the package.
@siteed/react-native-essentia provides React Native bindings for the Essentia audio analysis library, enabling advanced audio feature extraction on mobile platforms.
Key features:
- Advanced audio analysis algorithms (MFCC, Key, Spectrum, etc.)
- Native implementation using C++ for high performance
- Cross-platform support for iOS and Android
For more details, please refer to the README of the package.
@siteed/sherpa-onnx.rn is a React Native wrapper for the sherpa-onnx library, providing speech-to-text (STT) and text-to-speech (TTS) capabilities. This package is currently under development.
Key features:
- On-device speech recognition using ONNX models
- Text-to-speech synthesis
- Cross-platform support for iOS, Android, and Web
- Optimized native implementation with web compatibility
For more details, check out the sherpa-onnx demo or refer to the package source.
- Automate Changelog generation on monorepo.
- Implement dual audio stream (one RAW for analysis, one compressed for playback).
- Intelligent call interruption handling
- Integrate with react-native-whisper for real-time on device transcriptions. (demo in playground)
- Migrate audio analysis to c++ native library (implemented in @siteed/react-native-essentia)
- Cross-platform audio device detection, selection, and fallback handling
- Add Zero-Latency Audio Recording with
prepareRecordingAPI - Setup Beta channel to avoid regressions
- Improve native code quality and configure sonarcloud to prevent CI errors.
- Integrate sherpa onnx models into expo-audio-studio and playground
- Implement example app for custom VAD.
- Audio preview waveform component from 'uri'. ( similar to https://github.com/SimformSolutionsPvtLtd/react-native-audio-waveform )
- e2e validation for cross platform features extraction (make sure we get exact same values on all platforms).
- Speaker Diarization example app.
This project is licensed under the MIT License - see the LICENSE file for details.
Created by Arthur Breton • See more projects at siteed.net