SpeechSense

SpeechSense is an AI-powered mobile application that analyzes speech patterns to detect potential signs of intoxication or cognitive impairment. By having users read randomly generated tongue twisters aloud, the app captures audio, processes it through machine learning models, and provides feedback on cognitive function.

This project consists of a Flutter mobile app and a Python Flask backend server. The app is designed for educational and experimental purposes to explore speech-based cognitive assessment.

Features

• Speech Analysis: Records and analyzes audio while users read tongue twisters
• AI Detection: Uses machine learning to identify speech patterns indicative of impairment
• User Authentication: Firebase-based sign-in and registration
• History Tracking: Stores and displays past analysis results
• Cross-Platform: Built with Flutter for iOS and Android
• Privacy-Focused: Audio data is processed locally and optionally stored securely

How It Works

1. User Registration/Login: Users create an account via Firebase authentication
2. Script Generation: The app requests a random tongue twister from the server
3. Audio Recording: User reads the script aloud while the app records audio
4. Speech Recognition: Server verifies the spoken text matches the script (at least 50% similarity)
5. AI Analysis: Machine learning model analyzes audio features (MFCCs) to predict impairment status
6. Results Storage: Results are stored in Firebase Firestore with optional audio upload
7. Feedback Display: App shows analysis results and history

Prerequisites

• For Server (Raspberry Pi):

  • Python 3.7+
  • Internet connection for Firebase and speech recognition
  • Firebase project with Firestore and Storage enabled

• For Mobile App:

  • Flutter SDK (3.5.4+)
  • Android Studio or Xcode for building
  • Mobile device or emulator

Server Setup (Raspberry Pi)

The backend server handles script generation, speech recognition, AI prediction, and data storage.

1. Install Dependencies

cd server
pip install -r requirements.txt

2. Firebase Configuration

1. Create a Firebase project at https://console.firebase.google.com/
2. Enable Firestore Database and Storage
3. Generate a service account key (JSON file)
4. Place the JSON file in server/ directory as firebase_options.py

Example firebase_options.py:

import firebase_admin
from firebase_admin import credentials, firestore, storage

cred = credentials.Certificate('path/to/serviceAccountKey.json')
firebase_admin.initialize_app(cred, {
    'storageBucket': 'your-project-id.appspot.com'
})

db = firestore.client()
bucket = storage.bucket()

3. Train the ML Model (Optional)

If you need to retrain the model:

cd server
python main.py

This will train an SVM model using the audio data in data/ and save it as intoxication_model.pkl.

4. Run the Server

cd server
python server.py

The server will run on http://0.0.0.0:5000 (accessible from other devices on the network).

Mobile App Setup

1. Install Flutter

Follow the official Flutter installation guide: https://flutter.dev/docs/get-started/install

2. Get Dependencies

cd flutter_app
flutter pub get

3. Configure Firebase (Optional)

If using Firebase features:

1. Add Firebase to your Flutter project
2. Configure authentication and Firestore in the app

4. Build and Run

For Android:

flutter build apk
flutter install

For iOS:

flutter build ios
flutter install

For Development:

flutter run

5. Update Server URL

In the Flutter app code, update the server URL in API calls (e.g., in customHttpClient.dart) to point to your Raspberry Pi's IP address.

Usage

1. Start the Server: Run the Flask server on your Raspberry Pi
2. Install the App: Build and install the Flutter app on your mobile device
3. Register/Login: Create an account in the app
4. Perform Analysis: Tap "Perform Analysis" to get a script, read it aloud
5. View Results: Check your analysis history in the app

Tech Stack

• Frontend: Flutter (Dart)
• Backend: Python Flask
• AI/ML: Scikit-learn SVM, Librosa for audio processing
• Speech Recognition: Google Speech Recognition API
• Database: Firebase Firestore
• Storage: Firebase Cloud Storage
• Authentication: Firebase Auth

Project Structure

SpeechSense/
├── flutter_app/          # Flutter mobile application
│   ├── lib/             # Dart source code
│   ├── android/         # Android build files
│   ├── ios/             # iOS build files
│   └── pubspec.yaml     # Flutter dependencies
├── server/              # Python backend server
│   ├── main.py          # ML model training
│   ├── server.py        # Flask API server
│   ├── tongue_twister.py # Script generation and speech recognition
│   ├── firebase_options.py # Firebase configuration
│   ├── requirements.txt # Python dependencies
│   └── data/            # Training audio data
└── README.md

Disclaimer

This application is for educational and experimental purposes only. It does not provide medical diagnoses and should not be used as a substitute for professional medical advice. Results may not be 100% accurate.

Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.