MUSE: User-Sensitive musical Expression

Activity-Sensitive Generative Music

MUSE is a system designed to provide active, low-latency audio stimulation aimed at enhancing motivation, focus, and mood stabilization. By fusing real-time activity data with stimulatory audio generation, MUSE seeks to improve focus and task performance. The system leverages systematically finetuned mobile-data encoder linked to a pretrained audio decoder within a joint embedding (semantic latent-space) architecture.

Currently under commercial product development.

Product Overview

|

Repository Organization

This repository is structured as follows:

e6692-2025spring-FinalProject-MUSE-lm3963/
├── data/                   # Contains datasets used for training and evaluation
├── docs/                   # Documentation and project-related resources
│   └── service_archtr/     # Service architecture & runtime docs
├── dev/                    # Development: Model Training, Experiments and visualizations
├── models/                 # Pretrained models and model checkpoints
│   ├── decoder/            # Audio decoder finetunes (unused)
│   └── encoder/            # mobile-activity encoders
├── src/                    # Source code for the project
|   ├── edge_acq/           # Data Acquisition/Test Processing, Streaming on edge device (Jetson/wearable)
│   |   └── data_prcs/      # Scripts for data preprocessing and augmentation
|   ├── edge_client/        # Client streaming processes on local machine (Laptop/iphone)
│   ├── core_prcs/          # MusicGen architecture and generation scripts
│   └── utils/              # Utility functions and helper scripts
├── results/                # Generated results, logs, and analysis.
├── tests/                  # Unit tests for the project
├── requirements.txt        # High-level dependencies
├── LICENCE                 # PolyForm noncommercial license description & warnings
└── README.md               # Project overview and instructions

Key Links

• Project Report (5.13.2025)

• Project Presentation (5.13.2025)

• Project Proposal Document

• Facebook Research’s MusicGen Model Card

• MusicGen Fragment Streaming App (Sanchit Gandhi)

• SigLIP2 Zero-Shot Image Classification

• CoreML - Apple Developer Platform

• Spotify’s AI Playlist Generator

• State of Generative Music & Audio-Generative Architectures

• Prof. DiCarlo’s work on human visual-perceptive control

• MIT Media Lab’s Semantic Synth Programming

• NVIDIA Triton Temporal Fusion Transformer Architecture (OS)

• Mesgarani N.A.P. Lab work on BCI-Augmented Attentional Audio Control

How to Run (offline mode)

1. Clone the repository:

   git clone https://github.com/username/e6692-2025spring-FinalProject-MUSE-lm3963.git
   cd e6692-2025spring-FinalProject-MUSE-lm3963

2. Install dependencies:

   pip install -r requirements.txt

3. Run the main script:

   python src/main.py

4. For detailed instructions, refer to the docs/ directory.

Additional Notes

• This project is being developed for commercial use.
• For academic/research development, contact liam.mchugh@columbia.edu

Acknowledgements

Building on open-source work from Meta's MusicGen & Sanchit Gandhi's streamer application demo

https://huggingface.co/spaces/sanchit-gandhi/musicgen-streaming/blob/main/app.py#L52