Our vision was to create a tool, Orpheus, that could be used as a stem player. Users can upload an audio file and separate it into vocals, drums, bass, piano, and other instruments (each of which is called a stem). We have made our tool a website to facilitate easier distribution of the tool online as we plan to publish the site in the near future. We use a HTML/JS/CSS front-end to build the user interface in addition to a Python Flask backend with a SQL database. We chose a Python backend in order to implement two APIs, Spleeter and Pitch Shifter (which are explained in the Separation and Pitch Shifter).
The website has three primary functions: registration/login, uploading, listing songs, playing song stems, separating songs, pitching-up songs, and deleting songs. They are described as follows.
Users can make accounts (stored in a SQL database which takes a username and a hashed password) in order to keep track of their songs. Thus, we are able to keep track of which users have which songs uploaded while also conserving storage. We created a separate registration (register.html) and login (login.html) page for users, and we implemented Flask sessions to keep track of user logins. Moreover, once a user is logged in, we set up the website to show “Log Off” rather than a login page so that the user can exit their session.
Users are able to upload audio files (made possible by the upload_file() function in app.py) to the website on the Upload page (upload.html), which submits the user’s selected audio file to the server through a POST request. The upload_file() function includes checks to ensure the user is uploading a file with the correct extension and that the file exists. Once the file has been verified, the upload_file function saves the audio file in the static/uploads folder and inserts the name of the song into the songs database along with the id of the user that uploaded it. Our code also checks to see if any other users have uploaded an audio file with the same name before, in order to conserve storage. From there, the user can access it under a different page, My Songs.
To access uploaded songs, we added a My Songs page (mypage.html) where users can navigate the songs they have uploaded. We employed Jinja to iterate through an alphabetically sorted list of the songs associated with the user’s id in the songs table of the database. We made this decision to use Jinja to make our implementation more concise and generalizable to a large quantity of songs. Here, users are able to see buttons for getting the stems (using Spleeter) and shifting the pitch (using the pitch shifter). We made the decision to separate the functionality into separate columns of an HTML table in order to simplify the implementation of using two separate external APIs, allowing us to require the users to first retrieve the normal stems before pitching them upwards. From there, users are able to select the buttons (which is an HTML form, chosen because we can pass the names of the songs to the spleeter() function in app.py) based on which behavior they want to access. From there, the user is redirected to the player page (hidden on the navbar, represented by player.html) where the right song stems can be played.
The player page is another HTML page that extends the layout and employs jinja to iterate through each of the generated stems. We added three separate buttons (play, pause, and refresh audio) call three JavaScript functions, respectively. First, we have the playAudio() function, which gets the elements of base, vocals, drums, other, and piano from the HTML (these are the names of all stems generated by Spleeter). We then run the JavaScript play() function on each of these in order to allow them to start simultaneously. We chose to do this because it enabled us to play the complete song without having to juggle with individual play/pause controls for each of the stems. Similarly, the next button called the stopAudio() method, which followed a similar implementation with the primary difference being that we called the pause() function rather than the start() function. Finally, we also have the refreshAudio() function that also grabs the individual stems from the HTML. Subsequently, it pauses these elements and then uses the currentTime property to reset the song. We chose this feature as it kept our code concise, and we chose to first pause the song as certain browsers have a slight latency when modifying the currentTime property.
On the front end, we chose to play each stem using HTML audio components (we set src to the path of each stem under the right folder using Jinja and python. We chose to use the default HTML audio components, as they include volume controls that we can use to mute/augment/diminish the particular stem while also keeping our program concise (we do not have to write the component from scratch). On the front end, we extended a layout while also including audio components that provided controls for each stem (play/pause, volume controls). The stems can then be muted using the volume controls built into HTML to let us modify the listening experience as intended.
Once their song is uploaded, users can then navigate to the My Songs page (as mentioned). There is a form with a button that can split a song into its stems. We elected to use the Spleeter API as it was the state-of-the-art model for stem separation; the tool was implemented in the spleeter() method in app.py. We run Spleeter using OS commands in order to take a local audio file and then split it up into the five separate stems. However, we first check that another user has not already spleeted the same song. If they have, we can bypass the Spleeter API and simply load the precalculated stems, saving both time for the user and memory for us (in that we don’t have to save the same stems twice).
Similarly to the splitting, users can navigate to the My Songs page and press a button to pitch shift a song. If the song already has stems in the stems folder, then the Pitch Shifter API will be able to shift the individual stems by one semitone, which can then be saved to a static/pitched folder. From there, we can pass the directories of the pitched-up stems to the player page (player.html) and redirect the user to that page. From there, as mentioned above in the Separation section, the user can play the song and mute/edit stems as they see fit. More technically, this uses nearly the same implementation as the Spleeter route, with the distinction being that we use the pitch shift API through command line arguments rather than the Spleeter API.
On the front end, we added Delete buttons to each song’s row under My Songs (mysongs.html). We use the DELETE functionality in SQL for each user (where the user ID matches that of the current session) in order to remove that song’s name from the songs database. However, we would only remove the song locally if no other user also had that song’s name on their account. The reason we decided to do this was to prevent deleting songs (which we store locally) from other user’s accounts as our implementation sought to minimize storage requirements as well as maximizing speed. To delete files locally, we simply use the rm -rf command line argument on the song’s respective directory in order to remove it from the local files. This allows us to remove the user’s access to the specific song they want to delete.