Mono to Stereo: How AI Is Breathing New Life into Music | by Max Hilsdorf | Dec, 2024

Now that we mentioned how related mono-to-stereo know-how is, you is perhaps questioning the way it works underneath the hood. Turns on the market are completely different approaches to tackling this downside with AI. Within the following, I need to showcase 4 completely different strategies, ranging from conventional sign processing to generative AI. It doesn’t function a whole record of strategies, however relatively as an inspiration for a way this process has been solved during the last 20 years.

Conventional Sign Processing: Sound Supply Formation

Earlier than machine studying turned as in style as it’s at the moment, the sector of Music Data Retrieval (MIR) was dominated by sensible, hand-crafted algorithms. It’s no surprise that such approaches additionally exist for mono-to-stereo upmixing.

The basic thought behind a paper from 2007 (Lagrange, Martins, Tzanetakis, [1]) is easy:

If we are able to discover the completely different sound sources of a recording and extract them from the sign, we are able to combine them again collectively for a practical stereo expertise.

This sounds easy, however how can we inform what the sound sources within the sign are? How can we outline them so clearly that an algorithm can extract them from the sign? These questions are tough to resolve and the paper makes use of quite a lot of superior strategies to realize this. In essence, that is the algorithm they got here up with:

1. Break the recording into brief snippets and determine the height frequencies (dominant notes) in every snippet
2. Determine which peaks belong collectively (a sound supply) utilizing a clustering algorithm
3. Determine the place every sound supply must be positioned within the stereo combine (guide step)
4. For every sound supply, extract its assigned frequencies from the sign
5. Combine all extracted sources collectively to kind the ultimate stereo combine.

Though fairly complicated within the particulars, the instinct is kind of clear: Discover sources, extract them, combine them again collectively.

A Fast Workaround: Supply Separation / Stem Splitting

Rather a lot has occurred since Lagrange’s 2007 paper. Since Deezer launched their stem splitting instrument Spleeter in 2019, AI-based supply separation techniques have turn into remarkably helpful. Main gamers equivalent to Lalal.ai or Audioshake make a fast workaround doable:

1. Separate a mono recording into its particular person instrument stems utilizing a free or industrial stem splitter
2. Load the stems right into a Digital Audio Workstation (DAW) and blend them collectively to your liking

This method has been utilized in a analysis paper in 2011 (see [2]), nevertheless it has turn into way more viable since because of the latest enhancements in stem separation instruments.

The draw back of supply separation approaches is that they produce noticeable sound artifacts, as a result of supply separation itself continues to be not with out flaws. Moreover, these approaches nonetheless require guide mixing by people, making them solely semi-automatic.

To totally automate mono-to-stereo upmixing, machine studying is required. By studying from actual stereo mixes, ML system can adapt the blending model of actual human producers.

Machine Studying with Parametric Stereo

One very inventive and environment friendly approach of utilizing machine studying for mono-to-stereo upmixing was introduced at ISMIR 2023 by Serrà and colleagues [3]. This work is predicated on a music compression approach known as parametric stereo. Stereo mixes encompass two audio channels, making it arduous to combine in low-bandwidth settings equivalent to music streaming, radio broadcasting, or phone connections.

Parametric stereo is a method to create stereo sound from a single mono sign by specializing in the essential spatial cues our mind makes use of to find out the place sounds are coming from. These cues are:

1. How loud a sound is within the left ear vs. the precise ear (Interchannel Depth Distinction, IID)
2. How in sync it’s between left and proper by way of time or part (Interchannel Time or Part Distinction)
3. How related or completely different the alerts are in every ear (Interchannel Correlation, IC)

Utilizing these parameters, a stereo-like expertise could be created from nothing greater than a mono sign.

That is the strategy the researchers took to develop their mono-to-stereo upmixing mannequin:

1. Gather a big dataset of stereo music tracks
2. Convert the stereo tracks to parametric stereo (mono + spatial parameters)
3. Prepare a neural community to foretell the spatial parameters given a mono recording
4. To show a brand new mono sign into stereo, use the educated mannequin to infer spatial parameters from the mono sign and mix the 2 to a parametric stereo expertise

At present, no code or listening demos appear to be accessible for this paper. The authors themselves confess that “there may be nonetheless a spot between skilled stereo mixes and the proposed approaches” (p. 6). Nonetheless, the paper outlines a inventive and environment friendly approach to accomplish absolutely automated mono-to-stereo upmixing utilizing machine studying.

Generative AI: Transformer-based Synthesis

Now, we are going to get to the seemingly most straight-forward approach to generate stereo from mono. Coaching a generative mannequin to take a mono enter and synthesizing each stereo output channels instantly. Though conceptually easy, that is by far probably the most difficult strategy from a technical standpoint. One second of high-resolution audio has 44.1k knowledge factors. Producing a three-minute tune with stereo channels subsequently means producing over 15 million knowledge factors.

With todays applied sciences equivalent to convolutional neural networks, transformers, and neural audio codecs, the complexity of the duty is beginning to turn into managable. There are some papers who selected to generate stereo sign via direct neural synthesis (see [4], [5], [6]). Nonetheless, solely [5] practice a mannequin than can resolve mono to stereo technology out of the field. My instinct is that there’s room for a paper that builds a devoted for the “easy” process of mono-to-stereo technology and focuses 100% on fixing this goal. Anybody right here on the lookout for a PhD subject?