Monday, April 19, 2021
11:00 AM
https://rochester.zoom.us/j/97062984398
MS Thesis Defense
Gavin Baker
University of Rochester
Audio Up-Channeling with Deep Neural Networks in the Spectral and Waveform Domains
In the early days of recorded audio, all music was recorded and played back in a one-channel format. It took decades before the technology required for stereo sound was developed, and longer still for it to be widely available. The acceptance of stereo equipment spurred recording engineers to embrace the added expressiveness opened up by the newly widened stereo field. However, a significant set of music is unable to be rendered in stereo, as it was mixed on an analog system in mono, and the original session tracks are unavailable.

Recent developments in computer vision and audition techniques motivate a push towards intelligently converting these exclusively mono recordings to multichannel formats, although there are challenges inherent to producing results with equivalent quality to the work produced by an experienced mixing engineer. This thesis proposes several deep learning-based models for this 'up-channeling' process, spanning both direct waveform generation as well as spectral representation generation.

This research uses a U-Net architecture operating in both the magnitude spectral-domain and the time-domain, as well as a Generative Adversarial Network (GAN) that utilizes these U-Nets. The networks are trained on a dataset containing string quartet music, as classical music has standardized stereo recording techniques that produce very distinct sounding individual channels. While the spectral-domain models produce more distinct channel separations, the time-domain models produce audio with less noise.

Although this study focuses on converting from mono to stereo, these models would be feasible for up-channeling to more channels, such as stereo to 5.1 surround sound. However, training these models requires a large dataset of music in the desired output channel count, which does not exist for many musical genres.