By Jean-Julien Aucouturier, Boris Defreville, and François Pachet.
The Journal of the Acoustical Society of America 122(2), 881–891 (2007). DOI: 10.1121/1.2750160.
Read the paper (PDF) · Publisher version (DOI)
About the paper
A bag-of-frames model describes a recording by the distribution of spectral features from short audio frames, without retaining their order. The authors applied the same similarity measure to urban soundscapes and polyphonic music. It performed much better on the soundscapes they tested, prompting a closer look at how the two kinds of audio are structured.
Abstract
The "bag-of-frames" approach (BOF) to audio pattern recognition represents signals as the long-term statistical distribution of their local spectral features. This approach has proved nearly optimal for simulating the auditory perception of natural and human environments (or soundscapes), and is also the most predominent paradigm to extract high-level descriptions from music signals. However, recent studies show that, contrary to its application to soundscape signals, BOF only provides limited performance when applied to polyphonic music signals. This paper proposes to explicitly examine the difference between urban soundscapes and polyphonic music with respect to their modeling with the BOF approach. First, the application of the same measure of acoustic similarity on both soundscape and music data sets confirms that the BOF approach can model soundscapes to near-perfect precision, and exhibits none of the limitations observed in the music data set. Second, the modification of this measure by two custom homogeneity transforms reveals critical differences in the temporal and statistical structure of the typical frame distribution of each type of signal. Such differences may explain the uneven performance of BOF algorithms on soundscapes and music signals, and suggest that their human perception rely on cognitive processes of a different nature.
Related work
The Influence of Polyphony on the Dynamical Modelling of Musical Timbre · PDF (2007), by Jean-Julien Aucouturier and François Pachet, examines why the temporal models tested struggled with polyphonic music. Its experiments compare isolated notes, simultaneous notes, and longer textures.
Automatic Recognition of Urban Sound Sources · PDF (AES 2006) describes research on classifying urban sounds such as vehicles, birds, and voices using generated acoustic features.
Improving Timbre Similarity: How High’s the Sky? investigates limitations of a related approach to comparing songs by timbre.
Hubness in Music Similarity traces related work on why some songs appear unusually often as nearest neighbors in timbre-based music retrieval.
A Scale-Free Distribution of False Positives for a Large Class of Audio Similarity Measures (2008) examines how bag-of-frames models can repeatedly return the same songs as false matches in music retrieval.
The Bag-of-Frames Approach: A Not So Sufficient Model for Urban Soundscapes (2015) revisited the experiment with other soundscape datasets and found that the earlier result did not generalize to those datasets.