By Jean-Julien Aucouturier and François Pachet.
Journal of Negative Results in Speech and Audio Sciences 1(1), 1–13 (2004).
About the paper
The authors systematically tested variants of a method for comparing songs by their overall sound, or timbre. Many changes made little difference. In their experiments, performance appeared to level off around 65% R-precision for the family of methods they tested.
Abstract
We report on experiments done in an attempt to improve the performance of a music similarity measure which we introduced in [2]. The technique aims at comparing music titles on the basis of their global “timbre”, which has many applications in the field of Music Information Retrieval. Such measures of timbre similarity have seen a growing interest lately, and every contribution (including ours) is yet another instantiation of the same basic pattern recognition architecture, only with different algorithm variants and parameters. Most give encouraging results with a little effort, and imply that near-perfect results would just extrapolate by fine-tuning the algorithms’ parameters. However, such systematic testing over large, inter-dependent parameter spaces is both difficult and costly, as it requires to work on a whole general meta-database architecture. This paper contributes in two ways to the current state of the art. We report on extensive tests over very many parameters and algorithmic variants, either already envisioned in the literature or not. This leads to an improvement over existing algorithms of about 15% R-precision. But most importantly, we describe many variants that surprisingly do not lead to any substantial improvement. Moreover, our simulations suggest the existence of a “glass ceiling” at R-precision about 65% which cannot probably be overcome by pursuing such variations on the same theme.
Related work
Hubness in Music Similarity puts this paper’s observations about hub songs in context and links to later studies that cite the work.
In the follow-up study A Scale-Free Distribution of False Positives for a Large Class of Audio Similarity Measures (2008), Aucouturier and Pachet measured how often songs appeared as nearest neighbors and examined how the distribution varied across audio similarity models.
The earlier paper Music Similarity Measures: What’s the Use? (ISMIR 2002) introduces the timbre similarity measure evaluated here. That paper received a 2026 ISMIR Test-of-Time Honorable Mention.