Modelling musical similarity over time

through the variation principle

 Principal investigator: Dr. Anja Volk




in the media: e-data & research, March  2011, pdf                           


Project Description:


The aim of this project is to deliver a cognition-based computational model on music similarity that grounds in the variation principle employed in classical, folk and popular music. The project will integrate knowledge and methods from Music Information Retrieval, Musicology and Cognitive Science.

The assessment of similarity is fundamental for cognitive processes. Cognitive Science provides formal models for the theoretical framework and empirical measurement of similarity, demonstrating its crucial influence on our conception of the world. However, no comprehensive theory exists on how listeners use similarity to predict, categorize or appreciate music. This is a major problem in the rapidly growing discipline of Music Information Retrieval (MIR). MIR researches methods that allow users to retrieve music that is similar to musical queries representing their needs via the Internet. Developing content-based search methods for music faces the challenge of relating features of the music to listeners’ experience of similarity. Most methods developed in MIR extract low-level features from the music, severely limiting the cognitive plausibility of search results and hence impairing the usefulness of current retrieval systems.

This project will investigate the fundamental principle of variation in music studied in Musicology and Cognitive Science as a means to establish similarity. The project will research computational modelling of music similarity in the symbolic domain (i.e. using perception-related notation, such as MIDI or **kern) based on the variation of structural elements (such as motives, rhythms and chord sequences). Specifically, it will take into account the interaction between global and local features of the music and will address music as unfolding in time.

The project will deliver a model of music similarity that covers three major styles, namely classical, folk and popular music. The envisioned model will be based on cognitive and structural aspects of music, addressing high-level processes in establishing similarity. Hence, the project will make a major step towards cognition-based similarity models in music, which are urgently needed for the design of meaningful music retrieval systems. The development of a theoretic framework for similarity in music will contribute to the search for general principles of similarity across different domains envisioned in Cognitive Science.

Aims and Objectives

The overall aim of this project is the development of a computational model of music similarity in the symbolic domain (i.e. based on perception-related notation) that grounds in the variation principle employed in classical, folk and popular music. Specifically, there are four key objectives:

Objective 1:    To deliver a theoretic framework for similarity in music based on the variation principle across classical, folk and popular music. Test cases for different forms of similarity relations will be identified that serve as a ground truth for the computational model.

Objective 2:    To design segmentation methods that partition music into perceptual units in which variations of structural elements occur.

Objective 3:    To model local and global features of the music that are relevant for experiencing similarity based on variation.

Objective 4:    To model the integration of segments into their context by considering the unfolding of musical structure over time; this approach links local elements dynamically with their global contexts. A similarity model that is based on the interaction of local elements and global contexts will be developed.



Modelling similarity based on the variation principle is urgently needed to establish a general theory of similarity in music. This is crucial for the successful design of MIR systems and will open up new research lines in music cognition research. Integrating the concept of variation (Musicology), formal models of similarity (Cognitive Science) and computational modelling (Information Retrieval) into a comprehensive approach to music similarity will deliver important aspects for a general theory on similarity across different domains.


·         Music Information Retrieval: The largest part of modelling music similarity in Music Information Retrieval is currently in the audio domain. The extracted low-level features allow the comparison of broad categories of music, such as genres, but are less successful for establishing similarity for finer-grained data, such as for similar songs within the same genre. High-level features extracted in the symbolic domain from formats such as MIDI or **kern are closer to the way people perceive music. This project addresses the high-level processing in establishing similarity in music and hence will deliver an essential requirement for building cognitively plausible search algorithms in Music Information Retrieval. The lack of established concepts of similarity in music causes the lack of suitable ground truths on which algorithms can be tested (Downie et al, 2009). For the three selected music styles, test cases based on the variation principle will establish a ground truth for the computational modelling. The similarity model will deliver a theoretic framework that enables systematic strategies within the empirical cycle for quantitative approaches to similarity in MIR.


·         Musicology: The musicological discourse on the variation principle is based on small numbers of selected musical examples and has not led to a general concept of variation. The computational modelling allows to formalize the concept and to explicitly test it on a large collection of music.


·         Cognitive Science: While research in music cognition has strongly focussed on the experienced listener of Western classical music (Peretz, 2006), research on music similarity contributes an excellent topic on basic music skills. Similarity is used as a default method to reason about a domain, even if we do not have specific knowledge about it (Goldstone & Son, 2005). Thus, understanding music similarity will demonstrate that accessing music is not reserved to the highly trained specialist. Moreover, our model of music similarity will contribute an underrepresented domain to similarity research in Cognitive Science and hence contribute new aspects to the search for general principles of similarity across different domains.


Musicology provides analytic descriptions of music closely related to the phenomenon of similarity that are based on the variation principle. The term “variation principle” refers to the variation of musical patterns such that listeners are able to relate different components of a piece of music, or entire pieces, to each other and hence experience similarity relations. Although no formalized concept of the variation principle has been developed, studies on techniques to vary musical patterns such that they establish similarity in classical, folk and popular music strongly suggest that the variation principle is a universal principle in music. Recent studies in Cognitive Science show that similarity relations based on the variation of characteristic elements as described in Musicology are recognized both by experts and novices. This project will take into account different ways of interaction between global and local features of the music as a basis for modelling the concrete manifestations of the variation principle studied in Musicology and Cognitive Science. The data-rich approach of computational modelling within Music Information Retrieval allows to corroborate the generality of the variation principle in establishing similarity.

Overview over relevant workpackages (WP):

WP- 1: Domain modelling and test cases. 
Addressing objective 1, this workpackage will build domain models for the formalization of variation described for classical, folk and popular music in Musicology. Relevant musical features for experiencing music similarity based on variation will be identified. Appropriate test cases for similarity relations will be created and evaluated to serve as a ground truth for the computational modelling.

WP-2: Segmentation.
This workpackage aims to segment a musical piece into perceptual units addressing objective 2.  Stable segments will be determined that serve as local contexts of salient elements, such that the global structure of a musical piece is described as a patchwork of these local contexts.

WP-3: Relevant features.
Addressing objective 3, this workpackage designs local and global features that are relevant for the variation principle. Developing appropriate features for aspects such as melody, harmony and rhythm, and applying them to different segment sizes, determines change and persistence of features on different levels of the piece with respect to local and global information.

WP-4: Interaction between local and global components.
This workpackage will integrate segments into their global context and thus addresses objective 4. Integrating local segments dynamically into their global contexts will be realized through the unfolding of relevant musical features over time.Similarity measures based on the interaction of the local and global contexts will be derived. A computational model of music similarity will be designed that takes these multiple aspects into account for a hybrid approach to similarity in music.

WP-5: Theoretic model of similarity in music.
The evaluation of the computational modelling within WP-2 to WP-4 with respect to the domain modelling of similarity in WP-1 will lead to a theoretic model on similarity in music.

            Description: Description: Description: Description:

Research Team

The research team includes:

principal investigator: Dr. Anja Volk

Dr. Frans Wiering

Postdoc: Bas de Haas  

PhD student: Marcelo Rodriguez-Lopez

PhD student: Hendrik Vincent Koops

Contact for more detailed project description:

Anja Volk
Department of Information and Computing Sciences
Utrecht University
email and further contact details