The need for interdisciplinary input
in setting goals for Music Information
Retrieval
Don Knox
School of Engineering and Computing
Glasgow Caledonian University
Music Information Retrieval

Emerging discipline, themes/goals include:
◦ Audio content and score/symbolic music
analysis, music retrieval by query, music
similarity, rights management, music
classification, recommendation systems, music
visualisation, human interaction
◦ QBH, QBE, genre and style classification,
semantic classification
Music Information Retrieval

Methods
◦ Content analysis – acoustic and music
structure features, metadata, score/symbolic
analysis
◦ Statistical methods, signal analysis, machine
learning, pattern matching etc.
Symposium themes and MIR
•
Innovation
▫ MIR systems and approaches are innovative
▫ Tools allow e.g. Musicological evaluation of genre,
structure, novelty, similarity
•
Creativity
▫ Analysis and classification of music semantics –
attempts to evaluate creative intent (e.g.
emotion)
•
Labour
▫ Music recommendation now playing a central
role in economic exploitation of musical work
Informing MIR goals
•
‘Remarkable’ difference in the past between
MIR assumptions and user needs regarding
music query (Lee et al 2007)
▫ Example: QBH early focus of MIR
 Ignored user requirements, seeking behaviour – casual
users don’t want it
 Inappropriate format for musicological query
Informing MIR goals
•
Who are we innovating for?
▫ Intended audience, how will systems be used?
▫ Relevance of results – subjective views on what the
results mean to different users
▫ User interaction and query – useful?
•
Experiential issue – Downie (2003)
▫ ‘The single most important challenge facing MIR’
▫ ‘To ignore the experiential aspect of MIR is to diminish
the very core of MIR endeavour’
▫ Investigation of existing MIR practice should identify
distinct user communities and investigate what they
need from MIR systems (Futrelle and Downie 2002)
Users – who are they?

Casual, professional, researcher
◦ Researchers need MIR tools
◦ Industry – e.g. find music to match a specific need
– advertising, film, QBE
◦ Users need relevant applications to suit their
needs and music seeking behaviour

User type affects how MIR systems are used
◦ Music Psychology can tell us what type of user
someone is (Uitdenbogerd and Yap 03)
Users/Experiential issue

Different disciplines play a key role
around this issue
◦ Agreed from the beginning that this is key to
the future of MIR
◦ Still very little emphasis on subjectivity, users,
experience when setting goals for MIR
systems
Example: genre classification

Motivation:
◦ Industry use, user interaction with music,
researching musical similarity
Video clip. Marsayas genre meter:
http://www.youtube.com/watch?v=NDLhrc_WR5Q
Why pursue this?
Do we understand how users define and use
genre?
 Does this lead to meaningful results for
anyone?

◦ Techniques usually based on expert annotation
and then classification via feature extraction
 MIR studies have found only 70% human agreement on
genre (Tzanetakis and Cook 02)
 Bottom-up approaches are leveling out as regards
success rates
Genre classification

View of experts and ‘social’ view recognised as
being important
◦ Social, economic, cultural aspects affect genre identity
(Sordo et al 2008)
◦ McEnnis and Cunningham (2007) – social context is
important

View of artists
◦ Role of genre influence on artist– identification with
communities (Toynbee 2000)
◦ Very few MIR studies including artists participation

Recent call to evaluate these issues alongside
the statistical/analysis methods applied (Craft et
al 2007)
Musical similarity and recommendation

Similarity: Implicit in many MIR goals –
classification, music query, recommendation, rights
protection and tracking

Musicology influence:
◦ Need MIR tools for query of music databases
◦ Concerned with issues such as allusions to musical
themes, musical influence, evolution of tune families
◦ Can advise as to the nature of musical similarity
◦ Danger of engineers and computer scientists defining
‘the nature of music’ – possible source of
interdisciplinary tension (Futrelle and Downie 2002)
Recommendation systems

Huge industry interest (Genius, Amazon, Last.fm)
◦ Usually based on artist, genre, purchasing behaviour,
and ‘similarity’ (metadata)

Better similarity scores obtained by studying
user preference (Slaney and White 2007)
◦ i.e. It’s not just about musical similarity – it’s about like
and dislike
◦ Music psychology an invaluable source of information
in this regard
Recommendation systems

Few MIR studies on user - cultural, social,
contextual reasons for preference are ignored,
despite being of ‘real practical use’ to MIR
(Cunningham et al 2005)
◦ Current trend in MIR in this area is study of social
tagging and user music queries
 Helpful for e.g. automatic playlist generation – gap between
recommendation results and observed user behaviour
(Cunningham et al 2006)
Recommendation systems


Usually based on study of tags, queries.
How good is this data?
◦ One sociological study on motivation in social tagging
(McEnnis and Cunningham 2007)

Input from music psychology recognised as being
useful:
◦ Emotional and social connection as part of music query
(Downie and Cunningham 2002)
◦ Friends, family channel for music seeking rather than expert
opinion (Laplante and Downie 2006)

Despite this – there is little music psychology
influence as regards music seeking behaviour for e.g.
Activity, mood, well-being etc.
The semantic gap

Bottom up approach (structure, feature
extraction) alone is not sufficient
◦ But this is still a main focus for MIR efforts
◦ Top-down, user centred studies make results and goals
more relevant and accurate

Semantic/affective query, interaction identified as
being ‘a promising new area’ for MIR
◦ Content based MIR can benefit from study of user
subjectivity and background (Casey et al 2008)
The semantic gap

The need for multidisciplinary input
◦ How do we even begin to define these semantic
descriptions of music?
 Human factors essential to this task
 Effect of gender, age familiarity, musical training etc. on
semantic description (Hargreaves and North 99)
◦ Need for guidance re music structure, composition,
performance and semantic description – musical
communication
Emotion classification

A recent focus of some MIR systems – ‘affective’
interaction with music collections
◦ Driven by perception that listeners seek music
according to mood (Juslin and Sloboda 2001)
◦ Perhaps the most significant interdisciplinary overlap in
MIR – feature extraction, musicology, user behaviour,
composer intent, performer actions, cultural and social
factors, subjectivity
Emotion classification

Subjective and complex issue of emotional
effect of music
◦ MIR takes an essentially uninformed approach to
this important issue
 Issue is either fudged, or examined by social
tagging/collaborative filtering
 Debate still raging in music psychology about whether
this effect exists
◦ Music psychology input regarding user
experience is crucial if this is to be a valid goal
Emotion classification
Limited scope of MIR approaches thus far:
 Limited study of features

◦ Typically treated as a similarity/classification
problem based on feature extraction
 Expert annotation, subsequent comparison and
classification
Emotion classification

Continued adoption of bottom-up
approaches
◦ Only beginning to look at music structure and
emotion
◦ Although e.g. tension/expectation well understood
in musicology/psychology (Meyer 56)
Emotion classification

Limited participation by salient parties
◦ No existing MIR studies concerning composer
intent regarding emotion of the piece
 Musical communication issue - composition and
performance convey meaning
◦ No analysis of performance cues – despite this
being studied in music psychology (Juslin 2000)
Current MIR trends

Vast majority of ISMIR papers still about ‘how’

Still very few studies from user perspective – but
growing
◦ Recent social tagging studies, some analysis of music
queries and FOAF profiles (Lee et al 2007, Landone et
al 2007, Celma et al 2005)

Few calls for interdisciplinary input since Downie
and others around 2002/2003
◦ E.g. Crawford (2005) – MIR and the future of
musicology, Craft et al (2007)– issues surrounding
subjectivity of genre
Current MIR trends

Smiraglia (2006) study of all ISMIR publications
◦ Vast majority of papers still produced by computer
scientists
◦ Teams tend to come from same institution, and same
discipline
 No psychology input at all (up to 2006)

Do we cite across disciplines, but continue to work
within our own?
◦ E.g. McEnnis and Cunningham (2007) a review of sociology
literature by computer scientists
Recent MIR developments

Encouraging signs:
◦ Recent series of workshops on ‘semantics of audio’
◦ Increasing MIR contribution in cross-disciplinary
forums – e.g. ICMPC
◦ ‘Limited but increasing’ number of user studies
(Lee et al 2007)
 User communities and organisation of artists into genres
(Baccigalupo et al 2008)
 Jacobson et al (2008), Facebook links between artists –
alignment by genre
 User motivation behind social tagging (McEnnis and
Cunningham 07)
Conclusions
MIR efforts and goals must be informed by the
needs of the user
 Musicological/music theory contribution

◦ Tools for researchers – e.g. query format etc.
◦ Music structure
 The nature of musical similarity
 Structural effects upon listener, musical communication

Musicians, composers, performers
◦ Participation in studies concerning musical
communication
Conclusions

Music psychology contribution
◦ Semantic description of query terms
◦ User needs and music seeking behaviour
◦ Cultural and social issues behind e.g. genre,
music preference
◦ Music and emotion – experience, subjectivity
◦ Meaningful understanding of similarity and
recommendation – social, contextual
References

Baccigalupo, C., Donaldson, J. and E. Plaza. 2008. Uncovering Affinity of Artists to Multiple
Genres From Social Behaviour Data. Proceedings of the 9th ISMIR, 2008.

Casey, M.A., Veltkamp, R., Goto, M., Leman, M., Rhodes, C. and M. Slaney. 2008. Content-Based
Music Information Retrieval: Current Directions and Future Challenges. Proceedings of the IEEE,
Vol. 96, No. 4, April 2008.

Celma, A., RamÃrez, M. and P. Herrera-Boyer. 2005. Foafing the Music: A Music Recommendation
System based on RSS Feeds and User Preferences. Proceedings of the 6th ISMIR, 2005.

Craft, A., Wiggins, G. A., and T. Crawford. 2007. How many beans make five? the consensus
problem in music-genre classification and a new evaluation method for single-genre
categorisation systems. Proceedings of the 8th ISMIR, 2007.

Crawford, T. 2005. Music Information Retrieval and the future of Musicology. Online Chopin
Variorum Edition technical report. Available from http://www.ocve.org.uk [Accessed June 2009]

Cunningham, S.J., Downie, J.S. and D. Bainbridge. 2005. "The Pain ; the Pain": Modelling Music
Information Behavior and the Songs We Hate. Proceedings of the 6th ISMIR, 2005.

Cunningham, S.J., D. Bainbridge and A. Falconer. 2006. More of an Art than a Science: Supporting
the Creation of Playlists and Mixes. Proceedings of the 7th ISMIR, 2006.

Downie, J.S. and S.J. Cunningham. 2002. Toward a Theory of Music Information Retrieval Queries:
System Design Implications. Proceedings of the 3rd ISMIR, 2002. pp. 299-300.

Downie, J.S. 2003. Music Information Retrieval. Annual Review of Information Science and
Technology. 37 pp. 295-340. 2003.

Futrelle, J. and J.S. Downie. 2002. Interdisciplinary Communities and Research Issues in Music
Information Retrieval. Proceedings of the 3rd ISMIR, 2002.

Hargreaves, D.J. and A. North. 1999. The functions of music in everyday life: Redefining the social
in music psychology. Psychol. Music, vol. 27, no. 1, pp. 71–83. 1999.

Jacobson, K., Fields, B. and M. Sandler. 2008. Using Audio Analysis and Network Structure to
Identify Communities in On-Line Social Networks of Artists. Proceedings of the 9th ISMIR, 2008.

Juslin, P.N. 200. Cue Utilization in Communication of Emotion in Music Performance: Relating
Performance to Perception. Journal of Experimental Psychology: Human Perception and
Performance 2000,Vol. 26, No. 6, 1797-1813.

Juslin, P.N. and J. A. Sloboda, Music and Emotion:Theory and Research. Oxford University Press,
2001.

Landone, C., Harrop, J. and J.D. Reiss. 2007. Enabling Access to Sound Archives Through
Integration, Enrichment and Retrieval: The EASAIER Project. Proceedings of the 8th ISMIR, 2007.

Laplante, A. and J.S. Downie. 2006. Everyday Life Music Information-Seeking Behaviour of Young
Adults. Proceedings of the 7th ISMIR, 2006.

Lee, J.H., Downie, J.S. and M. Cameron Jones. 2007. Preliminary analyses of information features
provided by users for identifying music. Proceedings of the 8th ISMIR, 2007.

McEnnis, D. and S.J. Cunningham. 2007. Sociology and Music Recommendation Systems.
Proceedings of the 8th ISMIR, 2007.

Meyer, L.B. Emotion and meaning in music. The University of Chicago Press, 1956.

Selfridge-Field, E. 2006. Social Cognition and Melodic Persistence: Where Metadata and Content
Diverge. Proceedings of the 7th ISMIR, 2006.

Slaney, M. and W. White. 2007. Similarity based on rating data. Proceedings of the 8 th ISMIR,
2007.

Smiraglia, R.P. 2006. Music Information Retrieval: An Example of Bates' Substrate? In Information
Science Revisited: Approaches to Innovation: Proceedings of the Canadian Association for
Information Science annual conference. June 1st‐3rd, 2006.

Sordo, M., Celma, O., Blech, M. and E. Guaus. 2008.The Quest for Musical Genres: Do the
Experts and the Wisdom of Crowds Agree? Proceedings of the 9th ISMIR, 2008. pp. 255-260.

Tzanetakis, G. and P. Cook. 2002. Musical genre classification of audio signals. IEEE Transactions
on Speech and Audio Processing, 10(5):293–302, 2002.

Uitdenbogerd, A.L. and Yap,Y.W. 2003.Was Parsons right? An experiment in usability of music
representations for melody-based music retrieval. Proceedings of the 4th ISMIR, 2003, pp.75-79.