Speaker
Associate Professor Ning-Han Liu
What’s MIR
Music information retrieval (MIR) is
the interdisciplinary science of retrieving
information from music.
 MIR is a small but growing field of
research with many real-world
applications.

Applications of MIR
Recommender systems
 Track separation and instrument
recognition
 Automatic score creation
 Automatic categorization
 Music generation

Methods used in MIR-Data Source

Music Content
 Symbolic Data Formats
○ Scores
○ MIDI
 Digital Audio Formats
○ WAV
○ Mp3

Metadata




Song name
Singer
Performer
tags
Methods used in MIR-Feature
Representation

Mel-Frequency Cepstral Coefficient
(MFCC)
 is a measure of the timbre of a piece of
music.
Chords
 Harmonics
 Melody
 Main pitch
 Beats per minute

Methods used in MIR-Statistics and Machine
Learning
Music analysis and knowledge
representation
 Classification, clustering and modeling

Famous Application

iTunes: is a media player computer program, used for playing,
downloading, saving, and organizing digital music and video files on
desktop or laptop personal computers.
Famous Application

KKBOX
Famous Application

Nike+iPod
Previous research issue I

Query music efficiently
 Music representation
○ ‘U’,’D’,’S’
○ Rhythm
○ Chords
 Index structure
○ Link list
○ Suffix tree
 Query process
○ Exact matching
○ Approximate matching
Previous research issue I

String Matching
 Exact String Matching
○ E.g. sol-mi-mi-fa-re-re-do-re-mi-fa-sol-sol-sol
○ Query: do-re-mi
○ To find whether data contain the query melody
string
 Algorithms: KMP, Boyer-Moore
 Problem
○ Cannot expect users to precisely specify
music query
○ Need to retrieve all entire melody string
Previous research issue I

String Matching
 Approximate String Matching
○ Edit distance is often used as similarity
measure
○ E.g. sol-mi-mi-fa-re-re-do-re-mi-fa-sol-sol-sol
○ Query: do-mi matching do-re-mi with edit
distance=1
Previous research issue I

String Matching Algorithm Using Index
 Arbee L.P. Chen (ICMCS 1999)
Previous research issue I

Approximate Matching
Previous research issue I

Query by Music Segments
 Arbee L.P. Chen (ICME 2000)
Previous research issue I

Query by Music Segments
 Index: Suffix Tree
Previous research issue II

Music Recommendation
 Content based filtering (CBF)
○ predicts user preferred data items by matching the
representations of the data items relevant to the user
 Collaborative filtering (CF)
○ uses the correlations between users on the basis of
their ratings to predict items for users
 Mixed method
○ Combine CBF and CF
Previous research issue II

A music recommendation system based on music data
grouping and user interests
 Arbee (ICME 2000)
Previous research issue II
Novel research topic I

Playlist Generation
 An Intelligent Music Playlist Generator based on the
Time Parameter with Artificial Neural Networks (Liu,
2010)
Hip Hop
07:00
Rock&Roll
Blues
time
12:00
08:00
User A
Slow
tempo
07:00
07:30
Fast
tempo
09:00
Medium
tempo
Slow
tempo
12:30
time
User B
N.H. Liu, S.J. Hsieh, C.F. Tsai, "An Intelligent Music Playlist Generator based on the Time Parameter
with Artificial Neural Networks," Expert systems with applications. Volume 37, Issue 4, April (2010),
Pages 2815-2825 . (SCI, impact factor=2.9, EI)
Novel research topic I

The structure of the mixed artificial neural
networks
Input
Layer
Hidden
Layers
Music Item Mx
Feature 1
Feature 2
.
.
.
Feature n
Output
Layer
OutputANN1
Long term ANN
Time
parameter t
Fusion function
OutputANN2
Short term ANN
Output
Novel research topic II

JoMP: A Mobile Music Player Agent for Joggers
based on User Interest and Pace (Liu, 2009)
Smart Phone
Music
Player
Pace
Detection
Module
User
Feedback
Module
JoMP Server
Music Selection
Module
Pace
Prediction
Module
User Profile
Music Pool for Running
Music
Database
Music Filtering Module II
(Depends on user status)
Music Filtering Module I
(Depends on user interesting)
Music
Web
Server
Web Based Music Player
with Feedback Mechanism
User Client
N.H. Liu, H.Y. Kung, "JoMP: A Mobile Music Player Agent for Joggers based on User Interest and Pace,"
IEEE Transactions on Consumer Electronics, Volume 55, No. 4, Nov, (2009 (SCI, EI)
Novel research topic II

System prototype
Novel research topic III

Intelligent Music Player for Bike Sport Using
EEG & GPS Sensors
 Brainwave earphone (NeuroSky™ Minset) and wearing
method.
EEG Signal rhythm
α waves
β waves
γ waves
δ wave
θ waves
Frequency
8~13 Hz
14~30 Hz
> 21 Hz
0.5~3 Hz
4~7 Hz
N.H. Liu, H.M. Hsu, H.C. Chu, and S.H. Hsu "Intelligent Music Player for Bike Sport Using
Electroencephalogram and Global Positioning System Sensors" Sensor Letters, Volume 11, Number 5, pp.
772-780 (2013) (SCI).
Novel research topic III

The system is based on the Fuzzy Inference System (FIS),
combined with EEG and GPS sensors to measure the sport data of
cyclists.
EEG sensor
Tablet PC
with GPS
Novel research topic IV

Intelligent car audio system in cloud
N.H. Liu "Design of an Intelligent Car Radio and Music Player System," accepted by Multimedia Tools and
Applications. (2013).(SCI, IF=1.014) DOI: 10.1007/s11042-013-1467-z
Novel research topic V

Car audio system with doze prevention
Novel research topic V

Car audio system with doze prevention
N.H. Liu, C.Y. Chiang, H.M. Hsu, "Improving Driver Alertness through Music Selection Using a Mobile EEG
to Detect Brainwaves" Sensors, 13(7):8199-8221 (2013)(SCI, IF=1.953).
Novel research topic V

Car audio system with doze prevention
Novel research topic VI

Distance estimation methods
 Methods to calculate a personalized
distance measure between different pieces
of music based on user preferences.
 The questions ask the user to rate the
similarity between songs selected by the
system.
N.H. Liu "Comparison of Content-based Music Recommendation Using Different Distance
Estimation Methods," Applied Intelligence, Volume 38, issue 2, pp. 160-174, Mar. (2013).(SCI,
IF=1.853)
Novel research topic VI

Weighted squared Euclidean distance function
generated from maximum likelihood estimation
2
d
D (X , Q )   w n ( x n  q n )
n 1
P(M
p
 P (m
|W ,Q) 
mi M
W  arg max P ( M
i
|W ,Q)
ri
p
'
p
| W , Q)
W
{ P ( M
p
P ( M p | W , Q ) 
 P ( M p | W , Q ) P ( M p | W , Q )
| W , Q ), W }  
,
,...,

 w1
w 2
w d


r
i
w  wn 
'
n
m i M
 r (m
i
mi M
p
i ,n
 qn )
2
p
T
w
wn 
t 1
'
n ,t
'
T
, where w n , t is the re - weighted
value of n - th dimension
of W in round t
Novel research topic VI

Weighted squared Euclidean distance function
generated from genetic algorithm
W  arg min
'
W
bit0
bit1

Q i QS
bit15




Dif
(S
|
W
,
Q
)
x
i
S

 x  M qi

bit0
bit1
w1
bit15
w2
bit0
bit1
wd
Chromosome W
Define a fitness function to evaluate the chromosome performance
1
f (W ) 

Q i  QS


Dif(S
S

M
 x qi
x

| W ,Qi )  1


bit15
Novel research topic VI

Distance function generated by genetic programming
F  arg min
'

q i  QS
W




Dif
(
S
|
F
(
S
,
Q
)
x
x
i
S

 x  M qi

The fitness function is defined as follows:
1
f (W ) 

Q i  QS


Dif(S
S
 M qi
x

x

| F (S x , Qi )   1


Novel research topic VII

Music Selection Interface for Car Audio System
 In the architecture, users are able to select a genre of music
or a playlist from through a 2D interface. Self-organizing
map, depending on a personalized distance function and
music contents, is utilized to map music tracks to the
interface.
Novel research topic VII

Prototype in car
N.H.Liu, "Music Selection Interface for Car Audio System Using SOM with Personal Distance
Function," EURASIP Journal on Audio, Speech, and Music Processing, 2013:20. (2013) (SCI)
Novel research topic VIII

Query by Singing/Humming
 When a user cannot remember the title of a song, or
its related details, the most direct and convenient
method to search for the song is by humming a
section of it.
 The background of the user often influences the
genres of the songs being searched.
 We use the information from a user’s search
history, as well as the properties of genres
common to users with similar backgrounds, to
estimate the genre or style the current user may
be interested in based on a probability
calculation.
Novel research topic VIII

System flow chart
Humming
Query
Music
Database
Query
Processing
Users
Query
History
Music
Profile
Melody
Processing
Melody
Representation
Melody
Representation
Match
Processing
Rank List
Phase I
Re-ranking by
Hybrid
Recommendation
Method
New Rank List
User
Phase II
Novel research topic VIII

The Singing/Humming signals process
A2# (116.54Hz)
NS2
NS0
NS1
F2# (92.499Hz)
NS1 NS2
NS8
NS0
NS3 NS4 NS5
NS6 NS7
NS9
NS10 NS11 NS12
F2# (92.499Hz)
Twinkle, Twinkle, Little Star
Pitch String:
Contour String:
67 64 64
-3 0
65 62 62
+1
-3 0
60 62 64 65
-2
67 67 67
+2 +2 +1 +2
0 0
Novel research topic VIII

Ranking by User’s Preference
Novel research topic VII

Ranking through Similar Users’ Records
naïve Bayesian prediction
P (C i | A ) 
P ( A | C i ) P (C i )
P ( A)
6
P(A | Ci ) 
 P (a
m 1
m
| Ci )
Novel research topic VII

User Interface
N.H. Liu, "Effective Results Ranking for Mobile Query by Singing/Humming Using a Hybrid Recommendation
Mechanism," IEEE Transactions on Multimedia, Aug (2014)(SCI, IF=1.754)