Some projects in Real-World Sound Analysis Dan Ellis Laboratory for Recognition and Organization of Speech and Audio Dept. Electrical Eng., Columbia Univ., NY USA dpwe@ee.columbia.edu 1. 2. 3. 4. http://labrosa.ee.columbia.edu/ Real-World Sound Speech Separation Soundtrack Classification Music Audio Analysis Real-World Sound Analysis - Dan Ellis 2009-12-10 - 1 /30 LabROSA Overview Information Extraction Music Recognition Environment Separation Machine Learning Retrieval Signal Processing Speech Real-World Sound Analysis - Dan Ellis 2009-12-10 - 2 /30 1. Real-World Sound “Imagine two narrow channels dug up from the edge of a lake, with handkerchiefs stretched across each one. Looking only at the motion of the handkerchiefs, you are to answer questions such as: How many boats are there on the lake and where are they?” (after Bregmanʼ90) • Received waveform is a mixture 2 sensors, N sources - underconstrained • Use prior knowledge (models) to constrain Real-World Sound Analysis - Dan Ellis 2009-12-10 - 3 /30 Approaches to Separation ICA CASA • Multi-channel • Single-channel • Fixed filtering • Time-var. filter • Perfect separation • Approximate – maybe! separation target x interference n - mix proc x^ + n^ Real-World Sound Analysis - Dan Ellis mix x+n s stft proc mask istft x^ Model-based • Any domain • Param. search • Synthetic output? mix x+n params x^ param. fit synthesis source models 2009-12-10 - 4 /30 2. Speech Separation Roweis ’01, ’03 Kristjannson ’04, ’06 • Given models for sources, find “best” (most likely) states for spectra: combination p(x|i1, i2) = N (x; ci1 + ci2, Σ) model {i1(t), i2(t)} = argmaxi1,i2 p(x(t)|i1, i2) inference of source state can include sequential constraints... • E.g. stationary noise: In speech-shaped noise (mel magsnr = 2.41 dB) freq / mel bin Original speech 80 80 80 60 60 60 40 40 40 20 20 20 0 1 Real-World Sound Analysis - Dan Ellis 2 time / s 0 1 2 VQ inferred states (mel magsnr = 3.6 dB) 0 1 2 2009-12-10 - 5 /30 Speech Mixture Recognition Varga & Moore ’90 Kristjansson, Hershey et al. ’06 • Speech recognizers contain speech models ASR is just argmax P(W | X) • Recognize mixtures with Factorial HMM i.e. two state sequences, one model for each voice exploit sequence constraints, speaker differences model 2 model 1 • Speaker-specific models... Real-World Sound Analysis - Dan Ellis observations / time 2009-12-10 - 6 /30 Eigenvoices • Idea: Find Kuhn et al. ’98, ’00 Weiss & Ellis ’07, ’08, ’09 speaker model parameter space generalize without losing detail? Speaker models Speaker subspace bases • Eigenvoice model: µ = µ̄ + U adapted model mean voice Real-World Sound Analysis - Dan Ellis eigenvoice bases w + B weights h channel channel bases weights 2009-12-10 - 7 /30 Eigenvoice Bases additional components for channel Frequency (kHz) 10 20 6 30 4 40 2 b d g p t k jh ch s z f th v dh m n l 8 Frequency (kHz) • Eigencomponents shift formants/ coloration Mean Voice 50 r w y iy ih eh ey ae aa aw ay ah ao owuw ax Eigenvoice dimension 1 8 6 6 4 4 2 2 0 b d g p t k jh ch s z f th v dh m n l 8 Frequency (kHz) 280 states x 320 bins = 89,600 dimensions r w y iy ih eh ey ae aa aw ay ah ao owuw ax Eigenvoice dimension 2 8 6 6 4 4 2 2 0 b d g p t k jh ch s z f th v dh m n l 8 Frequency (kHz) • Mean model 8 r w y iy ih eh ey ae aa aw ay ah ao owuw ax Eigenvoice dimension 3 8 6 6 4 4 2 2 0 b d g p t k jh ch s z f th v dh m n l Real-World Sound Analysis - Dan Ellis r w y iy ih eh ey ae aa aw ay ah ao owuw ax 2009-12-10 - 8 /30 Speaker-Adapted Separation Weiss & Ellis ’08 • Factorial HMM analysis with tuning of source model parameters = eigenvoice speaker adaptation Real-World Sound Analysis - Dan Ellis 2009-12-10 - 9 /30 Speaker-Adapted Separation Real-World Sound Analysis - Dan Ellis 2009-12-10 - 10/30 Speaker-Adapted Separation • Eigenvoices for Speech Separation task speaker adapted (SA) performs midway between speaker-dependent (SD) & speaker-indep (SI) SI SA SD Real-World Sound Analysis - Dan Ellis 2009-12-10 - 11/30 Spatial Separation Mandel & Ellis ’07 • 2 or 3 sources in reverberation assume just 2 ‘ears’ • Model interaural spectrum of each source as stationary level and time differences: Real-World Sound Analysis - Dan Ellis 2009-12-10 - 12/30 ILD and IPD • Sources at 0° and 75° in reverb R i gh t IPD I LD Mixture Source 2 Source 1 L eft Real-World Sound Analysis - Dan Ellis 2009-12-10 - 13/30 Model-Based EM Source Separation and Localization (MESSL) Re-estimate source parameters Mandel & Ellis ’09 Assign spectrogram points to sources can model more sources than sensors flexible initialization Real-World Sound Analysis - Dan Ellis 2009-12-10 - 14/30 MESSL Results • Modeling uncertainty improves results tradeoff between constraints & noisiness 2.45 dB 0.22 dB 12.35 dB 8.77 dB 9.12 dB Real-World Sound Analysis - Dan Ellis -2.72 dB 2009-12-10 - 15/30 Parameter estimation and source separation MESSL-SP (Source Prior) Ron Weiss Underdetermined Source Separation Using Speaker Subspace Models Real-World Sound Analysis - Dan Ellis May 4, 2009 27 / 34 2009-12-10 - 16/30 MESSL-SP Results • Source models function as priors • Interaural parameter spatial separation Frequency (kHz) Frequency (kHz) source model prior improves spatial estimate Ground truth (12.04 dB) 8 DUET (3.84 dB) 8 8 6 6 6 4 4 4 2 2 2 0 0 0.5 1 1.5 MESSL (5.66 dB) 8 0 0 0.5 1 1.5 MESSL SP (10.01 dB) 8 0 6 6 4 4 4 2 2 2 0 0.5 1 Time (sec) 1.5 Real-World Sound Analysis - Dan Ellis 0 0 0.5 1 Time (sec) 1.5 1 0.8 0 0 0.5 1 1.5 MESSL EV (10.37 dB) 8 6 0 2D FD BSS (5.41 dB) 0.6 0.4 0.2 0 0 0.5 1 Time (sec) 1.5 2009-12-10 - 17/30 3. Soundtrack Classification • Short video clips as the evolution of snapshots 10-100 sec, one location, no editing browsing? • Need information for indexing... video + audio foreground + background Real-World Sound Analysis - Dan Ellis 2009-12-10 - 18/30 MFCC Covariance Representation • Each clip/segment → fixed-size statistics similar to speaker ID and music genre classification • Full Covariance matrix of MFCCs 8 7 6 5 4 3 2 1 0 VTS_04_0001 - Spectrogram MFCC Covariance Matrix 30 20 10 0 -10 MFCC covariance -20 1 2 3 4 5 6 7 8 9 time / sec 20 18 16 14 12 10 8 6 4 2 1 2 3 4 5 6 7 8 9 time / sec 50 20 level / dB 18 16 20 15 10 5 0 -5 -10 -15 -20 value MFCC dimension MFCC features MFCC bin Video Soundtrack freq / kHz maps the kinds of spectral shapes present 14 12 0 10 8 6 4 2 5 10 15 MFCC dimension • Clip-to-clip distances for SVM classifier 20 -50 by KL or 2nd Gaussian model Real-World Sound Analysis - Dan Ellis 2009-12-10 - 19/30 Latent Semantic Analysis (LSA) • Probabilistic LSA (pLSA) models each Hofmann ’99 histogram as a mixture of several ‘topics’ .. each clip may have several things going on • Topic sets optimized through EM p(ftr | clip) = ∑topics p(ftr | topic) p(topic | clip) = GMM histogram ftrs * “Topic” p(topic | clip) p(ftr | clip) “Topic” AV Clip AV Clip GMM histogram ftrs p(ftr | topic) use p(topic | clip) as per-clip features Real-World Sound Analysis - Dan Ellis 2009-12-10 - 20/30 AP Classification Results • Wide range: 0.8 Chang, Ellis et al. ’07 Lee & Ellis ’10 1G + KL 1G + Mah GMM Hist. + pLSA Guess 0.7 0.6 0.5 0.4 0.3 0.2 0.1 Da Ski nc in g Bo at Ba by Be a W ch ed di M ng us eu m Pa Pl ra ay de gr ou nd Su ns et Pi cn Bi ic rth da y Gr On e pe r ou son p of 3 + M us ic Ch e Gr ou er p of 2 Sp or ts Pa r Cr k ow d Ni gh An t im Si al ng in g Sh ow 0 Concepts audio (music, ski) vs. non-audio (group, night) large AP uncertainty on infrequent classes Real-World Sound Analysis - Dan Ellis 2009-12-10 - 21/30 Temporal Refinement • Global vs. local class models tell-tale acoustics may be ‘washed out’ in statistics try iterative realignment of HMMs: YT baby 002: voice baby laugh 4 New Way: Limited temporal extents freq / kHz freq / kHz Old Way: All frames contribute 3 4 2 1 1 5 10 voice 15 0 time / s baby 3 2 0 voice bg 5 voice baby 10 laugh 15 bg time / s laugh baby laugh “background” (bg) model shared by all clips Multiple-Instance Learning Real-World Sound Analysis - Dan Ellis 2009-12-10 - 22/30 Landmark-based Features • Describe audio with biggest Gabor elements extract with Matching Pursuit (MP) neighbor pairs as “features”? Real-World Sound Analysis - Dan Ellis 2009-12-10 - 23/30 Landmark models for similar events • Build index of Gabor neighbor pairs Cotton & Ellis ’09 recognize repeated events with similar pairs Real-World Sound Analysis - Dan Ellis 2009-12-10 - 24/30 Matching Videos via Fingerprints are a noise-robust fingerprint freq / kHz • Landmark pairs Cotton & Ellis ’10? VIdeo IMpLQaiHWbE at 195s 4 3 2 • Use to match 0 freq / kHz distinct videos with same sound ambience 1 195.5 196 196.5 197 197.5 198 198.5 199 time / sec VIdeo Yi1hkNkqHBc at 218 s 4 3 2 1 0 Real-World Sound Analysis - Dan Ellis 218.5 219 219.5 220 220.5 221 221.5 222 time / sec 2009-12-10 - 25/30 4. Music Audio Analysis freq / kHz Let it Be (final verse) Signal 4 20 0 2 -20 0 • Interested in all levels from notes to genres 162 Melody C5 C4 C3 C2 Piano C5 C4 C3 C2 164 166 168 170 172 level / dB 174 time / s Onsets & Beats G Per-frame chroma E D C 1 0.75 0.5 0.25 0 intensity A Per-beat normalized chroma G E D C A 390 Real-World Sound Analysis - Dan Ellis 395 400 405 410 415 time / beats 2009-12-10 - 26/30 Polyphonic Transcription • Apply the Eigenvoice idea to music eigeninstruments? Real-World Sound Analysis - Dan Ellis Grindlay & Ellis ’09 • Subspace NMF 2009-12-10 - 27/30 Chord Recognition Structural Support Vector Machine Joint features Learn weights describe match between x and y so that is max for correct y Tsochantaridis et al ’05 Weller, Ellis, Jebara ’09 … • Much better results Real-World Sound Analysis - Dan Ellis 2009-12-10 - 28/30 Melodic-Harmonic Mining • 100,000 tracks from morecowbell.dj as Echo Nest Analyze • Frequent clusters of 12 x 8 binarized eventchroma Beat tracking Music audio Chroma features Key normalization Locality Sensitive Hash Table Landmark identification #1 (3491) #2 (2775)) #3 (2255) #4 (1241)) #5 (1224)) #6 (1218)) #7 (1092)) #8 (1084)) #9 (1080)) #10 (1035)) # (1021) #11 # (1005)) #12 #13 (974) #14 (942)) #15 (936)) #16 (924)) #17 (920)) #18 (913)) #19 (901)) #20 (897) #21 (887) #22 (882)) #23 (881) #24 (881)) #25 (879)) #26 (875)) #27 (875)) #28 (874)) #29 (868)) #30 (844) #31 (839) #32 (839)) #33 (794) #34 (786)) #35 (785)) #36 (747)) #37 (731)) #38 (714)) #39 (706)) #40 (698) #41 (682) #42 (678)) #43 (675) #44 (657)) #45 (656)) #46 (651)) #47 (647)) #48 (638)) #49 (610)) #50 (593) #51 (592) #52 (591)) #53 (589) #54 (572)) #55 (571)) #56 (550)) #57 (549)) #58 (534)) #59 (534)) #60 (531) #61 (528) #62 (525)) #63 (522) #64 (514)) #65 (510)) #66 (507)) #67 (500)) #68 (497)) #69 (486)) #70 (479) #71 (476) #72 (468)) #73 (468) #74 (466)) #75 (463)) #76 (454)) #77 (453)) #78 (448)) #79 (441)) #80 (440) Real-World Sound Analysis - Dan Ellis 2009-12-10 - 29/30 Summary • LabROSA : getting information from sound • Speech monaural separation using eigenvoices binaural + reverb using MESSL • Environmental classification of consumer video landmark-based events and matching • Music transcription of notes, chords, ... large corpus mining Real-World Sound Analysis - Dan Ellis 2009-12-10 - 30/30