TOP TOP TOP The importance of auditory illusions for artificial listeners Computational Auditory Scene Analysis: An overview and some observations Dan Ellis International Computer Science Institute, Berkeley CA <dpwe@icsi.berkeley.edu> Dan Ellis International Computer Science Institute, Berkeley CA <dpwe@icsi.berkeley.edu> Outline “The organization of complex sound scenes according to their inferred sources” Computational Auditory Scene Analysis 1 Modeling Auditory Scene Analysis 2 A survey of CASA 2 A survey of CASA 3 Illusions & prediction-driven CASA 3 Prediction-driven CASA 4 CASA and speech recognition 4 CASA and speech recognition 5 Implications for duplex perception 5 Implications for other domains 6 Conclusions 6 Conclusions 1997oct24/5 - 1 CASA talk - Haskins/NUWC - Dan Ellis TOP 1997oct24/5 - 2 • Automatic sound organization? - convert an undifferentiated signal into a description in terms of different sources • Psychoacoustics defines grouping ‘rules’ - e.g. [Bregman 1990] - translate into computer programs? 1997oct24/5 - 4 TOP Human audition is very effective - unexpectedly difficult to model • ‘Correct’ analysis defined by goal - human beings have particular interests... - (in)dependence as the key attribute of a source - ecological constraints enable organization CASA1: CASA survey • Early work on co-channel speech - listeners benefit from pitch difference - algorithms for separating periodicities • Utterance-sized signals need more - cannot predict number of signals (0, 1, 2 ...) - birth/death processes • Hypothesis systems CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 5 Other approaches • Cooke & Brown (1991-3) - divide time-frequency plane into elements - apply grouping rules to form sources - pull single periodic target out of noise CASA talk - Haskins/NUWC - Dan Ellis • Blind source separation (Bell & Sejnowski) - find exact separation parameters by maximizing statistic e.g. signal independence • Klassner 1996 - search for a combination of templates - high-level hypotheses permit front-end tuning • HMM decomposition (RK Moore) - recover combined source states directly • Neural models (Malsburg, Wang & Brown) - avoid implausible AI methods (search, lists) - oscillators substitute for iteration? Perception is not direct but a search for plausible hypotheses • Ellis 1996 - model for events perceived in dense scenes - prediction-driven: observations - hypotheses CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 8 Data-driven... vs. Prediction-driven • 1997oct24/5 - 7 1997oct24/5 - 6 Prediction-driven CASA 3 Okuno et al. (1994-) - ‘tracers’ follow each harmonic + noise ‘agent’ - residue-driven: account for whole signal CASA talk - Haskins/NUWC - Dan Ellis Periodic pieces Weintraub 1985 - separate male & female voices - find periodicities in each frequency channel by auto-coincidence - number of voices is ‘hidden state’ TOP CASA3: • • 1997oct24/5 - 3 • Ultimately, more constraints needed - nonperiodic signals - masked cues - ambiguous signals TOP CASA2: • TOP 2 Motivations & Applications - it’s a puzzle: new processing principles? - real-world interactive systems (speech, robots) - hearing prostheses (enhancement, description) - advanced processing (remixing) - multimedia indexing (movies etc.) CASA talk - Haskins/NUWC - Dan Ellis Sounds rarely occur in isolation - getting useful information from real-world sound requires auditory organization CASA talk - Haskins/NUWC - Dan Ellis TOP Computational Auditory Scene Analysis (CASA) • • Outline 1 CASA talk - Haskins/NUWC - Dan Ellis Auditory Scene Analysis 1 Novel features - reconcile complete explanation to input - ‘vocabulary’ of noise/transient/periodic - multiple hypotheses - sufficient detail for reconstruction - explanation hierarchy CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 9 TOP TOP TOP Analyzing the continuity illusion • • • PDCASA example: Construction-site ambience Interrupted tone heard as continuous - .. if the interruption could be a masker CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 10 Problems - error allocation - source hierarchy 1997oct24/5 - 11 Recognize combined states? (Moore) - ‘state’ becomes very complex • Data-driven: CASA as preprocessor - problems with ‘holes’ (Cooke, Okuno) - doesn’t exploit knowledge of speech structure • Prediction-driven: speech as component - same ‘reconciliation’ of speech hypotheses - need to express ‘predictions’ in signal domain CASA talk - Haskins/NUWC - Dan Ellis TOP Example of speech & nonspeech 1997oct24/5 - 12 TOP Duplex perception as masking & restoration Prediction-driven analysis and duplex perception 5 • Single element → 2 percepts? - e.g. contralateral formant transition - doesn’t fit into exclusive support hierarchy • But: two elements at same position - hypotheses suggest overlap - predictions combine - reconciliation is OK • Order debate is sidestepped - .. not a left-to-right data path • Account for masking could ‘work’ for duplex - bilateral masking levels? - masking spread? - tolerable colorations? • Sinewave speech as a plausible masker? - formants hiding under each whistle? - greedy speech hypothesis generator • Problems: - where do hypotheses come from? (priming) - what limits on illusory speech? Problems: - undoing classification & normalization - finding a starting hypothesis - granularity of integration CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 13 TOP CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 14 CASA talk - Haskins/NUWC - Dan Ellis TOP Problem: inadequate signal data - hearing: masking - vision: occlusion - other sensor domains: noise/limits General answer: employ constraints - high-level prior expectations - mid-level regularities - low-level continuity • Hearing is a admirable solution • Prediction-driven approach suggests priorities CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 16 1997oct24/5 - 15 TOP Essential features of PDCASA Lessons for other domains • • - rating hypotheses - resynthesis CASA talk - Haskins/NUWC - Dan Ellis TOP • Speech recognition is very fragile - lots of motivation to use ‘source separation’ Prediction-driven can accommodate - special case or general principle? 5 • Data-driven just sees gaps • • CASA for speech recognition 4 • Prediction-reconciliation of hypotheses - specific hypotheses are pursued - lack-of-refutation standard • Provide a complete explanation - keeping track of the obstruction can help in compensating for its effects • Hierarchic representation - useful constraints occur at many levels: want to be able to apply where appropriate • Preserve detail - even when resynthesis is not a goal - helps gauge goodness-of-fit CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 17 Conclusions 6 • Auditory organization is indispensable in real environments • We don’t know how listeners do it! - plenty of modeling interest • Prediction-reconciliation can account for ‘illusions’ - use ‘knowledge’ when signal is inadequate - important in a wider range of circumstances? • Speech recognizers are a good source of knowledge • Wider implications of the prediction-driven approach - understanding perceptual paradoxes - applications in other domains CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 18 TOP The importance of auditory illusions for artificial listeners Dan Ellis International Computer Science Institute, Berkeley CA <dpwe@icsi.berkeley.edu> Outline 1 Computational Auditory Scene Analysis 2 A survey of CASA 3 Illusions & prediction-driven CASA 4 CASA and speech recognition 5 Implications for duplex perception 6 Conclusions CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 1 TOP Computational Auditory Scene Analysis: An overview and some observations Dan Ellis International Computer Science Institute, Berkeley CA <dpwe@icsi.berkeley.edu> Outline 1 Modeling Auditory Scene Analysis 2 A survey of CASA 3 Prediction-driven CASA 4 CASA and speech recognition 5 Implications for other domains 6 Conclusions CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 2 TOP Auditory Scene Analysis 1 “The organization of complex sound scenes according to their inferred sources” • Sounds rarely occur in isolation - getting useful information from real-world sound requires auditory organization • Human audition is very effective - unexpectedly difficult to model • ‘Correct’ analysis defined by goal - human beings have particular interests... - (in)dependence as the key attribute of a source - ecological constraints enable organization CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 3 TOP Computational Auditory Scene Analysis (CASA) • Automatic sound organization? - convert an undifferentiated signal into a description in terms of different sources • Psychoacoustics defines grouping ‘rules’ - e.g. [Bregman 1990] - translate into computer programs? • Motivations & Applications - it’s a puzzle: new processing principles? - real-world interactive systems (speech, robots) - hearing prostheses (enhancement, description) - advanced processing (remixing) - multimedia indexing (movies etc.) CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 4 TOP CASA survey 2 • Early work on co-channel speech - listeners benefit from pitch difference - algorithms for separating periodicities • Utterance-sized signals need more - cannot predict number of signals (0, 1, 2 ...) - birth/death processes • Ultimately, more constraints needed - nonperiodic signals - masked cues - ambiguous signals CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 5 TOP CASA1: frq/Hz Periodic pieces • Weintraub 1985 - separate male & female voices - find periodicities in each frequency channel by auto-coincidence - number of voices is ‘hidden state’ • Cooke & Brown (1991-3) - divide time-frequency plane into elements - apply grouping rules to form sources - pull single periodic target out of noise brn1h.aif frq/Hz 3000 3000 2000 1500 2000 1500 1000 1000 600 600 400 300 400 300 200 150 200 150 100 brn1h.fi.aif 100 0.2 0.4 0.6 0.8 CASA talk - Haskins/NUWC - Dan Ellis 1.0 time/s 0.2 0.4 0.6 0.8 1.0 1997oct24/5 - 6 time/s TOP CASA2: Hypothesis systems • Okuno et al. (1994-) - ‘tracers’ follow each harmonic + noise ‘agent’ - residue-driven: account for whole signal • Klassner 1996 - search for a combination of templates - high-level hypotheses permit front-end tuning 3760 Hz Buzzer-Alarm 2540 Hz 2230 Hz 2350 Hz Glass-Clink 1675 Hz 1475 Hz 950 Hz 500 Hz 460 Hz 420 Hz Phone-Ring 1.0 • Siren-Chirp 2.0 3.0 4.0 sec 1.0 2.0 TIME TIME (a) (b) 3.0 4.0 sec Ellis 1996 - model for events perceived in dense scenes - prediction-driven: observations - hypotheses CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 7 TOP CASA3: Other approaches • Blind source separation (Bell & Sejnowski) - find exact separation parameters by maximizing statistic e.g. signal independence • HMM decomposition (RK Moore) - recover combined source states directly • Neural models (Malsburg, Wang & Brown) - avoid implausible AI methods (search, lists) - oscillators substitute for iteration? CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 8 TOP Prediction-driven CASA 3 Perception is not direct but a search for plausible hypotheses • Data-driven... input mixture Front end signal features Object formation discrete objects vs. Prediction-driven Grouping rules Source groups hypotheses Noise components Hypothesis management prediction errors input mixture • Front end signal features Compare & reconcile Periodic components Predict & combine predicted features Novel features - reconcile complete explanation to input - ‘vocabulary’ of noise/transient/periodic - multiple hypotheses - sufficient detail for reconstruction - explanation hierarchy CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 9 TOP Analyzing the continuity illusion • Interrupted tone heard as continuous - .. if the interruption could be a masker f/Hz ptshort 4000 2000 1000 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 time/s • Data-driven just sees gaps • Prediction-driven can accommodate - special case or general principle? CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 10 TOP PDCASA example: Construction-site ambience f/Hz 4000 2000 1000 400 200 1000 400 200 100 50 Construction 0 1 2 3 4 5 6 7 Noise2 f/Hz 8 9 Wefts8,10 f/Hz 4000 2000 1000 400 200 4000 2000 1000 400 200 1000 400 200 100 50 Saw (10/10) Voice (6/10) f/Hz 4000 2000 1000 400 200 Click1 Clicks2,3 Click4 Clicks5,6 Clicks7,8 Wood hit (7/10) Metal hit (8/10) Wood drop (10/10) Clink1 (4/10) Clink2 (7/10) f/Hz 4000 2000 1000 400 200 1000 400 200 100 50 f/Hz 4000 2000 1000 400 200 Wefts1−6 Wefts7,9 −30 Noise1 −40 −50 −60 dB 0 • 1 2 3 4 Problems - error allocation - source hierarchy CASA talk - Haskins/NUWC - Dan Ellis 5 6 7 8 9 time/s - rating hypotheses - resynthesis 1997oct24/5 - 11 TOP CASA for speech recognition 4 • Speech recognition is very fragile - lots of motivation to use ‘source separation’ • Recognize combined states? (Moore) - ‘state’ becomes very complex • Data-driven: CASA as preprocessor - problems with ‘holes’ (Cooke, Okuno) - doesn’t exploit knowledge of speech structure • Prediction-driven: speech as component - same ‘reconciliation’ of speech hypotheses - need to express ‘predictions’ in signal domain Speech components Hypothesis management input mixture Noise components Predict & combine Periodic components Front end CASA talk - Haskins/NUWC - Dan Ellis Compare & reconcile 1997oct24/5 - 12 TOP Example of speech & nonspeech f/Hz 223cl 4000 2000 1000 400 200 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 0.4 0.6 0.8 1.0 1.2 1.4 Speech1 Click5 20 Clicks1−7 10 0 −10 dB • 0.0 0.2 time/s Problems: - undoing classification & normalization - finding a starting hypothesis - granularity of integration CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 13 TOP Prediction-driven analysis and duplex perception 5 • Single element → 2 percepts? - e.g. contralateral formant transition - doesn’t fit into exclusive support hierarchy • But: two elements at same position - hypotheses suggest overlap - predictions combine - reconciliation is OK • Order debate is sidestepped - .. not a left-to-right data path CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 14 TOP Duplex perception as masking & restoration • Account for masking could ‘work’ for duplex - bilateral masking levels? - masking spread? - tolerable colorations? • Sinewave speech as a plausible masker? - formants hiding under each whistle? - greedy speech hypothesis generator • Problems: - where do hypotheses come from? (priming) - what limits on illusory speech? f/Bark 15 80 60 S1−env.pf:0 10 5 40 0.0 0.2 CASA talk - Haskins/NUWC - Dan Ellis 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 1997oct24/5 - 15 TOP Lessons for other domains 5 • Problem: inadequate signal data - hearing: masking - vision: occlusion - other sensor domains: noise/limits • General answer: employ constraints - high-level prior expectations - mid-level regularities - low-level continuity • Hearing is a admirable solution • Prediction-driven approach suggests priorities CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 16 TOP Essential features of PDCASA • Prediction-reconciliation of hypotheses - specific hypotheses are pursued - lack-of-refutation standard • Provide a complete explanation - keeping track of the obstruction can help in compensating for its effects • Hierarchic representation - useful constraints occur at many levels: want to be able to apply where appropriate • Preserve detail - even when resynthesis is not a goal - helps gauge goodness-of-fit CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 17 TOP Conclusions 6 • Auditory organization is indispensable in real environments • We don’t know how listeners do it! - plenty of modeling interest • Prediction-reconciliation can account for ‘illusions’ - use ‘knowledge’ when signal is inadequate - important in a wider range of circumstances? • Speech recognizers are a good source of knowledge • Wider implications of the prediction-driven approach - understanding perceptual paradoxes - applications in other domains CASA talk - Haskins/NUWC - Dan Ellis 1997oct24/5 - 18