Recommendations Based on Speech Classification (and examples of what recommender systems can learn from signal processing) Christian Müller German Research Center for Artificial Intelligence International Computer Science Institute, Berkeley, CA Overview Speech as a source of information for non-intrusive user modeling Speech/signal processing Take-away messages Vocal aging -> features Knowledge-driven Recommendations Based on for speaker age feature selection recognition Speech Classification GMM/SVM supervector Classification methods approach for acoustic for independent “bag of (and observations” systems speech features examples of what recommenderfeatures can learn from signal processing) Detection task and Valid application- pseudo-NIST evaluation independent evaluation procedure Rank and polynomial Feature space warping rank normalization normalization Conclusions Speech as a Source for Non-Intrusive UM Now it’s time to Information about get to gate 38. adaptive the user speech dialog system speaker A ? classification user model speech = sensor adapts it's dialog inference from behavior sensors (e.g. detailed map with (not intrusive) shops vs. arrows) B explicit statement provides recommendations (intrusive) (e.g. a different route to the gate) Speaker Classification Systems Cognitive Load Best Research Paper Award UM 2001 Age and Gender Voice Award 2007 S Telekom live operation 2009 Audio segment y Language (telephone quality) s 14 languages + dialects NIST evaluation 2007 t e Identity m Project with BKA 2009 NIST* Evaluation 2008 Acoustic Events Project with VW 2008 Interspeech 2008 Recommendations Based on Speech Classification products media services actions strategies age gender emotions language dialect accent identity acoustic events Product Recommendations Based on Age and Gender Zur Anzeige wird der QuickTime™ Dekompre ssor „svq1“ benötigt. Product Recommendations Based on Age and Gender AM Michael Feld and Christian Müller. Speaker Classification for Mobile Devices. In Proceedings of the 2nd IEEE International Interdisciplinary Conference on Portable Information Devices (Portable 2008). 2008 How can you find features for building your models by explicitly studying the underlying phenomena? Proposing Knowledge-driven feature select the example of features for speaker age recognition Speaker Classification as an Interdisciplinary Area of Research Which are the manifestationsspeaker of a speaker be the How the the age (and the of age (and gender) in Which arecan requirements of a gender) classification system speaker’s voice automatically ? ? recognized on speaking style and how can they be solvedand the implementation layer ? Speech Speaker Phonetics Technology / Classificatio Voice Artificial n Pathology Intelligence Software- Technolog y Impact of Aging on the Human Speech Production Speech breathing effects: lower expirational volume more speech pauses lower amplitude thorax stiffer lungs lighter less elastic lower position Impact of Aging on the Human Speech Production laryngal area effects: rise of fundamental frequency (in men) reduced voice quality larynx calcification and ossification vocal folds loss of tissue stiffening Impact of Aging on the Human Speech Production supralaryngal area facial bones and muscles degeneration reduced elasticity effects: imprecise articulation for example vowel centralization Impact of Aging on the Human Speech Production neurological effects loss of tissue in the cortex reduced performance of the neuronal transmitters effects: reduced articulation rate defective coordination between the articulators vowel centralization Development of F0 in Men / Women F0 (Hz) 170 men 160 150 140 only non-smokers women 130 120 smokers and non- 110 smokers 100 Linville (2001) 90 age in years 20 30 40 50 60 70 80 90 Age Classes Female Male age CF CM Children <= 13 years YM Youth 14 - 19 years Adults AM 20 - 64 years Seniors >= 65 Jahren Age Classes Female Male age CF CM Children <= 13 years YM Youth 14 - 19 years Adults AM 20 - 64 years Seniors >= 65 Jahren Features fundamental frequency (pitch) mean pitch_mean standard deviation pitch_stddev min, max and difference pitch_min / pitch_max / pitch_diff voice quality shimmer shim_l / shim_ldb / shim_apq3 / shim_apq11 / shim_ddp jitter jitt_l / jitt_la / jitt_rap / jitt_ppq / jitt_ddp harmonics-to-noise-ratio harm_mean / harm_stddev articulation rate ar_rate speech pauses pause_num / pause_dur Features fundamental frequency (pitch) mean standard deviation min, max and difference voice voice quality shimmer jitter harmonics-to-noise-ratio articulation rate speaking style speech pauses Example Results C_YF AF SF YM_AM_SM C C Y Y A A S S F M F M F M F M high jitter value = low voice quality fundamental frequency (F0) Christian Müller. Zweistufige kontextsensitive Sprecherklassifikation am Beispiel von Alter und Geschlecht [Two-layered Context-Sensitive Speaker Classification on the Example of Age and Gender]. AKA, Berlin, 2006 C C Y Y A A S S F M F M F M F M speech pauses Hiearchical Feature Model High-level features (learned characteristics) semantics ? dialog A B b : b a e b d d e c : ideloect <s> how shall I say this <c> <s> yeah I know... phonetics /S/ /oU/ /m/ /i:/ /D/ /&/ /m/ / / /n/ /i:/ ... prosody spectrum Low-level features (physical characterstics) How can your features be modeled assuming that they are multi-dimentional represent repeating observations of the same kind can be assumed to be independent (“bag” of observations) Proposing the GMM/SVM Supervector Approach on the example of frame-by-frame acoustic features General Classification Scheme zk e.g. channel wkj compensation - multilayer perceptron support-vector machines 0.7 0,4 - (not addressed in this networks 1 Preprocessing talk) y1 y2 -1.5 0.5 1 1 1 wj Feature i Extraction 1 x1 x2 Classification Fusion Top-Down- Knowledge Modeling Acoustics and Prosodics semantics ? dialog A B b : b a e b d d e c : ideloect no ASR <s> how shall I say this <c> <s> yeah I know... phonetics /S/ /oU/ /m/ /i:/ /D/ /&/ /m/ / / /n/ /i:/ ... prosody spectrum Generative Approach: Gaussian Mixture Model (GMM) training “emergency vehicle” probability density “emergency feature vehicle” extraction model frame of speech test ? avg likelihood “emergency over all frames feature vehicle” for class extraction “emergency model vehicle” Generative Approach: Gaussian Mixture Model (GMM) test ? “emergency feature vehicle” extraction avg. log model likelihood ratio over all frame of speech frames for class “emergency vehicle” back- ground model A Mixture of Gaussians Means, variances, and mixtures weights are optimized in training Black line = mixture of 3 Gaussians Discriminative Method: Support Vector Machine (SVM) training “em. vehic.” (1) feature “em. vehic.” “not em. vehic.” (-1) extraction model Features are transformed into higher-dimensional space where problem is linear Discriminating hyper plane is learned using linear regression Trade-off between training error and width of margin Model is stored in form of “support vectors” (data points on the margin) Discriminative Method: Support Vector Machine (SVM) test ? feature score extraction (distance to hyper plane) Discriminative methods have shown to be superior to generative methods for similar tasks Features vectors have to be of the same lengths (sensitive to variable segment lengths) Solutions: feature statistics calculated over the entire utterance fixes portion of the segment sequential kernels GMM/SVM Supervector Approach feature extraction Gaussian means (MAP adapted) Combines discriminative power of SVMs with length independency of GMMs Very successful with similar tasks such as speaker recognition GMM is trained using MAP adaptation DCF Evaluation Results 25 23,41 19,55 20 14,58 15 10,22 8,09 10 3,45 5 0 t d d e se he he c c tir at at GMM-UBM en m nm u GMM-SVM Christian Müller, Joan-Isaac Biel, Edward Kim, and Daniel Rosario, “Speech-overlapped Acoustic Event Detection for Automotive Applications,” in Proceedings of the Interspeech 2008, Brisbane, Australia, 2008. How can you evaluate your multi- class models independently from the given application? How can you establish a appropriate evaluation in order procedure to obtain valid results? Proposing the detection task and the “pseudo NIST” evaluation procedure on the example of acoustic event detection and speaker age recognition. Background With multi-class recognition problems, many test/analyzing methods are very application specific. e.g. confusion matrices. we want a method that allows results to be generalized across a large set of applications. With home-grown databases, parameter tuning on the evaluation set often compromises the validity of the results/inferences. we want a fair “one shot” evaluation. The Detection Task system yes , 1.324326 emergeny vehicle ? Given a speech segment (s) and an acoustic event to be detected (target event, ET ) the task is to decide whether ET is present in s (yes or no) the system's output shall also contains a score indicating its confidence with more positive scores indicating greater confidence. Terminology Segment class e.g. segment event, segment age-class. ground truth (not known). Target the hypothesized class. Trial a combination of segment and target. Evaluation yes 1.32432 emergency vehicle ? system no -0.3212 music ? no 1.8463 talking ? no -2.5773 laughing ? yes 0.00132 phone ? no 2.20122 no event ? The system performance is evaluated by presenting it with a set of trials. Each test segment is used for multiple trials. The absence of all of all targets is explicitly included. Type of Errors segment “em. vehic.” system no target “em. vehic” ? “MISS” segment “em. vehic” system yes target “phone” ? “FALSE ALARM” Decision-Error Tradeoff misses “equal error rate” false alarms Selecting an operating point (decision threshold) along the dotted line trades misses off false alarms. Optimal operating point is application dependent. Low false alarm rates are desirable for most applications. Decision Cost Function C(ET, EN) = CMiss · PTarget · PMiss(ET) + CFA · (1-PTarget) · PFA (ET,EN) where ET and EN are the target and non-target events, and CMiss, CFA and PTarget are application model parameters. The application parameters for EER are: CMiss = CFA = 1 and PTarget = 0.5 Weighted sum of misses and false alarms using variable costs and priors. Application model parameters are selected according to the application. Example DET-Plot miss probability false alarm probability Christian Müller, Joan-Isaac Biel, Edward Kim, and Daniel Rosario, “Speech-overlapped Acoustic Event Detection for Automotive Applications,” in Proceedings of the Interspeech 2008, Brisbane, Australia, 2008. Example Cost Chart COSTS: (At, An) An C YF YM AF AM SF SM C -- 0.220 0.092 0.145 0.083 0.133 0.069 YF 0.166 -- 0.081 0.201 0.080 0.198 0.070 YM 0.076 0.084 -- 0.130 0.203 0.108 0.188 AF 0.088 0.161 0.110 -- 0.095 0.219 0.082 AM 0.064 0.083 0.254 0.139 -- 0.105 0.228 SF 0.096 0.150 0.100 0.249 0.091 -- 0.095 SM 0.065 0.085 0.238 0.117 0.246 0.118 -- Avg Cost 0.092 0.130 0.146 0.164 0.133 0.147 0.122 (At) Avg Cost 0.133 Acoustic GMM/SVM Supervector system on 7-class age task Pseudo NIST Evaluation Procedure ERL provided development and evaluation data as representative as possible for the application. Three months before the evaluation, ICSI was provided with the development data. At a pre-determined date, the blind evaluation data was provided to ICSI for processing. The system's output was submitted to ERL in NIST format. ERL downloaded the scoring software from NIST’s website, made the necessary modifications due to the changes in the labels. ERL ran the software on the submitted system output. The results were then disclosed to ICSI along with the keys (truth) for further analysis. --> Fair “one-shot” evaluation, no parameter tuning on the evaluation set. How can you normalize your features in order to obtain a uniform scale and a unifom distribution? Proposing rank normalization respectively polynomial rank normalization Background Fundamental frequency (pitch): 75-200 Hz Jitter: 0.001324 PPQ --> implicit feature weighing Mean/Variance Normalization 1 ai = vi − min(vi) max(vi) − min(vi) -1 1 uniform scale non-uniform distribution Rank-Normalization feature background model normalized feature 0101 0.01 0101 0 0 0101 0.75 ... 0101 0.01 0.25 ... 0101 0.06 0.5 0123 0.4 0101 0.13 0.75 2317 0.2 0101 0.29 1 ... 0101 0.06 ... ... 0101 0.13 ... create ordered list of values using bg 0101 0.29 data ... rank = position in list / number of values no occurrence mapped to 0 Rank Normalization 1 1 -1 1 -1 1 (+) uniform distribution (-) large three dimensional lookup tables (-) linear interpolation for unseen values larger values ? smaller values ? Polynomial Rank Normalization use ranks to train a polynomial apply polynomial instead of look-up tables better interpolation no need to store look-up tables Christian Müller and Joan-Isaac Biel. The ICSI 2007 Language Recognition System. In Proc eedings of the Odyssey 2008 Workshop on Speaker and Language Recognition. Stellenbosch, South Africa, 2008 Conclusions Speech as a source of information for non-intrusive user modeling Speech/signal processing Take-away messages Vocal aging -> features Knowledge-driven for speaker age feature selection recognition GMM/SVM supervector Classification methods approach for acoustic for independent “bag of speech features observations” features Detection task and Valid application- pseudo-NIST evaluation independent evaluation procedure Rank and polynomial Feature space warping rank normalization normalization Thank you!