Document Sample

Maximum Conditional Mutual Information Weighted Scoring for Speech Recognition Mohamed Kamal Omar, Ganesh N. Ramaswamy IBM T.J. Watson Research Center Present by: Fang-Hui Chu outline • Introduction • Problem formulation • Implementation • Experiments and results • Discussion 2 introduction • This paper describes a novel approach for extending the prototype Gaussian mixture model – achieved by estimating weighting vectors to the log likelihood values due to different elements in the feature vector • The weighting vectors which maximize an estimate of the conditional mutual information between the log likelihood score and a binary random variable representing whether the log likelihood is estimated using the model of the correct label or not 3 introduction • This estimate of the mutual information is conditioned on the maximum likelihood estimated HMM model • We show that maximizing this objective function is equivalent to maximizing the differential entropy of a normalized log likelihood score – under Gaussianity assumption of the log likelihood conditional PDF given the value of the binary random variable. 4 problem formulation • The mutual information of the log likelihood score and the binary random variable is I ( S , B) H ( S ) H ( S | B) • the acoustic log likelihood values for each frame in the training data is calculated using this HMM model as skt log P(Okt | ), 5 problem formulation • Using state-dependent weighting of the contributions to the likelihood due to different feature elements and replacing the sum over the Gaussian components by the maximum n log P(Okt | ) wj log P(Okt | m* ), j j 1 m where m* arg max H m P(Okt | m ) 6 problem formulation • To be able to compare the likelihood values estimated using different HMM states and , • To guarantee that the likelihood function will integrate to one over all the observation space • It can be shown that the following constraints are necessary and sufficient w 0 for 0 K , 0 j n, j 1 w j M n 2 jm H m j 1 for 0 K , m 1 j 1 w 7 problem formulation • the maximum conditional mutual information (MCMI) objective function N Tk K I kt log P skt | bkt log P skt , k 1 t 1 1 P( S ) q( B 0) P( S | B 0) q( B 1) P( S | B 1), I ( S , B) H ( S ) H ( S | B) p ( s , b) p( s, b) log s ,b p( s) p(b) p( s, b)log p( s | b) p( s) s ,b 8 problem formulation • an alternative approach to calculating an estimate of the objective function – By noticing that if both P(S|B=0) and P(S|B=1) are Gaussian PDFs with mean μ0 and μ1 and variance 0and 12 2 respectively – Using a normalization of the log likelihood score in the form skt b ~ skt kt , b kt – the conditional differential entropy of the normalized log ~ likelihood score, S , is constant – Therefore maximizing the conditional mutual information of the ~ normalized log likelihood score, S , and the binary random ~ variable B is equivalent to maximizing the differential entropy of S 9 problem formulation ~ • Since the variance of is constant, the differential S entropy of the normalized log likelihood score is maximized if and only if its probability density function (PDF) is Gaussian [ref: D. P. Bertsekas, Nonlinear Programming] • Maximizing the differential entropy of the the normalized log likelihood score becomes a maximum likelihood problem 10 problem formulation • we maximize the likelihood that the normalized log likelihood score is a Gaussian random variable • In this case, the maximum differential entropy (MDE) objective function to be maximized is N Tk K L kt 1 log skt 2 ~ , 2 k 1 t 1 1 2 2 2 11 implementation • Our goal is to calculate the state-dependent weights of the log likelihood scores, w 1, j 1 , which maximize the K , j n j MCMI and MDE objective functions – we can use an interior point optimization algorithm with penalized objective function • Alternatively to simplify the optimization problem, we impose the constraints in Equation 5 by normalizing the weights of the Gaussian components of each state model using the relation H m r H m 1 w jr n 2 jm jr j 1 w 12 implementation • This is an optimization problem over a convex set and we use the conditional gradient method to calculate the weighting vectors Wr 1 Wr r 1 (Wr Wr ), ˆ r W 0 W arg max W W r T I W | W Wr , – The Armijo rule is used to estimate the step size r 1 13 MCMI • The gradient of the MCMI objective function with respect to the state-dependent weighting vectors is ˆ I N Tk Mb kt kt b m skt mb Vkt W k 1 t 1 m1 mb 2 N Tk 1 Mf kt q( f ) kt mf skt mf Vkt , f 0 mf 2 k 1 t 1 m 1 – Vkt skt 0 , skt1,..., sktj ,..., sktn is the vector of log likelihood values for frame t of utterance k using the state ρ corresponding to different elements in the feature vector 14 MDE • The gradient of the MDE objective function with respect to the state-dependent weighting vectors is L N Tk skt kt Vkt W k 1 t 1 b kt 15 Experiments and results • Task: Arabic DARPA 2004 Rich Transcription (RT04) broadcast news evaluation data • 13-dimensional MFCC features computed every 10 ms. from 25-ms. frames with a Mel filter bank that spanned 0.125–8 kHz • The recognition features were computed from the raw features by splicing together nine frames of raw features (±4 frames around the current frame), projecting the 117-dim. spliced features to 60 dimensions using an LDA projection, and then applying maximum likelihood linear transformation (MLLT) to the 60-dim. projected features 16 Experiments and results • The acoustic model consisted of 5307 context-dependent states and 149K diagonal-covariance Gaussian mixtures • In the context of speaker-adaptive training to produce canonical acoustic models, we use feature-space maximum likelihood linear regression (MLLR) • The language model is a 64K vocabulary 30M n-gram interpolated back-off trigram language model • The estimation of the weights using the MCMI criterion converged after six iteration of the conditional gradient algorithm, while using the MDE criterion, it converged after four iterations 17 Experiments and results 18 Discussion • we examined an approach for state-dependent weighting of the log likelihood scores corresponding to different feature elements in the feature vector • we described two similar criteria to estimate these weights • This approach decreased the word error rate by 3% relative compared to the baseline system for both the speaker-independent and speaker-adapted systems • Further investigation of the performance of our approach on other evaluation tasks will be our main goal 19

DOCUMENT INFO

Shared By:

Categories:

Tags:
Conditional Random Fields, object recognition, hidden variables, gesture recognition, state sequence, conditional random ﬁelds, conditional random field, hidden Markov models, model parameters, speaker adaptation

Stats:

views: | 15 |

posted: | 9/15/2010 |

language: | English |

pages: | 19 |

OTHER DOCS BY fjwuxn

How are you planning on using Docstoc?
BUSINESS
PERSONAL

By registering with docstoc.com you agree to our
privacy policy and
terms of service, and to receive content and offer notifications.

Docstoc is the premier online destination to start and grow small businesses. It hosts the best quality and widest selection of professional documents (over 20 million) and resources including expert videos, articles and productivity tools to make every small business better.

Search or Browse for any specific document or resource you need for your business. Or explore our curated resources for Starting a Business, Growing a Business or for Professional Development.

Feel free to Contact Us with any questions you might have.