Department of Computer Science _ Engineering - PowerPoint by pengxiang


									                Project title : Automated Detection of Sign Language Patterns
     Faculty: Sudeep Sarkar, Barbara Loeding, Students: Sunita Nayak, Alan Yang
       Department of Computer Science and Engineering, Department of Special Education
           Goal and Impact Statement                                                Representation that does not require tracking
                                                                                                                                                                                                Movement Epenthesis Aware Matching
Goal:                                                                              • We have proposed a novel representation that
To advance the design of robust computer representations                             captures the Gestalt configuration of edges and                                                  Movement epenthesis is the gesture movement that bridges
and algorithms for recognizing American Sign Language from                           points in an image.                                                                              two consecutive signs. This effect can be over a long
video.                                                                             • It can work with fragmented noisy low-level outputs                                              duration and involve variations in hand shape, position, and
Broader Impact:                                                                      such as edges and regions                                                                        movement, making it hard to model explicitly these
    • To facilitate the communication between the Deaf and                         • It captures the statistics of the relations between the                                          intervening segments. This has been a problem when trying
       the hearing population.                                                       low-level primitives                                                                             to match individual signs to full sentences. We have
    • To bridge the gap in access to next generation Human                            • Distance and orientation between edge primitive.                                              overcome this with a novel matching methodology that do not
       Computer Interfaces.                                                           • Vertical and horizontal displacement                                                          require modeling of movement epenthesis segments.
Intellectual Merit:                                                                   • Relationships between short motion tracks
We are developing representations and approaches that can
    • Handle hand and face segmentation (detection) errors,
    • Learn, without supervision, sign models from                                   • Normalized RD is an estimate of
       examples,                                                                       Prob (Any two primitives in the
    • Recognize in the presence of movement epenthesis,                                image exhibit a relationship)
       i.e. hand movements that appear between two signs.                            • The shape of the RD changes as
                                                                                       parts of the objects move.
    Unsupervised Learning of Sign Models                                             • Relational distributions over time
                                                                                       model high-level motion patterns.
Learn sign model given example sentences with one sign in
common. In the following two sentences, the target sign
model to be learned is HOUSE (marked in red)                                                                                                                                            The error rates for enhanced Level Building (eLB) (our
                                                                                                    Segmentation Aware Matching                                                         method), which accounts for movement epenthesis, and
                                                                                                                                                                                        classical Level Building (LB) that does not account for
                                                                                                                                             Frag-Hidden Markov Models:                 movement epenthesis.
                                                                      S1                        S2                   S3                ...
                                                                                                                                             • Groups across frames are linked
                                                                                                                                             • Best match is a path in this induced
                                                                      O1                       O2                    O3
                                                                                                                                                                                                           Publications and Acknowledgement
                                                                                                                                               graph over groups
                                                                                                                                             • Matching involves optimization           •    R. Yang; S. Sarkar, B. Loeding, Enhanced Level Building Algorithm for the Movement Epenthesis Problem in
SHE WOMAN HER HOUSE FIRE                                                                                                                       over states AND groups for each          Sign Language Recognition, to be presented at IEEE Conf. on Computer Vision and Pattern Recognition, 2007.
                                                                                                                                                                                        •    R. Yang; Sarkar, S., “Gesture Recognition using Hidden Markov Models from Fragmented Observations,”
                                                                                                                                               frame                                         IEEE
                                                                g1  { p1 , p1 }
                                                                  1       1
                                                                                           g12  { p12 , p2 }
                                                                                                                  g13  { p13 , p2 }
                                                                                                                                       ...                                              Conference on Computer Vision and Pattern Recognition pp. 766- 773, 17-22 June 2006.
                                                                                                                                                                                        •    R. Yang and S. Sarkar, “Detecting Coarticulation in Sign Language using Conditional Random Fields,”
                                                              g 2  { p2 , p3 }
                                                                1      1    1
                                                                                         g 2  { p 2 , p3 }
                                                                                           2       2     2
                                                                                                                g 2  { p 2 , p3 }
                                                                                                                  3       3     3

                                                                ……                          ……                     ……                                                                   International Conference on Pattern Recognition vol.2, pp. 108- 112, 20-24 Aug. 2006
                                                                                                                                                                                        •    S. Nayak, S. Sarkar, and B. Loeding, “Unsupervised Modeling of Signs Embedded in Continuous Sentences,”
                                                                                                                                                                                        IEEE Workshop on Vision for Human-Computer Interaction, vol. 3, pp. 81, June 2005.
                                                                                                                                                                                        •    R. Yang, S. Sarkar, B. L. Loeding, A. I. Karshmer: Efficient Generation of Large Amounts of Training Data for
                                                                                                                                                                                        Sign Language Recognition: A Semi-automatic Tool. ICCHP 2006: 635-642
                                                                                                                                                                                        •    B. L. Loeding, S. Sarkar, A. Parashar, A. Karshmer: Progress in Automated Computer Recognition of Sign
                                                                                                                                                                                        Language. ICCHP 2004: 1079-1087

                                                                                                                                                                                        This work was supported in part by the National Science
                                                                                                                                                                                          Foundation under ITR grant IIS 0312993.

                                    Center of Excellence in Pattern Recognition

To top