Handwritten Word Recognition: AN ew CAPTCHA Challenge by 6k81IkJt


									Handwritten Word Recognition:
 A New CAPTCHA Challenge

   Amalia Rusu and Venu Govindaraju
          University at Buffalo

 Completely Automatic Public Turing test to tell Computers and Humans
 An automated test that humans can pass but current computer programs
  fail – beyond the state-of-the-art
 Exploits the difference in abilities between humans and machines
   (i.e. text, speech or facial features recognition)
 A new formulation of the Alan Turing’s test - “Can machines think?”
      Please enter the handwritten word as it is shown below:

              If you cannot read this image click here

Example of interface and handwritten CAPTCHA to confirm registration.
User Authentication Steps using HCAPTCHA
                The user initiate the
                dialog and has to be                                         Authentication Server
                authenticated by server

                                Challenge              User authentication

                     Automatic Authentication Session for Web Services.

i.     Initialization
ii.    Handwritten CAPTCHA Challenge
iii.   User Response
iv.    Verification
                     Desirable Properties
   CAPTCHA should be automatically generated and graded
   Test can be taken quickly and easily by human users
   Test will accept virtually all human users and reject software agents
   Test will resist automatic attack for many years despite the
    technology advances and prior knowledge of algorithms
                          Previous Work

   First CAPTCHA designed in 1997 (for AltaVista website URL filter)
   CMU
           Gimpy, EZ-Gimpy, Gimpy-R, Bongo, Pix, Eco
   PARC
   UCB & PARC
   Microsoft
   Bell Labs
           Reverse Turing test using speech
   GIT
           Character morphing
                                CAPTCHA Tests

       AltaVista URL filter uses isolated
       random characters and digits on a
       cluttered background.

BaffleText uses pronounceable character strings that are not   PessimalPrint uses a degradation
in the English dictionary and render the character string      model simulating physical defects
using a font into an image (without physics-based              caused by copying and scanning of
degradations); then generate a mask image as shown above.      printed text.
                       CAPTCHA Tests

                                          EZ-Gimpy uses real English words.

Type 3 different English words              Gimpy-R uses nonsense words.
appearing in the picture above.

          Character morphing algorithm that transforms a string
          into its graphical form.
              Why Handwritten CAPTCHA?
 No handwritten text based CAPTCHA exists - so far!!!
 Several machine printed text based CAPTCHA already broken
      Greg Mori and Jitendra Malik of the UCB have written a program that can solve
       Ez-Gimpy with accuracy 83%
      Thayananthan, Stenger, Torr, and Cipolla of the Cambridge vision group have
       written a program that can achieve 93% correct recognition rate against Ez-
      Gabriel Moy, Nathan Jones, Curt Harkless, and Randy Potter of Areté
       Associates have written a program that can achieve 78% accuracy against
   Machine recognition of handwriting is more difficult than printed text
   Handwriting recognition is a task that humans perform easily and reliably
   Research is in the early stages - a promising field
   Handwritten CAPTCHAs will challenge the KBCS community!

      Lexicon Lexicon Driven                               Grapheme Model

                   time         accuracy                   time         accuracy
                   (secs)                                  (secs)
                                Top 1         Top 2                     Top 1        Top 2
      10           0.027        96.53         98.73        0.021        96.56        98.77
      100          0.044        89.22         94.13        0.031        89.12        94.06
      1000         0.144        75.38         86.29        0.089        75.38        86.29
      20000        1.827        58.14         66.56        0.994        58.14        66.49

Speed and accuracy of a HR. Feature extraction time is excluded. Testing platform is an Ultra-SPARC.
       Source of Errors for HW Recognizers
 Image quality
   Background noise, printing surface, writing styles
 Image features
   Variable stroke width, slope, rotations, stretching, compressing
 Segmentation errors
   Over-segmentation, merging, fragmentation, ligatures, scrawls
 Recognition errors
   Confusion with similar lexicon entries, large lexicons
                  Creating H-CAPTCHAS
 Use handwritten word images that current recognizers cannot read
 Controlled “distortion” of existing handwritten word images
 Create handwritten images by concatenating handwritten character
     Use handwritten US city name images (4,000 from CEDAR CDROM)
     Character images were discretely printed to begin with
     Character images are automatically segmented out of handwritten word
     Use set of 20,000 handwritten character images (extracted by program)
 Synthesize sentence images by gluing together isolated upper and lower
  case handwritten characters or word images
          H-CAPTCHA Generation Algorithm
 Original (random) handwritten image (existing US city name image or
   synthetic word image with length 5 to 8 characters or meaningful sentence).
 Lexicon containing the image’s truth word.
 H-CAPTCHA image.
 Randomly choose a number of transformations
 Randomly establish the transformations corresponding to the given number
   from: add lines, circles, grids, arcs, background noise (multiplicative or
   impulse), random convolution masks, blur, wave, spread, median filters,
   thick or thin characters on vertical or horizontal fashion, etc.
 A priori order is assigned to each transformation based on experimental
   results. Sort the list of chosen transformations based on their priority order
   and apply them in sequence, so that the effect is cumulative.
         Handwritten text images

Examples of handwritten characters used to generate random words.

Examples of handwritten US city name images used as a base for

    Examples of synthetic handwritten sentence images.
H-CAPTCHA by Image Quality Transforms
Add lines, grids, arcs, background noise, convolution masks and special
 H-CAPTCHA by Image Features Transforms

Variable stroke
width, slope,
rotations, stretching,
H-CAPTCHA by Segmentation Transform

 Delete ligatures, use
 touching letters/digits,
 merge characters for
 over segmentation or to
 be unable to segment
         H-CAPTCHA by Lexicon Transform
                                             WMR          Accuscript
                              Truth          results        results    Image
                                          (Top choice    (Top choice
                                              first)         first)
                             Orlando     ovlando        ollando
                                         ovlavdo        ovlando
                                         onlando        orlanolo
Lexicon challenges: size,                orlanolo
                                         oviando        ovlavdo
density, availability                    orlahdo
                                          orlando       orlanda
                                         ovlanao        arlando
                            Lackawanna   lackaevana     lackawarna
                                         lackawawa      lactawana
                                         lackawaua      lackawarra
                                         lackowana      lackawawa
                                         lackawana      lackawana
                                          lackawanna    lackawaua
                                         lackawarna      lackawanna
                                         lackawanra     lackowana
                                         lackamama      locrawara
                                         lactawana      lackawanra
                             Clarence    clarlncl       claience
                                         clarlnce        clarence
                                         clarencl       clatence
                                         cearence       clarlnce
                                          clarence      cearence
                                         cbarence       clavence
                                         clorence       clarenxe
                                         clahence       clasence
                                         aarence        clorence
                                         clawce         claiexce
                              Buffalo    buffaio        ruffalo
                                          buffalo        buffalo
                                         butfalo        buffrlo
                                         buifalo        buffaio
                                         buffrio        buffrio
                                         ruffalo        bulfalo
                                         bulfalo        buifalo
                                         bufialo        butfalo
                                         buefaio        buefalo
                                         bullalo        bufialo
                 H-CAPTCHA Evaluation
 No risk of image repetition
    Image generation completely automated: words, images and distortions
     chosen at random
 The transformed images cannot be easily normalized or rendered
  noise free by present computer programs, although original
  images must be public knowledge
 Deformed images do not pose problems to humans
    Human subjects succeeded on our test images
 Test against state-of-the-art: WMR, Accuscript
    CAPTCHAs unbroken by CEDAR recognizers

Handwritten US city name images that defeat both WMR and Accuscript recognizers.
                         H-CAPTCHA Challenge
                                              Number of
                 Word Recognizers                                      Accuracy
                                           Recognized Images
                      WMR                          383                   9.28%
                    Accuscript                     182                   4.41%

Low accuracy of handwriting recognizers. The lexicons are created so as to contain all the
 truths of test images. Total number of tested images is 4,127 (and so is the lexicon size)

     Number of          Number of         Humans             WMR             Accuscript
      Students          Test Images       Accuracy          Accuracy         Accuracy
         12                 15               82%               0%                 0%

         Low accuracy of handwriting recognizers vs. humans on a subset of test images.
            CAPTCHA using Gestalt Psychology
 Gestalt psychology is based on the observation that we often experience things
  that are not a part of our simple sensations
 What we are seeing is an effect of the whole event, not contained in the sum of
  the parts (holistic approach)                                           OXXXXXX

 Organizing principles - Gestalt laws:                                   XOXXXXX
       law of closure              [   ][    ][   ]                     XXXXOXX
       law of similarity                                                XXXXXXO

       law of proximity                     **********
       law of symmetry
       law of continuity
       law of familiarity                   **********

       figure and ground
 Not restricted to perception
     memory
             H-CAPTCHA based on Gestalt Laws
Gestalt laws: law of proximity, symmetry, familiarity, continuity

Methods: create horizontal or vertical overlaps - for same words smaller distance overlaps
                                                - for different words bigger distance overlaps
        H-CAPTCHA based on Gestalt Laws
Gestalt laws: law of closure, proximity, continuity

Methods: create occlusions by circles, rectangles, lines with random angles
         H-CAPTCHA based on gestalt laws

Gestalt laws: law of closure, proximity, continuity

Methods: add occlusions by waves from left to right on entire image, with
various amplitudes / wavelength or rotate them by an angle
        H-CAPTCHA based on Gestalt Laws
Gestalt laws: law of closure, proximity, continuity, background

Methods: use empty letters, broken letters, edgy contour, fragmentation
         H-CAPTCHA based on Gestalt Laws
 Gestalt laws: memory, internal metrics, familiarity of letters

                                                   vertical mirror – difficult for

                                                    horizontal mirror – difficult for

                                                     flip-flop –OK for humans!!

Methods: change word orientation entirely, or the orientation for few letters only
                              Gestalt H-CAPTCHA Results
              Horizontal     Horizontal                                                    Less       More
  Word                                    Vertical   Occlusion   Occlusion    Empty                                Old
               Overlap        Overlap                                                   Fragment-   Fragment-
Recognizers                               Overlap    by waves    by circles   Letters                           Transforms
               (Small)        (Large)                                                     ation       ation

  WMR          24.35%         12.93%      27.88%      15.43%      35.93%      0.89%        0%        0.48%        9.28%

Accuscript      2.93%          2.42%      12.64%      10.56%      32.34%      0.06%      0.18%         0%         4.41%

                           Tested images is 4,127 for each type of transformation.
                                Future Work
Personalizing Email Addresses
              Original Email

              Apply Image

            Transformed Email

   Creates transformed alias e-mail addresses to prevent mining by software agents
                                           Future Work
     Adult vs. Child vs. Machine
   Few methods to differentiate between
    adult vs. child
     o   Asking a question that has the answer in
         the handwritten sentence
     o   Giving an incomplete handwritten
         sentence and asking to imply the missing
     o   Comparing the handwritten text with a
         standard word list
     o   Using     longer,     more     complicated
         handwritten sentences, using advanced
         topics from technical fields such as math,
         physics, or financial
   Useful on Internet services due to
    expansion of harmful minor websites
                                                      Reading abilities delimitation:
                                                      Machine vs. 1st grade child
                                                      Adult vs. 7th grade child
                            Future Work
 HCAPTCHA based on Handwritten Sentence Reading and Understanding
 Incorporate and adjust the image complexity factor as a parameter of error
 Try out more image transformations and compare results against humans
   Cognitive aspects of HCAPTCHA for adult vs. child protocol
   HCAPTCHA as a Challenge Response Protocol for Security Systems
   Online-Handwriting CAPTCHA
   HCAPTCHA as a Biometric?
   HCAPTCHA normalization concerns based on future technology development
Thank You
          Power of Context

Context                      Ranked

To top