CSIS
Pen-Centric Shorthand Handwriting
Recognition Interfaces
Charles C. Tappert1 and Jean R. Ward2
1 Schoolof CSIS, Pace University, New York, USA
2 Pen Computing Consultant, Massachusetts, USA
PLT 2007
CSIS
Background
• Famous writings throughout history were
effectively written in a style of shorthand
– Cicero’s orations
– Martin Luther’s sermons
– Shakespeare’s and George Bernard Shaw’s plays
PLT 2007
CSIS
Thesis: Pen-Centric Shorthand Interfaces
• Can provide critical infrastructure for natural pen-
centric interactions, enhancing many pen-centric
learning applications
• Can provide faster text input for pen-centric
teaching, studying, and learning applications
• Will have greatest impact on the utility of
applications running on small mobile devices
PLT 2007
CSIS
Agenda
• Fundamental Property of Writing
• Handwriting Recognition Difficulties
• Online (Pen-Centric) Handwriting Recognition
• Online more accurate than Offline Recognition
• Online Info Can Complicate Recognition Process
• Design Tradeoffs/Decisions
• Historical Shorthand Alphabets
• Pen-Centric Shorthand Alphabets
• Pen-Centric Word/Phrase Shorthand
• Allegro/Chatroom Shorthand System
PLT 2007
CSIS
Fundamental Property of Writing
• Differences between different characters are
more significant than differences between
different drawings of the same character
• This makes handwritten communication
possible
PLT 2007
CSIS
Fundamental Property of Writing
• Property holds within subalphabets of uppercase,
lowercase, and digits, but not across them
• “I”, “l”, and “1” written with single vertical stroke
• “O” and “0” written similarly with an oval
PLT 2007
CSIS
Handwriting Recognition Difficulties
• Shape, size, and slant variation
• Similarly shaped characters – U and V
• Careless writing
– in the extreme, almost illegible writing
• Resolving difficult ambiguities requires
sophisticated recognition algorithms,
syntax/semantics
PLT 2007
CSIS
Online (Pen-Centric)
Handwriting Recognition
• Machine recognizes the writing while the user
writes
• Digitizer equipment captures the dynamic
information of the writing
– Stroke number, order, direction, speed
– A stroke is the writing from pen down to pen up
PLT 2007
CSIS
Online (Pen-Centric) more accurate
than Offline (Static) Recognition
• Can use both dynamic and static information
• Can often distinguish between similarly
shaped characters
– E.g., 5 versus S where the 5 is usually written
with two strokes and the S with one stroke
PLT 2007
CSIS
Online Information Can
Complicate Recognition Process
• Large number of possible variations
– E can be written with one, two, three, or four strokes, and
with various stroke orders and directions
– Four-stroke E has 384 variations (4! stroke orders x 24 stroke
directions)
PLT 2007
CSIS
Online Information Can
Complicate Recognition Process
• Segmentation ambiguities
– character-within-character problem
– lowercase d might be recognized as a cl if drawn with two
strokes that are somewhat separated from one another
PLT 2007
CSIS
Design Tradeoffs/Decisions
• No constraints on the user
– Machine recognizes user's normal writing
• User severely constrained
– Must write in particular style such as handprint
– Must write strokes in particular order, direction,
and graphical specification
PLT 2007
CSIS
Simplified Design Tradeoffs/Decisions
for Graffiti and Allegro PDA Alphabets
• Small alphabet
– one case rather than both upper and lowercase
• Small number of writing variations per letter
– preferably only one
• One stroke per character (character = stroke)
– allows machine to recognize each character upon pen lift
• Separate writing areas for letters and digits
– avoids confusion of similarly shaped letters and digits
PLT 2007
CSIS
Historical Shorthand Alphabets
• We first review the history of shorthand
systems prior to pen computing
• Shorthand is “a method of writing rapidly by
substituting characters, abbreviations, or
symbols for letters, words, or phrases”
• Shorthand can be traced back to the Greeks
PLT 2007
CSIS
Historical Shorthand Alphabets
• We focus on shorthand alphabets that might
be appropriate for PDAs
• We review two types of shorthand
– Geometric shorthand
• Small number of basic shapes
• Shapes reused in multiple orientations
– Non-geometric shorthand
PLT 2007
CSIS
Tironian Alphabet, 63 B.C.
PLT 2007
CSIS
Stenographie Alphabet, 1602
PLT 2007
CSIS
Stenographie Alphabet, 1602
Geometric shorthand – basic shapes/orientations
PLT 2007
CSIS
Moon Alphabet, 1894
• Geometric shorthand – basic shapes/orientations
PLT 2007
CSIS
Other Historical Shorthand Systems
• Phonetic alphabets
– Pitman (1837)
– Gregg (1885)
• Systems for the blind
– Braille (1824)
• Cursive shorthands
– Gabelsberger (1834)
PLT 2007
CSIS
Pen-Centric Shorthand Alphabets
• Some of the earliest were for CAD/CAM
– symbols represent graphical items and commands
• Others developed for text input on small
consumer devices like PDAs that have limited
computing power
• We review geometric and non-geometric
shorthands appropriate for small devices
PLT 2007
CSIS
Pen-Centric Shorthand Alphabets
• Historical alphabets presented above
could be used for machine recognition
– symbols drawn with a single stroke (except
K in Tironian and + in Stenographie)
• In addition to shape and orientation,
online systems can use stroke direction
to differentiate among symbols
PLT 2007
CSIS
Allen Alphabet
PLT 2007
CSIS
Allen Alphabet
Basic Shapes and Orientations
PLT 2007
CSIS
Goldberg Alphabet
PLT 2007
CSIS
Goldberg Alphabet
Basic Shapes and Orientations
PLT 2007
CSIS
Graffiti and Allegro Alphabets
• Not geometric alphabets
• High correspondence to Roman alphabet
• Might not qualify as shorthand but
included here
PLT 2007
CSIS
Graffiti Alphabet
PLT 2007
CSIS
Allegro Alphabet
PLT 2007
CSIS
Commercially Successful Shorthands
• Similar to the Roman alphabet
– Easier to learn
– Graffiti used in Palm OS devices
• notably the Palm Pilot and Handspring models
– Allegro used in Microsoft Windows devices
• Geometric alphabets not successful
PLT 2007
CSIS
Pen-Centric Word/Phrase Shorthand
e.g., Chatroom Shorthand
• Further increase speed of text entry
• Potential applications
– Where input speed important
– Where word/phrase abbreviations occur
frequently – e.g., email
PLT 2007
CSIS
Allegro/Chatroom Shorthand System
• Developed for M.S. dissertation
– Student was hearing impaired
– Developed as output component of
communication system
• Handwriting to text to speech
• Two input writing areas
– One for Allegro (all-purpose)
– One for chatroom-like words/phrases
PLT 2007
CSIS
Allegro/Chatroom Shorthand System
Stroke acquisition GUI
a single stroke
is it
character word/phrase
allegro stroke allegro stroke other stroke user-defined
library recognition recognition stroke library
alphabet meaning
sentence accumulator
no
done?
yes
Sentence display
and spoken output
PLT 2007
CSIS
Allegro/Chatroom Shorthand System
PLT 2007
CSIS
Conclusions
• Pen-centric learning interfaces should use
shorthand, and word/phrase shorthand where
appropriate, for fast text input
• Benefit of shorthand interfaces
– Provides critical infrastructure for many pen-centric
learning applications
– Enhances natural pen-centric interactions for teaching,
studying, and learning applications
– Has greatest impact on the utility of applications
running on small mobile devices
PLT 2007