; From NLP - University of Minnesota Duluth
Learning Center
Plans & pricing Sign in
Sign Out
Your Federal Quarterly Tax Payments are due April 15th Get Help Now >>

From NLP - University of Minnesota Duluth


  • pg 1
									  Technology Transfer of
Automated Essay Evaluation:
 From NLP research through
  deployment as a business

                  Jill Burstein
         Educational Testing Service
Presented to the Computer Science Department
        University of Minnesota, Duluth
            Fall Colloquium Series
                October 4, 2004
CriterionSM, E-rater®, Critique,
      C-rater, & more …
             Jill Burstein
          Claudia Leacock
          Thomas Morton
      Educational Testing Service
         Martin Chodorow
        Hunter College, CUNY
           Susanne Wolff
         Princeton University
  Let’s Talk About
Writing & Assessment
       Educators’ Vision:
   Writing Skill Development

• Master basic skills in K-12
  – Grammar, spelling, punctuation, etc.
• Perfected the 5-paragraph essay
  – U.S. Concept
  – Thesis, 3 Main Points, Conclusion
• Writing within and beyond discipline
  – Address different audiences
  – Generate various genres: persuasive, compare-
    and–contrast, research writing within discipline
   Evaluation of Vision Through
      Writing Assessments

• High stakes: Undergraduate & Graduate
  – Admissions
  – Placement
• No/Low stakes: K-12
  – Statewide and national assessments
  – Classroom instruction
What do essays look like?
The Reality of Writing Quality

• Timed Assessments
   – Up to 500 words (grade-level dependent)
   – Not Literary Essays
• Instructional Uses
   – Maybe Longer Essays
   – Better Quality with Revision Facility
     Most essays look like this
“My position on school uniforms is as follows. School uniforms are
a violation on several students rights. School uniforms makes us,
the students, think that we do not have the write to express our
feelings through clothing. Many students show pride through the
clothing that they choose to wear.

For instance some students may be of a different religion. Their
relgion may require them to wear a clothing of such. When the
school forces us to wear a uniform it forces these people to go back
on their religion. wearing differnent types of clothing expresses a
student's inner child, if not more. Through clothing we can see a
students hobbies, joys, and loves of life. Putting uniforms on us
would violate the fact to actually to show an opinion. Does the
school want us to look exactly alike? Some students may not have
an open mind about the fact that they cannot show their youth and
personalty through clothing they will show it in another unhealthy

In closing uniforms are and unjustice act against all students alike."
    And others like this…
  “You are stupid. You are stupid because you can't
read. You are also stupid becuase you don't speak
English and because you can't add.
   Your so stupid, you can't even add! Once, a teacher
give you a very simple math problem; it was 1+1=?.
Now keep in mind that this was in fourth grade, when
you should have known the answer. You said it was 23!
I laughed so hard I almost wet my pants! How much
more stupid can you be?!
   So have I proved it? Don't you agree that your the
stupidest person on earth? I mean, you can't read, speak
English, or add. Let's face it, your a moron, no, an idiot,
no, even worse, you're an imbosol.”
     And this ….

And this…

“I don't know how to explain this
question because I took a nap
while listening.

Sorry. “
  And this …
This is my topic on presidents. The one i will be talking about is Bill
Clinton. He like most of us have fingers, with fingernails. He also has
two arms were his fingers are on, were teh fingernails connect.What
can a arm be without a hand? nothing thats the answer, so he obviously
has two hands in witch the fingers are on, with the fingernails connect
    He also has eyes.....EYES o yeah even him the big cheese has eyes,
its weird cuz not many people have eyes but this good president does,
and he can SEE you SEE you, you might not be able to see him , but
he can SEE you.
      One time we were on AOL chat and i was talking to him he said
he like to climb trees in his underwear well his arms were covered with
sauce....pizza sauce. He said it makes him feel free and good about
himself. Also in the chat he said he has a pigeon for a pet and the
things name is Frances, he said they like to make bacon together in the
mourning and at night.And they eat at his friends Y place mostly
And this …
        “Is true that in so many jobs people have to wear
dress codes for so many reazons.Like in the restaurant
the workers are obligaded to use a drees code because
they have to look differently and have a good looking to
impresionate the cutmers.Not necesesary at all the
schools have drees codes but the ones that havet is
          Arriba todos mis compas ya llego el rey del
cristal, y yo mismo lo cosino para mejor calidad por esos
mismos motivos me busca la federal la dea de estados
unidos tambien me quiere agarar. si ellos me buscan por
tierra yo me les pelo por mar si piensan que ando colima
yo me paseo en michoacan por la ruana y poir tepeque
aguililla y cuancoman ,cuantas libras va a llevarse “
[Descriptive Translation: “It’s a rhyme about a drug-
runner … the guy is basically saying that he's the king
of Cristal meth, wanted by the DEA and the Feds. ”]
Human Scoring

human reader score                                            human reader score
              (S1)                                            (S2)

                 YES                                     NO
                              Is |S1 - S2| > 1 ?

                                                                 Final Score
                                               Final Score            =
             expert human reader score (S3)         =               mean
                                              mode, or mean
                                               of closest
Building Automated Essay
Scoring Capabilities
             Some History
– PEG – 1960’s essay scoring (Page, 1966)
  • Transformation of essay length
  • Some syntactic analysis
  • Convincing results
– Writer’s Workbench (Cherry et al, 1982)
  • Editing tool for students
  • Diction, style, spelling
  • Discourse structure („topic sentence‟ identification)
– Intelligent Essay Assessor (Landauer et al,
  • Essay scoring with latent semantic analysis (LSA)
  • Style and mechanics measures
          Assessment Market
         Technologically Ready
• Increase in Internet & Computer Access
   – Instructional computers with Internet access in
     public schools: 12.1 to 1 in 1998 & 4.8 to 1 in
     2001 (NCES, 2002)
   – Web resources used in over 40% of college
     courses (Campus Computing Project, 2000)
   – 99% of public schools have internet access (NCES
     report, 2002)
• State Assessments: Increase in computer-
  based delivery
• Largest Test Publishers offer 850+ digital
  textbook titles
Motivation in Assessment
• Cost Savings for Large-Scale
• Classroom Integration for
  – More practice writing possible
  – Electronic writing portfolios
  – Individual performance assessment
  – Classroom assessments
    Educators’ Questions
      About Innovation
1. Reliability: Can automated essay
   assessments increase scoring
   consistency for authentic assessments?
2. Assessment Type: Can automated
   scoring introduce more varied high-
   stakes assessments?
3. Costs/Performance: Can scoring costs
   be reduced, but scoring performance
Starting Development
What should a good essay look like?

• Clearly states the author's position, and effectively
persuades the reader of validity of author's
• Well organized, with strong transitions helping to
link words and ideas.
• Develops its arguments with specific, well-
elaborated support.
• Varies sentence structures and makes good word
choices; very few errors in spelling, grammar, or
   Mapping Writing Features
        to NLP Tools
 Writing Features           NLP Tools
Grammar Errors &       POS Taggers;
Sentence Structures    Syntactic Parsers
Vocabulary Usage       Content Analysis; POS
Sentence & Word-       Spelling Tools; POS
Level Mechanics        Taggers
Organization &         Discourse Analyzers
Development of Ideas
                E-rater (2/99 – 9/04)
• 50+ Writing-Relevant Features
   –   Syntactic Structure Features: clause types
   –   Discourse Structure Features: cue words & terms
   –   Content: Content vector analysis
   –   Lexical Complexity: e.g., word length, unique words
• NLP Tools
   – Syntactic Parses
   – High-level discourse analyzer
   – tf*idf (essay level & argument level)
• Topic-Specific Models
   – Training with Human-Scored Essays
   – Stepwise Linear Regression (Variable Feature Set & Weights)
• System Performance
   – Agreement with Humans
   – Comparable to Two Humans
   – E-rater/Human agreement : 59% exact; 98% exact + adjacent

human reader score                                        E-rater score
              (S1)                                        (S2)

                 YES                                    NO
                              Is |S1 - S2| > 1 ?

                                                              Final Score
                                               Final Score         =
             expert human reader score (S3)         =            mean
                                              mode, or mean
                                               of closest
Outcomes of Early Success
 NY Times Headline Phobia
Can you spell imbecile?: E-rater® Gives Good Score
to Bad Essay
By A. Reporter
ETS’s automated scoring system thinks that this essay should
get something like a “B.” Would you want your child to do
well on this kind of writing? You be the judge.
 “You are stupid. You are stupid because you can't read.
You are also stupid becuase you don't speak English and
because you can't add.
   Your so stupid, you can't even add! Once, a teacher give
you a very simple math problem; it was 1+1=?. Now keep
in mind that this was in fourth grade, when you should have
known the answer. You said it was 23! I laughed so hard I
almost wet my pants! How much more stupid can you be?!
   So have I proved it? Don't you agree that your the
stupidest person on earth? I mean, you can't read, speak
English, or add. Let's face it, your a moron, no, an idiot, no,
even worse, you're an imbosol.”
   Anomalous Essay Detection
Statistical evaluation of word usage to flag
anomalous essays
  – “Your essay does not resemble others being
    written on this topic.”
  – “Your essay might not be relevant to assigned
  – “Your essay appears to be restatement of the
    topic with a few additional concepts.”
  – “Compared to other essays written on this topic,
    your essay contains more repetition of words.”
  – “Your essay shows less development of a theme
    than other essays written on this topic.”
   Positive Outcomes:
Changing Business Model

• Cost Savings
– 1995- 2000: ETS Research, some marketing,
  little sales
• Revenue Generation
– 2001-2003: Spin-off (ETS Tech), all
  marketing, all sales, all product development,
  all the time
– 2003 – present: Spin-back (to ETS) with
  vastly increase marketing & sales
What teachers really wanted:
   Qualitative Feedback
Learning from Assessment Experts
 • Holistic scores not meaningful to students
       Score 3:
   –   While a position may be stated, either it is unclear OR
   –   May have organization in parts, but lacks organization in
       other parts.
   –   The support of the position may be brief, repetitive, or
   –   Inconsistent control of sentence structure, and incorrect word
       choices; errors in spelling, grammar, or punctuation
       occasionally interfere with reader understanding.
 • Demos for focus groups with teachers,
   policy makers, assessment experts
   – Errors in grammar, usage, mechanics, and style
   – Organization & Development
 More Innovation – More Questions

– Meaningfulness: Is the feedback
  consistently related to a clearly-defined
– Self-Evaluation: Can instructional
  software help students understand
  evaluation of their writing?
– Improvement: Can writing practice
  with immediate feedback help
    CriterionSM Online Essay
       Evaluation Service
• Critique writing analysis tools
  – Grammar
  – Usage
  – Mechanics
  – Style
  – Organization & Development
• E-rater
      Motivation For
New Capability Development
• What’s free for commercial use
  - Spelling
• What’s not …free …
  - Grammar
  - Usage
  - Mechanics
  - Style
• What doesn’t exist
  - Organization & Development
         Grammar, Usage and
          Mechanics Errors
• Corpus of well-formed text: 30 million words
  from newspapers
• Features: function words and part-of-speech tags
     a_AT good_JJ job_NN during_IN
• Collect frequencies for:
  – Unigrams of tags and function words
  – bigrams of tags and function words
     a_JJ AT_JJ JJ_NN NN_during NN_IN
• Method: pointwise mutual information and log
  likelihood ratio used to detect unexpected
  sequences – likely violations of English grammar
    Grammar, Usage and
     Mechanics Errors
• Harvest low probability bigrams from a
  set of essays.
• Low probability bigrams:
  – *these pencil, *every teenagers
• Write Filters:
  – *These pencil is yellow.
  – but not These pencil erasers are dirty.
• Fragments
• Garbled Sentences
• Subject-Verb Agreement: the motel are …
• Verb form: They are need to distinguish …
• Pronoun Errors: Them are my reasons …
• Possessive Errors: the students grades
• Wrong or Missing Word: The should take the
• Proofread This!: I think my through problems

• Article Errors: I like these song
• Confused Words: Because of there
  different genres …
• Wrong Form of Word: the right choose
• Faulty Comparison: It is more easier
• Nonstandard Verb or Word Form
•   Spelling
•   Missing Capitalization
•   Missing Initial Capitalization
•   Missing Question Mark
•   Missing Final Punctuation
•   Missing Apostrophe: Thats about the only
•   Missing Comma
•   Missing Hyphen: a well deserved vacation
•   Fused Words
•   Compound Words
•   Duplicate Words: escape to the another town
• Short sentences, unusually long
  sentences, and passives?
• Automatic detection of repeated words
  – 300 essays manually annotated for
  – Word-based text features with C5.0
     • proportion of word use in essays
     • distance between repeated word
     • pronoun?
     • word length
   How Do We Identify Organization &
   Development in Essay Writing?
• Discourse Theories
      • Lacks Essay-Based Discourse Function
         –Cue-word & term detection (Cohen, Hirschberg &
         Litman, Hovy et al, Knott, Mann & Thompson, Vander
         Linden & Martin, Sidner, & Quirk, et al)
      • Topical Coherence, Not Discourse Function
         –TexTiling – (Hearst & Plaunt)
         –LSA (Landauer et al)
         –Select-A-Kibbitzer (Weimer-Hastings & Graesser)
      • Not Student Friendly
         –RST Trees (Mann & Thompson)
• Essay-Based Discourse Analyzer
  (Burstein, Marcu, & Knight)
      • Background, Thesis, Main Points, Supporting Ideas, and
 Organization and Development:
 Essay-Based Discourse Analyzer
• 1400+ essays manually annotated with pre-defined
• Voting Between 3 Systems: 2 Probabilistic & 1
  – Probable discourse label sequences
  – Essay sublanguage: agree, should, would, opinion, for
    example, because, however...
  – RST relations: contrast, elaboration, antithesis...
  – Syntactic structures: infinitive, complement,
    subordinating clauses...
• Identifies background text, thesis statement, main
  ideas, supporting ideas, & conclusion statement
           Evaluating Capabilities

• Precision, Recall, & F-measure
  – Trade-off Precision for Recall
  – Better not to show falsely identified errors
• Grammar, Usage, & Mechanics (Bigrams)
   – Minimum Precision for Deployment
• Style & Organization & Development
   – Human-annotated data
   – Develop baseline comparisons
   – Precision, Recall, F-measure outperform
     baselines & approach human agreement
            Some Numbers
• Grammar, Usage, & Mechanics
  – (Minimum) Overall System Precision = 0.90
• Discourse Capability (Org & Dev)
  – Baseline Precision = 0.71
  – Overall System Precision = 0.85
  – Human agreement = 0.95
• Repetitive Word Use (Style)
  – Baseline Precision: 0.27
  – Overall System Precision = 0.95
  – Human Agreement: 0.55
       E-rater v.2.0 – Release 8/04
• 12 Features: Relevant to Writing Standards
  – Grammar, Usage, and Mechanics : Error Types
  – Style: Sentence Type, Sentence Length, Repeated Words
  – Organization: Thesis, Main Points, Support, and
  – Content: Vocabulary Usage
• Topic-Specific & Grade-Specific Models
  – Training with Human-Scored Essays
  – Multiple Regression (Standardized Feature Set &
    Variable Weights)
• System Performance
  – Agreement with Humans
  – Comparable to Two Humans
  – E-rater/Human agreement : up to 62% exact (from
    59%) ; 98% exact + adjacent
CriterionSM Online
Essay Evaluation
       Student Independence


E-rater score               Critique Writing Analysis

                        G, U, M, S          Org & Dev
Effectiveness Studies
Criterion Field Data (Attali, 2004)

  • Research Questions
     – Can we evaluate the basic effectiveness
       of Critique writing analysis tools?
     – Can students understand and respond to
       system feedback?
  • Criterion Field Data
     – Multiple submissions from about 9,000
       6th to 12th grade essays
     – Available for analysis:
       • First and last essay submission
       • Total number of submissions
        Summary Results
• 25% error reduction across 30+error types
• Increased number of essay-based discourse
   – background
   – main point
   – supporting idea
   – conclusion
  CriterionSM & Standardized Testing
             (Shermis et al, 2004)

•Research Question
 – Can Criterion use have a positive impact on
 FCAT writing scores?
 – 36 10th Grade English classes in Miami-Dade
     • 18 used Criterion
     • 18 used “traditional” instruction
       Summary Results

• Bad News
 – No significant differences in FCAT scores
• Good News
 – Significant growth in writing performance
   across different topics; Reduced numbers
   of errors
 – Significantly more writing productivity
           Users & Volumes
– GMAT (1999 – present)
– 350K essays scored each year
– Moving into Statewide Assessment
Criterion: E-rater + Critique
– K-12, college, and graduate level practice
– End 2002: 200 clients & 50K subscriptions
– End 2003: 445 clients & 127K subscriptions
– March 2004: 544 clients & 437K subscriptions
– September 2004: 580 clients & 530K subscriptions
International Exposure
– Users in Canada, Mexico, Pakistan, India, Estonia,
  Puerto Rico, Egypt, Nepal, Taiwan, Hong Kong &
          Beyond Essay Evaluation

C-rater (Leacock)
  –   short-answer, concept-based evaluation
      •   morphological analyzer
      •   pronoun resolution
      •   syntactic chunker
      •   predicate argument structure generator
      •   automated spelling correction
      •   word similarity matrices
      •   part-of-speech tagger
Test Item Creation Assistants
  – key-distractor selection for word-based test items
      • Statistical word similarity tools (Deane & Higgins)
  – article/paragraph selection for reading
     comprehension items
      • part-of-speech tagger
      • rhetorical parser
Next Generation
Additional Grammar Error Detection
 (Chodorow, Leacock, Han & Wolff)
   •Missing (or extra) Determiner Errors
      I can do so for __ following reasons
   •Preposition Errors
      a knowledge of/*at math
   •Long Distance Subject-Verb Agreement
      The use of dress codes are becoming a popular
Word Salad Detector: Prevent System Gaming
 (T. Morton)
   •Rare p-o-s tag sequences (mixing up content!)
       quick The the over brown dogs fox. jumped
   •Abnormal p-o-s tag distributions
      kfdl afjidaoi djfd &&&&**
  Current Research Problems

•Evaluate Coherence in Organization &
Development (Higgins, Burstein & Marcu)
  – Does the thesis statement respond to the
  – Do the main points relate to the thesis
  – Are all sentences in a supporting idea

•E-rater Trait-Based Scoring
     CL Research Contributions to
      Automated Text Evaluation
               (1 of 2)
Paul Jacobs, Jacqueline Kud, & Lisa Rao (General Electric IR Tools)
ESSAY SCORING PROTOTYPE (1996 – 1999): Lisa Braden-
Harder, Simon Corston-Oliver, George Heidorn, Karen Jensen, &
Steve Richardson (Microsoft syntactic parser+); Robin Cohen,
Sidney Greenbaum, Julia Hirschberg, Ed Hovy, Alistair Knott, Julia
Lavid, Geoffrey Leech, Keith Vander Linden, Diane Litman,
William Mann, James Martin, Elisabeth Maier, Candace Sidner,
Sandra Thompson, & Randolph Quirk (discourse theory); *Gerard
Salton (Vector Space Analysis)
E-RATER® (1999 - present): *Steve Abney (CASS parser), *Eric
Brill (p-o-s tagger), Hoa Trang Dang, Mary Dee Harris, Karen
Kukich, Thomas Landauer (scholarly debate), Leah Larkey, Ralph
Grishman (COMLEX), *George Miller (WordNet), *Thomas
Morton,*Adwait Ratnaparkhi (p-o-s tagger), *Susanne Wolff
     CL Research Contributions to
      Automated Text Evaluation
               (2 of 2)
Giovanni Flammia (Kappa Tool), Peter Foltz, Walter Kintsch, and
Thomas Landauer (LSA & text coherence), Marti Hearst & Christian
Plaunt (TexTiling), Derrick Higgins, P. Kinerva, J. Kristoferson, and
A. Holtz (Random Indexing), Kevin Knight & *Daniel Marcu
(Rhetorical/Discourse Structure Parsers), *Dekang Lin (Word
Similarity Indices) & Andrew McCallum & Kamal Nigam
(multivariate bernoulli), Eleni Miltsakaki, Ross Quinlan (C5.0), Rob
Schapire (BoosTexter), Peter Weimer-Hastings & Arthur Graesser
(essay coherence theory), Magdalena Wolska, & Vladmir Vapnik
(Support Vector Machines), Linguistic Data Consortium.
C-RATER (2000 – present): Claudia Leacock, Rebecca Passoneau
Deane & Derrick Higgins, R. Soricut (Rhetorical parser)
More Publications:

Tom Morton‟s Freeware Parser:
      OpenNLP Tools, Download

To top