An Attempt at capturing Tutoring Style in Tutorial Dialog

Document Sample
An Attempt at capturing Tutoring Style in Tutorial Dialog Powered By Docstoc
					   An Attempt at capturing
Tutoring Style in Tutorial Dialog
       Conversational Interfaces
        Course Project: Fall 005

              Rohit Kumar
        rohitk @ cs . cmu . edu
     Layout of the Talk
o Introduction & Motivation o Simplistic Tutorial Dialog
o Examples of different tutor
  styles                      o Decision Model of the
                                Tutorial Dialog Process
                                 o Classification
o Definition of Style            o Clustering
                                 o Association Rules
o Literature Review
                             o Another Unit Size: Topics
o The Corpus
                             o Conclusions

o First Analysis: LIWC       o Future Work

o Revised set of Questions
Introduction & Motivation
  Different Human Tutors have different learning
  effects. Why ?

   o Different tutors know the subject to different extents

   o Different tutors have different tutoring styles (for that

      o In Tutorial Dialog, what characterizes these different
        tutoring styles ?

      o Do tutors really follow a characteristic style ?

      o Does following a characteristic style have a consistent
        learning effect (so that it should be implemented in a
        Tutorial Dialog System) ?
Examples of different styles
  Tutor: V
    <agent="tutor">but you cant inc it forever as it is constrained my material properties</sentence>
     <agent="tutor">so change the temp now to inc eff</sentence>
     <agent="tutor">did it inc?</sentence>
     <agent="tutor">go to the max tem p you can</sentence>
     <agent="tutor">no no</sentence>
     <agent="tutor">nice ~~</sentence>

  Tutor: M
     <agent="tutor">you can change the value of tmax</sentence>
     <agent="tutor">to the maximum possible value</sentence>
     <agent="tutor">for stainless steel</sentence>
     <agent="tutor">which is 570 c ~~</sentence>

     <agent="tutor">there is another constraint</sentence>
     <agent="student">and how do i know the high temperature</sentence>
     <agent="student">they didnt give me any tables</sentence>
     <agent="tutor">that is limited by how much temperature the boiler materials can withstand</sentence>
     <agent="tutor">on page 1</sentence>
     <agent="tutor">it is specified</sentence>
     <agent="tutor">that maximum t and p in the cycle are</sentence>
     <agent="tutor">570 c</sentence>
     <agent="tutor">and 20000 kpa respectively</sentence>
     <agent="tutor">did you see it?</sentence>
More Examples: tmax_analysis
 Tutor: V
     <agent="tutor">note the initial thermal eff then try to improve it</sentence>
     <agent="tutor">ya plot eff vs tmax</sentence>
     <agent="tutor">tools and sesitivity</sentence>
     <agent="tutor">select the state</sentence>
     <agent="student">which one</sentence>
     <agent="tutor">temp is maximum after the boiler right</sentence>
     <agent="tutor">plot is from 300 to 600</sentence>
     <agent="tutor">see your instructions</sentence>
     <agent="tutor">do you understand what this means</sentence>
     <agent="tutor">as the temp inc eff inc</sentence>

 Stylistic Characteristics:
 o Authoritative
 o Didactic Tutoring (Instructive)
 o Strong and Dominant
 o Often Negative
 More Examples (contd.)
Tutor: M
    <agent="tutor">ok now lets g to tools sensitivity</sentence>
    <agent="tutor">refer page 5</sentence>
    <agent="tutor">lets see what happens to efficiency when we change the heater temperature</sentence>
    <agent="tutor">do analysis 1</sentence>
    <agent="tutor">the independent parametrs are filled in already</sentence>
    <agent="tutor">you just need ti fill in dependent parametrs</sentence>
    <agent="tutor">i mean you need to fill in independent parametrs</sentence>
    <agent="tutor">select parameter to vary</sentence>
    <agent="tutor">so what does this plot tell you?</sentence>
    <agent="student">the higher the temp the higher the eff</sentence>
    <agent="tutor">which is what we thought</sentence>
    <agent="tutor">higher the steam temperature higher the efficiency ~~</sentence>

Stylistic Characteristics:
o Similar to Tutor V in some ways
o More collaborative than Authoritative
o Wordier
o More and continuous feedback
o Rarely Negative
 More Examples (contd.)
Tutor: E
    <agent="tutor">what max t did you use in your sensitivity analysis?</sentence>
    <agent="student">not tmax?</sentence>
    <agent="student">it says the temp leaving heater 2 =tmax from optimized simple cycle</sentence>
    <agent="tutor">ok there's a reason 4 that</sentence>
    <agent="tutor">we'd see that it needs to be changed</sentence>
    <agent="student">so then it's not tmax?</sentence>
    <agent="tutor">please go on</sentence>
    <agent="student">so i leave it as 575?</sentence>
    <agent="tutor">yeah ~~</sentence>

Stylistic Characteristics:
o Mostly confused and highly divergent from topic
o Often negative
o Rare Feedback
o Lesser content words & short utterances
o Mostly confused and highly divergent from topic
What is Tutoring Style ?
o Tutor has a global agenda of getting
  concepts through to the students while
  tutoring by solving problems

o Preference to a choice of way of
  delivering the conceptual content
  characterizes an individual style of the
  o What are choice of ways ?
     o Features & Dimensions of Style (Lit-Review)

o Choice of content is also made by
  tutor and is orthogonal (not exactly) to
      Literature Review
o Looking for feature dimensions considered for tutorial dialog
  in different studies
    o Relevant Tutor Turn Features (system of choices)
        o Type of Response
             o Type of Feedback
             o Type of Question
             o Type of Example
        o Error Attribution

    o Several other features
        o Student Turns
        o Tutoring models
        o Task Characteristics

o Wever et. al. 2005, Different Units of Content Analysis (Feature

o Literature about different tutoring models and their effects
    o Studies about effectiveness of individual tutoring models
        o E.g.: Rose et. al., 2001: Socratic vs. Didactic tutoring
    o Graesser & Person, 2003: “Tutors rarely adhere to ideal tutoring
                   The Corpus
                   Thermodynamic Cycle Tutoring by Human Tutors

                   3 different tutors (Mechanical Engineering Graduate Students)
                   17 students tutored (Mechanical Engineering Undergraduate Students)
Tutor    No. of
                   Students work in a Simulation Environment.
 V         5

 M         9       Task:
                   Analyzing given variations of Rankine cycles to
 E         3       get maximum efficiency while understanding
                   underlying concept

                   Tutor on a remote locating chatting over an
                   Instant Messenger.

                   Tutor can see and act on the Student screen if
LIWC Dimensions as Features
   First Analysis
       o To see relations of learning to the various shallow
         linguistic dimensions of the tutor’s dialogs
       o Unit of Analysis: Dialog

       o CogMech words predict post test better than PreTest
       o R-Sq: 83.8%
       o Significant effect size for Students tutored with High
         CogMech: 1.57

       Predictor    Coefficient         T               P

        PreTest       0.4772          4.01            0.001

       CogMech        2.3010          4.42            0.001
  LIWC Analysis (contd.)
               Tutor     High       Total

                V          5         5
                M          4         9
                E          0         3
Pairwise Comparison of Tutors
   •   Tutor V significantly better than tutors M and E
   •   Tutor M slightly better than tutor E

Reason to consider LIWC Dimensions (in general) like
CogMech (in particular) as features for further analysis of
Tutoring Styles.
     More Questions
o Having seen that some of LIWC Dimensions
  (like CogMech) can play an important role in
  capturing a particular Tutor’s style at the
  Dialog level of Content Analysis

  o We want to capture How does use of these style
    aspects happen at the utterance level so that it
    can be used to design real Tutorial Dialog Systems

  o We want to see How consistently do tutors follow
    these style aspects

  o We want to pick up the best of style aspects that
    lead to higher learning effects
Simplistic Tutorial Dialog Process
o At every turn in a tutorial dialog, the tutor constructs his
  utterance so as to further the global agenda while
  maintaining the continuum of dialog by appropriate
  choice of utterance (or topic) based on the earlier
  turns of the student and the tutor

o Every step requires a choice of content and a way of
  delivering the content (Stylistic Choice)

o Style is the preference to one of the possible choices
  for the given content scenario

Not that Simplistic really !

o What are the choices of way of delivery ?
  o Qualitative / Quantitative Listing ??
  o Multi - Dimensional ??
  o Dependence on content
Simplistic Tutorial Dialog Process (contd.)
     E                                           O
     F                                           A
                        Choice of using this
                        way of delivering        G
                        content is made by the
                        tutor at every turn      E
                        based on
                        • earlier choices
     R                  • student turns          D
     I                  • type of content
                        • global agenda          A

          Steps towards the Global Agenda
Decision Model for Tutorial Dialog
Inventory Building
   o Dialog corpora converted to a Turn taking
     sequence of Student and Tutor Dialogs

   o Tagged with LIWC based Features for each
     utterances along with features of context
      o LIWC Features:
          o CogClass, Negation, Optimism, Question
      o Context Features:
          o LIWC Features of previous Tutor utterance
          o LIWC Features of previous Student utterance

   o Extracted only Tutor Utterances

   o Grouped by Tutors, Learning (High/Low)
        Decision Trees
o Modeling by ID3

o Models for Each LIWC Dimension for Each Group

o Results in the XL Sheet

o Moderate values of Prediction Accuracy
   o Best Models for Tutor E and Low Learning

o Low values of Kappa (except for couple of cases)

o Why ?
   o Insufficient context features
   o Incorrectness of features
       o Due to conversion to turn taking
   o Inappropriate Modeling Technique
o Simple K Means

o For checking Consistency

o Results in XL Sheet

o High Clustering Errors (except for few cases)
   o Tutors are not making a Principled choice
   o Lack of consistency in Style

o Best Clusters for Tutor V and Low Learning
       Association Rules
Relation:     Squished-LowLearning-All-CogClass

/* 0.080631 0.000000 */
PS-Optimism = yes and PS-CogClass = OTHER and PT-Question = no ==> CogClass = INSIGHT

/* 0.077386 0.000000 */
PS-Optimism = yes and PS-CogClass = NONE and PT-Optimism = yes ==> CogClass = CERT

Relation:     Squished-V-All-Negation

/* 0.133767 0.000000 */
PS-Question = yes and PS-CogClass = NONE and PT-Optimism = yes ==> Negation = yes

/* 0.109806 0.000000 */
PS-Negation = yes and PT-Question = no and PT-Negation = yes ==> Optimism = yes

Relation: Squished-E-All-Negation

/* 0.126733 0.000000 */
PS-CogClass = OTHER and PT-Question = yes and PT-Negation = no ==> Negation = yes
Different Unit of Analysis: Topics
   o Tutorial Dialog segmented by Topic
      o 15 different topics
      o Different tutors have different ordering and number of
         topics that they talk about

   o Experiment with Decision Tree
      o 82.3529% (14 of 17) tutors correctly classified
      o Kappa: 0.7167*
      o 3 incorrect classifications
           o 2: V classified as M
           o 1: M classified as E

   o Experiment with Clustering
       o 35.2941% Clustering Error (by Tutors)
       o Tutor V most inconsistent with distribution of topics

   o Further need to work on modeling individual choice of topic
     based on state of the dialog and global agenda
o LIWC Dimensions are good at distinguishing Tutors

o Significant relation between CogMech and PostTest:
  Analysis with Unit as Dialog

o Fine Grained Analysis: Utterance as a Unit of Analysis
   o Modeling tutoring style (for considered dimension) using
     Decision Trees not often accurate
   o Most tutors don’t follow a style in a Principled way
   o Where are the lessons for building Tutorial Dialog Systems:
     Association Rules

o Another Unit of Analysis: Topic
   o Distinguishes Individual tutors very well in distribution of
     Conclusions (contd.)
o What have we got here
  o Tutors do not follow a style in a principled
    way to a large extent
  o Tutors follow a style to different extents

o Multiple applications to Tutorial Dialog
  o Decision Models for guiding Text (or
    content) generation
  o Getting principles for Authoring Dialog
    (from Association Rules)
           Future Work
o Lots of Features
   o   Different Dimensions
   o   Extended Contexts
   o   Accumulating Features
   o   Correct Features

o Lots of Modeling techniques

o Different Units of Analysis

o Modified model of Tutorial Dialog Process

Shared By: