Docstoc

CECS 6800

Document Sample
CECS 6800 Powered By Docstoc
					                                                                          KuangTing Shiue
                                                                             Summer 2006
                              CECS 6800 Scaling Method
                        Case Study of Natural Language Interface

Background:

       Is there a better way to communicate with computer using natural language

commands than using traditional command/menu interface on application? If a computer

interface can translate our natural language to commands and perform as directed, our life

would be much easier in this modern digital era.

       This case study tries to compare between two different types of language

command interfaces: the traditional command/menu interface (Lotus 1-2-3) and the

natural-language interface (Lotus HAL). The Lotus1-2-3 is a traditional command/menu

interface spreadsheet application, and the Lotus HAL is a program that works with Lotus

1-2-3, which allows user to communicate with the program using limited English

sentences or phrases.

Method:

       The experimental training session took 3 days for both applications conducted by

experienced training people on Lotus 1-2-3 and Lotus HAL. Forty-two subjects were

recruited by a participated employment agency. One group of 22 subjects used Lotus 1-

2-3 and the other group of 20 subjects used Lotus HAL. All of the subjects had little

prior knowledge working with computers, and neither of the groups’ participant had any

experiences with either Lotus 1-2-3 or Lotus HAL application.

       During the 3 days of training sessions, subjects were asked to resolve a series of

different problem sets in the afternoon of the second day, in the morning of the third day,

and the afternoon of third day. Also, in the beginning hour of the last day’s training

session, a general intelligence test – “Wonderlic Personnel Test” were given to all the



                                             1
                                                                      KuangTing Shiue
                                                                         Summer 2006
participants. The performance score was based on the correct answer of those problem

solving practice.

Data Resource:

       Related information and data source were acquired from Rice Virtual lab in

Statistics of Rice University webpage at:

http://www.ruf.rice.edu/~lane/case_studies/interface/index.html


Cond     W          Score
NL       22         6
NL       20         5.66667
NL       21         8
NL       32         7
NL       21         9
NL       27         8.66667
NL       22         7
NL       19         8.66667
NL       17         3.33333
NL       26         5.66667
NL       23         9.33333
NL       21         7.33333
NL       23         9
NL       23         7.33333
NL       19         7.66667
NL       15         0.33333
NL       21         9
C        26         7
C        18         7
C        27         12.33333
C        30         8.33333
C        24         5.33333
C        26         4.33333
C        18         2.33333
C        24         1
C        29         4.66667
C        30         6.66667
C        19         6
C        23         6.33333
C        16         1
C        18         4.33333
C        22         2.66667
C        26         8
C        23         1.33333
C        32         6
C        26         6.33333
C        26         12.66667
C        25         8.33333




                                            2
                                                                                     KuangTing Shiue
                                                                                        Summer 2006
  Descriptive Statistics:

            The first step of analysis was trying to reproduce the fundamental data set on

  SPSS and to confirm the same picture was obtain out of this data set. With the

  descriptive statistics result, the HAL group shows overall higher score of performance

  than 1-2-3 group overall score of performance. Also, the Std. deviation of HAL group is

  narrower than 1-2-3 group. This finding supports that HAL group has a better

  performance on problem solving test. The Boxplots of performance graphic supports the

  same result with a visual graphic outlook.



                                         Descriptive Statistics
                                                                                           Std.
                            N        Range    Minimum Maximum             Mean                     Variance
                                                                                         Deviation
GroupID
                                                                                 Std.
                         Statistic Statistic Statistic   Statistic   Statistic           Statistic   Statistic
                                                                                 Error
           w                    21    16.00      16.00       32.00 24.1905 .96762         4.43417      19.662
           performance          21    11.67       1.00       12.67    5.8095 .69942       3.20515      10.273
Lotus123
           Valid N
                                21
           (listwise)
           w                    17    17.00      15.00       32.00 21.8824 .94668         3.90324      15.235
           performance          17     9.00        .33        9.33    7.0000 .56592       2.33333       5.444
LotusHal
           Valid N
                                17
           (listwise)




                                                    3
                                                                        KuangTing Shiue
                                                                           Summer 2006
         Based on the output of Descriptive Statistic and Boxplots graphics, HAL group

has a slightly lower intelligence test score; yet HAL group has better performance on

problem solving test. This result implies that using natural language interface would help

user to perform better on solving problems.

                              Group: 1-2-3

                                                          Boxplot: Summaries for separated
                                                          Performance and Wonderlic score in two
 30.00                                                    groups.



 20.00




                      1-2-3
                      1-2-3
 10.00




  0.00


                performance                  w




                               Group: NL


                                                 NL

 30.00




 20.00


                                                 NL




 10.00




                      NL
  0.00


                performance                  w




                                                      4
                                                               KuangTing Shiue
                                                                  Summer 2006

              35.00
                                                Boxplot: Compare Wonderlic test score
                                                of two groups.
                                       25


              30.00
w




              25.00




              20.00




                                       37
              15.00


                      1-2-3           NL
                              Group




              14.00
                                                Boxplot: Compare Performance score
                         20
                                                of two groups.
              12.00      3



              10.00
performance




              8.00



              6.00



              4.00



              2.00


                                       37
              0.00


                      1-2-3           NL
                              Group




                                            5
                                                                            KuangTing Shiue
                                                                               Summer 2006
Clustering Method:

       The result of Hierarchical Cluster shows that Group ID and Performance score are

fall into the same cluster, it implies that the difference of interfaces use has more effect

on performance, not the intelligent score. As result, it also support that HAL group had

better performance was because the natural language interfaces help the group perform

better on problem solving task.



 Data
********************************
Group GroupID   W       Performance
1-2-3   .00     26.00   7.00
1-2-3   .00     18.00   7.00
1-2-3   .00     27.00   12.33
1-2-3   .00     30.00   8.33
1-2-3   .00     24.00   5.33
1-2-3   .00     26.00   4.33
1-2-3   .00     18.00   2.33
1-2-3   .00     24.00   1.00
1-2-3   .00     29.00   4.67
1-2-3   .00     30.00   6.67
1-2-3   .00     19.00   6.00
1-2-3   .00     23.00   6.33
1-2-3   .00     16.00   1.00
1-2-3   .00     18.00   4.33
1-2-3   .00     22.00   2.67
1-2-3   .00     26.00   8.00
1-2-3   .00     23.00   1.33
1-2-3   .00     32.00   6.00
1-2-3   .00     26.00   6.33
1-2-3   .00     26.00   12.67
1-2-3   .00     25.00   8.33
NL      1.00    22.00   6.00
NL      1.00    20.00   5.67
NL      1.00    21.00   8.00
NL      1.00    32.00   7.00
NL      1.00    21.00   9.00
NL      1.00    27.00   8.67
NL      1.00    22.00   7.00
NL      1.00    19.00   8.67
NL      1.00    17.00   3.33
NL      1.00    26.00   5.67
NL      1.00    23.00   9.33
NL      1.00    21.00   7.33
NL      1.00    23.00   9.00
NL      1.00    23.00   7.33
NL      1.00    19.00   7.67
NL      1.00    15.00   .33
NL      1.00    21.00   9.00




                                              6
                                                                                                            KuangTing Shiue
                                                                                                               Summer 2006
Multidimensional Method: (PROXSCAL)

                       Running Multidimensional Method on SPSS also shows the performance variable

is closer to the interface variable in both one dimension and two dimension output

graphic. The main finding in this case study is the effect of Group is significant, and all

three different analysis methods suggest that the natural language interface (Lotus HAL)

Leads better performance than the traditional command/menu interface (Louts 1-2-3).

                                                 Common Space




                         GroupID                                                     w




      -0.50                  -0.25        0.00        0.25       0.50         0.75             1.00
                                                  Dimension 1


                                                 Common Space


               0.3
                             GroupID
               0.2
                                                                                                  w
               0.1
 Dimension 2




               0.0

               -0.1

               -0.2

               -0.3

               -0.4
                      -0.6         -0.4    -0.2       0.0       0.2         0.4          0.6          0.8
                                                     Dimension 1




                                                                        7
                                                                                            KuangTing Shiue
                                                                                               Summer 2006
Multidimensional Method: (MDPREF)
MDPREF with data on preferences for four desserts

  # of       # of           # of       Plotted
Subjects    Stimuli        Factors     Factors         Data Form
  38          3              2           2               2


STIMULUS MATRIX
 FACTOR
    1    -0.574        0.790         -0.216
    2    -0.581       -0.207          0.787


               PLOT OF SUBJECTS IN FIRST TWO DIMENSIONS
     +-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
 1.0+                                    |                                    +
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
 0.6+                                    |                                    +
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                            K       -
 0.2+                                    |                            3T      +
    -                                    |                              #     -
    -                                    |                              #     -
    -------------------------------------0-----------------------------#-------
    -                                    |                              #     -
    -                                    |                              #     -
-0.2+                                    |                            #       +
    -                                    |                            /       -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
-0.6+                                    |                                    +
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
-1.0+                                    |                                    +
     +-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
   -1.2 -1.0 -0.8 -0.6 -0.4 -0.2        0.0   0.2   0.4   0.6   0.8   1.0   1.2

       *****IDENTIFICATION KEY FOR PLOTS WITH IDENTIFIED                         POINTS*****

PT #   1    2     3    4    5   6      7      8   9    10    11   12   13   14   15   16   17
CHAR   1    2     3    4    5   6      7      8   9     A     B    C    D    E    F    G    H

PT #   18   19   20   21   22   23    24   25     26   27    28   29   30   31   32   33   34
CHAR    I    J    K    L    M    N     O    P      Q    R     S    T    U    V    W    X    Y

PT #   35   36   37   38   39   40    41   42     43   44    45   46   47   48   49   50
CHAR    Z    +    /    =    *    &     $    @      %    ?     <    (    )    ~    ;    @

POINT NUMBERS ABOVE 50 IDENTIFIED AS              >
MULTIPLE POINTS IDENTIFIED AS #




                                                         8
                                                                                         KuangTing Shiue
                                                                                            Summer 2006

               PLOT OF STIMULUS POINTS IN FIRST TWO DIMENSIONS
     +-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
 1.0+                                    |                                    +
    -                                    |                                    -
    -                                    |                                    -
    -                              3     |                                    -
    -                                    |                                    -
    -                                    |                                    -
 0.6+                                    |                                    +
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
 0.2+                                    |                                    +
    -                                    |                                    -
    -                                    |                                    -
    -------------------------------------0-------------------------------------
    -                                    |                                    -
    -                                    |                                    -
-0.2+                                    |                       2            +
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
-0.6+                   1                |                                    +
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
-1.0+                                    |                                    +
     +-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
   -1.2 -1.0 -0.8 -0.6 -0.4 -0.2        0.0   0.2   0.4   0.6   0.8   1.0   1.2

       *****IDENTIFICATION KEY FOR PLOTS WITH IDENTIFIED                     POINTS*****

PT #   1    2    3    4    5    6    7    8    9    10   11   12   13   14   15    16   17
CHAR   1    2    3    4    5    6    7    8    9     A    B    C    D    E    F     G    H

PT #   18   19   20   21   22   23   24   25   26   27   28   29   30   31    32   33   34
CHAR    I    J    K    L    M    N    O    P    Q    R    S    T    U    V     W    X    Y

PT #   35   36   37   38   39   40   41   42   43   44   45   46   47   48   49    50
CHAR    Z    +    /    =    *    &    $    @    %    ?    <    (    )    ~    ;     @

POINT NUMBERS ABOVE 50 IDENTIFIED AS           >
MULTIPLE POINTS IDENTIFIED AS #




                                                     9
                                                                                KuangTing Shiue
                                                                                   Summer 2006

               PLOT OF STIMULI AND SUBJECTS IN FIRST TWO DIMENSIONS
     +-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
 1.0+                                    |                                    +
    -                                    |                                    -
    -                                    |                                    -
    -                              3     |                                    -
    -                                    |                                    -
    -                                    |                                    -
 0.6+                                    |                                    +
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                            N       -
 0.2+                                    |                            6W      +
    -                                    |                              #     -
    -                                    |                              #     -
    -------------------------------------0-----------------------------#-------
    -                                    |                              #     -
    -                                    |                              #     -
-0.2+                                    |                       2    #       +
    -                                    |                            &       -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
-0.6+                   1                |                                    +
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
    -                                    |                                    -
-1.0+                                    |                                    +
     +-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
   -1.2 -1.0 -0.8 -0.6 -0.4 -0.2        0.0   0.2   0.4   0.6   0.8   1.0   1.2

       *****IDENTIFICATION KEY FOR PLOTS WITH IDENTIFIED            POINTS*****

PT #   1   2   3   4   5   6   7   8   9   10   11   12   13   14    15   16   17
CHAR   1   2   3   4   5   6   7




                                           10

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:2
posted:10/12/2011
language:English
pages:10