An Overview of In Situ Self Report and the by lkn74367


									    An Overview of In Situ Self Report and the MyExperience Tool
           Jon Froehlich1, James Landay1,2                                 Mike Chen2, Sunny Consolvo2, Beverly
                       DUB Group                                                   Harrison2, Ian Smith2
          Computer Science and Engineering1                                         Intel Research Seattle2
               University of Washington                                         1100 NE 45th Street, 6th Floor
                Seattle, WA 98195-2350                                                Seattle, WA 98105
         {jfroehli, landay}                           {mike.y.chen, sunny.consolvo, beverly.harrison,
ABSTRACT                                                                provide richer forms of participant response. The data
We present the MyExperience tool, a software application                collected in the field (including multimedia content) can be
for in situ data collection to support the study of human               wirelessly transferred back to the researchers in near real
behavior and the evaluation of mobile computing                         time.
technologies. MyExperience extends prior work in
computerized self-report and context-aware experience                   The rest of this short paper is organized as follows: first, we
sampling done on PDAs to the mobile phone. The                          provide background on traditional in situ self-report
combination of a small mobile platform paired with                      methods and then show how technology has changed their
context-aware computing provides researchers with an                    implementation. We then introduce MyExperience and
opportunity to capture human behavior in ways previously                discuss its capabilities as an in situ self-report tool for
not possible. This paper serves as an introduction to in situ           naturalistic studies.
self report procedures and provides an overview of                      Traditional In Situ Self-Report
MyExperience.                                                           In the past, in situ self-report has been done via paper-and-
Author Keywords
                                                                        pencil; participants would carry around and fill out small
In situ self-report, experience sampling, ubicomp, mobile               notebooks, typically formatted with predefined questions,
computing, PDA and mobile phone                                         or scantron sheets containing a series of Likert-scale
                                                                        inquiries [1]. Traditionally, three different methodologies of
INTRODUCTION                                                            self-report have been employed: interval-contingent
In situ, self-report procedures such as the diary method and            recording, signal-contingent recording and event-contingent
the experience sampling method (ESM) have been used                     recording [9].
extensively in psychology and HCI to capture data on
participants’ thoughts, feelings, and behaviors as they are             Interval-contingent recording. In this method, participants
experienced [1, 3, 9]. Such procedures have a distinct                  are asked to record their experiences at recurring intervals.
methodological advantage over ex-situ inquiries in that they            These intervals could be situated at temporally meaningful
do not rely on the reconstruction of information from                   times (e.g., at the end of the day or during breaks at work),
memory, but rather involve reporting on experiences as                  or at set times during the day (e.g., every three hours).
they occur, thus minimizing recall bias.                                Event-contingent recording. Rather than being instructed to
We have built a software framework called the                           fill out notebooks at specified intervals, participants are
MyExperience tool to support computerized self-report in                asked to record their experiences as a result of a particular
the field. Although MyExperience runs on both TabletPCs                 event of interest. For example, in a study examining dietary
and PDAs, the tool was specially designed to run on mobile              practices amongst teenagers the participants would be asked
phones. We strongly believe that the mobile phone presents              to record food consumption details whenever they ate. One
an ideal platform for in situ data collection as it is a familiar       disadvantage of this method is that the quality of data is
device to most subjects and has a small, unintrusive form               contingent on the participant’s ability to be cognitively
factor. MyExperience supports a variety of data collection              engaged into his/her own actions so that the “event of
methodologies from diary studies to the experience                      interest” may be recognized and recorded accordingly.
sampling method. Preformatted questionnaires (“self-report              Signal-contingent recording. This method is most
surveys”) can be administered to subjects via the device                commonly associated with ESM. Participants are asked to
according to set parameters by the researcher. These                    report their experiences whenever signaled by the
parameters may, for example, involve regular prompting                  researcher, which may occur randomly or at fixed intervals
intervals or, more interestingly, may be tied to sensor data            or a combination of both. The signals themselves are
in the field. MyExperience also takes advantage of the                  typically triggered via technology. For example, in the past,
multimedia features now common in mobile phones such as                 participants would carry beepers that could be triggered by
audio recording, digital photography and digital video to

the researcher to signal the participant to fill out a               the self-report that captured them, providing richer context
questionnaire.                                                       for analysis.
Computerized In Situ Self-Report                                     A New Method Using Context-Aware Computing
Modern technology has allowed us to extend these self-               In addition to these advantages, computer-based self-report
report procedures to collect data in ways previously not             offers a completely new form of in situ data collection,
possible. With the popular adoption of mobile phones and             called “context-triggered sampling.” This technique,
PDAs, computerized in situ self-report is a natural                  pioneered by MIT’s Context-Aware Experience Sampling
progression from the paper-and-pencil method and offers a            (CAES) toolkit [7] for the PocketPC, uses sensors to infer
host of advantages over its non-digital counterpart.                 context, which then triggers a self-report questionnaire. For
                                                                     example, a research study on the influence of distance
First, participant compliance can be more accurately
                                                                     running on emotional state could use sensors (e.g., an
assessed [1]. Previously, it was not possible to determine
                                                                     accelerometer) to determine when the participant is running
the validity of in situ self-reports—participants could fake
                                                                     for at least a 20 minute interval and then prompt them with
or change entries post-hoc reducing the overall quality of
                                                                     a questionnaire a few minutes after they stop.
data collected. Computerized self-report allows the
researcher to, for example, have a better idea of exactly            Context-triggered sampling is most closely related to event-
when a self-report was completed, how long it took to fill           contingent sampling, where the participant is asked to fill
out, and if any of the answers changed before completion.            out a self-report as the result of some event of interest. The
                                                                     key difference, however, is that event-contingent sampling
Second, wireless connectivity provides the researcher with
                                                                     relies on the participant to be cognitively engaged into his
real-time knowledge of participant behavior and allows the
                                                                     or her own activities in order to recognize when a
researcher to access collected data in an ongoing fashion.
                                                                     recordable event occurs [9]. Context-triggered sampling, in
This provides a multitude of unique benefits: researchers
                                                                     contrast, uses sensor data to automatically infer when an
can prepare more knowledgeable and concise ex situ
                                                                     event of interest has occurred and prompts the user
inquiries (e.g. individualized mid-point interviews based on
                                                                     accordingly—thereby shifting the cognitive burden to the
in-progress data), preliminary data analysis can begin
                                                                     technology. This new technique has several advantages:
almost immediately after the study begins, and researchers
have the ability to identify and troubleshoot problems as              1. Sampling questionnaires become more targeted,
they occur during the course of the study. In addition, new               occurring only during events of interest, which reduces
data can be downloaded onto the participants’ devices in                  participant disruption when compared with other
real time. For example, researchers could remotely load a                 sampling methods.
new set of questions onto the devices or trigger a
                                                                       2. Context data can be continuously saved, allowing the
questionnaire dynamically.
                                                                          researcher to validate participant responses and,
Third, auxiliary inputs like cell-phone cameras or                        perhaps, to uncover new behavioral patterns not
microphones can be used to augment text answers. This is                  initially considered;
particularly important given that most electronic devices do
                                                                       3. Sensor data paired with participant responses can be
not offer the same ease of writing as pencil-and-paper.
                                                                          used to test the effectiveness and accuracy of new
Moreover, media captured by the participant can be used to
                                                                          UbiComp sensor technologies and to evaluate
gain a richer understanding of his or her experience and can
                                                                          UbiComp prototype designs in the field.
be captured during the normal course of a self-report
questionnaire.                                                       Though very promising, context-triggered sampling is still
                                                                     subject to some of the methodological issues that affect
Fourth, computerized self-reports can be much more
                                                                     interval-contingent and signal-contingent sampling. For
sophisticated in their presentation of questions. For
                                                                     example, it is probably not the most appropriate technique
example, self-report questionnaires can contain conditional
                                                                     for studying events related to when a participant cannot do
questions (i.e., ask question C only if the answer to question
                                                                     something because of a limitation. In this case, an event-
B was false), multi-media questions (e.g., video based
                                                                     contingent based method (i.e., a diary study) or a hybrid
questions), specific question frequencies (i.e., only ask
                                                                     between the two may be more appropriate because it would
question E once per unique answer to question D), and
                                                                     give the participant control of when to record an event.
question/answer order randomization (i.e., the presentation
of answer data is randomized from questionnaire to                   THE MYEXPERIENCE TOOL
questionnaire to reduce response bias [1].                           The MyExperience tool developed jointly by the University
                                                                     of Washington (UW) and Intel Research, Seattle provides
Finally, data collected via computerized self-report is              all of the benefits of computerized self-report and context-
intrinsically structured for computer-based analysis. This           triggered sampling highlighted above on a familiar,
reduces the data management burden on the researcher as              unintrusive platform—the mobile phone. See Figure 1.
well as the potential for coding errors. Non-textual data like       Both the UW and Intel have a precedence of building tools
audio recordings and pictures are automatically bound to             to study behavior in situ [4, 5]. MyExperience is the

culmination of these past effo and builds on the research
c                            orts         s                           can be used to provid sampling co
                                                                                          de                        hout the
                                                                                                      overage through
e             ers            lds
efforts of othe in the fiel of psychol                 ous
                                          logy, ubiquito              day.
computing, hu
c                            er
               uman-compute interaction, and medic     cal
science [1, 2, 5, 8]. MyE
s                           Experience is meant to be a               Multip questionnair can be load on the devi so that
                                                                            ple             res          ded             ice
comprehensive self-report to and is bu to support a
c             e              ool         uilt                         differe triggers can administer dif
                                                                            ent            n                             questions.
                                                                                                         fferent sets of q
v             hodologies and study designs.
variety of meth                           .                           For exa              udy
                                                                             ample, in a stu of travel beehavior one might use a
                                                                            on-aware senso like GPS to t
                                                                      locatio              or            trigger a “workkplace
A the time of this writing, MyExperience h been used in
At                          M             has
                                                                            onnaire” when the participant is observed to be at
                                                                      questio                            t               o
o study, whi explored the relationship between pla
one            ich          t             p             ace
                                                                      work a a “home qu                  when the partic
                                                                                           uestionnaire” w              cipant is
v                                        M
visit behavior and preference [6]. A GSM motion sens    sor
                                                                      observ to be at hom  me.
was used to trigger self-re
w                                         nnaires when a
                             eport question
subject became stationary fo 10 minutes. A random tim
s             e             or                          me
sensor was also used to augment the c
s                           a             context-triggered                ions
q                                                                     Curren ntly, MyExpe   erience suppo  orts three c classes of
                                                                             ons:           rm,
                                                                      questio closed-for open-form and multimed capture.dia
                                                                      Closed d-form questi                 o            at
                                                                                            ions refer to those tha provide
                                                                      predete                nse           .g.,         le,
                                                                             ermined respon options (e. Likert-scal Yes/No,
                                                                             Open-form qu
                                                                      etc.). O             uestions, on th other hand, allow the
                                                                      particip                            wn             See
                                                                              pant to freely enter their ow responses. S Figure
                                                                      2. Giv ven the cumb                                on
                                                                                            bersome input capabilities o mobile
                                                                      devices, open-form q                d             d
                                                                                             questions tend to be limited to simple
                                                                      text en (e.g., “plea enter the nu
                                                                            ntry            ase                          le
                                                                                                          umber of peopl with you
                                                                      right n               ons            re
                                                                             now”). Questio that requir a more sop      phisticated,
                                                                      prose-l                             er
                                                                             like response may be bette suited to us     sing audio
                                                                             e              ext
                                                                      capture rather than te entry.

                e              t               ovox SMT 5600
    Figure 1: The MyExperience tool on the Audio

Self-Report Qu
S            uestionnaire
             nce           t             re
A MyExperien self-report questionnair can range in
              ng                         ally
length dependin on response but is typica between on  ne
to twenty ques
 o                         estionnaires ar preconfigured
             stions. The que              re
by            her                        e
b the research to trigger in any of the following fo our
  1. Calendar-triggered sampl                pecific
                               ling permits sp
     scheduling of questionna times at fle
               g              aire           exible recurrencce
                                                                                              -completion based open-form ques
                                                                      Figure 2: (left) An auto-               d              stion. (right)
     patterns (e.g., 2PM each weekday, 11A every                                  An example of two closed-form question types.
  2. Context-tri              ing
                iggered sampli associates a   automatically                                 r
                                                                      Audio recordings (or “voice memo                 n
                                                                                                          os”) have been shown to
     detectable events to questionnaires (e.g., subject heartt        be a f                ive
                                                                            fitting alternati to text en ntry both in reecording a
     rate exceed 150 beats/m
                ded            minute)                                       pant’s free na
                                                                      particip                           nse           n
                                                                                           arrative respon [2] and in situations
  3. Random-tr  riggered sampling provides a mechanism to   o         where the subjects thhemselves are not comfortable with the
               estionnaires at random interva
     trigger que               r              als.                    existin mobile text entry methods (e.g., T9, Gra
                                                                            ng                           s              affiti, etc.).
  4. Manually-t               pling allows the participants
                 triggered samp               e                       Beyond audio, MyE    Experience also allows parti icipants to
     themselves to trigger que
                s             estionnaires.                                  e
                                                                      capture digital pho                nd             ll
                                                                                            otography an video. Al of the
                                                                      multim               s
                                                                            media features are seamles                 d
                                                                                                         ssly integrated into the
T triggering m
The              methods can be used alone or in                            ys                            ge
                                                                      survey themselves, making usag straightforw      ward. This
combination. For example, a diary study ma only need
c                                           ay                        also eases data ana  alysis as the multimedia re  esponse is
manually-trigge  ered sampling, which allows participants to
                              ,                            o          situated in the survey data. Multim
                                                                                            y            media content c also be
self-report at th discretion. In the past, we have used
s               heir                         e                        used in asking questi
                                                                             n              ions on the dev            mple, audio
                                                                                                          vice. For exam
r               red           nd
random-trigger sampling an context-trig     ggered sampling  g        recordiings of quest  tion text may play back when the
in combination to ensure that self-report que
 n                                                          ill
                                             estionnaires sti                on            d
                                                                      questio is displayed on the device. .
get             ed            o              e.
g administere in the case of sensor failure Both
c                             a
                ered sampling and random-triggered samplin  ng

Question Flow                                                        perform advanced sensor calculations and context-triggered
Questions can be asked in a predefined order or                      sampling. With the available multimedia features,
randomized. MyExperience also allows question branching,             MyExperience is not simply a quantitative data collection
which allows the questionnaire to react dynamically to               tool but also one that can be used qualitatively in photo-
participant response. Six “relational operators” are used to         elicitation and related studies.
establish conditions for branching (==, <. >, <=, >=, !=).
For example, ask follow-up question D if the answer to               Please contact         for   more
question C was greater than four. Researchers can also loop          information.
questions around previous responses. For example, in a               REFERENCES
study of chronic pain, an initial question may ask “Please           1. Barrett, L.F. and Barrett, D.J. (2001), An Introduction to
checkmark each area in which you are experiencing pain.”                Computerized Experience Sampling in Psychology. Social
A follow-up question may loop on each are marked and                    Science Computer Review., Vol. 19, No. 2, Summer 2001, pp.
inquire about the level of pain experienced. Finally,                   175-185.
questions can be assigned probabilities, which effect how            2. Carter, S., Mankoff, J. (2005). When Participants do the
often they are asked.                                                   Capturing: The Role of Media in Diary Studies. In Proc of
                                                                        CHI’05. April 2- 7, Portland, OR
Wireless Connectivity
                                                                     3. Consolvo, S., Smith, I., Matthews, T., LaMarca, A., et. al
A concern with many naturalistic data capture studies is                (2005). Location Disclosure to Social Relations: Why, When
sustained subject participation. The MyExperience tool                  & What People Want to Share. In Proc of CHI’05. Portland,
offers the ability to monitor participant behavior in real-             OR.
time via wireless uploads. In this way, researchers are better
                                                                     4. Consolvo, S., Walker, M. (2003). Using the Experience
able to understand the engagement of their participants and             Sampling Method to Evaluate Ubicomp Applications. IEEE
make proper adjustments as needed (e.g. a friendly                      Pervasive    Mobile and Ubiq. Systems: The Human
reminder e-mail or phone call). Additionally, real time                 Experience, pp. 24-31.
access to study data provides a wealth of opportunity for            5. iESP (Intel’s Experience Sampling Program for the Palm):
researchers to create more targeted ex situ inquiry – from    
better, more knowledgeable interview preparation to the
                                                                     6. Froehlich, J., Chen, M., Smith, I., & Potter, F. (2006) "Voting
creation of other contextualized inquiries like online
                                                                        With Your Feet: An Investigative Study of the Relationship
surveys or paper questionnaires. Finally, wireless                      Between Place Visit Behavior and Preference" (in submission)
connectivity allows the researcher to detect errors and
technology failures early before they result in high data            7. Intille, S. S., Rondoni, J., Kukla, C., Anacona, I., and Bao, L.
                                                                        (2003), A context-aware experience sampling tool. In Proc of
loss.                                                                   CHI '03 New York, NY: ACM Press, 2003, pp. 972-973.
CONCLUSION                                                           8. Stone, A., Turkkan, J., Bachrach, C., Jobe, J., Kurtzman, H.,
We believe that MyExperience offers researchers in a                    and Cain, V. (eds.), The Science of Self Report: Implications
breadth of fields the ability to conduct field studies that             for Research and Practice. Mahwah, NY: Lawrence Erlbaum
were simply not possible before. The mobile phone is                    Associates, 1999
quickly becoming the most pervasive technology in the                9. Wheeler, L., & Reis, H. (1991), “Self-recording of Everyday
world and its computational power is only increasing. We                Life Events: Origins, Types, and Uses,” Journal of Personality,
leverage the popularity of mobile phones to provide an easy             59, pp. 339-354.
self-report interface to subjects and the computation to


To top