Docstoc

kelly-results _08-34_

Document Sample
kelly-results _08-34_ Powered By Docstoc
					                                                                                                   112


                                             Chapter 5

                                               Results

        In this chapter data is reported that includes characteristics of subjects, such as

demographic data and previous computer and searching experiences, and characteristics

of the data collected, such as the size of the data collected from the various pieces of

logging software and the number of documents viewed and evaluated by each subject.

The tasks and topics identified by each subject are then presented, along with the

descriptive data describing the endurance, frequency and stage for each task, and the

persistence and familiarity for each topic. Subjects’ comments from the Exit Interviews

about the processes of creating task and topic classes and classifying documents follow.

A description of document usefulness and confidence is then presented and this is

followed by a description of the relationship between display time, retention and

usefulness.    Next, the relationship between display time, retention and information

seeking context is presented. Finally, the relationship between behaviors, information

seeking context and usefulness is presented.



5.1     Characteristics of the Subjects

        Seven subjects were recruited to participate in this study because pilot testing

indicated the possibility of a high dropout rate. All seven subjects completed the study in

its entirety. Of the seven subjects, five were male and two were female. All subjects

were Ph.D. students at Rutgers University, from the following seven departments:

history, geography, political science, electrical engineering, communication, mechanical

engineering and comparative literature. All subjects were between the ages of 26 and 37.



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   113

Table 5.1 provides a description of the seven subjects.                 For the remainder of this

dissertation, subjects will be referred to according to their specific number as presented in

the table.

Table 5.1

Description of Subjects

               Subject                Ph.D. Major                       Sex          Age
                 1                       History                         M           33
                 2                     Geography                         F           37
                 3                  Political Science                    M           26
                 4               Electrical Engineering                  M           30
                 5                  Communication                        F           32
                 6               Mechanical Engineering                  M           32
                 7               Comparative Literature                  M           29


        Subjects’ computer and search experience was assessed by the Entry

Questionnaire. Subjects were asked to indicate their computer and World Wide Web

experience on a seven point scale, where 1 = none; 4 = some; 7 = a great deal. These

results are displayed below in Figure 5.1a. All subjects claimed to have more than some

computer and World Wide Web experience and three subjects claimed to have a great

deal. Subjects were also asked to indicate their level of expertise with computers on a

seven point scale, where 1 = novice and 7 = expert. These results are displayed in Figure

5.1b. Although no subject indicated that they were an expert, most subjects considered

themselves beyond mid-level expertise.




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                                                                 114

                                       7                                                                                                   7



                                       6                                                                                                   6



                                       5                                                                                                   5




                                                                                                                Expertise with Computers
                                       4                                                                                                   4



                                       3                                                                                                   3

                                                                                           Experience with...
                                       2                                                                                                   2
                                                                                                Computers
        Rating




                                       1                                                        WWW                                        1
                                               1       2       3   4   5       6       7                                                             1   2   3   4   5   6   7


                                           Subject                                                                                             Subject

                                           (a)                                                                                                 (b)

Figure 5.1 Subjects’ experience using computers (a) and computer expertise (b)

              To assess search experience, subjects were asked to indicate the number of years

that they had been doing online searching. These results are displayed in Figure 5.2a.

Overall, subjects had a mean of 6.4 years of searching experience. All subjects had at

least five years of searching experience and one subject had nine years. Finally, subjects

were asked to indicate their level of expertise with online searching on a seven point

scale where 1 = novice and 7 = expert. These results are displayed in Figure 5.6b. All

seven subjects rated themselves beyond mid-level of expertise and two subjects indicated

that they considered themselves to be experts.

                                       10                                                                                                  7

                                       9
                                                                                                                                           6
                                       8


                                       7                                                                                                   5
        Years Doing Online Searching




                                       6
                                                                                                                                           4
                                                                                                                Expertise with Searching




                                       5

                                       4                                                                                                   3


                                       3
                                                                                                                                           2
                                       2

                                       1                                                                                                   1
                                                   1       2       3       4       5        6       7                                                1   2   3   4   5   6   7


                                            Subject                                                                                            Subject

                                            (a)                                                                                                (b)

Figure 5.2 Number of years doing online searching (a) and searching expertise (b)



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   115


5.2      Overview of the Data

         The quantity of data collected during the fourteen-week period of the study is

shown in Table 5.2. The first two items correspond to the two types of logging software

used during the study. WinWhatWhere collected data from the client and the proxy

logger collected data at the server.           WinWhatWhere recorded information such as

applications used, URLs visited, start, finish and elapsed times for interactions and all

keystrokes. The proxy logger saved a copy of each page requested by subjects. A more

detailed description of the data logged by each piece of software can be found in Section

4.3.2.    It should be noted that these values are approximate and in many cases

incomplete. The incompleteness resulted from the WinWhatWhere logger spontaneously

turning itself off during the study, which resulted in losing several days of data. As

evident from Table 5.2, this was a problem, in particular, for Subject 1, where the

proportion of data logged by the WinWhatWhere logger to that logged at the proxy is not

the same as it is for Subject 6, who requested a similar number of documents.

Table 5.2

Description of the Quantity of Data Collected

                                                          Subject
                         1           2           3            4           5           6           7
 WinWhatWhere         2.6 MB      6.8 MB      3.9 MB      2.0 MB      1.5 MB      21.7 MB      4.9 MB
  Proxy Logger        1.7 GB      83 MB       39 MB        42 MB       48 MB       2.9 GB      2.1 GB
 Docs Requested       15,499       5,319       3,157       3,205       3,404       14,586      11,657
   Docs Shown           870         802         384         353         200         1,328      1, 160
 Docs Evaluated         843         771         381         351         193         1,181       1,125
  (% Requested)        (5%)        (14%)       (12%)       (11%)        (6%)        (8%)       (10%)




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   116

        Documents requested (Docs Requested) is the total number of URL requests sent

by the subject. The contents of the files that resulted from these requests were recorded

at the proxy. It should be noted that this number does not correspond to unique pages

viewed. For instance, for pages which had an advertisement associated with them, two or

more items would be logged at the proxy, one for the page and one for the advertisement.

Images that were part of a page were not logged separately. For frames pages, which

might be made up of three files, three items would be logged at the proxy. Documents

shown (Docs Shown) is the number of documents shown to each subject for evaluation.

Recall that these were selected based on criteria described in Section 4.3.8 of the Method

chapter. Documents evaluated (Docs Evaluated) represents the total number of pages

that the subject classified and rated during the course of the study. This number is

presented in Table 5.2 as a raw number and as a percentage of the total number of

documents requested. Reasons that documents were shown to subjects but not rated were

because the subject did not remember viewing the document, or with Subject 6, the

subject allowed someone else to view documents on their laptop.

        The proportion of documents evaluated to documents requested is low, and

somewhat constant, for subjects. Most subjects evaluated approximately 11% of the

pages that they viewed. The exceptions are Subject 1, who looked at large amount of

pornography and Subject 5 whose primary online activity was email.                           The low

percentages for all subjects are primarily a result of the content-based classification

scheme that was used to identify candidate documents for evaluation. The number of

documents requested represents all requests, including those made for email messages

and advertisements. Further differences between the number of documents evaluated



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   117

between subjects is evident in Table 5.2.              The subject who evaluated the fewest

documents evaluated 193, while the subject who evaluated the most evaluated 1,181.

These differences indicate that a user modeling system might not be equally as useful for

all people, since some people are heavier users than others, and that such a system needs

some way to distinguish between candidate documents for modeling and noise, such as

advertisements and email, unless the goal is to model these things.



5.3     Tasks

        Subjects identified a range of tasks during the fourteen weeks that they

participated in this study. The tasks identified by each subject are displayed in Tables 5.3

to 5.9. Next to each task is a number that is used in the remainder of this dissertation to

refer to the specific task. Tasks are presented in the chronological order in which

subjects identified them; thus, subjects identified tasks with smaller numbers before tasks

with larger numbers. Tasks were identified with the initial Task Questionnaire and with

subsequent Update Questionnaires, although subjects added the majority of tasks during

their document evaluation activities.

Table 5.3

Subject 1’s Tasks

  1. Researching            3. Read News               5. Preparing Course
     Dissertation                                         materials
  2. Shopping               4. Movie Reviews and       6. Entertainment
                               Schedules




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                     118

Table 5.4

Subject 2’s Tasks

 1. Get insurance           4. Academic Research      7. Hobbies               10. Get Information
    quotes
 2. Get course materials    5. Shopping               8. General Interests     11. Check Email
 3. Read the news           6. Check weather          9. Teaching
    online



Table 5.5

Subject 3’s Tasks

 1. Studying Quals          6. Research Paper - IPR 11. Weather                16. Pet Search
                                                                                   Assistance
 2. Scheduling              7. News                  12. Amherst Alumni        17. Computer
    Resources                                            Activities                Maintenance
 3. General Knowledge       8. Shopping              13. Political Activism    18. Student Government
                                                                                   Work
 4. Entertainment           9. Person Search         14. Funding               19. Research - General

 5. Research Paper -       10. Translation           15. Teaching Stuff
    Interest Groups



Table 5.6

Subject 4’s Tasks

 1. Searching and           8. Searching Health      15. Searching for         22. To Know Holidays
    Browsing for               Information               Employment
    Computer Interests
 2. Searching for           9. Searching Auto        16. Install Software      23. Check Rutgers
    Project - Brain            Repair Information                                  Network Utilization
    Images
 3. Online Shopping        10. Browsing Friends'     17. Find Books in         24. Learn Python
                               Homepages                 Rutgers Library           Language
 4. Viewing News           11. Checking Email        18. Find Companies        25. Learn to Use SIP
                               Online                    Information
 5. Writing an             12. Language Studying     19. Check What's on
    Academic Paper                                       TV
 6. Checking the           13. Checking University   20. Check my own
    Weather                    Webpages                  Homepage
 7. Looking up             14. Sports Searching      21. Check my lab
    Directions                                           Webpage




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                     119

Table 5.7

Subject 5’s Tasks

 1. Travel                 4. Staying in Touch        7. Legal                  10. Teaching
                              with People                Trouble/Conflict           Instruction
 2. Applying for           5. Shopping for            8. Weather                11. Additional
    Fellowships, Grants       Material Possessions                                  Teaching Gigs
    and Awards
 3. Submitting Papers      6. Writing Dissertation    9. Development as a       12. Housing Options
    for conferences                                      Scholar



Table 5.8

Subject 6’s Tasks

 1.   Dissertation         7. Household Method       13.   Research            19. Recreation
 2.   Publication          8. Shopping               14.   Database Systems    20. English Learning
 3.   News                 9. Booking Tickets        15.   Compiler            21. Registration
 4.   Reading             10. Music -                16.   Change new I20
                              Downloading
 5. Job Searching         11. Download Software      17. Look up Directory
 6. Registration          12. Project in Operating   18. Review Maps
                              Systems                    Online



Table 5.9

Subject 7’s Tasks

 1. Selling on Ebay       10. Purchase Mandolin      19. Get Address [1]       28. Check on Rutgers
                                                                                   Parking
 2. Shopping for Books    11. Research for Exit 9    20. Consult Teaching      29. Check Transcripts
    for Oral Exam                                        Resources
 3. Writing reviews for   12. Record Music           21. Follow Conference     30. Get Address [2]
    Amazon.com                                           Proceedings
 4. Printing Chords and   13. Check News             22. Compile Reading       31. Research Napster
    Tablatures                                           Packet
 5. Setting up Personal   14. Shopping               23. Check on Jobs         32. Check Train
    Website                                                                        Schedule
 6. Running Fantasy       15. Correct Greek          24. Locate a Friend       33. Purchase Concert
    Soccer League             Homework                                             Tickets
 7. Research for          16. Purchase Guitar        25. Find Housing
    Dissertation
 8. Download Updates      17. Check on Flight        26. Shop for Profs for
    for Digital               Arrival                    Exam
    Recording Studio
 9. Reviewing for         18. Purchase Theater       27. Rent Movies
    Journal                   Tickets




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   120

        As is evident from the tables, the numbers and kinds of tasks that subjects

identified varied considerably. For instance, Subject 1 identified a low number of very

broadly defined tasks, while Subject 4 identified a high number of more specifically

defined tasks. Some tasks, such as reading the news, shopping, travel and working on

dissertation, were identified by almost all subjects, while others, like “learn python

language,” “get insurance quotes,” and “political activism” were only identified by a

single subject.     Many subjects identified a generic task class; Subject 2 identified

“general interests” and Subject 3 identified two generic categories of tasks: “general

knowledge” and “research – general.” Many of the labels that subjects created for their

tasks are difficult to understand without further elaboration or context.

        Subjects provided further clarification of their tasks during individual Exit

Interviews, which were conducted at the end of the study. Of interest, is that three

subjects indicated that their tasks of “entertainment” were used to describe documents

that provided them with entertainment while they were online, not those that assisted

them with planning entertainment, like movie reviews. The entertainment task was

described as “entertaining oneself with what is online” and “being entertained by

whatever is on the screen.” Subjects further elaborated on the tasks of general interests or

general knowledge, which were described as “random stuff; people send me lots of little

pages” and “just a kind of curiosity topic.”              Many subjects distinguished between

shopping and conducting product research, “I was doing general research on different

kinds that are available. Shopping was looking for a page to actually buy it.” One

subject noted that many of his tasks were monitoring tasks, where he would visit a web

page to see if information had changed.



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   121

        Another interesting distinction that some subjects made were different instances

of the same task. While most subjects considered a task like “review maps online” as a

single task, occurring perhaps multiple times depending upon specific geographic

location of interest, Subject 7 made finer distinctions between tasks, as can be seen in

“Get address,” which occurred as two separate tasks, each corresponding to a different

address. Further, Subject 7 identified “purchase theater tickets” and “purchase concert

tickets” as two separate tasks, although it is possible to imagine these as a single task,

“purchase tickets,” or as “shopping.” In the Exit Interview, Subject 7 clarified this

distinction, “I guess there are two distinctions that I would make here. One would be the

differences between ongoing tasks which shopping would be and single instant tasks like

‘purchase theater tickets’ which I only did once and ‘purchase concert tickets’ which I

looked into once but didn’t even actually purchase.”

        It was also the case that subjects used the specific topic of the task to distinguish

between two tasks. For instance, Subject 3 identified “Research Paper – General Interest

Groups” and “Research Paper – IPR” as two separate tasks; their subject matter is the

only visible distinction between the two.              This example further demonstrates the

difficulty subjects sometimes had with distinguishing between tasks and topics. Subject

5 described some difficulties with her travel task, “it seems that the task has multiple

levels associated with it, so that it was hard for me to distinguish, okay I completed this

topic, but I didn’t complete the task, because the task was much broader than that single

topic.” The variation in the ways that subjects identified tasks is no surprise considering

that this was a totally novel and subjective technique for understanding subjects’ online

activities and instructions for doing so were loosely defined.



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   122


5.3.1 Endurance

        Subjects were asked to characterize each task that they identified according to

three attributes: endurance, frequency and stage. This section reports on subjects’

endurance ratings. Frequency and stage ratings are presented in subsequent sections.

Subjects were asked to make these characterizations for each week of the study that the

task was active (i.e. not completed). Endurance represented the length of time that

subjects expected to be working on the task. This scale ranged from “one day” to

“several years.” Endurance ratings for each task are displayed in Figure 5.3.

        To better understand Figure 5.3 and all subsequent figures displaying task and

topic characteristics, several things should be noted.              With respect to the specific

endurance ratings, only those ratings that the subject ever assigned to any task are

displayed in the “Endurance” legend. Along the axis displaying “Week,” the values of –2

and 16 appear only to mark the axis; ratings occurred during Week 0, which was the start

of the study, through Week 14, which was the completion of the study. Spring break

occurred during Week 6, so no ratings appear for this week. Further information that is

available from this set of figures includes when the task was first identified by subjects

and how long the task remained active.              For instance, Task 11 for Subject 2 was

identified during Week 10 of the study, while Task 1 was first active for a single week of

the study, but then reactivated during Week 5. Task 2 was completed during Week 11.




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                                                           123


                         Subject 1                                                                             Subject 2
                7                                                                                    12

                                                                                                     11
                6
                                                                                                     10

                                                                                                     9
                5
                                                                                                     8

                4                                                                                    7

                                                                                                     6
                3                                                                                    5
                                                                                                                                                        Endurance
                                                                                                     4
                2                                                                                                                                         several years
  Task Number




                                                                                       Task Number
                                                                                                     3
                                                                    Endurance                                                                             one year
                                                                                                     2
                1
                                                                      several years                                                                       several months
                                                                                                     1
                0                                                     several months                 0                                                    several weeks
                    -2     0    2   4   6   8   10   12   14   16                                         -2    0   2   4   6   8   10   12   14   16


                     Week                                                                                  Week




                         Subject 3                                                                             Subject 4
                20                                                                                   26
                19                                                                                   25
                                                                                                     24
                18                                                                                   23
                17                                                                                   22
                16                                                                                   21
                                                                                                     20
                15
                                                                                                     19
                14                                                                                   18
                13                                                                                   17
                12                                                                                   16
                                                                                                     15
                11                                                                                   14
                10                                                                                   13
                 9
                                                                    Endurance                        12
                                                                                                     11
                 8                                                                                   10                                                 Endurance
                                                                      several years
                 7                                                                                    9
                 6                                                                                    8
                                                                      several months                  7                                                   several years
  Task Number




                                                                                       Task Number




                 5
                                                                                                      6
                 4                                                    one month                       5                                                   one year
                 3                                                                                    4
                 2                                                    one week                        3                                                   several months
                                                                                                      2
                 1                                                                                    1
                 0                                                    one day                         0                                                   one month
                     -2     0   2   4   6   8   10   12   14   16                                         -2    0   2   4   6   8   10   12   14   16


                         Week                                                                              Week




                         Subject 5                                                                             Subject 6
                13                                                                                   22
                                                                                                     21
                12                                                                                   20
                11                                                                                   19
                                                                                                     18
                10                                                                                   17
                                                                                                     16
                9                                                                                    15
                                                                    Endurance                        14
                                                                                                                                                        Endurance
                8
                                                                                                     13
                                                                      several years                  12                                                   several years
                7
                                                                                                     11
                6                                                     one year                       10                                                   one year
                                                                                                      9
                5                                                     several months                                                                      several months
                                                                                                      8
                4                                                                                     7
                                                                      one month                       6                                                   one month
  Task Number




                                                                                       Task Number




                3                                                                                     5
                                                                      several weeks                   4                                                   several weeks
                2                                                                                     3
                                                                      one week                        2                                                   one week
                1
                                                                                                      1
                0                                                     several days                    0                                                   several days
                     -2     0   2   4   6   8   10   12   14   16                                         -2    0   2   4   6   8   10   12   14   16


                         Week                                                                              Week




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                   124


                                                       Subject 7
                                             34
                                             32
                                             30
                                             28
                                             26
                                             24
                                             22                                                 Endurance
                                             20
                                                                                                  several years
                                             18
                                             16                                                   one year
                                             14
                                                                                                  several months
                                             12
                                             10                                                   one month




                               Task Number
                                             8
                                                                                                  several weeks
                                             6
                                             4                                                    one week
                                             2
                                             0                                                    one day
                                                  -2    0   2   4   6   8   10   12   14   16


                                                   Week




Figure 5.3 Weekly task endurance

        From Figure 5.3 it can be observed that task endurance did not change all that

much during the study. Some tasks, such as “researching dissertation,” “job searching,”

and “funding” were on-going tasks whose boundaries for initiation and completion

extended beyond the time-frame of this study. Other tasks, such as “shopping” and

“check news” were of indefinite interest.                                    Only a few tasks, such as “pet search

assistance,” “check transcripts,” and “change new I-20” lasted for one week or less.

        It is interesting to note that the length of time that subjects expected to be working

on tasks did not necessarily correspond to the length of time that subjects actually worked

on tasks. For instance, Subject 6’s Tasks 2 and 4, were first characterized as having an

endurance of “several weeks.” For the next three weeks of the study, these tasks were

characterized as having an endurance of “one week.”                                                          In another case, Subject 5

indicated that the endurance of Task 7, “legal trouble/conflict” was “several days” for

three consecutive weeks of the study, for the fourth week indicated it was “several

weeks,” and for the final three weeks indicated it was “one month.” Then the subject

suddenly completed the task! This demonstrated discrepancy between subjects’ estimate

of endurance and the actual endurance of the task occurred in many different cases. It


Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   125

seemed that subjects anticipated finishing specific tasks, but for various reasons did not.

Comments from the Exit Interview support this observation, “I’m in the mindset that I

might be doing this a lot because of I’m thinking about having just done it. In reality,

that’s often not the case,” “so what each of these represent is a mistaken impression of

how long I thought I was gonna be interested in the task,” and “my own expectations of

how long I was gonna be working on a certain task were often inaccurate.”

        The results of the Exit Interviews further indicated that subjects had few problems

understanding and using the endurance scale. One problem that several subjects noted

was rating the endurance of the general or specific instance of the task. For instance,

teaching a course was identified by several subjects. Some subjects referred to teaching

as something that they would do for numerous years, while others viewed it within the

context of whatever particular course they were teaching during the semester. Subjects

acknowledged that most of their tasks had high endurance and tasks with shorter

endurance often were associated with a particular deadline, “so that was pretty discrete,

the exams had a date and I knew when it would end.”

        Although endurance did not change that much, subjects identified several reasons

why it changed, including the immediacy of the task (“it was actually grounded in much

more immediate activities that needed to be taken care of”); new knowledge (“but then I

found out there was [a new fantasy soccer league] and so the number went back up”); and

a change in interest level (“it was easier than I thought it was going to be, and I thought

‘well this is something that I will probably do for the rest of my life’”).




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   126


5.3.2   Frequency

        For each task identified, subjects were asked to indicate the frequency with which

they expected to conduct online activities related to that task. The eight-point frequency

scale ranged from “once or twice a day” to “yearly.” Recall that the values for this

variable were collapsed into four points because of confusion over the labels. The new

labels were daily (1), weekly (2), monthly (3) and yearly (4). Frequency ratings for each

task are displayed in Figure 5.4. All subjects used a range of frequency scores, except for

Subject 1 who used two scores. Frequency ratings for each task did not change that often

for Subjects 1, 2, and 3. However, for Subjects 4, 5, 6, and 7, the frequency for many

tasks changed multiple times during the course of the study. For instance, Subject 5

assigned two different frequencies to Task 5, “writing an academic paper,” which

alternated between “weekly” and “monthly.” Subject 7 assigned a number of different

frequencies to Task 1, “selling on Ebay,” which likely corresponded to the availability of

items to sell, interest in purchasing an item and time constraints. Changes in frequency

across weeks indicated subjects’ perception of how often they would conduct online

activities related to the specific task. It is likely that this estimate is influenced by a

number of factors, including the salience of other tasks and the occurrence of particular

life events (e.g. vacations, holidays).




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                                             127


                         Subject 1                                                                      Subject 2
                7                                                                             12

                                                                                              11
                6
                                                                                              10

                                                                                              9
                5
                                                                                              8

                4                                                                             7

                                                                                              6
                3                                                                             5
                                                                                                                                                 Frequency
                                                                                              4
                2                                                                                                                                  Yearly
  Task Number




                                                                                Task Number
                                                                                              3
                                                                    Frequency                                                                      Monthly
                                                                                              2
                1
                                                                      Weekly                                                                       Weekly
                                                                                              1
                0                                                     Daily                   0                                                    Daily
                    -2      0   2   4   6   8   10   12   14   16                                  -2    0   2   4   6   8   10   12   14   16


                     Week                                                                           Week




                          Subject 3                                                                     Subject 4
                20                                                                            26
                19                                                                            25
                                                                                              24
                18                                                                            23
                17                                                                            22
                16                                                                            21
                                                                                              20
                15
                                                                                              19
                14                                                                            18
                13                                                                            17
                12                                                                            16
                                                                                              15
                11                                                                            14
                10                                                                            13
                 9                                                                            12
                                                                                              11
                 8                                                  Frequency                 10                                                 Frequency
                 7                                                                             9
                 6                                                                             8
                                                                      Yearly                   7                                                   Yearly
  Task Number




                                                                                Task Number




                 5
                                                                                               6
                 4                                                    Monthly                  5                                                   Monthly
                 3                                                                             4
                 2                                                    Weekly                   3                                                   Weekly
                                                                                               2
                 1                                                                             1
                 0                                                    Daily                    0                                                   Daily
                     -2     0   2   4   6   8   10   12   14   16                                  -2    0   2   4   6   8   10   12   14   16


                         Week                                                                       Week




                          Subject 5                                                                     Subject 6
                13                                                                            22
                                                                                              21
                12                                                                            20
                11                                                                            19
                                                                                              18
                10                                                                            17
                                                                                              16
                9                                                                             15
                                                                                              14
                8
                                                                                              13
                7                                                                             12
                                                                                              11
                6                                                                             10
                                                                                               9
                5                                                   Frequency                  8
                                                                                                                                                 Frequency
                4                                                                              7
                                                                      Yearly                   6                                                   Yearly
  Task Number




                                                                                Task Number




                3                                                                              5
                                                                      Monthly                  4                                                   Monthly
                2                                                                              3
                                                                      Weekly                   2                                                   Weekly
                1
                                                                                               1
                0                                                     Daily                    0                                                   Daily
                     -2     0   2   4   6   8   10   12   14   16                                  -2    0   2   4   6   8   10   12   14   16


                         Week                                                                       Week




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                             128


                                                       Subject 7
                                             34
                                             32
                                             30
                                             28
                                             26
                                             24
                                             22
                                             20
                                             18
                                             16
                                             14
                                                                                                 Frequency
                                             12
                                             10                                                    Yearly




                               Task Number
                                             8
                                                                                                   Monthly
                                             6
                                             4                                                     Weekly
                                             2
                                             0                                                     Daily
                                                  -2     0   2   4   6   8   10   12   14   16


                                                   Week




Figure 5.4 Frequency of online activities

        Results from the Exit Interviews support the observations that the saliency of the

task and external factors affect the frequency ratings. Subjects indicated that frequency

fluctuated depending on the immediacy of the current task and other tasks, “frequency

depends on other tasks for this period. This frequency shows the importance of the

objective,” and “you could see my fluctuation in interest in that, I had a big chunk of

selling stuff right before my oral exam, and you could see my interest in selling waning

when I had my oral exam.” Frequency ratings were also impacted by external events,

like holidays, “my first frequency is once or twice a month and it changed because over

spring break I checked more often.”

        Subjects also described their behaviors as clustered and opportunistic: “a lot of

stuff I did was really sporadic; I’d do a whole bunch of it in a couple weeks and then not

much later,” “realistically, I wasn’t searching once or twice a month, my searches were

very clustered in that they had a tendency to deal with immediate need,” and “I didn’t do

anything about it all semester until about a month and a half ago, and then just went

ballistic.” This caused some problems with maintaining the ratings, especially since the

frequency ratings were projected, “if I was working on something a lot one week and


Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                                                         129

then not at all the next week I felt like I had to balance them out.” However, overall, few

subjects indicated any problems understanding the frequency rating scale, with the

exception of the overlapping scale-points described earlier. Like endurance, subjects

noted a difference between their perception of frequency and reality: “I think it, as much

as anything reflects how much I would like to be working on the task … I wouldn’t be

surprised if you saw me always thinking I did stuff more frequently.”



5.3.3                     Stage

                          Finally, for each task identified, subjects were asked to characterize it according

to the stage with which they were in with regard to completing the task. Stage was

assessed on a seven point scale, where 1 = starting and 7 = finished. Subjects were also

provided with the option of responding “not applicable” if they felt that a specific task

could not be characterized according to stage. Stage characterizations are displayed in

Figure 5.5. As can be seen in the figure, with the exception of Subject 6, for most tasks,

stage was considered to be not applicable. Example tasks whose stage many subjects

described as “not applicable” included “reading the news” and “shopping.”

                         Subject 1                                                                           Subject 2
                7                                                                                  12

                                                                                                   11
                6
                                                                                                   10

                                                                                                   9                                                  Stage
                5
                                                                                                   8                                                    not applicable

                4                                                                                  7
                                                                                                                                                        7 [finished]
                                                                                                   6
                                                                                                                                                        6
                3                                                                                  5
                                                                  Stage                                                                                 5
                                                                                                   4
                2                                                   not applicable                                                                      4
  Task Number




                                                                                     Task Number




                                                                                                   3
                                                                    3                                                                                   3
                                                                                                   2
                1
                                                                    2                                                                                   2
                                                                                                   1
                0                                                   1 [starting]                   0                                                    1 [starting]
                    -2    0   2   4   6   8   10   12   14   16                                         -2    0   2   4   6   8   10   12   14   16


                     Week                                                                                Week




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                                                                                                              130


                     Subject 3                                                                                               Subject 4
                20                                                                                                 26
                19                                                                                                 25
                                                                                                                   24
                18                                                                                                 23
                17                                                                                                 22
                16                                                                                                 21
                                                                                                                   20
                15                                                              Stage                              19
                14                                                                                                 18
                13                                                                not applicable                   17
                12                                                                                                 16
                                                                                  7 [finished]                     15                                                                                 Stage
                11                                                                                                 14
                10                                                                                                 13
                                                                                  6                                12                                                                                      not applicable
                 9
                                                                                                                   11
                 8                                                                                                 10
                                                                                  5                                                                                                                        7 [finished]
                 7                                                                                                  9
                 6                                                                4                                 8
                                                                                                                    7                                                                                      6
  Task Number




                                                                                                     Task Number
                 5
                                                                                                                    6
                 4                                                                3                                 5                                                                                      5
                 3                                                                                                  4
                 2                                                                2                                 3                                                                                      3
                                                                                                                    2
                 1                                                                                                  1
                 0                                                                1 [starting]                      0                                                                                      2
                     -2   0   2   4   6   8   10   12         14           16                                           -2    0        2        4        6           8       10      12    14    16


                      Week                                                                                               Week




                     Subject 5                                                                                               Subject 6
                13                                                                                                 22
                                                                                                                   21
                12                                                                                                 20
                11                                                                                                 19
                                                                                                                   18
                10                                                                                                 17
                                                                                Stage                              16
                9                                                                                                  15
                                                                                  not applicable                   14
                                                                                                                                                                                                           Stage
                8
                                                                                                                   13
                                                                                  7 [finished]                     12                                                                                          7 [finished]
                7
                                                                                                                   11
                6                                                                 6                                10                                                                                          6
                                                                                                                    9
                5                                                                 5                                                                                                                            5
                                                                                                                    8
                4                                                                                                   7
                                                                                  4                                 6                                                                                          4
  Task Number




                                                                                                     Task Number




                3                                                                                                   5
                                                                                  3                                 4                                                                                          3
                2                                                                                                   3
                                                                                  2                                 2                                                                                          2
                1
                                                                                                                    1
                0                                                                 1 [starting]                      0                                                                                          1 [starting]
                     -2   0   2   4   6   8   10   12         14           16                                           -2        0        2        4        6           8      10    12    14        16


                      Week                                                                                               Week




                                                                                Subject 7
                                                                      34
                                                                      32
                                                                      30
                                                                      28
                                                                      26                                                                                     Stage
                                                                      24
                                                                      22                                                                                         not applicable
                                                                      20
                                                                                                                                                                 7 [finished]
                                                                      18
                                                                      16                                                                                         6
                                                                      14
                                                                                                                                                                 5
                                                                      12
                                                                      10                                                                                         4
                                                        Task Number




                                                                      8
                                                                                                                                                                 3
                                                                      6
                                                                      4                                                                                          2
                                                                      2
                                                                      0                                                                                          1 [starting]
                                                                           -2     0      2       4   6             8         10       12       14       16


                                                                            Week




Figure 5.5 Stage of activities

                          It is also clear from the figure that very few tasks were completed during the

course of this study.                                   These tasks were often those whose time for completion

corresponded to a university semester. Examples included “teaching” (Subject 1, Task

Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   131

9), “studying quals” (Subject 3, Task 1), “searching for project” (Subject 4, Task 2),

“teaching instruction” (Subject 5, Task 10), and “project in operating systems” (Subject

6, Task 12). For several tasks, subjects’ stage ratings changed both in a forward and

backward direction. For instance, Subject 2’s Task 2, “get course materials,” moves

sequentially through Stage 2 to Stage 7. Considering this study lasted for an entire

semester, it is unsurprising that this task progressed in this manner. Conversely, Subject

5’s Task 7, “legal trouble/conflict,” moves forward from Stage 3, to Stage 4, to Stage 5

and then back to Stage 3. The variability of this particular task was also evident in

Subject 5’s endurance ratings which were quite volatile as presented in Section 5.3.1.

        Subjects made the fewest comments about stage during the Exit Interviews; most

noted that for the majority of their tasks, stage was not applicable and that the concept of

stage was “highly subjective.” Subject 5 indicated some confusion over the use of stage,

which resulted from her distinction between general and specific instances of her

teaching task: she indicated that she was “finished” at the end of the study with a course,

but noted that she would be teaching more courses in the future, so she was not truly

finished with teaching, just with this particular instance. Subject 6 used the stage ratings

more than any other subject and even used it to describe his news task, “for stage, I think

it is a misunderstanding for the news, it is a short-term task. For example, I read the

news about the tornado, so I think it is specific news. After you read it, you are finished.”

This comment indicates that Subject 6, like Subject 5, often encountered some difficulties

using the stage scale, which could have been a result of the level of specificity with

which tasks were defined or with trying to use the scale to characterize tasks whose stage

could not be easily characterized.



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   132

        Finally, one subject observed that while ideally a task should progress linearly

through the various levels of stage, reality was often different: “I know at the time I was

thinking, tasks are supposed to proceed very linearly, but the way this works is, here you

are at three and next week you’re at five, but then you get thrown back to four because

you hit a little snafu.”



5.4     Topics

        As with tasks, subjects identified a range of topics with which they were

interested in during the course of this study. The topics identified by each subject are

displayed in Table 5.10 to 5.16. Next to each topic is a number that is used in the

remainder of this dissertation to refer to the specific topic. As with tasks, topics are

presented in the chronological order in which subjects identified the topic. Thus, subjects

identified topics with smaller numbers before they identified topics with larger numbers.

Topics were identified with the initial Topic Questionnaire and with subsequent Update

Questionnaires, although subjects added the majority of topics during their document

evaluation activities.

Table 5.10

Subject 1’s Topics

 1. History of              4. Evil Republicans        7. Outdoor Recreation
    Archaeology
 2. American Indians        5. Housing Industry and    8. Trivia
                               Laws
 3. Museums and             6. Politics                9. Entertainment
    Monuments




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   133

Table 5.11

Subject 2’s Topics

  1.   West Africa          21. Poetry               41. Antiques              61.   Campus Information
  2.   Yoga                 22. Course Web Page      42. Email                 62.   French Knives
  3.   Beer and Wine        23. NY Weather           43. Cellphones            63.   Automobiles
  4.   Martial Arts Classes 24. Human Interests      44. Car Stereo            64.   Internet Connection
                                                         Components
  5. Video Editing          25. Maps                 45. Auto Accessories      65. Funding
     Software                                                                      Oportunities
  6. Automotive Repair      26. Dams                 46. Employment Ops        66. Academic
                                                                                   Association
  7. Urban Growth           27. Fire Monitoring      47. Taxes                 67. TV Scheduling
  8. Political Ecology      28. General Interests    48. Clothing              68. Track Shipping
  9. Politics               29. Graduate Student     49. Structural            69. Buying Property
                                Association              Adjustment
10. Office Chair            30. People               50. Shipping Costs        70. Calling Codes
11. Native American         31. Satellite Data       51. Currency              71. Fat Loss
    Flutes                                               Conversion Rates
12. War                     32.   Research Methods   52. House Plants          72. Store Location
13. Entertainment           33.   Niger River        53. Slide Projector       73. Displacement
14. DC Sniper               34.   Astrology          54. Housewares            74. Location Direction
15. Bedding                 35.   Utility Bill       55. Depression            75. Cancer
16. Desk Lamp               36.   Banking            56. General Health        76. Vacumns
17. NJ Weather              37.   Malta              57. Birds                 77. Editing
18. Classic Cars            38.   Biophysical Data   58. Coffee                78. Sewing
19. Down Winter Jacket      39.   Contraception      59. Real Estate           79. Veterinary
                                                                                   Information
20. Winter Gloves           40. Water Resources      60. Camping               80. Zip Codes
                                                         Equipment



Table 5.12

Subject 3’s Topics

  1. Political Theory        6. News                  11. War/Politics          16. Coffee
  2. IP Law                  7. Humor                 12. Graduate School       17. Food
  3. Interests Groups and    8. Recreation            13. Teaching Materials
     Democratic
     Institutions
  4. Sports                  9. Political             14. Public Law
                                commentary
  5. Rutgers Facilities     10. Communications        15. Computer
                                                          Assistance




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                    134

Table 5.13

Subject 4’s Topics

  1. Image Processing       10. Computer              19. Graphics Calculator   28. Graphics Books and
     and Computer               Information                                         Papers
     Graphics
     Techniques
  2. Graphical              11. Automobile            20. Movies                29. RU Books about
     Skeletonization of         Information                                         Morse Function
     3D Model
  3. fMRI Brain Images      12. Health Information    21. My own Homepage       30. Campus Bus
                                                                                    Information
  4. Online Shopping        13. Sports Information    22.   My lab Webpage      31. Server Firewall
  5. Weather                14. Cell Phone            23.   Check Jewish        32. Python
                                                            Holiday of Passover     Programming
                                                                                    Language
  6. English Language       15. University            24.   Rutgers Resident    33. SIP Programming
                                Employment                  Network
  7. University             16. Software              25.   Medical Mask        34. Secure Shell
     Information                Installation/Hard                                   Software
                                Driver Information
  8. News                   17. IRIS Library Search   26. About SARS News       35. Math: Gamma
                                                                                    Function
  9. Friends' Information 18. Commercial              27. Morse Function
                              Companies                   (Math)
                              Information



Table 5.14

Subject 5’s Topics

  1. Travel for Singles      8. Housewares            15. Snow                  22. Community
  2. Travel for              9. Dogs                  16. Money for Last        23. Renting an
     Conferences                                          Year in School            Apartment
  3. Dissertation Grants,   10. Community, Work,      17. Graduate Student      24. Buying a Home
     Funding, Awards for        and Identity              Teaching Award
     Teaching, Grants
     from Organizations
  4. National               11. Interpersonal         18. Preliminary           25. Government
     Communication              Communication             Bibliography              Programs to help
     Association,                                                                   with buying/renting
     International
     Communication
     Association, Eastern
     Communication
     Aassociation
  5. Friends, Family,       12. Organizational        19. Committee Member
     Professional               Communication             Selection
     Contacts
  6. Clothes                13. NB Parking Ticket     20. Family Tree
  7. Furniture              14. State Police          21. Developing Survey


Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                        135

Table 5.15

Subject 6’s Topics

  1. Nanotechnology          11. Chinese Literature    21. Check Daily            31. Download
                                                           Weather                    QuickBooks
  2. Information             12. Economy               22. Memory                 32. Review Jobs
     Retrieval                                             Management
  3. Computer Structure      13. Military              23. Daily Life             33. Check Company
                                                                                      Information
  4. Application of          14. Auto-repairing        24. Microsoft Access       34. Download
     Modern Technology                                                                GSViewer
     in Traditional Fields
  5. Finite Element          15. Review Items Online 25. Verification of          35. Read Movie Review
     Method                                              Compiler
                                                         Optimization
  6.   Theoretical Analysis 16. Check Status         26. OS: File System          36. Simulated Banking
       in Mechanics                                                                   Service
  7.   Digitization of Small 17. Thread Management 27. Entertainment              37. Check Facilities for
       Instruments               in Operating System                                  ESL
  8.   3G Wireless           18. Politics            28. Look for Address         38. Health
       Communication                                     Online
  9.   Security in           19. Technology          29. Read INS                 39. Review Courses
       Computers and                                     Immigration Rules            Information
       Network
10.    Biomechanics          20. Experimental        30. Word Meaning and         40. View Lyric of a
                                 Mechanics               Pronunciation                Song



Table 5.16

Subject 7’s Topics

 1. Singers/Songwriters           8. Academy Awards      15. Greek                 22. Fellowships
 2. Campaign Finance              9. Digital Music       16. Online Auctions       23. Entertainment
                                     Production
 3. Venues for Guitar and        10. Current Events      17. Travel                24. Parking
    Vocal Performances in
    NYC and UK
 4. World Soccer              11. Leisure Reading        18. Theater               25. Grades
 5. Literature Departments at 12. 19th Century           19. Teaching              26. Software
    North American                Realism                    Literature Courses
    Universities
 6. Acoustic Guitars          13. Graduate Student       20. Friends
                                  Comparative
                                  Literature Journal
 7. Live Music                14. CDs                    21. Housing



          Unlike tasks, there were few topics which appeared on almost everyone’s topic

list. However, most subjects identified a general topic category of “news” to describe

Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   136

current events. There were only a couple of instances of a subject identifying a specific

current event as a topic; for example, Subject 4 identified “about SARS news.” Thus,

some topics were lumped together, while others retained a unique identity.                       Other

subjects identified topics that could be interpreted as tasks; for example, Subject 5

identified “committee member selection.” This perhaps indicates confusion and/or

ambiguity with the process of thinking about one’s activities in terms of tasks and topics

or just reflects the idiosyncratic nature of subjects when performing this activity.

        Most topics differed between subjects and many were discipline specific. Indeed,

without specific training in a discipline, one might have a difficult time understanding

many of the topics.          For instance, Subject 6, a mechanical engineer identified

“nanotechnology,” “finite element method,” and “3G Wireless” as topics. Subject 3’s

discipline, or area(s) of expertise, also appeared to have influenced the kinds of topic

distinctions made. Subject 3 studies political science and at least 5 of the 17 topics

identified by this subject could be considered to be about politics: “political theory,”

“interests groups and democratic institutions,” “political commentary,” “war/politics,”

and “public law.”

        Topics ranged from academic to recreational interests.                  There was a large

difference in the number of topics identified by each subject. Subject 2 identified eighty

unique topics during the course of the study, while Subject 1 only identified nine. An

examination of the topics identified by each of these subjects reveals that Subject 1

identified very broad topic classes: “trivia,” “outdoor recreation,” and “entertainment,”

with only a couple of instances of more refined topic classes: “evil republicans” as a

more refined version of “politics.” Conversely, Subject 2 identified very specific topic



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                                                              137

classes, which were quite variable: “calling codes,” “contraception,” “utility bill,” “fire

monitoring,” “beer and wine,” “birds,” and “classic cars,” just to name a few.



5.4.1                      Persistence

                           Subjects were asked to characterize each topic that they identified according to

two attributes: persistence and familiarity. This section reports on subjects’ persistence

ratings. Familiarity ratings are presented in the next section. Subjects were asked to

make these characterizations for each week of the study that the topics were active (i.e.

still of interest). Persistence represented the length of time that subjects expected to be

interested in information on the topic. This scale ranged from “one day” to “several

years.” Persistence ratings for each topic are displayed in Figure 5.6.

                           As with endurance, persistence did not change all that much. In most cases

subjects indicated that persistence was “several years” and this characterization remained

constant throughout the study. This is most likely because many of the topics were

related to subjects’ academic (and usually career) interests and/or subjects’ recreational

interests.                         It should be noted that Subject 2’s figure looks different from the other

subjects’ figures because of the number of topics that were identified.

                      Subject 1                                                                                  Subject 2
                 10                                                                                    80
                                                                                                       75
                 9
                                                                                                       70

                 8                                                                                     65
                                                                                                       60                                                 Persistence
                 7                                                                                     55
                                                                                                                                                             several years
                                                                                                       50
                 6
                                                                                                       45                                                    one year
                 5                                                                                     40
                                                                                                                                                             several months
                                                                                                       35
                 4
                                                                                                       30                                                    one month

                 3                                                                                     25
                                                                                                                                                             several weeks
  Topic Number




                                                                                        Topic Number




                                                                                                       20
                 2                                                  Persistence                        15                                                    one week

                                                                       several years                   10                                                    several days
                 1
                                                                                                       5
                 0                                                     several months                  0                                                     one day
                      -2   0   2    4   6   8   10   12   14   16                                           -2    0   2   4   6   8   10   12   14   16


                       Week                                                                                  Week




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                                                                                                        138


                      Subject 3                                                                                                 Subject 4
                 18                                                                                                   36
                 17                                                                                                   34
                 16                                                                                                   32
                 15                                                                                                   30
                 14                                                                                                   28
                 13                                                                                                   26
                 12                                                                                                   24
                 11                                                                                                   22
                 10                                                                                                   20                                                                            Persistence
                 9                                                                                                    18
                                                                                                                                                                                                       several years
                 8                                                                                                    16
                 7                                                            Persistence                             14                                                                               one year
                 6                                                                                                    12
                 5                                                                 several years                      10                                                                               several months
  Topic Number




                                                                                                       Topic Number
                 4                                                                                                    8
                                                                                   several months                                                                                                      one month
                 3                                                                                                    6
                 2                                                                 one month                          4                                                                                several weeks
                 1                                                                                                    2
                 0                                                                 one week                           0                                                                                several days
                      -2   0   2   4   6   8   10   12   14              16                                                -2        0        2        4        6     8        10    12   14   16


                       Week                                                                                                 Week




                      Subject 5                                                                                                 Subject 6
                 26                                                                                                   42
                 25                                                                                                   40
                 24                                                                                                   38
                 23
                 22                                                                                                   36
                 21                                                                                                   34
                 20                                                                                                   32
                 19
                                                                              Persistence
                                                                                                                      30
                 18
                 17                                                                several years                      28
                 16                                                                                                   26
                 15                                                                one year                           24                                                                            Persistence
                 14                                                                                                   22
                 13
                 12                                                                several months                     20                                                                               several years
                 11                                                                                                   18
                 10                                                                one month                          16                                                                               one year
                  9                                                                                                   14
                  8                                                                several weeks
                  7                                                                                                   12                                                                               several months
  Topic Number




                                                                                                       Topic Number




                  6                                                                                                   10
                  5                                                                one week                            8                                                                               one month
                  4                                                                                                    6
                  3                                                                several days                                                                                                        several weeks
                  2                                                                                                    4
                  1                                                                                                    2
                  0                                                                one day                             0                                                                               one week
                      -2   0   2   4   6   8   10   12   14              16                                                -2        0        2        4        6     8        10    12   14   16


                       Week                                                                                                 Week




                                                                                  Subject 7
                                                                        27
                                                                        26
                                                                        25
                                                                        24
                                                                        23
                                                                        22
                                                                        21
                                                                        20
                                                                        19
                                                                        18                                                                                      Persistence
                                                                        17
                                                                        16
                                                                        15                                                                                          several years
                                                                        14
                                                                        13                                                                                          one year
                                                                        12
                                                                        11
                                                                        10                                                                                          several months
                                                                         9
                                                                         8                                                                                          one month
                                                         Topic Number




                                                                         7
                                                                         6
                                                                         5                                                                                          several weeks
                                                                         4
                                                                         3                                                                                          one week
                                                                         2
                                                                         1
                                                                         0                                                                                          one day
                                                                             -2     0        2     4   6              8         10       12       14       16


                                                                              Week




Figure 5.6 Weekly topic persistence

                           As with endurance, subjects’ estimated persistence did not always correspond to

the actual number of weeks that the topic was active, and was variable, representing

subjects’ uncertainty with respect to when they would no longer be interested in


Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   139

information on the topic. For instance, Subject 6 indicated that the persistence of Task

17, “thread management in operating system,” was “one week” for one week, “several

months” for three weeks, “one month” for one week, and finally, at the end of the study,

the persistence was “several weeks.” Subject 7 indicated that the persistence of Task 21,

“housing,” was “one month” for three weeks, “several weeks” for one week and then

“one week” for the final two weeks of the study.

        Subjects’ comments from the Exit Interviews indicated few problems with

understanding and using the scale to characterize their tasks. Subjects indicated that

changes in persistence primarily were the result of a deadline or a change in personal

interest in the topic. Deadlines were either related to a specific task with which the topic

was associated or with another task: “and this was also right when my oral exam was, I

think that was my wake up call not to worry about this stuff now,” “persistence is going

down because thank heavens, it’s almost done,” and “it became salient right before I

actually had to do it, because it says that I’m going to be interested in it for several days,

several days…” Increases and decreases in personal interest in the topic were also given

as reasons for changes in persistence: “so my persistence has gone way up because I am

pretty sure that I want to study that area,” “and then I switched to eight because I realized

I have looked up stuff on coffee and I will be interested in it for a number of years,” and

“even though I put my persistence would be a year, one day I decided I was sick of it and

I just stopped looking, so that was the end of the topic.”

        Some subjects also expressed uncertainty with how long they would be interested

in a topic, “the sevens indicate that I know I’ll be interested in this until I finish my

dissertation, and after that I probably will be, but I’m not so sure.” Another subject stated



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                                                                140

that persistence was more difficult than endurance because “I’ll be really interested in

stuff and then the next day I couldn’t care less.” This subject further noted that difficulty

and volatility primary arose with short-term topics and not long-term topics.



5.4.2                      Familiarity

                           Finally, subjects were asked to characterize their familiarity with each topic for

each week of the study. Familiarity was assessed on a seven point scale, where 1 =

unfamiliar and 7 = familiar. Familiarity characterizations are displayed in Figure 5.7.

                      Subject 1                                                                               Subject 2
                 10                                                                                 80
                                                                                                    75
                 9
                                                                                                    70

                 8                                                                                  65
                                                                                                    60
                 7                                                                                  55
                                                                                                                                                          Familiarity
                                                                                                    50
                 6
                                                                                                    45                                                       7 [familiar]
                 5                                                                                  40
                                                                                                                                                             6
                                                                                                    35
                 4                                                 Familiarity                      30                                                       5

                 3                                                                                  25
                                                                      7 [familiar]                                                                           4
  Topic Number




                                                                                     Topic Number




                                                                                                    20
                 2                                                    6                             15                                                       3

                                                                      5                             10                                                       2
                 1
                                                                                                    5
                 0                                                    3                             0                                                        1 [unfamiliar]
                      -2   0   2   4   6   8   10   12   14   16                                         -2    0   2   4   6   8   10    12    14    16


                       Week                                                                               Week




                      Subject 3                                                                               Subject 4
                 18                                                                                 36
                 17                                                                                 34
                 16                                                                                 32
                 15                                                                                 30
                 14                                                                                 28
                 13                                                                                 26
                 12                                                                                 24
                 11                                                                                 22
                 10                                                                                 20                                                      Familiarity
                 9                                                                                  18
                                                                   Familiarity                                                                                   7 [familiar]
                 8                                                                                  16
                 7                                                    7 [familiar]                  14                                                           6
                 6                                                                                  12
                 5                                                    6                             10                                                           5
  Topic Number




                                                                                     Topic Number




                 4                                                                                  8
                                                                      5                                                                                          4
                 3                                                                                  6
                 2                                                    4                             4                                                            3
                 1                                                                                  2
                 0                                                    3                             0                                                            2
                      -2   0   2   4   6   8   10   12   14   16                                         -2    0   2   4   6   8    10    12    14    16


                       Week                                                                               Week




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                                                                                                   141


                      Subject 5                                                                                                     Subject 6
                 26                                                                                                       42
                 25                                                                                                       40
                 24                                                                                                       38
                 23
                 22                                                                                                       36
                 21                                                                                                       34
                 20                                                                                                       32
                 19                                                                                                       30
                 18
                 17                                                                Familiarity                            28
                 16                                                                                                       26
                 15                                                                   7 [familiar]                        24                                                                     Familiarity
                 14                                                                                                       22
                 13
                 12                                                                   6                                   20                                                                        7 [familiar]
                 11                                                                                                       18
                 10                                                                   5                                   16                                                                        6
                  9                                                                                                       14
                  8                                                                   4
                  7                                                                                                       12                                                                        5
  Topic Number




                                                                                                           Topic Number
                  6                                                                                                       10
                  5                                                                   3                                    8                                                                        4
                  4                                                                                                        6
                  3                                                                   2                                                                                                             3
                  2                                                                                                        4
                  1                                                                                                        2
                  0                                                                   1 [unfamiliar]                       0                                                                        2
                      -2   0   2   4   6   8   10   12           14           16                                               -2     0    2        4        6       8      10    12   14   16


                       Week                                                                                                     Week




                                                                                  Subject 7
                                                                        27
                                                                        26
                                                                        25
                                                                        24
                                                                        23
                                                                        22
                                                                        21
                                                                        20
                                                                        19
                                                                        18                                                                                   Familiarity
                                                                        17
                                                                        16
                                                                        15                                                                                       7 [familiar]
                                                                        14
                                                                        13                                                                                       6
                                                                        12
                                                                        11
                                                                        10                                                                                       5
                                                                         9
                                                                         8                                                                                       4
                                                         Topic Number




                                                                         7
                                                                         6
                                                                         5                                                                                       3
                                                                         4
                                                                         3                                                                                       2
                                                                         2
                                                                         1
                                                                         0                                                                                       1 [unfamiliar]
                                                                             -2      0       2         4    6              8         10   12   14       16


                                                                              Week




Figure 5.7 Weekly topic familiarity

                           Subjects used a wide range of familiarity scores. For Subjects 1 and 4, familiarity

for topics was somewhat constant throughout the study. For Subjects 2, 3, 5, 6 and 7,

familiarity changed for a number of topics and in a variety of directions. While in most

cases, familiarity increased over time, in a few cases it decreased. For instance, for four

consecutive weeks of the study Subject 5 indicated that her familiarity with Topic 25,

“government programs to help with buying/renting,” increased each week from “1” to

“2” to “3” to “4,” at which point the topic was no longer of interest. Conversely, Subject

7 indicated that his familiarity with Topic 2, “campaign finance,” decreased over the

course of the fourteen weeks of the study; for the first eight weeks of the study, Subject 7



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   142

indicated that his familiarity was “3,” for weeks nine through twelve, it was “2,” and for

the final two weeks of the study, it was “1.”

        In the Exit Interviews, subjects identified a variety of reasons for both increases

and decreases in familiarity, and for ratings remaining stable over time. Reasons for

increases were obvious, as subjects learned more about a topic their familiarity increased.

However, in many cases, learning more about a topic caused familiarity to decrease:

“this is a topic I started out thinking I was pretty familiar with and then I realized I didn’t

know anything about it at all,” “you think that you have a grasp, but I think there may be

something you didn’t know,” and “the more I learned about the topic, the more I realized

there was so much more out there I didn’t know about. So there are a lot of times where I

didn’t feel like I was getting any closer to a 7, I felt like as I moved to the endpoint, it

moved farther away.”

        Subjects also observed that learning more about a topic caused familiarity to

remain stable or to rise slowly for some topics, “my familiarity, even though I looked at it

all semester, didn’t go up that much, because the more I read the more I realize that I

don’t know.” A separate reason given by many subjects for stable familiarity ratings

across time was related to rapidly changing topic content. For instance, “politics – that

pretty much stayed the same. It’s something I’ve always been interested in and I put the

familiarity in the middle because it changes all the time.” Other topics identified with

rapidly changing content include news and computers. In commenting on news and

sports, one subject observed, “as far as familiarity, they kind of hit equilibrium, because

this is how much I keep up with it because … most of these things change all the time,

and I just keep up to a certain level.”



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   143

         Overall, subjects indicated few problems understanding and using the familiarity

scale.    However, one problem that was mentioned by several subjects involved

distinguishing between one’s familiarity with a topic and one’s familiarity with the task

with which the topic was associated. When discussing shopping for various products,

one subject observed, “[I] could be unfamiliar with a specific topic like vacuums, but

very familiar with the process of finding information or with shopping.” Another subject

observed, “I’m highly familiar with clothes – I’ve been clothing myself for however

many years. Realistically, I’m not that familiar with different shops that you can buy

clothes online. So, my knowledge of this topic is high, but my knowledge of how to find

out information about this topic online is actually lower.”



5.5      Task and Topic Combinations

         The particular task and topic combinations that subjects used to classify

documents can be found in Appendix L. Most subjects had many instances of distributed

task and topic combinations. That is, a single task was often associated with multiple

topics or a single topic was associated with more than one task. Tasks which were almost

always associated with multiple topics included those about shopping, checking the news

and researching one’s dissertation. Also, tasks which were generic, for example “general

interests,” often had a number of topics associated with them.

         For all subjects, many tasks and topics only occurred in relation to one another

and some subjects were exclusive in their task/topic associations, primarily associating

single tasks with single topics. Viewing the task and topic combinations provide some

insight into how subjects’ construed their activities, how tasks and topics were related,



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   144

and how a single topic was explored in a variety of ways. However, this dissertation

excludes analysis at the level of the task and topic combination, so no further discussion

of these results is presented.



5.6     Creating Task and Topic Classes and Classifying Documents

        Finally, subjects were asked in the Exit Interview to comment on the process of

creating tasks and topics and classifying documents into these classes.                Most subjects

indicated that they had few problems creating task and topic classes and classifying

documents: “it was pretty easy,” and “most of it fit, I knew where it goes.” Although

some subjects also expressed that they had to be creative at times, “I had to think pretty

creatively about where they fit,” and “it was kind of a mental stretch sometimes.”

        One problem that almost all subjects identified was with determining how broad

or specific to be with creating task and topic classes: “I think, sometimes I wondered

how specific to be. Looking at 300 pages, it can be maybe 200 topics. No two pages can

be exactly alike. I just didn’t know how many topics to put.” Another subject comments

on this same issue: “My biggest problem I think was I had a tendency to want to lump

things together rather than having a topic list or task list that would go on endlessly” and

“I think my difficulties with categorizing pages were largely created by myself. By my

too narrow creation of certain categories.” On the other hand, one subject felt that his

broadly defined categories allowed him to easily classify things.

        A special case of the broad versus specific distinction was the identification of

specific instances of tasks and topics, or single general instances, which subjects

commented on numerous times in their discussions of the task and topic attributes: “there



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   145

were times when I had trouble distinguishing between the immediate task or the

immediate topic, in this case, and kind of like a meta level of how I was thinking about

these categorizations.” Subjects also expressed some confusion over how to structure

sub-tasks in relation to a very broad task like writing dissertation. One subject made sub-

tasks out of topics: “I guess, kind of looking at it in terms of the word task vs. topic, it

would make more sense that those would be tasks.                   Because I was looking at the

dissertation as an umbrella, that’s why they’re associated with that particular task and are

called topics.”

        Many subjects observed that they felt constrained by the task and topic classes

that they created, “I felt constrained by my own creation of the categories, because they

set me to think about things in a certain way.”               As the study progressed, subjects

indicated that they learned more about the relationship between their various task and

topic classes, “as I was kind of building this way understanding, I realized that some of

my categories were overlapping,” and how better to specific tasks and topics, “I also feel

like by the end of the semester I learned how to create a better topic category.”



5.7     Usefulness and Confidence

        Subjects rated the usefulness of the documents that they viewed during the weekly

evaluation sessions. Usefulness was assessed on a seven point scale, where the anchors

were “not useful” and “useful.” Responses were coded such that 1 = not useful and 7 =

useful. The results are displayed in Figure 5.8. Overall, the usefulness ratings that

subjects assigned to documents were high.               The standard deviations indicate some




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   146

variance in the usefulness ratings for most subjects: Subject 1 (1.65); Subject 2 (2.0);

Subject 3 (2.2); Subject 4 (.80); Subject 5 (2.4); Subject 6 (.80); and Subject 7 (2.4).

                                        7.0


                                        6.0
                                                         6.1         6.0


                                        5.0                    5.3         5.3
                                                                                       5.0
                                                   4.8
                                                                                 4.6
                                        4.0


                                        3.0
                      Mean Usefulness




                                        2.0


                                        1.0


                                        0.0
                                                   1     2     3     4     5     6     7


                                              Subject



Figure 5.8 Mean usefulness of documents evaluated

        The distributions of usefulness scores for each subject are displayed in Figure 5.9.

The distribution of scores were quite different for each subject, although both Subject 2

and Subject 7 used Usefulness 7 to describe most of the documents that they evaluated.

Subject 1’s distribution is perhaps the most even, although it peaks at Usefulness 5.

Subject 2 classified nearly three-fourths of the documents that she viewed as Usefulness

7; the score that was used with the next highest frequency was Usefulness 1 and it was

only used to make roughly 75 judgments. Subject 3 also used Usefulness 7 with the

highest frequency, although his use of the other ratings was somewhat evenly distributed.

Subject 4 only used Usefulness 3 to 7 to describe the documents that he viewed, with the

majority of documents being assigned Usefulness 6.

        Usefulness scores that received the highest frequencies for Subject 5 were

Usefulness 1 and Usefulness 7, with the intermediate scores receiving less use. Subject 6


Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                    147

assigned over half of the documents that he viewed Usefulness 5, with Usefulness 4

receiving another 330 documents assignments. Subject 6 used the remaining usefulness

score infrequently. Finally, Subject 7’s distribution of usefulness scores is also skewed

toward the high end of the scale, with approximately half of the documents evaluated by

Subject 7 assigned Usefulness 7. The next highest frequency was Usefulness 1 and it was

used to make roughly 165 judgments.

                                              660
                                              630
                                              600
                                              570
                                              540
                                              510                                    Subject (N)
                                              480
                                              450
                                              420                                        1 (843)
                                              390
                                              360                                        2 (747)
                                              330
                                              300
                      Number of Evaluations




                                                                                         3 (377)
                                              270
                                              240
                                              210                                        4 (349)
                                              180
                                              150                                        5 (188)
                                              120
                                               90                                        6 (1181)
                                               60
                                               30
                                                0                                        7 (1125)
                                                     1      2    3   4   5   6   7


                                                    Usefulness



Figure 5.9 Distributions of usefulness ratings for each subject

        Subjects were also asked about their confidence with respect to their usefulness

ratings. Confidence was assessed on a seven point scale where the anchors were “low”

and “high.” Responses were coded such that 1 = low and 7 = high. As can be seen in

Figure 5.10, subjects were, in general, highly confident in the ratings that they assigned to

documents. The standard deviations indicate little variance in these scores: 0.66, 0.54,

0.59, 0.79, 0.94, 0.41, and 0.42, respectively.




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                     148


                                        7.0
                                                         6.9   6.8                     6.9
                                                                           6.6
                                        6.0
                                                                     5.9
                                                   5.7
                                        5.0                                      5.2


                                        4.0


                                        3.0



                      Mean Confidence   2.0


                                        1.0


                                        0.0
                                                   1     2     3     4     5     6     7


                                              Subject



Figure 5.10 Mean confidence in usefulness ratings assigned to documents

        Subjects were asked to reflect on their interpretation and use of the usefulness and

confidence scales during individual Exit Interviews at the completion of the study. All

subjects indicated that they had no problems understanding and using the usefulness

scale, although one subject did indicate that the meaning of the mid-point was sometimes

ambiguous. Subjects indicated that they used the scale liberally and in a completely

subjective way; a document that was found useful was marked as useful regardless of its

relevance to a query. This usage likely contributed to the overall high usefulness ratings

made by many subjects.                             Two subjects commented that they liked the number of

responses because the scale “really accommodated nuances and how you really feel about

the page.” One subject described an additional usefulness scale that he would like to use

to rate those pages that he visited only to see if information had been updated or changed.

        All subjects reported that they did not have any problems understanding and using

the confidence scale.                         Most subjects indicated that they almost always rated their

confidence high, unless they “didn’t remember viewing a page that well” or “didn’t


Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   149

completely trust my memory of why I had been at the page.” One subject stated, “the

confidence scale allowed me to indicate my sureness of the decision … it enabled me to

indicate my level of commitment to my answers.”



5.8     Behaviors and Usefulness

        The results reported in this section address the first and second research questions

of this dissertation, which sought to explore the relationship between observable

behaviors and documents preference. Two types of behaviors were explored: display

time and retention. Document preference was measured by the usefulness rating given to

a particular document by a subject.



5.8.1   Display Time

        Because of a failure with the client-side logging software, in many cases, display

time had to be collected from the proxy logs. The extent of this problem is illustrated in

Figure 5.11. This figure shows for each subject, how much of the overall display time

data came from the client and how much came from the proxy. For some subjects, such

as Subject 1, Subject 5, and Subject 7, over half of the display time data had to be

computed from the proxy logs.




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                  150


                                            1200
                                                                                     970
                                            1100

                                            1000                                           489

                                            900

                                            800       301

                                            700              424

                                            600




                      Number of Documents
                                            500                                            564
                                                      535

                                            400
                                                                   272
                                            300              330         226
                                                                                                 Display Time
                                            200
                                                                                     218
                                                                               76                   Client-side
                                            100
                                                                   102   115   113
                                              0                                                     Proxy-side
                                                       1     2     3     4     5     6     7


                                                   Subject



Figure 5.11 Origin of display time data

        While this might be viewed as a rather devastating loss of data, the mix of proxy-

and client-side display times provided an excellent opportunity to directly examine the

relationship between display times collected from a proxy and display times collected

from a client-side logger. The distributions of client- and proxy-side display time for all

subjects combined are displayed in Figures 5.12a and 5.12b. As evident from the figures,

both distributions are skewed, with a majority of points being described by less than ten

seconds and the distance between points increasing as display time increases. For both

distributions, the maximum display times are a little over 30 minutes, which is in contrast

with maximum display times typically collected from proxies, where large outliers often

have to be screened from the data before analysis can be performed. However, an

examination of the proxy time distribution does suggest that perhaps low display times

are a problem. Note that in Figure 5.12a there are a larger number of documents being

described by very low display times and then a huge drop, while in Figure 5.12b the

distribution declines more steadily as display times increase.


Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                           151

                              280                                              280
                              260                                              260
                              240                                              240
                              220                                              220
                              200                                              200
                              180                                              180
                              160                                              160
                              140                                              140

        Number of Documents




                                                         Number of Documents
                              120                                              120
                              100                                              100
                              80                                               80
                              60                                               60
                              40                                               40
                              20                                               20
                               0                                                0
                                     0:
                                     0: :00
                                     0: :13
                                     0: :26
                                     0: :39
                                     0: :52
                                     0: :05
                                     0: :18
                                     0: :32
                                     0: :45
                                     0: :00
                                     0: :16
                                     0: :33
                                     0: :04
                                     0: :23
                                     0: :58
                                     0: :47
                                     0: :18
                                     0: :30
                                     0: :44




                                                                                      0:
                                                                                      0: :01
                                                                                      0: :12
                                                                                      0: :22
                                                                                      0: :30
                                                                                      0: :40
                                                                                      0: :51
                                                                                      0: :04
                                                                                      0: :18
                                                                                      0: :31
                                                                                      0: :44
                                                                                      0: :59
                                                                                      0: :13
                                                                                      0: :30
                                                                                      0: :53
                                                                                      0: :13
                                                                                      0: :37
                                                                                      0: :04
                                                                                      0: :52
                                                                                      0: :06
                                       00
                                       00
                                       00
                                       00
                                       00
                                       01
                                       01
                                       01
                                       01
                                       02
                                       02
                                       02
                                       03
                                       03
                                       03
                                       04
                                       06
                                       08
                                       11
                                       20




                                                                                        00
                                                                                        00
                                                                                        00
                                                                                        00
                                                                                        00
                                                                                        00
                                                                                        01
                                                                                        01
                                                                                        01
                                                                                        01
                                                                                        01
                                                                                        02
                                                                                        02
                                                                                        02
                                                                                        03
                                                                                        03
                                                                                        04
                                                                                        04
                                                                                        06
                                                                                        11
                                          :3




                                                                                           :2
                                             0




                                                                                              0
                                    Proxy Display Time                               Client Display Time

                                (a)                                                  (b)

Figure 5.12 Distributions of proxy- and client-side display time

            One-second display times from the proxy might be expected since the proxy

created a unique record for each URL that it encountered, and display times were

computed from the proxy by subtracting request times; pages which had advertisements

with them would often have a display time of one second or less if the advertisement

loaded second. However, even for the client-generated display times, 261 documents, or

10% of the total number of documents, were described by a display time of one second.

            Table 5.17 presents the descriptive statistics for display time for each subject and

for each type of display time. Overall, the mean display time of the client-generated data

was less than the proxy-generated data. When examining the means on a subject-by-

subject basis, this remained true for all subjects but Subject 1. In most cases, the standard

deviations of the proxy-generated display times were larger than the client-generated

display times, and median display times differed more greatly from the mean for the

proxy data than for the client data. An examination of the descriptive statistics in

combination with the distributions in Figure 5.12 indicate that much more extreme

variability exists in the proxy-generated data.




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   152

Table 5.17

Descriptive Statistics for Display Time for Each Subject

                                           Descriptive Statistics
                   Mean            Standard Deviation        Median                     Mode
              Client Server         Client   Server     Client Server              Client  Server
       1      00:53    00:07        01:43     00:40     00:20     00:02            00:02   00:02
       2      00:55    01:19        02:08     03:10     00:21     00:25            00:01   00:01
       3      00:44    00:53        01:53     01:50     00:17     00:15            00:03   00:09
 Subject




       4      00:36    01:15        01:43     03:35     00:07     00:09            00:02   00:01
       5      00:45    01:09        02:14     03:30     00:10     00:30            00:02   00:01
       6      00:09    01:17        00:18     03:30     00:04     00:25            00:01   00:01
       7      00:50    01:11        01:28     01:46     00:48     00:17            00:18   00:04
   Total      00:38    00:48        01:31     02:23     00:12     00:08            00:01   00:01


           Finally, a Pearson’s correlation was computed between the proxy- and client-

generated display times in cases where both types of data were available (n=2,776). The

results of this analysis found no statistically significant relationship between the two

types of display times, r =.013, p=.497.            This suggests that the proxy- and client-

generated display times should not be pooled, and that proxy-generated display times, at

least as computed in this study, are not very valid substitutes for client-generated display

times. Thus, all analyses reported in the remainder of this chapter involving display time

were conducted using client display times, which comprised 58% of the all display time

data. Unfortunately this excluded 42% of the display time data from these computations,

and over half of the display time data for Subjects 1, 5 and 7. However, analyses of

proxy-generated display times are briefly presented for comparison purposes only, and to

illustrate the potential danger in using unreliable data for these computations.

           Overall, for the client-side data, the general trend was for mean display times to

be considerably lower than the standard deviations, which indicates that the mean is not

the best descriptor of the distribution. It is clear when considering the mode why this is

Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   153

the case; with the exception of Subject 7, the mode for most subjects is one to three

seconds. Table 5.18 presents the observed display times for various percentages of the

data. With the exception of Subject 7, mean display times only describe 25% or less of

the display time data for subjects.

Table 5.18

Distributions of Client Display Times According to Percentage of Data

                                                      Percentile
                                     25%           50%        75%            100%
                              1      00:07         00:20      00:14          14:54
                              2      00:08         00:21      00:55          30:57
                              3      00:07         00:17      00:34          16:48
                    Subject




                              4      00:03         00:07      00:25          16:00
                              5      00:05         00:10      00:26          13:52
                              6      00:02         00:04      00:09          03:45
                              7      00:20         00:48      01:24          11:30


        To adjust these distributions, a logarithm transformation was performed using the

natural log. The log distribution of display time is displayed in Figure 5.13. Documents

which had a display time of one second (n=261) appear at point zero on the log scale.

The result of this transformation is a more normal distribution, since logarithm

transformations effectively minimize the differences between scores.




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                              154


                     300




                     200




                     100



                                                                                                       Std. Dev = 1.52
                                                                                                       Mean = 2.49
                       0                                                                               N = 2758.00
                           0.
                                .5
                                     1.
                                          1.
                                               2.
                                                    2.
                                                         3.
                                                              3.
                                                                   4.
                                                                        4.
                                                                             5.
                                                                                  5.
                                                                                       6.
                                                                                            6.
                                                                                                 7.
                                                                                                      7.
                            00
                                 0
                                     00
                                          50
                                               00
                                               50
                                                         00
                                                              50
                                                                   00
                                                                        50
                                                                             00
                                                                                  50
                                                                                       00
                                                                                            50
                                                                                                 00
                                                                                                 50
                           Log of Display Time



Figure 5.13 Distribution of log client display time

        The distributions of display time according to usefulness score is displayed in

Figure 5.14. It should be noted that when the proxy data was excluded, the overall

distribution of usefulness scores did not change all that much from that displayed in

Figure 5.9, except that the total number in each group decreased. While no general

relationship between display time and usefulness can be gleaned from Figure 5.14, it is

clear that display times differ for subjects. For instance, Subject 7 consistently displayed

documents for a longer period of time than other subjects, while Subject 6 consistently

displayed documents for a shorter period of time than other subjects. While there is some

cross-over amongst display times for the other five subjects, the figure demonstrates no

clear relationship between display time and usefulness.                                                     The means and standard

deviations for natural display time according to usefulness score can be found in

Appendix M.




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                          155


                                              5.0

                                              4.5

                                              4.0
                                                                                                     Subject
                                              3.5
                                                                                                               1
                                              3.0
                                                                                                               2
                                              2.5




                      Mean Log Display Time
                                                                                                               3
                                              2.0
                                                                                                               4
                                              1.5
                                                                                                               5
                                              1.0

                                                                                                               6
                                               .5

                                              0.0                                                              7
                                                      1      2    3       4       5      6       7


                                                    Usefulness



Figure 5.14 Mean log display time according to usefulness score

        One-way ANOVAs were conducted to explore the relationship between display

time and usefulness score for each subject. For this analysis, usefulness was treated as an

independent variable and display time was treated as a dependent variable. The results of

these tests are presented in Table 5.19. Table 5.19 displays the F values for all significant

relationships and the η2 which was used to assess the strength of the relationship. Non-

significant results are noted by “NS.” An alpha level of .01 was used for all statistical

tests reported in this chapter.

Table 5.19

ANOVA Results for Display Time and Usefulness

                                                                       Subject
       1             2                                     3              4               5               6             7
     NS            NS                                    NS              NS             NS              NS         F(6,485)=
  F(6,300)=     F(6,412)=                             F(6,267)=       F(4,224)=       F(5,71)=       F(6,958)=       6.35*;
     1.20          1.42                                  1.88            1.42           1.26            2.29        η2=0.07
                                                                       *p<.01




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   156

          There was only one statistically significant relationship between mean log display

time for usefulness and this was for Subject 7. However, the η2 indicates that even this

statistically significant relationship was not very strong, with only 7% of the variance in

display time being explained by usefulness score.                While it should be noted that

accounting for 10% of the variance in social science research is considered modest and

“probably would cause a celebration among social scientists” (Witte, 1993; p.379), when

designing techniques that would affect information system responses and performance,

accounting for more of the variance is desirable.

          Follow-up tests were conducted to evaluate pair-wise differences between display

time and usefulness using a Scheffe’s post-hoc test. The Scheffe’s post-hoc test is a

conservative test that allows for pair-wise comparisons when there are unequal cell sizes

and is thus very appropriate for this analysis. The result of this test identified a single

statistically significant pair-wise difference for Subject 7, between usefulness 1

(M=0:00:51; SD=0:01:36) and usefulness 6 (M=0:01:45; SD=0:01:10) and usefulness 7

(M=0:01:16; SD=0:01:23). This result demonstrated that the documents that Subject 7

rated as highly useful were displayed significantly longer than those rated as not very

useful.

          One-way ANOVAs were conducted to investigate the relationship between

proxy-generated display times and usefulness for each subject. Statistically significant

relationships were identified for Subject 1 and Subject 7, F(6,502) = 6.60, p<.01 and

F(6,555) = 3.66, p<.01.         Further analysis of the combination of client- and proxy-

generated display times and usefulness revealed no statistically significant relationships

at alpha levels of .01. These results indicate the potential danger of Type I and Type II



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   157

errors when using proxy-generated data alone, or pooling it with the client-generated

data.



5.8.2     Retention

          Retention behaviors that were of interest in this study included saving,

bookmarking and printing. It should be noted that retention behaviors can only be

collected via client-logger; when the client-logger stopped working in this study,

retention was not recorded. Overall, retention behaviors occurred very infrequently, as

can be observed in Table 5.20. Printing is excluded from this table and from further

analysis because most subjects did not print anything from their laptops. This was

verified through follow-up emails to subjects. Subjects reported that they printed things

at their offices because it was cheaper and the printers found there were often better than

the ones distributed to the subjects at the beginning of the study. There were some

instances of subjects printing documents from email or from a word processing

application, but no instances were recorded by the client-logger of the printing of the

online documents that were eventually evaluated by subjects.

Table 5.20

Number of Retained Documents (Percent of Total Documents Evaluated)

                                                        Subject
                       1           2           3           4            5           6           7
                      12          26          17           4            1           9           3
 Bookmarked
                    (1.4%)      (3.4%)      (4.5%)      (1.1%)       (0.5%)      (0.8%)      (0.3%)
                                   2                       5                       13           2
        Saved          0                        0                       0
                                (0.3%)                  (1.4%)                   (1.1%)      (02.%)
                      12          28          17           9            1          22           5
        Total
                    (1.4%)      (3.7%)      (4.5%)      (2.5%)       (0.5%)      (1.9%)      (0.5%)




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                           158

        The relationship between retention and usefulness for each subject is displayed in

Figure 5.15. While the mean usefulness of documents that were retained was high for all

subjects except Subjects 6 and 7, it did not differ significantly from the mean usefulness

of documents that were not retained. This is most likely because the overall usefulness

scores for all subjects across all documents were, in general, very high. Thus one

possible approach to using retention is to simply assume that all retained documents are

useful, which is what many studies of implicit feedback have done by using retained

documents as a benchmark for relevance. However, in this study the documents retained

by Subjects 6 and 7 were not rated as particularly useful, so the validity of this

assumption is questionable.

                                        7.0
                                                                                                    7.0


                                        6.0
                                                            6.0 6.2               6.1
                                                                            5.7         5.8
                                                      5.7

                                        5.0                           5.3                     5.3
                                                                                                                    5.0
                                                4.7
                                                                                                          4.6 4.6
                                                                                                                          4.4
                                        4.0



                                        3.0
                      Mean Usefulness




                                                                                                                                Retained
                                        2.0
                                                                                                                                    No

                                        1.0                                                                                         Yes
                                                  1           2         3           4           5           6         7


                                              Subject



Figure 5.15 Relationship between retention and usefulness



5.9     Behaviors and Information-seeking Context

        This section explores the third research question of this dissertation, which sought

to understand the relationship between information seeking context and observable


Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                                         159

behaviors. In this dissertation, information seeking context was defined as the specific

task and topic associated with the document, as well as the task attributes of endurance,

frequency and stage, and the topic attributes of persistence and familiarity. As reported

in the preceding section, the observable behaviors investigated were display time and

retention.



5.9.1        Display Time, Task and Topic

             The mean log display time according to task and topic for each subject is

displayed in Figures 5.16 to 5.22. For cases with extreme standard deviations, and thus,

extreme variance bars, the axis is truncated to better highlight the differences in display

time (e.g. Figure 5.18). In the majority of cases, these large standard deviations occurred

when there were only two documents classified into a particular task or topic. For tasks

and topics with a greater number of documents, the variances were much less, and thus,

the bars are shorter. It should also be noted that a few tasks and topics only had a single

document classified into them. For these cases, no variance bars appear on the figure

(e.g. Figure 5.17).

                                4.0                                                                   5.5


                                                                                                      5.0
                                3.5
                                                                                                      4.5


                                                                                                      4.0
                                3.0

                                                                                                      3.5
        Mean Log Display Time




                                                                              Mean Log Display Time




                                2.5
                                                                                                      3.0


                                                                                                      2.5
                                2.0
                                                                                                      2.0

                                1.5                                                                   1.5
                                  N=      33        127   78   17   30   16                             N=      34    10   16   13   51   88   39   49

                                           1        2     3    4    5    6                                      1     2    3    4    6    7    8    9


                                      Task Number                                                           Topic Number




Figure 5.16 Display time according to task and topic for Subject 1



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                                                                                                                                                                                                                      160

                                5.5                                                                                                                                                                       16
                                                                                                                                                                                                          15
                                5.0
                                                                                                                                                                                                          14
                                4.5                                                                                                                                                                       13
                                                                                                                                                                                                          12
                                4.0
                                                                                                                                                                                                          11
                                3.5                                                                                                                                                                       10
                                                                                                                                                                                                          9
                                3.0

        Mean Log Display Time
                                                                                                                                                                                                          8




                                                                                                                                                                                  Mean Log Display Time
                                2.5                                                                                                                                                                       7
                                                                                                                                                                                                          6
                                2.0
                                                                                                                                                                                                          5

                                1.5                                                                                                                                                                       4
                                                                                                                                                                                                          3
                                1.0                                                                                                                                                                       2
                                 .5                                                                                                                                                                        1
                                     N=        9             16             49            65              62        10        1               56            155          1                                -1
                                               1             2              3                 4           5         6         7               8             10           11




                                                                                                                                                                                                                 12
                                                                                                                                                                                                                 14
                                                                                                                                                                                                                 16
                                                                                                                                                                                                                 18
                                                                                                                                                                                                                 24
                                                                                                                                                                                                                 26
                                                                                                                                                                                                                 28
                                                                                                                                                                                                                 30
                                                                                                                                                                                                                 32
                                                                                                                                                                                                                 35
                                                                                                                                                                                                                 37
                                                                                                                                                                                                                 39
                                                                                                                                                                                                                 41
                                                                                                                                                                                                                 43
                                                                                                                                                                                                                 46
                                                                                                                                                                                                                 59
                                                                                                                                                                                                                 64
                                                                                                                                                                                                                 67
                                                                                                                                                                                                                 69
                                                                                                                                                                                                                 72
                                                                                                                                                                                                                 74
                                                                                                                                                                                                                 76
                                                                                                                                                                                                                 78
                                                                                                                                                                                                                 80
                                                                                                                                                                                                                  1
                                                                                                                                                                                                                  6
                                                                                                                                                                                                                  8
                                      Task Number                                                                                                                                                               Topic Number




Figure 5.17 Display time according to task and topic for Subject 2

                                6                                                                                                                                                                         5



                                5
                                                                                                                                                                                                          4


                                4
                                                                                                                                                                                                          3

                                3
        Mean Log Display Time




                                                                                                                                                                                  Mean Log Display Time
                                                                                                                                                                                                          2
                                2


                                                                                                                                                                                                          1
                                1


                                -1                                                                                                                                                                        0
                                     N=   20       23   131        10           16       15       5        7   9         14       4       4        3         7       2        2                           N=      19   28     17       58       3    25       17       37    18     22       4        15   5         1        3

                                          1        2     3             4        8        9        10 11        12 13          14 15 16                      17 18            19                                   1     4     5        6        7    8        9        10    11     12       13       14   15        16       17


                                      Task Number                                                                                                                                                              Topic Number




Figure 5.18 Display time according to task and topic for Subject 3

                                10                                                                                                                                                                        10

                                 9                                                                                                                                                                        9

                                 8                                                                                                                                                                        8

                                 7                                                                                                                                                                        7

                                 6                                                                                                                                                                        6

                                 5                                                                                                                                                                        5
        Mean Log Display Time




                                                                                                                                                                                  Mean Log Display Time




                                 4                                                                                                                                                                        4

                                 3                                                                                                                                                                        3

                                 2                                                                                                                                                                        2

                                 1                                                                                                                                                                        1

                                 0                                                                                                                                                                        0
                                -1                                                                                                                                                                        -1
                                     N=   37   14       44        11       42        8        8       9    2   18        4    4       4       1        10        2   6        2                                N = 38 13 26   8    2   17   9   9   25    8   4    4    5   10 17   2    2   2    2    1   1    11   6    2       1

                                          1    2        3         4        5         6        9 10 12 13 14 15 16 17 18 19 24 25
                                                                                                                                                                                                                  10
                                                                                                                                                                                                                  11
                                                                                                                                                                                                                  13
                                                                                                                                                                                                                  15
                                                                                                                                                                                                                  16
                                                                                                                                                                                                                  18
                                                                                                                                                                                                                  19
                                                                                                                                                                                                                  20
                                                                                                                                                                                                                  26
                                                                                                                                                                                                                  27
                                                                                                                                                                                                                  28
                                                                                                                                                                                                                  29
                                                                                                                                                                                                                  30
                                                                                                                                                                                                                  31
                                                                                                                                                                                                                  32
                                                                                                                                                                                                                  33
                                                                                                                                                                                                                  34
                                                                                                                                                                                                                   2
                                                                                                                                                                                                                   3
                                                                                                                                                                                                                   4
                                                                                                                                                                                                                   5
                                                                                                                                                                                                                   6
                                                                                                                                                                                                                   7
                                                                                                                                                                                                                   8
                                                                                                                                                                                                                   9




                                      Task Number                                                                                                                                                               Topic Number




Figure 5.19 Display time according to task and topic for Subject 4




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                                                                                                                                                                                                       161

                                7                                                                                                                                               8


                                                                                                                                                                                7
                                6

                                                                                                                                                                                6
                                5

                                                                                                                                                                                5
                                4
                                                                                                                                                                                4

        Mean Log Display Time




                                                                                                                                                        Mean Log Display Time
                                3
                                                                                                                                                                                3

                                2
                                                                                                                                                                                2

                                1
                                                                                                                                                                                1

                                0                                                                                                                                               0
                                 N=           37             26         1                7             1             2            1         1                                    N=           28       20           5       13           1            1           5             2            1

                                              1              2          3                4             7             9            10        12                                                2        3            4       5            11       13              16        20               25


                                     Task Number                                                                                                                                     Topic Number




Figure 5.20 Display time according to task and topic for Subject 5

                                5                                                                                                                                               5



                                4                                                                                                                                               4



                                3                                                                                                                                               3



                                2                                                                                                                                               2
        Mean Log Display Time




                                                                                                                                                        Mean Log Display Time

                                1                                                                                                                                               1



                                0                                                                                                                                               0


                                -1                                                                                                                                              -1
                                     N=   2        3   144   10   225   1       97   8       169 216       23    8       3   13   23   20   2       2                                N = 21 1917160152 2 16 14 29 2 87 6 72 25 8 6 31 10 27 19 8 20 2 27 1 12 1 1 16 4 8 66 2 5 2 1 16

                                          1        2   3      4   5     7       8    11 12 13 14 15 16 17 18 19 20 21
                                                                                                                                                                                        10
                                                                                                                                                                                        12
                                                                                                                                                                                        13
                                                                                                                                                                                        14
                                                                                                                                                                                        15
                                                                                                                                                                                        16
                                                                                                                                                                                        17
                                                                                                                                                                                        18
                                                                                                                                                                                        19
                                                                                                                                                                                        20
                                                                                                                                                                                        21
                                                                                                                                                                                        22
                                                                                                                                                                                        23
                                                                                                                                                                                        24
                                                                                                                                                                                        25
                                                                                                                                                                                        26
                                                                                                                                                                                        27
                                                                                                                                                                                        28
                                                                                                                                                                                        29
                                                                                                                                                                                        30
                                                                                                                                                                                        31
                                                                                                                                                                                        32
                                                                                                                                                                                        33
                                                                                                                                                                                        34
                                                                                                                                                                                        35
                                                                                                                                                                                        36
                                                                                                                                                                                        37
                                                                                                                                                                                        38
                                                                                                                                                                                        39
                                                                                                                                                                                        40
                                                                                                                                                                                        41
                                                                                                                                                                                         1
                                                                                                                                                                                         3
                                                                                                                                                                                         4
                                                                                                                                                                                         5
                                                                                                                                                                                         6
                                                                                                                                                                                         7




                                     Task Number                                                                                                                                      Topic Number




Figure 5.21 Display time according to task and topic for Subject 6

                                10                                                                                                                                              10

                                 9                                                                                                                                               9

                                 8                                                                                                                                               8

                                 7                                                                                                                                               7

                                 6                                                                                                                                               6

                                 5                                                                                                                                               5
        Mean Log Display Time




                                                                                                                                                        Mean Log Display Time




                                 4                                                                                                                                               4

                                 3                                                                                                                                               3

                                 2                                                                                                                                               2

                                 1                                                                                                                                               1

                                 0                                                                                                                                               0
                                -1                                                                                                                                              -1
                                     N = 2 47 14 13 25 39 57 3          5   9   3 3 132 18 5      2        4 2   6   6 27 8 16 15 6 16 3        2   1                                N = 18    1   45 75    2   2   3   15 120 30   62    3   7   5       7   7   30   5   17       2   17   3    1
                                          10
                                          11
                                          12
                                          13
                                          14
                                          15
                                          16
                                          17
                                          19
                                          20
                                          21
                                          22
                                          23
                                          24
                                          25
                                          26
                                          27
                                          28
                                          31
                                          32




                                                                                                                                                                                                                    10
                                                                                                                                                                                                                    11
                                                                                                                                                                                                                    12
                                                                                                                                                                                                                                         13
                                                                                                                                                                                                                                         14
                                                                                                                                                                                                                                         15
                                                                                                                                                                                                                                         16
                                                                                                                                                                                                                                         17
                                                                                                                                                                                                                                         19
                                                                                                                                                                                                                                                                       20
                                                                                                                                                                                                                                                                       21
                                                                                                                                                                                                                                                                       22
                                                                                                                                                                                                                                                                       23
                                                                                                                                                                                                                                                                       24
                                                                                                                                                                                                                                                                                                  26
                                           1
                                           2
                                           3
                                           4
                                           5
                                           6
                                           7
                                           8
                                           9




                                                                                                                                                                                         1
                                                                                                                                                                                              2
                                                                                                                                                                                              4
                                                                                                                                                                                              5
                                                                                                                                                                                              6
                                                                                                                                                                                              7
                                                                                                                                                                                                                     8
                                                                                                                                                                                                                     9




                                      Task Number                                                                                                                                     Topic Number




Figure 5.22 Display time according to task and topic for Subject 7

             One-way ANOVAs were conducted to explore the relationship between display

time and task, and display time and topic for each subject. The results of these tests are



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   162

displayed in Table 5.21, which presents the F values for all significant relationships, and

η2 that was used to assess the strength of the relationship. Means and standard deviations

are reported in Appendices N and O. To aid in interpretation, recall that previous

ANOVA results only found a statistically significant relationship between usefulness and

display time for Subject 7.

Table 5.21

ANOVA Results for Display Time and Task and Topic

                                 Task                                   Topic
                 1    F(5,309) = 2.98*, η2=0.05               F(7,300) = 6.26*, η2=0.13
                 2    F(9,424) = 5.20*, η2=0.10               F(52,424) = 2.26*, η2=0.24
                 3      NS: F(15,272) = 1.19                  F(14,272) = 2.43*, η2=0.12
       Subject




                 4    F(17,226) = 2.28*, η2=0.16              F(24,225) = 2.50*, η2=0.23
                 5       NS: F(7,76) = 1.16                      NS: F(8,76) = 1.55
                 6    F(17,969) = 4.14*, η2=0.07              F(36,969) = 3.04*, η2=0.11
                 7    F(28,489) = 3.54*, η2=0.18              F(22,477) = 4.89*, η2=0.19
                                           *p<.01

         The relationship between task and display time was statistically significant for

five subjects, while the relationship between topic and display time was statistically

significant for all subjects but one. An examination of η2 demonstrates that for all

subjects, the stronger relationship was between topic and display time rather than task

and display time. It should be noted that topic appears to explain a large portion of the

variance in display time. For Subjects 2 and 4, this variance is above 20%, which is a

very high variance for social science research. Task appears to explain less of the

variance, although for Subject 4 and 7, these values approach 20%.

         Scheffe’s follow-up tests were conducted to evaluate pair-wise differences

between the significant relationships. The results of these tests are reported in Table

5.22. It should also be noted that pair-wise comparisons cannot be performed with

Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   163

Scheffe’s post-hoc test if one or more cells has only a single data point. Thus, tasks with

only one document were combined with other similar tasks, and topics with only one

document were excluded from the post-hoc analysis. Tasks that were combined with

others, and topics that were excluded from Scheffe’s tests can be found in Appendix P.

In many cases, the Scheffe’s test was non-significant, indicating that while the F

demonstrated at least one mean difference, there was not enough data or strong enough

evidence to identify pair-wise differences.

Table 5.22

Pair-wise Differences Between Display Time, Task and Topic

                                  Tasks:                                    Topics:
                     Significant Pair-wise Differences         Significant Pair-wise Differences
                                                           Trivia > History of Archaeology, Outdoor
           1                  Scheffe’s NS
                                                                   Recreation, Entertainment
           2               News > Shopping                               Scheffe’s NS
 Subject




           3                       -                                     Scheffe’s NS
           4                       -                                     Scheffe’s NS
           5                       -                                            -
           6              News > Job Searching                           Scheffe’s NS
           7                 Scheffe’s NS                       Academy Awards > Live Music



               It is clear from Table 5.22 that many of the post-hoc tests failed to pinpoint

differences in mean display time. This is most likely the result of the large number of

levels of the variable for some subjects. For instance, Subject 2 had approximately 53

topics represented in the client-generated display time data; 9 of these were excluded

from the post-hoc test because they only described a single document. While exclusion

of topics might also appear to be a plausible explanation for this lack of significance, the

exclusion of these topics did not change the overall significance of the ANOVA.

Furthermore, attributing significant differences to cell sizes of one is always

inappropriate.


Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   164

        One-way ANOVAs were conducted to investigate the relationship between

proxy-generated display times, and task and topic for each subject.                       Statistically

significant relationships were only identified for Subject 5, for both task and topic,

F(7,111) = 3.58, p<.01 and F(12,111) = 2.11, p<.01. Note that no statistically significant

relationships were found between client-generated display time, and task and topic for

Subject 5. These differences are most likely the result of the distributions of client- and

proxy-generated display times for Subject 5; 113 (62%) came from the proxy, while 76

(38%) came from the client. Although a similar split in the data was observed for Subject

1, with 535 (64%) of the display time data coming from proxy and 301 (36%) coming

from the client, the larger overall number of data points for Subject 1 probably prevented

the erroneous results.

        Further analysis of the pooled combination of client- and proxy-generated display

times and task revealed statistically significant relationships at alpha levels of .01 for

Subjects 2, 5, 6 and 7; for topic, statistically significant relationships were identified for

Subjects 1, 6 and 7.         Overall, these varying results are likely to be caused by a

combination of the unreliability of the proxy data, the varying number of data points

included in analysis, and the distribution of these data points across subjects’ task and

topic classes. Again, these results demonstrate the potential danger of Type I and Type II

errors, especially Type II errors, when using pooled data or proxy-generated data only.



5.9.2   Display Time and Task and Topic Attributes

        The relationship between display time and task attributes of endurance, frequency,

and stage, and topic attributes of persistence and familiarity are displayed in Figures 5.23



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                   165

to 5.27. None of these figures demonstrates a clear, general relationship between the

various task and topic attributes and the various levels of usefulness. However, as with

Figure 5.14, which displayed display time and usefulness, the figures do indicate that

display time varies considerably for each subject.

                                              5.5

                                              5.0

                                              4.5
                                                                                                     Subject
                                              4.0

                                              3.5                                                              1

                                              3.0                                                              2
                      Mean Log Display Time




                                              2.5                                                              3

                                              2.0
                                                                                                               4
                                              1.5
                                                                                                               5
                                              1.0
                                                                                                               6
                                               .5
                                              0.0                                                              7
                                                     1      2        3       4   5       6   7   8


                                                    Task Endurance



Figure 5.23 Mean log display time and task endurance

                                              4.0


                                              3.5

                                                                                                     Subject
                                              3.0
                                                                                                               1

                                              2.5
                                                                                                               2
                      Mean Log Display Time




                                              2.0                                                              3

                                                                                                               4
                                              1.5
                                                                                                               5

                                              1.0
                                                                                                               6

                                               .5                                                              7
                                                     1                   2           3           4


                                                    Task Frequency



Figure 5.24 Mean log display time and task frequency



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                                          166


                                              6.5

                                              6.0

                                              5.5

                                              5.0
                                                                                                            Subject

                                              4.5                                                                     1
                                              4.0
                                                                                                                      2
                                              3.5




                      Mean Log Display Time
                                                                                                                      3
                                              3.0

                                              2.5                                                                     4

                                              2.0
                                                                                                                      5
                                              1.5
                                                                                                                      6
                                              1.0
                                               .5                                                                     7
                                                      1         2       3       4       5       6       7


                                                    Stage



Figure 5.25 Mean log display time and stage

                                              6.0

                                              5.5

                                              5.0

                                              4.5
                                                                                                            Subject

                                              4.0                                                                     1
                                              3.5
                                                                                                                      2
                                              3.0
                      Mean Log Display Time




                                                                                                                      3
                                              2.5

                                              2.0                                                                     4

                                              1.5
                                                                                                                      5
                                              1.0
                                                                                                                      6
                                               .5
                                              0.0                                                                     7
                                                      1     2       3       4       5       6       7   8


                                                    Topic Persistence



Figure 5.26 Mean log display time and topic persistence




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                          167


                                              4.5


                                              4.0

                                                                                            Subject
                                              3.5
                                                                                                      1

                                              3.0
                                                                                                      2




                      Mean Log Display Time
                                              2.5                                                     3

                                                                                                      4
                                              2.0
                                                                                                      5

                                              1.5
                                                                                                      6

                                              1.0                                                     7
                                                      1        2        3   4   5   6   7


                                                    Topic Familiarity



Figure 5.27 Mean log display time and topic familiarity

        One-way ANOVAs were conducted to explore the relationship between display

time and each of these attributes. For this analysis, attributes were treated as independent

variables and display time was treated as a dependent variable. Table 5.23 displays the F

values for all significant relationships and the η2 that was used to assess the strength of

the relationships. For non-significant relationships, NS appears in the table cell. A dash

appears in the persistence cell for Subject 1 because only a single level of the variable

was observed in the documents that were described by client-generated display times;

thus analysis could not be conducted using persistence and display time for this subject.




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   168

Table 5.23

ANOVA Results for Display Time and Task and Topic Attributes

                              Task Attributes                                Topic Attributes
                Endurance       Frequency         Stage               Persistence      Familiarity
                   NS                NS            NS                                   F(3,300) =
           1    F(1,301) =      F(1,301) =      F(1,63) =                   -            16.38*,
                  0.003            2.36           1.58                                   η2=0.08
                   NS           F(2,424) =         NS                      NS           F(6,424) =
           2    F(3,424) =        9.65*,        F(2,71) =             F(7,424) =          5.51*,
                   2.26          η2=0.02          3.32                   2.07            η2=0.04
                   NS                NS            NS                      NS              NS
           3    F(3,272) =      F(3,272) =      F(3,23) =             F(1,272) =        F(3,272) =
                   1.08            1.58            .57                     .07             2.34
                F(3,226) =           NS            NS                 F(4,225) =        F(4,225) =
 Subject




           4      5.28*,        F(3,226) =      F(4,70) =               9.13*,            5.49*,
                 η2=0.07           2.72           2.39                   2
                                                                       η =0.07           η2=0.09
                   NS                NS            NS                      NS              NS
           5    F(4,76) =       F(3,76) =       F(4,76) =              F(7,76) =        F(6,76) =
                   1.57              .92           .26                   1.12              1.13
                F(5,969) =      F(2,969) =     F(6,969) =                  NS           F(5,953) =
           6      8.70*,         15.94*,         7.25*,               F(4,953) =          4.60*,
                 η2=0.04           2
                                 η =0.03        η2=0.04                  1.91            η2=0.02
                   NS                NS            NS                      NS           F(4,477) =
           7    F(6,489) =      F(3,489) =     F(6,437) =             F(5,477) =          4.68*,
                   1.97            1.36           2.30                   2.42            η2=0.04
                                            *p<.01

           As can be observed from the table, there were a number of statistically significant

differences, most notably for familiarity. Overall, stage and persistence appear to be the

least related to display time, although a few main effects were observed for endurance

and frequency as well. For Subjects 3 and 5, there were no main effects for any of the

variables. For Subjects 1 and 7, there was one main effect for familiarity and for Subjects

2 and 4, there were two and three main effects, respectively. Finally, for Subject 6, there

were main effects for endurance, frequency, stage, and familiarity. For most of these

relationships, the variance explained was quite low. As compared to the results in Table

5.21, it is clear that task and topic attributes were less strongly related to display time

Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                         169

than task and topic. It is also clear that different attributes relate to display time in

different ways, for different subjects.

               Scheffe’s follow-up tests were conducted to evaluate pair-wise differences

between the levels of each attribute and display time. The results of these tests are

reported in Table 5.24. As with task and topic, some levels of the attributes were

excluded from the post-hoc analysis because the cell size was less than one. Excluded

levels can be identified in Appendix P by locating cells which have a mean display time,

but no standard deviation.

Table 5.24

Pair-wise Differences Between Display Time and Task and Topic Attributes

                                       Task                                         Topic
                   Endurance        Frequency         Stage           Persistence            Familiarity
           1           -                -               -                  -                 3 > 5, 6, 7
           2           -              1>2               -                  -                Scheffe’s NS
           3           -                -               -                  -                      -
 Subject




           4         7>8                -               -                7>8                 2 > 3, 4, 6
           5           -                -               -                  -                      -
                                                     1 < 3, 5
           6          6<8            1 > 2, 3                              -                  6 > 5, 7
                                                     3>6>5
           7           -                -               -                  -                   4>5



               There is not enough data to identify general trends, although it appears that lower

levels of familiarity are most often associated with higher display times. For frequency,

it appears that display times are greater for those tasks that have higher frequencies. For

endurance, stage, and persistence, the results are mixed. Although the results with

respect to familiarity are encouraging, the difficulty with identifying reliable, statistically

significant pair-wise differences for the other contextual variables can be attributed to the

large number of levels of each variable and the uneven distributions of data points across

these levels. Thus, these results should be interpreted as providing support for the


Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   170

potential effects of these contextual variables on display time, rather than as establishing

firm relationships.

        One-way ANOVAs were conducted to investigate the relationship between

proxy-generated display times, and task and topic attributes. The results are not reported

here, although statistically significant relationships were identified for various attributes,

for various subjects. Overall, approximately the same number of statistically significant

results were identified when using the proxy-generated data (9) and when using the

client-generated data (11). However, results differed with regard to which attributes were

significant and for which subjects these were significant. For instance, when using the

proxy-generated data, a main effect for familiarity was only found for Subjects 5 and 7;

when using the client-generated data, a main effect for familiarity was found for all

Subjects except 3 and 5. For Subject 6, there were statistically significant main effects

for all attributes except persistence when using client-generated data, but only two

statistically significant main effects when using proxy-generated data.                    Again, the

majority of display times for Subject 5 were generated by the proxy, while the majority of

display times for Subject 6 were generated by the client. The combination of client- and

proxy-generated display times produced yet another set of results, with seven statistically

significant relationships identified. These varying results can be partially explained by

the various percentages of client- and proxy-generated display data for each subject, and

the overall number of documents that were evaluated by subjects. Overall, these results

provide further support of the importance of valid and reliable behavior-based metrics.




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                           171


5.9.3          Retention, Task and Topic

               Since there were so few occurrences of retention, it was inappropriate to conduct

statistical tests on the distribution of retention according to task and topic. However, an

examination of these distributions indicate some differences for task and topic for some

subjects. Table 5.25 displays the specific tasks and topics that contained at least one

retention behavior. For some subjects, retention occurred only for a small number of

tasks and topics, while for others, it occurred across a wider range of tasks and topics. It

does appear that documents for some tasks, especially for Subjects 1, 2, 4 and 6, are more

likely to be retained. For instance, 54% of the documents that Subject 2 retained were

associated with the task of “Academic Research,” and 45% of the documents that Subject

6 retained were associated with the task of “Research.” Because of the low number of

occurrences, retention is not discussed in relation to task and topic attributes.

Table 5.25

Tasks and Topics with Retained Documents

                 #                    Tasks                     #                    Topics
                 4   Shopping                                   4   American Indians; Entertainment
           1     3   Preparing Course Material; Entertainment       History of Archaeology; Evil
                                                                1
                 1   Researching Dissertation; Read News            Republicans; Trivia
                14   Academic Research                          4   Dams
                 6   Get Information                            3   Entertainment; Displacement
                                                                2   West Africa; Urban Growth; Buying
                3    General Interests
           2                                                        Property
                2    Shopping                                       Automotive Repair; War; Bedding;
                                                                    Classic Cars; Maps; Fire Monitoring;
                                                                1
 Subject




                1    Get Course Material; Hobbies                   People; Research Methods; Banking;
                                                                    Contraception; Coffee
                     General Knowledge; Scheduling
                5                                               4   Rutgers Facilities; News
                     Resources
                3    Entertainment                              2   Recreation; Graduate School
           3
                2    Studying Quals                                 Political Theory; Communications;
                                                                1   War/Politics; Teaching Materials; Public
                1    Weather; Teaching Stuff
                                                                    law
           4         Writing Academic Paper; Checking               Graphical Skeletonization; fMRI Brain
                3                                               2
                     University Webpages                            Images; University Information
                2    Searching for Project – Brain Images       1   Online Shopping; Graphics Books &


Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   172

             1   Online Shopping                               Papers; Campus Bus Information
       5    1    Development as a Scholar                  1   National Communication Association, etc
            10   Research                                  7   Finite Element Method
             6   Project in Operating Systems              4   Theoretical Analysis in Mechanics
             3   Job Searching                             3   Simulated Banking Service
       6                                                       Applications of Modern Technology in
            2    News                                      2
                                                               Traditional Fields
                                                               Thread Management in Operating
            1    Database Systems                          1
                                                               Systems; Microsoft Access
            2    Setting up Personal Website               2   Literature Departments
       7         Writing Reviews for Amazon; Check             Campaign Finance; Current Events;
            1                                              1
                 News; Shopping                                Online Auctions



5.10       Predicting Usefulness

           The final question of this dissertation was concerned with how a user’s document

preference, in this case usefulness, could be predicted from observable behavior, in this

case, display time and retention.               The previous analysis demonstrated little direct

relationship between display time and usefulness; indeed, display time appeared to be

more related to characteristics of the information seeking context than to usefulness.

Thus, this section explores the interaction between elements of the information seeking

context, display time and usefulness, to identify what types of information might be

necessary to effectively predict document usefulness from display time. Because there

were no significant results with respect to retention, it is excluded from the analyses

reported in this section.

           Two-way ANOVAs were used to explore the relationships between display time,

task and usefulness, and display time, topic and usefulness. No statistically significant

relationships were found for any combination of the variables for any subject. An

analysis of the proxy-generated data found statistically significant interaction effects for

task and usefulness for Subjects 5 and 7, and for topic and usefulness for Subject 5. An

analysis of the combination of proxy- and client-generated data found many more


Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   173

statistically significant interaction effects for both task and usefulness and topic and

usefulness.

        Two-way ANOVAs were used to explore the relationship betweens display time,

usefulness and each of the five attributes of task and topic: endurance, frequency, stage,

persistence and familiarity. No statistically significant relationships were found for any

combination of the variables for any subject. An analysis of the proxy-generated data

found statistically significant interaction effects for endurance and usefulness for Subjects

5; frequency and usefulness for Subject 7; stage and usefulness for Subject 5; persistence

and usefulness for Subjects 5 and 7; and familiarity and usefulness for Subjects 1 and 5.

An analysis of the combination of proxy- and client-generated data found similar

interaction effects to those computed using proxy-generated data, most notably for

Subjects 1, 5 and 7, where the majority of display times came from the proxy (64%, 62%

and 54%, respectively). Again, the uneven distribution of client- and proxy-generated

display time data for Subject 5 was most likely a factor in the number of significant

relationships identified for this subject when the proxy-generated display time data is

analyzed in isolation, and in combination with the client-generated data. In other cases,

the source of the data did not matter: for Subjects 3 and 4, no interaction effects were

identified regardless of the source of the data.



5.11    Summary of Results

        Overall, the results presented in this chapter indicate that there is no direct

relationship between display time and usefulness, and that display time varies according

to some aspects of information seeking context. Results further demonstrate that there is



Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.
                                                                                                   174

no correlation between display times collected from a proxy and display times collected

from a client, and that Type I and Type II errors are likely to result from using proxy-

generated display times alone, or in combination with client-generated display times.

        Display time was found to vary according to task for all subjects but two, and

according to topic for all subjects but one. Further, display time was found to vary

according to endurance, frequency, stage, and persistence, in different ways, for different

subjects. For familiarity, display times varied in a consistent direction for almost all

subjects, with lower levels of familiarity associated with higher display times. Results

also showed that retention was not always a good indicator of the documents that

subjects’ preferred, and that subjects were more likely to retain documents belonging to

particular tasks and topics. In trying to determine which variables are important for

predicting usefulness, no significant interaction effects were found between task, topic

and usefulness, nor for endurance, frequency, stage, persistence, familiarity and

usefulness.

        In sum, the results of this study suggest that for an online information system to

use behaviors to implicitly infer document preference, it is necessary for the system to

model the user’s information-seeking context, and that approaches to modeling should be

personal rather than general. Furthermore, the integrity of behavioral-based metrics used

by the system is an important and significant issue, which deserves special attention.




Kelly, Diane (2004). Understanding implicit feedback and document preference: A naturalistic user study.
Ph.D. Dissertation, Rutgers University.

				
DOCUMENT INFO
Categories:
Tags:
Stats:
views:6
posted:10/24/2010
language:English
pages:63