Document Sample
CASE Powered By Docstoc
					System Prototype
 Evaluation Plan
             Part 3
      Winter Quarter 1999


       Toya Nicole Baggett
        Christoph Gauger
         Jan Hannemann
    Anand Lakshminarayanan
     Barbara Patricia Napper
CASE – System Prototype and Evaluation Plan                                          -2-

1. Project Description

1.1. Overview
The Computer Aided Shopping Expert (CASE) serves as a home shopping assistant. It is in-
tended to help making the task of shopping easier particularly for busy, elderly and handicapped
people and families.

CASE is a convenient system, which lets people shop from the comfort of their kitchen. This sys-
tem provides facilities to make up shopping lists, create and manipulate Meal groups (Meal Plan-
ning), Browse the store inventory for including items to the shopping list, support serendipitous
shopping, order online and receive immediate feedback about orders from the store.

With CASE, waiting in long lines at the cashier, dozens of tiny little shopping notes, forgotten
products or shopping with small children will belong to the past. The system saves a lot of time
for the busy shopper, which can be better utilized. In countries where shopping hours are re-
stricted (e.g. 9am - 8pm) it will be of great help to all-working people.

The user can customize the system to his personal shopping needs. CASE also keeps track of user
preferences for various products, commonly ordered goods, and “grouping” items together for
convenience and clarity.

1.2. Description of Final Design
Our final design is based upon an evolved version of the voice interface plan discussed in the
second project. It encompasses many of the features addressed previously, although we have
modified some aspects of the design in order to address some of the difficulties expressed by our
potential users and our audience at the poster session.

We have decided to use a screen output device which will provide many graphical aspects re-
quired to convey illustrations of products. In addition, as the user will be wearing a headset to
communicate voice commands, he will also receive audio feedback in most cases. The audio
feedback will be optional, as it may become distracting to have the system repeat several func-
tions back to the user along with many other activities occurring in the kitchen. The combination
of audio and visual output must be coordinated such that the user can get a clear understanding of
system feedback whether using the screen directly or wandering around the kitchen, remotely
interacting with our system.

Input will be channeled to the system through a strict dialog of potential commands combined
with a wide variety of products which the user must specify to a recognizable degree. The voca-

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                             -3-

bulary allowed by the system will be dictated according to the menu design of the system. More
specifically, the user does not have at his disposal an entire list of all possible verbal commands
for the system, instead a subset of the system language is available depending on the relevant
"rooms" that the user visits. At all times the available commands are displayed, with some anno-
tated buttons indicating that the systems expects additional parameters such as product name, in
the event of an Add. Some of the difficulties with this dialogue design that were expressed by
potential users are included in the discussion of our evaluation plan.

Again the data mountain metaphor is used to interact with the user, although several modifica-
tions have been made. In general the room is used to graphically convey a 3 dimensional aspect
of our products, with physical grouping rather than lists of products as the touch screen interface
design recommended. The change comes however in the representation of tables on which to
place the food items. Several complaints with our previous design were that the food items in the
room metaphor were placed on the floor, and association that many users did not wish to make.
The addition of tables, however presents other problems since we need to view as much area as
possible without having products at the front mask the products behind it. It is for this reason that
we changed the orientation of the view to see the room from slightly overhead.

Other changes included making an appropriate interface for browsing, an issue that was ad-
dressed by the touch screen group in a list, but must now be incorporated into the graphical data
mountain metaphor. This task describes the spontaneous shopping method in which items are
displayed according to sales or group according to main product groups. The browsing room was
created with the metaphor of the grocery store shelves in mind. Various products are displayed on
shelves and are organized either by sale items, or food groups, and are further subdivided when
the user verbally selects an item, such as chicken, in which shelves are modified to show the vari-
ous items available. Since the browse room is separate from the list room, we have a temporary
shopping cart in which items selected from the store inventory are placed. Items in this cart are
merged to the shopping list upon returning to the list room. As with the meal groups, we have a
similar problem with consistency when merging two lists together. For example, one problem
that can arise is duplicating items, which involves either increasing the quantity of the item or
perhaps increasing the product size. Again the user must be prompted to address these issues.

The final design of the CASE system consisting of a screen output and voice interface, is consis-
tent with many of our initial usability goals. The data mountain is a model which allows CASE to
graphically represent store items in excruciating detail, an advantage over the mostly text based
touch screen interface. However some clarity is lost since the user must now articulate his or her
choices in a clear manner. Feedback generated by our interface will help guide the user and clari-
fy ambiguities. In general CASE is a learnable system as it models current shopping methods, but
also offers some additional features, in meal planning, customizable store inventory, and default
shopping lists, all of which make shopping a little easier.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                           -4-

2. Summary Design Rationale

Throughout the design process we evaluated and then prioritized, added and adapted require-
ments and usability criteria. We learned more about the tasks, the users‟ way of accomplishing
their shopping needs and the technology to be applied.

During the requirements collection phase we thought of a system that would basically perform
the everyday shopping functionality - making up a list and ordering.
From the task analysis, the analysis of the present system, from a questionnaire and interviews
with potential users we found out, that there are a lot of tasks around this pure shopping. Only by
integrating and providing this functionality in an understandable way, CASE can be the desirable
tool we wanted it to be within the intelligent house. On top of that, CASE has to address all the
problems arising from the harsh environment it works in.

In the following we describe how our final design developed and how we reached these deci-

2.1. Our Users, the Environment and Usability
We are designing for a typical American family unit, which may or may not include children.
Thus the language of choice is English. From typical users we expect the following characteris-
   Literacy
    We expect users to have a moderate reading level, since they have to read dialog boxes and
    the specification of products if they want to. However, to make the interface more intuitive
    and to support users with low reading skills we include graphical representations of products
    and shopping items. Basic understanding of measurements is also required so as to decide the
    quantity of an selected item.

   Computer skills
    CASE does not require you to be an expert computer user. Some familiarity with basic termi-
    nology (“Lock”, “Exit”, etc.) helps but is not at all assumed. The system does not preempt the
    user, so the initiative for a dialogue lies with the user.

   Physical skills
    As we use a voice interface with graphical output the user should have usual capabilities con-
    cerning vision, hearing and verbal expression.
    As the device is mounted to a surface but may also be detached and used over a wireless link
    within the house, the degree of motor skills necessary depends mainly on the user‟s habits in
    shopping with CASE.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                           -5-

   Cognitive abilities
    Because the shopping decisions made with CASE definitely have real world impact with or-
    dering (the purchase is billed to your credit card) users should be able to make these deci-
    sions. Hence, we included an authentication feature so that the right to use CASE can easily
    be restricted to certain family members.

The system will operate in the kitchen environment. It may be exposed to dirt, grease, wetness
and splattering of oil etc. Background noise and frequent interruptions are also characteristics of
the kitchen environment. This harsh environment influenced several of our design decisions and
finally led us to the hands-off voice interface.

As the system will be applied in every-days household environment, CASE has to have an easy to
use, comfortable, reliable, quite fast, flexible and safe device. As such a specialized system that
also appeals the expert user cannot be completely intuitive like an airport kiosk should be we
stressed learnability as a usability criteria. With the authentication we achieve safety, with the
support for frequently purchased goods and customized product selections we support speed and
convenience for shopping. On top of that CASE helps you to accomplish such complex things as
meal planning and buying things when on sale.

2.2. Technological Properties
For the technological aspect of CASE we decided to incorporate a voice controlled graphical in-
terface. Most of the principles and features argued upon here will be illustrated in the scenarios
and prototype sections.

   Voice
    A voice interface gives us the freedom to use CASE hands-off. This is a great advantage in
    the harsh kitchen environment. Roaming is made possible by a wireless headset. So we still
    have high audio input quality even if the user is walking around the kitchen, checking his cab-
    inets. But a voice interface also poses the challenge how to manipulate the graphical list re-
    presentation in an easy and error save way. This will be described in a section below.

   Screen
    We decided to use a quite big (approximately 15”) but light flat graphics display as an output
    and feedback device. This is essential as our users need extensive and persistent feedback and
    guidance in this environment. Audio alone would not be sufficient because a user can be in-
    terrupted every other second. On top of that an expert user can perceive the system state after
    an action much faster visually than by waiting for an complete audio feedback. So we aimed
    at providing a clear system model and easing the gulfs of execution and evaluation.
Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                            -6-

   Communication Environment
    To be able to integrate CASE in all kitchens without major changes to the electrical installa-
    tion and for the sake of user convenience we decided to use wireless connections for all
    communication. So the headset, the central processing machine, an optional remote control
    device (like a PDA) and a usual PC printer will be addressed over a digital wireless channel,
    either indoor or for the PDA via a commercial cellular network.

   Physics
    The screen will be mounted to a surface. To reach a position that is convenient for all users no
    matter how tall they are, the CASE screen can be adjusted in mounting height and angle.
    It will be waterproof and therefore easy to clean (a dishwasher proof device will be the chal-
    lenge for further research).

   Authentication
    To restrict the system access to a number of authorized users we use their (acoustical) voice
    patterns for an authentication procedure. The screen can be locked at any time and easily un-
    locked just by greeting the system.

2.3. Metaphor and Manipulation

In this section we describe the overall metaphor we use for CASE, for its representation of prod-
ucts and its control. Illustrations provided with the prototype will give you a real-world expe-
rience on this metaphor.

To give the user a clear graphical representation, that provides grouping capabilities and also dif-
ferent degrees of importance we implemented the data mountain technique proposed by Micro-
soft™. The data mountain representation is based on a three-dimensional room, in which items
are presented in groups. The room and the tables/shelves we introduced into it give us the oppor-
tunity of grouping items even more clearly. We can also arrange them following the present im-
portance using the depth of the room. With the graphical representation of products as icons we
take the burden from users to read through textual lists.

There are three different rooms within the CASE system: The “List Room”, the “Meal Room”
and the “Browse Room”. While the first manages the shopping list itself, the second supports
meal planning and the latter gives you the opportunity to browse the store inventory. This brows-
ing includes all the flavors of today‟s grocery shopping. You can see sales and purchase items

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                             -7-

Unlike the original data-mountain idea, we manipulate the icons with voice instead of with some
pointing device. Actually we found out that most of the manipulation is performed on the entire
list rather than on a single item or group. For example, the grouping of goods in food-groups is
given or customized, you don‟t have to care about this when adding items to the list.

The screen shows the room and two bars which show all the commands available in this context.
The bar at the left screen border contains commands related to this special room. In the bar at the
bottom you find commands that control the general behavior of CASE (EXIT, Lock, etc.). We
print the commands in speech bubbles to show the user that these are the voice input options.
Bubbles that contain the “…” will prompt you if you do not complete the command properly, e.g.
“Add …” by a product name. We had to accept more commands on the screen at the same time to
achieve the goal of showing availability of commands and of giving feedback.

In the room we give intensive feedback: In the right upper corner you find the present “Total”,
the command bubble and the item most recently selected are also high-lighted. The group that
was affected by the most recent operation will have moved to the front of the room.
The feedback is designed to clearly represent the state of the CASE system. You can immediately
see the effects of your commands in the room. In addition you hear verbal responses. Error re-
covery with the “Undo” command is easy and always available.

During the design of the voice interface we had long discussions regarding the feasibility of a
special vocabulary and if adopted, how many words should be available. As the system will most-
ly be used over a longer period of time we think the burden of having a restricted vocabulary is
acceptable. The novice user can always find the commands on the screen, the expert user will
memorize them and use them in a natural way. The notion of a vocabulary is only applied to
commands, for products CASE recognizes all descriptions for the products available.
From this decision we gain a more robust interface in the noisy kitchen environment. We intend
to reduce substitution errors and insertion errors by keeping rejection and deletion errors low at
the same time. This is possible because the usual trade-offs are based on the assumption that you
either reject a word or risk substituting it. With a restricted vocabulary we decrease the number of
similar words for a certain situation. The same is true for the insertion/deletion trade-off.

Anyway, the voice interface will be smart enough to understand variations of a command and still
perform the right operation. So users can say “Edit a list”, “Edit my list”, Edit list” etc. and will
always get the same result.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                               -8-


For this final version of CASE we decided to support the following functions: Making up a list,
meal planning, browsing and ordering. Apart from these specified tasks, there are some minor
support functions we do not describe here in detail. It is, for example, possible to print out the
shopping list or your order receipt in a detailed textual form.

We narrowed the design space by analyzing the present system for shopping, interviewing poten-
tial users and comparing the benefits of various options. The following paragraphs will only give
a short overview over the functionality implemented. A more detailed illustration is given with
the prototype and the scenarios.

   Making up a list
    The items in the room are grouped on tables. Each group carries its name. There are different
    views the grouping of the room can look like: Grouping in food-groups, locations in kitchen
    cabinets and in meals.
    You can add and delete items, let CASE show you groups or the properties of special prod-
    ucts. In adding items you can decide whether you want to specify the item or take your pres-
    pecified selection.
    We decided to make these frequent operations fast (“make the common case fast”) but still
    give enough feedback to prevent errors effectively.

   Meal planning
    We included this set of features to support the complex task of shopping for meals. The prob-
    lem is to come up with a robust strategy for merging meals with the list and to guide the user
    in case of the unavailability of certain items within a meal. We solved this by keeping the re-
    lations between products and meals even for quantities smaller than package size. So CASE is
    able to show these consistency problems to the user in case of unavailability.

   Browsing
    To allow our users to shop any product in the store‟s inventory, we included a special browse
    room. Because customers usually only select form a subset of the inventory, we call this their
    favorites, this browse room can be used to browse the complete store inventory or to go
    through your n most frequently purchased goods, your favorites. This more shallow structure
    helps you getting to a certain item faster.

   Ordering
    For the ordering we decided on a one time dial-up connection with secure transmission. At
    this time the list will be transferred to the store the user subscribed to. There will be an avail-
    ability check immediately. If there are some problems, the user will be prompted to react and
    decide if the products should be substituted or cancelled.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                          -9-

3. Prototype for the List Management
Shop with Melanie Gale
For the last ten years, Melanie Gale has been the president of the weight-watchers association in
Houston, Texas. This has changed her life quite a bit: She became a rigorous vegetarian and
spends all her spare time on a private crusade convincing people to become vegetarians and live
healthy (including her husband Tom, who has refused so far). Her busy life leaves her little time
to do the shopping and since her husband refuses to do it, she bought the famous CASE system
recently. Right now she is going to restock, planning to get some vegetables, fruits, and a frozen
pizza for Tom.

As she approaches the system, it is in sleep mode. She activates it by saying "Hello CASE!". The
system recognizes her voice and responds with "hello Melanie!", showing the main menu. She
quickly selects the list room to add item to her shopping list by saying "list room". CASE dis-
plays the List room with her current shopping list:

Thinking about the pepperoni pizza Tom asked her to order, she shudders but decides to order his
stuff first. "Add pepperoni pizza", she says and CASE responds with "frozen pepperoni pizza
added", moving the table for frozen/canned food to the front and adding the picture of a pizza.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                           - 10 -

"Add two tomato soups" is her next command, and CASE responds with "two tomato soups add-

That's it for Tom, Melanie thinks and moves on to her personal favorite: ordering fruit and vege-
tables. As she says "add cauliflower", the table for fruits/vegetables moves back to the front, dis-
playing her current selection plus a fresh cauliflower. CASE responds with "cauliflower added".
In the same manner she adds four bananas, a green pepper and some other fruits and vegetables
she likes.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                         - 11 -

Looking at her current order, this terrific fruit salad from last week comes to her mind. However,
she would need another pineapple to prepare it. To change the quantity, she says "show pineap-
ples" and a box pops up that contains all specifications of her pineapple choice so far: Quantity,
brand name, etc.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                           - 12 -

The system starts reading the specifications to her "Pineapples - your choice is two ...", but Mela-
nie interrupts the voice output with: "change quantity to three pieces". Dutiful, CASE responds
with "quantity changed to three", and the information in the specifications box changes appro-

Satisfied, she says "OK" and reviews her planned purchase. After checking the items, she moves
on to order them, but that is another story...

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                         - 13 -

4. Prototype for Meal Planning
Cook with Iris Romaine
Iris Romaine loves to cook. Everyone in her family knows of her culinary skills and they fre-
quently give her recipes, with the hope that she‟ll share the fruits of her labor with them.

On Sunday, she‟s having several cousins and an aunt over for old Uncle Charlie‟s birthday bash.
They‟ll be having spaghetti with spicy tomato sauce and lots of garlic bread, sausages, fresh sal-
ad, and wine – his absolute favorite dinner, and a recipe passed down to Iris from his mother!

She‟s already picked up the wine, sausages, and some hard Italian bread, now she needs to check
the spaghetti meal she‟s stored in CASE, to be sure she‟s ordered all the key ingredients.

She goes into the kitchen, says “Morning CASE!” and gets the now familiar “Good morning
Irene” and the CASE welcome screen. She knows this means that the system had completed an
automatic voice authentication check on her. Selecting from the displayed choices, she says “Plan
a meal” and sees her favorite room, the Meal Room, come up on the screen. She looks at the ar-
ray of bubbles displayed outside of the Meal Room and says “Open the Spaghetti Meal”.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                           - 14 -

She used to worry about saying the wrong thing to the system, because she‟d always had a small
fear of machines, but after owning this system for two months, she‟d become more comfortable
with it. She reflects now, on how natural it feels to just say what you want to do, rather than hav-
ing to type things in on a keyboard, or something.

She checks the Spaghetti Meal Group for all the ingredients she‟ll be needing for the spaghetti
sauce. She has to smile when she sees the flaming icon for the hot chili peppers, something she
already purchased last week for her shrimp creole. She does need some fresh mushrooms, how-
ever, so she immediately says “ Select mushrooms”, and the icon is highlighted in a hazy shade of
yellow. She next selects spaghetti, then finishes with the Parmesan cheese. That is a must! Uncle
Charlie has to have freshly grated Parmesan, or the meal is over for him! She supposes, at the
grand old age of 90, his eccentricities can be excused.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                            - 15 -

That‟s it, then. Time to merge the list and get going with the garlic bread. She quickly scans the
displayed total, and tells CASE to “Merge” the list to the current shopping list. As usual, she in-
tends to check to see if everything she selected is now displayed in the list room. It doesn‟t hurt to
be careful. Even now, the Merge function seems a little mysterious to her.

At this point, CASE reminds her that she‟s got an existing order for “12 ozs of Parmesan cheese”.
As one option, the system suggests combining it with the new Spaghetti Meal order for “6 ozs of
Parmesan cheese”, and getting the economical 1 lb pkg - the available size in her favorite Bertoldi
brand. She really doesn‟t want to skimp for Uncle Charlie‟s birthday, though, so she decides to
go with the alternative of two 1 lb pkgs. She knows that Parmesan cheese will never go to waste
in her house.

She checks the items in the List Room, noting that icons for each of her selected items are now
shown in the Spaghetti Meal List Group. Satisfied, she says “Exit”, and turns to get the bread out
of the fridge. She uses a special blend of fresh garlic, oregano, and …. uh oh, she‟d forgotten to
order some fresh garlic! And it had been right there under her nose the whole time. Oh well,
she‟d just have to wake CASE up again and get some garlic. Garlic bread without garlic is like a
chicken without lips – it has absolutely no bite!

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                            - 16 -

This time she goes to the List Room and quickly updates the list by saying “Add 8 ozs of garlic”.
When CASE confirms “8 ozs of garlic added to the list”, she goes ahead and places the order. She
needs the garlic now so, for an extra $5, she opts to have everything delivered in the store‟s 2
hour minimum delivery time. CASE confirms the delivery date and time, telling her that “all
items on the list are available as ordered”. The bread will sit in the freezer for a couple of days to
absorb all the potency of the garlic and spices. Hopefully it‟ll be as delicious as ever!

5. Prototype for Browsing
The Weekend Party
Sylvia walked up the CASE system in her kitchen. She started the system up with her usual “Hi
CASE”. Immediately the CASE system came up with a big welcome message on the screen
“Good Evening, Sylvia”. Sylvia smiled contently. The CASE system had made the shopping ex-
perience much more enjoyable, interesting and easy.

Sylvia was in an upbeat mood today. Her husband John had just burst in with the news that he‟d
promoted at work. They were excited! John wanted to throw in a big party this weekend to cele-
brate the occasion.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                         - 17 -

Sylvia was doing their shopping for the weekend party through the CASE system. Right now, she
looked at the List Room view on her CASE system. She proceeded to include all the items that
she wanted for her party to her shopping list, using the CASE speech interface.

The CASE system comes with online store browsing capabilities, she knew. But she‟d never real-
ly used that feature so far, in these few weeks that she has owned the system. No time like now to
try out the browse features on CASE!

“Change to Browse Room” she said to the system. CASE displayed a view of the “Browse
Room” which consisted of food items of broad food categories placed on shelves of the “virtual”
store. Vegetables, Meat, Canned Foods, Diary, Snacks…she read off the screen. Sylvia remem-
bered her friend telling her that Lay‟s had come out with a new variety of chips just for parties.
But she couldn‟t remember the name of the product.

Here goes, she thought. “Snacks” she said.
Now, she saw a shelf of Snack items on her screen. She picked out chips in that list . “Chips” she

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                       - 18 -

All chip varieties carried by the store, came up on this new list on the screen. She scanned the
icons quickly. Lay‟s classic, Lay‟s Barbeque….. Lay‟s Toasted Onion and Cheese! The icon said
“New” and she knew this was what she‟d been looking for.

“Add Lay’s Toasted Onion and Cheese”.

The system came up with the message “ Lay’s Toasted and Cheese Not found”. Sylvia realized
that the system had omitted a word in her previous command. She said it again. “Add Lay’s
Toasted Onion and Cheese”.

The system popped up a dialog box which asked her “Size?” and listed the various possible Sizes

Sylvia chose the big packet. “12 Oz” she said.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                            - 19 -

Another dialog box prompted her for the number of packets. When she‟d chosen “7” packets, a
final box appeared on screen indicating that she‟d added

7 nos. of 12Oz, Lay‟s Toasted Cheese and Onion for $25.83 @$3.69ea. added to Shopping Cart

Now Sylvia decided to add some beans to her shopping list. She knew that she could directly
browse for beans instead of going through the various food category shelves that the system pro-

“Beans” she said. The system came up with “Beans or Beef? ”. She realized that the system
hadn‟t heard her clearly. “Beans” she clarified again. This time, she got a list of the various types
of beans available at the store. She chose the “Fresh Beans” and added this to her list specifying
the Amount/Size to be bought.

Syliva waned to check her list. “Show Shopping Cart” she said. And CASE displayed her the list
of current items on her Shopping Cart. She could see icons for her Chips and Beans in the list.

She got back to browsing and continued to browse the store to her heart‟s content and finished
making her shopping list. Now, she proceeded to order the list from her store. She was done,

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                            - 20 -

shopping for the party! Amazing she thought, and thanked the developers of CASE in her mind
for coming up with such a useful and convenient system.

6. Prototype Assessments
6.1. List Management Prototype
In general the graphical representation of products on the list is very clear. The user can see ex-
actly what items have been added to the list, as well as the current total for these items. Based on
user feedback, it is clear that there are some ambiguities with the design of the buttons. An initial
think aloud session revealed that users had some difficulty determining the differences between
some of the commands, such as Add Special and Add. We feel that this distinction can be clarified
and is quite learnable with minimal instruction. In fact in our previous designs we had one more
Add Common option which yielded a total of three different add buttons and it was difficult to
keep track of which button would produce the desired result.

Other aspects of the design that caused problems consistently for users was the “triple dot” nota-
tion which gives the user some idea that they should say something in addition to the commands,
but they were unclear at times the exact vocabulary necessary to convey their selection to the
system. Graphically CASE offers no suggestions, until it encounters some error and the product is
not understood or found in the product inventory. Many users offered suggestions such as dialo-
gue boxes, or drop down menus which suggested available commands. One such example is the
Change View option. The user may not be aware of the three different types of views. Each view
will change the grouping of the items on the table. More specifically, group by Meals, group by
food categories, or by location such as refrigerator, freezer or pantry. As a point and click inter-
face would offer a list of possible suggestions, the voice interface should incorporate these op-
tions as well to bridge the gap in the gulf of execution that this voice interface presents.

It was clear from each user evaluation technique that the feedback was plentiful, whether pre-
sented in audio or visual, but most felt that the pure graphical interface needed to offer more clues
to its usage. In some cases more descriptive menu buttons are effective enough to offer additional

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                            - 21 -

6.2. Meal Planning Prototype
Our Meal Planning design was meant to provide the user with a more natural and complete way
of doing his/her shopping. Not everyone uses a formal approach to planning meals, but for those
users who do, it‟s important that this capability is provided by the system.

The tradeoff between making possible meal planning actions visible in the interface, and
inundating the user with too much information, was an important one. The whole meal planning
issue is so detail oriented, that we thought it necessary to include more bubbles than we would
have liked to. Yet it appears from the results of our cognitive walkthrough, that we didn‟t include
enough. For first-time users, it‟s important that the interface provides enough information for
them to be able to translate their intentions into doable system actions. At the same time, too
many options can tend to cause user confusion. We think the number of visible command options
we decided to display, is a relatively successful attempt at solving this dilemma. More (future)
user feedback would be very helpful in resolving this issue. We are by no means done with our
evaluation phase in the CASE system.

Another initially thorny issue, was that of whether to provide online recipes. We opted not to. Our
decision was based on the fact that there are so many possible versions of any given recipe, that it
would create either an undesirable amount of ambiguity, or a headache for users tying to make a
quick choice. Obviously, this is a decision that can very easily be revised later, when more
fundamentally important considerations have been taken into account. We finally decided that it‟s
actually more of a nice-to-have-for-certain-users kind of feature, and we did not waste any more
time agonizing over it.

Like we discussed in Part II, we think the data mountain representation was ideal for meal
planning. It allows users to specify, see, recognize and manipulate a multitude of individual
items, with high quality graphics, and very recognizable icons. We believe this medium helps the
user feel like he/she is in control of the system, since the meals are exactly what the user likes,
and are further customizable at the user‟s whim. The ability to effect changes by using the very
natural speech interface, should encourage the user to quickly become comfortable with the

Lastly, the system‟s ability to remember details of the user‟s past preferences and choices ought
to make it a lot easier for someone in a hurry to simulate a real life “rush” scenario. For example,
if there‟s no time to check individual items, either in the cabinets or in a particular meal group,
one could simply say “Add a certain Meal”. All the Meal ingredients would then be added to the
list, saving the harried user a lot of frustration and time. This, of course, might not be the way to
go in every situation, but is a viable option when time is of the essence.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                          - 22 -

6.3. Browsing Prototype
An assessment evaluating the Browsing capabilities of the system was performed using the Think
Aloud evaluation method.

In this assessment plan, a novice user is asked to use the CASE system to browse the store
inventory and add items to the store list. At every step, the user expressed his views on actions
undertaken to achieve his goal, interaction with the system, system responses and ease of usage.

Browse the store for Potato Chips (Lay‟s) and add it to the shopping list.

Jim Wilkinson, Bill Brady, Srikanth Patury, Manoj Philip, Thomas Classen

The metaphor of the shopping room was immediately apparent to the users who “tried” our
system. The transition from the Main Menu to “List Room” was understood easily. Once in the
“List Room” users instinctively tried the “Add” option to perform the given task. After realizing
that their task of browsing the store was different from simply adding an item to the list, most
users took a few seconds to hit upon the next step.

From the command bubble “Change Room …” most users gathered that they should experiment
with this option, to achieve their desired objective. In the options listed for “Change Room”, they
located “Browse Room” and were immediately presented with the Store‟s inventory.

The metaphor of the store as just another room on the system, seemed logical to users and this
helped them in understanding the layout of the system better. We have assumed that users who
would browse the store, have already used the system to create and manipulate a list on the
system. Though this assumption may not hold true all the time, it presents a tradeoff between
ease of usage for the novice user and learnability. The concept of a “Browse Room” provides
logical consistency and helps in improving the learnability of the system.

Users were then able to select “Snacks” from the browse room shelves, choose “Chips” from the
list of snacks and reach the shelves for “Potato Chips” with ease.The command bubbles‟ on the
menu ( on the left of the screen) helped them choose “Add Lay‟s Sour Cream and Onion chips”
and add this to the shopping cart. The response from most users seemed to indicate the system
was effective in providing a easy-to-use interface for browsing tasks.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                            - 23 -

7. Evaluation Plan
7.1. Heuristic evaluation
This technique would have served us better, had we formally used it earlier in our design process,
say in Part II. At that time, when we were struggling to refine our usability criteria, it would have
been helpful to employ the skills of five or six expert evaluators. The ten heuristics provided in
the text, describe basically the same criteria we‟ve been trying to use as we iterated through the
various stages of CASE‟s development, and extended and modified our initial ideas for the
system. As the project grew, so did our expertise in recognizing what design deficiencies
translated directly into usability problems. We learned to develop a sort of radar for system
shortcomings that would make CASE less flexible, less learnable, or error-prone for our potential

In fact, as we reflect on the entire process, it seems that we were, albeit unwittingly, the heuristic
evaluators of the CASE system. The only heuristic that we did not consider very closely, was the
tenth: “Help and documentation”. That would have come in the next phase of our work. We tried
to provide built-in help functions by making the system menus and visual aids self-explanatory,
even for the novice user. But of course, a lot of work remains to be done in that area. Had we
been continuing further with CASE‟s development, we would certainly have brought in some
evaluators to critique our system against Nielsen‟s heuristics.

7.2. Cognitive Walkthrough
The Cognitive Walkthrough evaluation technique was used by two former HCI students to step
through the action sequence involved in adding frozen pizza and two pineapples to the shopping
list. To include error consideration in the walkthrough, the task includes later changing the
number of pineapples three. The evaluators‟ job was to walk through the sequence of necessary
actions, checking for potential usability problems, and trying to determine how easy it is to learn
CASE. The emphasis was on having them learn the system by exploring its functionality, based
on the prototype they were given.

The two evaluators used the following forms to document stories of their exploratory walk
through the system: the Cognitive Walkthrough (CW) cover sheet, the CW evaluation form, and
lastly, the CW usability problem report form. An example of each of these forms is presented
below (for Action A).

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                           - 24 -


Date March 9, 1999                   Time 10:00 am


Tanisha Hall
Shanda Harper

Representative Task: T1

Use the system to add the following items to the shopping list:

Add list items
Pepperoni Pizza

Modify item
Change the quantity of pineapples to 3.

We assume the user speaks, reads, and understands English and has a basic ability to recognize
and distinguish specific screen images. We also assume this to be the user's first time walking
through the system, so no available "short-cuts" are invoked. (For example, an expert user would
change several properties of a shopping list item in one fluid action.)

All user actions are specified by "Action" and system responses are described by "Response".

Action A: Greet the system by saying "Good morning Case".
Response A: System authenticates voice and replies "Good morning username". The MAIN
screen appears, showing six bubble prompts with allowable user commands.
Action B: Say "Edit the List".
Response B: CASE says "Entering List Room", and the List Room is displayed, with a
highlighted back wall "List Room" label. Groups of items are displayed on tables, and the
shopping list's current total is displayed. Several "control" bubbles are shown on the side and base
of the List Room indicating system-understandable user commands.
Action C: Say "Add Pepperoni Pizza".

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                        - 25 -

Response C: CASE realizes that the user has used the system to order Pepperoni Pizza before.
From its records, CASE pulls out Brand name, Quantity, and other information related to
Pepperoni Pizza and assumes these values for the current session. The "Add…" menu bubble is
highlighted and the Pizza icon in the “Frozen/Canned Goods Group” is highlighted. The group‟s
table is shifted to the front of the room.

Action D: Say "Add Pineapples".
Response D: CASE has a previous record for Pineapples too. It retrieves this record and finds
information about Brand Name, Quality, Amount etc. The “Add…” menu bubble is selected and
the Pineapple icon in the Fruits/Veggies Group is highlighted. This table moves to the foreground
indicating the result of the user‟s action.
Action E: Say "Show Pineapples".
Response E: Case highlights the Fruits/Veggies group (which is at the front of the current
display), highlights the "Show…” menu bubble, highlighting the Pineapples icon. The System
also opens an information window showing the information about the fruit selected, including
quantity, brand and price.
Action F: Say "Change quantity to 3".
Response F: CASE highlights the "Change…" bubble, says "Quantity changed to 3", and updates
the window with the new quantity highlighted, and the updated total price including this selected
Action G: Say "OK".
Response G: System highlights the "OK" menu bubble, then closes the Pineapples window; says
"3 Southern Sun Pineapples have been added to the shopping list" and increases the shopping list
total by $1.79.
Action H: Say "Exit".
Response H: CASE says "Goodbye username", closes the List Room, and returns to the CASE
start-up screen.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                            - 26 -


Evaluation Form

Representative task: T1

Action C: Say “Add Pepperoni Pizza”.

Instructions: For the stated action, critique the CASE system by writing a story about why that
action is, or is not good for a new user, based on the following four usability questions:

Will the user be trying to produce the effect described in Response F?

Will users be able to notice that Action C is available?

Once users find Action C at the interface, will they know that it is the right action to
produce the effect described in Response F?

After Action C is taken, will users understand the feedback they get?

Please help us by documenting negative answers to any of these questions for Task T12 Action C
on the "Usability Problem Report Form" provided.

We greatly appreciate your cooperation in this evaluation of the CASE system.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                         - 27 -


Usability Problem Report Form

Date March 9, 1999


Tanisha Hall
Shanda Harper

Representative Task: T1

Action C: Say “Add Pepperoni Pizza”.

Please give a detailed description of the usability problem(s) you identified while evaluating the
above stated action. Feel free to specify the severity of the problem you encountered. This will
help us to do a better job of addressing the issues you raised, in the next stage of CASE's

How often do you think the problem will occur when a user attempts this action?

How serious do you think the problem will be for system users?

We greatly appreciate your feedback.
Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                            - 28 -

Analysis of the Cognitive Walkthrough evaluation

The CW evaluators gave us quite a bit of feedback about the ease of learning the CASE system
by exploration, using the prescribed sequence of actions. Following is a summary of their

Notice Available Actions

   Our start up screen did not convey the necessity of greeting the CASE system. It was thought
    to be an unnecessary option for interacting with the system until CASE‟s voice authentication
    of users was explained. It was suggested that we somehow indicate this need to users.

   Upon adding the frozen pizza item to the shopping list, some difficulty was encountered in
    the vocabulary to use. The “Add…” option was clearly available, but it wasn‟t evident what,
    exactly, could be “Added”. Possible options available to the user were not specified. For
    example, should the user say „Add pizza‟ or „Add frozen pizza‟ or further specify by saying
    „Add 1 large Tombstone Pepperoni Pizza‟. From the menu items it was unclear which, if not
    all, would produce the desired results.

   Both evaluators felt that the “Show…” option could be more self explanatory such as “Show
    product Details” or something similar, in order to provide more clues as to exactly what
    action is associated with using this option. Again, as with the Add… case the Show…does not
    give any clue as to what parameters the system expects.

   When asked to “Change the quantity of Pineapples from 2 to 3”, it is clear that this option is
    available from the Show… menu, however it was noted that, from the List Room, this option
    is not clearly available. In fact the user is not aware that this sequence of events is necessary
    to change the quantity. However, once inside the Show dialogue box, this option is clearly
    available yet still presents some difficulty. Again the Change…option is available, but it is
    unclear from the graphical representation of the dialogue box, what dialogue should follow.
    For example, should the user say, „Change quantity‟ or „Change size‟? It was pointed out that
    this can also occur with changing the brand name or any other list item parameter. Should the
    user should say “ Change Brand,” or “Change Product Name? Clearly, product details such
    as size, quantity, and so forth should be clearly labeled. Another suggestion was that
    Change… could have a drop down menu, which made suggestions such as Change Quantity,
    Change Brand Name, or Change Size.

Trying to produce the Effects that are Generated
Evaluators thought that all of the results produced were the desired result. Most of the problems
occurred in understanding what actions and speech command would generate the desired results.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                        - 29 -

The right action to Produce effects
Some difficulties encountered included:
   Greeting the system to view the main menu

   Ambiguity of the Show… button

   Okay button in the Show product details dialogue box. It was unclear when the results of the
    change were effective. Did the changes take place as soon as the user said “OK”? Or when
    the user said “Exit”? They did understand the feedback however, since the quantity of items
    on the table changed, and the total price was increased when quantity increased.

Understanding Feedback
The graphical feedback, as well as audio, definitely helped in understanding the responses to all
commands. Some questions raised were the following:
   Errors – under what conditions would the system generate errors if the products were not
    clearly specified, brand names not found, commands invalid, etc.

   We did not show any error scenarios.

7.3. Empirical evaluation
We believe that empirical evaluation could be used very effectively, in the CASE system, to
compare two different types of audio interface: a free speech audio interface, where all user
prompts and user-system interaction is verbal, and another audio interface depending much more
heavily on visual user support. The following section describes how we would conduct this
evaluation. In it, we list two independent groups of (hypothetical) evaluators, who will evaluate
one of the audio interfaces.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                      - 30 -


System to be evaluated: Basic functional prototype of CASE version 1.0

Date: March 10, 1999        Time: 1:00 pm

Subjects to be tested:

All subjects are new users from the intended user population (covering the expected primary age
range of 20 – 65 years of age).

Group 1
Lynette Gibson
Habib Rahman
Jose 0sario
Hyacinth Wizzard
Jochen Eisen
May Ling Chen
Adam Efron
Kim Ho
Shari Withrow
Jim Gibson

Group 2
Yvette Larkin
William Sananthanan
Kenya Johnson
Desmond Smith
John Akimoto
Robert White
Kirtan Gill
Mary Bledsoe
Gary Martinez
Lynn Kozlowski

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                                    - 31 -


First Independent Variable:                Free speech interface

Second Independent Variable: Visually-supported speech interface

First Dependent Variable:                Time it takes to satisfactorily complete the task.

Second Dependent Variable: Number of errors generated while performing the task

Experimental hypothesis:                  Using the visually-supported (menu driven) speech
                                          interface, the task takes longer , but generates fewer errors.

The null hypothesis is that there will be no difference in task performance between the
two types of audio interface.

DESIGN To safeguard against the user learning the system during the first of two sequential
experimental conditions, the “between – groups” experimental technique will be used. Members
of group 1 will perform the task using a free speech interface, and members of group 2 will
simultaneously perform the same task, using a visually- supported interface.

TASK The free speech interface will depend heavily on a dialog established between the user
and the system, verbally prompting the user, and providing vocal options for each action
necessary for completion of the task. The visually-supported interface will, at each step of the process,
provide the user with visual cues about allowable system commands; verbal cues will be limited to
prompts not visible in the interface, and will not have any direct bearing on execution of the task. Before
either test begins, the user will be given verbal instructions to change the existing shopping list item “3 lbs
ground chuck” to “4 lbs ground turkey”. The system interface will appear identical to both groups, with
the exception of how the user is made aware of possible system commands. Failure to locate the
appropriate item in the shopping list, and failure to complete the change are both considered errors.

ANALYSIS For each user, the time taken to complete the task and the number of errors made
will be measured. We will assume that the data is normally distributed, and since the main
independent variable, the audio interface type, is two valued we use a simple difference of means
with Student‟s t test to analyze the data.

Since we don‟t have the facilities available to perform this evaluation, we were unable to
implement it or gather any useful data with which to test our hypothesis. We do feel, however,
that this would be a useful test to perform, since each of the two types of audio interface have
distinct advantages, disadvantages, and implementation issues.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999
CASE – System Prototype and Evaluation Plan                                           - 32 -

7.4. Think aloud evaluation
This technique lends itself so well to an understanding of the user‟s cognitive process in using the
system, that we used it in evaluating all three of our scenarios. The process is discussed in the
individual design areas in section 6.

8. Assessment of the Final Design, Summary and Out-
We assessed the overall system by intensively assessing all major parts of the system as described
in section 6 and 7. Please refer to these sections for detail.

During the design phase we iterated over ideas and requirements to refine them and to extract the
really important and strategic ones. In part 2 we explored the design space in 3 independent direc-
tions and set the basis for the prototype you find illustrated in this document.
From the work in part one and two as well as from the feedback we learned that the voice inter-
face is the technology of choice. To be able to reach the goal of visibility and feedback we use the
voice interface together with a graphical representation.

In the future we would have to address the remote device more precisely. But we decided to con-
centrate on the more common user interface for now. From user feedback we learned that we
might have to provide even more visibility in terms of available commands.. In particular, the
“Add…”-command puzzled our first time users a little bit. They asked for some cue about what
the dots mean. With an executable prototype and long term users we could learn the optimal
combination of visibility and an abbreviated operation.
Our users found it also unfamiliar to greet the system. More specifically, because we didn‟t give
any indication that it was expected on the start-up screen – which we don‟t have yet.
In the dialogue boxes to specify product properties users thought it would be helpful to indicate
categories that they could change. For example, when I say “Change…” there should be a menu
of possible command completions like “… quantity”.

Baggett, Gauger, Hannemann, Lakshminarayanan, Napper 1999

Shared By: