Team

Report Team A B C D E F G H J

Executive summary. Title must include word Page 1 Page 1 - - - Page 2 - Page 3 Page 2

"Summary". "Conclusion" not accepted

Executive summary. Length in pages 0,5 1,5 0 0 0 0,5 0 0,7 1

Number of pages in main report 16 36 10 5 36 19 18 11 22

Number of pages in appendices, not including 16 0 0 0 0 15 13 5 9

addendum

Test log included No No No No No No No No No

Number of full or partial screen shots in report 0 8 0 1 2 1 2 0

Number of levels on severity scale 2 3 2 0 2 0 0 3 4



Usability Number of users per comment specified No No Yes No No Partly Partly Yes No

comments

Number of basic problems 26 150 17 10 68 75 30 18 20

Number of basic problems classified as serious 9 26 6 - 7 - - 0 5

Number of basic problems classified as not-so-serious 17 124 8 - 61 - - 18 15

Number of unclassified basic problems 0 0 2 10 0 75 30 0 5

Number of basic problems found only by this team 11 107 4 1 33 38 10 10 12

Number of core problems found (there are 26 core 10 19 9 2 15 14 13 7 8

problems)

Number of basic positive comments 0 8 4 7 24 25 14 4 6



Resources Number of hours 136 123 84 - 130 50 107 45 218

Number of hours used for preparation 37 40 57 - - 10 34 - 70

Number of hours used for testing 43 36 9 0 - 10 18 18 72

Number of hours used for analysis and reporting 56 47 18 16 - 30 55 27 76

Number of usability specialists 2 1 +6 1 1 3 1 1 3 6+1

Number of test participants, total 7 6 6 50 9 5 11 4 6

Number of test participants with Hotmail experience 4 3 3 - 5 0 1 Unclear Excluded

Tasks Team A B C D E F G H J

Connect to Hotmail (locate the Hotmail web-site on the 1 1a 1+2

net)

Locate Hotmail from the MSN web-site 1+7 1b

Find out how to advertise on Hotmail

Find out if you can use Hotmail even if you do not have

your own computer

Familiarize yourself with Hotmail using the information 3

available from the login page

Create a new account 2 1 2 2 1 6a 1 1

Find out what your e-mail address is 3 2

Send mail to one person 3 2 3 6 4 6b+14b 3 4

Send mail to address list 5

Send mail with attachment 11b 12+13 5 5 7

Send mail with signature 13b

Save mail as draft 14

Explore the mail tool 7

Locate e-mail address of another Hotmail user 20 12 8 14a

Check for new mail 2+10 6 2+5

Read new mail 5 10b+12a

Retrieve and read old mail from in-box 7a+9a+11 3

a

Retrieve and read old mail from folder 9

Locate last message received 19

Open attachment (without saving) 9b 10 9 12b

Save attachment 11

Recognize and use link to another home page (URL) in 7d 9 11

e-mail text

Sort mail in in-box 10b 19 8+10 11

Reply to mail 6

Forward mail

Forward mail with attachment 13

Delete mail

Delete all messages in in-box 11

Create e-mail address in address book 8 10

Save e-mail address in address book 7 7 8 10

Save postal address in address book 7c

Retrieve e-mail address from address book 8 9 6

Check for e-mail address in address book 6

Edit address book 11

Create address list 4

Create folder 7e+9c 15 20a 9

Move one e-mail from in-box to a specified folder 7b+9d 8

Move several e-mails from in-box to a specified folder 10a 16 20b

Create signature 13a

Edit personal settings, e.g. own address

Edit preferences, e.g. disable confirmation message or 4 18 4 7 8

screen settings

Use Hotmail with and without frames 5 24

Find e-mail address of Hotmail support

Try the help system 12

Register POP-mail account 5 11 12 23

Locate information explaining how to forward e-mail 22

from other accounts (POP)

What does the Reminders function do? 22

What does the My default switch do? 25

Send reminder to user on a specified date 12 21

Block given e-mail addresses 17

Filter messages from given e-mail address into 21

separate folder

Log in to Hotmail 4 3 3 10a

Log out from Hotmail (or leave Hotmail pc for a while) 6a 23 6+13 4 13 9+15 12

Change Hotmail password 7

Look up forgotten password 6b 24 7 16

Look up forgotten login name 17

Delete account 8

Problems Team A B C D E F G H J

General: If the text appears to be catchy and x

gimmicky, users become unbelieving. Phrases like "no

gimmicks" make it worse

General: Users noted that "Related" is different on x

different pages. Unclear why contents different (related

to 22, 135)

General: Hotmail should be able to handle international 1

character sets. Outgoing mail is ok, but incoming

headers are not.

General: In Europe: Very slow after 3 PM; in US: 3 1 x 3 x x

Sometimes fast and sometimes unacceptably slow.

Sometimes denial of service

General: Unclear what happens when you reach the 1,5

2.000 k maximum account size

General: Appropriate feedback is often missing when 3

Hotmail is processing

General: Secondary windows can get lost behind the 3 x

main browser window and then can only be displayed

using the task bar (related to 72)

General: Users have a hard time finding Hotmail from 1 x

the MSN..com home page. MSN.com is "cluttered"

General: Some users do not see boundaries between x

Hotmail, the browser and the operating system. Look

for Hotmail help in browser help etc.

General: The browser "Stop" function is inactive during x

most actions

General: Should be available in native language x

General: Users with a dial-up connection consider x

Hotmail too expensive and fear that it will overload

their telephone line.

General - Trust: Users fear that their personal x x

correspondence will be accessible to some Hotmail

personnel

General - Trust: Users fear that their personal x

information will be sold by Hotmail to others for

promotion

General - Trust: Users are suspicious because the x

service is free. There must be more to it than

advertisements. "What is the catch?"

General - Trust: Users are suspicious because the x x

service is owned by Microsoft

General - Trust: Some users consider advertising in x

private messages an intrusion in their private mail

General - Keyboard: Users expected carriage return 1,5 1,5

(Enter) to work for submitting information, for example

on the home page

General - Keyboard: Users sometimes use Enter to 1 x x

move from field to field in a form (instead of Tab). In

particular on login screen

General - Functionality: Favorite Quotes and Press x

Release are not worth looking at - they contain only

"promotion"

General - Functionality: Users miss Import and Export 1,5

e-mail from and to other systems

General - Functionality: Many power features are not 3

used by experienced users. Examples: Reminders,

sorting, block and filter

General - Error handling: If the user makes several x

errors in a form, only the first error is reported (in

particular during registration)

General - Error handling: "Internal Error" message 1,5 2 x 2

occurs quite often

General - Error handling: "Internal Error" message 1,5

difficult to understand

Navigation: MSN logo should link to the MSN site 1,5

Navigation: Use Back-button, left side navigation bar to 3 3 x

leave form instead of OK. Confused that changes are

not carried out. E.g. Create folder

Navigation: URLs are so complicated that they do not x

provide an alternate means for navigation

Navigation: There is no Home or "Back to login" button x

Navigation: Remove duplication of menu items, e.g. 1,5

Inbox appears twice in the Folders screen.(related to

238)

Layout: Poor color choices in some places, for 1,5

instance invalid password message, e-mail greetings

Layout: Use of color in Navigation tabs appears to 1,5 x

have a meaning, but does not.

Layout: Users perceived any item that is blue or purple x

as a potential link

Layout: Low contrast in some places, for instance e- 1,5

mail greetings, visitors banner on home page,

information banner on home page

Layout: Adapt pages so they work even for small 1,5

screen sizes. Examples: Home page (enter cut off), In-

box, addresses

Layout: Inconsistent page layout. Deviations should be 1,5 1 x

purposeful. Example: Inconsistent location of options

Layout: On many pages it is hard to discern useful 1,5

information

Layout: Users complain about inconsistent use of MSN 1,5

and old Hotmail logo

Layout: Use of graphics on top left of some pages, but 1,5

not others

Layout: The left panel of the home page contains 1,5

nothing useful

Layout: The web-site is unusable with graphics turned 3

off

Layout: Add more color, highlighting, or other graphical 1

elements to the screen

Layout - Fonts: In many places the font is too small. 3

Examples: Terms of service and links on many pages

Layout - Fonts: Users should be able to modify font 1,5

size

Layout - Fonts: The sizes of the fonts used for the 1

headlines on the home page are too small

Layout - Buttons: Provide a single look for all buttons 1,5

Layout - Buttons: Clarify what should be a button, what 1,5 x x

a link and what something else, and change

appearance accordingly

Layout - Buttons: The buttons used in Help are visually 1,5

different from those in the rest of the system

Layout - Buttons: It should be possible to push 1 x 1

anywhere on the buttons in the navigation bar - not just

on the text

Layout - Buttons: Looks unprofessional that the x

application contains links in places where buttons

should be used

Layout - Banners: Banners are overlooked or ignored 3 x

(no further explanation)

Layout - Banners: Banners are overlooked or ignored - 1,5

Not personalized

Layout - Banners: Banners are overlooked or ignored - 1,5

Not in native language

Layout - Banners:: Overabundance of advertisements - 3 x

sometimes two boxes on a single page

Layout - Banners: Users hate banners or find them x x

irritating, distracting and pointless

Layout - Banners: Advertising seems deceptive x

because it lurks users to click on an advertisement

when that it not their intent

Layout - Banners: It may be useful to provide a place x

where users can find old ads

Layout - Banners: Moving ads are annoying. Stop the x

movement after a certain period

Layout - Frames/Noframes: Significant and confusing 1,5

difference between appearance of frame and non-

frame version

Layout - Frames/Noframes: Differences between the 3 x

Frames and No Frames versions are difficult to predict.

Expect immediate feedback

Layout - Frames/Noframes: "No frames" is ugly 1,5

Home page: The Frames option "My default" does not 3

make sense. Users do not understand what is being

referred to. Expected feedback

Home page: "Who should sign up" has too many Back 1,5

to the Top links. They overwhelm the content

Home page: The tab-sequence is incomplete - you can 1,5

not operate the page without a mouse

Create new account: Registration takes too long (10- 1,5 x x

26 minutes. Acceptable goal is less than 10 minutes.

Achieved by Netscape mail)

Create new account: The headline "Visitors" on the 1

home page is not intuitive. It does not relate to "Create

new account"

Create new account 1 - Terms of service: This page 1,5

and actual registration page are both labelled "Hotmail

Registration"

Create new account 1 - Terms of service: Test 1,5 1,5 1,5 1

subjects do not read it

Create new account 1 - Terms of service: Too long, in x 1

particular sections 7 and 8

Create new account 1 - Terms of service: Small font 1,5

size signals that Hotmail does not really want the user

to read the information

Create new account 1: -Terms of service: Not available 1,5

in native language

Create new account 1 - Terms of service: Important 1,5

question not answered clearly: "Is Hotmail free of

charge"

Create new account 1 - Terms of service: Important x

question not answered clearly: "What is the security of

my information"

Create new account 1 - Terms of service: Make it clear 1

where the Terms of Service can be referenced later

Create new account 2: Some users do not enter 1,5 1,5 x

correct personal information - inconfident about

security or purpose of giving information

Create new account 2: Unclear which fields are 1,5 1,5 x

mandatory and which are optional

Create new account 2 - Login name: Maximum length 1,5

of login name not indicated. Length of text box does

not provide clues

Create new account 2 - Login name: Should tell users 1,5

earlier and better that login can not be the same as

password

Create new account 2 - Login name: Does not accept 1,5

"." in login name, such as firstname.lastname

Create new account 2 - Login name: Users had a hard x x

and frustrating time getting a unique login name

Create new account 2: Some users do not understand x

why they have to re-type their login information

Create new account 2: Some users enter password of x

less than four characters

Create new account 2: Password disappears when 1

returning from error message. Users think that

password was not accepted

Create new account 2: Password disappears when 1 x

returning from error message. Users do not notice, and

this results in another error

Create new account 2: Hotmail does not explain why 3 x 1 x

information about country, age, sex, occupation etc. is

required and what it is used for

Create new account 2: Change "last name" to "family 1,5

name"

Create new account 2: Meaning of "U.S. Residents x

only" and "Non-U.S. Residents Only" is unclear

Create new account: "State/Province" textbox is 1,5 1,5

required but does not make sense in many countries

Create new account: Pull down list for states is very 1,5

inefficient. Allow users to enter state name

abbreviation directly

Create new account: The selection for Country is lost 1

when returning to the form after an error

Create new account: Some users could not find a 1,5 3 x

relevant occupation in the list

Create new account: List of occupations sends a clear 1,5

message about who Hotmail wants as users

Create new account: Occupation titles are country 1,5

dependent. Example: Doctor, engineer may mean

different things in different countries

Create new account: "Get listed on our Hotmail 1,5

member directory" easy to gloss over. Small font size,

bottom of page

Create new account: "Get listed on our Hotmail 1,5

member directory" - default is "listing". User must

actively decide

Create new account 2: Users click Clear instead of x x

Submit

Create new account 3: Separate input of user profile 1,5

from password definition and login. Login failure

causes loss of profile.

Create new account 3: Inconsistent terminology in 1,5

login confirmation page. "Hotmail login" vs. "Hotmail

login name"

Create new account 3: Invalid password error 1,5

message: Not constructive, not adapted for first-time

registration, allow re-entering password

Create new account 3: Does not understand the 3 3 3 x x 1 1,5

meaning of the Password hint question

Create new account 3: Password hint question - x

should not be mandatory

Create new account 4: After the user has created a 1,5

new account, he wants to get or send messages - not

work with Web Courier

Create new account 4: News subscription, and 1,5 x

Classifieds offer not interesting

Create new account 4: News subscription generates a 1,5

lot of incoming e-mail, but nothing interesting

Create new account 4: Users have trouble finding out 1

which item is wrong when they make an error

Create new account 4: Users do not read the web 1,5

courier commercials

Create new account 4: Web courier commercials not 1,5

adapted to user profile

Create new account 4: Users are requested to note the 1,5

address webcourier_help@hotmail.com. How should it

be noted? Put it in address book

Create new account: Users do not read the "Welcome" 1,5 1,5

mail which Hotmail initially places in their in-box

Create new account: The contents of the "Welcome" 1,5

mail looks like a web-site, not like an e-mail

Login: Users attempt to use Carriage Return to 1,5

activate the Enter button. This does not work

Send Mail: Term "Compose" difficult to understand. 3 1,5 x 2 1,5

Use "Create new message" or "Write Mail"

Send Mail: Not clear for some users how to send mail x

to multiple adressees. Some send several identical

mails, some use cc

Send Mail: The navigation bar should be retained on 1,5 3 x x 1

the Compose Mail and Compose Attachment pages

Send Mail: Link text in Related (upper right corner) is 3

too small and was overlooked

Send Mail: Unclear how to insert e-mail address from 1,5

address book in To-box (click "To" ). Change "To" into

a button

Send Mail: Users do not understand bcc 1,5 1,5 x

Send Mail: "To:", "cc:" and "bcc:" links unexpectedly 1,5

bring up a separate window. The visual appearance of

the links does not suggest this

Send Mail: When users click on "To" but don't yet have 1,5

any nicknames entered, system should display Create

page, not error message

Send Mail: Addresses do not default to the Hotmail 1,5

domain

Send Mail: "Save Outgoing Message" check box not 1,5

used (related to 26)

Send Mail: Users overlook "Save Outgoing Message" 3 x 1

check box. Unexpected: Message not saved.(related to

209)

Send Mail: Unclear where outgoing messages are 3

actually saved

Send Mail: Messy layout of buttons (two rows of four 1,5 1,5

buttons)

Send Mail: The difference between Dictionary and x x

Thesaurus may not be clear to all users. Replace with

"Spelling" and "Synonyms"

Send Mail: The spell check feature does not provide x

feedback to confirm it is checking words

Send Mail: Visual indication of mail status missing in 1,5

Compose window (Draft, Saved sent mail, New mail)

Send Mail: Clicking Cancel on the Compose page to 1,5

return to the in-box is non-obvious. Use a more

concrete name

Send Mail: Users say they appreciate Dictionary, 1,5

Thesaurus, but they do not use them

Send Mail: Missing feature: Inform friends of new e- 1,5

mail address by providing button and simple-to-use

template (would also solve problem 38)

Send Mail - Confirmation screen: Information on how 1,5 1,5

to deselect the confirmation is hard to remember while

doing it. Provide direct link

Send Mail - Confirmation screen: Unclear where OK 1,5

will take the user. Use a more concrete name like "In-

box"

Send Mail - Confirmation screen: Causes confusion x

when users send mail to an incorrect address (they still

receive the confirmation)

Send Mail - Confirmation screen: Tip causes x

confusion. Users expect bold words to be clickable

(related to 114)

Send Mail - Save Draft: Difficult to link the term "drafts" 1,5 1,5

with the action it performs

Send Mail - Save Draft: Function difficult to locate 1,5

Send Mail - Save Draft: It is possible to move an 1,5

unread letter from thei n-box to "draft" without any

warning

Send Mail - Save Draft: Difficult to relocate draft 1,5

messages

Send Mail - Save Draft: Need one-click access to the 1,5

saved draft location

Send Mail - Save Draft: Users should be able to name 1,5

draft before it is saved

Send Mail - Save Draft: Should not take the user to the 1,5

In-box page

Send Mail with attachment: Procedure not intuitive. 3 1,5 x

Unclear why several steps: Choose file, attach file

Send Mail with attachment: Specifies file name, not 1,5

path name

Send Mail with attachment: The "Browse" button 1,5

referred to in the help text is named differently in non-

American operating systems

Send Mail with attachment: The imperative character 1,5

of the button text "Attach to Message" caused users to

ignore step three

Send Mail with attachment: Attached files not 1,5 2

displayed on the compose-screen

Send Mail with attachment: Some users interpreted the 1,5

final button "Done" on the compose-screen as a button

that actually sent the mail

Send Mail with attachment: Any feature requiring 10 1,5

minutes or more to activate needs to be re-engineered

or removed

Send Mail with attachment: Users missed the message 1,5

"… up to 10 minutes" although they saw "may require

30 seconds"

Send Mail with attachment: Feedback that file has 1,5

actually been attached is not clear. Users attach file

twice

Send Mail with attachment: Terms "browse", "file", 1,5

"type the path" are jargon and unlikely to be

understood by general users or platform specific

Send Mail with attachment: In step one, the prompt 1,5

"Attach" should be replaced by "Type the path"

Send Mail with attachment: Choose file dialog is 1

confusing. Users do not understand the term Open

(use Choose). Users attempt drag and drop

Send Mail with attachment: Forwarding a mail that x

includes an attachment does not forward the

attachment

Send Mail with attachment: Change the default file 2

type on the "browse" option from HTML to "all files"

Send Mail with attachment: The second step from the 2

attachment procedure automatically should occur

automatically

In-box: Not possible to forward incoming messages 1,5

automatically. Useful when not travelling

In-box: "Related" link not used. MSN dropdown menu 1,5

not used

In-box: Caption "Check for New Hotmail ; POP Mail" 1,5

makes sense to experienced users. Novices are

confused by term "New Hotmail"

In-box: Caption "Check for New Hotmail ; POP Mail" x

makes sense to experienced users. Novices are

confused by term "POP mail"

In-box: Delete all messages done by deleting 1,5

messages one by one. Checkboxes hard to

understand. Same with Move

In-box: Unclear how to open e-mail. Confuses Open x x

and Checkbox. Meaning of underlined sender name

unclear

In-box: The underlining of messages that users already x

read remains blue

In-box: Unclear distrinction between trash can in 3

dropdown menu and Delete button

In-box: Purpose of "The Microsoft Network" pulldown is x

not clear. Is it a bookmark?

In-box: Users who started using the pages available x

through "The Microsoft Network" pulldown were unable

to return. Cryptic URL

In-box: The purpose of the column after " New" is not x

clear to all users (checkbox used for selecting

messages for processing)

In-box: Some users do not understand what "1k" x

means (size of message in in-box)

In-box - Sort: New mail added at the end of the list - 1,5 x

users have to scroll to see new mail

In-box - Sort: Unclear how to sort mail in in-box and 1,5 1,5 1 1,5 x

folders according to subject, date, sender etc. Clues

(arrow) missing. Buttons not recognizable

In-box - Sort: Unclear how to sort mail in in-box x

according to subject, date, sender etc. Sort feature

missing in similar windows, e.g. address book

In-box - Sort: Sort mail feature superfluous. The 1,5

amount of e-mail permitted does not require such a

powerful feature

In-box - Sort: Some users confused because links 1,5

move around to indicate which column controls sort

In-box - Sort: Hotmail should remember users' sorting x

preference

In-box - Move: Clicks "Move to" before choosing where x 3

to move mail. Unable to relate "Move to" to dropdown

menu. Uses "Move to" as navigation tool

In-box - Move: Unable to relate "Move to" to the 3

objects (e-mails) above

In-box - Move: If the user moves an open mail Hotmail 3

automatically opens and displays the next mail

In-box - Move: Inappropriate feedback (no error 1,5

message or generic error message) if user clicks

"Move To" without selecting a folder

In-box - Messages with attachments: Experienced 1,5

users say this feature is not trustworthy

In-box - Messages with attachments: Restriction on 1,5

maximum file size not known even by all experienced

users

In-box - Messages with attachments: Users should 1,5 1,5

have the choice of viewing the attachment directly or

storing it (view failure occurs often)

In-box - Messages with attachments: Restriction on 1,5

maximum file size is too low both for sending and

receiving

In-box - Messages with attachments: If the attachment 1,5

is too large, the message is rejected. The Hotmail

recipient should be notified

In-box - Messages with attachments: The inbox should 3 1,5

show whether a message contains attachments or not

In-box - Messages with attachments: Images (.gif) are 3

treated differently than text (.txt). Gif are displayed, txt

are not

In-box - Messages with attachments: Unclear how to 1,5 x

view attachment

In-box - Messages with attachments: Attached image 1,5

is displayed below the conceptual bottom of the

message; confused with ads or missed

In-box - Messages with attachments: "Download" to 1,5

view attachments is jargon and incorrect. It means

save on local computer, not view in browser

In-box - Messages with attachments: Not obvious how 1,5

to save image attachment (download for txt easy to

understand)

In-box - Messages with attachments: While viewing 1,5

attachments, instruction to quit extra browser is not

sufficiently obvious

In-box - Messages with attachments: Attachments do x

not work in MS IE 3.0

Read Mail: Clicking a link in an e-mail causes Hotmail 3

to open a new browser window. Users confuse new

and old window

Read Mail: Hotmail inspects the e-mail text in order to 1,5

format links - smart but break of tacit agreement that

mail is private

Read Mail: Link to "Block Sender" should be made 1,5

more visible so users can find it

Read Mail: There is so much stuff mixed in with the 1,5

messages (buttons, advertising) - it is hard to

determine where the message begins and ends

Address book: Too many overlapping functions; 3

different terms, visual designs and access methods

(Quidclist, e-mail lookup, member directory)

Address book: Unable to create new entry. Could not 1,5 x 1

find Create button.

Address book: Users do not notice the possibility of 1,5 x

saving an address directly from an e-mail

Address book: Users expect "E-mail lookup" to be a 3 3

superset of "Hotmail Member Directory" . Exact

difference unclear

Address book: Users expect to find own name in 1,5 x

member directory. If they have chosen not to be listed,

they should be alerted to that

Address book: Missing feature: Opportunity to import 1,5 1,5

or export address information from or to other

programs

Address book - Hotmail member directory search: If 3

user not listed, then Hotmail displays text and not

search entry fields

Address book - Hotmail member directory search: 3

Accept listing during registration not enough. Must

confirm again

Address book - Hotmail member directory search: 3

Unclear what to do after second accept of listing

Address book - Create Individual Nickname: Unclear 1,5 x

which fields are mandatory and which are optional

Address book - Create Individual Nickname: Users 1,5 x

only fill in the Nickname and E-mail address fields.

Hesitate to provide other info

Address book - Create Individual Nickname: Users did x

not understand the term "Nickname"

Address book - Create Individual Nickname: Edit and x

Delete caused minor mistakes. Move Edit and Delete

columns closer to Nickname column

Address book - Create Group Nickname: The format 1

for entering e-mail addresses is unclear. Provide

example

Address book - Quicklist: Adding new names is not 3

easy or obvious

Address book - Quicklist: Most were unsure what the 1,5 x

word meant

Folders: Are of limited use in an e-mail system with 3

limited storage. Normally, In-box is sufficient (related to

365)

Folders: More direct and explicit links to folders are 3

required.

Folders: In-box is really one type of folder, but is 1,5

labelled differently (simply "In-box" vs. "Folder: Sent

messages")

Folders: The subfolder "Sent messages" should be 1,5 x

displayed on the global menu together with In-box

Folders: Users had difficulties in locating sent x

messages. Users were looking for a folder named

"Outbox" instead of "Sent messages"

Folders: Some users had difficulties in finding the 1,5 1,5

"Create folder" button

Folders: No users saw the "Find message" link in the 1,5

upper right corner - they found the messages by

manual search

Folders: No confirmation of mail move ("You have 1,5 2

moved x e-mails from y to z")

Folders: One user prefers Netscape folders because 1,5

he can drag and drop e-mail into them

Folders: Some users thought that creating a folder 1,5

would automatically filter e-mail into it (creating a folder

Kris would direct mail from Kris into it)

Folders: "Edit" option only allows user to edit folder 1,5

name - no other edit-like functionality

Folders: The function of the radio box next to the 1,5

message is not clear

Folders: The function of the hyperlink on the message 1,5

is not clear

Options - General - The possibility of customizing the 1,5

default look and feel of Hotmail is not intuitive

Options - Change password - Unclear if you have to 1,5

provide new hint or if new hint and answer overrides

old hint and answer

Options - Preferences - Appearance: Users overlook 1,5 3 1,5 x

that they also have to click the radio button in front of

"Customize"

Options - Preferences - Appearance: Users overlook 1,5

that they have to click OK in the bottom of the long

window to activate the changes

Options - Preferences - Appearamce: Customization is 3

useless as the modifications it supports are minor and

difficult to see and locate

Options - Preferences - Appearance: The terms used 1,5

for the color choices for modifying Navigation Button

Styles are unclear

Options - Preferences - Appearance: Not clear what 1,5

the "background styles" on the customize page refers

to

Optioos - Pop-mail: The name "pop-mail" is not 1,5 1,5 1,5

understood by users.

Optioos - Pop-mail: The feature "pop-mail" is not 1,5 1,5 x

understood by users

Options - Pop-mail: After reading the explanations, x

users said that they did not know their POP servers

and would not know how to find out

Optioos - Pop-mail: Users do not know what to write in x 1,5

the input fields, such as "Server time-out" and

"Portsettings"

Options - Pop-mail: Some experienced users do not 1,5

understand the reason why there are several New Mail

Indicators available

Options - Pop-mail: After reading help, user concluded x

that POP mail costs money (Premium service external

access to Hotmail costs money)

Options - Reminders: Users did not discover it 1,5 1,5

because it is accessible only from the in-box, not from

Options

Options - Reminders: Users did not expect this 1,5

function - it belongs in a calendar program

Options - Reminders: Some users said that they would 1,5

never use a function like this (related to 174)

Options - Reminders: Does not match the actual 3

behavior of long-term users because it requires regular

monitoring of in-coming e-mail

Options - Reminders: Would be nice to allow user to 1,5

set the number of times to remind

Options - Reminders: Choosing a day of the week 3

does not set a weekly reminder (related to 78)

Options - Blocked Senders List: Feature should be 1,5

made more visible

Options - Blocked Senders List: Separate from Filter 1,5

so users will not be distracted by the complexity of

filters while entering Blocked addresses

Options - Blocked Senders List: Should have it's own 1,5

access point on the Options page

Options - Blocked Senders List: "Blocked senders list" 1,5

and "Block sender option" both refer to the same

feature on the Options page

Options - Filters: Unduly complex 1,5

Options - Filters: Not needed in a simple e-mail system 1,5

like Hotmail

Options - Filters: Explain the cascading aspect better, 1,5

or remove it

Options - Filters: Explain, better label or remove the 1,5 x

Move Up/Down buttons

Options - Filters: Make more apparent the difference 1,5 x

between the OK and the Apply Filters Now buttons

Options - Filters: Inconsistent naming - Filter is called 1,5

"spam" or "unsolicited e-mail" in Help, also "Filter" or

"Incoming Mail Filters"

Options - Filters: Filter descriptions talk primarily about x

trashing junk mail - mention that it can be used to

direct mail to other folders also

Options - Filters: Consider providing links for creating x

folders directly from the instructions

Options - Filters: Users overlook the "Enable" field x

Account management: Users do not know what their 1

own Hotmail e-mail address is. Put it in the title of the

in-box

Account management: Distinction between login-name 1

and e-mail address unclear. Attempt login using e-mail

address

Account management: Difficult to find information 1,5 x

about how to cancel an account (related to 30)

Account management: Information about how to 1,5

cancel an account (or availability of cancel) should be

provided during registration

Account management: Not possible to cancel account 2

instantaneously

Account management: Provide search facility for those x

users who have forgotten their logins (user may get

inconvenient login-name)

Account management - Password hint: Unclear what 1,5

users should do if they have not provided a hint

question

Account management - Password hint: Worried about 1,5 1,5

insecurity of procedure. Password hint question is too

simple to answer, e.g. for relatives

Account management - Password hint: Access to 1,5 1,5

password-service should be possible through the

browser, not only via alternate e-mail

Account management - Password hint: Reply from 1,5

password-service to request for retrieval of password

does not contain useful information

Help: Some users thought [contact us ! help ] was one x

function - they saw the two functions as one button

Help: Help and supporting text does not sufficiently 3

differentiate between Frames and No Frames, and

does not always refer to the current version

Help: Should let you get back to where you want to be 1,5

when finished. Users have to click Back multiple times

to return to task

Help: Should be in separate window so users can 1,5

continue with task in the other window

Help: FAQ list missing 1,5

Help: E-mail safety page should highlight the headers 1,5

for the tips

Help: Link from home page to Privacy Policy puts you 1,5

into the middle of the dense Hotmail Terms of Service

Help: It would be nice if the Favorite Quotes page 1,5

contained links so the reader could read the whole

story

Help: Tips are missing x

Help: Should be available on every screen 1,5

Help: Online help text is too wordy x

Help - Context Sensitive: Distinction between Help 3

Center and context sensitive Help not clear to users

Help - Context Sensitive: Links from context-sensitive 1,5

help to topic based help are needed

Help - Center: It should be possible to go directly from 3 2

any help page to the help center

Help - Center: Help Center requires too much work to 3

find information

Support: Autoresponses from Hotmail Customer 1,5

Support would be better presented as web page

Support: E-mails sent to Hotmail support frequently 1,5

result in e-mail replies telling users to write yet another

e-mail to yet another address

Support: Give some assurance that feedback is 1,5

appreciated and will be read

Support: The various e-mail addresses you need to 1,5

contact for various problems are not in a central

location

Support: "Internal Error" message asks user to send 1,5

message to Hotmail web-master. Why not send it

directly to the webmaster?

Support: "Internal Error" message asks user to send 1,5

message to Hotmail web-master. Impossible during

registration

Support: If user sends e-mail about internal error as 1,5

requested, this results in e-mail that does not resolve

problem and asks for another e-mail

E-mail greetings: Link to e-mail greetings signal that 1,5 1,5 x

you stay inside Hotmail, but the appearance of e-mail

greetings signals otherwise

E-mail greetings: Pages look different from "ordinary" 1,5

Hotmail pages

E-mail greetings: Occasional problems in returning to 1,5 x

Hotmail from e-mail greetings ("File not found")

E-mail greetings: Most were unsure what the word 1,5

meant

E-mail greetings: Users did not like the feature - there 1,5

are other sites for this

E-mail greetings: Greetings do not recall names from x

the user's address book

E-mail greetings: It is not clear why Greetings is x

considered so important that it has its own button.

Consider moving it to the left frame

E-mail greetings: Some users confuse Greetings with x

"greeting information" such as address, or signature

file

Logout: User closes browser or links to another web- 1,5 1,5 1,5 x x

site or locks pc-access instead of clicking "Logout"

Logout: Confused about meaning of Logout: Use of 1,5

Back-button after logout displays personal Hotmail

pages (from cache).

Logout: Users did not like the redirect to MSN upon 1,5 x

logout. Expected Hotmail home page.

Logout: Confirmation of successful logoff is missing 1,5

Positive Team A B C D E F G H J

findings

General: The "flat" structure of Hotmail makes it easy x

to use

General: Relevant content and features x



General: Universal Accessibility. Available everywhere. x x x x



General: Link labels are intuitive x



General: The No Frames version provides direct links x

to the Options choices which encourages use of these

features

General: Hotmail is free x x



General: Easy to navigate and to move around quickly x



General: Straight forward and easy to understand x



General: Easy to learn x x



General: Users liked layout and information of the x

Features section

General: Users were satisfied with the information x

provided in the FAQ

General: In most cases the original data entry is still x

there after an error message

General: The quick responsiveness of Hotmail (this x

from users using Hotmail at low-traffic times)

Layout: Clean, nice x x x



Layout: Simple, means fast download x



Layout: Users accept advertising as "neccessary evil" x x x

in return for free service. Some users click banner ads

"just to keep the service going"

Layout - Banners: Interesting, a good way to find out x

about things

Layout - Banners: Attractive, short and to the point x



Layout - Banners: Frames work well. Users liked the x

layout of the page with frames, and preferred it over

the non-frame version

Create new account: Users succeeded in signing up x x

for a Hotmail account without too many difficulties

Create new account: Hotmail helps the user in finding x x

an appropriate login-name

Create new account: Users liked the "Forgot your x x x

password" feature (although they did not all use it as

intended)

Login: Simple procedure x



In-box: Good basic understanding x x



In-box: Sort mail feature x



In-box: Sort mail feature easy to discover x



In-box: Users succeeded in reading a note x



In-box: Subject, From, Date clearly displayed (unlike x

some other tools)

In-box: Delete button very visible (unlike some other x

tools where Delete is only available from the keyboard

Delete button)

In-box: Messages with attachments: Usable x



Send mail: Satisfying functionality x



Send mail: The functions of the controls are clear to x

users

Send mail: Users succeeded in sending mail - easy to x x

use

Send mail: Tasks Read, Compose, Forward, Reply are x

similar to other mail programs

Send mail: Notification when e-mail is sent or x x x

forwarded

Send mail: Notification when e-mail is sent or x

forwarded can be turned off

Send mail: Users liked the organization and layout of x

the Compose Window

Send mail: Buttons are standard Macintosh buttons - x

they look pushable, have black text on gray, etc.

Send mail with attachment: Excellent feedback x



Send mail with attachment: Usable (related to 48). x

Users had problems, but were able to recover

Send mail with attachment: Attachment button located x

right on the Compose screen

Read mail: The button-bar makes a kind of border x

around the message

Read mail: Banner ads are far away from the x

interesting context

Read mail: Links in the e-mail text are easy to x x

recognize and clickable

Address book: Address book and Quicklist work well x



Address book: Users appreciate the Quicklist, and the x x x

Address book

Address book: Users appreciate the Save Address x

function

Address book: Create page easy to understand and x

useful

Address book: The use of "nickname" is handy x



Address book: Thorough - it lets you add considerable x

biographical information about people

Address book - Create group nickname: Group is a x

good term - even for users who are used to similar

terms like "alias", "list"

Folder: Easy to create new folder x x



Folder: First select folder destination, then press the x x

"Move to" button, works well (related to 40)

Folder: Easy to store and retrieve folders x



Folder: Easy to discover and use x



Folder: Useful feature (related to 171) x



Options - General: Users appreciate the possibility to x

make changes in the appearance of Hotmail

Options - General: Once the users are aware that x x

customizing Hotmail is possible, the function is easy to

find

Options - Pop-mail: Users appreciate this function x x



Options - Preferences: Useful feature x



Options - Preferences: Possibility of disabling default x

confirmation after sending an e-mail

Options - Blocked Senders list: Users liked this feature x



Options - Filters: Users liked this feature x x



Help: Users liked this feature x



Help: Help is given for the current page x



Support: Support texts service users well. They x

anticipate many relevant user problems.

Logout: Confirmation message makes it clear that x

users have closed their e-mail