Get documents about "ppt - Slide 1 - Download as PowerPoint - PowerPoint
Document Sample
Automatically Generating High-Quality
User Interfaces for Appliances
Jeffrey Nichols
Human-Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Ph.D. Dissertation Defense
December 1, 2006
Motivation
1
Motivation
Nov 27, 2006
Economist – Oct 2004
CNN – Jan 2004
April 29, 1991
2
Inconsistent Interfaces
3
Multiple appliances used together
4
Approach: Move Interfaces to Mobile Devices
Common, cheap, and often have ability to communicate
203 million mobile phone subscribers (US)
75% always within reach of phone while awake
17.8 million smart phones and 3.65 million PDAs shipped in Q1’06
5
Personal Universal Controller PUC
[Nichols, UIST 02]
Personal Mobile
Automatically Device
Generated
Interface
Control
Abstract
Specification State Feedback
6
Automatic Generation of UIs
Benefits
• Generated interfaces can be better than
today’s appliance interfaces
As shown by user study
>
7
Automatic Generation of UIs
Benefits
• Generated interfaces can be better than
today’s appliance interfaces
As shown by user study
• All interfaces personally consistent for
each user Original
Across functionally similar appliances
Uniform
[Nichols, CHI 06]
Consistent 8
Automatic Generation of UIs
Benefits
• Generated interfaces can be better than
today’s appliance interfaces
As shown by user study
• All interfaces personally consistent for
each user
Across functionally similar appliances
• Aggregate interfaces for systems of
multiple appliances
Functions from multiple appliances
combined into task-based interfaces
Flow-Based Interface Aggregate Interface
Huddle
[Nichols, UIST 06]
9
Related Work: Universal Remotes
Most Universal Remotes
• Require substantial programming
• Support a subset of all functionality
Logitech Harmony Remote Control
• Task-based remote control
• Attempts to track current appliance state
• Tasks are pre-specified by Harmony designers
• Allows execution of only one task at a time
• Adding new tasks requires lots of programming
10
Related Work, cont.
INCITS/V2 Standardization Effort
Alternative Interface Access Protocol (AIAP) [Zimmermann, CHI 2002]
Controlling Appliances
• Xweb [Olsen Jr., UIST 2000]
• Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001]
• Speakeasy (Obje) [Newman, UIST 2002]
• Roadie [Lieberman, IUI 2006]
Model-based User Interface Work
HUMANOID [Szekely, CHI 1993]
• UIDE [Sukaviriya, CHI 1993],
• MASTERMIND [Szekely 1995], TRIDENT [Vanderdonckt 1995]
• Interface Description Languages:
XIML [Puerta, IUI 2002], UIML [Abrams, W3C 1999], USIXML [Limbourg, 2004], etc.
• SUPPLE [Gajos, IUI 2004]
None of these systems have generated consistent user interfaces or combine user interfaces
from multiple appliances
11
Outline
• Introduction
• Personal Universal Controller
• Uniform: Generating User Interfaces with Consistency
• Huddle: Combining Interfaces from Multiple Appliances
• Evaluation
• Discussion and Future Work
12
Overview
Started by exploring appliance user interfaces
• Created handheld interfaces for two appliances
• Found these interfaces to be better than
manufacturers’ interfaces
• Analyzed the interfaces for functional
information
Designed appliance specification language
• Support complete functionality of appliances
• No specific layout information
• Minimize work needed to write specification
Built interface generators
• Uses dependency information to infer structure
of user interface
• Smart Templates allow rendering of high-level
design conventions
13
Specification Language
Language
• Functional information
• XML-based
Elements
• State variables & commands
• Group tree
• Multiple labels per object
• Dependency information
Full documentation available at:
http://www.pebbles.hcii.cmu.edu/puc/ 14
Interface Generation
Graphical Interface Generators
• Runs on today’s handheld devices
• .NET Compact Framework 2.0
Speech Interface Generator
• Masters work of Thomas Harris
• Built on top of the PUC framework
• Implemented using Universal Speech
Interface (USI) techniques
[Rosenfeld 2001]
15
Dependency Information
Useful for determining structure of user
interface
• Some groups are never available
simultaneously
• Nondependent controls can be split into
more accessible side-panels
• Mutually exclusive groups can be placed
on overlapping panels
Improves usability and robustness of
specification language
• Reduces emphasis on group tree for
structuring interfaces
• Dependency information is objective
16
Smart Templates
Designed Smart Templates technique
[Nichols, IUI 04]
• Allows generated interfaces to include high-
level design conventions
• Flexible parameterization specified using
primitive elements of specification language
Without Smart
Templates
With Smart Templates
17
Outline
• Introduction
• Personal Universal Controller
• Uniform: Generating User Interfaces with Consistency
• Huddle: Combining Interfaces from Multiple Appliances
• Evaluation
• Discussion and Future Work
18
Problem
Users cannot transfer knowledge from previous interfaces!
We want interfaces to be personally consistent
Leverage previous work on consistency
• Loosely defined as “doing similar things in similar ways” [Reisner 90]
• “Common elements” theory of knowledge transfer applies to consistent
interfaces [Polson 86, Polson 88]
Steps to complete a task must be similar on each interface
External cues must indicate to user that previous knowledge may apply
• Previous automatic generation systems have been able to achieve some
consistency within an application or family of applications
ITS [Wiecha 90], PUC [Nichols 02], and others…
19
Specification Authoring Study
Study of Authoring Appliance Specifications
• How can specifications vary for different
appliances with similar functions?
• How can specifications vary for different
authors? High-End: Mitsubishi DVCR
Two User Studies
• Study #1: An expert user (me!) wrote specs for
three different VCRs
• Study #2: Three subjects wrote specifications Mid-Range: Samsung DVD-VCR
for the same VCR
All subjects instructed to:
• Include all functions
• Be faithful to the existing appliance design
Low End: Panasonic VCR
What consistency issues arise in our domain? 20
Results (of both studies)
Functions
• Some were identical
• Others were similar but had different labels
• Others had very different specifications
Structure
• Structure for the first study was very similar across appliances
• Many structural differences were found in specifications from the second
study
Results show that specifications can differ substantially, leading to inconsistent
interfaces
What consistency issues arise in our domain? 21
How to achieve consistency?
1. Understand similarities between appliances
• Developed a language for describing mappings between specifications
Six types of mappings based on results of authoring study
XML-based
• Mappings may be automatically found or specified by
users/manufacturers
• Knowledge base stores mappings in multiple mapping graphs
Mapping graphs group similar functions together, store usage history and
allow users to control system
• Function-based, not appliance-based
2. Apply knowledge of similarity to generate new interfaces taking
into account previous interfaces
22
Uniform Architecture
PUC Appliance
Specification
Build Abstract Interface
Build Concrete Interface
Fix Layout Problems
User Interface
How do we modify an interface to achieve consistency? 23
Uniform Architecture
PUC Appliance
Specification
Build Abstract Interface
Previous Specifications
Build Concrete Interface Functional Mappings
Between Specifications
Fix Layout Problems Previously Generated UIs
Knowledge Base
User Interface
How do we modify an interface to achieve consistency? 24
Uniform Architecture
PUC Appliance
Specification
Find Mappings
Build Abstract Interface
Modify Abstract Interface
Previous Specifications
Build Concrete Interface Functional Mappings
Between Specifications
Fix Layout Problems Previously Generated UIs
Modify Concrete Interface
Knowledge Base
Personally
Consistent
User Interface
How do we modify an interface to achieve consistency? 25
Modifying the Abstract Interface
Two parts:
Functional modification
• Similar functions are assigned similar labels
• Similar functions are converted, if possible, to a consistent representation
Abstract structural modification
• Similar functions are moved to similar groups
• Groups are reordered to match previous
Heuristics for unique functions
• Move unique functions only with their siblings
• Unique functions at the beginning or end of a group are not reordered
• Unique functions in the middle of a group follow their preceding sibling
How do we modify an interface to achieve consistency? 26
Moving
Base
Base Base
Samsung DVD-VCR Mitsubishi DVCR
Status Setup
Clock
Channel
How do we modify an interface to achieve consistency? 27
Moving
Base
Setup Setup
Base Base
Samsung DVD-VCR Mitsubishi DVCR
Status Setup
Clock
Channel
How do we modify an interface to achieve consistency? 28
Moving
Clock Clock
Base
Setup Status
Base Base
Samsung DVD-VCR Mitsubishi DVCR
Status Setup
Clock
Channel
How do we modify an interface to achieve consistency? 29
Moving
Clock Clock
Base
Status
Setup Status
Base Base
Samsung DVD-VCR Mitsubishi DVCR
Status Setup
Clock
Channel
How do we modify an interface to achieve consistency? 30
Moving
Channel
Channel
Clock Channel
Base
Setup
Status Setup
Base Base
Samsung DVD-VCR Mitsubishi DVCR
Status Setup
Clock
Channel
How do we modify an interface to achieve consistency? 31
Re-Ordering
How do we modify an interface to achieve consistency? 32
Re-Ordering
Timed Recordings for the Samsung DVD-VCR
When Length Type Source Channel Speed Conflict
Date Time
Timed Recordings for the Mitsubishi DVCR
Conflict VCR+ Channel Day Start Stop Speed
How do we modify an interface to achieve consistency? 33
Re-Ordering
Timed Recordings for the Samsung DVD-VCR
When Type Channel Speed Conflict
Timed Recordings for the Mitsubishi DVCR
Conflict VCR+ Channel When Speed
How do we modify an interface to achieve consistency? 34
Re-Ordering
Timed Recordings for the Samsung DVD-VCR
Conflict Source Channel When Length Type Speed
Date Time
Timed Recordings for the Mitsubishi DVCR
Conflict VCR+ Channel Day Start Stop Speed
How do we modify an interface to achieve consistency? 35
Uniform
Original (PUC) Interfaces
Copier A Copier B
36
Uniform
Original (PUC) Copier A used first
Consistency enabled – Interfaces
Copier A Original Copier BConsistent Copier
Copier B B
37
Outline
• Introduction
• Personal Universal Controller
• Uniform: Generating User Interfaces with Consistency
• Huddle: Combining Interfaces from Multiple Appliances
• Evaluation
• Discussion and Future Work
38
Multi-Appliance Interfaces
How can we address this problem?
• Generate task-based interfaces that span all appliances
• Need to know how appliances are used together
… in a scalable way
39
Content Flow Model
Home Theater
DVD
Receiver
Broadcast
Television
TV
VCR #1 VCR #2
40
Modeling Tasks
Home Theater: Watching Television
DVD
Receiver
Broadcast
Television
TV
VCR #1 VCR #2
41
Modeling Tasks
Home Theater: Watching Television
DVD
Receiver
Broadcast
Television
TV
VCR #1 VCR #2
42
Modeling Tasks
Home Theater: Watching Television
DVD
Receiver
Broadcast
Television
TV
VCR #1 VCR #2
43
Modeling Tasks
Home Theater: Watch DVD
DVD
Receiver
Broadcast
Television
TV
VCR #1 VCR #2
44
Modeling Tasks
Home Theater: Watch DVD + Copy Tape
DVD
Receiver
Broadcast
Television
TV
VCR #1 VCR #2
45
Modeling Tasks
Presentation Room
VCR
PowerPoint
Projector
Laptop Task External Video
Manager Control
Media Player
46
Modeling Tasks
Presentation Room: Presenting PowerPoint
VCR
PowerPoint
Projector
Laptop Task External Video
Manager Control
Media Player
47
Where does the model come from?
DVD
Receiver
Broadcast
Television
TV
VCR #1 VCR #2
48
Where does the model come from?
DVD
Receiver
TV
VCR #1 VCR #2
49
Where does the model come from?
VCR #1 VCR #2 DVD Receiver TV
50
Interfaces From Content Flow
Flow-Based Interface Aggregate User Interfaces
51
Flow-Based Interface
52
Flow-Based Interface
53
Flow-Based Interface
54
Flow-Based Interface
55
Flow-Based Interface
56
Flow-Based Interface
57
Flow-Based Interface
58
Flow-Based Interface
59
Aggregate User Interfaces
Allow users to manage low-level details
Three kinds:
• Control of active flows
• Setup of active flows
• General setup of all appliances
Also explored merging similar functions
from all appliances into single controls
• E.g. one clock control sets all clocks on all
appliances
60
Aggregate User Interfaces
Control of Active Flows
DVD
Receiver
Broadcast
Television
TV
VCR #1 VCR #2
61
Aggregate User Interfaces
Control of Active Flows
DVD
Receiver
Broadcast
Television
TV
VCR #1 VCR #2
62
Aggregate User Interfaces
Control of Active Flows
DVD
Receiver
Broadcast
Television
TV
VCR #1 VCR #2
63
Aggregate User Interfaces
Control of Active Flows
DVD
Receiver
Broadcast
Television
TV
VCR #1 VCR #2
64
Aggregate User Interfaces
Control of Active Flows
DVD
Receiver
Broadcast
Television
TV
VCR #1 VCR #2
65
Aggregate User Interfaces
Setup of Active Flows
Two screens of “Watch DVD” setup
66
Aggregate User Interfaces
General Setup of all Appliances
Two screens of General Setup for the home theater config.
67
Aggregate User Interfaces
Merged Functions
• Most functions shouldn’t be
merged
• Some should:
clock, language, sleep timer…
• Future work…
Baudisch’s flat volume control
General merging
Some merged functions
68
Outline
• Introduction
• Personal Universal Controller
• Uniform: Generating User Interfaces with Consistency
• Huddle: Combining Interfaces from Multiple Appliances
• Evaluation
• Discussion and Future Work
69
Evaluation
Dimensions
• Completeness
What appliances can we write specifications for?
What specifications can we generate interfaces for?
• Usability of Generated Interfaces
How good are the generated interfaces?
• Usability of Specification Language
• Performance
70
Completeness
We have written 33 complete specifications
for a wide variety of appliances
• Home theater appliances
InFocus Television, VCRs, DVD players,
Sony receiver, Audiophase shelf stereo,
Sony camcorder, Windows Media Player
• Copiers, All-in-one printers
• Lighting controls (X10, Lutron, UPnP)
• AT&T telephone/answering machine
• Alarm clocks
• Elevator
• Desktop computer applications
PowerPoint, Photo browser, To-do list
• In-vehicle systems for GMC Yukon Denali
Driver Information Console, Climate Controls,
Navigation System
PUC can generate interfaces for all of these
specifications
71
Completeness: Navigation Interface
72
Usability
Evaluation of Generated Interfaces
• Users perform tasks with two all-in-one
printer interfaces (HP and Canon)
• Two comparisons
Physical interface to PUC interface
Without consistency and with consistency
• 48 participants (28 male, 20 female)
Divided into 6 groups, 8 per group
Recruited through CBDR web page
• Metrics
Completion time
Failures
HP printer Canon printer
73
Procedure
1. Subject performs all 8 tasks on appliance #1 (HP or Canon)
Five minutes maximum allowed for each task
2. Subject is instructed on the optimal way to perform each task
Subject performs the task again until completed correctly.
Additional instruction given as necessary
3. Subject performs all 8 tasks on appliance #2 (Canon or HP)
74
Three Interface Conditions
• Built-In
User sees only physical appliance interfaces
Physical HP → Physical Canon
Physical Canon → Physical HP
• AutoGen
User sees only PUC interfaces without consistency
PUC HP → PUC Canon
PUC Canon → PUC HP
• Consistent
User sees a PUC interface without consistency followed by a Uniform
interface generated to be consistent with the previous PUC interface
PUC HP → Uniform Canon generated to be consistent with HP
PUC Canon → Uniform HP generated to be consistent with Canon
75
Conditions
3 interface conditions * 2 appliance orderings = 6 groups
#1 - HP #2 - Canon #1 - Canon #2 - HP
Built-In Physical Physical Built-In Physical Physical
AutoGen PUC PUC AutoGen PUC PUC
Consistent PUC Uniform Consistent PUC Uniform
76
Comparing Usability
Focus primarily on the first set of tasks
First set not affected by earlier tasks or instruction
#1 - HP #2 - Canon #1 - Canon #2 - HP
Built-In Physical Physical Built-In Physical Physical
AutoGen PUC PUC AutoGen PUC PUC
Consistent PUC Uniform Consistent PUC Uniform
77
Usability Results
#1 - HP #2 - Canon #1 - Canon #2 - HP
Built-In Physical Physical Built-In Physical Physical
AutoGen PUC PUC AutoGen PUC PUC
Consistent PUC Uniform Consistent PUC Uniform
* * * †
* * * * * * †
Failures per subject also significantly less for PUC than Physical
(Fisher’s Exact Test, p < 0.05)
HP: 1.125 Built-In vs. 0.125 PUC
Canon: 2.0 Built-In vs. 0.625 PUC
* *
* = p < 0.05 † = p < 0.1 78
Comparing Consistency
Focus only on the second set of tasks
Previous experience during study should influence results
#1 - HP #2 - Canon #1 - Canon #2 - HP
Built-In Physical Physical Built-In Physical Physical
AutoGen PUC PUC AutoGen PUC PUC
Consistent PUC Uniform Consistent PUC Uniform
79
Consistency Results
#1 - HP #2 - Canon #1 - Canon #2 - HP
Built-In Physical Physical Built-In Physical Physical
AutoGen PUC PUC AutoGen PUC PUC
Consistent PUC Uniform Consistent PUC Uniform
* * † * * †
Very few failures in these conditions: 4 total for all tasks performed by
32 subjects (256 tasks, 1.5% failure rate)
* *
* = p < 0.05 † = p < 0.1 80
Outline
• Introduction
• Personal Universal Controller
• Uniform: Generating User Interfaces with Consistency
• Huddle: Combining Interfaces from Multiple Appliances
• Evaluation
• Discussion and Future Work
81
Summary of Contributions
Designed an abstract appliance specification language
• Can describe the complete functionality of appliances
• Easy to learn and use
Built interface generators for multiple platforms and
modalities
• Use of dependencies to understand interface structure
• Smart Templates technique for rendering design
conventions
• Generates usable interfaces for complex specifications
Generation of personally consistent interfaces
• Means of describing similar functions across appliances
• Algorithms for modifying interfaces to achieve consistency
and allowing user control
• Improves usability for users with related experience
Generation of combined interfaces for multiple appliances
• Uses content-flow model to understand tasks across
multiple connected appliances
• Flow-based interface allows easy specification of high-level
tasks
• Task-specific aggregate interfaces can be generated based
on content flows
82
Future Work
Further benefits of automatic generation
Assistive technology
• Many different types of disabilities each with different needs
• Today solutions must be developed separately for each user
• Challenges:
Developing general ways of specifying the user’s condition
Writing rules that can manipulate the interface to address the user
User-guided design
• Interface customization is hard
Making interface changes, when possible, is often tedious and time-consuming
Hard for users to know what they will do when getting started
Once users are experienced, the cost of changing that interface is higher
• How could automatic generation improve this?
Allows customization to be performed at a higher-level
System can collaborate with user to iteratively arrive at a better interface
83
Future Work, cont.
Automatic modification vs. generation
• Can the benefits of automatic generation be achieved for all interfaces?
Consistency
Aggregation (mash-ups)
• Challenges
Getting a model of the hand-designed interface
Understanding the conventions of the hand-designed interface
Creating rules that balance previous design with new design
84
Conclusion
Automatic generation of usable interfaces for end users is
possible
• Automatic design may be viable for some applications
Automatic generation allows interfaces to be customized to the:
• user’s previous experience
• user’s environment
• more?
85
Acknowledgements
Thesis Committee PUC Project Members Collaborators & Friends
• Brad A. Myers (chair) • Brandon Rothrock • Naomi Ramos
• Scott Hudson • Duen Horng Chau • Desney Tan
• John Zimmerman • Kevin Litwack • Daniel Avrahami
• Dan Olsen Jr. • Thomas K. Harris • Gaetano Borriello
• Michael Higgins • Laura Dabbish
Funding • Joseph Hughes • Andrew Faulring
• National Science Foundation • Roni Rosenfeld • James Fogarty
• Microsoft • Rajesh Seenichamy • Krzysztof Gajos
• General Motors • Pegeen Shen • Darren Gergle
• Intel • Htet Htet Aung • Andy Ko
• Pittsburgh Digital Greenhouse • Mathilde Pignol • Amy Nichols
• Suporn Pongnumkul • Mick Nichols
Equipment Grants • Stefanie Shriver • Sally Nichols
• Mitsubishi (MERL) • Jeffrey Stylos • Trevor Pering
• VividLogic • Peter Lucas • Fleming Seay
• Lucent • Irina Shklovski
• Lutron • Rachel Steigerwalt
• Lantronix • Roy Want
• Nokia • Jake Wobbrock
• and many others…
86
Thanks for listening!
For more information…
http://www.jeffreynichols.com/
http://www.pebbles.hcii.cmu.edu/puc/
Automatically Generating High-Quality
User Interfaces for Appliances
Jeffrey Nichols
Human-Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Ph.D. Dissertation Defense
December 1, 2006
89
90
Tasks
1. Send a fax to the number stored in the third speed dial
2. Configure the fax function so that it will always redial a number that is busy
3. Configure the fax function so that any document received that is larger than the
default paper size will be resized to fit the default
4. Configure the fax function so that it will only print out an error report when it has a
problem receiving a fax
5. Make two black-and-white copies of the document that has already been placed on
the scanner of the printer
6. Imagine you find the copies too dark. Improve this by changing one setting of the
device
7. Given a page with a picture, determine how to produce one page with several
instances of the same picture repeated (demonstrated with actual paper copies)
8. The device remembers the current date and time. Determine where in the interface
these values can be changed (do not change them)
91
Modifying the Concrete Interface
Address visual inconsistency problems
• Side panels may have different orientations
Re-ordering within panels
• Flattening of abstract hierarchy into panels
sometimes creates ordering inconsistencies
Concrete organization decisions
• Different organization may have been used
from the abstract interface
How do we modify an interface to achieve consistency? 92
Complex Phone Interface
a.
b. c. d.
e. f. g.
93
Motivation
These devices are hard to use
94
Content-Flow Template Pieces
TV TV TV TV
TV TV VCR
95
What should specifications contain?
What information is needed about the appliance
to automatically generate remote control
interfaces?
Investigate via a design process
Create interfaces by hand
• AIWA Shelf Stereo
• AT&T Telephone/Answering Machine
Improve quality with heuristic analysis and think-
aloud studies with several users
Compare interfaces with actual appliance
interfaces to validate PUC concept
• Users were twice as fast, and made half as many
errors with our interfaces
Analyze interfaces for functional information
96
UNIFORM
B
Consistency disabled – Copier A used first
Copier A Copier B
97
UNIFORM
enabled
Consistency disabled – Copier B used first
Copier B Copier AConsistent Copier
A
98
Interface Generation, cont.
Graphical Interface Generators User options
System control x10, control audiophase
• Runs on today’s handheld devices User control x10
• .NET Compact Framework 2.0 System control x10 (x10 voice)
User options
Speech Interface Generator System global off, all lights on, all lights
off “...”
• Masters work of Thomas Harris User control audiophase
System control audiophase (stereo voice)
• Built on top of the PUC framework User options
System stereo power, x-bass, volume “…”
• Implemented using Universal Speech User stereo power
Interface (USI) techniques System <stereo is turned on>
[Rosenfeld 2001]
User options
System stereo power, x-bass, volume “…”
User more
System mode, tuner, CD
User stereo power is, options
System on, off
99
Interface Generation
Process
1. Build abstract interface from specification
• Tree structure, including dependency
structure
• Abstract choice of controls
2. Build concrete interface from abstract
interface
• Custom to screen size, resolution,
platform, etc.
3. Fix layout problems
• Insufficient height or width, etc.
4. Display interface to user
10
Knowledge Base
Functional Mappings
• We support six types of mappings, based Previous Specifications
Functional Mappings
on the results of the specification Between Specifications
Previously Generated UIs
authoring studies
Knowledge Base
general, state, node, group, list, and
template
• Specifies that a function in one
specification is similar to a function in
another
Mappings are grouped within the knowledge
base in mapping graphs
How do we describe the similarity between specifications? 10
Where do mappings come from?
Three ways mappings may be found
Automatically extracted immediately before interface generation
• Challenging because of limited semantic information in specifications
• We have implemented two mapping algorithms, one based on work in
automatic schema mapping from the database community
• Current best algorithm finds about 60% of mappings
• We are also exploring other techniques
Specified by the user
• Manually entered as text
• Interaction techniques?
Downloaded from the Internet
• Manufacturers?
• Internet communities? (e.g. www.remotecentral.com)
How do we find similar functions across appliance specifications? 10
Generated Phone Examples
Copier A without Copier B without
consistency consistency
10
Consistent Phone Interfaces
Copier A
consistent with
Copier B
10
Consistent Phone Interfaces, cont.
Copier B
consistent with
Copier A
10
Aggregate User Interfaces
Three kinds
• Control of Active Flows Volume for TV Volume for Receiver
“Copy Tape” “Watch TV “Watch TV
using TV speakers” using Receiver speakers”
10
Flow-Based Interface
10
What’s next?
Consistency
• Studies of consistency
• Unique functions
• Trade-offs between dimensions of consistency
(e.g. visual appearance vs. structure)
• Additional domains
• Mobile phones
• Old phone and new phone
• Intelligent design assistants & analysis
10
What’s next?
Combining User Interfaces
• Task-based Mash-ups
• User-guided adaptive interfaces
10
Task-Based Mash-Ups
Today’s mash-ups mainly combine data
Use mash-ups to streamline tasks
• Arranging travel
• Paying bills
• Various business tasks…expense reports, etc.
Questions
• How to identify tasks?
• How to find application functionality for each task step?
• How do we then build an interface from these different pieces of application
functionality?
• Security, Privacy, etc…
11
Combining Interfaces, cont.
User-guided adaptive interfaces
Process
1. System observes a user’s interactions
2. Display a visualization of interaction history
3. User can select collections of interactions that represent a task
4. An interface is automatically generated for that task
5. User can iterate on their selection to refine interface
11
Completeness, cont.
How complex are they?
Vars & Labeled Smart Max Tree Ave. Tree
Cmds Groups Groups Templates Depth Depth
Max* 136 64 46 30 11 6.20
Average* 36.25 16.93 11.6 6.21 4.79 3.59
GM Nav 388 171 136 79 11 7.00
*not counting GM Navigation system
We can generate interfaces for all of these specifications
11
Interfaces
Max Ave.
Vars & Labeled Smart Tree Tree
Cmds Groups Groups Templates Depth Depth
Max* 136 64 46 30 11 6.20
Average* 36.25 16.93 11.6 6.21 4.79 3.59
GM Nav 388 171 136 79 11 7.00
HP 87 35 28 15 7 5.59
Canon 136 64 46 30 11 6.20
HP printer Canon printer HP printer Canon printer
without consistency without consistency consistent with Canon consistent with HP
11
Interfaces
HP printer Canon printer
Consistent Canon printer 11
Usability Results
#1 - HP #2 - Canon #1 - Canon #2 - HP
Built-In Physical Physical Built-In Physical Physical
AutoGen PUC PUC AutoGen PUC PUC
Consistent PUC Uniform Consistent PUC Uniform
* * * † * * * * * * †
* *
Failures per subject also significantly less for PUC than Built-In (Fisher’s Exact Test, p < 0.05)
HP: 1.125 Built-In vs. 0.125 PUC
Canon: 2.0 Built-In vs. 0.625 PUC
* = p < 0.05 † = p < 0.1 11
Consistency Results
#1 - HP #2 - Canon #1 - Canon #2 - HP
Built-In Physical Physical Built-In Physical Physical
AutoGen PUC PUC AutoGen PUC PUC
Consistent PUC Uniform Consistent PUC Uniform
* * † * * †
* *
Very few failures in these conditions: 4 total for all tasks performed by 32 subjects (256 tasks, 1.5% failure rate)
* = p < 0.05 † = p < 0.1 11
Similarity and Mapping Graphs
One-to-one functional mappings
between specifications
Six types of mappings determined
from authoring study
Similar functions across specs are Previous Specifications
grouped in “mapping graphs” Functional Mappings
Between Specifications
• Power
Previously Generated UIs
• Volume Answering Machine
• Media Controls State: Mode Mitsubishi VCR
• Etc.
{ Play, Stop, Pause } State: Base
Knowledge Mode
Command: Play New Messages { Play, Stop, Pause, Rewind, F-Fwd, Record }
Node counts 0 3
• Store generation history
• Used to find basis for consistency Cheap VCR
Command: Play
Edges 1
Command: Stop
Command: Pause
• Track whether representation can 0 Command: Rewind
be modified to achieve consistency DVD Player Command: F-Fwd
Command: Record
State: Mode
Choice in consistency supported { Play, Stop, Pause }
by manipulating the node counts Command: Next Track
Command: Previous Track
How do we design our system so that the user has control? 11
Achieving consistency with Uniform
Uniform is the first system to automatically achieve personally
consistent interfaces
Many hard problems to address:
• What is consistency?
• What consistency issues arise in our domain?
• How do we describe the similarity between specifications?
• How do we find similar functions across appliance specifications?
• How do we modify an interface to achieve consistency?
• How do we address unique functions?
• How do we incorporate consistency constraints with all appliances?
• How do we design our system so that the user has control?
11
What is consistency?
Previous work
• Many definitions of consistency
[Nielsen 88, Kellogg 87]
• Explored as knowledge transfer by psychologists
[Keiras & Polson 85, Polson 88]
• Previous systems have achieved consistency within an application or for
families of applications:
ITS [Wiecha 90], PUC [Nichols 02], others…
My Studies of Specification Authoring
• Multiple authors wrote complex VCR specifications
• Controlled for author, VCR model
• Result: Specifications can differ substantially across appliances and
authors, leading to inconsistent interfaces
What is consistency? 11
Summary of Huddle
Huddle automatically generates interfaces
for systems of connected appliances
• Content-flow model tells us how appliances
are used together in a scalable way
• Flow-based interface allows user to quickly
achieve high-level goals
• Aggregate interfaces are generated based
on flows for low-level control
Flow-Based Interface Aggregate Interface
12
Summary of Uniform
Uniform automatically generates personally
consistent interfaces
• Consistency helps users transfer
knowledge between interfaces
• Similarity is described by one-to-one
functional mappings grouped in mapping
graphs
Original
• Mappings can be found automatically or be
specified by manufacturers or users
• Similarity information is used to modify
new interfaces to achieve consistency
• Functional and structural modifications
• Heuristics for addressing unique functions
Uniform
• Most changes made to abstract interface [Nichols, CHI 06]
Consistent 12
