Docstoc

ppt - Slide 1 - Download as PowerPoint - PowerPoint

Document Sample
ppt - Slide 1 - Download as PowerPoint - PowerPoint Powered By Docstoc
					Automatically Generating High-Quality
User Interfaces for Appliances



Jeffrey Nichols
Human-Computer Interaction Institute
School of Computer Science
Carnegie Mellon University

Ph.D. Dissertation Defense
December 1, 2006
Motivation




             1
Motivation

                                              Nov 27, 2006




                              Economist – Oct 2004




                                                     CNN – Jan 2004




             April 29, 1991
                                                                  2
Inconsistent Interfaces




                          3
Multiple appliances used together




                                    4
Approach: Move Interfaces to Mobile Devices
Common, cheap, and often have ability to communicate
    203 million mobile phone subscribers (US)
    75% always within reach of phone while awake
    17.8 million smart phones and 3.65 million PDAs shipped in Q1’06




                                                                       5
Personal Universal Controller                        PUC
                                                     [Nichols, UIST 02]




                                   Personal Mobile
      Automatically                Device
         Generated
          Interface




                         Control
           Abstract
         Specification    State Feedback




                                                                     6
Automatic Generation of UIs
Benefits
•   Generated interfaces can be better than
    today’s appliance interfaces
       As shown by user study




                                              >


                                                  7
Automatic Generation of UIs
Benefits
•   Generated interfaces can be better than
    today’s appliance interfaces
       As shown by user study

•   All interfaces personally consistent for
    each user                                                        Original
       Across functionally similar appliances




                                                Uniform
                                                [Nichols, CHI 06]


                                                                    Consistent   8
Automatic Generation of UIs
Benefits
•   Generated interfaces can be better than
    today’s appliance interfaces
       As shown by user study

•   All interfaces personally consistent for
    each user
       Across functionally similar appliances

•   Aggregate interfaces for systems of
    multiple appliances
       Functions from multiple appliances
       combined into task-based interfaces
                                                Flow-Based Interface   Aggregate Interface

                                                 Huddle
                                                 [Nichols, UIST 06]
                                                                                             9
Related Work: Universal Remotes
Most Universal Remotes
 •   Require substantial programming
 •   Support a subset of all functionality

Logitech Harmony Remote Control
 •   Task-based remote control
 •   Attempts to track current appliance state
 •   Tasks are pre-specified by Harmony designers
 •   Allows execution of only one task at a time
 •   Adding new tasks requires lots of programming




                                                     10
Related Work, cont.
INCITS/V2 Standardization Effort
     Alternative Interface Access Protocol (AIAP) [Zimmermann, CHI 2002]
Controlling Appliances
     • Xweb [Olsen Jr., UIST 2000]
     • Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001]
     • Speakeasy (Obje) [Newman, UIST 2002]
     • Roadie [Lieberman, IUI 2006]

Model-based User Interface Work
                                 HUMANOID [Szekely, CHI 1993]
     • UIDE [Sukaviriya, CHI 1993],
     • MASTERMIND [Szekely 1995], TRIDENT [Vanderdonckt 1995]
     • Interface Description Languages:
         XIML [Puerta, IUI 2002], UIML [Abrams, W3C 1999], USIXML [Limbourg, 2004], etc.
     • SUPPLE [Gajos, IUI 2004]

None of these systems have generated consistent user interfaces or combine user interfaces
from multiple appliances
                                                                                             11
Outline
• Introduction

• Personal   Universal Controller

• Uniform:   Generating User Interfaces with Consistency

• Huddle:   Combining Interfaces from Multiple Appliances

• Evaluation

• Discussion   and Future Work


                                                           12
Overview
Started by exploring appliance user interfaces
  •   Created handheld interfaces for two appliances
  •   Found these interfaces to be better than
      manufacturers’ interfaces
  •   Analyzed the interfaces for functional
      information

Designed appliance specification language
  •   Support complete functionality of appliances
  •   No specific layout information
  •   Minimize work needed to write specification

Built interface generators
  •   Uses dependency information to infer structure
      of user interface
  •   Smart Templates allow rendering of high-level
      design conventions
                                                       13
  Specification Language
  Language
       •   Functional information
       •   XML-based

  Elements
       •   State variables & commands
       •   Group tree
       •   Multiple labels per object
       •   Dependency information




Full documentation available at:
http://www.pebbles.hcii.cmu.edu/puc/    14
Interface Generation
Graphical Interface Generators
  •   Runs on today’s handheld devices
  •   .NET Compact Framework 2.0

Speech Interface Generator
  •   Masters work of Thomas Harris
  •   Built on top of the PUC framework
  •   Implemented using Universal Speech
      Interface (USI) techniques
      [Rosenfeld 2001]




                                           15
Dependency Information
Useful for determining structure of user
interface
   •   Some groups are never available
       simultaneously
   •   Nondependent controls can be split into
       more accessible side-panels
   •   Mutually exclusive groups can be placed
       on overlapping panels


Improves usability and robustness of
specification language
   •   Reduces emphasis on group tree for
       structuring interfaces
   •   Dependency information is objective



                                                 16
Smart Templates
Designed Smart Templates technique
[Nichols, IUI 04]
   •   Allows generated interfaces to include high-
       level design conventions
   •   Flexible parameterization specified using
       primitive elements of specification language




         Without Smart
          Templates


                             With Smart Templates
                                                      17
Outline
• Introduction

• Personal   Universal Controller

• Uniform:   Generating User Interfaces with Consistency

• Huddle:   Combining Interfaces from Multiple Appliances

• Evaluation

• Discussion   and Future Work


                                                           18
Problem
Users cannot transfer knowledge from previous interfaces!
We want interfaces to be personally consistent
Leverage previous work on consistency
  •   Loosely defined as “doing similar things in similar ways” [Reisner 90]
  •   “Common elements” theory of knowledge transfer applies to consistent
      interfaces [Polson 86, Polson 88]
       Steps to complete a task must be similar on each interface
       External cues must indicate to user that previous knowledge may apply

  •   Previous automatic generation systems have been able to achieve some
      consistency within an application or family of applications
       ITS [Wiecha 90], PUC [Nichols 02], and others…


                                                                               19
Specification Authoring Study
Study of Authoring Appliance Specifications
    • How can specifications vary for different
      appliances with similar functions?
    • How can specifications vary for different
      authors?                                          High-End: Mitsubishi DVCR



Two User Studies
    • Study #1: An expert user (me!) wrote specs for
                three different VCRs
    • Study #2: Three subjects wrote specifications    Mid-Range: Samsung DVD-VCR
                for the same VCR


All subjects instructed to:
    • Include all functions
    • Be faithful to the existing appliance design
                                                         Low End: Panasonic VCR


What consistency issues arise in our domain?                                        20
Results (of both studies)
Functions
     •   Some were identical
     •   Others were similar but had different labels
     •   Others had very different specifications

Structure
     •   Structure for the first study was very similar across appliances
     •   Many structural differences were found in specifications from the second
         study



Results show that specifications can differ substantially, leading to inconsistent
interfaces



What consistency issues arise in our domain?                                         21
How to achieve consistency?
1.   Understand similarities between appliances
      •   Developed a language for describing mappings between specifications
             Six types of mappings based on results of authoring study
             XML-based
      •   Mappings may be automatically found or specified by
          users/manufacturers
      •   Knowledge base stores mappings in multiple mapping graphs
             Mapping graphs group similar functions together, store usage history and
             allow users to control system

      •   Function-based, not appliance-based

2.   Apply knowledge of similarity to generate new interfaces taking
     into account previous interfaces

                                                                                        22
Uniform Architecture

                          PUC Appliance
                          Specification



                                  Build Abstract Interface


                                  Build Concrete Interface
                                  Fix Layout Problems




                               User Interface




How do we modify an interface to achieve consistency?        23
Uniform Architecture

                          PUC Appliance
                          Specification



                                  Build Abstract Interface

                                                              Previous Specifications
                                  Build Concrete Interface    Functional Mappings
                                                               Between Specifications
                                  Fix Layout Problems         Previously Generated UIs


                                                              Knowledge Base



                               User Interface




How do we modify an interface to achieve consistency?                                     24
Uniform Architecture

                          PUC Appliance
                          Specification

                                  Find Mappings
                                  Build Abstract Interface
                                  Modify Abstract Interface
                                                               Previous Specifications
                                  Build Concrete Interface     Functional Mappings
                                                                Between Specifications
                                  Fix Layout Problems          Previously Generated UIs
                                  Modify Concrete Interface
                                                               Knowledge Base


                               Personally
                               Consistent
                               User Interface




How do we modify an interface to achieve consistency?                                      25
Modifying the Abstract Interface
Two parts:
    Functional modification
         •   Similar functions are assigned similar labels
         •   Similar functions are converted, if possible, to a consistent representation

    Abstract structural modification
         • Similar functions are moved to similar groups
         • Groups are reordered to match previous



Heuristics for unique functions
    •   Move unique functions only with their siblings
    •   Unique functions at the beginning or end of a group are not reordered
    •   Unique functions in the middle of a group follow their preceding sibling


How do we modify an interface to achieve consistency?                                       26
Moving

                                            Base

          Base                                                                    Base

     Samsung DVD-VCR                                                         Mitsubishi DVCR
                                Status                  Setup




                                                                Clock




                                                                        Channel

How do we modify an interface to achieve consistency?                                          27
Moving

                                            Base
          Setup                                                                   Setup
          Base                                                                    Base

     Samsung DVD-VCR                                                         Mitsubishi DVCR
                                Status                  Setup




                                                                Clock




                                                                        Channel

How do we modify an interface to achieve consistency?                                          28
Moving

          Clock                                                                   Clock
                                            Base
          Setup                                                                   Status
          Base                                                                    Base

     Samsung DVD-VCR                                                         Mitsubishi DVCR
                                Status                  Setup




                                                                Clock




                                                                        Channel

How do we modify an interface to achieve consistency?                                          29
Moving

          Clock                                                                   Clock
                                            Base
          Status
          Setup                                                                   Status
          Base                                                                    Base

     Samsung DVD-VCR                                                         Mitsubishi DVCR
                                Status                  Setup




                                                                Clock




                                                                        Channel

How do we modify an interface to achieve consistency?                                          30
Moving
         Channel
         Channel
          Clock                                                    Channel
                                            Base
          Setup
          Status                                                    Setup
          Base                                                      Base

     Samsung DVD-VCR                                            Mitsubishi DVCR
                                Status                  Setup




                       Clock




                               Channel

How do we modify an interface to achieve consistency?                             31
Re-Ordering




How do we modify an interface to achieve consistency?   32
Re-Ordering
                                Timed Recordings for the Samsung DVD-VCR




         When          Length        Type          Source         Channel    Speed    Conflict



  Date          Time



                                 Timed Recordings for the Mitsubishi DVCR




    Conflict      VCR+          Channel          Day             Start      Stop     Speed


How do we modify an interface to achieve consistency?                                        33
Re-Ordering
                               Timed Recordings for the Samsung DVD-VCR



            When             Type           Channel             Speed      Conflict




                                Timed Recordings for the Mitsubishi DVCR




           Conflict          VCR+           Channel              When      Speed



How do we modify an interface to achieve consistency?                                 34
Re-Ordering
                               Timed Recordings for the Samsung DVD-VCR




       Conflict      Source       Channel          When           Length     Type    Speed



                                            Date           Time



                                Timed Recordings for the Mitsubishi DVCR




    Conflict      VCR+        Channel           Day             Start      Stop     Speed


How do we modify an interface to achieve consistency?                                        35
Uniform
             Original (PUC) Interfaces




          Copier A                Copier B

                                             36
Uniform
                  Original (PUC) Copier A used first
          Consistency enabled – Interfaces




              Copier A     Original    Copier BConsistent Copier
                           Copier B                    B
                                                                   37
Outline
• Introduction

• Personal   Universal Controller

• Uniform:   Generating User Interfaces with Consistency

• Huddle:   Combining Interfaces from Multiple Appliances

• Evaluation

• Discussion   and Future Work


                                                           38
Multi-Appliance Interfaces




How can we address this problem?
  •   Generate task-based interfaces that span all appliances
  •   Need to know how appliances are used together
       … in a scalable way
                                                                39
Content Flow Model
Home Theater




                             DVD
                                     Receiver




      Broadcast
      Television
                                       TV
                   VCR #1   VCR #2




                                                40
Modeling Tasks
Home Theater: Watching Television




                             DVD
                                     Receiver




      Broadcast
      Television
                                       TV
                   VCR #1   VCR #2




                                                41
Modeling Tasks
Home Theater: Watching Television




                             DVD
                                     Receiver




      Broadcast
      Television
                                       TV
                   VCR #1   VCR #2




                                                42
Modeling Tasks
Home Theater: Watching Television




                             DVD
                                     Receiver




      Broadcast
      Television
                                       TV
                   VCR #1   VCR #2




                                                43
Modeling Tasks
Home Theater: Watch DVD




                             DVD
                                     Receiver




      Broadcast
      Television
                                       TV
                   VCR #1   VCR #2




                                                44
Modeling Tasks
Home Theater: Watch DVD + Copy Tape




                             DVD
                                      Receiver




      Broadcast
      Television
                                        TV
                   VCR #1   VCR #2




                                                 45
Modeling Tasks
Presentation Room




                                                VCR


  PowerPoint
                                                      Projector


                 Laptop Task   External Video
                  Manager         Control

  Media Player



                                                              46
Modeling Tasks
Presentation Room: Presenting PowerPoint




                                                VCR


   PowerPoint
                                                      Projector


                 Laptop Task   External Video
                  Manager         Control

  Media Player



                                                              47
Where does the model come from?



                          DVD
                                  Receiver




   Broadcast
   Television
                                    TV
                VCR #1   VCR #2




                                             48
Where does the model come from?



                  DVD
                          Receiver




                            TV
        VCR #1   VCR #2




                                     49
Where does the model come from?




  VCR #1   VCR #2   DVD   Receiver   TV




                                          50
Interfaces From Content Flow




     Flow-Based Interface   Aggregate User Interfaces


                                                        51
Flow-Based Interface




                       52
Flow-Based Interface




                       53
Flow-Based Interface




                       54
Flow-Based Interface




                       55
Flow-Based Interface




                       56
Flow-Based Interface




                       57
Flow-Based Interface




                       58
Flow-Based Interface




                       59
Aggregate User Interfaces
Allow users to manage low-level details

Three kinds:
  •   Control of active flows
  •   Setup of active flows
  •   General setup of all appliances

Also explored merging similar functions
from all appliances into single controls
  •   E.g. one clock control sets all clocks on all
      appliances




                                                      60
Aggregate User Interfaces
Control of Active Flows




                         DVD
                                 Receiver




  Broadcast
  Television
                                   TV
               VCR #1   VCR #2




                                            61
Aggregate User Interfaces
Control of Active Flows




                         DVD
                                 Receiver




  Broadcast
  Television
                                   TV
               VCR #1   VCR #2




                                            62
Aggregate User Interfaces
Control of Active Flows




                         DVD
                                 Receiver




  Broadcast
  Television
                                   TV
               VCR #1   VCR #2




                                            63
Aggregate User Interfaces
Control of Active Flows




                         DVD
                                 Receiver




  Broadcast
  Television
                                   TV
               VCR #1   VCR #2




                                            64
Aggregate User Interfaces
Control of Active Flows




                         DVD
                                 Receiver




  Broadcast
  Television
                                   TV
               VCR #1   VCR #2




                                            65
Aggregate User Interfaces
Setup of Active Flows




                        Two screens of “Watch DVD” setup


                                                           66
Aggregate User Interfaces
General Setup of all Appliances




              Two screens of General Setup for the home theater config.



                                                                          67
Aggregate User Interfaces
Merged Functions
 •   Most functions shouldn’t be
     merged
 •   Some should:
      clock, language, sleep timer…
 •   Future work…
      Baudisch’s flat volume control
      General merging




                                       Some merged functions


                                                               68
Outline
• Introduction

• Personal   Universal Controller

• Uniform:   Generating User Interfaces with Consistency

• Huddle:   Combining Interfaces from Multiple Appliances

• Evaluation

• Discussion   and Future Work


                                                           69
Evaluation
Dimensions
 •   Completeness
      What appliances can we write specifications for?
      What specifications can we generate interfaces for?
 •   Usability of Generated Interfaces
      How good are the generated interfaces?
 •   Usability of Specification Language
 •   Performance




                                                            70
Completeness
We have written 33 complete specifications
for a wide variety of appliances
   •   Home theater appliances
         InFocus Television, VCRs, DVD players,
         Sony receiver, Audiophase shelf stereo,
         Sony camcorder, Windows Media Player
   •   Copiers, All-in-one printers
   •   Lighting controls (X10, Lutron, UPnP)
   •   AT&T telephone/answering machine
   •   Alarm clocks
   •   Elevator
   •   Desktop computer applications
         PowerPoint, Photo browser, To-do list
   •   In-vehicle systems for GMC Yukon Denali
         Driver Information Console, Climate Controls,
         Navigation System


PUC can generate interfaces for all of these
specifications
                                                         71
Completeness: Navigation Interface




                                     72
Usability
Evaluation of Generated Interfaces
  •   Users perform tasks with two all-in-one
      printer interfaces (HP and Canon)
  •   Two comparisons
       Physical interface to PUC interface
       Without consistency and with consistency

  •   48 participants (28 male, 20 female)
       Divided into 6 groups, 8 per group
       Recruited through CBDR web page

  •   Metrics
       Completion time
       Failures
                                                  HP printer   Canon printer

                                                                               73
Procedure
1.   Subject performs all 8 tasks on appliance #1 (HP or Canon)
       Five minutes maximum allowed for each task

2.   Subject is instructed on the optimal way to perform each task
       Subject performs the task again until completed correctly.
       Additional instruction given as necessary

3.   Subject performs all 8 tasks on appliance #2 (Canon or HP)




                                                                     74
Three Interface Conditions
• Built-In
     User sees only physical appliance interfaces
     Physical HP → Physical Canon
     Physical Canon → Physical HP

• AutoGen
     User sees only PUC interfaces without consistency
     PUC HP → PUC Canon
     PUC Canon → PUC HP

• Consistent
     User sees a PUC interface without consistency followed by a Uniform
     interface generated to be consistent with the previous PUC interface
     PUC HP → Uniform Canon generated to be consistent with HP
     PUC Canon → Uniform HP generated to be consistent with Canon

                                                                            75
Conditions
3 interface conditions * 2 appliance orderings = 6 groups


               #1 - HP    #2 - Canon                 #1 - Canon   #2 - HP
    Built-In   Physical    Physical       Built-In    Physical    Physical
   AutoGen      PUC          PUC        AutoGen         PUC        PUC
 Consistent     PUC        Uniform     Consistent       PUC       Uniform




                                                                             76
Comparing Usability
Focus primarily on the first set of tasks
First set not affected by earlier tasks or instruction


               #1 - HP    #2 - Canon                 #1 - Canon   #2 - HP
    Built-In   Physical    Physical       Built-In    Physical    Physical
  AutoGen       PUC          PUC        AutoGen         PUC        PUC
 Consistent     PUC        Uniform     Consistent       PUC       Uniform




                                                                             77
Usability Results
                            #1 - HP       #2 - Canon                       #1 - Canon           #2 - HP
                 Built-In   Physical          Physical          Built-In       Physical         Physical
               AutoGen       PUC               PUC            AutoGen           PUC                 PUC
              Consistent     PUC              Uniform        Consistent         PUC             Uniform




                            *      *      *     †
                                                         *        *        *        * *     *   †




             Failures per subject also significantly less for PUC than Physical
             (Fisher’s Exact Test, p < 0.05)
                  HP:      1.125 Built-In vs. 0.125 PUC
                  Canon:      2.0 Built-In vs. 0.625 PUC
     *   *
                                                                 * = p < 0.05             † = p < 0.1 78
Comparing Consistency
Focus only on the second set of tasks
Previous experience during study should influence results


               #1 - HP    #2 - Canon                 #1 - Canon   #2 - HP
    Built-In   Physical    Physical       Built-In    Physical    Physical
  AutoGen       PUC          PUC        AutoGen         PUC        PUC
 Consistent     PUC        Uniform     Consistent       PUC       Uniform




                                                                             79
Consistency Results
                          #1 - HP          #2 - Canon                 #1 - Canon        #2 - HP
               Built-In   Physical          Physical       Built-In    Physical         Physical
             AutoGen       PUC                PUC        AutoGen         PUC             PUC
            Consistent     PUC              Uniform     Consistent       PUC            Uniform




                                     * *            †                 * *          †




               Very few failures in these conditions: 4 total for all tasks performed by
               32 subjects (256 tasks, 1.5% failure rate)


    *   *
                                                            * = p < 0.05           † = p < 0.1 80
Outline
• Introduction

• Personal   Universal Controller

• Uniform:   Generating User Interfaces with Consistency

• Huddle:   Combining Interfaces from Multiple Appliances

• Evaluation

• Discussion   and Future Work


                                                           81
Summary of Contributions
Designed an abstract appliance specification language
   •   Can describe the complete functionality of appliances
   •   Easy to learn and use

Built interface generators for multiple platforms and
modalities
   •   Use of dependencies to understand interface structure
   •   Smart Templates technique for rendering design
       conventions
   •   Generates usable interfaces for complex specifications

Generation of personally consistent interfaces
   •   Means of describing similar functions across appliances
   •   Algorithms for modifying interfaces to achieve consistency
       and allowing user control
   •   Improves usability for users with related experience

Generation of combined interfaces for multiple appliances
   •   Uses content-flow model to understand tasks across
       multiple connected appliances
   •   Flow-based interface allows easy specification of high-level
       tasks
   •   Task-specific aggregate interfaces can be generated based
       on content flows

                                                                      82
Future Work
Further benefits of automatic generation
  Assistive technology
    • Many different types of disabilities each with different needs
    • Today solutions must be developed separately for each user
    • Challenges:
          Developing general ways of specifying the user’s condition
          Writing rules that can manipulate the interface to address the user

  User-guided design
    •   Interface customization is hard
          Making interface changes, when possible, is often tedious and time-consuming
          Hard for users to know what they will do when getting started
          Once users are experienced, the cost of changing that interface is higher
    •   How could automatic generation improve this?
          Allows customization to be performed at a higher-level
          System can collaborate with user to iteratively arrive at a better interface

                                                                                         83
Future Work, cont.
Automatic modification vs. generation
  •   Can the benefits of automatic generation be achieved for all interfaces?
        Consistency
        Aggregation (mash-ups)

  •   Challenges
        Getting a model of the hand-designed interface
        Understanding the conventions of the hand-designed interface
        Creating rules that balance previous design with new design




                                                                                 84
Conclusion
Automatic generation of usable interfaces for end users is
possible
  •   Automatic design may be viable for some applications

Automatic generation allows interfaces to be customized to the:
  •   user’s previous experience
  •   user’s environment
  •   more?




                                                                  85
Acknowledgements
Thesis Committee                      PUC Project Members       Collaborators & Friends
  •   Brad A. Myers (chair)             •   Brandon Rothrock      •   Naomi Ramos
  •   Scott Hudson                      •   Duen Horng Chau       •   Desney Tan
  •   John Zimmerman                    •   Kevin Litwack         •   Daniel Avrahami
  •   Dan Olsen Jr.                     •   Thomas K. Harris      •   Gaetano Borriello
                                        •   Michael Higgins       •   Laura Dabbish
Funding                                 •   Joseph Hughes         •   Andrew Faulring
  •   National Science Foundation       •   Roni Rosenfeld        •   James Fogarty
  •   Microsoft                         •   Rajesh Seenichamy     •   Krzysztof Gajos
  •   General Motors                    •   Pegeen Shen           •   Darren Gergle
  •   Intel                             •   Htet Htet Aung        •   Andy Ko
  •   Pittsburgh Digital Greenhouse     •   Mathilde Pignol       •   Amy Nichols
                                        •   Suporn Pongnumkul     •   Mick Nichols
Equipment Grants                        •   Stefanie Shriver      •   Sally Nichols
  •   Mitsubishi (MERL)                 •   Jeffrey Stylos        •   Trevor Pering
  •   VividLogic                        •   Peter Lucas           •   Fleming Seay
  •   Lucent                                                      •   Irina Shklovski
  •   Lutron                                                      •   Rachel Steigerwalt
  •   Lantronix                                                   •   Roy Want
  •   Nokia                                                       •   Jake Wobbrock
                                                                  •   and many others…
                                                                                           86
Thanks for listening!



For more information…
http://www.jeffreynichols.com/
http://www.pebbles.hcii.cmu.edu/puc/
Automatically Generating High-Quality
User Interfaces for Appliances



Jeffrey Nichols
Human-Computer Interaction Institute
School of Computer Science
Carnegie Mellon University

Ph.D. Dissertation Defense
December 1, 2006
89
90
Tasks
1.   Send a fax to the number stored in the third speed dial
2.   Configure the fax function so that it will always redial a number that is busy
3.   Configure the fax function so that any document received that is larger than the
     default paper size will be resized to fit the default
4.   Configure the fax function so that it will only print out an error report when it has a
     problem receiving a fax
5.   Make two black-and-white copies of the document that has already been placed on
     the scanner of the printer
6.   Imagine you find the copies too dark. Improve this by changing one setting of the
     device
7.   Given a page with a picture, determine how to produce one page with several
     instances of the same picture repeated (demonstrated with actual paper copies)
8.   The device remembers the current date and time. Determine where in the interface
     these values can be changed (do not change them)

                                                                                               91
Modifying the Concrete Interface
Address visual inconsistency problems
    •   Side panels may have different orientations

Re-ordering within panels
    •   Flattening of abstract hierarchy into panels
        sometimes creates ordering inconsistencies

Concrete organization decisions
    •   Different organization may have been used
        from the abstract interface




How do we modify an interface to achieve consistency?   92
Complex Phone Interface



                            a.




             b.        c.        d.




   e.   f.        g.




                                      93
Motivation
These devices are hard to use




                                94
Content-Flow Template Pieces


  TV             TV    TV      TV




                 TV    TV      VCR




                                     95
What should specifications contain?
What information is needed about the appliance
to automatically generate remote control
interfaces?

Investigate via a design process
   Create interfaces by hand
     • AIWA Shelf Stereo
     • AT&T Telephone/Answering Machine

   Improve quality with heuristic analysis and think-
   aloud studies with several users

   Compare interfaces with actual appliance
   interfaces to validate PUC concept
     •   Users were twice as fast, and made half as many
         errors with our interfaces

   Analyze interfaces for functional information

                                                           96
UNIFORM
                                    B
      Consistency disabled – Copier A used first




          Copier A                 Copier B

                                                   97
UNIFORM
                  enabled
      Consistency disabled – Copier B used first




          Copier B                  Copier AConsistent Copier
                                                    A
                                                                98
Interface Generation, cont.
Graphical Interface Generators              User    options
                                           System   control x10, control audiophase
  •   Runs on today’s handheld devices      User    control x10

  •   .NET Compact Framework 2.0           System   control x10 (x10 voice)
                                            User    options

Speech Interface Generator                 System   global off, all lights on, all lights
                                                    off “...”

  •   Masters work of Thomas Harris         User    control audiophase
                                           System   control audiophase (stereo voice)
  •   Built on top of the PUC framework     User    options
                                           System   stereo power, x-bass, volume “…”
  •   Implemented using Universal Speech    User    stereo power
      Interface (USI) techniques           System   <stereo is turned on>
      [Rosenfeld 2001]
                                            User    options
                                           System   stereo power, x-bass, volume “…”
                                            User    more
                                           System   mode, tuner, CD
                                            User    stereo power is, options
                                           System   on, off
                                                                                            99
Interface Generation
Process
 1.   Build abstract interface from specification
       • Tree structure, including dependency
         structure
       • Abstract choice of controls

 2.   Build concrete interface from abstract
      interface
       •   Custom to screen size, resolution,
           platform, etc.
 3.   Fix layout problems
       •   Insufficient height or width, etc.
 4.   Display interface to user




                                                    10
Knowledge Base
Functional Mappings
     • We  support six types of mappings, based              Previous Specifications
                                                             Functional Mappings

       on the results of the specification                    Between Specifications
                                                             Previously Generated UIs

       authoring studies
                                                             Knowledge Base
           general, state, node, group, list, and
           template

     • Specifies that a function in one
       specification is similar to a function in
       another
           Mappings are grouped within the knowledge
           base in mapping graphs



How do we describe the similarity between specifications?                                10
Where do mappings come from?
Three ways mappings may be found
   Automatically extracted immediately before interface generation
        •   Challenging because of limited semantic information in specifications
        •   We have implemented two mapping algorithms, one based on work in
            automatic schema mapping from the database community
        •   Current best algorithm finds about 60% of mappings
        •   We are also exploring other techniques

   Specified by the user
        •   Manually entered as text
        •   Interaction techniques?

   Downloaded from the Internet
        •   Manufacturers?
        •   Internet communities? (e.g. www.remotecentral.com)


How do we find similar functions across appliance specifications?                   10
Generated Phone Examples




  Copier A without   Copier B without
    consistency        consistency
                                        10
 Consistent Phone Interfaces




   Copier A
consistent with
   Copier B




                               10
Consistent Phone Interfaces, cont.




                               Copier B
                            consistent with
                               Copier A



                                              10
Aggregate User Interfaces
Three kinds
  •   Control of Active Flows      Volume for TV         Volume for Receiver




           “Copy Tape”              “Watch TV               “Watch TV
                                using TV speakers”   using Receiver speakers”

                                                                                10
Flow-Based Interface




                       10
What’s next?
Consistency
  •   Studies of consistency
       •   Unique functions
       •   Trade-offs between dimensions of consistency
           (e.g. visual appearance vs. structure)

  •   Additional domains
       •   Mobile phones
            •   Old phone and new phone
       •   Intelligent design assistants & analysis




                                                          10
What’s next?
Combining User Interfaces
  •   Task-based Mash-ups
  •   User-guided adaptive interfaces




                                        10
Task-Based Mash-Ups
Today’s mash-ups mainly combine data

Use mash-ups to streamline tasks
  •   Arranging travel
  •   Paying bills
  •   Various business tasks…expense reports, etc.


Questions
  •   How to identify tasks?
  •   How to find application functionality for each task step?
  •   How do we then build an interface from these different pieces of application
      functionality?
  •   Security, Privacy, etc…

                                                                                 11
Combining Interfaces, cont.
User-guided adaptive interfaces

Process
  1.   System observes a user’s interactions
  2.   Display a visualization of interaction history
  3.   User can select collections of interactions that represent a task
  4.   An interface is automatically generated for that task
  5.   User can iterate on their selection to refine interface




                                                                           11
Completeness, cont.
How complex are they?

           Vars &                   Labeled    Smart       Max Tree     Ave. Tree
           Cmds       Groups        Groups     Templates   Depth        Depth
   Max*         136            64         46         30           11         6.20
Average*      36.25      16.93          11.6       6.21         4.79         3.59
  GM Nav        388        171           136         79           11          7.00
                                                 *not counting GM Navigation system




We can generate interfaces for all of these specifications


                                                                                      11
Interfaces


                                                                                          Max     Ave.
                                                       Vars &            Labeled Smart    Tree    Tree
                                                       Cmds     Groups   Groups Templates Depth   Depth

                                               Max*      136       64       46       30      11    6.20
                                            Average*   36.25    16.93      11.6    6.21    4.79    3.59
                                             GM Nav      388      171      136       79      11    7.00
                                                 HP        87      35       28       15       7    5.59
                                              Canon      136       64       46       30      11    6.20




    HP printer           Canon printer             HP printer                       Canon printer
without consistency   without consistency    consistent with Canon                consistent with HP




                                                                                                       11
Interfaces




     HP printer        Canon printer
                  Consistent Canon printer   11
Usability Results
                                          #1 - HP    #2 - Canon                       #1 - Canon       #2 - HP
                           Built-In       Physical       Physical          Built-In    Physical        Physical
                         AutoGen           PUC            PUC            AutoGen         PUC            PUC
                        Consistent         PUC           Uniform        Consistent       PUC           Uniform




             *      *                 *   †          *              *            *     * *         *      †


                                                                                                                     *    *

Failures per subject also significantly less for PUC than Built-In (Fisher’s Exact Test, p < 0.05)
    HP:    1.125 Built-In vs. 0.125 PUC
    Canon:    2.0 Built-In vs. 0.625 PUC

                                                                                                   * = p < 0.05   † = p < 0.1 11
Consistency Results
                                   #1 - HP     #2 - Canon                  #1 - Canon   #2 - HP
                        Built-In   Physical     Physical        Built-In    Physical    Physical
                       AutoGen      PUC           PUC          AutoGen        PUC        PUC
                     Consistent     PUC         Uniform      Consistent       PUC       Uniform




                  * *              †                            * *           †


                                                                                                            *     *


Very few failures in these conditions: 4 total for all tasks performed by 32 subjects (256 tasks, 1.5% failure rate)



                                                                                        * = p < 0.05   † = p < 0.1 11
Similarity and Mapping Graphs
One-to-one functional mappings
between specifications
Six types of mappings determined
from authoring study
Similar functions across specs are                                                 Previous Specifications
grouped in “mapping graphs”                                                        Functional Mappings
                                                                                    Between Specifications
  •   Power
                                                                                   Previously Generated UIs
  •   Volume                               Answering Machine
  •   Media Controls                       State: Mode                                      Mitsubishi VCR

  •   Etc.
                                               { Play, Stop, Pause }                    State: Base
                                                                                   Knowledge Mode
                                           Command: Play New Messages                             { Play, Stop, Pause, Rewind, F-Fwd, Record }


Node counts                                                                   0              3
  •   Store generation history
  •   Used to find basis for consistency                                                                          Cheap VCR
                                                                                                                  Command: Play
Edges                                                                                                    1
                                                                                                                  Command: Stop
                                                                                                                  Command: Pause
  •   Track whether representation can                                    0                                       Command: Rewind
      be modified to achieve consistency     DVD Player                                                           Command: F-Fwd
                                                                                                                  Command: Record
                                             State: Mode
Choice in consistency supported                   { Play, Stop, Pause }

by manipulating the node counts              Command: Next Track
                                             Command: Previous Track




How do we design our system so that the user has control?                                                                                        11
Achieving consistency with Uniform
Uniform is the first system to automatically achieve personally
consistent interfaces

Many hard problems to address:
  •   What is consistency?
       •   What consistency issues arise in our domain?

  •   How do we describe the similarity between specifications?
  •   How do we find similar functions across appliance specifications?
  •   How do we modify an interface to achieve consistency?
       •   How do we address unique functions?
       •   How do we incorporate consistency constraints with all appliances?

  •   How do we design our system so that the user has control?

                                                                                11
What is consistency?
Previous work
   •   Many definitions of consistency
       [Nielsen 88, Kellogg 87]
   •   Explored as knowledge transfer by psychologists
       [Keiras & Polson 85, Polson 88]
   •   Previous systems have achieved consistency within an application or for
       families of applications:
        ITS [Wiecha 90], PUC [Nichols 02], others…

My Studies of Specification Authoring
   •   Multiple authors wrote complex VCR specifications
   •   Controlled for author, VCR model
   •   Result: Specifications can differ substantially across appliances and
       authors, leading to inconsistent interfaces
What is consistency?                                                             11
Summary of Huddle
Huddle automatically generates interfaces
for systems of connected appliances
  •   Content-flow model tells us how appliances
      are used together in a scalable way

  •   Flow-based interface allows user to quickly
      achieve high-level goals

  •   Aggregate interfaces are generated based
      on flows for low-level control




                                                    Flow-Based Interface   Aggregate Interface


                                                                                            12
Summary of Uniform
Uniform automatically generates personally
consistent interfaces
  •   Consistency helps users transfer
      knowledge between interfaces
  •   Similarity is described by one-to-one
      functional mappings grouped in mapping
      graphs
                                                                             Original
  •   Mappings can be found automatically or be
      specified by manufacturers or users
  •   Similarity information is used to modify
      new interfaces to achieve consistency
       •   Functional and structural modifications
       •   Heuristics for addressing unique functions
                                                        Uniform
       •   Most changes made to abstract interface      [Nichols, CHI 06]


                                                                            Consistent 12

				
DOCUMENT INFO