nicholsslides

							Recent Work in Model-Based User Interfaces




      Jeffrey Nichols
      Lecture #13
      05-830: Advanced User Interface Software
Last time…

   Model-based User Interfaces
       Automatic generation of the user interface so the
        programmer won’t do a bad job.
       Dialog boxes are relatively easy to generate
       The full application interface is hard to generate
       Abstract descriptions of the interface can be
        longer and harder to generate than implementing
        the interface itself.
       Interface builders turned out to be easier…
But work continued…

   Focus Changed
       Task models were leveraged more
       Design assistant aspect emphasized
   A Couple Projects of Interest:
       Trident
       Mecano & Mobi-D
       FUSE
       AIDE
TRIDENT
   Vanderdonckt, J., Knowledge-Based Systems for Automated User Interface Generation: the
    TRIDENT Experience, Technical Report RP-95-010, Facultes Universitaires Notre-Dame de la
    Paix, Institut d’Informatique, Namur, 1995.

   An interface design assistant
   Interesting features:
       Knowledge-based approach (i.e. expert system)
           Choosing Widgets
           Doing Layout
       Use of Task Models
           Decides where separate windows are needed
Choosing Widgets

   Used a decision tree
       Chose abstract
        interaction objects
        (AIO)
            Similar to Brad’s
             Interactor Model
       Lots of parameters
            Continuous?
            Capacity
            Etc.
Choosing Layout

Uses Right/Bottom Strategy
     Next component is placed to the right or below
      the current component
     Decision made by heuristics or designer
Right/Bottom Strategy
Windows from Task Models

   Basically used for constructing wizard-like interfaces
       What information should be on the first screen, etc.
What are task models, anyway?

   Description of the process a user takes to reach a
    goal in a specific domain
   Typically have hierarchical structure
       Introduced by GOMS


   Number of different task modeling languages
       GOMS
       UAN
       ConcurTaskTrees
ConcurTaskTrees

   Developed by Fabio Paterno et
    al. for the design of user
    interfaces
   Goals
       Graphical for easy interpretation
       Concurrent model for
        representing UI tasks
       Different task types
           Represent all tasks, including
            those performed by the system
Task Building Process

   Three phases                          Temporal Relationships
       Hierarchically decompose              T1 [] T2 - Choice
        the tasks                             T1 ||| T2 - Interleaving
       Identify the temporal                 T1 |[]| T2 - Synchronization
        relationships among tasks
                                              T1 >> T2 - Enabling
        at same level
                                              T1 []>> T2 - Enabling with
       Identify what objects are
                                               Information Passing
        manipulated and what
        actions can be performed              T1 [> T2 - Deactivation
        on them, and assign these             T1* - Iteration
        to the tasks as appropriate.          T1(n) - Finite Iteration
                                              [T1] - Optional
                                              T – Recursion
Example

   Note: First example is ambiguous
Another Example
Building/Editing Task Models

   Tools are available
       ConcurTaskTrees
        Environment




http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”
Recent Systems

   XIML – eXtensible Interface Markup Language
       Developed by the makers of Mecano/Mobi-D and Trident
       Kitchen-sink language for modeling any part of the
        interface design process
   XWeb
       Now known as ICE – Interactive Computing Everywhere
   ICrafter
       A system for integrating user interfaces from multiple
        devices
   Personal Universal Controller
       My research…
XIML

eXtensible Interface Markup Language
     Designed by RedWhale Software
     Intended to support the
      full lifecycle of interface
      building
XIML Requirements

   Central Repository of Data
       For one user interface or many
   Comprehensive Lifecycle Support
   Abstract and Concrete Elements
   Relational Support
   Underlying Technology
       XIML must be independent of particular tools
Models in XIML

   An XIML document can contain any type of
    model
       Task
       Domain
       User
       Presentation
       Dialog
Example Use for XIML

Multi-platform interface development
Status of XIML

   Used by RedWhale Software to drive their
    interface consultant business
       They have developed many tools
           move interaction data to/from XIML
           Leverage data in XIML to better understand various
            interfaces
           Automate parts of the interface design process
Model-Based Interfaces for Control

   XWeb
   ICrafter
   PUC
XWeb

   Work by Dan Olsen and group at BYU
   Premise:
       “Pervasive computing cannot succeed if every device must
        be accompanied by its own interactive software and
        hardware…What is needed is a universal interactive
        service protocol to which any compliant interactive client
        can connect and access any service.”
       The web comes close to solving this problem, but is
        interactively insufficient.
XWeb Protocols

   Based upon the architecture of the web
       XTP Interaction Protocol
       Server-side data has a tree structure
       Structured Data in XML
       URLs for location of objects
           xweb://my.site/games/chess/3/@winner
           xweb://automate.home/lights/livingroom/
           xweb://automate.home/lights/familyroom/-1
XWeb & XTP

   CHANGE message (similar to GET in HTTP)
       Sequence of editing operations to apply to a sub-tree
           Set an attribute’s value
           Delete an attribute
           Change some child object to a new value
           Insert a new child object
           Move a subtree to a new location
           Copy a subtree to a new location
Platform Independent Interfaces

   Two models are specified
       DataView – The attributes of the service
       XView – A mapping of the attributes into high-level “interactors”
       Interactors are somewhat like abstract interaction objects
           Atomic
               Numeric
               Time
               Date
               Enumeration
               Text
               Links
           Aggregation
               Group
               List
XWeb Example

DataView
XWeb Example

XView
XWeb Example

Interface
Other XWeb Details

   Has simple approach for
    adjusting to different screen
    sizes
       Shrink portions of the
        interface
       Add additional columns of
        widgets
   Also capable of generating
    speech interfaces
       Based on a tree traversal
        approach like Universal
        Speech Interfaces
ICrafter

   Part of the Interactive Workspaces research project
    at Stanford
   Main objective:
       “to allow users of interactive workspaces to flexibly interact
        with services”
   Contribution
       An intelligent infrastructure to find services, aggregate them
        into a single interface, and generate an interface for the
        aggregate service.
       In practice, much of the interface generation is done by
        hand though automatic generation is supported.
ICrafter Architecture
How is aggregation accomplished?

   High-level service interfaces (programmatic)
       Data Producer
       Data Consumer
   The Interface Manager has pattern
    generators
       Recognize patterns in the services used
       Generate interfaces for these patterns
           This means that unique functionality will not be available
            in the aggregate interface
Automatic Generation in ICrafter
Manual Generation in ICrafter
Personal Universal Controller

   My work with Brad
   Problem:
       Appliance interfaces are too complex and too
        idiosyncratic.
   Solution:
       Separate the interface from the appliance and use
        a device with a richer interface to control the
        appliance:
           PDA, mobile phone, etc.
Idea
                    Specifications


                       Control
                      Feedback


   Control existing appliances
   Generate multi-modal interfaces
Architecture - Comm. Protocol
               Interface Generators
               Appliance Adaptors
               Specification Lang.




              XML-based
 Language Design
Approach
     Create reference interfaces
          AIWA Shelf Stereo
          AT&T Telephone/Answering
           Machine
     Test interfaces with subjects
          Users twice as fast and made half
           the errors with reference interfaces
           as compared to manufacturers’
           interfaces
     Analyze interfaces for functional
      information
Language Elements

State Variables and Commands
      Represent functions of appliance
      State variables have types
           Boolean, Enumeration, Integer, String, etc.
      Variables sufficient for most functions but not all
           “seek” button on a Radio

Label Information
      One label not suitable everywhere
           The optimal label length changes with screen size
           Speech interfaces may benefit from pronunciation and
            text-to-speech information
Language Elements, cont.

Group Tree
     Specify organization
      of functions
     We use n-ary tree
      with variables or
      commands at
      leaves
Language Elements, cont.

Dependency Information
     Formulas that specify when a variable or
      command is active in terms of other state
      variables
           Equals, Greater Than, Less Than
           Linked with logical operators (AND, OR)

     For example,
            <and>
              <equals state=“PowerState”>true</equals>
              <equals state=“RadioBand”>AM</equals>
            </and>
 Interface Generators
Generators for Two Modalities
     Graphical
          Implemented for PocketPC in Java 1.1
          Uses dependency information to generate panel structure of
           interface
     Speech
          Implemented using Universal Speech Interface (USI)
           techniques [Rosenfeld 2001]
          Uses dependency information to disambiguate shortcut
           words (e.g. “play”) and resolve pre-conditions for a
           requested function (e.g. “play CD”)
 Graphical Interface Generator

Focuses on panel structure
of user interface
     Small groups of controls
      have basic layouts
     Complexity comes from
      structure of groups
     Structure can be inferred
      from dependency info!
Inferring Structure

Find sets of variables that are
  “mutually exclusive”
      Every variable in a set will
       never be active at the same
       time as a variable in another set

Create structure with sets, using
  overlapping panels
Choosing Panel Types
     a)         b)            c)




 full screen   tabbed   partial screen
Making the Interface Concrete

Finish conceptual layout
     Choose controls (decision
      tree)
     Choose row layouts
       (one column, two column, etc.)

Allocate space
     Examine panel contents and
      choose sizes

Instantiate and place controls
Generating Speech Interfaces

Automatically build USI tree from dependencies
      Allows verbal navigation of functional groups
Automatically generate grammar for parser
      Phrases for query and control
        “What is playmode?”
        “Set playmode to play”
        “play”

Automatically generate language model and
pronunciation for recognizer