Tools for Online Collaborative Decision Making

Document Sample
Tools for Online  Collaborative Decision Making Powered By Docstoc
					   Tools for Online
Collaborative Decision
          Richard Barber

   Information Technology and New Media


         Supervisor: Weigang Wang

             Third Year Project

          University of Manchester
         School of Computer Science
Tools for online collaborative decision making
Richard Barber
May 5th 2010
Supervisor: Weigang Wang

This report focuses on the design of a piece of groupware called PowerMeeting. The report
evaluates PowerMeeting's design from two viewpoints, that of a single user interacting with the
interface and that of multiple users using the interface to collaborate. To do this two evaluations
were done on the interface, one using Nielsen's heuristics and the other using heuristics that are
groupware specific. User centred evaluations were also used to evaluate the design. This report
publishes how these methods were used, the results of them and how they may be used to improve
the system. Screens of a prototype shown in the report suggest possible improvements that could
be made to the interface to increase its usability.


 1.      Introduction ................................................................................................................................ 3
      1.2        PowerMeeting..................................................................................................................... 3
      1.3        Aims and Objectives ............................................................................................................ 4
 2.      Literature Review ........................................................................................................................ 6
      2.1        PowerMeeting..................................................................................................................... 6
      2.2 Usability Design ......................................................................................................................... 7
      2.3        Groupware Design .............................................................................................................. 9
            2.3.1 Groupware Heuristics..…………………………………………………………………………………………………9

            2.3.2 Other Considerations..……………………………………………………………………………………………….10

      2.4        Conclusion ......................................................................................................................... 11
 3.      Method ..................................................................................................................................... 13
      3.1 Usability Evaluation................................................................................................................. 13
      3.2 User Centred Evaluation ......................................................................................................... 23
      3.3 Groupware Evaluation ............................................................................................................ 28
 4.      Redesign .................................................................................................................................... 37
 5.      Conclusion ................................................................................................................................. 50
 6.      References ................................................................................................................................ 52
 Appendix A: Evaluation of Error Messages ....................................................................................... 53
 Appendix B: Results from evaluation using Nielsen's heuristics....................................................... 56
 Appendix C: Consent Form ............................................................................................................... 61
 Appendix D: User workshop tasks and post observation questionnaire .......................................... 62
 Appendix E: User Workshop Results ................................................................................................. 66
 Appendix F: results from evaluation using Groupware heuristics .................................................... 69

   1. Introduction
This report will look over the work that I, Richard Barber have been carrying out over the
past year. The overall purpose of the project was to improve the design of the groupware
PowerMeeting using different methods of evaluation. It is hoped that the results from this
project can be used to improve future builds of PowerMeeting, increasing its usability.

   1.2 PowerMeeting
The project focused on a piece of groupware that has been built by my project supervisor
Weigang Wang. PowerMeeting is a fully functioning prototype. The website of
PowerMeeting defines it as:

“A Web-based synchronous groupware framework. It also refers to a research prototype that
    demonstrates the kind of groupware built with the framework. Such groupware allow
      participants to plan, perform, and coordinate their synchronous collaboration”[1]

A screen shot of PowerMeeting is shown in figure 1. As you can see it runs in a standard web
browser and is split into five sections. It is designed to be used like a desktop application[2],
this is reflected by the menu bar running along the top of the interface. There are then four
panels that make up the rest of the interface. The three panels on the left consist of an agenda
panel that display each of the agenda items (tools) that have been created. There is the
participant panel that lists the online participants and then a panel that allows group chat to
occur via an instant messaging tool. Finally there is the main large panel which is where open
tools are displayed and worked on.
                                Figure 1 – PowerMeeting's current design

   1.3 Aims and Objectives
The focus of this project was to give a formative evaluation on PowerMeeting to identify
where the system could be improved and create a prototype of an improved version of the
interface. This prototype would be created from the findings of various evaluation methods.
Two separate heuristic evaluations were carried out on PowerMeeting in the project. The first
evaluation was completed using the popular set of heuristics proposed by Nielsen. Another
heuristic evaluation was completed using a set of heuristics focussed specifically on
groupware systems. The findings from each of these evaluations are summarised in
spreadsheets. (Appendices B and F)

The reason that two separate heuristic evaluations have been carried out is that it allowed me
to look at the system from two different viewpoints. As PowerMeeting is a groupware it is
used to collaborate online with different users. It is therefore important that this form of
interaction is allowed in the system and works effectively, this is what the second set of
heuristics covers. Equally there is going to be some interaction between the system and a
single user.

Looking at Norman‟s model of interaction (figure 2) we can see how a single user interacts
with a system.

Figure 2 – Norman’s model of interaction [3]

As we know this model is split into two gulfs, the gulf of execution and the gulf of
evaluation. In a collaborative environment the gulf of evaluation is going to be experienced
by all users who are on the same tool. What is meant by this is that all the users are going to
be looking for what effect a certain action has had on the workspace by an individual. So if
one participant has made a change to the workspace then the rest of the participants need to
be able to see this change (perceive it), interpret the system state and evaluate the system
state with regards to their goals and intentions. The goals and intentions of the different users
should be the same as they are working within a group towards the same goal, however if
there are any conflicts between these goals and intentions then a discussion of this should be
facilitated. Figure 3 shows my adaptation of Norman‟s model of interaction with respect to
groupware tools.

Figure 3 – Adaption of Norman’s model of interaction

The main changes to this model to adapt it for groupware systems are that the changes made
to the system need to be perceived by all the users and some level of discussion needs to take
place between the users in order for the group to establish a new goal. Discussion is one of
the key aspects in groupware that needs to be allowed for. This model was used as an aid in
the evaluation.

   2. Literature Review
This section will cover the main sources used in the project explaining why they were used.
This section is split into four sub sections explaining the literature for each of the three
sections of the project and PowerMeeting as a whole. First to be discussed will be
PowerMeeting itself so that it is understood exactly what PowerMeeting is and what the
history behind it is. Second to be discussed will be the individual usability side of the project,
looking at what literature has been used to get information in how to improve
PowerMeeting's interface and overall usability. Thirdly the user centred design literature will
be discussed, explaining where the methods used came from and the background to user
centred design. The final part of this section will look at the literature on designing for a
collaborative design looking at how past literature can be used to help to improve
PowerMeeting‟s design to make it more effective for online synchronous collaboration.

   2.1 PowerMeeting
PowerMeeting is an online piece of groupware created by Weigang Wang. PowerMeeting is a
prototype that is being worked on incrementally and is therefore developing over time with
improvements made and tools added at various stages. There are different types of groupware
available, these are shown in the matrix (figure 4). Online Groupware such as PowerMeeting
tends to be a hybrid of synchronous distributed interaction (same time, different place) and
asynchronous interaction (different place, different time). There are three papers available on
the PowerMeeting website, [1] which explain the reasoning behind PowerMeeting's

               Figure 4 – Matrix showing different groupware types [4]

In the paper „PowerMeeting: GWT-Based Synchronous Groupware‟ [2], Wang explains that
by using new technologies such as Google Web Toolkit (GWT) and Ajax, a rich user

experience can be gained. This rich user experience is achieved by the set of graphical user
interface widgets available from using GWT to develop the system. For example GWT
allows web pages to have drag and drop functionality creating “web applications that have
the features and functionality of traditional desktop applications” [2]. Wang‟s work tends to
focus on the technical side of things i.e. the programming and architecture used for
PowerMeeting and so not much can be drawn from the papers about the actual interface
design. It is clear from his work that PowerMeeting's aim is to be a desktop like application,
providing a rich user experience. In parts he touches upon some important issues that need to
be considered in the design of groupware. For example in another paper title „A Multi-
Dimensional Framework for Facilitating Wide Participation and Common Understanding‟
[5], it is suggested that “the history of a collaborative process should be maintained to help
people keep track of their progress or for those not presented on a part of the collaborative
work to see how the progress has been made”. Relating this extract to interface design it
suggests that there is a need for each collaborator (the users on the groupware) to be able to
see not only the work that has been developed within the groupware but also how it has been
developed. This raises an issue of what could be done to show the history of the collaboration
in the interface so that the users know how something has developed and can keep track of
future developments. In the same paper it is also suggest that the method of direct
manipulation is a good interaction technique to use in groupware systems as this method
could help develop an element of common ground[12] between the collaborates. As they
would like to see things that other participants are doing on the system.

   2.2 Usability Design
There is a plethora of literature available on usability and many different theories and best
practices. It would be impossible to discuss them all in this report. In the project there were a
number of these usability literatures used to aid the redesign. Jackob Nielsen is one of the
main individuals when it comes to designing usable systems. His heuristics[6] which were
first developed in 1990 and later refined in 1994[7] are still used today in order to create
usable systems. Nielsen is defiantly an expert when it comes to usability issues. However it
can be argued that due to their age the heuristics he proposed are not relevant in modern day
systems. Although computer systems and there interfaces have undeniably moved on, many
of their core principles remain unchanged. Usability lies in the way humans interact, our
interaction has not changed, rather systems have changed to allow for more natural

interaction. If we look at the heuristics then we can see that the principles of them still count
in today‟s interfaces. For example let us look at the following heuristic proposed by Nielsen.

                                      “Error prevention-
   Even better than good error messages is a careful design which prevents a problem from
  occurring in the first place. Either eliminate error-prone conditions or check for them and
        present users with a confirmation option before they commit to the action.” [6]

The basic rule here is that conditions that are going to cause errors to the users of the system
should not be presented to the user. This would stop the error being caused which may then
reduce user stress. This example shows, in a simple form that the heuristics are still relevant.
Error prone conditions should still be removed from systems as users don‟t like errors.
Moreover the heuristics were originally developed with a focus on desktop applications. As
discussed earlier there is an emphasis on PowerMeeting being a rich internet application
(RIA) [2] this meaning that the interface takes on more of the look of a desktop application.
This means that the heuristics can apply quite well in this situation and therefore should not
need adapting as much as they may need to be with other systems. A more recent publication
defines Nielsen‟s heuristics as a set of heuristics that are used to conduct a heuristic
evaluation. This publication is the book „The Resonant Interface, HCI Foundations for
Interaction Design‟ by Steven Heim [3]. This book was published in 2007 which shows that
although these heuristics are old they are still viable and used with modern systems.
Furthermore the heuristics are now even being used to evaluate iPhone applications, showing
that they are still relevant to the most modern systems. [8]

With regards to interface design there is much more available as an aid than Nielsen‟s
Heuristics. Ben Schneiderman is another man well known in the field of computer science.
He defined the eight golden rules which are defined in his book „Designing the user interface‟
(1998) [9]. These golden rules have much the same underlying principles as Nielsen‟s
heuristics, for example the sixth golden rule is defined „Permit easy reversal of actions‟
which is much the same as Nielsen's heuristic „User control and freedom‟ which says that the
system should support undo and redo functionality. The golden rules are younger when
compared to Nielsen‟s heuristics, this again shows that even though times have changed there
is still a need to design with the same principles in mind.

   2.3 Groupware Design
Groupware such as PowerMeeting is relatively new. Most groupware that has been developed
in the past has been separate from a web browser and needed a download or installation.
PowerMeeting‟s purpose is to give a groupware experience by simply using a standard
browser such as Internet Explorer of Firefox as a client [2].

   2.3.1 Groupware Heuristics
Groupware is difficult to evaluate [10] and so ways of evaluating it have been scarce over the
years. Kevin Baker, Saul Greenberg and Carl Gutwin are all computer scientists who have
worked on creating a set of heuristics that are solely for use on groupware systems using
Gutwin‟s framework for the mechanics of collaboration[10]. They produced a set of eight
heuristics that can be used in the same way as Nielsen‟s to evaluate the efficacy and
effectiveness of groupware applications. These heuristics are published in the paper
„Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration‟[10]. In this
paper they discuss methods that can be used to achieve the heuristics. However some of the
methods suggested are not feasible when it comes to groupware applications like
PowerMeeting as they require some external hardware. For example in order to satisfy the
third heuristic „Provide consequential communication of an individual‟s embodiment‟ the
paper suggest that a good way of achieving this heuristic would be to have some external
hardware that represents other users. The paper describes such a hardware created at MIT
Media Labs which takes the form of a transparent mask on a motorised platform that has a
video image of a user‟s face projected onto it. Sensors are used to show the persons head
movements with the mask. Although this may be a good way to meet the heuristic as it would
show other user‟s embodiment, it is not very feasible. PowerMeeting's purpose is to begin to
get groupware widely used. Having expensive hardware would not achieve this and so
another method of showing users embodiment needs to be used.

To support these heuristics a paper was produced by the same academics explaining how the
heuristics were to be used [11], how they used them and what results were gained from this.
This paper reports positive results from using the heuristic‟s to evaluate two different pieces
of groupware. The paper also speculated that the method may have done better if the experts
carrying out some of the evaluations would have been more motivated. What this paper
shows is that this new set of heuristics for evaluating groupware does have some viability in
that they have proven that a good proportion of usability problems have been found. It does
go on to say that heuristic evaluations are worth doing with one evaluator as a number of

problems are still found. However it is suggested that a number of evaluators are used to find
more problems. As this is not a group project I am unable to have other people working with
me. This is why I am using three methods of evaluation including user evaluation to help me
to find the problems with PowerMeeting.

   2.3.2 Other considerations
There is more to designing groupware than the heuristics discussed above. Other things to
consider are Clarks common ground theory [12], the media richness theory[13][14] and
social presence[13]. These are topics that I am currently studying in a course unit titled
„Advance HCI‟. Establishing common ground between members of a group allows them to
each have a mutual understanding. This mutual understanding then allows the group to
communicate at a level which they know the group will understand. Therefore to have an
effective piece of groupware it will need to allow a group to establish a degree of common
ground. The media richness theory can help with this. Media richness has been explained as
„the ability of a medium to carry information‟[13]. Good media richness results in a good
understanding between users, for example face to face conversation is the most media rich
form of communication[13][14]. The reason for this is that a lot of common ground is
established between two people conversing in the same environment. They both are aware of
their surroundings and the other person‟s emotion and therefore the understanding of the
conversation. There is a large social presence[13] in this environment that allows the people
conversing to gain an understanding of each other. Groupware should try and replicate this
face to face interaction to allow common ground to develop in the group and give some form
of social presence so that the users have more understanding of each other. Systems such as
Second Life incorporate these theories into their design. For example, having an avatar in a
virtual environment gives social presence and common ground. The media richness theory
suggests that the second best to face to face interaction is video conferencing as you can still
see a moving image of the person in real time and also talk in real time. However some
common ground is lost here as the other person is no longer in the same environment and
your view is restricted to only seeing the other person down to their shoulder and not their
surroundings. The media richness theory grades different types of communication media
according to four criteria, these are feedback, multiple cues, message tailoring and emotions.
Table 1 shows how different types of communication media match to this criteria.

              Table 1- how different communication media match to the media richness criteria[13]

With PowerMeeting video may be unfeasible as having many videos on the screen will cause
the workspace size to be compromised. Having a synchronous audio channel is something
that would work well in PowerMeeting. To an extent this already exists as users may use
Skype to chat while using PowerMeeting but an internal audio channel would be an

   2.4 Conclusion
From the literature on PowerMeeting an assumption may be formed that the appearance of
the system is supposed to be desktop like and simple to use as its purpose is to allow wide
spread use of the system. From the literature on usability, Nielsen‟s heuristics are a very
established tool for carrying out evaluations on systems, and so by using them some good
results should be able to found. There are many aspect of interface design that need to be
looked at. There is much more available than that mentioned above for example other
principles should be taken into consideration when designing interfaces such as Gestalts‟
principles [3]. Principles such as these are described in books by Heim[3] and Preece et
al[15]. The heuristics created specifically for the evaluation of groupware are much newer
than those created by Nielsen. They are, however, created to be used as Nielsen‟s heuristics
are and early evaluations completed with them show positive results[11].

In order to create a good formative evaluation of PowerMeeting each aspect mentioned above
should be considered. It is important to take into consideration the theory behind
PowerMeeting and its purpose. It is also important to take interface design principles and

practices into consideration as well as groupware design issues. The design should also allow
for social presence by considering media richness as discussed above. When creating the
proposed redesign these topics should again be taken into account.

   3. Method
This section of the report covers the methods that were used in each evaluation that was
carried out on PowerMeeting. There are three subsections to this section. Each section
discusses a different evaluation methodology in the order they were completed. The first
subsection looks at the evaluation of PowerMeeting from an individual users point of view
using Nielsen's Heuristics. The next subsection discusses the user centred evaluation and
finally the evaluation using groupware specific heuristics is discussed. The methods that were
used and how they were used shall be discussed here. Also a summary of the results from
each evaluation summarising the findings will be presented.

   3.1 Usability Evaluation
Nielsen's heuristics[6] and various usability principles[3][9][15] were used to evaluate
PowerMeeting's usability with regards to how a single user would interact with the system..
The main focus was on Nielsen's heuristics as they cover a wide range of usability principles.
The referenced books were mainly used as background research to learn about certain
usability principles and their importance.

Listed in Heim‟s book[3] is some advice from Nielsen on how a good heuristic evaluation is
carried out. One piece of this advice was not followed in my project. The advice is that a
heuristic evaluation should be carried out by more than one user, this would give a more
accurate evaluation and also find more usability problems. However, as this was an
individual project it was not feasible for there to be more than one evaluator and so it was
only possible to conduct an expert evaluation by myself. The user workshops and focus
groups should help to discover more problems and therefore make up for the lack of
evaluators somewhat.

Another piece of advice listed is that an evaluator should go through the systems interface
many times. First off the evaluator should go through the system in order to get a feel for it
and understand it. As the evaluator goes through the system again they should start to be
focussed on precise issues. To follow this advice I went through the system multiple times
starting with a general pass through the system moving onto going through the system for a
specific purpose for example to evaluate the system against a specific heuristic. A further
piece of advice given supports my reasoning for the groupware specific evaluation being
carried out, it says „the evaluator may use heuristics that are more specific to the design that
is being evaluated‟[3]. This point is fundamentally saying that another set of heuristics may
be used for the evaluation that more closely links to the system under investigation and so

Nielsen's heuristics are not the only heuristics that can be used for this type of evaluation.
Another point stated is to document the evaluation as a list of usability problems with the
associated heuristic and an explanation stating in what way the interface does not adhere to
that heuristic. This advice was followed and is shown in the results later in this report.

The first pass through the system was used to get a feel for the system as suggested. I noted
down a few first impressions I had about the system in case I missed them in future passes
through the system. The main issues that were picked up on in the first run though was the
amount of white space and how isolated the interface looks. It is tucked away in the top left
corner of the browser window which doesn‟t seem natural for a webpage or a desktop
application. Webpages tend to be centred or use up most of the screen and desktop
applications tend to fill up the screen. PowerMeeting doesn‟t conform to either of these
designs. Although there is no rule or principle stating that the application should be
centralised or full screen I feel that the interface should be centralised in the browser window
to conform to most other webpages. Nevertheless this is not reason enough alone to make this

After this initial pass through the system subsequent passes through became more focussed as
is the advice by Nielsen. For the subsequent passes I followed another piece of advice from
Heim‟s book. This advice says that the evaluator should compare the interaction experienced
when using the interface to a list of predefined general heuristics, for this I am using the
heuristics proposed by Nielsen himself for this purpose[6]. Nielsen's heuristics are shown in
table 2.

   Table 2 – Nielsen's Heuristics [6]

As there was only one evaluator the „passes through‟ the system needed to be as thorough as
possible. In order to do this not only did care need to be taken to try and not miss any
contradictions to the heuristics, it was also vital that each heuristic was understood. The main
focus of the course that I am studying is Human Computer Interaction. At least two of the
modules have focussed on Nielsen's heuristics heavily with the coursework being to evaluate
systems using them. Therefore I do have some background knowledge of these heuristics
having studied them and used them previously.

The more focussed passes through the system were semi structured. I passed through the
system carrying out some tasks that a user may expect to be able to complete, for example
creating and moving tools. As this is a system developed for collaboration there are some
tasks that cannot be tested by one evaluator. These tasks are therefore covered in the user
workshops and analysed further in the evaluation of PowerMeeting using a set of groupware
heuristics. The tasks completed were typical tasks that a single user would expect to be able
to do using the interface that do not require collaboration. PowerMeeting allows for users to
use each aspect of the system without the need to collaborate. For example a user can create
an agenda item and use that agenda item without the need for a second user. Also a user is
able to select menus and options without the need for collaboration. This emphasises the need
for an evaluation of the system from an individual‟s prospective. Moreover PowerMeeting
has two different types of users, a normal or general user and the session chair. These users
have different access rights to some options and so the system needs to be looked at from the
view of both these users.

For each pass through the system tasks were carried and critiqued against one heuristic. As I
went through the system I would take down notes on where PowerMeeting did and did not
adhere to the heuristic, sometimes taking screenshots to illustrate where the problem lay.
Every so often I came across a problem that did not relate to the heuristic I was checking for
and so I jotted this down. Jotting these problems down enabled me to come back to them later
and not lose my focus.

Once I had completed each pass through the system with a specific heuristic I moved on to
looking at certain aspects that made up the interface such as the error messages and icons. My
evaluation of the error messages is shown in appendix A and the icons are discussed later.
These evaluations were considered when creating the prototype interface. The menus were
also evaluated. This evaluation is shown in the redesign section of this report.

The results from this heuristic evaluation can be seen in appendix B. From this evaluation 42
unique usability problems were found. The report will now discuss some of these problems in
more detail, outlining the problem and suggesting some possible solutions to these problems.
The solutions suggested use theories described in the literature that was discussed previously.

The results show that when checking PowerMeeting adheres to the heuristic „Match between
system and the real world‟ the most usability problems were identified (21%). This number
could be reduced if the recycle bin that is used within some agenda items in the system is

changed. Three usability problems identified related to the recycle bin with regards to this
heuristic. These problems mainly resulted from the functionality of the recycle bin not
matching the user‟s mental model.

The recycle bin allows users to drag and drop elements to it in order for them to be deleted. In
one way it does support the heuristic in that the icon itself looks like a recycle bin and so
matches the real world. However the actual functionality of the bin does not match that of a
real world recycle bin. The icon is a static image with one state. A similar icon is used with
operating systems such as Windows except this icon changes its state. For example the
recycle bin can be empty or have contents, figure 5 shows this. The first image shows the
recycle bin in the empty state, and so the bin looks empty. The second image shows the icon
after a file or folder has been dragged and dropped into it, now the bin looks like it has some
contents. This follows real world convention and would match the users mental model.
Furthermore in this example contents can be retrieved from the recycle bin, as it is a recycle
bin. This cannot be done with PowerMeeting even though the icon would suggest otherwise
which is another contradiction to the heuristic in question.

                           Figure 5- A recycle bin empty, then with contents [16]

Another way in which PowerMeeting does not adhere to this heuristic was found when using
the calender tool. Two buttons in the calender tool allow a user to navigate through the
months. However these buttons do not function as buttons would in the real world. In the real
world when a button is pushed it depresses, this depression of the button is a form of
feedback to the user. In many buttons used in software the buttons used follow this
convention, however the two identified in PowerMeeting do not (the buttons are shown in
figure 6).

                               Figure 6- Buttons featured in PowerMeeting's Calendar tool

A further simple change that would make PowerMeeting adhere to the heuristic more would
be to change some of the labels within the system. The label used on a pop up box that allows
users to change the date says „Datum‟. Obviously this should say „Date‟, this change would
be one less error in the system. Further ways in which PowerMeeting does not adhere to this
heuristic result from some of the icons present in the system being ambiguous.

19% of the usability problems found were from PowerMeeting not conforming to the
heuristic „Visibility of System Status‟. PowerMeeting allows there to be a session chair who
is the participant responsible for chairing the meeting that is taking place. They have some
extra controls that they can access such as the ability to re-order agenda items and to turn
meeting mode on or off. Therefore PowerMeeting works in two separate states, the session
chair state and the general participant state. However these states are not reflected in the
interface itself. Both the session chair and the general participant have exactly the same
interface with exactly the same options visible on the screen. The only difference is that the
options can be used by the session chair and not by the general participant. This should be
changed so that the general participant is shown an interface that represents the system state
for them, so options unavailable to them should not be shown. The problems identified here
could be overcome if the interface was made so it was strongly typed. The idea of making the
interface strongly typed was suggested by an academic member of staff at the University of
Manchester (Chris Harrison). This would mean defining different types of the interface which
could be displayed at different times depending on which interface is needed. One type of
interface could be displayed to the session chair. The session chair is the user who takes
charge of the meeting and has more options available to them, another type of interface that
does not have these options visible would then be displayed to users not logged in as the
session chair.

Other problems resulting from this heuristic are that there is no feedback to the user about
certain actions. For example there is no indication that an agenda item has been created by a

user. When a tool is created it is added to the bottom of the list in the agenda item panel. If
the list goes over the panel boundary then there is no feedback to the user that their tool has
been created as they cannot see that it has been added to the list. This problem could be
solved by either opening the agenda item on creation creating feedback to the user from this
or by adding the agenda item to the top of the list so that the new agenda item is visible to the

The „Consistency and Standards‟ heuristic identified 17% of the usability problems. These
problems tended to ones that could easily be fixed and may be considered minor. For
example one of the problems identified was that the calendar tool goes over the normal
workspace size. This does not adhere to the heuristic as the calendar tool is not consistent
with the rest of the system. Fitting into the normal workspace size would overcome this
problem. Another problem identified was that there is no add button available on the
brainstorming tool to add an element to the workspace, this button is available on other tools.
Again this could simply be overcome by the inclusion of an add button on this tool.

The main area of discussion with this heuristic is that there are multiple ways within
PowerMeeting‟s interface that elements in agenda items can be deleted. For example, in the
slide and brainstorming agenda items there is a recycle bin available to delete elements by
dragging and dropping them into the bin. However, in tools such as the task diagram,
elements are deleted by clicking the „x‟ in the top right of the element (shown in figure 7). To
adhere to the heuristic „Consistency and Standards‟ there should be one application wide way
of deleting objects. Although the x for deleting is an effective way, this is not possible in the
slide tool whose items are not in boxes but more „free‟. The recycle bin is quite obtrusive on
the tools, and can get in the way but would fit in with each of the tools. Therefore I propose a
trade-off whereby the recycle bin is used but in a different position to make it less obtrusive.

                            Figure 7- An element from PowerMeeting's Task diagram tool

14% of the usability problem were identified when using the „User Control and Freedom‟
heuristic. This heuristic states that sometimes users do things by mistake. When these
mistakes occur the system should allow for a clearly marked emergency exit to reverse the
effects of the mistake. One of the more prominent problems found by evaluating with this
heuristic was that the HTML tool doesn‟t allow for any user control or freedom. There are no

navigation buttons available on tool. This means that users are not able to navigate between
web pages and so if they click on a link that takes them to another page either by accident or
on purpose then there is no clear way of getting back to the page they were just on. To
overcome this issue standard browser navigation buttons should be added to the tool.

Another example of PowerMeeting not conforming to this heuristic is with the logout option.
A user may also select this option accidently. If this is the case they have no way of quickly
correcting the mistake, instead they are immediately logged out of the system so they would
have to log in again. Having an „Are you sure?‟ prompt appear when the logout option is
clicked would solve this issue.

Furthermore agenda items can be created with no name. These cannot then be selected for use
and are not easily deleted. Therefore there is no clearly marked emergency exit from this
error. Applying the „Error Prevention‟ heuristic would overcome this problem as using it
would mean that this error wouldn‟t occur (a prompt stopping the user from creating an
agenda item with no name would prevent this). Also even when an agenda item has been
created with a name the icon itself is not a selectable part of the element. The user must select
the title of the item to open it. To give some user freedom the icon should also open the tool
as some users may click on the icon rather than the text.

The error prevention heuristic also contributed to 12% of the usability problems found. A
point to mention here is that the recycle bin in PowerMeeting doesn‟t allow users to retrieve
previously deleted elements and therefore once an element has been deleted then there is no
way to get it back without creating it again. With the recycle bin being on the workspace it
may be possible for users to accidentally drag their element to the bin and delete it accidently.
This would probably cause a level of frustration to the user. A simple prompt would help
overcome this problem. When a user goes to delete an element then they could be informed
that it will be permanently deleted and ask if they are sure they want to commit to this action.
This should prevent any accidental deletions. Another solution would be to make the recycle
bin function as a recycle bin i.e. allow things to be retrieved from it. In this way if an element
was accidently deleted then users could simply enter the bin and retrieve it. This would be a
more complicated solution to implement but may be more effective in the long term.

A further error that could be prevented on PowerMeeting occurs when a user who is not a
session chair attempts to select an option that only the session chair has the authority to use.
This problem has been discussed previously. Options not available to users should not be

displayed. If certain functionality is not available then Heim suggests that the options that are
not available should be greyed out “functionality can be greyed out, which helps to reduce
user errors due to inappropriate commands out of the range of the current system state”[3].
However a more effective solution may be to implement the idea of the strongly typed
interface to overcome this problem as the type of interface displayed to the general user
would not have these options available on it. Greying out options should be used when those
options can at some point be used by that user. The options should be removed if there is
never any condition in which the user is able to use them, this will remove the error prone

Another 12% of the usability problems identified were from the heuristic „Help users
recognize, diagnose and recover from errors‟. This heuristic focussed on the error messages
of PowerMeeting. For this heuristic I found each error message present in the system and
copied them into a word document. I used guidelines suggested by Schneirdemann[9]
evaluate each error message and recorded the evaluation in the word document. This
evaluation of the error messages can be seen in appendix A.

10% of the usability problems were found when evaluating the system against the heuristic
„aesthetic and minimalist design‟. The main issue found here is the amount of white space on
the system which makes the interface look bare and somewhat uninteresting. This space
should be used more effectively. Another problem identified through this was the use of too
many icons used in the task manager tool. Figure 8 shows an element from the task diagram
tool. This figure shows the element expanded but when the element is actual size it can look
quite full due to there being four icons present on the element. When there are more of these
elements added to the interface then the user may become overwhelmed somewhat by the
number of icons visible to them e.g. adding 10 task diagram elements would mean that there
were 40 icons in the main section of the interface. An alternative way of allowing the
functions that the icons represent needs to be found in order to reduce the number of icons
needed on the task manager element. The „x‟ can be removed if the recycle bin is used to
delete elements. The rename icon can be removed if renaming is allowed by the same method
used in operating systems such as Windows where clicking on the name of an element allows
you to rename it. The linking icons could also be removed. The removal of these icons links
to the heuristic „Flexibility and Efficiency of Use‟. This heuristic found 7% of the overall
usability problems discovered with this method of evaluation. One of the problems found
with the heuristic was that the method of linking two elements within the task diagram tool

was overly complicated. Especially for novice users as the method used is not a natural way
of creating links. First of all it is hard to figure out exactly how to link the task, the icons
available don‟t give too much indication of what you need to do. There are tool tips available
to help with this task but those too are ambiguous and don‟t explicitly tell the user what
exactly needs to be done in order to link tasks. In order for this tool to adhere to this heuristic
it would be advisable to give a new way of linking tasks.

                           Figure 8- Element in PowerMeeting's Task Diagram tool with icons

The heuristic „recognition rather than recall‟ also found 7% of the problems. A problem of
note found with this heuristic that can easily be solved is that there is no way of mapping the
icons for created agenda items to the named agenda items in the create menu. The create
menu is shown in figure 9. As you can see it lists the tools that are available for users to
create. Once the tool has been created it is displayed in the agenda panel with an icon.
However the create menu itself contains no icons. An idea would be to add the icons that
represent the tools being created to the create menu to promote the recognition rather than
recall heuristic (improved menu shown in figure 10). This is because the user will be able to
then map the icon they have seen on the create menu with the icon representing the tool they
have just created. In addition when the user creates a new agenda item it may be an idea to
show that this agenda item is new by indicating this through the icon, maybe by adding a star
to the icon.

Figure 9- PowerMeeting's       Figure 10- ‘Create’ Menu with
current ‘Create’ menu          suggested improvements

Visibility of system status could also relate to a groupware principal. In groupware it is
necessary for participants to recognise who is online so that they know who they are working
with. PowerMeeting achieves this through the participant‟s panel, although this panel is not
currently functioning properly and if a participant doesn‟t log off properly then they are still
shown as offline. This list should be refreshed at intervals to keep it updated. It may also be
useful for participants to know which participants are working on which tools, if this was also
displayed in the participants panel then it would allow users to recognise who is doing what
in the group otherwise they would have to recall this information from the chat panel. This is
going to be discussed in more detail in the Groupware evaluation section.

The final heuristic to look at is „Help and Documentation‟. 5% of the problems were
identified from this heuristic. One problem would be solved if the tool tips were redefined to
be more helpful and less ambiguous. Also accessing the user guide isn‟t as easy as it should
be. Any help documentation should be easily accessible as the user will probably already be
frustrated by the fact that they need help. Clicking on help in PowerMeeting opens a pop up
box (figure 11). The main feature on this box is the OK button which a user may
automatically press without thinking, assuming that this is going to take them to the user
guide. With the OK button also being in line with the user guide link this assumption is
understandable. However pressing OK simply closes this box. Rather than this I would
suggest having a button that the user presses and the user guide is immediately presented to

                    Figure 11- PowerMeeting's pop up asking for confirmation of deletion

   3.2 User Centred Evaluation
The user evaluation was the second stage in the evaluation process. The decision was made to
do user workshops before the groupware evaluation as looking at how users collaborate
would aid the groupware evaluation and provide vital information for it. To decide upon what
the best way to approach the user evaluation was, some preliminary work was conducted.
This enabled me to see what methods would work best to evaluate the system. There are
some obvious drawbacks to groupware such as lag increasing as the number of users on the

system increases. Problems such as this were not what I wanted to investigate. The purpose
of this project was to see how the interface could be improved to improve both the level of
collaboration and usability with the system. For this reason it was important to evaluate the
system from both an individual and collaborative perspective.

Some good information is available in the books by Preece et al [15] and Schneidermann [9]
mentioned in the literature review. Nielsen also has some useful input into this area of work.
He suggests doing usability tests with no more than five participants as this will find the
majority of the usability issues with a system[17]. His graph shown in figure 12 illustrates
this. The graph shows that using fifteen participants should find 100% of the usability
problems; however he suggests using only five participants in order to avoid budget
restrictions. The budget was not a problem in this project and so more than five participants
can were used.

Figure 12- Graph showing that the majority of the usability issues are found using a rather small amount of participants

The graph is based on one user using a system at a time. Due to the nature of PowerMeeting
there will be at least two participants on the system in each workshop. At least one of these
users will need to be logged on as a session chair to enable the workshop to test all of
PowerMeeting's functionality. For this reason the plan was to conduct 15 workshops in order

to attempt to get 100% of the usability problems discovered. However due to some
participants not being able to attend the workshops only 10 were conducted.

The workshops conducted were done with two participants. In total ten workshops were
carried out in this way meaning that there were twenty participants. The decision to only have
two participants in the workshops at one time was made to make it easier to observe the
participants and it also reduced the problems associated with lag from too many participants
being active on the system at the same time. Two focus groups were set up after the user
workshops which consisted of a group of four participants and a group of six participants
who all used the system collaboratively then were allowed to discuss the system at the end of
their tasks. In doing each of these sessions it is hoped that not only problems experienced at
an individual level are found but also at a collaborative level. The focus groups allowed me to
get some further feedback about the main problems that the participants faced through the

An adaptation of the think aloud technique [15] was used to allow me to observe the
participants and drill for any problems that they may come across from interacting with the
system. The process of the think aloud technique is basically a user saying aloud there
thought process as they carry out a task on the system they are evaluating. "The technique
requires people to say out aloud everything that they are thinking and trying to do, so that
their thought processes are externalized[15]", this would allow me, the observer to know
exactly what the user is thinking and what problems, if any, the user is having. However there
are some limitations identified with this technique. Often users fall silent during this method
and 'forget' to speak as they will be concentrating too much on the task in hand, this is
particularly the case when users are trying to work out a situation that they deem difficult or
confusing, these are exactly the problems that the observer wants to know about. Preece et al
[15]suggests that a way of overcoming this problem is to have two users working together so
that they communicate with each other. This is more natural to the user who now wouldn‟t
feel as though they are talking to themself and will discuss the problems with the other user
and vice-versa in order to help each other along. It is also suggested that this is very effective
for evaluating systems that involve users working synchronously, which is exactly what
PowerMeeting is. Both Schneiderman[9] and Preece[15] suggest that two participants
working together produce more helpful responses using the think allowed technique as they
communicate. For example one participant may explain a procedure to the other. By its very
nature PowerMeeting wants to allow for communication of its users. There are two ways that

PowerMeeting allows communication, one is via the chat tool present at all times on the
interface and the other is via Skype which needs to be set up externally from PowerMeeting.
Other methods that could be used would be to record the workshops and create transcripts
that could be analysed and compared. This method is good but would have been very time
consuming and involves going through a lot of irrelevant data. Observation allows for
immediate analysis as well as reflection on the workshop.

Before conducting the workshops an initial workshop was done to see how exactly they
would work and what improvements could be made to my current ideas for the workshops.
For the initial workshop I used myself as a participant logged into PowerMeeting as the
session chair and had another participant logged in as a normal user. The setup was very
informal which worked well however in the real workshops I could not act as session chair,
as this part of the system also needs to be evaluated by the users. The user discussed their
actions with me throughout the workshop. This gave me lots of feedback on the system.

For the workshops I created a consent form. The creation of the consent form followed the
suggestions from Schneiderman[9] this form is shown in appendix C. A scenario was created
along with a series of tasks that needed to be completed by the users. This sheet (appendix D)
was only given to the session chair who had to use PowerMeeting to communicate it to the
other participant. A similar scenario was used in the focus groups. The tasks were developed
to make the participants use each aspect of PowerMeeting so that an accurate evaluation
could be made. This scenario was created so that the participants had a goal to work towards.
If there was no context given for them to work in then the participants may not use the system
properly and so the results from the workshops would be compromised. After the workshops
a short questionnaire was given to the participants to help gain further feedback. This is also
shown in appendix D.

The users of PowerMeeting are likely to be IT literate. Therefore it makes sense to test the
system using IT literate participants. A short description was given to the participants before
the workshops and focus groups so that they knew exactly what PowerMeeting was and what
the purpose of the evaluation was. The participants were mainly made up of computer science
and business school students. This meant they were IT literate.

The workshops with two users were set up so that the users could not see each other‟s
screens. Each participant had their own PC or laptop with PowerMeeting and Skype loaded
and a headset connected to allow for verbal communication. In some workshops I requested

that verbal communication was not used to see how users coped with using the text chat
function. The focus groups were set up to allow for discussion. As there were many users it
would have been more difficult for me to observe them had they been separated around a
room. Instead they set up around a table (most of the participants had laptops) and were
allowed to communicate with each other by talking. This meant Skype was not used which
meant that there was no evaluation of how using Skype would affect the participants.

The results from the workshops and the focus groups are shown in appendix E. I recorded the
results in an excel spreadsheet in much the same way as the results gathered from the
heuristic evaluation. I have recorded the issue and the number of times the issue had been
identified along with a note if the issue was brought up in the focus groups. In total there
were 51 problems identified with this method of evaluation (from both the workshops and the
focus groups). Interestingly some of the problems identified in the heuristic evaluation were
not brought up in the user workshops. One drawback associated with heuristic evaluations is
that some „false positives‟ may be found that are usability problems that are only identified
because the evaluator is looking for problems, whereas in an environment such as a user
workshop the participants are using the system without this heightened sense of awareness
and so some of the minor problems may go undiscovered.[7]

Of the 30 participants 27 of them identified that they did not know what four of the icons
meant. These icons included the pin card board icon, the brainstorming icon and both the user
icons (session chair and general user icons). The three who did not identify this problem
informed me that they had used PowerMeeting recently and remembered what the icons had
meant. It was clear from this that some of the icons needed to be changed.

22 participants complained that there was no indication that the HTML tool was loading the
page they had navigated to. Also 15 users identified that having to type in http://www into the
HTML tool to go to a page was not user friendly and some did not manage to figure out that
this had to be done without guidance. A further problem identified with the HTML tool was
that it was not coupled. So when a user successfully navigated to a page using the tool, others
would not see this and would still be on the home page of the tool. Also 15 users complained
that there was no navigation buttons available in the HTML tool, this was a problem that was
also identified in the heuristic evaluation.

10 participants identified that the telepointer showed up on their tool even if the person using
the telepointer was on a different tool. This clearly should not be the case as the participant

would be pointing at something on their agenda item. This may have caused some confusion
among the other users. This problem was only identified within the focus groups as there was
no need for the users working in twos to use the telepointer if the participants were on
different tools.

14 participants found the method of linking within the task manager tool to be problematic.
This was discussed at length in the focus groups with many participants calling this method
poor. A new method of linking these elements should be used.

Limitations of this method were that although there were 30 participants not all of them were
able to evaluate every part of the system. For example in the focus group the participants
were split up to complete tasks, so only two participants from each focus group may have
used the html tool. If every participant used every aspect of the system then there may have
been more problems identified and the results may have been more significant.

    3.3 Groupware Evaluation

This evaluation method followed much of the same principles as the usability heuristic
evaluation method discussed previously. In order to understand the heuristics as they were
defined three papers were analysed. These papers were published by the individuals
responsible for the creation of these heuristics. Table 3 shows the heuristics proposed for
groupware applications. This evaluation focused more on the collaborative side of the system
rather than its individual usability. The heuristics are designed to look for groupware-specific
usability problems[11]. The user workshops and focus groups already gave some insight into
the problems that PowerMeeting has in aiding collaboration between its users. This next
evaluation aims to look deeper into what could make PowerMeeting better as a piece of
groupware by evaluating key aspects that make groupware effective.

Table 3- Heuristics proposed for use on groupware systems [11]

To carry out this heuristic evaluation the same steps that were followed when using Nielsen's
heuristics were followed here. Before starting the evaluation I needed to familiarise myself
with this new set of heuristics. The papers published by Baker et al [10][11] talk about these
heuristics in good detail giving examples of how each heuristic could be achieved. This
helped me understand the heuristics more. Once I had become familiar with the heuristics I
moved onto evaluating the system, starting with a general pass through the interface. Later
passes would again be focussed on evaluating the system against a specific heuristic.

In total I discovered 24 problems using this set of heuristics. This is nearly half the number of
problems that were found using Nielsen's heuristics. However as the two evaluations are
looking for different types of usability problems they cannot be compared as different
problems were found with each. However my lack of familiarity with this new set of
heuristics may also have had an effect on the number of problems found. Some of these
problems will now be discussed. The results themselves can be seen in appendix F and follow
the same layout as the previous heuristic results. Later I shall discuss the suggested redesign
and why certain aspects of it should be implemented. The redesign depended heavily on the
work done when researching these heuristics.

The heuristic „Provide the means for intentional and appropriate verbal communication‟
identifies three groupware usability problems within PowerMeeting. The paper suggests three
forms of this communication. Summarised these forms are explicit verbal communication,
overhearing verbal communication and out loud or running commentary communication.

Baker et al[11] suggests that textual chat can help to achieve this heuristic and while it‟s
limited it can be very useful, “text chat can be useful for short or sporadic interactions, or
where it is impractical to provide an audio connection.[11]” Also suggested is that an audio
channel of communication is required for lengthy or highly interactive meetings and so it
should be available.

Currently PowerMeeting does support chat functionality through the chat panel available to
the right of the interface. However this chat panel is quite small and after a few lines are
typed into it the user must scroll up to see what else has been said. Furthermore this chat is
not logged or saved so when a new participant enters the session or when an existing member
logs out and back in the chat that has occurred is not available to them. These two problems
may reduce the chance of participants „overhearing‟ the communication. By this I mean that
the users who have just entered the session will not be able to see the discussion that has been
going on and so will not know exactly what has been said and what the on-going discussion
is about. A possible solution here would be to log the chat and save it, displaying the history
of the chat in the chat panel. This way new participants can scroll up and see what has been
discussed. This may also prevent unnecessary communication taking place as new users will
not need to ask what has happened in the session so far as they can check the discussion.
Another idea would be to make the chat area larger. This would mean that more of the history
of the conversation can be displayed to the users of the system which would give the current
conversation more of a context and help to establish more common ground[12] between the

Also PowerMeeting allows for audio communication. However this is done through an
external piece of software, Skype. This means that the users now have to control two systems
and may have to switch back and forth between windows which could cause user frustration.
An audio link should be made available within PowerMeeting's interface so that it can be
controlled by users without the need to minimize the window and take their attention away
from the group activity. It should be noted however that the audio link should not be the sole
form of communication as there are problems related to audio links highlighted by Baker et

al[10] “(audio links) currently suffer problems due to poor bandwidth, latency and quality”
such problems may be overcome by high speed broadband but it also must be considered that
maybe not all the users have a microphone and so may be forced to use the chat panel.
Therefore audio chat should be optional. Not including this internal audio connection was
highlighted when evaluating the system against three heuristics. The other two were „Provide
consequential communication of an individual‟s embodiment‟ and „Support people with the
coordination of their actions‟. The inclusion of the audio channel will increase the media
richness of the system[13].

Provide the means for intentional and appropriate gestural communication identified two
further problems currently existing with PowerMeeting. They both stem from PowerMeeting
not being able to display intentional gestures that users may want to make. PowerMeeting
does support this heuristic to some extent by allowing a single telepointer. This telepointer
can be used by a participant as a pointing gesture to point at an element and draw attention to
a certain element. However this cannot be used by more than one user at the same time and
so has limited use. To help support this heuristic more each participants cursor should be
made visible at all times to participants within the same tool. This will allow people to point
at elements without the need to wait to use the single telepointer. This would mean that the
telepointer can be deleted from the system.

Not all gestures can be modelled in groupware but another form of gesture taking place in a
face to face meeting may be for one participant to want another attention to come and look at
something that they are working on, as they made need help with it. This would currently
need to be achieved with the group chat function in PowerMeeting and the request may go
unnoticed. If a function allowing a participant to select another from the participant list and
invite them to their tool was available, then this would be the equivalent of the “come here”
gesture. The recipient of the invite could then either accept the invite and go to the other
agenda item to help out or decline the invite as they are too busy. These solutions link
directly with the next heuristic „Provide consequential communication of an individual‟s
embodiment‟. One problem resulting from using this heuristic was that no unintentional
gestures are picked up and displayed in PowerMeeting at all. This is hard to achieve fully
without using external hardware or videos. Videos would be good at establishing more
common ground[12] between the participants. The media richness theory[13][14] would
suggest that video would be a good media to use for computer mediated communication

(CMC) [13] as it would provide more feedback to the users in terms of emotion. Also more
common ground could be gained from the use of videos so that users can see one another and
their surroundings giving more social presence [13]. However videos are unfeasible in this
case as, if each user has a video of them, then the screen can become quite full. This may
overload the users working memory due to lots of different media being shown to them at one

The audio link would help to achieve this heuristic to a certain extent as unintentional
utterances may be picked up by the users. Moreover the telepointers discussed give the users
some idea of where each other is. The cursor position for each user could relate to the users
gaze and therefore indicate where the user is looking though this is not always the case it
does give some way of knowing what another user is doing. These cursors would need to be
colour coded to match each participant; this would also mean the participant icon in the
participant panel would also need to be colour coded so that the relevant icon can be mapped
to the relevant participant. This would also give PowerMeeting more media richness[13][14]
as the audio will show some of the users‟ emotions which could put into context what they
are saying more than plain text chat could.

The heuristic “Provide consequential communication of shared artefacts (i.e. artefact feed
through)” found 7 problems. The problems identified should also benefit from the new
telepointers. The main problems identified with this heuristic are the problem of users not
knowing what other users are doing. They are blinded during some interactions. For example
the heuristic gives the concept of action feed through. This “involves information
unintentionally given off by physical artefacts as they are manipulated” [10]. In
PowerMeeting this concept is not apparent. For example when a user is dragging an element
towards the bin to delete it other users watching do not get any feed through. They simply see
that the element is there one second and gone the next. There is also no way of knowing who
has deleted the element. A solution to this would be to have the colour coded telepointers,
this would let the users know who is interacting with the element and also show the users the
intermediate stages of the interaction i.e. show it moving towards the recycle bin. This would
give the users watching more context as to what is happening and allow them to stop the
action if necessary by talking to the user via either the chat or the audio link. This relates
back to unintentional gestures as moving the element towards the bin informs the other users
that it is your intention to delete the element.

Also users are unaware of when meeting mode is being activated by the session chair and are
immediately taken away from what they are doing when meeting mode is activated. Again it
is this notion of not knowing that is causing problems. Giving a delay and informing the users
when meeting mode is activated may help this issue as the users can then plan for their return
rather than have the shock of immediately being taken away from their agenda item. This was
something that was found in the user workshops. In a real life meeting if someone wants to
address the group then you are not normally dragged away from what you are doing in an
instant, you normally finish what you are doing so you can come back to it and understand it
before giving your attention to the group leader.” This was an opinion of a participant in a
user workshop. They make a valid point in that there is normally a short delay in the request
for your attention and you giving the chair your attention. Another way to increase
knowledge of what is occurring in PowerMeeting would be to include some labels next to
each participant that informs everyone what they are working on so people are able to see at a
glance who is doing what.

The heuristic „provide protection‟ found 4 problems. In groupware participants may
concurrently access the same elements at the same time, this needs to be controlled somehow
through turn taking. Social protocol may help in reducing this kind of conflict and knowing
where other participants are should also help though sometimes social protocol is not enough.
Also some agenda items may be created and be finished with. The creator of this item may
not want any changes to be made to the item so some form of locking should be enforced.

A problem that was noted in the user workshops was that PowerMeeting allows participants
to concurrently access elements within agenda items. More than one user may select to make
different changes to an element simultaneously but only the changes that are completed last
will be saved. PowerMeeting should make it more explicit to participants that an element is in
use. This could be done by giving the element in use a border in the colour of the participant
that is using it. This would show who is using that element. The element in use should also be
locked so that it cannot be manipulated at the same time that another participant is editing it.
This will avoid any conflicts and in effect introduce turn taking to the interface. An error
message will need to appear to inform users who attempt to use the element that is already in
use that this is the case though as its state is already made explicit then this error may be
prevented most of the time. This again links back to Nielsen‟s Heuristics in that we are
making the status of the elements visible to the participants and preventing errors. In the user

workshops one complaint was that people were able to move things that someone else may
have already been moving or editing. In one case one participant was renaming an element in
the task diagram tool while another deleted the element which proved frustrating for the user
renaming it. Decreasing user frustration is a good thing and so implementing the solution
discussed would be a good idea.

With regards to the heuristic „Manage the transition between tightly and loosely-coupled
collaboration‟ no real problems with PowerMeeting not adhering to this heuristic were found.
PowerMeeting is good at supporting this heuristic. When meeting mode is turned off
participants have the freedom to work together in groups or work individually by either
opening an agenda item that has active participants working on it or by opening an agenda
item where no one else is working. There is no explicit control for participants to turn off this
coupling action, they either work in an agenda item with others or don‟t. This is a good
system as it means that all the work carried out is in real time. If participants were allowed to
turn off the coupling and start working individually on an agenda item that other people were
already working on then this would cause conflict and changes that other participants could
not see. Therefore this should not be allowed. The only time PowerMeeting conflicts with
this heuristic is when meeting mode is on as participants are not allowed to shift away from
this mode. However I don‟t see this as negative as this mode is only used for the session chair
to address the group and its purpose is to bring all the participants together and not allow
them to work on their own while the session chair gives out information or shows the group

This heuristic also talks about the need for participants to have awareness of each other. The
simplest way to do this has been discussed and is to show what agenda item people are
working on in the agenda panel and what they are doing by a label in the participant panel.
The paper suggests other methods of showing other active agenda items on the interface or
giving an overview of each agenda item. This solution would be poor in the case of
PowerMeeting as this would cause a clash of media and may overload the participants
working memory. This lack of awareness links to the next heuristic „Support people with the
coordination of their actions‟ awareness could be given with the telepointers and the labels.
Also it is important to give the participants awareness of what the goal of the meeting is so
they are able to coordinate their actions according to this goal. Currently PowerMeeting does
not do this and the goal may only be mentioned in the chat window or verbally through

Skype and so can be missed by some participants, especially those who may enter the session
at a later time or date. Having an area on the interface that explicitly tells the group the goal
of the session or meeting will help collaboration and allow people to coordinate their actions
towards this goal. Furthermore it may be an idea to add a similar thing within agenda items
that tells the participants within that item the goals of that item. This would give each agenda
item a specific purpose which may be defined on creation. In turn this would allow
participants within the agenda items to coordinate their actions with regards to the group goal
that should be met within that item.

The final heuristic to discuss is “Facilitate finding collaborators and establishing contact”.
The problems identified here are mainly due to the participant list not being updated regularly
and so it is not know exactly who is online at what times in PowerMeeting. Furthermore there
is no information on the participants and so no contact can be made to participants who are
not actively online. If there was some kind of business card feature available in the system
then contact could be made to the participants at all times.

    4. Redesign
After each evaluation some redesign ideas were jotted down and sketches were drawn up.
Some ideas for the redesign came from the evaluations of PowerMeeting whereas others
came from other design principles and good practice. Redesigning the interface was a process
that ran throughout the project. Ideas would immerge during each evaluation process and so I
would jot down these ideas or create mock-ups to see how the ideas would look. The final
step in the project was to create a prototype that could illustrate these improvements. Due to
my limited knowledge of programming a fully functioning prototype was not created. Instead
presented are screens of the suggested redesign that can be navigated through to show
different functionality. An HTML image map was used for this purpose. An example of the
code used to create these pages is shown below:

<html xmlns="">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" />
<title>Untitled 1</title>
<map name="interface">
<area href="createmenu.html" alt="create menu" title="create menu" shape="rect" coords="5, 3, 56, 20" />
<area href="toolchat.html" alt="maximize tool chat" title="maximize tool chat" shape="rect" coords="1243,
582, 1256, 594" />
<area href="stanBCwotc.html" alt="Business card" title="Business card" shape="rect" coords="0, 436, 244,
457" />
<area href="voiceonwotc.html" alt="Group voice" title="Group voice" shape="rect" coords="1225, 33, 1241,
46" />
<area href="toolchatvoiceon.html" alt="Tool Voice closed" title="Tool Voice closed" shape="rect"
coords="1212, 586, 1230, 602" />
<!-- Agenda selection -->
<area href="brainstorm.html" coords="1, 50, 232, 71, 71" shape="rect"/>
<area href="slide.html" coords="0, 67, 231, 88, 71" shape="rect"/>
<area href="taskdiagramlockwtc.html" alt="create menu" title="create menu" shape="rect" coords="0, 87, 232,
108" />
<area href="html.html" coords="0, 107, 231, 128, 71" shape="rect"/>
<area href="cal.html" coords="0, 122, 231, 143, 71" shape="rect"/>
<area href="pincard.html" coords="0, 142, 231, 163, 71" shape="rect"/>
<!-- click session chair to see their view -->
<area href="session chair.html" alt="Tool Voice closed" title="Tool Voice closed" shape="rect" coords="0,
474, 76, 490" />
<img src= "images/version 2.gif" alt="interface" border="0" usemap="#interface" />

The images were created using GIMP, an open source photo shopping application. Some of
the code above was generated using Microsoft Expression Web 3.

This section of the report is going to run through some screenshots of the prototype of
PowerMeeting created by improving the design with regards to the problems identified
throughout the report. Screen one shows the general interface running in a web browser.

               Screen 1- The suggested redesing of PowerMeeting in a standard browser window.

From this screen various improvements can be seen. First off the interface itself has been
centred within the browser window. This makes it look more presentable as there is less
white space. There is still some space beneath the interface. This is because this particular
screen shot was taken on a large screen. On a smaller screen this interface would fill most of
the screen. The systems should conform to the widely used resolution of 1024 x 768
pixels[18] The next improvement to note is the change to the chat panel. I have suggested
that the chat panel is moved to the right hand side, this allows the chat panel to increase in
size and also means that the agenda item panel can be made bigger. This makes more created
agenda items available at one time which could mean there is less searching needed by users
to find there desired agenda item. Furthermore the design shows two types of chat, the group
chat and a tool chat. The group chat is the same chat function that was available before that
allows every participant to communicate via this panel. The tool chat is an optional panel that
can be opened by the participants if they choose. It was found that in larger groups the group
chat panel became full rather quickly meaning that users had to constantly scroll back

through the panel to keep up with the conversation. Having more of the conversation visible
may reduce the need for this. The tool chat has been added to allow users in big groups to use
if the work has been divided between the group. For example there may be a group of ten
using PowerMeeting. This group may be divided so that two participants create a task
manager tool, two create a slide to present to the rest of the group and so on. There will be
some messages that only need to be passed between the smaller groups as the message is only
relevant to their task. This can be sent through the tool chat. The redesign also suggests the
internal audio link and shows possible adaptions that may be added to the interface in order to
incorporate it into the design. Screen 2 shows the tool voice chat enabled and screen three the
group voice chat enabled. Again the tool voice chat is optional as in smaller groups it will be
more feasible to talk amongst the whole group, this is what is suggested with the groupware
heuristics in the heuristic „Provide the means for intentional and appropriate verbal
communication‟. However, again this may not always be a feasible option as in larger groups
a lot of participants may want to communicate at the same time. If they have broken off to
form smaller groups to work on different tools then this crowded audio chat may become
frustrating and so the tool voice chat is available for people who are within that tool to talk.

The screen also shows that the recycle bin has relocated to an area where users may be less
likely to accidently delete elements. Having it in this location creates a task bar like area at
the bottom of the agenda items. The icon in this design represents a trash can which implies
that icons cannot be retrieved once deleted. Of course a recycling function that allows
retrieval of items would be much better and should be incorporated as explained in the results
of the heuristic evaluation using Nielsen's heuristics.

                              Screen 2- Redesign showin Tool chat function

                                 Screen 3- Redesign showing voice chat

These tool chat and voice tool chat options allow the agenda items to behave more like
rooms. It is important for participants to know who is in these „rooms‟ so they know exactly
what each user is doing. The labels that can be seen in the agenda panel (top left of the
interface) next to the agenda items name, displays the names of the participants currently
working on that tool. This allows people to see at a glance who are where. Users can then
tailor their actions according to this. Having these labels allows gives some more knowledge
to each participant. One of the main things that are needed to create good group dynamics is
some common ground[12] within the group. Common ground is established in this design by

making more visible to each user. The increase in the group chat panel makes more
conversation visible to the users. The labels showing where each participant is gives the
group knowledge of what each person is working on and where they are. Also the addition of
the panel at the bottom of the interface increases common ground within the group as its
purpose is to allow the session chair to define the group‟s goal. This is constantly displayed
to the group to remind them of what they have to work towards. This panel can be adapted
further to be included within each tool so that participants know exactly what each tool is for.

Logging is another idea that could be incorporated into PowerMeeting to increase common
ground within the group. Currently the chat is not logged and so if a participant logs in after
the meeting has started they cannot catch up on any conversation that may have occurred.
Furthermore logging the chat allows the group to come back to the session at a later date and
remind themselves of what has been discussed in the previous session. Of course logging is
not feasible for this purpose if only audio chat has been used. Much more useful information
could be logged in PowerMeeting. Baker et al[10][11] suggest that in a physical environment
the artefacts tend to display information on how they have been created and there history. In a
virtual environment much of this history is lost and finding out who has created or edited a
certain element or agenda item is not possible in PowerMeeting unless the participant
responsible for the creation or edit remembers it was them and states to the rest of the group.
Of course if the edit was a negative edit that the group would rather not have happened then
this participant may well not own up and the history is not known. Problems like this could
be resolved by logging actions. If PowerMeeting logged actions that take place within the
agenda items then this would create a timeline for that agenda item. The creation of the
agenda item would start the log along with the name of the creator and the time. Further
details logged would be elements added to the agenda item and edits made. This could be
made accessible to the session chair. An example of this is shown in figure 13. This idea
relates to the heuristic „Provide consequential communication of shared artefacts‟. Also this
is an idea suggested by Wang himself, discussed in the literature review to keep track on the
progress of the collaboration[5].
                 Figure 13- Example of a log that could be used in PowerMeetings tools

An improvement with regards to Nielsen's heuristics is that now the interface explicitly says
which state it is with regards to the meeting mode being off or on. This can be seen by the
label in the top centre of the interface. This improves the system by giving more visibility of
the system status. This label should also be made to change not only to on or off but also to
inform users that meeting mode is activated. As discussed, in the user workshops one of the
things that came out was that there should be some form of delay to warn users that they are
going to be taken away from the system. This label can warn users that this is going to
happen. The screen below (screen 4) shows an example of how the label could look when the
session chair selects meeting mode. Screen 5 then shows how the label would change once
the delay has passed.

       Screen 4-Redesign showing PowerMeeting informing users that they are going to enter meeting mode

                    Screen 5- Redesign from general users view when meeting mode is on

Once in meeting mode the navigation of general users becomes limited as they are not able to
leave the agenda item that the session chair has called them to. Above you can see that the
agenda items have been greyed out this is to stop users trying to navigate to another agenda
item while in meeting mode as this option is not available in this state[3].

Screen 5 also shows more prominently the telepointers. The addition of these is explained in
the results section of the groupware heuristic evaluation. These telepointers should help the
participants communicate their location within the interface, their focus of attention and
movement [20]. The telepointers in this design are colour coded to match the colour of the
icon given to each participant. This colour coding will help make it easier to identify which
user is in control of which telepointer. The telepointers in this design are cross hairs rather
than cursors. The reason for this is that users may get confused of which cursor they are
controlling if the telepointers were the same shape as there standard cursor.

Further improvements that can be seen in the screens are the reduction in the amount of
menus and the change to the icons. The reason for the change with the icons is that in the user
workshops it was found that many of the icons were ambiguous and many of the participants
did not know what they meant. There were some where there was no problem identifying
their meaning, however others were quite obscure and did not have any meaning that linked
them to their tool. Icons are very important in a system, they have become more and more
common in GUI‟s and are relied on. They are important aspects of a GUI that need to be

designed correctly in order to get the best use out of them. There are different views of icons
and different ways that people interact with icons. [3]

Icons are seen as metaphorical representations of their real-world counterparts (the object that
they are representing). It is important for the design of an icon to take into account the
physical representation of an object that the icon is being designed for[3]. The recycle bin is
present in most operating systems and also in PowerMeeting. It is represented in a similar
way in most GUI‟s. This is because the icon of the recycle bin is representing a real world
object and therefore is designed to look like this object.

Heim goes on to suggest that research has shown that image is a more effective
communication tool than text as a person‟s ability to recall images is greater than that of text
and so if used correctly icons are a very effective way to aid the use of a system.

Some of the icons have been redesigned due to the theory suggested by Heim [3] and some
were changed to fit in with the new design by being colour coded. The new icons can be seen
below alongside the old icons in figure 14. As can be seen from the figure, some of the icons
have remained unchanged. This is because some, e.g. the calendar icon was good and
represented the tool well. Others, such as the brainstorming icon did not represent its tool at
all. The new icon represents this tool more as it depicts a brain inside a bulb. Bulbs are
generally associated with ideas which are what brainstorming generates.

       Figure 14- Current PowerMeeting Icons and suggested PowerMeeting Icons

The reason for the reduction in the amount of menus is that some of the menus were not
needed. There were five titles in the menu bar. Only four of these were menus the other was a

button titled „Logout‟, this closed PowerMeeting. As this was not a menu the decision was
made to remove it from the redesign. Logging out can be done by simply closing the browser
window, or if preferred the log out option could be moved next to the „?‟ icon in the top right
of the interface. The „?‟ icon is another change to the menu structure. Figure 15 shows the
menus that are present in PowerMeeting. The last menu shown is the help menu. This only
has one option that can be selected which is the User Guide option. Selecting this option
takes you to the user guide. Heim[3] suggests that a menu should have at least two options
within it. If this is not the case he suggests that the options should be combined within
another menu. There is no menu in this case that the option could fit in to. Instead the „?‟ icon
has been added to replace this menu. The pointer menu has remained to give the users the
choice to turn off the telepointers that have also been added to the design.

The edit menu has been removed from the design as many of its functions could be made
easier and they are only accessible to the session chair. If the option to rename agenda items
is taken away from the general users then the edit menu has no functionality in the interface
for the general user. The interface for the session chair would still need this functionality.
However as the functions available in this menu allow the session chair to rename, remove or
move the order of the agenda items. If the system was redesigned so that these functions were
made easier then the Edit menu could be removed from the interface. They could be made
easier by enabling the agenda items in the agenda panel to be moved round via drag and drop,
they could be deleted by selecting an „X‟ in the corresponding tool, and also they could be
renamed by simply clicking on the tool name, like in Windows interfaces. Of course these
functions would only be available to the session chair but as they would not be visible to the
other users they would not cause any problems to them. Also making these functions easier
would make PowerMeeting adhere to Nielsen's heuristic „Flexibility and efficiency of use‟ as
dragging and dropping an agenda item to the position required is much more efficient than
accessing the edit menu to move the item one place at a time. The create menu remains in the
redesign. This menu is needed to create agenda items and not much can be changed on it.
One change that has been made to this menu is the inclusion of the icons as discussed earlier
in the report.

Figure 15- Menus in PowerMeeting's current design

                                                                Help Menu
                                         Pointer Menu

                       Edit Menu
   Create Menu

Figure 16 shows an image of the business card feature that I suggest is incorporated into
PowerMeeting. Selecting a user from the participant panel will open a card like this. From
this card users will be able to see contact details of the other users should they need to get in
contact with them if they are not available online. The Skype icon could be a link that opens
up Skype in order to phone the user. The email when clicked would open up the email client.
Also on the card is a small area called Notes. Here users can input any information they deem
necessary. In the example the user informs the others of the times they are available. The
reason for the addition of this feature is to allow the users to easily establish contact with
each other. This is a point identified in the groupware heuristic „Facilitate finding
collaborators and establishing contact‟[10]. This figure also shows the invite button which
was discussed in the groupware evaluation. A user could select someone in the participant
panel and then click the invite button. This would send I message to the corresponding
participant inviting them to go to the tool where the sender of the invite is.

                             Figure 16- Example of a ‘business card’ that could be used

Other changes to the design lie in the agenda items themselves. The following screens show
some possible improvements that could be made to the task manager tool. In both the
heuristic evaluations and the user workshops it was identified that the method used to link

two elements in this tool was over complicated and not very user friendly. An approach
similar to that used in Smart Draw could be used. Here there is a drawing tool that allows the
user to draw a link from one box to the target box. Implementing a similar method in
PowerMeeting would improve the method of linking tasks greatly. Screen 6 shows an
example of this. The user would select the drawing icon in the bottom right then link the
desired boxes. The cursor would change to a pencil to let the users know they are in the
drawing mode.

                Screen 6- Redesign showing new approch to linking elements in task diagram tool

Screen 7 shows further improvements relating to the task diagramming tool. Here an element
is being displayed to a user as being locked. This shows them that another participant is
currently using this tool and so access to it is not allowed. Screen 8 shows the error message
that would appear if a user attempted to access a locked element. This will stop simultaneous
access to the same element by more than one user, resulting in user not being able to change
elements names at the same time.

           Screen 7- Locking elements in task diagram tool

Screen 8- Error message when a users attempts to use a locked element

                                  Screen 9- Redesign of HTML tool

Screen 9 is an example of how the html tool may be improved. The addition of a navigation
bar will allow users to navigate more easily through the tool and make it more like a standard
web browser. The navigation pane on this image is taken from Google Chrome[21].

   5. Conclusion
To conclude, the evaluation methods used in this project found a good number of usability
problems with PowerMeeting from both perspectives identified. Two of the main clusters of
problems could be solved by changing the method of deletion in PowerMeeting by given an
application wide deletion method of using the recycle bin and improving the recycle bin as
described. The other problem was with the general interface of PowerMeeting. There are two
types of users who have access to different functions within PowerMeeting but there is only
one type of interface. This results in users trying to access options that they do not have
access to. Both problems identified here could be resolved by giving types to the interface.
Giving the recycle bin types would make it behave as a recycle bin should. It would have the
type „Empty‟ which could display an icon of the bin without contents. The bin would also
have the type „Has Contents‟ which would change the icon to look like a bin with contents
and also the behaviour of the element. With the recycle bin in this state users should be able
to access the recycle bin and either empty its contents or retrieve some of its contents.
Equally the overall interface should be typed. What I mean by this is that there should be two
types of interface that are defined by the type of user. There should be a session chair type
interface and a general user type interface. The general user type interface will not display
options that are not available in that type, the session chair type interface will contain these
options as they can only be used by the session chair. Having these types means that the
interface would have less error prone conditions.

This projects main aim was to give a good formative evaluation of PowerMeeting and
identify areas for improvement. This has been achieved and it is hoped that the problems
identified and the possible solutions that have been suggested can be used to improve
PowerMeeting. More could be done to move this project forward. The prototype that has
been suggested should now be evaluated itself to see if it adheres to the heuristics used in this
report. Also users should test out this prototype. To do this the prototype would need to be
given functionality in order to be tested as the current build of PowerMeeting was.

An improvement that could be made to the user workshops is to ensure that all participants
evaluate every piece of functionality in the system. In the focus groups the participants where
split up to work on different tools at different times and therefore the results could not show
what 100% of the users thought about a certain part in the system.

A limitation with this project was that the heuristic evaluation was carried out with only one
evaluator due to the reasons explained. It is advisable for heuristic evaluations to be carried
out with more than one evaluator. In fact Nielsen suggests using 3-5 evaluators[19]. However
due to the user centred evaluations also being carried out it is hoped that problems that may
have been missed in the heuristic evaluations may have been picked up in these sessions.

The redesign that has been suggested solves most of the problems that were identified with
PowerMeeting. Improving it further would result in adding more media to the system to
increase its media richness[13]. To do this the entire interface may need to be redesigned to
allow for multiple Windows displaying videos to fit on the interface. Although the addition of
video would increase media richness, it may also result in losing some workspace and be
distracting. A improvement that could be added to the current design is the addition of user
images. Images could be added to the interface to increase media richness and social
presence[13] to an extent. Having a profile image of participants may give the other
participants a mental model of who they are collaborating with which could result in common
ground being better established in the group.

Overall I feel that the suggested improvements could greatly help PowerMeeting become a
more effective piece of groupware if implemented. Of course more can always be done and
so if these improvements do get implemented then they should themselves be evaluated and
improved upon in the future.

6. References
1., [PowerMeeting] [accessed on 5/02/10].
2. Wang, W. PowerMeeting: GWT-Based Synchronous Groupware. 2008
3. Heim, S. The Resonant Interface, HCI Foundations for Interaction Design, Addison
   Wesley, 2007
4., [Groupware by Ifeoluwa
   Idowu] [accessed on 12/04/10]
5. Wang, W, French, S., A Multi-Dimensional Framework for Facilitating Wide Participation and
   Common Understanding, 2008
6. [Ten Usability Heuristics by
   Nielson, J.] [accessed on 22/12/09]
7. [Heuristic Evaluation and its
   alternatives] [accessed on 13/04/09]
8. [iPhone App Usability Heuristics] [accessed on
9. Schneiderman, B. Designing the User Interface, Strategies for Effective Human-
   Computer Interaction, Addison Wesley, 1998. 3rd edition.
10. Baker, K., Greenberg, S. and Gutwin, C. Heuristic Evaluation of Groupware Based on the
    Mechanics of Collaboration. 2001. Proceedings of the 8th IFIP Working Conference on
    Engineering for Human-Computer Interaction (EHCI'01). (May 11-13, Toronto, Canada)
11. Baker, K., Greenberg, S. and Gutwin, C. Empirical Development of a Heuristic Evaluation
    Methodology for Shared Workspace Groupware, 2002
12. Clark, H, H., Schreuder, R., Buttrick, S., Common Ground and the Understanding of
    Demonstrative Reference, 1983
13. [Media Richness,
    Social Presence and Technology Supported Communication Activities in Education by Brain
    Newberry] [accessed on [01/05/10]
14. [Media
    Richness Theory] [accessed on [01/05/10]
15. Preece, Rogers and Sharp., Interaction Design, Beyond Human-Computer Interaction, Wiley,
16. Windows 2007 operating system
17. [Why You Only Need to Test with 5 Users by
    Nielsen, J] [accessed on 16/01/2010]
18. [Screen Resolution and Page Layout
    by Nielsen, J] [accessed on 22/04/2010]
19. [How to Conduct a
    Heuristic Evaluation] [accessed on 16/01/2010]
20. Greenberg, S., Gutwin, C., Roseman, M. Semantic Telepointers for Groupware

Appendix A: Evaluation of Error Messages
Preece et al suggests that the level of frustration caused to the user by error messages is high,
also suggested is that error messages should be more like „how-to-fix-it‟ messages instead of
simply saying what has happened. Schneiderman, 1998 suggests a set of guidelines for
designing good error messages in his book „Designing the User Interface, Strategies for
Effective Human-Computer Interaction‟. These guidelines are listed below:

      Rather than condemn users, messages should be courteous, indicating what users need
       to do to set things right
      Avoid using terms like FATAL, ERROR, INVALID, BAD, and ILLEGAL
      Avoid long code numbers and uppercase letters
      Audio warnings should be under the user‟s control, since they can cause much
      Messages should be precise rather than vague
      Messages should provide a help icon or command to allow users to get context-
       sensitive help
      Message should be provided at multiple levels, so that short messages can be
       supplemented with longer explanations.

Not all the guidelines set out by Schneiderman are relevant to the design of error messages in
PowerMeeting; they are more geared towards errors of non-web-based systems such as
operating systems. However they are important to consider as error messages are universal
and should be treated the same independent of the application they run on. The main point to
take from Schneiderman‟s guidelines is that the error messages should be precise and
courteous giving the user information on how to get back to using the system, avoid shocking
the user by using uppercase letters or terms such as those listed, the reason being that this
would induce frustration in the user further and this is unnecessary and not wanted. Other
points suggested are not required. For example the point made about audio will depend on the
user‟s settings for the operating system they are running on. Also as the errors caused in
PowerMeeting are not fatal or system errors then the context sensitive help is not a necessity
as the majority of the errors simply require the users to choose a different name or refresh the
page rather than take any drastic actions.

Figure 1 displays all the error messages that appear in PowerMeeting. Looking at the alerts
they are not too bad. They are expressed in plain language, they indicate what the problem is
an in some cases suggest an alternative. However they can be improved upon. I have
numbered each of the error messages so that they can be easily referenced when being

                                          Figure 1






Error message 1 is displayed when the connection is lost to PowerMeeting. This can happen
if the internet connection goes down, or if the computer in use loses the internet connection
itself or any other number of things. Error message one informs the user that the connection
is lost, and then tells the user to reload – this is the solution to the problem. Although the
error message is clear it isn‟t very precise and may introduce some ambiguity as simply
saying please reload doesn‟t inform the user of exactly what action they need to take to
overcome the error. Also the explanation mark that is used in the message may cause the user
to feel more frustrated as explanation marks are generally used to show strong feelings

including anger and joy.1 The use of the explanation mark is similar to the use of uppercase
letters, it creates an unnecessary state of urgency. Therefore the explanation mark may make
the user feel like the message is being almost shouted at them rather than informing them.

To improve this error message I would include more of an explanation to the user of the error
and give a better way of informing the user of the solution. “Sorry. The connection to
PowerMeeting has been lost, please try refreshing or reloading your browser and re-login to
PowerMeeting.” This seems a much friendlier way to put across the same message while still
being precise. Also being apologetic to the user makes the user feel less like it was their fault
that the error has occurred and should make them feel less frustrated at the error. Furthermore
the suggested message rather than simply saying reload asks the user to reload the browser
then try reclogging in to the system therefore the user knows exactly what they need to do in
order to overcome the error.

Error message 2 appears in two situations. One situation is when the user is trying to create a
new agenda item and the name they have entered is already in use and the second is when the
user is attempting to rename an agenda item and an agenda item of that name already exists.
This error message is very precise and indicates the problem and suggests the solution in a
clear and concise way. I see no clear cut problem with this error message with regards to the
heuristic, however it may be too brief an alternative to this error message could read “The
name you have entered is already in use, please enter a different name”. This expresses the
error in more of a natural way, as though the computer is suggesting the solution rather than
demanding, the idea being that it appears less forceful on the user.

Error messages 3, 4 and 5 all appear due to a user who is not the session chair being allowed
access to options that they are not allowed to use. With error messages 3 and 4 this can easily
be avoided as these two error messages are shown when the user tries to move or delete
agenda items. This problem has been covered a number of times in this document, removing
the conditions which allow the user to access options that are not available to them will then
stop this error from happening and therefore mean that these error messages are no longer
required. Reducing the number of error messages should then reduce user frustration. Error
message 5 is shown when a user clicks on an agenda item as they are trying to open the item.
PowerMeeting only allows the session chair to do this as they control what the other users
can see. In this case it may be best to leave the actual conditions that cause the error as we do
not want to remove the list of agenda items from the interface as it is important for all users
to know what items are already created. Also if the session is not in meeting mode then the
users are then able to select the agenda items to change their view. A solution for this is to
grey out the names of the agenda items to indicate to the user that they are not selectable. A
problem with error message 5 is that it refers to the meeting mode. This „meeting mode‟ is
not a term used anywhere else in PowerMeeting and refers to the mode that is set by the
session chair by checking or un checking the tick box, this tick box has no label, if this tick
box did have a label saying „Meeting Mode‟ then the users would have some knowledge of
this mode and so the error message would make more sense. Again this error message uses

1 (Accessed on 17/01/2010)

the explanation mark, which as discussed may be viewed by the user as the system is
shouting at them. A possible improved error message for this situation could be “Meeting
mode is currently turned on and so you cannot change the agenda item. Please talk to the
session chair if you would like to view another agenda item.”

A further message I found in PowerMeeting is shown in figure 2, it is not an error message as
such but it is worth inspecting. Here we see the text “Warning” which is quite alarming to a
user and is a term that should be avoided, especially when the message is not an error. This
box appears when a user selects to delete a task element in the task diagramming tool. It is
good in that the user is being given the chance to prevent the deletion of the task and its links
if they clicked it accidentally but it is not very friendly and may alarm the user. Instead a
simple message saying “Are you sure you would like to delete the task and all its links?”
should be used as it is much more friendly than “Warning”.
Figure 2- a warning in PowerMeeting

Appendix B: Results from evaluation using Nielsen's heuristics
No.   Issue                          Heuristic           Possible Solution
1     The loading status text is     Visibility of       Give a dynamic status indicator e.g. A
      static, this may not reflect   System Status       status bar.
      the system status if the
      system crashes.
2     No indication to the user      Visibility of       Give a status bar showing that the tool is
      that the tool is being         System Status       loading
3     the participant panel is       Visibility of       Refresh the participant list at a defined
      not updated, users who         System Status       time interval to update the list of
      are not online may be                              participants and remove those who are
      shown as online                                    inactive, or change their status to offline
                                                         and show all participants who have
                                                         taken part in the session
4     Session and General user       Visibility of       Create two views, one for the session
      states are not reflected in    System Status       chair and one for the general user
      the system, both are
      shown the same interface
5     options are visible to the     Visibility of       Create two views, one for the session
      general user that they         System Status and   chair and one for the general user
      cannot use such as the         Error Prevention    discarding unnecessary options from the
      edit menu                      and Aesthetic and   general user view
                                     Minimalist Design
6     no obvious indication          Visibility of       Add a label that says when the system is
      whether or not the             System Status       in meeting mode and when it is not
      system is in meeting
7     Agenda items with no           Visibility of       Don’t allow the system to create items
      name are displayed in the      System Status       with no name
      agenda panel but cannot
      be opened
8     Participants can create        Error Prevention    Don’t allow the system to create items
      agenda items with no                               with no name
9     The recycle bin is a static    Match Between       Change the bin icon
      image with one state,          System and the
      mental model would             Real World and
      suggest it has some            Visibility of
      contents                       System Status
10    The recycle bin doesn’t        Match Between       Change the bin icon
      function as a recycle bin-     System and the
      cannot recycle anything        Real World
      from it
11    when deleting something        Match Between       Change the bin icon and create an 'are
      with the recycle bin the       System and the      you sure?' prompt to inform the user
      user is not informed that      Real World and      they will permanently delete the object
      it will permanently be lost    Error Prevention
12    the two arrow buttons in       Match Between       Change the cursor on hovering over
      the calendar tool used for     System and the      these buttons, and depress them
      navigation don’t behave        Real World

     like buttons
13   user is asked to create an    Match Between       Change to 'create agenda item'
     'agent item'- this is not a   System and the
     natural term                  Real World and
                                   Consistency and
14   when hovering over an         Match Between       Change cursor behaviour to change to
     agenda item title the         System and the      hand icon rather than text editor
     cursor changes to the text    Real World and
     editor cursor                 Consistency and
15   The popup box in the          Match Between       Change to 'Date'
     calendar agenda item          System and the
     labels the date 'Datum'       Real World
16   The pin card board,           Match Between       Find new icons
     brainstorming and HTML        System and the
     agenda item icons don’t       Real World
     match their respective
17   No easy way of deleting       User Control and    Stop the system from allowing items
     agenda items created          Freedom             with no name being created.
     erroneously with no
18   no way of navigation          User Control and    include some standard navigation
     through webpages in html      Freedom             buttons
19   no are you sure prompt        User Control and    Add an are you sure prompt before
     for logging out, may have     Freedom             logging a user out of the system
     been accidently selected
20   User is unable to select      User Control and    Make the agenda icon selectable to
     the agenda icon to go to      Freedom             open the agenda item
     that agenda
21   No 'Go To' date function      User Control and    Include a Go To date function in the
     meaning only a long way       Freedom and         calendar
     to get to the date            Flexibility and
                                   Efficiency of Use
22   There are different ways      Consistency and     Have one application wide way of
     to delete elements in         Standards           deleting elements
     different agenda items
23   The agenda items: slide,      Consistency and     Include an add button on the
     task diagram and pin card     Standards           brainstorming agenda item
     board all have an add
     button, the brainstorming
     agenda item does not
24   The calendar tool goes        Consistency and     Resize the calendar agenda item to fit
     over the normal               Standards           the work area
     workspace size
25   The icons used to             Consistency and     The icons should be included on the
     represent agenda items        Standards           create menu next to their respective
     don’t represent the items                         agenda items
     until creation

26   Users can accidently           Error Prevention     Create an Are you sure prompt to give
     delete the wrong tool          and User Control     users a second chance
                                    and Freedom
27   No way of mapping the          Recognition          The icons should be included on the
     icons for created agenda       Rather than Recall   create menu next to their respective
     items to the named                                  agenda items
     agenda items in the
     create menu
28   No easy way of finding the     Recognition          Have an icon with a star for new tools
     newly created icon             Rather than Recall
29   Participants cannot            Recognition          Include information of where the
     recognise who is on what       Rather than Recall   participants are on the interface
     tool without remembering
     the information from the
     chat panel
30   creating a link in the task    Flexibility and      Change the way links are created
     diagram agenda item is         Efficiency of Use    perhaps with a drawing tool
     over complicated
31   Moving an agenda item          Flexibility and      Change to a drag and drop system in this
     up / down in the agenda        Efficiency of Use    panel to reorganise items
     panel can take some time
     and many steps
32   lots of unused white           Aesthetic and        Change the position of the interface,
     space, makes the               Minimalist Design    make some panels larger
     interface look 'lost' on the
33   Newly created agenda           Aesthetic and        Order the agenda items alphabetically,
     items are put to the           Minimalist Design    or put new agenda items on top or
     bottom of the list by                               group agenda items by type
34   The screen can become          Aesthetic and        Altering the way we use this tool will
     quite full with icons on       Minimalist Design    mean these icons can be removed
     the task manager tool as
     there are many icons on
     the task elements
35   the lost connection error      Help Users,          Change the message to read "sorry. The
     message isn’t very precise     Recognize,           connection to PowerMeeting has been
                                    Diagnose, and        lost, please try refreshing or reloading
                                    Recover from         your browser and logging back in"
36   Some of the error              Help Users,          Remove this explanation mark
     messages contain an            Recognize,
     explanation mark which         Diagnose, and
     may cause unnecessary          Recover from
     urgency                        Errors
37   the error message              Help Users,          Change to "The name you have entered
     displayed when a name is       Recognize,           is already in use, please enter a different
     already in use may be too      Diagnose, and        name." (this is less forceful and more
     brief and demanding            Recover from         natural
                                    Errors and Match
                                    Between System

                                              and Real World
        38      Error messages informing      Error Prevention   Remove conditions that allow the user
                a user that they cannot do                       to select options unavailable to them
                something as they are the
                session chair could be
        39      One error message refers      Help Users,        Label the Meeting mode tick box, and
                to the Meeting Mode, this     Recognize,         inform users that this is on or off
                is not referred to            Diagnose, and
                anywhere else in the          Recover from
                system                        Errors and
                                              Consistency and
        40      The message that appears      Help Users,        Remove the warning title and make the
                when a user selects to        Recognize,         message overall friendlier "Are you sure
                delete a task in the task     Diagnose, and      you would like to delete the task and all
                diagram agenda item says      Recover from       its links?"
                "Warning", this is not        Errors
                actually an error so should
                not be so alarming
        41      Access to the user guide is   Help and           Change the way of accessing the user
                not very straightforward      Documentation      guide to simply clicking a help button
                                                                 and bringing up the user guide.
        42      Tool tips can be              Help and           Improve the tool tips
                ambiguous                     Documentation

Heuristic                                                               Number of problems found          Percentage
Visibility of System Status                                                                        8            19%
Match Between System and the Real World                                                            9            21%
Consistancy and Standards                                                                          7            17%
User Control and Freedom                                                                           6            14%
Regocnition Rather than Recall                                                                     3             7%
Error Prevention                                                                                   5            12%
Felxibility and Effciency of Use                                                                   3             7%
Aesthetic and Minimalist Design                                                                    4            10%
Help Users, Recognize, Diagnose, and Recover from Errors                                           5            12%
Help and Documentaiton                                                                             2             5%

Appendix C: Consent Form

                                       Consent Form

Project Title:           Tools for supporting Web-Based Collaborative Decision Making –
                         Evaluation of PowerMeeting

Name of Researcher:      Richard Barber

      I have read and understand the information sheet relating to this workshop

      I have freely volunteered to participate in this workshop

      I have been informed in advance what my tasks are and what procedures will follow

      I have had the opportunity to ask questions about the study and have had my
       questions answered to my full satisfaction

      I am aware that my participation is voluntary and that I am free to withdraw at any
       time, without reason and without prejudice to my further treatment

      My signature below may be taken as affirmation of the above statements; it was given
       prior to participation in this workshop

Print Name: ___________________                     Signed: _____________________
Date: ___/___/_____

Appendix D: User workshop tasks and post observation questionnaire

You are part of a team who have been assigned a project to be completed by May 17th 2012.
The project is to create a website for selling children‟s toys. Features the website must
include are e-commerce functionality and a user friendly interface. Navigation through the
website should be aided by categorising the products into departments (e.g. a slide would be
in the outdoor department and a ball may be in both the outdoor and sports departments).

It is important that only PowerMeeting is used to plan this project. Below are a set of
deliverables that you should complete. You should work on each task together except for the
final task which is only for the session chair.

Throughout the workshop I urge you to write down anything you find difficult or dislike
about the system. If there are any problems please firstly write them down and ask me for any
help. I‟ll be logged into PowerMeeting to see how you progress.

Start by logging in and deciding who should be the session chair

Deliverable One
Create a slide informing the other participants of the task set. This slide should outline the
key points in the brief. Start Meeting mode and explain what needs to be done.

Deliverable Two
Key dates should be decided upon and added to a calendar tool.

Deliverable Three
A task diagram indicating how the tasks are going to be carried out for the project i.e. the
order of the tasks.

Deliverable Four
Ideas for the navigation of the site should be made so you know what categories to include
and what products may be available in the categories.

Deliverable Five
Research web design using the HTML tool, in particularly how ecommerce could be
incorporated into this site and create a slide explaining this research.

Deliverable Six
Each leave a note on the Pin card Board saying what you have done for the project.

Final Task (session chair)
Sort the tools in the Agenda panel into alphabetical order. Delete the bottom tool after sorting
and rename the top tool “Toy Website”.

Workshop Post Observation Questionnaire

Please fill in the following questions the best you can. If you need more space please
continue you answer on the back of the sheet and number it. Remember it is not you who is
being tested it is the system.

   1. What type of participant were you?

       Session Chair         □
       General Participant   □
   2. Place a tick in the box next to the tools that you used

       Slide           □
       Pincard Board   □
       HTML            □
       Task Diagram    □
       Brainstorming   □
       Calendar        □
   3. How did you find navigating around the system? Did anything hinder you moving
      around the system?


4. How easy did you find it complete tasks? Was everything available for you to
   complete these tasks efficiently?
5. Was there any aspect of the system that confused you in anyway? For example did
   you not understand some of the icons or phrases used in the system?

6. Consider the icons used in the system. Where there any that meant nothing to you?
   Please suggest some alternative icons for this you have identified (feel free to go back
   and look at the icons)

7. Write down anything that may have frustrated you during the use of the
   PowerMeeting. Why where these frustrating? Could you suggest a way to improve the
   system to minimize these frustrations?

   8. Do you have any suggestions on improving PowerMeeting?

The following questions are for the session chair, general participants skip to question

   9. Did you find it easy to turn on meeting mode? If not why not?

   10. How easy was it rearrange the tool in the agenda panel for your final task?

The following questions are for the general participants

   11. Where there any options you tried to access but could not? How did you feel about

   12. How did you feel about the meeting mode? Was there anything that could be done to
       make this mode better for you i.e. make you understand what was happening

Appendix E: User Workshop Results
No.   Issue                                                              No of      Notes
1     Some Compatibility issues. Reload connection message               4
      appears when user logs out when they are not using Internet
2     No are you sure prompt for logging out, could be an accidental     6
3     "Logging on is not explained" - No indication of required fields   8
4     The interface is not fitted e.g. It’s got no clear border or end   4
5     Don’t know what Brainstorming icon means                           27
6     Don’t know what HTML icon means                                    15
7     Don’t know what Pin card Board icon means                          27
8     Don’t know what Task Diagram icon means                            6
9     The menus didn’t fit users mental model, should be one click       4
      open one click close
10    Unsure of meaning of user icons - didn’t know pink meant           27
      session chair, natural assumption was that pink meant female
      and blue male
11    Some users were shown as being online even though they             13
      were offline this caused some confusion
12    User thought the participants could be selected but was            4
      unable to do this. The user wanted to get the other participant
      over to their tool
13    User used telepointer to show another participant something        1
      but was confused when the menu didn’t show the telepointer
      was activated (no tick)
14    Other users became confused as telepointer was also shown          10        Focus groups only
      on their agenda item even though the participant using it was
      not using it on that agenda item.
15    Tried to access menu options available only to chair, didn’t       13
      know why these where shown if not available
16    elements were allowed to overlap on the brainstorming tool         3         Only found in
      which covered and up some elements so they could not be                      focus group
      easily accessed
17    elements were allowed to overlap on the pin card board tool        3         Only found in
      which covered and up some elements so they could not be                      focus group
      easily accessed
18    elements were allowed to overlap on the task manager tool          4         Only found in
      which covered and up some elements so they could not be                      focus group
      easily accessed
19    elements were allowed to overlap on the slide tool which           2         Only found in
      covered and up some elements so they could not be easily                     focus group
20    Participants were confused when meeting mode was activated         16
21    Participants were frustrated at being taken away from what         16
      they were working on when meeting mode was activated
22    There was a random tick box on the calendar tool which had         3         discussed in focus

     no apparent meaning. (users believed that it may be to select           group 1
     the duration of an event for all day, but it doesn’t do this)
23   Says Datum instead of Date on calendar. Users asked why           9     discussed in focus
24   Calendar not coupled in meeting mode and so people did not        9     Mainly found in
     stay on same view and so couldn’t see what was going on                 focus group
25   No easy way to go to a date far in the future, have to scroll     13
     through by month.
26   No indication of the year that the calendar tool is on            13
27   In task diagram the 'add' text box is not cleared when the        18
     element has been added. This has to be cleared by the user
28   Poor way of linking elements in task diagram                      14    Discussed at
                                                                             length in focus
29   no direction arrow shown on links created in the task diagram     4
     until the elements had been moved
30   in the task diagram lines can overlap and look confusing          6     Found in focus
31   participants are able to add tasks of the same name in the task   4     Discussed in
     diagramming tool, this should not be allowed - also in other            focus group
32   User tried to access user guide, pressed OK on the prompt         2
     rather than the user guide link.
33   tried double clicking on task name to rename- mental model        4     Discussed in
                                                                             focus group
34   Couldn’t add an element on the brainstorming tool as there        8     Discussed in
     was no add button                                                       focus group
35   Empty elements where allowed to be added erroneously              15    Discussed in
                                                                             focus group
36   no go to button on HTML page                                      8     Discussed in
                                                                             focus group
37   no navigation buttons on HTML tool                                15    Discussed in
                                                                             focus group
38   HTML tool was not coupled and so participants could not see       12    Discussed in
     what each other saw.                                                    focus group
39   the calendar tool does not fit in interface                       7
40   Popups stay open to participants when meeting mode is             3     Discussed in
     activated even if they are taken to a different tool.                   focus group
41   HTML tool was not user friendly as users had to type in           15    Discussed in
     'http://' In the web address, this is not the norm                      focus group
42   There is no indication of loading in the HTML tool                22
43   Users sometimes accessed the same element at the same time        4     Discussed in
     which caused confusion and meant one participant did not get            focus group
     the desired outcome for their actions
44   Icons on the agenda panel were not clickable                      13
45   Was some lag when using verbal communication and at times         n/a   Only found in
     people found it hard to know when they could speak                      focus group and
                                                                             only discussed
46   group chat was used sparingly when verbal communication           n/a   Observed
     was allowed - used to note particular things such as dates

47   participants complained about the size of the group chat panel   n/a   Only found in
     when they stopped using the verbal communication                       focus group and
                                                                            only discussed
48   Participant did not like having to leave PowerMeeting to         4     Only found in
     access Skype to change microphone or speaker volume                    focus group
49   No feedback to user to say that another user is typing a         2
     message to them lets the user know they are still there.
50   In the calendar agenda item some of the buttons don’t            10
     function as button- cursor doesn’t change
51   Cannot see what have typed in chat, unable to easily scroll      6
     back through the message you have typed

Appendix F: results from evaluation using Groupware heuristics
No.   Issue                        Heuristic          Possible Solution
1     Chat is not logged and the   Provide the        Log the chat and save it, displaying it in the
      history of the               means for          chat panel at all times
      communication is not         intentional and
      shown to all participants    appropriate
2     Chat panel is small so       Provide the        increase the size of the chat panel
      people may have to scroll    means for
      to keep up with              intentional and
      conversation after only a    appropriate
      small amount of text is      verbal
      entered                      communication
3     telepointer cannot be        Provide the        multiple telepointers can be used to more
      used by more than one        means for          than one participant can 'point at something
      user                         intentional and
4     no gestures are support      Provide the        multiple telepointers, and invite people to
      apart from one user being    means for          agenda items, same as come here gesture
      able to point using the      intentional and
      telepointer                  appropriate
5     no way of displaying         Provide            telepointers lets you know where
      unintentional gesture        consequential      participants are so lets you have some idea
                                   communication of   of what they are doing
                                   an individual's
6     no idea of whom you are      Provide            colour coded pointers would let you know
      collaborating with in a      consequential      who you are with and also stating who is in
      specific agenda item         communication of   working on the agenda item would let the
                                   an individual's    participants know who they are collaborating
                                   embodiment         with
7     no inclusion of internal     Provide            include the audio channel, will allow users to
      audio channel                consequential      pick up on unintentional utterances
                                   communication of
                                   an individual's
                                   embodiment and
                                   Provide the
                                   means for
                                   intentional and
                                   and Support
                                   people with the

                                  coordination of
                                  their actions

8    no way of knowing what       Provide            Provide some form of automate
     agenda item people are       consequential      communication to the participants from the
     working on                   communication of   interface that informs them what everyone is
                                  shared artefacts   doing within the session (labels on
                                                     participant panel and agenda panel)
9    No way of knowing what       Provide            Provide some form of automate
     someone is doing within      consequential      communication to the participants from the
     an agenda item               communication of   interface that informs them what everyone is
                                  shared artefacts   doing within the session (labels on
                                                     participant panel and agenda panel)
10   incomplete action feed       Provide            Show intermittent states of elements as well
     through                      consequential      as initial and final states
                                  communication of
                                  shared artefacts
11   Participants are unaware     Provide            Give a delay
     that the session chair is    consequential
     about to change to           communication of
     meeting mode                 shared artefacts
12   Participants are taken       Provide            Give a delay
     immediately to meeting       consequential
     mode                         communication of
                                  shared artefacts
13   Don’t know the history of    Provide            Log information on agenda items
     agenda items                 consequential
                                  communication of
                                  shared artefacts
14   Don't know history of        Provide            Log information on agenda items and there
     elements within agenda       consequential      life cycle
     items                        communication of
                                  shared artefacts
15   Allows participants to       Provide            Lock items when a user is editing them
     concurrently access an       Protection         and/or make it explicit that the item is in use
     element in the task
     diagram tool and both
     select to rename it
16   Allows participants to       Provide            Lock items when a user is editing them
     concurrently move            Protection         and/or make it explicit that the item is in use
17   Allows one user to start     Provide            Lock items when a user is editing them
     editing an element and       Protection         and/or make it explicit that the item is in use
     another user to delete the
     element while someone is
     still editing it
18   no way of protecting         Provide            supply a locking facility on agenda items
     complete agenda items        Protection
19   doesn’t give awareness of    Support people     telepointers and other mentioned methods
     other participants           with the
                                  coordination of

                                               their actions
       20     doesn’t show where other         Support people       telepointers and other mentioned methods
              users are in relation to         with the
              other users                      coordination of
                                               their actions
       21     doesn’t show where users         Support people       telepointers and other mentioned methods
              are in relation to elements      with the
                                               coordination of
                                               their actions
       22     participants availability        Facilitate finding   Give status of idle, online or offline
              not explicit                     collaborators and
       23     no contact information for       Facilitate finding   Include this information on the user’s name
              participants                     collaborators and    (small profile?)
       24     don’t know who is in the         Facilitate finding   telepointers and updates participant list
              meeting                          collaborators and

Heuristic                                                                    Number of problems found            Percentage
Provide the means for intentional and appropriate verbal communication                                       3       12.50%
Provide the means for intentional and appropriate gestural communication                                     2           8%
Provide consequential communication of an individual's embodiment                                            3       12.50%
Provide consequential communication of shared artefacts                                                      7          29%
Provide Protection                                                                                           4          17%
Manage the transitions between tightly and loosely-couple collaboration                                      0           0%
Support people with the coordination of their actions                                                        4          17%
Facilitate finding collaborators and establishing contact                                                    3       12.50%


Shared By: