In pursuit of excellence in research program
                  management: the journey so far!
      Dr Lewis Atkinson°, Mr Ian Jensen°, Dr Catherine Manathunga#, & Ms Christine Raward°
                                 °Meat & Livestock Australia Ltd,
                         Locked Bag 991, NORTH SYDNEY, NSW, 2059.
                Teaching and Educational Development Institute/UQ Graduate School,
                                 University of Queensland, Q 4072
                         Tel: 0419 240 979 / email:

           Researchers are highly trained professionals, who bring to their work a
           range of academic, industrial and life experience. They are generally self-
           directed employees seeking to extend the boundaries of scientific and
           commercial knowledge. They usually value professional autonomy and
           respect for their research judgment. The nature of their role as research
           program managers, however, is often unspecified and many researchers have
           not had formal training in program management. This paper outlines an
           innovative professional development program that defines the characteristics
           of effective research program management and of successful program
           managers. This program, Program Excellence, was developed by the Meat &
           Livestock Australia’s (MLA) as part of a systematic organisational
           restructure and professional development plan for its Client & Innovation
           Services business unit (CIS). The elements of the Program Excellence
           Initiative could be adaptable to other industrial and research contexts.

Researchers are highly trained professionals, who bring to their work a range of academic,
industrial and life experience. They are generally self-directed employees seeking to extend the
boundaries of scientific and commercial knowledge. They usually value professional autonomy
and respect for their research judgment. The nature of their role as research program managers,
however, is often unspecified and many researchers have not had formal training in program
management. The Meat & Livestock Australia’s (MLA) Program Excellence Initiative is part of
a systematic organisational restructure and professional development plan for the Client &
Innovation Services business unit (CIS). It was first introduced as part of MLA’s Good to Great
program in 2003. It was designed to define the essential elements and qualities of a successful
program and a successful program manager. The initiative is basically a response to the rapid
growth in industry demand for the innovation services provided by CIS, along with an “upshift”
in the complexity of the role of the Program Manager. It is apparent that this role has changed
over the past 15 years from one focused on program coordination and decision implementation
(consensus by committee) to a much more strategic/investment oriented form of management
(business driven R&D). This transition was described in a paper presented at ARMS’03 in
Auckland 1(reference??)

This paper reviews the management and organisational psychology literature that was drawn
upon to develop the Program Excellence Initiative. It outlines how these understandings have
shaped the program’s definition of effective research program management and of successful
program managers. It also explores the rationale for this particular approach to the professional
development of highly skilled and autonomous employees. The initial implementation of the
program is reviewed, drawing upon the commendations and recommendations made by the
program’s external reviewer 2. In particular, the paper investigates the further improvements
required to transform this innovative idea into a practical and effective reality.

Defining excellent research program management and

There is a wealth of literature from management and organisational psychology disciplines that
suggests a range of models for effective program management and excellent program managers.
In a functional sense, program management may be thought of as a management framework that
is applied to business initiatives, consisting of a series of related projects. Program Managers are
industry leaders with responsibility for strategic development (focusing on the big picture) and
also have responsibility for the execution of a group of projects designed to meet a program’s
goal. Program Management at Meat Livestock Australia (MLA) Ltd is a core organisational
capability. Program managers are (or need to be) industry leaders that develop and communicate
a shared vision for strategic development in their area of responsibility. They are also
responsible for project execution and the translation of program outcomes into a benefit to the

In view of this, the Program Excellence initiative was created to provide respectful and
innovative leadership and mentorship to a team of ‘dynamic and ambitious program managers
and co-ordinators’ by a ‘small group of senior managers ... with considerable technical skills;
broad industry and commercial experience; and a depth of wisdom and understanding’. In
particular, it has sought to ‘enhance the leadership and creativity within the team and to ensure
alignment with MLA’s organisational development program’3.

At the heart of program management is a passion for the program goal. There should be no
scrimping on the vision for a program; it needs to be large, exciting, saleable and achievable. A
program must also be holistic in its vision of the industry, its capability, needs, and priorities.
There is little point delivering a research outcome with great fanfare before the industry is ready
and able to use it, and maybe even before they realize the need. There is a need to conduct a
program, in the same sense that a conductor conducts the orchestra. The qualities of the music
and the value to the listener must be considered, so the music must be at the correct tempo, the
timing of entrances of various instruments need to be correct, and the balance of dynamics and
timbres must be managed with finesse. Program management is a process of creative midwifery.

There are some similarities between the role of the program manager and the conduct of action
research, in which the researcher is both taking actions and learning at the same time. Action
research is a flexible spiral process, which allows action (change, improvement) and research
(understanding, knowledge) to be achieved at the same time. The understanding allows more
informed change and at the same time is informed by that change. People affected by the change

  Manathunga, C. (2005). Review of the Meat and Livestock Australia’s Program Excellence Initiative.
Brisbane: The University of Queensland.
  CIS (2005) A new model for leadership and creavitity: the journey continues. Sydney: MLA, p. 1.
are usually involved in the action research. This allows the understanding to be widely shared
and the change to be pursued with commitment.4

Since a program is attempting to achieve certain benefits for the industry and other stakeholders,
and there is a need for innovation to achieve these benefits, a model can be developed in how the
program works with the interested parties to achieve the goals. A series of interactions can be
described, a series of interventions and responses, or actions and results. Figure1 applies these
understandings of action research to what the MLA has called its ‘Industry Innovation Cycle’.

         Innovation Cycle:
                                   De pa        Strategy

         anatomy of a                                                D nt
                                                                  R& tme
                                     sir ct



                        Evaluation                                  Mechanisms

                                           e      n              En
                                       ctiv tatio               Ad ter
                                    fe n                          op pri
                                  Ef me                             tio se
                                    pl              Innovation

                           Figure 1: MLA Industry Innovation Cycle

This abstract model can be developed towards MLA’s reality by the inclusion of functional steps
such as consultation with stakeholders, the engagement of industry in project delivery and
selecting and developing suppliers (figure2)

 Dick, B. (2002) Action research: action and research [On line]. Available at
    MLA functional approach to innovation

         SALES                CONSULTATION
        PROCESS                 PROCESS

       SUPPLIER                 PROJECT
                                                         AND ADOPTION

                        Figure 2: Functional definition of program blocks

Program and project management

Project Management is a major activity for MLA, and probably consumes more than 50% of the
time and effort of program managers, project coordinators, contract and administrative support
staff. Within project management, strategy can be one of the most difficult parts of a program to
conceptualise, but its clear articulation is foundational to the long term sustainability of the
program. There are a number of approaches to strategy5 and the usefulness of a particular
approach probably depends on the particular problem or issue that a program intends to address.
The Kellogg program logic model is a generic model used widely in social development
programs and considered state-of-the-art in program development.6 It suggests that the
development of a program requires a number of activities, which may be called the strategic
process or the development of a logic model for the program7. These include the development of
      problem or issue statement that describes the problem that the program is attempting to
      planned range of future, short- and long-term achievements (outputs, outcomes and

  Henry Mintzberg and Joseph Lampel (1999) Reflecting on the Strategy Process. Sloan Management
Review 40(3):21-30.
  W. K. Kellogg Foundation (2001) Logic Model Development Guide. and personal comments from New Zealand Foundation
for Research Science and Technology.
  W. K. Kellogg Foundation (2001) op. cit.
       list of influential factors, including consumer and customer expectations and
        requirements, the policy environment, the industry’s capabilities and situation that will
        influence adoption and uptake in the industry
       list of assumptions behind how and why the change strategies will work in the industry
        from previous experience.

The Bennet model has been used in agricultural research and extension, and has particular
applicability to agricultural extension services.8 This model has been developed for both
program development and evaluation. The steps of the model that involve understanding of the
environment in which the industry operates and the practices, knowledge, attitudes, skills and
aspirations of the industry may be particularly helpful in understanding how a program can
achieve its objectives and the changes in these facets of the industry and operating environment
that are necessary for success of the program.

Developing evaluation tools

The subject of evaluation is as large again as the rest of program management. Evaluation is so
specialized that professional societies exist for its promotion9 and Guides have been written on
how to do it.10 Evaluation is closely tied to reporting. Outputs are the direct results of program
activities, whether conducted by contract or MLA’s own activities. They are usually described in
terms of the size and/or scope of the services and products delivered or produced by the
program11 Outputs are usually relatively easy to measure because they are usually a direct output
of the activities of the current program. Outcomes, on the other hand, can be seen as specific
changes in attitudes, behaviour, knowledge, skill status or level of functioning expected to result
from program activities over the longer term12. Therefore, there will be projects within a program
which focus on communication and adoption because support is needed over the longer term for
program outcomes to develop into outcomes through the process of industry adoption. Therefore
program outcomes are not measured in the same way as program outputs.

Within MLA, emphasis is placed on reporting of Key Performance Indicators (KPIs) for
programs. Key Performance Indicators are quantifiable measurements, agreed to beforehand, that
reflect the achievement of program objectives (impact). The tendency has been to require
outcomes as KPIs, but it is not realistic to expect that a program will always have outcomes to be
measured against, particularly in the early years of a program. Therefore, it makes sense to use a
combination of output and outcome related KPIs in any given reporting period. Whatever KPIs
are selected, they must reflect the program’s goals, they must be key to its success, they must be
quantifiable and they must be significant and meaningful to external stakeholders. KPIs may be
set for the life of a program as quantifiaible measures of progress toward a program outcome.
Then over a number of years a series of outcome-based KPIs may be developed and achieved as
achievement of the program objective comes closer.

  Claude Bennett, Shaun Coffey, Bronwen McDonald and Brian McNeal (2001) Planning and Evaluating
Collaborative Research and Extension. And Kay Rockwell and Claude Bennett Targeting Outcomes of
Programs (TOP) march 29,2004.
  European Evaluation Society, , American Evaluation Association, , Australasian Evaluation Society,
   W. K. Kellogg Foundation (1998). Evaluation Handbook.
   W. K. Kellogg Foundation (2001) Logic Model Development Guide.
   Kellogg op. cit.
In order to facilitate this kind of systematic evaluation of research program management, the
Program Excellence Initiative team developed a useful diagnostic tool called Program Health
Check, which enables program managers to assess the effectiveness of their program and identify
areas for improvement.

Assessment of program manager competency tool

So too, a reflective professional development tool was also developed to identify the
characteristics of an effective program manager. Using this tool, program managers are able to
chart their specific strengths as program managers and identify areas that need improvement.
This tool was developed by exploring the literature on worker competencies. Competency may
be defined as ‘the ability to perform tasks and duties to the standard expected in employment.’ 13

One approach to defining program manager competency used an innovation framework14 and
recognized critical roles for program managers in the four program actions:
     Innovation Strategy Development
     Best Practice Innovation Processes
     Innovation Culture & Capability Building
     Impact Evaluation

Another approach has been through the Baldridge Award15 criteria, which are based on a generic
business excellence framework. The Baldridge Award criteria are designed to deploy
performance management a cross the organization that results in:
     Delivery of continuously improving value to customers
     Improvement of overall organisational effectiveness and capability, and
     Organizational and personal learning.

There are clear parallels between the organizational excellence framework employed by
Baldridge these criteria useful in defining the ‘best practice’ Program Manager it is necessary to
extract from the criteria those aspects that, in the MLA context, are the responsibility of, or
depend most completely, on the competence and attitudes of the individual program manager.

One method for assessing the competence of existing program managers is to assess each
criterion against the factors: Approach, Deployment, Learning, and Integration (A-D-L-I)16. This
emphasises that, beyond establishing an approach and deploying it, to achieve results (outputs
and outcomes) high-performing individuals also focus on process that includes aspects of
organisational improvement (learning) and process integration with other organisational
imperatives within MLA. This may be viewed as an application of the Total Quality
Management Plan-Do-Check-Act (P-D-C-A) cycle.

Therefore, these criterion are based upon two evaluation dimensions; process and results. In the
context of a MLA program, the results that could be attributable to the program manager would

   Atkinson, Jenson and Raward (2003) Program Managers are made- not born! ARMS Conference
Auckland NZ.
   Baldridge National Quality Program (2004) Criteria for Performance Excellence
   Baldridge op. cit.
typically be those derived as a measurement of program achievement against relevant program
KPI’s for both outputs and outcomes. These would be based upon industry adoption and impact
benchmark targets that were established during the initial strategy development phase of the
program. .

The ultimate selection of the assessment criterion for the “ideal” program manager was largely
based upon a consultation process with current program managers and coordinators within the
CIS business unit. The key methodology employed to generate the final list of assessment
criteria was a group-based creative “mindmapping” technique from Buzan17. However, there is
also a nice fit between these criteria for the process dimension and the key actions/process
undertaken by a program:
      Leadership = Enabling Leadership
      Strategic Planning = Enabling Strategic Management
      Customer & Market Focus = Enabling Customer Communications
      Process Management = Enabling Processes & Systems
      Human Resource Focus = Enabling Technical Resource Management

There is also a fit with the results dimension and the reactions/results, or outcomes of the
program (see figure 1):
     R&D investment by the stakeholders
     Adoption by enterprises- (initial trials, early-adopters)
     Effective implementation throughout the industry
     Desired impact on the industry

The Baldridge assessment tool above is primarily a measure of the program manager’s ability to
achieve outcomes through effective management of an existing program. Its generic nature will
allow for benchmarking of program manager performance against those in similar organisations.

An extract (see figure 3) has also been taken from the model of situational leadership 18 with a
particular focus on assessing the individual development of “know how, mindsets, and attitudes”
which indicate a level of commitment that is critical to successful program management. This
assessment can be used to determine the ‘readiness level’ of the individual program manager.
The results can then be used to guide the manager/mentor of the program manager in responding
appropriately with direction, training and support. These criteria will also find use as a screening
tool for the recruitment and selection, with respect to the development levels, of new program

  Buzan, Tony (2002) How to Mind Map ® London: Thorsons
  Kenneth Blanchard, Patricia Zigarmi and Drea Zigarmi (1987) Leadership and the One Minute Manager.

                                                          SUPPORTING                   S2

                                                           praise, listen,
                                                             facilitate        direct and support

                                                     responsibility for day

                                                                                structure, supervise,
                                                       to day decision

                                           Low                      DIRECTIVE BEHAVIOUR                  high

                                                      R3                            R2
                                      R4                                                                   R1
                                               high competence                     some
                               high competence                                                      low competence
                                                    variable                    competence
                               high commitment                                                      high commitment
                                                  commitment                  low commitment

                                                                 S3                                          S1
                                   DELEGATING                                         S2
                                                           SUPPORTING                                   DIRECTING
                                responsibility for day                           COACHING
                                                            praise, listen,                         structure, supervise,
                                  to day decision                             direct and support
                                                              facilitate                                   control

                           Figure 3: Situational Leadership for leadership development and performance

Importantly, the Program Excellence Initiative recognises that program managers cannot possibly
be expected to be ‘sufficiently competent in all areas ... and have sufficient time to devote to all
tasks’ and also recommends a number of strategies to deal with this (see illustration below).
          Profile of an Ideal Program Manager

The Implementation

Many of the initial cohort of R&D program managers that were trained in the use of the
framework in December 2004 are still using the Program Excellence framework because of its
value as a professional development tool. Below is a testimonial from one such program

           The Program Excellence document highlights what to consider when
           assessing a program’s performance and capacity to deliver genuine
           outcomes for the red meat industry19 Reference??

           The tools developed in the Good to Great this year have enabled me to
           objectively assess both programs’ performance as well as my own. I plan to
           incorporate Program Excellence into the management of both the
           Environment and Co-products programs and my development as a program
           manager and industry leader20 . Reference??

Based on the feedback of program managers, the Program Excellence framework will continue to
evolve by implementing a number of changes that would simplify some of the developmental
tools and scales to enable ongoing support for the implementation of the framework.

The Program Health Check and Program Manager Competence Assessment Scales provided with
the Program Excellence framework have been found to be practical and informative.21 They
provide the context for strategic discussions to be held between program managers and their
manager/mentors and the identification of areas to direct individual program managers’ efforts to
improve. They also assist with the systematic evaluation of research programs.

Two workshops were held with MLA Program Managers and other staff interested in the
operations of MLA programs. The purpose of these workshops was to introduce the model,
provide a basis for program health check and program manager competence self-assessment and
obtain feedback on the usefulness of the model.

Comments from the first workshop included:
    ‘more strategic approach to discuss with [my] mentor’
    ‘seeing where I need to focus my efforts’
    ‘demonstrated there is a significant, detailed process for generic program excellence.
     Hopefully adapting it to my specific programs will yield fruit’
    ‘great for evaluation re end of the year or 3yr program before preparing a new

Comments from the second workshop included:
    ‘clear directions but at the same time allows space [for] creativity’,
    ‘guideline not a set of rules’
    ‘put things into perspective’
    ‘it facilitates development of shared values and culture’
    ‘anything that makes a team reflect and that encourages action learning is a positive
     move forward for the team and the organisation as a whole’23

An external review has been completed and the following commendations were received from
the reviewer:
     Clear and broad definition of a successful program that captures both the strategic big
         picture as well as the smaller details
     Synthesis of wide range of research-based management models
     Flexibility that provides direction while simultaneously encouraging creativity and self-
     Allows for shared values and helps to generate a positive and supportive team culture
     Program Health Check scale is practical and informative
     Clear, holistic identification of the characteristics of a successful program manager
     Recognition that program managers cannot be expected to be highly competent in all
     Program Manager Competence Assessment scale for measuring levels of readiness in the
         development of specific skills and know how, especially as it includes commitment as
         well as competence
     Enables strategic discussions with manager/mentors and the identification of areas to
         direct individual program managers’ efforts to improve
     Helps the systematic evaluation of programs

   Manathunga, 2005, op.cit.
   ibid, p. 6.
   Ibid, p. 8.
         Provides a realistic, supportive basis for ongoing professional development 24.

The external reviewer also identified through some follow up online focus groups and telephone
conversations some problems in the implementation of the Program Excellence Initiative that
require attention. In particular, 2 of the initial 4 program managers to experience the program
had either found the framework not applicable to their program or not a helpful exercise25.

Specifically, one program manager indicated that s/he had completed the Program Health Check
and Program Manager Competence Assessment tools but that s/he had received no feedback from
senior staff or those implementing the framework. Another program manager had also not found
it useful because s/he felt it was less applicable to his/her program.

All four program managers commented on the time required to use the tools and recommended
that they be simplified and streamlined in order to make them more time-effective to use in
practice. One program manager had translated the tools into Excel spreadsheets for ease of use.
Two of the program managers also felt some of the questions in the tools were ambiguous and
needed to be clarified or turned into opportunities for qualitative answers where questions
required more than a yes/no answer.

Three of the program managers also believed that the Program Excellence framework needed to
be more effectively integrated into the operations of the CIS Business Unit, with one manager
suggesting that support for its ongoing use could be provided by the new Research Operations
section. S/he also thought that the Strategic Review Team seemed to use different ratings of
programs to those used in the Program Excellence framework and recommended that these
needed to correspond in the future.

Another program manager also indicated that the Program Health Check tool needed to be
modified to mirror the different stages of program implementation because programs changed
significantly over their entire life cycle 26. Two managers recommended that the tools be
revisited every six months and action plans be developed and reviewed in collaboration with
mentors and the strategic review team. There was a belief among some of the program managers
that mentors also required additional training in facilitation and mentoring as opposed to only
offering advice or giving directions. There could also be room for additional tools that would
support the existing Program Excellence framework. For example, one program manager
suggested the development of a tool to identify significant stakeholders for various programs and
how you to effectively balance their different needs.

Where to from here?

It will be particularly important to address these implementation concerns. As a result, the
reviewer also made some significant recommendations for improving the Program Excellence
Initiative. These included the need to:
      Simplify:
             o Induction to the framework
             o Number and complexity of diagrams
             o Program Health Check tool (remove columns on sub-elements and indicators)
   Ibid, p. 9.
   Ibid, p. 7.
   Ibid, pp. 7-8.
                o    Program Manager Competence Assessment tool (remove process scales and
                     incorporate reflective components instead)
                o The tool questions to remove ambiguity
           Change the term ‘competency’ to ‘attributes’ or another term that does not have a
            pass/fail, lowest common denominator nuance to it
           Ensure adequate follow up and resourcing
           Integrate the use of the Program Excellence framework more effectively in the normal
            operations of the CIS Business Unit, perhaps using the new Research Operations section
            to provide ongoing support and assistance in using the framework
           Ensure that the ratings used by the Strategic Review Team match those used in the
            Program Excellence tools
           Constantly seek feedback about which aspects of the framework work or do not work
           Provide mentors with additional training in facilitating and mentoring as opposed to
            providing advice and direction
           Modify the Program Health Check tool to mirror the different stages of program
           Revisit the Program Excellence tools every six months and develop and review
            successive action plans in collaboration with mentors and the strategic review team
           Incorporate case studies and presentations by experienced program managers who can
            share real life positive and negative experiences of being a program manager 27.

If these issues are addressed, the Program Excellence Initiative could become the foundation for
good practice amongst our Program Managers, who are responsible for delivering innovation to
the Australian red meat industry. Over the past two years it has become part of “who we are”
and how we work, and has spawned complementary approaches in how we manage research
operations and adoption and commercialization services. But it is expected to continue to evolve
as further feedback is received during implementation.

A key component of the framework enables the employee and their mentor to develop a common
understanding of where the employee wishes to direct their overall development using an
“iterative action learning cycle” (see Figure 4), whilst enabling the mentor, and MLA, to explore
succession planning and talent development opportunities where appropriate.

     Ibid, pp. 9-10.
      Action Learning Framework

                                         New Role

                                               New skill and

                     Evidence: such as
                     PAR review

                                           Support; mentoring;                   Such as
                                           project work; role                    secondments


                                                                       CIS Team Presentation – 9 August 2005

               Figure 4: Action learning framework for mentoring research managers


The Program Excellence framework has been developed to ensure that the critical organisational
capability of “Program Management” is enhanced even further at MLA. This paper has outlined
the management and organisational psychology literature that was drawn upon to develop the
Program Excellence Initiative. It reviewed how these understandings have shaped the program’s
definition of effective research program management and of successful program managers. It
also explored the rational for this particular approach to the professional development of highly
skilled and autonomous employees. The initial implementation of the program was reviewed,
drawing upon the commendations and recommendations made by the program’s external
reviewer 28. While a number of improvements need to be made to the implementation of the
program, the MLA team believes that the framework has broad applicability to Cooperative
Research Centers and other research organizations with commercially focused outcomes.

References from Program Excellence handbook – document dated 16/9/04 that reviews the management lit
that I’ve quoted here??

ARMS 2003 paper

Employee quotes on p. 9

Client and Innovation Services (CIS). (2005). A new model for leadership and creativity: the journey
continues. Sydney: MLA.

Manathunga, C. (2005). Review of the Meat and Livestock Australia’s Program Excellence Initiative.
Brisbane: The University of Queensland.
ARMS 2005 Conference
      Canberra, ACT

             page 16

To top