Learning from the Case Studies

Document Sample
Learning from the Case Studies Powered By Docstoc
					                               Learning from
            the retrospective Case Studies:
                         A Synthesis of Lessons
                    for the PROTEE Instrument




                                       Ruth McNally,
                                       Steve Woolgar




                      Brunel University, Uxbridge, UB8 3PH
                                     PROTEE project


                                     September 1999




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc               Page 1 of 34
                                                                  Contents
1. INTRODUCTION ............................................................................................................................................. 3

2. THE PROTEE CASE STUDIES ..................................................................................................................... 4

    2.1. INDUSTRIAL PARTNERS ................................................................................................................................. 5
    2.2. RETROSPECTIVE CASE STUDIES .................................................................................................................... 5

3. THE EVOLVING PROTEE INSTRUMENT................................................................................................. 6

    3.1. STEMM ....................................................................................................................................................... 6
    3.2. THE PROTEE INTERVIEW GUIDE ................................................................................................................. 7
    3.3. DEVELOPING THE INDICATORS: PROTEE IN OTHER WORDS ......................................................................... 7
    3.4. THE PROJECT DESCRIPTION SUMMARY SHEET AND THE PROJECT LEARNING CURVE SUMMARY SHEET ..... 9

4. THE PROTEE PROCESS................................................................................................................................ 9

    4.1. PROBLEMS ENCOUNTERED ........................................................................................................................... 9
    4.2. THE 5-STEP PLAN ....................................................................................................................................... 10
    4.3. TIME „ZERO‟ .............................................................................................................................................. 12
    4.4. STRUCTURING THE PROTEE INTERVIEW ................................................................................................... 13
    4.5. CHOOSING TIME POINTS ............................................................................................................................. 15
    4.6. THE PROTEE REDESCRIPTION .................................................................................................................. 15
    4.7. THE PROTEE EVALUATION ....................................................................................................................... 16
    4.8. THE ROLES OF THE INNOVATOR AND THE EVALUATOR .............................................................................. 17
    4.9. PROTEE AS SOCIO-TECHNO-THERAPY ..................................................................................................... 17

5. THE PROTEE CONCEPT OF INNOVATION ........................................................................................... 18

    5.1. SOFTWARE AND HARDWARE INNOVATION ................................................................................................. 18
    5.2. THE UNIT OF ANALYSIS .............................................................................................................................. 19
    5.3. TECHNOLOGICAL ESSENTIALISM? .............................................................................................................. 20
    5.4. INNOVATION VERSUS DEVELOPMENT .......................................................................................................... 22

6. WHAT THE INNOVATORS THINK........................................................................................................... 23

    6.1. „WHAT DIFFERENCE WOULD PROTEE HAVE MADE?‟ ................................................................................ 23
    6.2. WILL THE INNOVATORS USE PROTEE? ..................................................................................................... 24

7. CONCLUSION: PROTEE AS PUNISHMENT OR AS DISCIPLINE? .................................................... 25

APPENDIX 1: PROTEE METHODOLOGY................................................................................................... 27

    PROJECT DESCRIPTION SUMMARY SHEET ............................................................................................ 29
    PROJECT LEARNING CURVE SUMMARY SHEET ................................................................................... 32




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                                                                     Page 2 of 34
                           Learning from the Case Studies:
                               A Synthesis of Lessons
                             for the PROTEE Instrument

                          Ruth McNally1 & Steve Woolgar2


1. INTRODUCTION


This report is based on lessons from the four retrospective case studies undertaken in
PROTEE: the ECT, the ETTC, and the KFHS and Commutor. 3
PROTEE aims to develop an instrument which can record the learning curve of innovation
projects in intermodal freight transport. The method used in PROTEE to develop this
instrument is a case study approach. PROTEE has undertaken five case studies on intermodal
freight transport projects. These case studies are not conventional case studies in that their
primary aim is to develop the PROTEE instrument rather than to investigate the transport
projects per se. This Report is a synthesis of lessons learned from the (available) PROTEE
case studies.
It is in the nature of the methodology used in the PROTEE project that what is tested in later
case studies is derived from lessons learned in earlier case studies. In other words, the lessons
reported in later case studies are from the experience of testing the PROTEE instrument as
developed through lessons derived from earlier case studies. This poses a challenge in the
writing of this synthesis which I hope we have succeeded in meeting.


1
    Research Fellow, Centre for Research into Innovation, Culture and Technology (CRICT), Department of
          Human Sciences, Brunel University, Uxbridge, Middlesex UB8 3PH, UK.
2
    Professor, Centre for Research into Innovation, Culture and Technology (CRICT), Department of Human
          Sciences, Brunel University, Uxbridge, Middlesex UB8 3PH, UK.
3
    Unless otherwise indicated, lessons on the KFHS case study are from McNally, R. and Sondermann, S.
          „PROTEE as socio-techno-therapy‟ (16 March 1999), Sections 2 and 6; lessons on the ETTC case study
          in the case of ZIV are from Rehse, S. „Experiences and chances in the application of PROTEE‟s
          methodology from the viewpoint of the ETTC innovator‟ (4 March 1999) and in the case of Maastricht
          are from Hommels, A. „PROTEE lessons from the ETTC case study‟ (11 March 1999); and lessons on
          the ECT case study are from Peters, P. „Analysis of the ECT case study, using a PROTEE approach‟ (16
          March 1999); lessons on the Commutor case study are from Martin S., Conclusions, Commutor case
          (May 1999).




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                          Page 3 of 34
The aim of PROTEE case studies is to develop procedures and indicators for the PROTEE
instrument. In addition to this, there are several assumptions about the PROTEE instrument
which the case studies can be used to test. Such assertions and assumptions include the
following. Innovation projects are different from other projects, for example, infrastructure
projects. This means that innovation projects cannot be assessed using instruments (for
example STEMM) designed for non-innovation projects. Rather, the assessment of innovation
projects requires a different instrument - an instrument which can measure their learning
curve - and no such instrument currently exists. PROTEE aims to develop such an instrument.
Only innovation projects will have a learning curve, therefore the PROTEE instrument will
only be useable on innovation projects.




2. THE PROTEE CASE STUDIES


The four retrospective case studies in PROTEE are in two Work Packages - Work Package 1
and Work Package 2A. The difference between the two Work Packages is that the case
studies in Work Package 1 do not have an industrial partner in the PROTEE Consortium,
whilst those in Work Package 2A do. The case studies without an industrial partner are on
Commutor and on the Europe Combined Terminals Delta/Sea-Land terminal (ECT). These
case studies were undertaken by an academic or analyst partner alone. 4 The case studies with
an industrial partner are on the Krupp Fast Handling System (KFHS) whose industrial partner
is Krupp Fordertechnik, and the European Transport and Trade Centre (ETTC) whose
industrial partner is the Zentrum fur Innovative Verkehrslosungen. The partners responsible
for each of the case studies are summarised in the Table below. It should be noted, however,
that more partners than just those responsible played a part in developing all of the PROTEE
case studies.

Table 2.1: Partners responsible for the PROTEE case studies

Case Study             Industrial partner                   Academic/Analyst partner
Commutor                                                    Technicatome 5
ECT                                                         Maastricht University
KFHS                   Krupp Fordertechnik                  Brunel University
ETTC                   ZIV                                  Maastricht University




4
    With respect to this case study, Technicatome functioned as an analyst partner rather than an industrial partner.
5
    Ibid.



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                                Page 4 of 34
2.1. INDUSTRIAL PARTNERS


The inclusion of industrial partners provided several benefits to the case studies. These
include:
a) The industrial partners are experts on their own innovation projects and provide access to
important source material for the case studies. In this way they are an efficient way of
collecting data on an innovation project.
b) The collaboration of both academic/analyst partners and industrial partners on a case study
provides the opportunity to model the key PROTEE relationship which is the relationship
between the Evaluator and the Innovator. It seems to us that a close cooperation with an
industrial partner (as in KFHS) comes closest to the situation of the „learning pact‟ between
innovator and evaluator, and would in retrospect have been preferable.
c) However, it should be noted that the role of „evaluator/innovator‟ changed depending on
whether the topic was the industrial innovation or the PROTEE innovation.
d) The presence of industrial partners in the Consortium helped to make the descriptions of
the PROTEE instrument itself risky. This was both through the critical feedback they provided
on individual case studies involving their own innovation project and in the Working Groups
where the case studies and the PROTEE instrument came under discussion.


2.2. RETROSPECTIVE CASE STUDIES


PROTEE is designed as a real-time management tool for innovation projects which are
ongoing in the present. The historical nature of some of the projects used as case studies
limited the extent to which they could be used to test PROTEE Indicators and procedures.
This was for two main reasons.

One reason is that a PROTEE evaluation is the outcome of comparing two redescriptions
made at different time points in the project‟s history. What you have in retrospective case
studies is a snapshot of the past taken from the present. It is a description from a single time
point and it is not possible to evaluate the learning curve of a project from a single time point.
This is the reason why case studies were carried out in sequence, rather than in parallel. It was
then decided to apply to Commutor a full analysis of the indicators developed through KFKS
and ECT at four key times during the life of the project. This was possible since the project
was terminated and the full documentation available. The modified indicators and procedures
will be tested on CIMP, the sole on-going case undertaken in the PROTEE project, see later.

The second reason is that it is extremely difficult, if not impossible, to make a judgment in the
present about what difference it would have made had certain decisions been taken, or not
been taken, in the past. This makes the evaluation of the retrospective thought experiments
that were attempted in some case studies difficult because both parties to the case study -


b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                 Page 5 of 34
Innovators/industrial partners and Evaluators/academic-analyst partners - have the benefit of
hindsight. In the case of the Innovators, hindsight might lead to a tendency to produce „Whig
histories‟ or „Just So‟ stories whereby the present is portrayed as following logically and
unavoidably from the past. In the case of the Evaluators, with the benefit of hindsight they
might see possibilities and risks which were simply not visible at the time the decisions were
taken. This risk is even greater when the project has effectively been terminated for several
years, as was the case for Commutor. However, the aim was not to study the well known
management problems of French state owned companies, it was to test the feasibility of using
the indicators for describing the dynamics of the project and to make a set of « thought
experiments » at given turning points to see what such descriptions could have brought to the
fore.




3. THE EVOLVING PROTEE INSTRUMENT



3.1. STEMM


The starting point for developing the PROTEE instrument was STEMM, a set of Indicators
and procedures developed for transport infrastructure projects. STEMM was available to the
Consortium as a set of documents produced by Technicatome in March 1998. 6 These were
the starting point of the PROTEE project, the aim of which was to arrive at a new set of
indicators relevant for innovative projects (and no longer for infrastructure development). The
first set of PROTEE Indicators was written in November 1998. 7
Therefore, most of the work on the first case studies was an evaluation of and adaptation of
STEMM as a stepping stone towards the development of PROTEE.
The set of STEMM Indicators from the March documentation comprised four Classes each
comprising three or four Indicators, most of which were further subdivided into two or more
Variables. For each Variable, a set of options, corresponding to possible features of the
project, were described in the documentation. Each such feature was given a value which was
colour-coded. The colours used in the coding were shades of red, orange and green, their
symbolism being acquired from the meanings of those colours in traffic lights.
The concept in STEMM is that the Evaluator (Administrator) should use the documentation
both as a framework for organising and evaluating features of the project. That is to say, he or
she should organise the various features of the project in relation to STEMM Variables so that
they can be colour coded. The result would be a visual guide to the state of the project at a

6
    Technicatome, St-cls-1.doc, St-cls-2.doc, St-cls-3.doc, St-cls-4.doc. (1 March 1998).
7
    Latour and Martin, PROTEE Interview Guide: Being a Presentation of PROTEE’s Principles and Indicators
           (18 November 1998), pp. 13-27.



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                              Page 6 of 34
particular point in time. A large number of red codings would be a warning signal for the
project at that time.
What STEMM had to commend it was clear guidance on how it was to be applied to a case
study or project. If anything, it was over comprehensive and over prescriptive. In STEMM,
project evaluation was a task undertaken by the Evaluator (Administrator) using
predetermined values. Through the experience of the case studies, the concept of PROTEE
started to diverge from STEMM procedures in at least three respects. Firstly, there was a
move away from comprehensive use of the STEMM Indicators towards their intuitive use;
secondly, in PROTEE project evaluation was conceptualised as a joint enterprise, involving
both an Evaluator and an Administrator; and thirdly, PROTEE moved away from the
provision of a priori evaluation of concrete features of a project to the contextualised
evaluation of the learning curve of a project.
These were the lessons that had been learned by September as a result of testing and adapting
STEMM on the case studies. These lessons informed the drafting of the first set of PROTEE
Indicators and Procedures in November 1998 in the Interview Guide. 8


3.2. THE PROTEE INTERVIEW GUIDE


The writing of the first draft of the PROTEE Interview Guide was certainly a milestone in the
project. However, its title of Interview Guide was something of a misnomer. It was more of a
theoretical document than a practical guide to undertaking PROTEE interviews. It comprised
a set of principles, or propositions, and the first draft of a set of Indicators designed to
describe the learning curve of an innovation project. Whilst the propositions, or principles, in
the Guide captured the essence of the developing PROTEE approach, it was far from a
practical guide for case studies or projects.
Feedback from the case studies was that it was not user-friendly, and that it was not useable as
a practical guide in the form it was drafted in November 1998.
Lack of clarity in the Interview Guide gave rise to specific comments from the case studies
which are discussed in the Sections below, especially Section 4.


3.3. DEVELOPING THE INDICATORS: PROTEE IN OTHER WORDS


With regard to the Indicators themselves, feedback was that the way they are presented in the
Interview Guide makes them hard to apply to innovation projects and case studies. It was
suggested that what was needed was a set of questions which mediated between the type of
question one might ask in an interview situation and the sort of information required for the



8
    Latour and Martin, op cit..



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                               Page 7 of 34
Indicator. The PROTEE Indicators needed to be expressed in other words so that they could
be used in an interactive interview situation without becoming an obstacle. 9

For Class I (Realisability) Indicators, which are about the Innovator‟s description of the
realisability of the project, such questions would be:
        What is the project about?
        What is the project going to become?
        How is the project going to progress?
        What is the progress of the project dependent on?
        Why is the project the way it is?
        What could change the project?
        What are the uncertainties of the project?
        How many ways ahead are there?
With respect to Class I Indicators, it was observed that care has to be taken because the quality
of the Innovator‟s description is directly dependent on the quality of the questions asked by
the Evaluators. 10 The lesson here is that it is important that the Evaluator is PROTEE-trained.
It also points to another feature of PROTEE which is that the interaction between the
Innovator and the Evaluator is central to the PROTEE evaluation process. Indeed, one
observation is that the roles of Innovator and Evaluator interchange (see below).

For Class II (Negotiability) Indicators, the Evaluator might ask questions such as:
        Who are your opponents/customers/allies?
        Why is there opposition/controversy?
        Could you change your project to enrol your opponents/potential customers?
        What are the pros and cons of changing your project for the sake of negotiation?
It was also suggested that it should be made clear that this is the Class that analyses whether
the Innovator has considered strategies to exclude competitors, for example, as was suggested
in respect of the Glossary in the KFHS case study.

The ECT case study does not have an industrial partner in PROTEE (Table 2.1). However,
Maastricht University organised a Round Table discussion of the ECT case study with some
of the people interviewed. Feedback from this Round Table was that some participants had
difficulty in understanding the difference between Class II Indicators (Negotiability) and Class
III Indicators (Falsifiability). 11
Class III Indicators are about disinterestedness. These Indicators were redrafted in the light of
feedback from the European Commission on an early draft. that the redraft attempted to
provide PROTEE Evaluators with an in-built safeguard against Innovators who tried to
manipulate the PROTEE process by producing false project descriptions in order to achieve

9
    See McNally, R. „PROTEE Indicators in other words‟, (9 Dec 1998).
10
     Hommels op cit
11
     Peters, op cit



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                Page 8 of 34
good PROTEE evaluations. Questions an Evaluator might ask in order to analyse the project‟s
falsifiability might be:
        How are decisions and judgements about this project made?
        What are they based on?
        Who makes them?
        How many interests are represented among the decision-makers?
        What interests are excluded from the decision-makers?


3.4. THE PROJECT DESCRIPTION SUMMARY SHEET AND THE PROJECT
LEARNING CURVE SUMMARY SHEET


In addition to the material in the Interview Guide, there was demand for a manual giving clear
guidance and a standardised set of rules to produce descriptions of the projects.
One move towards this was the reformulation of the Indicators themselves in three different
ways so that they could be used as part of the PROTEE methodology. 12 These three ways
were:
 as questions
 as part of a Project Description Summary Sheet;
 as part of a Project Learning Curve Summary Sheet.
These new formulations of the Indicators are in Appendix 1 to this Report.
They were subsequently tested by Maastricht at the Round Tables with the European
Commission.




4. THE PROTEE PROCESS


4.1. PROBLEMS ENCOUNTERED

The lack of clarity and guidance in the PROTEE Interview Guide produced a number of
difficulties in the execution of some of the case studies. Examples of such difficulties are:
1) Proposition 6 of the PROTEE Interview Guide states that at time zero, when the learning
pact begins, the Innovator should be able to describe the world in his or her own terms, and
that the evaluator should do nothing more but listen. This proposition posed problems for
several case studies. With regard to the ECT case study which had no industrial partner,
Maastricht pointed out how difficult it was for an academic partner to simulate time zero by


12
     McNally, R. „Comments on the Interview Guide‟ (20 November); McNally, R. „Further comments on the
          Interview Guide‟ (30 November); McNally, R. „PROTEE Methodology: Project description summary
          sheet and project learning curve summary sheet‟, Maastricht Working Group, 14 January 1999.



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                          Page 9 of 34
writing a neutral description of a project, and commented that providing the description and
analysing it at the same time proved to be too difficult.
2) Unlike the ECT case study, the ETTC case study had an industrial partner. Nonetheless,
with respect to the ETTC case study Maastricht commented that it was difficult for the
Evaluator to elicit a risky description of the project at the same time as allowing the Innovator
to talk about the project in his or her own words.
3) ZIV noted that making a PROTEE description required a huge effort on the part of the
both the Innovator and the Evaluator. The Evaluator had to learn a vast amount about the
project and the Innovator had to master the STEMM or PROTEE principles or procedures.
One factor which contributes to the difficulties noted above is the attempt to execute several
different steps in the PROTEE process at once.
Another factor is the attempt, of necessity in the ETC case study, of trying to perform the roles
of both the Evaluator and the Innovator simultaneously.
The KFHS developed a method for operationalising the PROTEE Interview Guide which
avoided both of these two factors. This is summarised in the 5-Step Plan below (section 4.2).


4.2. THE 5-STEP PLAN


The Interview Guide conceptualises the Innovator and the Evaluator as having separate, but
complementary competences. The Innovator is an expert on his or her own project. The
Evaluator is an expert in developing PROTEE type analysis and evaluation of an innovation
project. The KFHS case observed this division of labour by exploiting the fact that the work
was a collaboration between an academic and an industrial partner to recreate the roles of
Innovator and Evaluator within the case study. In the KFHS case study, the PROTEE process
was conceptualised as comprising five distinct and non-overlapping steps, or moments. The
five steps are described in Box 4.1. The five steps incorporate this complementary division of
labour and competences.

Maastricht made suggestions in accordance with Brunel‟s 5-Step Plan. On the basis of the
ECT study, Maastricht pointed out that since the descriptions at t0 and t1 are the key basic
material of the PROTEE analysis, the production of these accounts deserves much attention.
There is a tension between the requested open and risky character of the description (a
description of a world as a heterogeneous ensemble of actors, entities and elements in the
innovators terms) and the highly standardised evaluation sheets with which the descriptions
are assessed. To overcome this tension, it is suggested that the production of the description is
completed in two phases. In the first phase, the Innovator gives an account of the project „in
his own terms‟ (when the first time point is being chosen at the beginning of a project, the
technical proposal might be a suitable document to work from). In the second phase (which
might in practice be on the same day) the Innovator and the Evaluator work on a more




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                Page 10 of 34
structured description in which elements that are to be judged by the indicators are already
present. This structured second phase description should be negotiated and agreed upon.



Box 1. The PROTEE 5-Step Plan


1. The Innovator’s Story. The Innovator tells a story to the Evaluator. This is contained in the
documentation and other material supplied to the Evaluator prior to a project meeting. The subsequent
procedures, described in points 2-5 below, take place at the project meeting itself.

2. The Consensus Story. The Evaluator retells the story of the project to the Innovator in the form of
a summary - a chronology of events. Through this process they agree on a Consensus Story - an
agreement about what has happened on the project since it started, or the last meeting.

3. The Socio-Techno-Therapeutic Dialogue. The Evaluator and the Innovator enter into an analytic
dialogue structured by questions formulated by the Evaluator to encourage the Innovator to make
risky descriptions of the project. This dialogue is designed to lead to a PROTEE redescription of the
project.

4. The Redescription. The Evaluators and Innovators each make a record of the PROTEE
redescription. The PROTEE redescription would be a record of the quality and quantity of the
descriptions of the project made within the framework of the PROTEE Indicators. The Innovator and
the Evaluator do not have to agree on the project‟s redescription, but their respective redescriptions
should address the same points. At the Working Group in Maastricht we proposed a method for
making the PROTEE redescription which involves the completion of Project Description Summary
Sheets. 13 The Sheets would record a summary of how the parties redescribed the project at the time of
the meeting with respect to the quality and quantity of its descriptions.

5. The Evaluation. The Evaluation is the outcome of the comparison of project redescriptions made
at two consecutive meetings. Clearly, it is not possible to make an Evaluation at the first meeting. At
the second and subsequent project meetings, however, the Innovator can compare his two
redescriptions and the Evaluator can compare his.14 What is important for the Evaluation are the
differences between the two sets of redescriptions.




13
     McNally, R. PROTEE Methodology: Project Description Summary Sheet and Project Learning Curve
          Summary Sheet, Presentation at the PROTEE Working Group, Maastricht, 14 January 1999.
14
     We have designed a form, or sheet, for recording this Evaluation. See „PROTEE Project Learning Curve
          Summary Sheet‟, ibid.



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                       Page 11 of 34
4.3. TIME ‘ZERO’


Maastricht found it difficult for the Evaluator to elicit a risky description of the project at the
same time as allowing the Innovator to talk about the project in his or her own words. In the 5-
Step Plan this problem does not arise because these two tasks are separated. The Innovator‟s
story is Step 1, whilst the risky description is not attempted until the Step 3.

In Step 1 the Innovator describes the project from his own point of view and the Evaluator just
listens. In the Interview Guide, this is the situation envisaged at the first meeting between the
Innovator and the Evaluator at time zero in a PROTEE project. However, in the five-step plan
it is also envisaged that each PROTEE project meeting will be preceded by this moment.
The aim in Step 2 is to establish a common story before commencing the analysis. This Step,
which involves the Evaluator retelling the story of the innovation project before commencing
the analytic dialogue, was strongly endorsed in the KFHS case study. Indeed, the feedback
from the industrial partner was that in order to assist the orientation and further discussion by
the various people involved in a project such structuring is mandatory for analysing projects
retrospectively. 15

In preparation for Step 2 of the 5-Step Plan, the Evaluator summarises the Innovator‟s Story in
the form of a chronology of events. The experience of the KFHS case study was that it was
possible for the industrial and academic partners to agree on a consensus story starting from a
chronology prepared and presented by the academic partner. As mentioned above, Maastricht
experienced difficulty in writing a neutral description. However, neutrality is, of course, in
the eye of the beholder and is not what is aimed for in Step 2. Step 2 aims for consensus rather
than for neutrality. It may be that in order to achieve this aim, formatting the project summary
as a chronology of events is relatively less controversial than other narrative formats.
This last comment is also relevant to feedback from Maastricht (ECT) about the suggested
lack of useful previous knowledge during the innovation process. For pedagogical purposes it
makes sense to highlight this „state of total ignorance‟ on the possible outcome of the project,
but we should not overdo this. Project managers and engineers do know quite a lot already at
the beginning of a project. The question is: how do we operationalise this knowledge,
pertaining both to the technology and to the context, to the project management and to doing
research? Almost all the ECT-interviewees underlined the fact that the DSL-innovation did
not start at a point zero in terms of knowledge and experience. On the contrary, on various
places and moments during the process, known technology proved to be of decisive
importance.




15
     Sondermann, U. (January 1999), op cit



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                 Page 12 of 34
This feedback is useful because it reminds us that time zero is just a convention. As was
explained in the KFHS case study, it is a point mutually agreed upon as the basis for dialogue.
The Evaluator selects a time zero on the basis of material supplied by the Innovator. At the
beginning of the project meeting they then clarify that this is an agreed time zero for the
purposes of PROTEE.
There will of course always be prior knowledge, competences and technology. Whether they
are brought to the project by the Evaluator or the Innovator, these are brought into the
description via the dialogue. What is redescribed at the end of the dialogue is what is known
about the project at time zero in the light of both what was known at the start of the dialogue
and what was revealed through it.
Commutor was developed without any interaction with its industrial promoters, apart from
traditional interviews. The a posteriori redescription thus ignores the problem of a shared
description between evaluator and innovator at T0. The question is however whether an
evaluation is needed at this first encounter or whether it is mainly focused on the project
description, and on the questions it is going to explore. This questions the necessity of having
two summary sheets, an “absolute” one evaluating for instance the richness per se of the
project future world, and a “relative” one focused on the transformations between two
interactions of the hoped future world.

It is only from Step 3 onwards of the 5-Step Plan that Indicators and project story are put
together. These Steps are a joint exercise involving the Innovator and the Evaluator. This
collaboration addresses ZIV‟s concerns which the amount work required of the Innovator and
the Evaluator in mastering each other‟s expertise and competences. Under the 5-Point Plan,
the application of the Indicators to the project is a collaboration in which the Innovator is only
required to be an expert on the project and the Evaluator is only required to be an expert on
PROTEE. Steps 3-5 - the processes of project analysis, redescription and evaluation - employ
a division of labour whereby the Innovator and the Evaluator each benefit from the other‟s
skills and knowledge without having to duplicate it for themselves.
In the 5-Step Plan, analysis, redescription and evaluation are three separate activities. Case
study lessons on these three Steps are included in the sections below.




4.4. STRUCTURING THE PROTEE INTERVIEW


PROTEE project analysis is the result of applying Indicators to the project material.
Through the case studies, a methodology for PROTEE analysis has been developed. In
PROTEE, this analysis is an interaction between the Innovator and the Evaluator rather than
an analysis undertaken by either party alone. Partly for this reason, step 3 emphasises the
dialogic interaction, in the manner of “therapy” between Innovator and Evaluator. It takes the



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                 Page 13 of 34
form of an interview in which the Evaluator asks the Innovator questions about the project.
The aim of PROTEE project analysis is to move towards a PROTEE redescription of the
product.
The question is, what should guide the way in which the interview is structured? Maastricht
(ETTC) found that there was insufficient guidance in the Interview Guide on this point for
them to use it, and so they used STEMM instead.
The ETTC and the KFHS case studies structured their interviews/dialogues through an
internalised knowledge of STEMM and selective use of the STEMM Indicators. Both these
case studies found STEMM useful in an adapted form, and Brunel suggested that the STEMM
Indicators, which cover more substantive issues than the PROTEE Indicators, could be a
useful adjunct to PROTEE. This is not to suggest that the STEMM Indicators should measure
the learning curve of the project. The STEMM Indicators would be used at the analytic
interview stage to transform the Innovator‟s Story to a riskier description of the project.
Subsequently, the PROTEE Indicators would be applied to this riskier description resulting in
the PROTEE redescription of the project.
The ETTC case study provided a limited test of this suggestion. Having used the STEMM
Indicators in the analytic interviews, Maastricht then subsequently attempted to apply the
PROTEE Indicators to generate a PROTEE redescription. Maastricht comments that, although
the redescription required some adjustment because the Indicators were not fine-tuned to the
PROTEE Indicators, used selectively, the STEMM Indicators were useful as a guideline for
the dimensions relevant for the production of a PROTEE- like redescription of the project.
This could mean that the STEMM Indicators are a useful adjunct to PROTEE. More likely,
however, it could mean that STEMM merely provided a necessary stop gap for the early case
studies whilst awaiting the first drafting of the PROTEE instrument.
Having experienced the STEMM approach, those involved in the ETCC project seem to
favour drafting a new set of questions specifically for PROTEE, rather like the „PROTEE in
other words‟ above. ZIV proposed the development for every PROTEE class of a set of
questions which can be asked by the Evaluator to each Innovator independent from each other
class. Based on the answers descriptions of the project can be worked out which are helpful
for the evaluation. ETTC (Hommels) reported the need for a standardised set of rules to
produce descriptions of projects.
However, feedback from the ECT project, which did not use STEMM, seemed to go in the
opposite direction. Maastricht (ECT) commented: „It is clear that the description should be
prestructured along the lines of the Indicators, but not too much since the whole operation
would then become circular (a description based on certain Indicators is judged in terms of
these Indicators)‟. 16
We shall see in the next chapter dealing with CIMP that a description solely based upon
PROTEE indicators was tested and judged powerful for initiating a description between the
project promoter and the PROTEE “investigator”.


16
     Peters, P. op. cit.



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                             Page 14 of 34
4.5. CHOOSING TIME POINTS


Maastricht (ECT) stressed the problem of choosing time points. The core of PROTEE
methodology is the comparison between two descriptions produced at two different points in
time. Assessing this differential might lead to some practical problems, some of which were
brought up during the Round Table. One participant asked how the two time points are
actually chosen? What should be the time interval? If this is not arbitrary (as was suggested in
the meeting held in Mondragon), what should then be the criteria for picking the two time
points?
For Commutor, turning points were chosen because they corresponded to major decisions to
be taken. Deciding or not to build a prototype provides for a before and an after which are
radically different. This was thus never a problem to select them. And PROTEE, aiming at
being a management tool for the monitoring of live projects, will have to be inserted into the
time frame of the wider decision making process in which the projects will be embedded. For
example, in „real‟ projects, interim reports are due every six months or so, at dates determined
by the starting date of the contract. These reports would be the Innovator‟s Story of the project
at the date of the interim report, covering progress between the present and the previous
report. The reporting period would provide the time points.




4.6. THE PROTEE REDESCRIPTION


An outstanding problem is that a redescription judged favourable within one frame may be
regarded as unfavourable within another. For example, standardisation and globalisation may
mean more transport, but may relate only problematically to different criteria such as the
project‟s contribution to environmental and sustainable growth? Thus, a project thought to be
innovative in one sense (standardisation) might be judged conservative in another context
(sustainability).
A further, more complicated issue is that there may be hidden or unexpected virtues in
imprecise descriptions. For example, a definition of the situation which is wilfully ambiguous
may have the distinct advantage of offering protagonists the flexibility subsequently to
connect to (more and) different networks. In this circumstance, the innovativeness of the
project might be overlooked if too much weight is given to the precision of the initial
description. This however should not be a major issue since PROTEE is about learning and
the quality of the learning curve. Whatever initial description, it is the differential between
descriptions that will support the joint agreement about future relevant actions.




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                Page 15 of 34
4.7. THE PROTEE EVALUATION


PROTEE aims to detect the learning curve of innovation projects and advocates rewarding
projects with a steep learning curve.
PROTEE sees stabilisation and learning potential as being inversely related. At the beginning
of an innovation project, there is a lot of flexibility but not very much is known and so the
learning potential is high. At the end of the project, the innovation is stabilised, but the
learning potential is low. From this it follows that as the innovation is stabilised, the amount
of learning detected by the PROTEE instrument will decline. It points to a central issue: when
to abandon PROTEE as a monitoring tool to turn back to the traditional quantitative methods
of the evaluator‟s institution?
Some of the Innovators commented on this aspect of PROTEE. Consider PROTEE‟s
valorisation of branched paths, number of alternatives, and maintenance of flexibility, for
example. In private enterprises, what is valued is a focused concentration and a timely
realisation. Maintaining a large amount of openness and flexibility until the very end of a
development runs the risk of loosing track and not meeting the concrete objective. More
concretely, starting with a High Rack system and ending with a gantry crane was not an
acceptable outcome or a large plant making company, even if the process generates a lot of
learning and collaboration which satisfies public promoters (KFHS).
With regard to the point of stabilising the project, it is not good to provide risky,
heterogeneous descriptions in all phases of the project. There is a point where something has
to be built. The question is how to get from ever more risky descriptions to a more stabilised
structure without loosing the openness of the initial phase? Learning can be very important in
one phase, but a threat in another (ECT).
During the Round Table, A similar point was put forward when one participant stressed the
fact that learning and risk are good for some time, but there has to be a moment when the
project stabilises. In his view, PROTEE was not successful in identifying that point because it
focuses mainly on the learning part of an innovation process (ECT).
These elements have been well illustrated by the Commutor case study, where the issue was
not to maintain all variants open at any time, but to weigh the pros and cons of closing
variants, and entering into a scaling up process through prototypes and demonstrations. What
PROTEE should help at is valuing the relevance of the choices made by comparing
information acquired to capabilities mobilised and risks taken (new opponents, etc).

Can the instrument only be reflexive on the learning curve, or does it also have prognostic
possibilities? In other words, can it be used to suggest a certain strategy? This might be a
spontaneous result from the production of the description. This urges the Innovator and the
Evaluator to identify elements in the process that would otherwise have gone unnoticed
(ECT). This brings to the fore one of the “hard” tests for a real time case: for PROTEE to be
useful to managers (of project or of the evaluating institution), the evaluation of the learning
curve must be of help in identifying the next explorations to undertake.



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                               Page 16 of 34
4.8. THE ROLES OF THE INNOVATOR AND THE EVALUATOR


ECT pointed out that within the PROTEE methodology the learning pact between Innovator
and Evaluator is essential. During the Round Table, one participant underlined the fact that
Innovator and Evaluator should, at least in theory, be able to change roles. Both Innovator and
Evaluator are subject to a „double inclusion‟ within the networks they are assessing. This
means that they are acting within and without their respective networks at the same time. To
be successful, the Innovator in some way or another has to relate to the network of the
Evaluator and vice versa.
The key implication is that the traditional roles of the Innovator - whose fate depends on the
judgement of the Evaluator - and the Evaluator - who stands in a hierarchic relationship to the
Innovator - are not ones that match the requirements of a PROTEE assessment. In this double
inclusion of Innovator and Evaluator, the „willingness to be influenced‟ on both sides is more
important than the hierarchic power structures.
The fact of double inclusion is clear. After all, PROTEE is a learning pact. The PROTEE
redescription and evaluation are collaborative activities involving the Evaluator and the
Innovator. But, on the basis of experience of KFHS case study, this does not have to entail
them changing roles. They have different competences. There is a division of labour based on
this difference in competences.
Also, whilst the process of deriving the PROTEE evaluation is „therapeutic‟ for the Innovator,
in the way described in the KFHS case study, the relationship is hierarchical. The Evaluator
does have a different role to the Innovator, and the fate of the Innovator does depend on the
opinion of the Evaluator.


4.9. PROTEE AS SOCIO-TECHNO-THERAPY


Looking back over the methods used for the KFHS case study it is possible to conceptualise it
in terms of three moments in the transfer and transformation of information. During the first
moment, information was transferred from Krupp Fordertechnik to Brunel University and to
other members of the Consortium as Krupp Fordertechnik told the story of the KFHS project
in their own words. During the second moment, the roles were reversed. Brunel University
used the information provided by Krupp Fordertechnik to retell the story, in the form of a
chronology, resulting in a consensus version of the chronology of events in the history of the
project. In the third moment there was analytic dialogue.
These three moments have resonance with moments in the interaction between client and
therapist in psychotherapy and other forms of „talking cure‟. In the first moment of the
interaction, the client tells the therapist about his or her life, a subject upon which he or she is
the expert. During the second moment the therapist retells parts of the client‟s story back to


b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                  Page 17 of 34
the client, perhaps asking questions for clarification, and allowing the client to correct
inaccuracies and misapprehensions. This second moment prepares the ground for the third
moment, the moment of dialogue. Here the therapist asks the client questions about what the
client has told him, questions that are based on the therapist‟s expert knowledge and
experience, questions which might bring new perspectives, unexplored explanations, and
unsuspected dimensions on the past, present and future.
In PROTEE, as in therapy, two types of actors are involved, each possessing complementary
competences. The client/Innovator is an expert on his or her own situation; and the
therapist/Evaluator has expertise in the use of a set of analytic tools. In both cases the
therapeutic value is in the interaction.
It is important to clarify the concept of „therapy‟ in the context of PROTEE. PROTEE therapy
has a lot in common with marriage guidance counselling or relationship counselling. Such
counselling or therapy does not guarantee the survival of the relationship. Rather, the aim with
such counselling or therapy is that if the relationship is a „white elephant‟, the parties involved
will realise this sooner rather than later, and that couples who receive such therapy will have a
better understanding of what is happening to them and will be better equipped to manage
future relationships. It is this concept of therapy which invites the conceptualisation of
PROTEE, particularly the analytic dialogue, as socio-techno-therapy (STT).
The function of the PROTEE dialogue is socio-techno-therapeutic: aided by questions from
the Innovator, its purpose is to encourage the Innovators to enrich their description of the
project - to entertain riskier aspects of the project in their descriptions and to imagine other
possibilities. This is the idea. However, feedback from some project managers is that making
risky descriptions might not be worth the risk, as described below.




5. THE PROTEE CONCEPT OF INNOVATION


5.1. SOFTWARE AND HARDWARE INNOVATION


The original project proposal aimed to assess if PROTEE was suitable for both software and
hardware innovation projects and to identify any differences between the use of PROTEE on
these two types of project.
However, ECT in particular, reported they did not make a distinction between software and
hardware components in the project. It was difficult to make this distinction. The difference
between hardware innovation projects and software innovation projects that had been made in
the initial PROTEE proposal, proved to be not very useful in the context of the ECT-DSL. In
the case of the development of the AGV‟s and ASC‟s, the engineering part of the hardware
and the software development went inseparably together. In the case of the ECT Delta/Sea-
Land terminal, the joint tuning of the software and the hardware innovations appeared to be
one of the most difficult tasks of the project to achieve.


b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                 Page 18 of 34
ECT also reported that it was unclear whether innovations in terms of a changing labour
organisation – seen by some of the interviewees as the most important and difficult innovation
– should be categorised under „hardware‟ or „software‟ or both.
All the PROTEE projects were a combination of both software and hardware innovations.
Indeed, each of the projects concerned an innovative system, which combined several
innovations, rather than a single innovation.


5.2. THE UNIT OF ANALYSIS


The multiplicity of innovations raises a key question about the unit of analysis for PROTEE.
In a project in which there are several different innovations, could (or should) there be several
different learning curves?
Some of the interviewees in ECT said it was possible to identify many learning curves during
a project with the complexity of the ECT-DSL. What seems to be a useful learning curve for
one actor in the project, might be regarded as a draw back for another. Rijsenbrij (project
leader for the building of DSL) highlighted the discrepancy between the need to build a
functioning terminal before the deadline on the one hand and the perceived need for a more
flexible automation concept among the software developers on the other. The latter option
would take more time and money. The actual innovation was a compromise. The criteria for
success are not only the amount of innovation that goes into the new terminal and the quality
of the learning curve; but also the effectiveness of the innovation process in terms of meeting
the specifications (including the deadline) that were laid down in the technical
proposal/annex.
It became clear that the development of the software took much more time and money then
had been expected. Seen from the PROTEE point of view, one might argue that there was a
better learning curve in the software innovation trajectory because there was less proven
technology to start with and therefore the differential between any t0 and t1 was bigger.
The same general question arose with respect to the ETTC case study. What should the unit of
analysis be when applying the PROTEE instrument to an innovation project? Innovation
projects are complex, and that means that PROTEE could be applied to the whole, but also to
sub-projects. Maastricht/ZIV decided to describe the ETTC project as a whole, and then
focused on one particular part of it, the terminal technology.
One possibility is that the effects of this are more exaggerated when considering a case study
problem than they might be with a „real‟ problem. When PROTEE is used in „real life‟ the
unit of analysis may be clearer. It is, simply, whatever the Innovator seeks support for from
the Evaluator.
This pragmatic argument is, of course, complicated by the point made by Maastricht (ETC)
about the learning curve. The result of the comparison of two descriptions at two time points
is a statement about the learning curve of a project. One of the problems already mentioned in
Maastricht‟s comments on the Interview Guide relates to this learning curve. Does it make



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                Page 19 of 34
sense to talk about one learning curve? Are there not as many learning curves to be
distinguished as there are, for instance, relevant social groups? Even if the user of the
instrument decides to single out one learning curve, an explicit choice at the start of the
assessment process is probably more in line with our own view of the innovation process.
Another way to phrase the problem, as is shown in the Commutor case, is to consider each
indicator or groups of indicators as a learning process per se. The issue is then not addition
which would be meaningless and turn back to another form of quantitative approach, but
aggregation. This poses a set of questions about class 4 and the conditions under which a
difference can be made between a “hopeful monster and a “white elephant”. This will be
addressed again in the lessons derived from the CIMP case study.


5.3. TECHNOLOGICAL ESSENTIALISM?


Individual technological innovations are parts of systems that may contain other technological
innovations. The unit of PROTEE analysis may well be the entire system - for example the
KFHS - rather than individual components of that system. Thus, for example, a key lesson
from the KFHS case study is that PROTEE considered the innovation to include both the
technological side and the market side of the KFHS.
In this case study, PROTEE seemed to uphold a definition of innovation which was that a new
technology is not an innovation until it is introduced into commercial or social activity. In this
case study it is clear that PROTEE considered that the technology is only part of the
innovation.
However, does there have to be a technological component at all? One definition of
innovation is: “The first introduction of a new product, process or system into the ordinary
commercial or social activity of a country”.
Can a new system, whose innovativeness is in the combination of elements, be an innovation
if introduced into commercial or social activity, if none of those elements is technologically
novel? Is PROTEE only applicable to innovative systems with a new technology or is it also
applicable to innovative systems without a new technology?
The case study which poses this question very clearly is the ETTC case study. Both ZIV and
Maastricht argue that the ETTC is an innovation project.

The first argument they make is that a system which requires an innovative combination of
elements but which does not include a new technology is a risky project in the PROTEE
sense.
PROTEE provides an instrument for the evaluation of innovative projects (no limited only to
the field of freight transport). It characterises innovation by the absence of certainty, the
impossibility to know and to describe the world in which the project would become viable.
Innovations in PROTEE language are “hopeful monsters”: “they are by necessity „monsters‟




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                 Page 20 of 34
since they do not resemble inhabitants of the present state of affairs”. To what extent can the
PROTEE-instrument be usefully applied to projects like ETTC?
The ETTC-project is not concerned with innovation in a technological sense. The protagonists
have deliberately chosen a conventional, well-known, widely used transhipment system. The
reason was that this would be a less risky choice: it is easier to implement, investors would be
easier to convince. But this is the opposite of what PROTEE sees as innovation, as
characterised by not knowing and taking risks. If a less risky choice is made, less will be
learnt and thus PROTEE will be less capable of improving the learning curve. On the other
hand, PROTEE explicitly not only focuses on the ”technical” part of projects. It can also be an
innovation ”about usage, control, accounting, practice, law, as well as pieces of machinery or
composition of existing machinery”.
According to the innovators themselves, the innovative part of the project consists of
connecting different functions with one another: normal freight transport for the Oder region,
border freight transport centre (FTC) for traffic with Eastern Europe, integration of a centre
for combined intermodal transport, planned as gateway and supplemented with a conception
for city logistics. Others point out that while there has not been so much risk in the choice for
the terminal technology, it is still uncertain how the city, the region and the companies using
the system will react after the FTC has been built. Moreover, the precise form of the mobile
system and crane technology has not been yet decided on. There is also innovation going on at
that level, for example in the field of automated portal cranes.
There is always a lot of uncertainty in complex projects like these on different levels. In this
case, uncertainty does not relate so much to the technology chosen, but more to economic
aspects, the support they will have from citizens, policy makers and companies, the growth or
decline of freight transport etc. Thus PROTEE might be well applied to projects such as the
ETTC.

The second argument is as follows: PROTEE is only supposed to be applicable to „high risk‟
projects where there is a need to learn about the nature of the risks in order to steer the project
in the most successful way possible but without of course guaranteeing success. If PROTEE
„works‟ on the ETTC project, then the ETTC project is, by definition, an innovation project.
In this way, PROTEE becomes a criteria of innovation, rather than just an indicator of it.
In PROTEE only such projects can be innovative for which it is possible to follow a learning
trajectory. With that PROTEE tries to separate projects which are not able to cope with a
learning procedure and which are based on traditional technics [technology]. These projects
avoid risks but they do not result in progress in the sense of societal development.
It could be shown and proven, that PROTEE‟s methodology is also applicable for evaluations
on systems with a high complexity and integrated approach. The ETTC Frankfurt (Oder)
should be grouped in this hierarchy in the last category of the highly complex technical
systems.
PROTEE works on the ETTC: ZIV sees immediate benefit of using PROTEE: With
PROTEE‟s knowledge the present work can be done more efficiently. For example,



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                 Page 21 of 34
understanding and decision making in the field of the know-how of transhipment technics,
negotiations with pro and contra representatives all stand to be enhanced (ZIV).

The suggestion, therefore, from the ETTC/ZIV case study, is that PROTEE methodology
should contain a defined discussion with the term “innovation” that does not limit it to
technologically based innovations. This will open up applications of PROTEE to complex
technical systems. A problem with this, however, is the difficulty of evaluating PROTEE on
the basis of the ETTC case study, because PROTEE was not really applied to it. Rather, the
analysis was undertaken using STEMM indicators. It was retrospective and therefore
judgments about what difference PROTEE would have made in the past were problematic
owing to the unavoidable influence of hindsight. In addition, there was only one time point,
which makes the evaluation of the learning curve very difficult.


5.4. INNOVATION VERSUS DEVELOPMENT


The Innovators on the ECT project made an important distinction between innovation projects
and development projects. It was thus said that there are two kinds of projects. Some are true
innovation projects in that they are indeed aimed at some learning curve and thus subject to
PROTEE evaluation. However, other projects might be explicitly and consciously formulated
in terms of the final result and thus not assessable by the PROTEE instrument.

During the Round Table, it was suggested that the ECT-case was more a development rather
than a research project. In other words, there was very little open and risky research, because it
had been decided beforehand that the terminal would have to be built, at whatever cost, thus
independent of any learning curve and without an alternative (of course there was the
alternative of the standard terminal, but that option was politically ruled out). However, within
the Delta terminal project, several sub-projects would have been PROTEE-assessable: the
AGV, the ASC, the operating systems, etc. This relates to the more or less implicit definition
of an innovation within PROTEE, where a high quality learning curve suggests an innovative
project. It might be interesting to try to differentiate between the research-elements and
development-elements present in any innovation project (ECT). This question will be further
addressed with the lessons about CIMP.




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                 Page 22 of 34
6. WHAT THE INNOVATORS THINK


6.1. ‘WHAT DIFFERENCE WOULD PROTEE HAVE MADE?’


KFHS reported that it was interesting and pleasant to listen to the history of the KFHS project,
but they did not think the analysis revealed any major additional aspects. The decisions taken
would have been similar, they said, if they had been PROTEE-equipped.

ECT asked several interviewees if having the PROTEE-instrument would have made a
difference at the time. They referred to more traditional techniques of quality control that had
been used (Besselink and Holtz), but argued that the specificity of possible future failure
could of course not be detected by these instruments. Without exception, all participants at the
Round Table were enthusiastic about the possibilities of using PROTEE as a real time
monitoring instrument.

ETTC reported that they did in any case use a partial PROTEE like approach. Basically,
PROTEE methodology was taken into account in developing the ETTC. This did not happen
unconsciously, but target-oriented, derived from logical connections and from the
development process itself. Essential items of PROTEE working method had been applied to
the review of planning development phases of the ETTC (identification of targets and
problems (risk) - learning process on local, state-political, etc…). The argument was
decisively influenced by the high demand for information that exists in the case of evaluating
freight transportation centres. But if they had used a total PROTEE approach, the promoters
said, a complete knowledge of PROTEE methodology at the start of the ETTC project in 1994
would have had four main advantages:
1. The internal project structure would have been noted more conclusively and more
    deliberately. Necessary tasks and decisions could have been derived earlier. The
    development process would have become shorter and become more continuous. Processes
    can be designed more innovative through a consistent application of the PROTEE
    methodology. This supports a shorter planning period. In the case of ETTC the assessment
    process between Innovator and Evaluator took more than 5 planning years until the release
    of financial means. The process could have been reduced to a period of 3 years if the
    criteria for release had been sooner and better satisfied.
2. The project would already have included in its basic concept alternatives to development
    as well as a gradual extension of the ETTC. Alternatives and the modular extension were
    taken into account only as a result of the assessment.
3. The description of the project would have made it possible openly to represent risks from
    the start, thereby shortening the process of decision-making. In the ETTC case, risks were
    recognised and submitted to the Innovator for solution by the Evaluators (ETTC/ZIV).




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                               Page 23 of 34
ZIV thus sees immediate benefits from using PROTEE. With knowledge of PROTEE their
present work could be done more efficiently, particularly as regards understanding and
decision making in the field of transhipment technicology and in negotiations with supporters
and opponents.


6.2. WILL THE INNOVATORS USE PROTEE?


As we indicated in section 4.9, the concept of „therapy‟ is at the heart of PROTEE. Through
interaction between Innovator and Evaluator, construed as a form of socio-techno-therapy
(STT), Innovators are encouraged to enrich their description of the project - to entertain riskier
aspects of the project in their descriptions and to imagine other possibilities. It is evident that
the success of the therapy depends crucially on the interaction which, just as in any form of
counselling, depends in turn on the relation between Innovator and Evaluator.
The advantage of PROTEE for the Evaluator is that he/she learns more about the innovation
projects. Instead of just knowing that a project succeeded or failed, the Evaluator will also
know more about the development process in innovation projects. In addition to the benefit of
the acquisition of information per se, use of PROTEE could mean a saving of resources if the
information gleaned meant that a project were terminated sooner rather than later. For a
PROTEE Evaluator, a successful innovation project is one that is described as richly as
possible.

The interests of the Innovator may well differ from those of the Evaluator. For the Innovator, a
successful project is one that continues to receive support until it becomes a successful
product. In general, making a risky description of a project may result in its not receiving
support. KFHS thus observed that it is not usually in the Innovator‟s interest to provide risky
descriptions, either internally or externally. Internally, the aim is to reduce risks by not
mentioning them or highlighting positive effects. Externally, the project description has to
strike a balance between confidence and risk, if it is to be supported. ETTC similarly noted
that their experiences lead them to believe that only "smooth" descriptions of the progress of
projects would be welcomed by an Evaluator. Especially in times of scarce financial
resources, a risk-orientated description has little chance of being accepted by an Evaluator
attuned to market acceptance. This difference in interests between Innovators and Evaluators
makes the advantages for the Innovator of using PROTEE more controversial and ambiguous.

PROTEE rewards rich descriptions which include risky descriptions. However, if PROTEE
becomes the instrument used by public bodies which support innovation projects, then
Innovators seeking such support would come to provide risky descriptions. Indeed, in this
circumstance, providing risky descriptions would be less risky for the Innovator than
providing non-risky descriptions. Both KFHS and ETTC concurred that if this were the case,




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                 Page 24 of 34
then Innovators would provide PROTEE like (risky) descriptions. Thus Krupp/Brunel
fashioned the following exchange as part of their proto-therapeutic dialogue17:
Academic Partner: In a future PROTEE world, where projects are given public support
because their descriptions are risky, then making a risky description will be less risky for the
Innovator.
Industrial Partner: If this were the case, project Teams might be obliged to have two
complementary sets of arguments about the project. The normal ones, which are missing from
the PROTEE set of Indicators but which constitute the substantive description of the project
and which would be used for relating to a company‟s hierarchy; and „procedural ones‟ which
produce risky descriptions for discussing the project with PROTEE-equipped public
promoters who reward risky descriptions.
Academic Partner: This implies that Krupp Fordertechnik would only generate risky
descriptions in order to satisfy an external PROTEE Evaluator, and that it does not consider
that risky descriptions are in and of themselves of value for internal use when managing an
innovation project.

ETTC similarly concurred that PROTEE would be particularly applicable where both
Evaluator and “the designer” accept the methodology of PROTEE as highlighting the benefits
of working with the risks and the learning process. Outside this situation, Innovators may not
be convinced that risky descriptions are valuable per se, since under conventional project
evaluation schema, including those that exist inside the Innovators‟ own organisation, risky
descriptions may be a liability with respect to obtaining support for the project.




7. CONCLUSION: PROTEE AS PUNISHMENT OR AS DISCIPLINE?


The lessons from the available case studies are encouraging. With only 4 retrospective case
studies to make these lessons, it is difficult to generalise with great robustness. However, there
is perhaps enough to conclude that PROTEE provides a fresh take on a long standing and
difficult problem. Its freshness is that it is an instrument based on alternative criteria of
innovation, upon risks, learning and networks.
The available evidence from the case studies highlights a number of key reservations, notably
about the process of identifying the unit of analysis, the applicability of the instrument and the
determination of time frames. It is nonetheless highly suggestive in moving our understanding
of innovation to an entirely new footing. In particular, we suggest the implementation,
successive application and eventual institutionalisation of the PROTEE instrument would free
the concept of innovation from its current over bureaucratisation. This in turn would give rise
to and encourage new and more daring forms of innovation.


17
     McNally, R. and Sondermann, S. „PROTEE as socio-techno-therapy‟ (16 March 1999).



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                        Page 25 of 34
As is suggested by our remarks above, a key to the acceptance of PROTEE is the
institutionalised relationship between Innovators and Evaluators. To follow Foucault, this
could go in either of two ways. The negative outcome would be the perception of Evaluators
as guardians of unchanging and inflexible standards of practice. They would be perceived as
the agents of “punishment” – Insufficient risk? You lose your funding!
Alternatively, Evaluators would be perceived as agents of a consensually adopted form of
discipline. The discipline would comprise the internalised acceptance, by Innovators and
Evaluators alike, of risk as a necessary basis of innovation. In this model, the role of
Evaluators is not just to articulate the agreed fundamentals of innovation, but also to advise,
guide and assist Innovators through dialogue. This interactive dialogue will work best when
construed as Socio-Technical-Therapy (STT).
PROTEE provides an instrument for the evaluation of innovative projects in the field of
freight transport. Yet its design and testing suggests it is eminently applicable to a wide range
of different fields involving innovative development.




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                Page 26 of 34
APPENDIX 1: PROTEE METHODOLOGY


The Project Description Summary Sheet and the Learning Curve Summary Sheet were
designed as part of Brunel University‟s work on the methodology for the practice of PROTEE.
The challenge was to develop a methodology which not only used PROTEE Indicators, but
which also captured the key features of the PROTEE approach/philosophy, as described in the
PROTEE Interview Guidei.
The particular features of the PROTEE approach/philosophy which the Sheets and their use
are designed to accommodate are listed below. (Linked features are grouped together.)

PROTEE aims to produce a description of the project; a description is an extract of the project
story.
The PROTEE description is the product of an interaction between the innovator and the
evaluator; it is a common description; this does not mean that they agree, but that they share a
common reference to situate their disagreement the interaction and the description should be
realised according to a set of principles

PROTEE should produce a Protocol Book - an archive or file on the project; it should leave a
paper trail.
the Protocol Book should be a way of evaluating the evaluator
the Protocol Book should be useable by others involved with other projects.
A key item in the Protocol Book is the Minutes; PROTEE should be a way of taking Minutes
at a project meeting.

A PROTEE evaluation involves a comparison of two descriptions of the same project at
different time points
A PROTEE project is evaluated only through the quality of the trajectory of the learning curve
- the differential between the descriptions at two different time points

The Sheets and their method of use, described below, are designed to meet all of the above
objectives from the Interview Guide. When completed, it is envisaged that the Project
Description Summary Sheet and the Project Learning Curve Summary Sheets will be part of
the Minutes of a PROTEE meeting.




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                               Page 27 of 34
Methodology

For the methodology, each PROTEE Indicator is expressed in three ways:
  1. as a question
  2. as a spectrum of possible answers to the question
  3. as the difference between the answers to the question at two different time points

These three ways of expressing the Indicators are described below. The Indicators used in the
sections below are based on the Indicators described in the PROTEE Interview Guideii. The
questions and the Sheets can, of course, be revised to correspond to whatever set of PROTEE
Indicators is finally developed.

1. The Indicators as questions

The boxes below give the PROTEE Indicators expressed as questions. Each question
corresponds to a specific Indicator and is based on the Indicator‟s description in the Interview
Guide.

I: REALISABILITY

   a.   How rich is the description of the elements in the project?
   b.   How diverse is the range of elements described?
   c.   How contingent is the description?
   d.   How differentiated is the treatment of elements with different levels of uncertainty?
   e.   How branched is the imagined future trajectory of the project?


II: NEGOTIABILITY

   a.   How many antiprograms are described?
   b.   How coherently are they described?
   c.   How big is the scope for negotiation?
   d.   How coherently are the pros and cons of negotiation and non-negotiation articulated?


III: FALSIFIABILITY

   a.   How diverse is the range of experts who shape the project?
   b.   How arbitrary are evaluations of the project?
   c.   How coherently are the pros and cons of decisions about the project articulated?
   d.   How relevant are the trials which have been undertaken to explore the pros
        and cons of decision about the project?


b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                               Page 28 of 34
                                 PROJECT DESCRIPTION SUMMARY SHEET

Use the sheet to record the summary of ONE description of the project

Code the description by date:
for a project which is being studied in real time, use the date when the description is
made;
for a project which is being studied retrospectively, use the date which corresponds to
the stage of the project which is being described.

Project name:..........................................................................................
Date of description: ...............................................................................
..................................................................................................................
Summary by Innovator/Evaluator*:.........................................................
*delete as appropriate



CLASS                                Record the evaluation of the description, for example, by making
                                     a mark along the scale

                                     0                 1                   2                  3                  4              5


I: REALISABILITY                     a. Description is very poor                                           Description is very rich
(Anti-ballistic)                     b. Elements are homogeneous                               Elements are very heterogeneous
                                     c. Description is necessary                                  Description is highly contingent
                                     d. Uncertainty is totally undifferentiated Uncertainty is highly differentiated
                                     e. Trajectories are totally unbranched                       Trajectories are highly branched


II: NEGOTIABILITY                    a. No antiprograms                                                   Very many antiprograms
(Anti-paranoia)                      b. Antiprograms are incoherent                                Antiprograms are very coherent
                                     c. No flexibility for negotiation                            Lots of flexibility for negotiation
                                     d. Pros & cons are incoherent                                 Pros and cons are very coherent


III: FALSIFIABILITY                  a. Experts are homogeneous                                    Experts are very heterogeneous
(Anti-                               b. Judgments are arbitrary                     Judgments are based on relevant evidence
manipulation)                        c. Alternatives: Pros & cons incoherent                             Pros & cons very coherent
                                     d. Trials are irrelevant                                             Trials are highly relevant




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                                                        Page 29 of 34
2. The Project Description Summary Sheet:

   The Project Description Summary Sheet (above) is designed to correspond precisely with
    the list of Indicator questions given in the boxes above. For example, in answer to the
    question asked in Indicator Ia „How rich is the description of the elements in the project?‟,
    the Sheet makes provision for an answer ranging from „very poor‟ to „very rich‟. Similarly
    for Indicator Ib, „How diverse is the range of elements described?‟, the Sheet makes
    provision for an answer ranging from „homogeneous‟ to „very heterogeneous‟.

   The interaction in which the Sheets would be used is envisaged as follows. Towards the
    end of a project meeting, each of the parties present should attempt to record a summary of
    the PROTEE description of the project using a Project Description Summary Sheet. The
    discussion which accompanies the filling in of the Sheets is considered to be an important
    part of the heuristic process of the meeting. The Sheet-filling exercise should be discursive
    and should be an integral part of the meeting. The innovator(s) and evaluator(s) may not
    agree in their descriptions of the project, but the Sheets should help them to discuss the
    project in a principled way and should help to identify specific points of agreement and
    disagreement. The completed Summary Sheets should then be included in the Protocol
    Book as part of the Minutes of the meeting.

   The Project Description Summary Sheets thus become an important part of the paper trail
    of a project. They record the date of the meeting, provide a transparent summary of the
    common description of the project at that point in time, and make a record of the parties
    who have made that description. It is envisaged that a useful starting point for a PROTEE
    project meeting would be the Project Description Summary Sheets completed and
    discussed at the previous project meeting.

3. Project Learning Curveiii Summary Sheet

   Once Project Description Summary Sheets have been completed for two consecutive time
    points (or meetings) on the same project, it is possible to fill in a Project Learning Curve
    Summary Sheet (below). These Sheets are designed to meet the PROTEE objective that
    what is evaluated is not a description of a project, but the difference between two
    descriptions over time. Thus, it is not the completed Project Description Summary Sheets
    that form the basis of the evaluation, but the differences between two such Sheets
    completed at different stages of the project.

   To complete a Project Learning Curve Summary Sheet, two consecutive Project
    Description Summary Sheets should be compared, Indicator by Indicator. The outcome of
    this comparison is recorded on the Project Learning Curve Summary Sheet. For each
    Indicator, a mark is made in either the left or right hand column or, if there is no difference
    between the two descriptions, a mark is made in the centre.


b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                 Page 30 of 34
   As with the Project Description Summary Sheets, this process is envisaged as a collective
    activity and an integral part of the project meeting, performed by both the innovator(s) and
    evaluator(s).




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                               Page 31 of 34
                           PROJECT LEARNING CURVE SUMMARY SHEET

Use the sheet to record the DIFFERENCES between TWO descriptions of the same project

Project name:.............................................................................................
Comparison of:
a)     description dated: ..........................................................................
       by evaluator/innovator*:..................................................................
b)     and description dated: ...................................................................
       by evaluator/innovator*: .................................................................
*delete as appropriate


CLASS                                       Compared to the earlier description, the later description is:
                                            (Select either the right hand or the left hand option)


I: REALISABILITY                   a.       Poorer                                              Richer
(Anti-ballistic)                   b.       More homogeneous                                    More diverse
                                   c.       More necessary                                      More contingent
                                   d.       Degrees of uncertainty are less differentiated      Degrees of uncertainty are more differentiated
                                   e.       Trajectory is less branched                         Trajectory is more branched


II: NEGOTIABILITY                  a.       Fewer antiprograms                                  More antiprograms
(Anti-paranoia)                    b.       Antiprograms are less coherent                      Antiprograms are more coherent
                                   c.       Project has less flexibility for negotiation        Project has more flexibility for negotiation
                                   d.       Pros & cons of negotiation/non-negotiation          Pros & cons of negotiation/non-negotiation
                                            are less coherent                                   are more coherent


III: FALSIFIABILITY                a.       Range of experts judging the project is more        Range of experts judging the project is more
(Anti-                                      homogeneous                                         heterogeneous
manipulation)
                                   b.       Judgments are more arbitrary                        Judgments are more based on relevant
                                                                                                evidence
                                   c.       Pros     &   cons   of     decisions   concerning   Pros     &   cons   of   decisions   concerning
                                            alternatives are less coherent                      alternatives are more coherent
                                   d.       Trials are less relevant                            Trials are more relevant


CONCLUSION
At this stage, this                         not learning                                        learning

project is:                                 not an innovation                                   is an innovation
                                            hopeless                                            hopeful
                                            badly managed                                       well managed




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                                                     Page 32 of 34
   Once completed, the Learning Curve Summary Sheet will comprise a series of Indicators
    each of which is marked; marks on the right hand column indicate increased learning,
    whilst marks on the left hand column indicate decreased learning. Thus, the position of the
    marks on the completed Learning Curve Summary Sheet should suggest a visual
    diagnostic for the project.

   Using this visual diagnostic, future activity on the project can be discussed and agreed
    upon. For example, a lot of marks on the left hand column in a particular Class of
    Indicators would suggest that this is an aspect of the project where future effort should be
    focused.

Discussion

   The methodology presented here is designed to incorporate the features of the PROTEE
    approach/philosophy listed above. These include the generation of a transparent paper
    trail; a record of the common description of a project reached at each meeting; a way of
    comparing two PROTEE descriptions; a way of recording the difference between two
    PROTEE descriptions; a record of the project learning curve which is also a visual
    diagnostic.

   One of the advantages of using paper Sheets is that they can be used in any meeting room.
    This is an important consideration because the PROTEE description and evaluation of a
    project should be a joint exercise involving the innovator and the evaluator. It should be
    able to be done at the meeting, rather than afterwards by the evaluator alone.

   The questions and the two Sheets described here are based on the Indicators in the
    Interview Guideiv, and the descriptions and diagnosis they provide are only as good as the
    Indicators The questions and the two Sheets described here are only as good as the
    Indicators they use. As the Indicators are improved, so the improvements can be
    incorporated in the questions and Sheets.

   Another function of these Sheets is within the PROTEE project itself, because they raise
    some questions about the design of the PROTEE approach. For example, does every
    Indicator have to be used at each time point for every project? For this to be the case then
    the Indicators would have to be both sufficiently general and sufficiently comprehensive to
    cover every eventuality in every project, which would be a very difficult goal to achieve.
    The alternative to using every Indicator every time would be to use the Indicators on an
    intuitive basis, with perhaps even the possibility of adding new Indicators, which are
    specific to a project, on an ad hoc basis. However, if different Indicators were used at
    different time points to describe the same project, then comparison of the project between
    time points would become difficult, if not impossible. Moreover, it would also make
    comparisons between projects less informative.



b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                               Page 33 of 34
Notes
i
      Latour, Bruno and Martin, Solange, PROTEE Interview Guide, draft for discussion
           among members of the Consortium, 18 November 1998.
ii
      op. cit. 1
iii
      ‘Learning curve’ is the terminology used in the Interview Guide.
iv
      op. cit. 1




b112aeb3-080b-4b9a-ada4-e365f5c3a6ef.doc                                 Page 34 of 34