Brussels_ 15 October 2007 by abstraks



Workshop 1: Rigorous impact evaluation using counterfactuals

Counterfactual impact evaluation (the use of comparison and control groups) is a powerful
technique, with the potential to generate convincing evidence on the impacts of Structural
Funds. In this workshop, practitioners presented and discussed the potentials, challenges and
limitations of such techniques.

The workshop was chaired by Fabrizio Barca (Italian Ministry of Economy and Finance).
The presentations were given by Alberto Martini (Director of Progetto Valutazione),
Gerhard Untiedt (GEFRA – Society for regional economic analysis) and Kaspar Richter
(World Bank). Daniele Bondonio (University of Eastern Piemonte) acted as discussant.

Presentations & key messages:

Counterfactual Impact Evaluation (CIE) is a tool of growing importance for the evaluation
of Cohesion Policy. Professor Martini noted that it is the tool of choice for measuring the
size and significance of the effect of an intervention – the fundamental evaluation question.
You cannot ask how something works until you know if it worked in the first place. Mr
Richter added that control and comparison group work is the centrepiece of the World
Bank's "result agenda".

The various techniques and statistics often associated with CIE can obscure the fact that it is
based on simple common sense: the comparison group (however simple or complex the
method used to construct it) stands in for the "missing counterfactual", ie what would have
happened without assistance. This latter can never be observed directly – hence the need for
a comparison or control group.

There was general agreement on a useful rule of thumb for the division of labour:

      Output and result indicators from the monitoring process as the tool of choice for
       day-to-day programme management.

      Counterfactual Impact Evaluation for finding out "what works" and "what doesn't".
       With enough data to distinguish sub-groups, it can also identify "for whom it

      Theory-based (or "realist") evaluation for "how and why it works".

Despite its importance, all present agreed that the use of the tool in regional development
measures is a developing field. Professor Untiedt noted how few counterfactual studies exist
for enterprise support. And despite the World Bank's 220 studies of this kind (mostly on
labour market issues), they are still "feeling their way across the river".

While random assignment is the best method from a scientific point of view, most often it
will not be practical in a European context. Some participants pointed out that the World
Bank operates in a context where the challenge is so vast that strong rationing is inevitable –
and it is therefore reasonable to harness it for evaluation purposes. Mr Richter added that
even in this context the World Bank has a pragmatic approach – randomisation where
possible, but other evidence is good too.
Practically, this means that a comparison group will have to be constructed somehow. There
is a wide variety of ever developing methods for this, ranging from the very simple to the
very complicated. However, the take home message is that appropriate methods can be
found for a very wide range of situations and needs.

Another practical consideration is data. All comparison group methods require good data –
preferably panel data – and large samples. This restricts the frequency with which the
method can be used. It also restricts the fields to which it can be applied:

      There was general agreement that enterprise support, health, education, training
       and active labour market measures are appropriate fields, where data exist.

      There was also general agreement the method is unlikely to work for transport
       interventions or for regional R&D strategies and networks.

      The jury is out on R&D measures at the firm level, urban and local regeneration,
       energy efficiency measures.

The data requirement also means that the method requires preparation. Ideally, it will be
built in right from the start. The fact that target variables must be carefully selected and
tracked also the benefit of focussing the attention of both policy-makers and beneficiaries on
the fundamental objectives.


Counterfactual impact evaluation is an essential tool in the evaluator's armoury, helping to
answer the key question of what works and what doesn't. It already has a strong record in
improving the effectiveness of World Bank projects over time – partly by knowing which
measures are working, partly by improving targeting.

In fact, when there is enough data it can be incredibly instructive to review the relative
performance of various subgroups, eg to young people in training programmes, SMEs of
various sizes and micro-enterprises within general enterprise support.

It is important to make CIE not just scientifically rigorous but also policy relevant. To do
this it is essential to select target indicators in collaboration with policy-makers and
stakeholders, so that the indicators chosen clearly reflect their policy questions, rather than
academic questions or (even worse) being based solely on ease of data gathering. This
selection should ideally occur at the start of the project, focussing attention on the
fundamental goals as well as enabling thorough data collection.

For maximum policy relevance CIEs should also be accompanied by theory-based
evaluation, to understand what determines the impact (the "how" and "why" questions).


To top