Evidence – based evaluation – the way forward for the learning and skills agenda in Wales
The National Council ELWa is the strategic funding body responsible for all publicly supported post 16 education and training, except Higher Education, in Wales. Upon its formation in April 2001, the National Council ELWa took over the responsibilities of the former Further Education Funding Council for Wales (FEFCW) and the 4 Welsh Training and Enterprise Councils (TECS). The NC ELWa is responsible for maximising the value derived from its multi-million pound investment in post 16 education and training in Wales. It has given a firm commitment to evaluate all that it does and to learn from its actions in order to improve policies, programmes and decision making. Within NC ELWa’s Operational Plan for 2001 – 2002, there was a firm commitment to 2 guiding principles, 1). Measure NC ELWa’s contribution against the real needs of individuals, communities and businesses 2). Maximise the effectiveness and efficiency of all that it does and funds These are laudable statements of intent but how will they translate into the new organisation’s operational culture? The clear stages in the transition process and the mechanisms necessary to its implementation have been identified. These mechanisms and some of the issues they raise will constitute the focus of this paper. There has been a necessary time lapse between the official formation of NC ELWa, the expressed commitment to the above evaluation principles and the production of a concrete Evaluation Strategy and Guide in March of this year. This time lapse represented a complex period of corporate reorganisation and evolution that NC ELWa had to undergo to transform itself from 4 separate entities into one unified and united national organisation. It is true that ideally, ex ante evaluations should signpost the strategic direction of an organisation or programme. However, given the unprecedented and inherited complex circumstances from which NC ELWa was emerging, the overriding priority was to establish a reformed yet stable organisational structure with a clear corporate identity. And now, it is from this new stable platform, clearly underlined by specific goals and priorities, that the energies of the new organisation can legitimately focus upon a topic as pressing to its operational integrity and success as evaluation. The process of absorbing successfully the resources, culture and working practices of 4 different organisations nevertheless highlighted a serious issue for the future operational success of NC ELWa as there was an inherited lack of internal evaluation expertise and capacity within the newly formed organisation. The Corporate Plan for 2002 - 3 stipulates NC ELWa’s concrete plan to begin to overcome this gap in expertise, by implementing, “a comprehensive evaluation framework to gain a clearer understanding of the successes and failures within inherited programmes and ensure that all new programmes contain a robust evaluation process” This statement illustrates how evaluation has passed from being a principle to being an acknowledged practical pillar of NC ELWa’s working practices. Here evaluation is both retrospective and forward looking as there is recognition of the urgent need to
learn from established practice and channel this learning into future programmes. As the Corporate Strategy states, to formulate, “an evidence based policy – making framework to inform all forthcoming policies and strategies for effective learning”. This is intended to ensure that there is a meaningful and purposeful link between evaluation at programme level and its role as a determinant of NC ELWa’s future strategies and policies. It is perhaps useful, at this stage, to stress why learning from evaluation is particularly important for NC ELWa. As stated at the outset, NC ELWa has an all Wales remit to deliver learning and training to transform Welsh society. Such ambitious national objectives bring the responsibility of a substantial budget, and this year alone, NC ELWa, in the realisation of its domestic objectives, must deploy efficiently and equitably a budget of circa £500 million. My specific role of Monitoring and Evaluation within NC ELWa lies in the European Partnership Unit where I am primarily required to evaluate the effectiveness and added value of current Objective 1 + 3 EU funded Human Resource Development projects to inform not only the mid term review and key domestic documents but proposals for Stuctural Funds in Wales post enlargement. Since January of this year, sound structures and standards regarding NC ELWa’s evaluation of European programmes with an Human Resource Development focus have been established. My practical experience of this process has provided me with an unique perspective on the parallel complementary implementation of evaluation taking place with NC ELWa’s domestic programmes. As a member of the Evaluation Steering Group, charged with overseeing the formulation of NC ELWa’s Evaluation Strategy and Guide, I have had the opportunity to discuss with colleagues from across the organisation the next and perhaps most telling stage in NC ELWa’s integration of an evaluation culture - translating principles into practice. This stage can be usefully summed up in 2 practical questions: • • Does the organisation have the resources, what resources can be made available to embed evaluation into programme management as well as policy development and implementation? Can the distinctive case for evaluation be made to staff as opposed to the more established practices of performance monitoring and review?
Some of the overarching persuasive arguments in favour of well deployed evaluation are summed in the UK Government’s definition of evaluation as, “the process of checking how far project (or programme) objectives have been achieved, how efficiently and economically, and learning from experience how project (or programme) specifications can be improved in future (HM Treasury). In short, efficient use of public funds, effective implementation of key objectives - and lastly, by combining the above, achieving the aim not so much of to prove as to improve. Let us briefly examine, the key document that underpins NC ELWa’s new learning culture. There are four main components to NC ELWa’s Evaluation Strategy – summative evaluation, individual project and programme evaluations, longitudinal cohort studies and continuous evaluation. The management of the Evaluation
Strategy proposes a mechanism for resolving the organisation-wide issue of evaluation capacity. It is foreseen that it will be managed as one coherent programme both nationally and regionally by a team of evaluation experts. In each of the 4 regions into which NC ELWa’s operations are divided, a dedicated staff member should be responsible for supporting and guiding staff to undertake smaller scale evaluation activities. These 4 regional “evaluation champions” should in part be responsible to the national lead on evaluation thus ensuring the quality and coordination of all NC ELWa evaluation activities. As a long term solution to the inherited internal gap in evaluation expertise, NC ELWa’s Evaluation Strategy recommends that those staff currently engaged in evaluation activities participate in a programme of professional development to ensure that NC ELWa can rightfully assume the mantle of a “learning organisation. The use of web based technology is furthermore seen as an useful tool in reinforcing and disseminating this body of organisational knowledge. To create a space for organisational memory if you will, which would provide all staff with a regular evaluation update and encourage their input where appropriate. Staff alone cannot ensure the effectiveness of the evaluation process it will also ultimately be determined by the quality of NC ELWa’s Management Information Systems. Conscious of the necessary marriage between evaluation and information systems, NC ELWa is currently undertaking a review of its data recording systems. NC ELWa’s future information systems will be required to cover a vast range of learning contexts among which interventions with businesses, quality assessments of learning providers, the tracking of baseline socio-economic indicators / labour market information and last but not least, Future Skills Wales. The FSW 2003 for example survey will be used as the evidence base for population and workforce skills. If Senior Management within NC ELWa intend to create an evaluation culture within the organisation then over and above the specific capacity building actions mentioned above, it is adviseable that that they initiate a series of internal workshops whose intention is to create a shared understanding of evaluation but also to provide a forum for dialogue with staff. For example, how does one win the hearts and minds of middle managers who will have responsibility for implementing evaluation at programme level and managing its consequent impact upon resources, day to day working practices, deployment of staff time in an increasing culture of performance related pay, individual and Team performance appraisal? Building on this issue, if an evaluation campaign is ultimately to prove successful, it is also wise to be cognizant of some of the general “preconceptions” that exist regarding evaluation as these can disproportionately condition staff responses. Firstly, evaluation can be seen as an invasive form of surveillance, imposed from “on high” which is in the long term a “threat” to and in the short term an impediment to tried and tested working practices. It can be perceived as a misdirection of staff resources. The addition of another layer of fruitless, burdensome administration which distracts valuable staff time into duplicating the monitoring and checking systems which already exist in most publicly managed programmes. These arguments are based not only upon tradition and status quo but also on the presumption, frequently borne out, that the evaluation undertaken will not result in recommended action. Equally, it is important to persuade staff that perpetuating “safe” routine practices may be comforting but it is ultimately illusory. There is clearly a case here for effective internal communication and dialogue spearheaded by Senior Management.
I have referred to the Evaluation Strategy’s recommendation for staff professional development. Awareness raising and continuous training offered to staff is one way of counterbalancing anxieties re the responsibility and accountability inherent in carrying out evaluation effectively. The Evaluation Strategy and Evaluation Guide are tangible evidence of NC ELWa’s potential to graft a far reaching and transformational evaluation culture on to its newly formed corporate structure. However, the results of the evaluations undertaken both domestic and European, will only have value if they can demonstrate whether and illuminate how NC ELWa is fulfilling its key objectives. NC ELWa’s Corporate Plan is based upon 5 interdependent goals which sum up its key activity areas: • • • • • Essential Skills Creating Lifelong Learners Developing the Ability to Apply Knowledge Skills for Business Learning Communities
Within these overarching goals, 3 specific priorities for the period 2002 / 5 have been identified. • • NC ELWa’s first priority must be given to learning which prepares the people of Wales to meet the needs of the local, national, UK and global economies. NC ELWa’s second priority is to drive up the quality of learning in Wales, to reward provision that is responsive to demand, to increase understanding of what works and to develop models of effective learning for all future programmes across all five goals. NC ELWa’s third priority is to stimulate a learning culture amongst those in education and training and those in or available for work and to develop mechanisms which will empower learners by giving them purchasing power.
•
It is clear that evidence - based evaluation will play a crucial role in demonstrating to what extent NC ELWa achieves these key priorities. Applied to the first priority, defined as the “economic priority”, it will enable NC ELWa to “determine the balance of activity and spending between the five goals and within each of them”. Evaluation applied to the second and third priorities will illuminate areas of targeted investment: “investment in essential actions will allow us to improve the supply of and demand for, learning”. It is significant moreover, that within the overarching timeframe of 2002-5, NC ELWa has already identified March 2003 as a landmark date, a yardstick by which to assess the progress of its evaluation programme, but also within the context of the above priorities, “to have begun to improve our services in light of results”. It is also important that evaluation take account of NC ELWa’s 4 Cross – Cutting Themes, • • • Equal Opportunities Sustainable Development Social Inclusion
And the theme which is Wales specific: • Bilingualism
There are high expectations in Wales for NC ELWa to deliver in the field of post – 16 education. NC ELWa’s learning programmes are multi faceted and multi – layered. Above all they require effective collaboration with multiple partnerships. During this operational year, 2002 – 3, an investment of 487 million will be made in learning. This will involve working with 26 Further Education Institutions, more than 200 providers of Work Based Learning, 170 Secondary School Sixth Forms and Community Consortia for Education and Learning (CCETS) as well as many partners in the voluntary and other sectors. NC ELWa’s commitment to establishing an internal learning culture will enable it to spearhead a consistent agenda for change in Wales and an evidence based policy will allow it to, “be open and accountable to those we serve”. For it is clear that failure to embrace evaluation fully, will diminish not only an organisation’s capacity to deliver in the long term but ultimately impoverish the relevance of all that it plans to do and actually does. As indicated above, the delivery of NC ELWa’s regional and national learning programmes is dependent upon strong relationships with many diverse partner organisations. Working in partnership is a characteristic of the learning agenda, domestic and European in Wales. In this context, NC ELWa’s ongoing commitment to evaluation can be used to sign-post the stages in the direction of its strategy. Communication of evaluation findings will engage partners more effectively and consequently, reinforce current as well as forge future partnerships which are so key to the fulfilment of NC ELWa’s objectives. Geraldine. K. Fazio Senior European Monitoring and Evaluation Manager NC ELWa