2006213 ASW Consulting Report on CWQA Evaluation Final 31st Jan 06 by HC120303161043


									  ASW Consulting

           CAREERS WALES


            FINAL REPORT

             January 2006
Careers Wales                                                           Evaluation of CWQA


Chapter                                                                              Page

I:       BACKGROUND, PURPOSE AND METHOD                                                   4
     Introduction                                                                         4
     Background                                                                           4
     Terms of reference                                                                   5
     Methodology                                                                          6
     Structure for remainder of report                                                    8

II:      TAKE UP AND REASONS FOR PARTICIPATION                                            9
     Introduction                                                                        9
     Current take up                                                                     9
     Reasons for non-participation                                                       9
     Benefits from participation                                                        11
     Costs and benefits for schools and colleges                                        12
     Costs and benefits for Careers Wales                                               13
     Potential incentives for more institutions to participate                          14
     Feasibility of levying a fee                                                       15
     Concluding summary                                                                 16

III:     SCOPE AND CONTENT OF CWQA                                                      17
     Introduction                                                                       17
     Overall approach                                                                   17
     Principles and Criteria                                                            18
     Boundaries between the two modules                                                 20
     Relevance of model to different types of institution                               21
     Welsh language                                                                     22
     ACCAC frameworks                                                                   23
     Concluding summary                                                                 23

IV:      DELIVERY PROCESSES AND STRUCTURES                                              24
     Introduction                                                                       24
     Feedback on the process in general terms                                           24
     Role of Careers Wales                                                              25
     Three stage process                                                                26
     Consultancy                                                                        28
     Assessment                                                                         29
     Local and national co-ordination                                                   31
     Concluding summary                                                                 33

V:       IMPACT ON INSTITUTIONS                                                         34
     Introduction                                                                       34
     Overview                                                                           34
     Content and delivery of CEG and WRE programmes                                     37
     Planning, management and evaluation processes                                      38
     Profile and status                                                                 39
     Involvement of Senior Managers and external partners                               40
     Institutional objectives                                                           41

ASW Consulting                                    2              Final Report – January 2006
Careers Wales                                                        Evaluation of CWQA

    National standards                                                               41
    Staff development                                                                41
    Other impacts                                                                    42
    Concluding summary                                                               42

VI:     IMPACT ON PUPILS AND STUDENTS                                                43
    Introduction                                                                     43
    Methodological challenge                                                         43
    Learner views                                                                    43
    ‘Third party’ views – questionnaires                                             44
    ‘Third party’ views – fieldwork                                                  45
    Concluding summary                                                               46

VII:    RECOMMENDATIONS                                                              47
    Introduction                                                                     47
    Rationale                                                                        47
    Shorter term recommendations                                                     47
    Longer term recommendations                                                      49

A       Evaluation framework
B       Participants in fieldwork discussions
C       Written comments received
D       CWQA costings

Table 1.1: Breakdown of questionnaire respondents, by extent of involvement           8
Table 2.1: Take-up of CWQA by type of institution, as at September 2005               9
Table 2.2: Reasons given for not participating in the award (non-participating
           institutions only)                                                        10
Table 2.3: Headline views on whether benefits of participation outweigh the costs    13
Table 2.4: Views on whether introduction of a fee would affect take up of the CWQA   15
Table 3.1: Views on an alternative ‘compliance’ model                                17
Table 3.2: Overall view on appropriateness of Principles and Criteria                19
Table 3.3: Views on clarity and appropriateness of boundaries between the two
           modules                                                                   20
Table 3.4: Views on appropriateness of single model for use in diverse types of
           institution                                                               21
Table 4.1: Views on role differentiation                                             25
Table 5.1: Assessed impact of CWQA on institutions in general terms                  35
Table 5.2: Impact of CEG module on participating institutions                        35
Table 5.3: Impact of WRE module on participating institutions                        37

ASW Consulting                                  3            Final Report – January 2006
Careers Wales                                                                   Evaluation of CWQA



101      This report presents the findings from a study, commissioned by Careers Wales
         (CW), to evaluate the content, process and management of the Careers Wales Quality
         Award (CWQA). The study was undertaken by ASW Consulting (ASW).
102      This first chapter:-
              sketches in the background to the project;
              sets out its purpose and terms of reference;
              summarises the methodology adopted;
              describes the structure for the remainder of the report.

Policy context
103      Excellence in careers education and guidance (CEG) and work related education
         (WRE) is central to a number of key agenda in Wales, including raising standards of
         attainment, enhancement of skills, economic development and social inclusion. The
         CWQA is the centre point of CW’s drive for higher standards in CEG and WRE.
104      It has been developed during a time of change. A number of policy initiatives have
         influenced the nature and scope of this work. These have included:-
              the statutory requirement for CEG at KS3 and KS4, as well as post-16;
              the statutory requirement for the provision of WRE activities (from September
              ESTYN inspection criteria;
              1ACCAC’s Framework for Personal and Social Education (PSE) at KS3 and KS4;

              Progress File2;
              ACCAC’s Framework for WRE 14-19;
              ACCAC’s Learning Outcomes for CEG;
              the Entrepreneurship Action Plan for Wales3;
              Welsh Baccalaureate;
              other, more strategic developments, albeit with an implication for the way that
               services are delivered in Wales4.
Development and delivery structure
105      Until 2000, four quality awards for CEG were in operation in Wales. As a result of
         the recommendations outlined in the Wales Education Training and Action Plan,
         Careers Wales set up an all-Wales group with the remit to review current best
         practice. This was to be followed by the production of one, all-Wales standard for
         CEG called the Careers Wales Quality Award: Careers Education and Guidance.

1   Qualifications, Curriculum and Assessment Authority for Wales
2   An online version is now available through Careers Wales.com
3   A flagship initiative fronted by the Welsh Development Agency (WDA)
4   A key driver here is ‘Making the Connections: Delivering Better Services in Wales’ NAW, Oct 2004

ASW Consulting                                     4                   Final Report – January 2006
Careers Wales                                                                        Evaluation of CWQA

106       The all-Wales quality award for CEG was launched in 2001. At the same time, a Co-
          ordination and Development Working Group was set up to manage the day-to-day
          delivery of the award. A link Chief Executive (currently Careers Wales Gwent) acts
          as overseer for the group’s activities and provides the key link to the remaining
          Careers Wales Chief Executives.
107       Also in 2001, a national award co-ordinator was seconded from one of the Careers
          Wales companies to provide stability to the delivery of the award. The role includes
          carrying out the development of all-Wales training provision and ensuring consistent
          and fair assessment standards.
108       In 2003 Careers Wales undertook the development of a second award module – Work
          Related Education (WRE). This was added to the award portfolio alongside a
          revision and update of the original award criteria; this updated award - the CWQA -
          was launched in 2004.
Scope and content
109       The CWQA is described as a means of ‘recognising high quality, effective careers
          education and guidance and work related education within any establishment where
          14-19 year old students are being educated’. Further, it is positioned as ‘equally
          relevant for secondary schools, further education colleges, special schools and off-
          site units’.5
110       CWQA has two main modules, one each for CEG and WRE. There are 5 Principles
          for both modules, each of which is supported by Criteria which schools and colleges
          need to meet in order to achieve the award.
111       The five Principles are:-
              Principle 1: Student achievement of learning outcomes
               Principle 2: Overall management
               Principle 3: People management
               Principle 4: Partnerships, processes and resources
               Principle 5: Evaluation and continuous review.
Current position
112       Training and development of staff involved in supporting establishments to develop
          their curriculum to achieve the award has been undertaken both for the CEG award
          and subsequently for the WRE module.
113       As of September 1st 2005, 39 establishments had achieved the CWQA for CEG and
          around 170 schools, colleges and special schools were working towards either CEG,
          WRE or both award modules; as of the same date, 2 schools had achieved the CWQA
          for both CEG and WRE. The current position (December 2005) is summarised in
          Table 2.1 in Chapter 2.

114       The credibility of the CWQA is central to its success. It is crucial, therefore, that
          extremely high standards are demanded of establishments - and that the award
          itself is rigorously assessed. This provides the stimulus for the current project.
115       The starting point for the evaluation was the aims set for the CWQA itself. These

5   Both these quotations are from the pages on CWQA in the CW website Careers Wales Online.

ASW Consulting                                        5                    Final Report – January 2006
Careers Wales                                                             Evaluation of CWQA

            to ensure that all young people in Wales have an equal entitlement to excellent
             work related education and careers education and guidance;
            to acknowledge and celebrate high quality and well integrated work related
             education and careers education and guidance;
            to act as a mechanism for supporting the continuing improvement of
116   Turning to this specific project, the terms of reference were defined as:-
            to evaluate the impact that working towards and achieving the award has had
             on learners, schools and colleges;
            to evaluate key aspects of the award process.
117   At the next level of detail, the ‘key aspects’ referred to in the terms of reference were
      outlined as follows:-
            impact on learners:
             - impact of CWQA on the quality, range and depth of CEG and WRE provision
               in schools and colleges;
            impact on establishment processes:
             - content of the CEG/WRE programme
             - management and planning for subject delivery
             - management and utilisation of staff involved in the subject area
             - use of resources and involvement of key partners
             - evaluation and forward planning;
            management by CW of the award:
             - CW delivery and administration
             - CW support for establishments working towards the award
             - impact of CW service provision to establishments achieving the award;
            processes and structure:
             - consultancy process
             - assessment process.

118   The study was managed on behalf of CW by a Steering Group comprising Mary
      Jeans (Chair, CW Quality Award Group), Trina Neilson (Chief Executive ‘Champion’
      for CWQA) and Sally Farr (CWQA National Co-ordinator). The ASW Consulting team
      comprised Andrew Watson, Brian Ellis, Nia Lynn Jones, and Siôn Aled Owen.
119   The methodology had five main components, as follows:-
         desk research;
         consultation with national groups;
            visits to all 6 CW companies;
            fieldwork with a sample of 6 contrasting schools and colleges;
            briefing paper and questionnaire.
Desk research
120   Desk research was undertaken throughout the entire study. Material reviewed was
      of two main types:-
            papers relating to the policy context (e.g. ACCAC frameworks, Learning
             Pathways 14-19 working documents);
            CWQA material (e.g. assessment reports, panel papers, consultants manual).

ASW Consulting                                6                  Final Report – January 2006
Careers Wales                                                                              Evaluation of CWQA

121     An evaluation framework was prepared early in the project. Its principal purpose
        was to identify those issues to be addressed during the study and to segment them
        into groups of related topics. This framework was used to distil checklists for use
        during fieldwork visits. A copy is attached as Appendix A.
Consultation with national groups
122     At the national level, discussions were held with:-
              ACCAC;
              CW Chief Executives Group;
              CW Quality Award Group.
Visits to CW companies
123     Visits were undertaken to all CW companies. Although the precise details varied,
        discussions were typically held with the following:-
              the CWQA Co-ordinator;
              the Chief Executive or nominated Senior Manager;
              CWQA Consultants and Assessors.
Fieldwork with schools and colleges
124     Fieldwork was also carried out in a sample of six schools and colleges. These were
        selected by the research team from a long list of participating institutions6 supplied
        by CW. Schools and colleges were chosen on the basis of covering as many variables
        as possible (e.g. geographical areas, pre and post-16, English and Welsh medium,
        mainstream and special, long standing and recent involvement).
125     As with the visits to CW companies, the precise details of the programme followed
        varied between institutions. Typically, however, discussions were held with:-
              the Headteacher/Principal (or his/her nominee from the senior management
              the staff responsible for delivering the CEG and WRE components of the
              a small group of pupils/students (where feasible).
Briefing paper and questionnaire
126     The study team was acutely aware that (unintentional) bias could arise from the
        particular cross-section of views picked up through the visits and fieldwork. A
        bilingual briefing paper was therefore drafted and distributed to all schools and
        colleges in Wales, setting out the key issues and inviting written contributions. 7
127     In practice, the response rate to the briefing paper was very high. At the cut-off date
        (5th December 2005), 51 completed questionnaires had been received; these provide
        the principal material for the quantitative data quoted in the text and also contribute
        to the qualitative commentary. Since that date, a further 11 questionnaires have
        been received; these have also been read and taken into account in the preparation
        of this report (but not included in the quantitative summaries).

6 We use the general term ‘institution’ throughout to indicate mainstream and special schools, colleges, pupil
referral – and other – units. Where a point is linked to one type of institution only this is made explicit in the
7 We have not added the briefing paper as an appendix to this report, as the content would replicate much of

the material included in subsequent chapters. It can be made available immediately on request, however.

ASW Consulting                                           7                       Final Report – January 2006
Careers Wales                                                              Evaluation of CWQA

128     The table below indicates the extent of involvement among the 51 institutions
        included in the quantitative analysis.
Table 1.1:    Breakdown of questionnaire respondents, by extent of involvement

                                                                                NO OF
STAGE REACHED                                                                RESPONSES IN
                                                                            THIS CATEGORY

Not yet registered interest in either module (‘non-participants’)                  17

In the early stages of preparing for at least one module (e.g. self
assessment and action planning)
Well on the way to achieving at least one module (e.g. development
phase or preparing for panel)

Already awarded the certificate for at least one module                            15

Total                                                                              51

129     The names and organisations of all those contributing to the fieldwork and national
        consultations are added as Appendix B. Those institutions providing written
        comments are listed in Appendix C. The helpful contribution of all participants is
        gratefully acknowledged by the study team.

130     The structure for the remainder of the report is as follows:-
            Chapter 2 reports on take-up of the award and the reasons given for both
             participation and non-participation;
            Chapter 3 comments on the scope and content of the award (e.g. the Principles
             and Criteria);
            Chapter 4 provides feedback on the structures underpinning delivery of the
             award (e.g. deployment of Consultants and Assessors);
            Chapters 5 and 6 assess the impact of the award on institutions and on pupils/
             students respectively;
            Chapter 7 presents the study team’s recommendations (both short and long
131     Brief summaries of the key points are added at the end of Chapters 2 to 6 inclusive.

ASW Consulting                                8                     Final Report – January 2006
Careers Wales                                                                   Evaluation of CWQA



201     This chapter considers issues around take-up of the award and reasons given for
        both participation and non-participation. The sections are headed:-
            current take-up;
            reasons for non-participation;
            perceived benefits from participation;
            costs and benefits for schools and colleges;
            costs and benefits for Careers Wales;
            potential incentives for more institutions to participate;
            feasibility of levying a fee.


202     The table below provides an overview of the current position, as regards take up by
        type of institution, across Wales. In broad brush terms, around a half of institutions
        have some involvement, although only 2 out of 330 have completed both modules.
Table 2.1:     Take-up of CWQA by type of institution, as at September 2005

                           Mainstream                                    PRU & other
                           11-16 & 11-       Colleges                     specialist        Totals
                           18 schools                                       units
Number with award
                                  2              0              0               0              2
for both modules
No with award for at
                                 38             10              5               0             53
least one module
No working towards at
                                 81             24             20               0             125
least one module
Non-participating                               *See
                                107                            20              32            *159
institutions                                   below

Total    number     of
                                228             *25            45              32            *330
institutions in Wales

Note:   *Colleges may participate on a faculty, campus or whole college basis. The figure in the
        bottom (totals) row reflects the number of colleges – as whole entities. Figures in rows above
        (status re CWQA) reflect the number of faculties, campuses or whole colleges which have
        been assessed.
Source: Careers Wales


203     Given the participation rates evident from Table 2.1, it was clearly important to
        understand the reasons why many schools and colleges were not participating. This
        question was raised specifically in the questionnaires and the responses are
        summarised in Table 2.2 below.

ASW Consulting                                    9                   Final Report – January 2006
Careers Wales                                                                          Evaluation of CWQA

204       Several participating institutions also commented on this point, seeking to explain
          why some of their peers were not actively involved. We decided not to include these
          responses in this table on the grounds that perceptions were bound to be influenced
          by experiences gained and it was essential to hear the voice of non-respondents
Table 2.2:       Reasons given for not participating in the award (non-participating
                 institutions only)

    REASONS FOR NON-                       AGREE                 DISAGREE
                                                  AGREE DISAGREE          TOTAL
    PARTICIPATION                        STRONGLY                STRONGLY
    We have reservations about the
    relevance and/or validity of the           1              2             6               2             11
    We are concerned re the amount
    of staff time needed to work               7              8             1               0             16
    towards the CWQA
    The CWQA is a good idea in
    general terms but it has not been          6              6             2               0             14
    a priority for us
    We definitely intend to opt in to
    the CEG module within the next             1              4            3*               1             9
    2 years
    We definitely intend to opt in to
    the WRE module within the next             1              3            3    *           1             8
    2 years

Note: 1 In this – and subsequent – tables the figures indicate the raw number of responses. Given
      the relatively small sample size, it would have been misleading to convert them to percentages.
      2 Not all statements were marked by all respondents.
      3 *One response in both of these cells had a question mark by it.

205       The clear conclusion from this table (albeit from a small sample size) is that the
          main reason for non-participation is concern about the amount of staff time needed
          to work towards the award. This will be a recurrent theme throughput this report
          and is echoed also by many participating schools.
206       The table above also indicates that a significant number of institutions agreed with
          the statement that ‘the CWQA is a good idea in general terms but it has not been a
          priority for us’. This suggests that some institutions would be open to persuasion if
          the case for participation could be argued convincingly.
207       Non-participants were also invited to add their own comments. A representative
          selection is now added:-
                ‘As Teacher in Charge, responsible for the day to day running of [name of
                institution] and also with a full teaching timetable, I feel I have insufficient time
                to even work towards the CWQA.’8

8 Throughout this report, inverted commas are used to signify a quotation (or translation of a quotation) from a
questionnaire respondent or fieldwork interviewee. Editorial license has been applied in some instances where
the comment would not be understandable without some fleshing out – or vice versa (where a comment is
abbreviated). Quotations from fieldwork are based on notes taken at the time but will not always constitute the
precise words used.

ASW Consulting                                        10                        Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

            ‘Far too time consuming. I have quality awards for interim, full and gold
             awards. The school is restructuring and I now have to take aboard work
             experience as well as CEG. I simply have no time for extras.’
            ‘The involvement in a Quality Award is not down to its ultimate 'worth' as such
             - the school’s own developmental priorities have to be taken into account.’
            ‘The degree of control by CW is a problem for me.’
            ‘Would look into the award at some future date when it seems more feasible.’
            ‘Careers Co-ordinator has attended a course to look at what is needed but we
             haven't really moved from this yet.’


208   In parallel with ascertaining reasons for non-participation, it was important to distil
      the key benefits identified by those institutions already involved with the award.
      Respondents to the questionnaire were invited to indicate in their own words the
      main benefits from participation. We have synthesised and analysed the comments
209   There were five main themes, which are now summarised:-
          The most frequent theme (13 mentions out of the 31 narrative responses
           analysed) was around benchmarking, identifying strengths and weaknesses,
           monitoring and evaluation (e.g. ‘to enable systematic monitoring and evaluation
           of CEG and WRE activities’).
            The second largest group was around the theme of increasing awareness and
             raising the profile of CEG and/or WRE within the institution and the wider
             community (e./g. ‘raises college profile with clients and raises status of CEG’).
            The next was around improving the quality of CEG and WRE (e.g. ‘improvement
             of delivery methods and programme for pupils’).
            The next was around cohesion, linkages and a whole school approach (e.g.
             ‘ensuring a cohesive approach within PSE programme and elsewhere in the
            The remaining theme (4 mentions) was around help with preparation for Estyn
             inspections (e.g. ‘participation was valuable … and resulted in a very good
             inspection report’).
210   We asked a similar question as part of our fieldwork visits. Detailed comments from
      a school and college perspective are provided in Chapter 5 below (‘Impact on
      Institutions’). Here, we add a few examples of comments from CW staff about the
      benefits, as perceived externally:-
            ‘Has raised status of Careers Co-ordinators. Their work is better acknowledged
             by heads and governors’ (CWQA Co-ordinator).
            ‘Improved systems in both a curriculum and management context’ (CWQA Co-
            ‘Quantum improvement in CEG/WRE provision and delivery’ (CW Senior
            ‘Promoted cross-curricular working’ (CW Senior Manager).
            ‘Encouraged schools to reflect on areas needed for improvement’ (CW

ASW Consulting                                11                  Final Report – January 2006
Careers Wales                                                          Evaluation of CWQA


211   It was a very consistent view that participation in the award is a resource intensive
      process for school and college staff. The questionnaire therefore sought headline
      views as to whether ‘the benefits of participation in CWQA outweigh the costs (i.e.
      staff time and resources in kind)?’
212   The responses are summarised in the table overleaf.

ASW Consulting                             12                 Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

Table 2.3:    Headline views on whether benefits of participation outweigh the costs

Yes, benefits outweigh costs by some way                                  19
Benefits probably outweigh costs                                          10
No, benefits don’t outweigh costs                                          2
Not sure                                                                   3
Total                                                                     34

213     This is a positive finding: 29 out of 34 respondents thought the investment in kind
        was ‘worth it’. However, it is also essential to bear in mind that these respondents
        were already participating in the award and as such were already demonstrating
        their commitment.
214     We raised the cost:benefit question with Teachers, Careers Co-ordinators and Senior
        Managers we met as part of our visits to participating schools and colleges. Whilst
        they recognised the clear benefits from the award, they did not minimise the
        resource implications of the process.
215     For example:-
            one Deputy Head stressed that the Co-ordinator had put in many hours after
             school and at weekends. The award ‘would not have been achieved without her
             dedication’ [similar comments were made at several institutions];
            there was a concern expressed about changes in the contractual position of
             Teachers and the danger of ‘CWQA-type activities being squeezed out’. One Co-
             ordinator pointed out that he has only 5 non-contact periods under the new
             contractual arrangements to cover both his CEG/WRE role and his [other
             subject] responsibilities;
            a Senior Manager commented that, although CW were offering an excellent
             package, the costs effectively fell onto the schools and that the Welsh Assembly
             Government needed to put additional resources behind the award.


216     Much of the study focused on resource implications from an institutional
        perspective. However, parallel questions arise from a CW perspective. We raised
        these points in our discussions with CWQA Co-ordinators and CW Senior Managers.
217     It was common ground that CW derives many benefits from the award. In addition
        to the direct benefits (e.g. most obviously, improved CEG and WRE programmes in
        local institutions), there were several indirect benefits (e.g. raised profile for CW,
        increased credibility of CW staff).
218     For some, this was a sufficient justification:-
            ‘The resources dedicated to CWQA is a price worth paying, when weighing up
             the positive changes witnessed in schools and colleges’ (Co-ordinator).
            ‘Yes, the cost to CW is definitely worth it’ (Co-ordinator).
            ‘CW must be in a position to provide schools with the level of support they need
             – every institution is different and develops at a different pace. The adverse
             publicity of withdrawing support from a school would be very damaging’ (Senior

ASW Consulting                                13                 Final Report – January 2006
Careers Wales                                                                              Evaluation of CWQA

219     Other interviewees were concerned about the extent of resources available to
        support the award and expressed doubts about the sustainability of the current
              ‘It is important that schools are prepared to undertake their full part in the
               process and not rely on CW carrying out the work on their behalf’ (CWQA Co-
              ‘Colleges tend to require additional consultancy support, reflecting their larger
               size, and disparate faculties and campuses’ (CWQA Co-ordinator).
              ‘Consultancy time will need to be rationalised to reflect where the award is, in
               terms of maturity and available good practice’ (Senior Manager).
              ‘Too many schools are currently going through the process [to be able to
               sustain the entitlement to consultancy support] (Senior Manager)’.
              ‘Need to reduce the 6 days consultancy’ (Senior Manager).
220     The question of capacity within CW is clearly crucial here. There would be little
        point in actively promoting the award and seeking to increase numbers of
        participating institutions if there were insufficient consultant days to support
        growth. We therefore asked CW to provide a summary of costs. These are set out in
        Appendix D.
221     The costings suggest that the directly attributable costs to CW are in the order of
        £2.1k per ‘new’ institution and £0.35k per reassessment. These headline figures
        exclude costs borne by the school (notably staff time) and indirect costs borne by CW
        (e.g. consultant and assessor training).
222     If the target of ‘a half of all institutions in Wales achieving the CWQA’ is to be met 10,
        this will require nearly all of those currently working towards one or both modules
        (i.e. 165 out of 178) to complete the award in full. If the average cost of this were
        £1.4k per institution11, the notional cash cost would be £231k. Whether this is a
        realistic position, bearing in mind there will also be on-going reassessment costs, is
        a matter for CW Chief Executives and their Boards, and the Welsh Assembly

223     At present there is no incentive (in terms of levels of subsequent support by CW staff
        for institutions’ CEG and WRE programmes) for schools and colleges to participate.
        The briefing paper noted that it would be possible to argue that, following the award
        of the CWQA, support should be increased (commitment has been demonstrated and
        should be rewarded) or decreased (proven excellence so less help needed).
        Respondents were asked to comment on these propositions.
224     In outline:-
              9 respondents thought award holders should be entitled to greater support (e.g.
               ‘[Co-ordinators] need support to continue this good practice’);

9 Participating schools and colleges are entitled to up to 6 days of ‘free’ consultancy time for the first module (or
8 days if both are tackled in parallel), although it was widely accepted that, in practice, some institutions
received a lot more than this.
10 This target has been quoted by Jane Davidson, Minister for Education and Lifelong Learning in the Welsh

11 The figure used is slightly above the mid point since some institutions have hardly started the process; it is

also reasonable to assume some will drop out and any new starters would incur the full cost of £2.1k. In our
view, these figures still underestimate the likely costs to CW, since several institutions receive more than 6 [or
8] days consultancy and several colleges participate on a campus by campus or faculty by faculty basis,
resulting in much higher costs per institution. £250k may be a more realistic figure.

ASW Consulting                                          14                       Final Report – January 2006
Careers Wales                                                                             Evaluation of CWQA

              only 1 respondent thought holders should receive less support (‘CW needs to
               support establishments who are working towards the award, thus enabling
               them to improve standards’);
              by far the majority (23) thought that achievement of the award should not be a
               factor in determining the extent of support received (e.g. ‘I believe support
               should be on-going. Achieving the award is not an end result – it is a
               continuous process.       A good department continually strives for further
               improvement and needs support’).
225      Fieldwork confirmed the majority view above. Indeed, very few interviewees, either
         in CW or institutions, thought that such incentives (in either direction) would be
         prudent. CW staff also pointed out that the dividing line between consultancy for
         CWQA and mainstream support for CEG and WRE was blurred.
226      The sense was that, if there was to be any incentive for schools and colleges, it
         should perhaps take the form of accreditation for the work undertaken by Co-
         ‘Some recognition of Careers Co-ordinators’ work in developing the school for CWQA
         would be useful. Accreditation could be achieved in units gained in parallel with the
         work in preparing the school for the award. CWQA would therefore be linked with
         professional development: it could be modularised and made available via Careers
         Wales on Line’ (CW Consultant and Assessor).12

227      One suggestion raised early in the project was that, if resources in CW were to be a
         serious constraint on ability to support an expansion in coverage, income could be
         generated by levying a small charge on participating schools and colleges. This
         could cover some or all of the costs of consultancy, or assessment, for example. We
         therefore added this to the list of points we raised in the questionnaire and during
228      The question, as posed in the questionnaire, was: if a modest fee were introduced for
         the assessment and award process, would this affect take up? Responses are
         summarised in the table below. It will be apparent that a clear majority thought
         that a fee would have a negative impact on participation.
Table 2.4:      Views on whether introduction of a fee would affect take up of the CWQA

 No, institutions would still participate                  3
 Yes, fewer institutions would opt in                     20
 No difference either way                                  1
 Not sure                                                  9
 Total                                                    33

229      Views expressed during fieldwork varied between outright rejection and factors to
         bear in mind if considering the introduction of fees. Comments in the former
         category included:-

12 We recognise that accrediting the work of school Co-ordinators would require significant new work.
Nevertheless, it would be possible to print a certificate for the lead member of staff so that his or her work could
be celebrated at award events, alongside the achievement of the institution. This would help to generate good
PR within the institution and wider community for the work of the Co-ordinator.

ASW Consulting                                          15                      Final Report – January 2006
Careers Wales                                                               Evaluation of CWQA

            ‘Isn’t it like shooting yourself in the foot?!’ (CWQA Co-ordinator).
            ‘As I only have a budget of £1,000 for the whole of CEG and WRE, there would
             not be much scope for paying fees!’ (school Careers Co-ordinator).
            ‘We should not overlook the amount of effort put in by schools (and especially
             Co-ordinators) to get the award; it would appear ‘mean’ of CW to ask for any
             financial contribution. It could have the effect of reducing the goodwill
             generated between both parties for what would be a fairly modest amount of
             money’ (CW Consultant and Assessor).
230   It is perhaps significant that as many as 3 of the 6 institutions we visited said that
      they would not have participated had a fee been imposed at the time. CW staff also
      noted the difficulty of defining boundaries between CWQA and ‘mainstream’ work
      (see paragraph 225 above).
231   Among those at least willing to consider the idea, the reservations expressed
      included the following:-
          ‘It would be possible to charge for the trophy itself, but the bureaucracy and
           risk of alienation outweigh the benefits’ (CWQA Co-ordinator).
            ‘If a fee were to be introduced, it would be essential to identify clearly what was
             being purchased and have a business case to justify the sum charged’ (Senior
             Manager in college).
            ‘Traditionally CW has adopted a partnership approach. Charging a fee would
             risk compromising goodwill’ (CW Consultant).
            ‘Schools might consider paying for the initial assessment, but too few are
             undertaken for it to be a viable commercial proposition’ (CWQA Co-ordinator).
            ‘Any decision on charging would have to be agreed and applied on a pan-Wales
             basis’ (CWQA Co-ordinator).


232   The key points from this chapter are these:-
          although only two schools currently have completed both modules of the
           award, around a half of all institutions in Wales have made a start;
            the main reason for non-participation is concern about the amount of staff time
             needed to work towards the award;
            the benefit from participation cited most often was identifying strengths and
             weaknesses and benchmarking against national standards;
            both CW and institutions have some reservations about the cost:benefit ratio;
            there was little enthusiasm for charging a fee for participation in the award.

ASW Consulting                                 16                  Final Report – January 2006
Careers Wales                                                               Evaluation of CWQA



301     This chapter focuses on the scope and content of the award. The points discussed
            overall approach;
            the Principles and Criteria;
            boundaries between the two modules;
            relevance of model to different types of institution;
            Welsh language;
            ACCAC frameworks.


302     Perhaps the most significant feature of the design of the award is that it stresses the
        ‘journey’ (progress from where we are to where we might reasonably aspire to) rather
        than a ‘gate’ (which you either pass through or not). Similarly, the CWQA is highly
        participative across the institution as a whole, rather than a ‘project’ which could be
        undertaken by a single member of staff. This is a resource intensive model, both for
        Careers Wales and the institution. An alternative approach would for CWQA to
        adopt a compliance model with more of a ‘tick box’ approach.
303     We sought the views of schools and college staff on whether they would favour a
        shift in the overall design of the award. Questionnaire responses are summarised in
        Table 3.1 below.
Table 3.1:    Views on an alternative ‘compliance’ model

Yes, would favour a shift towards a compliance model                           12

No, would prefer to retain existing approach                                   19

Not sure                                                                       2

Total                                                                          33

304     The headline finding is that a clear majority favour the existing approach. However,
        it is perhaps significant that as many as 12 out of 33 (around a third) would favour
        a shift towards a simpler model. It will be recalled that all those ‘voting’ on this
        point are participants in the award whose commitment can reasonably be taken for
305     Several respondents to the questionnaire added comments on this point. Given that
        we also had rich material from our fieldwork visits, we reviewed both sets of
        qualitative data together. This is now summarised.
306     In response to open questions as to whether the current model is ‘along the right
        lines’, the typical response was ‘yes’, but that some streamlining or rationalisation
        would be welcomed. Selected comments include:-
            ‘Needs a rethink, taking into account availability of resources and its
             ‘bureaucratic’ approach (CW Senior Manager).
            ‘Simplify whole process. Some areas extremely detailed’ (participating school).

ASW Consulting                                 17                    Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

            ‘The award aims for ‘perfection’ - greater flexibility would be welcome within
             some elements’ (school Senior Manager).
            ‘Closer alignment needed with the Estyn inspection framework for self
             evaluation’ (participating school).
307   At the next level of detail, there was a divergence of views on the compliance model,
      as reflected in Table 3.1 above. The majority expressed misgivings about any shift
      towards what was characterised as a ‘tick box’ approach.
308   Reservations included the following:-
            ‘A tick box approach would not secure the same outcomes. There would be
             little focus for Teachers and no way of monitoring the impact on students’ (CW
             Senior Manager).
            ‘A compliance model would be driven by bureaucracy - probably at the expense
             of student needs’ (CWQA Co-ordinator).
            ‘Special schools would find a compliance model difficult as they need tailored
             support’ (CW Consultant and Assessor).
            ‘[A compliance model] might work, but danger of losing more than is gained.
             Schools do need help from Consultants if they don’t have systems and policies
             in place’ (CWQA Co-ordinator).
            ‘We needed the support from Consultants. We would have been unlikely to
             cope with a compliance model. We welcomed the regular visits, setting of
             achievable targets and the review process’ (participating school).
            ‘Best schools would sail through with a compliance model but others might
             struggle. If the project management element were removed, there would be a
             danger of some schools stagnating during the process’ (CW Senior Manger).
            ‘Much depends on how schools value the award. At the moment it has
             insufficient standing to allow a compliance model to stand on its own feet. But
             if Estyn or ELWa (WAG) ‘pushed’ the award, that might provide the impetus for
             a compliance approach to take root’ (CW Consultant).
309   However, as already noted, a significant minority favoured a compliance model. The
      following comments provide a representative sample of these latter views:-
           ‘[What is needed is] less paperwork, more support, clarity on the amounts of
            evidence required - in short, an alternative to a portfolio system’ (participating
            ‘Certainly, a 'tick-box' approach would be simpler’ (participating school).
            ‘A tick box approach would be less time consuming to complete and would be
             easier to use across business areas’ (participating college).
            ‘It would be easier to have an evaluation list and tick boxes rather than have to
             collect all the evidence - which makes it as bad as an inspection with regard to
             paperwork involved’ (participating school’.
310   A suggestion made on 3 separate occasions during fieldwork was that the current
      model should be retained for schools and colleges working towards their first
      module, but that a simpler version could be used for the second module - and also
      for reassessments.


311   The core of the CWQA model is a set of 5 Principles, each supported by a list of
      (between 7 and 17) Criteria. These are derived from the learning outcomes for CEG
      and WRE, as defined in the respective ACCAC frameworks.

ASW Consulting                                18                 Final Report – January 2006
Careers Wales                                                              Evaluation of CWQA

312     We canvassed views on the appropriateness of this design through both
        questionnaires and fieldwork discussions.        The headline question was ‘how
        appropriate are the existing set of Principles and Criteria?’ Table 3.1 summarises
        the questionnaire responses. There is a clear majority in support of the existing set.
Table 3.2:    Overall view on appropriateness of Principles and Criteria

Yes, the existing set are appropriate                                               26
No, we have reservations about the existing set                                      3
Not sure                                                                             5
Total                                                                               34

313     At the next level of detail, we sought views on where changes might usefully be
        introduced. The two points explored in detail were:-
            whether any additions were needed;
            whether any of the Principles or Criteria could be omitted without major
314     On the first point, the majority view (by some way) was that the existing set was
        comprehensive and no additions were needed. The only suggestions received (by
        questionnaire or during fieldwork) were that Criteria might also take into account:-
            pupil responsibilities in school e.g. School Council; peer support; support of
             extra curricular activities (all from the same contributor – participating school);
            pupils’ learning of CEG/WRE (school Co-ordinator);
            the student recruitment/enrolment process: ‘this is one of the big differences
             between schools and colleges’ (FE respondent);
            succession planning for Careers Co-ordinators (CW Consultant and Assessor).
315     Turning to the second point (potential rationalisation), there were three types of
        contribution. The most numerous were those arguing for a general ‘simplification’,
        ‘rationalisation’ or ‘slimming down’. Examples of these contributions include the
            ‘Please don't add any more as the current batch are difficult enough to achieve
             at the moment’ (participating institution).
            ‘There is a huge amount in the Award for institutions to 'grapple' with. It
             demands too much paper work - especially in the light of the Teaching Review’
             (participating institution).
            ‘There are too many elements in the Principles and Criteria. They are in need
             of rationalisation’ (school Co-ordinator).
316     The middle group provided comments on specific elements. These will be helpful in
        taking forward the work of this evaluation and we therefore give a full account of
        them, as follows:-
            ‘It should be taken as read that 1.15 and 1.16 are covered by all institutions’
             (participating institution).
            ‘Combine Criteria 2.1 and 2.4. Why have Criteria 1.12. and 2.9 as separate
             categories?    Combine 1.15 and 1.16’ (all 3 suggestions from the same
             contributor in a participating school).
            ‘Principle 1 (and Criteria) is in need of most modification to suit colleges –
             needs to reflect the vocational nature of the college delivery. The college found
             the ACCAC wording difficult to understand’ (Consultant comment following
             visit to FE college).

ASW Consulting                                19                  Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

            ‘Amalgamate Principles 2 and 3. If an establishment has Investors in People
             then there should be a dispensation to waive the staff management element of
             CWQA. This would provide some valuable ‘breathing space’ to allow the college
             to concentrate on the CEG-related criteria’ (participating college).
            ‘Omit 2.9, on the grounds that it is covered in 1.13. Criteria 4.8, 4.9 and 4.10
             are all very similar’ (participating school).
            ‘Principle 1 could be reduced. Principle 4 is huge across all business areas of a
             college. This is very different in schools’ (participating college).
            ‘There should be greater emphasis on the first section [Principle], as teaching
             and learning is the main thrust for education, but less emphasis on Sections
             2,3 and 4. Section 5 [evaluation] needs revising’ (participating institution).
            ‘Principles 1 and 5 are the most difficult, with the evaluation section
             particularly difficult for schools’ (CW Consultant and Assessor).
            ‘Principles 2 and 3 are similar and can overlap - potential for amalgamation?’
             (CW Consultant).
            ‘Principle 2 needs diplomatic handling as some criteria could imply criticism of
             senior management. Principle 3 has some overlap with Investors in People’
             (CW Consultant).
317     The smallest group of respondents argued (albeit with some passion and conviction!)
        that the current set had a validity which should not lightly be tinkered with. The
        following two quotes gives a flavour of these contributions:-
             ‘There should be no watering down of the Award. The high standards are
              essential if it is to retain its high esteem’ (CW Consultant).
            ‘Nothing [should be changed]. To ensure rigour and high standards the award
             should not be reduced in any way’ (participating institution).
318     One contributor added this helpful postscript:
        ‘The Principles and Criteria have been very useful for the initial development of the
        award, but it may be time for some rationalisation now. There is a need for a ‘purist
        versus pragmatist’ discussion. This should involve Teachers as well as CW staff’.

319     The award comprises two modules (CEG and WRE) which can be tackled singly or in
        parallel. Another key question about the model design was whether the two
        modules were well defined and mutually supportive, or whether there were issues
        around the boundaries between the two.     Table 3.3. below summarises the
        responses to the headline question.
Table 3.3:    Views on clarity and appropriateness of boundaries between the two
The scope of the two modules is well defined                                       15
The boundaries between the two modules needs further thought                       16
Total                                                                              31

320     The above table suggests that opinion is almost equally divided on this point. At the
        very least, this suggests that there is some unease about the current boundaries.
        However, analysis of the fieldwork discussions (alongside the supporting comments
        from the questionnaire) suggests that, in practice, the position may be more stark
        than the table suggest. Of those providing qualitative comments, 4 were predicated
        on the status quo, but no less than 21 argued for change.

ASW Consulting                                20                 Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

321     One argument for retaining a clear boundary was from a post-16 institution:
        ‘It is difficult in FE to combine CEG with WRE. There is no PSE time in college, only
        tutorial time, but this is non-funded and is very overloaded already.’
322     The following provides examples of contributors arguing for some rationalisation
        between the two modules. They are all from institutions which are participating in
        the award :-
            ‘I think CEG/WRE/PSE are interlinked, as they are skills as well as ‘content’
             based. I would welcome one award to encompass all three as I feel that PSE
             would benefit from the same rigorous standards.’
            ‘Many elements of the CEG and WRE are interlinked and trying to unpick
             them is a difficult process.’
            ‘There are several similarities between the awards – the same evidence is
             required for both. There are also several instances when PSE topics are also
            ‘There are implications here within The Learning Country and Learning
             Pathways. Rather than two awards, one would be more appropriate, as WRE
             and CEG are requirements for each child’s learning pathway.’
            ‘Why distinguish between CEG and WRE? The crossover is such that I believe
             there should only be the one: namely Careers and Work Related Education.’
            ‘It would be more effective in terms of value and time to produce one file for
             both CEG and WRE. Some work is duplicated.’
            ‘Need there be boundaries? Isn't this historical? Surely WRE is an integral
             part of CEG?’
            ‘Our preparatory work for the WRE module requires us to re-submit evidence.
             Accreditation of prior material should be considered if schools apply for the two
             modules separately.’
            ‘Difficult to know which elements are the responsibility of the Careers Co-
             ordinator and which fall to the WRE Co-ordinator where the two posts are held
             by different people.’
            ‘Why two awards? Why not one, say, ‘Careers and Work Related Learning’?
             This would make the procedure less complex. Instead of differentiating
             between CEG and WRE, we should recognise that they have many similarities’.


323     The same basic structure and process is adopted for all institutions, whether pre-16
        or post-16; mainstream or specialist. Participants were asked their views on the
        appropriateness of this design principle. In short, does CWQA work equally
        effectively in all institutions? The headline responses from questionnaires are set
        out in Table 3.4. below.
Table 3.4:    Views on appropriateness of single model for use in diverse types of

Yes, the single approach works well                                              20
No, the single approach is not always effective                                  6
Not sure                                                                         8
Total                                                                            34

ASW Consulting                                21                 Final Report – January 2006
Careers Wales                                                           Evaluation of CWQA

324   It will be apparent that the majority view is that the single approach is working well.
      There is also evidence of informal flexibility being adopted within the single national
      model. For example:-
            ‘Some adaptations have taken place at local level (e.g. for special needs
             schools). There would be a risk of complication and confusion if different
             models were introduced nationally’ (CW Consultant).
            ‘Post-16 has to be regarded differently to recognise internal structures but we
             found the award flexible to accommodate this (participating college).’
325   However, the high level of satisfaction with the single model, as implied in Table 3.4
      may be a misleading picture, as those agreeing with the proposition were largely
      from mainstream schools.         Satisfaction is far less clear-cut from post-16
      institutions, special schools and other units, although the numbers of respondents
      in these categories is small. Our fieldwork would support this latter view.
326   It would, perhaps, be safer to say that the principle of a single model is widely
      supported, but in practice the need for customising may have been underestimated
      thus far. The following are some of the contributions from post-16 institutions:-
            ‘Customising needed for vocational courses – and general approach to evidence
             gathering in FE.’
            ‘Support in interpretation/understanding is essential - language/terms not
             always appropriate for FE.’
            ‘Award needs to be structured to take account of different types and structures
             of FE colleges.’
            ‘The principles and criteria are very biased towards schools rather than an
             institution that is primarily vocationally based.’
327   There are fewer comments from special schools and other units.          The following
      provide a flavour:-
            ‘Some elements are less relevant for our special school.’
            ‘Given the specialist nature of special needs schools why should there be the
             need to produce the same evidence as mainstream?’


328   We also asked for feedback on the extent to which the CWQA was accessible through
      the medium of Welsh. One point commanding general agreement among Welsh
      speakers to whom we spoke was that the terminology used in the CWQA
      documentation is highly technical and often not easy to translate into Welsh.
329   On other points, views were more divergent. Two respondents (both school Co-
      ordinators, but from different areas) indicated broad satisfaction with the current
           ‘good Welsh language provision’ ;
            ‘every aspect of the award dealt with through the medium of Welsh, although
             some of the supporting CW materials were not (e.g. a DVD to prepare young
             people for interviews)’.
330   A further two respondents were more critical, citing difficulties in accessing Welsh
      speaking Consultants and Assessors; and concern over some of the documentation
      (including electronic versions) not being available in Welsh.

ASW Consulting                               22                 Final Report – January 2006
Careers Wales                                                               Evaluation of CWQA


331   Although it was not a point we explored directly, issues around ACCAC frameworks
      and future changes anticipated, were raised several times, not least in the context of
      the boundaries between CEG and WRE and the importance of any future changes to
      CWQA being compatible with ACCAC developments. It is perhaps important to note
      that we were told many mutually exclusive versions of what was to happen. There
      appears to be much confusion across institutions and CW companies, which in turn
      is making future planning more complex and confusing than it need be.
332   Our discussions with ACCAC confirm that the CEG and WRE frameworks are to be
      merged. The timetable is planned as follows:-
            consultation with schools in Summer 2007;
            introduction of frameworks to schools at the start of the 2007/8 academic year,
             for preparatory work;
            full implementation in the academic year 2008/9.
333   We further understand that:-
            it will be a much smaller document than those being used currently, and with
             a simpler set of priorities.     The current CEG framework is seen as
             overburdened with ‘general statements’ that need substantial refinement;
            it is possible that the term ‘careers’ will not appear in the title;
            its design will be informed by Learning Pathways developments;
            the new framework will not contain any learning outcomes as currently used.
             They will take the form of a list of skills to be developed by students and will be
             mapped against the Learning Core;
            the scope of the new framework will be set in the context of the Teachers’
             Workload Agreement;
            in general terms ACCAC would like to see an alignment of PSE with CEG/WRE,
             but this does not necessarily mean a merger. Indeed, there are no plans to do
             so for the foreseeable future.


334   The key points from this chapter are these:-
            many participants feel the current model should not be tinkered with, but
             others find it resource intensive and argue for some streamlining;
            the basic design features (Principles and Criteria) are widely supported,
             although there is scope for rationalisation;
            there is a degree of overlap between the CEG and WRE modules;
            there is a case for customising the model more to cater for post-16 and
             specialist institutions;
            there is considerable confusion about the future plans for ACCAC frameworks.
             Our understanding is that a merged CEG/WRE framework is to be introduced
             in 2007/8, but that there are no current plans to combine it with the PSE

ASW Consulting                                 23                   Final Report – January 2006
Careers Wales                                                             Evaluation of CWQA



401   This chapter explores points concerned with implementation and delivery of the
      award. The aspects discussed are:-
            feedback on the process in general terms;
            the role of Careers Wales;
            the three stage process;
            consultancy support;
            assessment process;
            local and national co-ordination.


402   Both in the questionnaire and – especially – during fieldwork visits, we sought an
      overview about how the process ‘felt’ from a user perspective. Wherever possible, we
      have linked the points made in responses to these open questions with the most
      relevant section in this report, to assist readers with accessing all the points on the
      same topic. However, this runs the risk of failing to give a sense of the feedback as a
403   In this section, therefore, we present a flavour of the general comments. In fact, a
      very clear picture emerges. The sense is that:-
            the award is something of a gold standard – difficult to obtain but highly prized
             by those who attain it;
            support from Careers Wales is professional and generally much appreciated;
            (but) the process is perceived as time consuming and overly complex.
404   We now provide a cross section of comments to illustrate the general picture (all
      from participating institutions):-
            ‘I think this award provides an accolade for participating institutions. It is a
             marked improvement upon the award it replaced - which was local. This gives
             consistency across Wales. The award should not be devalued.’
            ‘This is a rigorous process which results in a robust evaluation of all aspects of
             CEG within the establishment. The level of support from CW has been of a
             very high standard and the assessment process, whilst challenging and
             demanding, brought an overall focus to the whole process.’
            ‘I found the process of attaining the award very rewarding. It helped focus the
             whole school on CEG/WRE and was a way of recognising the hard work done
             at [name of institution]. The benefits to our pupils of well run CEG/WRE
             learning programmes is immense and I hope the award will continue. It is well
             run and very worthwhile.’
            ‘The help/advice given by Careers Wales was excellent. The problem is the
             amount of work required. At my school, staff are snowed under and therefore
             the idea of spreading the CWQA around the institution is, in my experience,
             unrealistic. I would favour a slimmed down award that can be delivered by the
             Careers Co-ordinator with some help from elsewhere.’
            ‘Assessment process is completely over the top. We are back to producing the
             unmanageable levels of portfolio evidence which nearly brought IiP to a

ASW Consulting                                   24               Final Report – January 2006
Careers Wales                                                                   Evaluation of CWQA

              standstill. The emphasis should be as with Estyn - come and see if what we
              say about ourselves is true!’
             ‘The assembling of the CWQA portfolio and meetings with Consultants took up
              an enormous amount of allocated careers time so that development of the
              programme was put 'on hold'. I would prefer a ready made structure,
              preferably electronic, with tick boxes and references to where evidence is kept,
              rather than assembling a portfolio. This may require additional time spent in
              school by the Assessor finding the evidence but would show a working system
              rather than a manufactured portfolio. Having completed the CEG award I
              would be extremely reluctant to undertake the WRE award because of the
              considerable amount of time it would involve.’
             ‘I feel that taking part in the assessment for CWQA has been a valuable and
              worthwhile experience. However, I am concerned about the outcomes when
              Teachers' workload is reassessed - will Careers Education still have such a
              high profile?’


405      The briefing paper     pointed out that the award was designed and launched by
         Careers Wales and       that their role now extends across marketing, support for
         institutions through   consultancy and training, assessment of evidence, and award of
         the certificate. The   question arises as to whether it is appropriate that the same
         organisation should    fulfil all these roles: in short, are there any issues around role
406      The table below summarises the responses to this question.              The headline
         conclusion is that participants are not overly exercised by CW’s multiple roles.
Table 4.1:      Views on role differentiation

 There are no issues around role differentiation - CW should continue as now                   30
 It would be more effective to split the role in some way                                       3

 Total                                                                                         33

Note:    Two respondents had not filled in this summary box, but had ticked ‘CW to retain’ for each of
         the individual statements.
407      The following comments (all from participating institutions) are examples of those
         arguing in favour of the status quo:-
             ‘It is too complex to split areas of the award to another organisation.
              Assessment is undertaken out of area - therefore there is no complicity.’
             ‘Careers Wales are doing an excellent job - why change it? This award works!’
             ‘Schools/colleges know of CW and they are seen as the main body for
             ‘It would seem more sensible for one organisation to retain these
              responsibilities. The aim is to raise standards of CEG/WRE provision and this
              is best co-ordinated by one organisation.’
             ‘CW is most certainly the most appropriate organisation to do this. It’s what
              they are about, have all the resources, and have good network links (WAG,
              Estyn, ACCAC).’
408      The following comments are from those who have considered potential alternative

ASW Consulting                                    25                   Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

            ‘LEAs are a possibility but with 22 there would be difficulties around
             consistency and economies of scale. It is unlikely that any other body could
             deliver as economically’ (CW Senior Manager).
            ‘Specialist CEG knowledge makes CW the best placed to deliver this. The only
             other possibility is Estyn, but they would opt for a compliance model. In any
             case there are very few people within Estyn with a careers background’ (CWQA
            ‘WAG could take over the assessment process and issuing of awards’
             (participating institution).


409   Attaining the CWQA is a three stage process, supported by Careers Wales
      Consultants. We asked participants to provide feedback on their experience of each
      stage in turn.
410   The stages are:-
            1: Self-assessment and action planning;
            2: Commitment and development;
            3: Assessment and moving on.
Stage 1: Self-assessment and action planning
411   The majority of the comments received on this stage stress its importance and the
      extent of the challenge; support from CW Consultants was also much welcomed.
      The following are examples of this type of response:-
          ‘This is probably the most difficult stage as there is so much to do and it
           involves the whole of the staff.’
            ‘This was very useful to establish foundation of where institution was in terms
             of CWQA.’
            ‘Consultancy process was excellent. We worked well together and the school
             benefited from her advice. A very useful process.’
            ‘Stage 1 dovetails well into our ongoing process of self evaluation/action
412   More individual comments included the following:-
            ‘Support from [named CWQA Co-ordinator] was crucial in this early stage, as
             we grappled with the ideas and principles. A small group of Careers Co-
             ordinators worked together, sharing ideas and good practice. This made the
             award easier to achieve as we were able to help each other.’
            ‘Consultants have been excellent throughout process. However, lack of clarity
             in criteria presents issues in identifying appropriate evidence and quantities of
             evidence, for both Co-ordinators and Consultants.’
            ‘There appears to be an overload of paper work in this section which could
             easily be streamlined and thus speed up the process overall.’
            ‘Perhaps a more structured timetable [was needed] to keep the lead Co-
             ordinator focused on the award.’
            ‘The audit is helpful in seeing the whole picture. A standardised format would
             have helped, as each Consultant had different ideas on this. There were also
             language problems - Teachers had difficulty understanding learning outcome

ASW Consulting                                26                 Final Report – January 2006
Careers Wales                                                                       Evaluation of CWQA

Stage 2: Commitment and development
413       Fewer comments were received on the second stage, probably for the simple reason
          that fewer respondents had had direct experience. In practice, they followed a
          similar pattern to those quoted above for Stage 1.
414       The following is a representative sample:-
               ‘It took some time to get started - it seemed like a huge task; but once started it
                was relatively straightforward. Needed support from Consultant throughout.’
               ‘Once the school is on board and improvements are noticed in areas such as
                PSHE, the staff understand the importance of the input.’
               ‘Good development for whole school. It was a valuable tool for recognition and
                development, but the process was long and time consuming.’
               ‘Time to complete the evidence portfolio was an issue.                   Perhaps more
                Consultant-Teacher time could be included?’
               ‘Development has had to be revisited due to curriculum changes and has
                proven more of a problem than first envisaged in CEG.’
               ‘It is hard to commit to and develop a process which mainly appears to be a
                paper exercise at the end of the day.’
Stage 3: Assessment and moving on
415       Comments on Stage 3 were more mixed.                       Those reporting a largely positive
          experience included the following:-
               ‘Support from Careers Wales was vital, useful to see where programme can be
                improved and developed.’
               ‘Good support from Consultant and CWQA Co-ordinator.’
               ‘[The assessment visit] was a challenging yet satisfying day. I have also had the
                18 month check for CEG and - in preparation for this - I updated the portfolio.
                This is an on-going process and keeps the department on its toes!’
               ‘Rigorous assessments - not relished but it was good to hear the positive
                feedback. Unfortunately we have not 'moved on', owing to so many other
                commitments and a new job description (extra responsibilities).’
               ‘Assessment is the most difficult area to develop as this usually means reports
                and extra work for the whole staff. We have developed an easy numerical
                assessment which is accurate and user friendly.’
               ‘As with Progress File - I feel that centres who achieve the award should be
                monitored annually and assistance provided for departmental development
                planning. ESIS13 provide assistance for other subjects: CEG should be no
416       There were some critical comments, however. The following is a sample:-
               ‘Results of assessment were slow. Feedback needs to be quicker.’
               ‘Very disappointing. External Assessor working to different criteria from 'home'
               ‘Less paperwork needed. Consultant shouldn't have 'just in case' attitude to
                adding evidence. Assessor could spend longer in institution assessing verbally
                rather than assessing paperwork in portfolio.’
               ‘Marketing and promotion support was weak when we received our award; also
                the confirmation of the award was very poorly managed.’

13   Education and School Improvement Service, based in Caerphilly

ASW Consulting                                        27                    Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

            Since 2004 (the award), no contact has been made by Careers Wales or their


417   Feedback from an institutional stance on the support provided by CW was
      summarised in the previous section. This section focuses on the CW perspective
      and in particular issues around the deployment and training of Consultants.
418   CW staff are mainly very positive about experience to date with the award. It has
      enabled them to increase their knowledge and experience base, especially in the
      fields of in-service teacher training and curriculum development. It has also enabled
      them to raise their profile with key agencies such as ACCAC, Estyn and Fforwm.
Deployment of Consultants
419   In practice, it has proved difficult to recruit, train and retain those with the optimal
      blend of skills and experience; and especially those able to operate through the
      medium of Welsh.
420   The approaches taken by different companies to deployment of Consultants vary
      widely. To be specific:-
            Cardiff and Vale deploys two serving Teachers on a part-time secondment
             basis, in addition to a number of its own staff;
            West deploys a team of mainstream Team Leaders and Careers Advisers on the
             basis of providing a few days consultancy input each as part of their workload;
            Mid Glamorgan and Powys used to deploy a larger cohort of external Associates
             but have recently moved to using a much reduced number, while at the same
             time training a group of Middle Managers to become Consultants;
            Gwent uses four external Associates (all with a CW background), in addition to
             internal staff;
            North East has a small, but discrete team of its own staff, some with recent
             experience in the education sector;
            North West have used a mix of external Associates and employed staff in the
             past but are currently using only employed staff who take on consultancy as
             part of their workload.
421   We think it premature to draw any firm conclusions on the relative strengths of the
      various models: in any case, inter-area comparison was not a part of our brief. We
      would merely comment that, on the basis of our fieldwork discussions:-
            Consultants with current or recent direct experience in the education sector are
             especially valued by schools and colleges (an issue of perceived credibility, not
            the majority view among CW managers was that smaller teams with discrete
             roles and clear targets were more effective than larger teams with diffuse roles
             (whilst scoring less on considerations of breadth and skills transfer).
Consultant training
422   Consultant training is provided by the National Co-ordinator. Feedback was very
      positive, although some Consultants said they would have liked the opportunity to
      train in the medium of Welsh; a similar point was made about assessor training.
423   The timing of training for Consultants is another important factor. Those involved
      stressed that the new skills learned need to be put into practice as soon as possible
      after the training; having Consultants trained on a more speculative basis was not

ASW Consulting                                28                 Final Report – January 2006
Careers Wales                                                                       Evaluation of CWQA

       seen as optimal. We understand that there has been a small number of cases where
       repeat training has had to be organised.
424    Mention has already been made of the entitlement of schools and colleges to 6 days
       of ‘free’ consultancy during the period they are working towards the first unit of the
       award.14 Many CW managers feel this position is untenable in the medium term,
       especially if take-up from institutions continues to rise.
425    Indeed, in at least one area, managers put a limit on the number of institutions to
       be supported at any one time. This is entirely understandable from a pragmatic,
       operational perspective, but it effectively rules out a strategy of rapid expansion.
Links with mainstream work
426    Mention has already been made of the near impossibility of defining exactly where
       mainstream support for the CEG and WRE curriculum ends, and discrete ‘CWQA
       consultancy’ begins (see, for example, paragraphs 225 and 230). This makes precise
       costing challenging, to say the least; it also makes it difficult for Middle Managers to
       refuse any reasonable request for teacher or curriculum support, whether made
       under a CWQA banner or not.
427    One way forward (where this is not already the case) would be to include CWQA
       support as part of the annual discussions between CW and institutions on roles and
       inputs (as enshrined in partnership agreements for CEG and EBL services), rather
       than view the two processes as parallel but distinct exercises.15
428    CW managers in some areas felt there was scope for mainstream Careers Advisers to
       be better briefed on CWQA developments. It was less than professional when CW
       staff were unable to speak knowledgably about the award, even though this may not
       be their particular specialism.
Networking and self help
429    Two fieldwork discussions in different areas commented on the increasing
       knowledge and skills held by school and college staff in institutions which have
       already achieved the CWQA. There was a strong prima facie case for establishing
       more self help groups and informal mentoring arrangements, thereby making better
       use of teacher/lecturer expertise and freeing up some of the CW consultancy time
       from one to one support.

430    Up to 3 days are required to carry out assessments on schools and colleges
       recommended for the award. Clearly it is important that those carrying out the
           are independent of the institution;
            are fully acquainted with all aspects of the award;
            carry credibility with both the institution and CW.

14Up to 8 days are offered if both modules are being tackled.
15In this section (Consultancy) and the one which follows (Assessment) some comments are added from an
ASW perspective. Our sense was that splitting the discussion on these detailed points between these sections
and the respective passages in Chapter 7 (Recommendations) would give rise to confusion – or overlap.

ASW Consulting                                      29                    Final Report – January 2006
Careers Wales                                                                         Evaluation of CWQA

Deployment of Assessors
431    In practice, it is difficult to find people who meet these criteria, other than
       experienced CWQA Co-ordinators and/or Consultants from other areas.16 Until
       recently an informal arrangement applied through which Co-ordinators or
       Consultants from one company undertook assessments in another, on the
       understanding that a reciprocal service would be supplied at a future time.
432    This has recently broken down, not least since the numbers of institutions going
       through the system vary significantly between CW companies. In practice, therefore,
       some companies are always consuming more assessor time than they are making
       available to others – or vice versa (making more time available than they recouped).
       This was felt to be inequitable and so the pattern now is for each company to pick
       up the costs of assessments for institutions in their own area.
433    As noted, the situation is made more difficult by the lack of an available pool of
       individuals competent to carry out the task. In a minority of instances, assessments
       are carried out by Consultants and Assessors from the same company but who have
       no day to day professional involvement with the institution under review. However,
       in most instances, assessments are carried out by staff of other CW companies who
       take annual leave from their own company and are paid on a freelance basis by the
       commissioning company.
434    This was widely felt to be an unsatisfactory situation: assessment arrangements
       would benefit from a pan-Wales decision (at Chief Executive level). Whilst it would
       be simpler to operate on the basis of an inter-company fixed fee per assessment
       carried out by staff from another area, this would always result in the greatest costs
       falling on those areas with the largest number of participating institutions (a
       negative incentive, as regards marketing to non-participants). A linked factor is that
       those companies employing the most experienced and sought after Assessors would
       actually enjoy less of their professional contribution in-house (the income generated
       would not entirely offset this loss).
435    Another model would be for assessment to be managed on a single, pan-Wales basis
       – perhaps through CWA or led by one company on behalf of all. However, this would
       bring its own controversies, not least the funding arrangements. If it were funded by
       a levy on all companies costed pro rata to the number of participating organisations,
       the net result would be similar to the status quo, but if it were financed by a flat rate
       contribution, there would always be net winners and net losers.
436    Ironically, the arrangement most criticised at present (i.e. deploying Assessors                 from
       the same company but with no day to day involvement with the institution)                        may
       offer a pragmatic way forward in the short term, although far from ideal.                         The
       criticisms we picked up of this arrangement tended to be from professional                       staff
       elsewhere, rather than institutions in the area concerned.
Assessor training
437    Similar issues (re apportionment of costs) arise over the training of Assessors. The
       practice was for ‘home’ companies to fund their own assessor training.
438    However some Chief Executives are now questioning whether they obtain value for
       money from this investment, given the fact that benefits accrue either to the ‘other’
       companies and/or to the individuals themselves (if they are taking annual leave to
       carry out assessments elsewhere on a freelance basis). This will need to be
       considered alongside the ‘deployment’ issue, described above.

16 Assessors able to operate through the medium of Welsh were in particularly short supply. Those in post
often have to travel long distances.

ASW Consulting                                       30                     Final Report – January 2006
Careers Wales                                                                        Evaluation of CWQA

Assessment panels
439       Panels are convened on a national basis with membership drawn from a list of
          senior individuals from contrasting backgrounds with a dispassionate but informed
          view. The process was characterised as rigorous but fair by participants (see
          comments on Stage 3 of the award process in the earlier section of this chapter).
440       There were criticisms as well, however. We refer, for example, to:-
               apparent inconsistencies in the interpretations of local Consultants and
                external Assessors;
               delays in receiving feedback;17
               emphasis on the written word more than observed practice.
441       Our own view was that the assessment process as a whole (not the Panel in
          isolation) seemed on the heavy side. There would appear to be scope for freeing it up
          a little, without losing focus or rigour. For example, the balance of assessment visits
          could be shifted away from reviewing policies and systems themselves and more
          towards discussing their impact with staff and students; and the role of the panel
          could be shifted towards moderating Assessors’ decisions rather than being the
          principle decision maker.
442       In the short term, what is required is a change in emphasis rather than fundamental
          reform; doubtless the Quality Award Group would be best able to recommend ways
          of simplifying the process and generally lightening the load.
443       At a second level of detail, we note a small point about the Assessment Reports, as
          presented to the Panel. Under Section 2 (Methodology) there is an item about
          student focus groups, where Assessors feed back on the views put forward by young
          people during their visit. The numbers and year groups of the young people are
          then listed, and the section ends with the standard wording: ‘the make up of these
          groups reflects the ability, ethnic and gender mix represented in the school [or
444       In practice, the numbers seen are often very small. For example, in 3 of the 6
          reports we reviewed, the number of students met by Assessors were 16 or under –
          and these were drawn from Years 7 to 13 (i.e. an average of just over 2 students per
          year in a 11-18 school). Clearly the sample could not be deemed representative. We
          would suggest the wording be omitted unless it can be demonstrated to be an
          accurate reflection.


445       There are four further elements of the delivery structure for CWQA which have not
          been discussed specifically thus far. These are local CWQA Co-ordinators, the
          National CWQA Co-ordinator, the national Quality Award Group, and the CEO
          Champion for CWQA. We now comment briefly on each of these in turn.
Local Co-ordinators
446       Local Co-ordinators are responsible for the promotion of the award in their area,
          deployment of Consultants, organisation of assessments and generally acting as the
          focal point for local CWQA activities. Feedback on their work was almost universally
          positive across institutions, Consultants and Senior Managers.
447       The boundaries around their roles, and the other duties they perform within their
          employing company, vary widely. This is not necessarily a ‘problem’, although the

17   There can also be delays between local assessment visits and panel meetings.

ASW Consulting                                          31                    Final Report – January 2006
Careers Wales                                                               Evaluation of CWQA

      ambiguity noted several times around the definition of CWQA activities is reflected in
      their job content.
National Co-ordinator
448   The enthusiasm and dedication of the National Co-ordinator was widely recognised.
      The role was seen as particularly important in the earlier stages of the award, but
      perhaps less crucial during the steady state (if such a position is ever reached in the
      future!). The recent reduction in hours from 3 days to 1 day per week in the
      national role was regretted by some interviewees, but seen as largely inevitable by
449   The main benefits from the role were perceived as:-
            providing a single point of reference for national activities (e.g. organising panel
            heading up development work (e.g. working on the WRE module);
            taking on tasks commissioned by the Quality Award Group, ensuring progress
             does not stagnate between meetings;
            keeping a watching brief on consistency of interpretation between areas.
450   Ironically, the very enthusiasm and commitment of the National Co-ordinator does
      bring its own difficulties. Not least it has tended to mask the actual resource
      implications of the reduction in her number of days in the national role; some
      interviewees felt the current position was not sustainable. Others pointed out that
      demands for ever increasing quality can place excessive demands on the system;
      occasionally a more pragmatic approach may bear dividends.
Quality Award Group
451   The Quality Award Group brings together the National Co-ordinator, CEO Champion
      and local CWQA Co-ordinators and meets 10 times per year. Its explicit role is to
      take forward the CWQA agenda on a pan-Wales basis but it also fulfils an important
      role in promoting the exchange of good practice, networking, and acting as a
      sounding board for new ideas.
452   Until recently it was facilitated by the National Co-ordinator but, at the suggestion of
      the CEO Champion, the group now elects a Chair from within its membership
      (currently the CWQA Co-ordinator from CW Cardiff and the Vale). Members of the
      group spoke positively about its contribution and welcomed the creation of a Chair
      distinct from the National Co-ordinator.
453   The two suggestions noted about the group were firstly that it should widen its remit
      to include curriculum development and teacher training (and perhaps Progress File);
      and secondly that it should meet less frequently (perhaps quarterly).
CEO Champion
454   The Chief Executive of CW Gwent acts as the CEO Champion for the CWQA and
      Quality Award Group. This is not a role exclusive to the award: all CW working
      groups have a nominated company Chief Executive whose remit is to provide a clear
      steer on strategic issues and act as a point of communication between the Chief
      Executives Group and CWA Board on the one hand, and the specialist working
      group on the other.
455   Group members welcomed the ‘hands on’ approach of the current Champion. This
      ensured that issues were prioritised (even though this was not always a comfortable
      process!) and CWQA developments positioned appropriately.

ASW Consulting                                 32                  Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

456   One interviewee questioned whether 3 separate roles were all needed in the current
      structure (i.e. National Co-ordinator and Quality Award Group Chair and CEO
      Champion). This was not a widely held reservation, however.


457   The key points from this chapter are these:-
            the award is something of a gold standard - difficult to obtain but highly prized
             by those who attain it;
            support from Careers Wales is professional and generally much appreciated;
            (but) the process is perceived as time consuming and overly complex;
            the case for splitting up the multiple roles currently held by CW is not
            there is some unease within CW about capacity issues (i.e. whether there are
             sufficient Consultants – and sufficient funding – to support a wholesale
             expansion in participating institutions);
            the boundaries between ‘CWQA support’ and ‘mainstream support for CEG and
             WRE’ are far from clear;
            the training and deployment of Assessors – and perhaps the assessment
             process itself – would benefit from a rethink on a pan-Wales basis;
            current arrangements for local and national co-ordination are largely
             supported, although there is scope for reducing the number of Quality Award
             Group meetings.

ASW Consulting                                33                 Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA



501   This chapter focuses on the impact of the award on schools and colleges as
      institutions, as opposed to the impact on individual learners; the latter is considered
      in the next chapter.
502   Impact on institutions is discussed by reference to the following:-
            overview;
            content and delivery of CEG and WRE programmes;
            planning, management and evaluation processes;
            profile and status;
            involvement of Senior Managers and external partners;
            institutional objectives;
            national standards;
            staff development;
            other impacts.


503   Assessing the impact of the CWQA on institutions was one of the central objectives
      for this study. In practice, this is complex since the reasons for an institution’s
      relative success or failure will be attributable to several intricate and subtle factors
      (including catchment area, tradition, calibre of staff, and many other influences and
      variables), only one of which will be the award. Indeed, even distinguishing the
      impact of the award itself (defined exclusively), from the impact of the CEG and WRE
      programmes generally is far from straightforward.
504   This relatively small scale study considers impacts from a mainly qualitative
      perspective, with data generated from written comments and discussions during
      fieldwork visits. This approach enables general conclusions to be drawn but cannot
      replicate a more quantitative study with large data sets and control groups. This
      chapter needs to be read with this important caveat in mind.
505   We present first the headline data from the questionnaires. Commentary on the
      significance of the content of these tables is added in the relevant sections below;
      but first we explain the approach used.
506   The first table summarises participants’ views on the impact of the award by
      reference to 5 possible effects on institutions, as listed in the first column (e.g.
      promotion of staff development). The second column indicates the number of
      respondents commenting on each potential effect (33, in the case of promotion of
      staff development). The next 3 columns summarise the views of participants as to
      the assessed impact of the award on each item (i.e. is the effect major, modest or
      zero); figures in each cell indicate the ‘votes’ for each rating (hence 12 respondents
      thought CWQA had a major effect on promotion of staff development; 16 thought it
      had a minor effect and 5 were not convinced it had any effect at all).
507   The ‘notional score’ in the final column is achieved by:-
          ascribing 2 points to each ‘major’ rating and 1 point for each ‘modest’ rating;
            adding up these two raw ‘scores’ (24+16=40);

ASW Consulting                               34                 Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

            in parallel, calculating the maximum total theoretically achievable for each
             potential effect. In the case of staff development this is 66 (if all respondents
             assessed the impact of CWQA as ‘major’, the total score would be 33x2=66);
            dividing the actual subtotal into the maximum possible score and converting
             this into a percentage (40÷66x100=60.6%, rounded up to 61%).
508    This is a rough and ready way of assessing the comparative weight attached to each
       potential impact. The actual percentages have no absolute significance; but there
       are conclusions it would be legitimate to draw from the data presented.
509    For example, it would, be fair to say that:-
            across the factors considered, participants thought the impacts of the award
             were quite high (between 55% and 71%);
            the assessed impact of CWQA on ‘profile and PR’ (71%) was higher than the
             equivalent view on ‘prompting questions in other curriculum areas’ (55%).
510    The three tables below summarise the assessed impact on institutions of:-
            the CWQA as a whole;
            the CEG module specifically;
            the WRE module specifically.
511    Participants were also invited to suggest additional impacts, not included in the
       tables in the questionnaire. In practice, relatively few were suggested - but those
       which were, are added underneath the relevant table.
Table 5.1:     Assessed impact of CWQA on institutions in general terms
                                             NO OF        ASSESSED IMPACT          NOTIONA
                                            RESPONS                                   L
                                              ES       Major    Modest     None    ‘SCORE’
 Helps achieve broader institutional
                                              33         16        14        3        70%
 Raises profile of institution, good PR       33         20        7         6        71%
 Prompts interesting questions in
                                              31         7         20        4        55%
 other curricular areas
 Promotes staff development                   33         12        16        5        61%
 Improves planning and review
                                              33         13        15        5        62%
 processes generally
Other : ‘Key Skills’ (major impact)

Table 5.2:     Impact of CEG module on participating institutions
                                             NO OF        ASSESSED IMPACT          NOTIONA
                                            RESPONS                                   L
                                              ES       Major    Modest     None    ‘SCORE’
 Content of the CEG programme                 33         15        15        3        68%
 Management, planning and delivery
                                              33         12        17        4        62%
 of CEG programme
 Involvement of key partners in
                                              33         10        18        5        58%
 planning and delivery of CEG
 Internal processes re evaluation and
                                              33         16        12        5        67%
 review of CEG programme

ASW Consulting                                35                 Final Report – January 2006
Careers Wales                                                                     Evaluation of CWQA

Others: ‘PSE’, ‘Co-ordinators’, ‘Use of time allocated for careers work’ (all major impact); and ‘Senior
       staff’ (modest impact)

ASW Consulting                                     36                   Final Report – January 2006
Careers Wales                                                             Evaluation of CWQA

Table 5.3:    Impact of WRE module on participating institutions
                                           NO OF          ASSESSED IMPACT           NOTIONA
                                          RESPONS                                      L
                                            ES         Major     Modest    None     ‘SCORE’
Content of the WRE programme                  16          7         6        3          63%
Management, planning and delivery
                                              17          9         5        3          68%
of WRE programme
Involvement of key partners in
                                              16          8         5        3          66%
planning and delivery of WRE
Internal processes re evaluation and
                                              17         10         4        3          71%
review of WRE programme

512   The tables confirm that CWQA has a positive impact on the content and delivery of
      school and college programmes, with ‘scores’ of around 65%. It is, perhaps, a little
      surprising that the ‘scores’ are not higher, but the comments suggest that in many
      cases this is because there was good coverage of CEG and/or WRE within the
      institution even before participating in the award, and so the net additional impact
      was modest rather than substantial.
513   Additional comments to explain the ratings given for the CEG module include the
            ‘Re-written tutorial programme’ (rated a major impact).
            ‘Has made CEG an element in tutorial programme’ (major).
            ‘Improved programme in content and quality’ (major).
            ‘The CEG programme has been rewritten to cover ACCAC learning outcomes
             and improve learners’ experiences’ (major).
            ‘Gave me a chance to review the CEG programme and consolidate’ (modest).
            ‘Already had RoQA but the CWQA (CEG and WRE) enables the process to be
             reviewed/improved’ (modest).
            ‘Mapping identified duplication of areas across the curriculum e.g. equal
             opportunities, CVs etc’ (modest).
            ‘Audit of departments to ensure all Learning Outcomes covered’ (modest).
            ‘Already covering framework but sharpened up what we do’ (modest).
            ‘Assessment identified some areas for further development’ (modest).
514   Turning to comments on the WRE module, the following is a representative sample:-
            ‘Full audit undertaken - information from all curriculum areas sought’ (major).
            ‘Excellent Estyn report’ (major).
            ‘Adoption of master classes and on-line placements’ (modest).
            ‘Need to focus on building in enterprise/mentoring activities’ (modest).
            ‘WRE undertaken already across most business areas’ (modest).
515    Frustratingly, respondents indicating that the CEG or WRE module had no impact
      on the school or college programme have tended not to add explanatory notes. One
      exception was the following:

ASW Consulting                                37                 Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

      ‘The work takes place in this area whether we work towards the Award or not. We
      are trying to adapt to the Award’s requirements but the process is so complicated
      and boring.’
516   Inevitably, notes from fieldwork visits cover similar ground. The following are a few
      selected additional comments:-
            ‘Significant improvements, especially in the way that the whole school (in terms
             of teaching staff) are now involved. The Deputy and Assistant Heads have been
             encouraged to deliver some of the CEG modules.’
            ‘Programmes are now richer in content and are integrated in a far better way.’
            ‘The CWQA has led to staff contributing from other departments.’


517   The notional ‘scores’ for the impact of the CEG module on management, planning
      and delivery, and on evaluation and review, were 62% and 67% respectively. The
      equivalent ‘scores’ for WRE were 68% and 71%.
518   Additional comments on the impact of the modules were as follows. The types of
      comments were similar for both modules. All those cited below relate to CEG (other
      than the last one):-
            ‘Departmental evaluations and reviews in place. More comprehensive event
             evaluation now undertaken e.g. parental feedback sought’ (major).
            ‘The whole system has been evaluated and improved by using the CWQA
             principles’ (major).
            ‘[Internal monitoring and evaluation processes] reviewed and revised
             substantially’ (major).
            ‘Prevents any laziness in updating and modernising programme. Assessment
             with specially designed certificates increases pupil performance and staff can
             identify weaknesses’ (major).
            ‘The award improved systems already in place’ (modest).
            ‘More members of staff involved’ (modest).
            ‘Processes for evaluation and review were established and implemented. Have
             not been implemented since, though they are available for future use’ (modest).
            ‘Greatest impact is the introduction of Key Skills’ (modest).
            ‘Mainly changes caused by curriculum needs but streamlining and delegation
             have occurred’ (modest).
            ‘Structures formalised and a greater range of evaluation procedures’ (modest).
            ‘WRE cross curricular team revived and used’ (modest).
519   It is significant that respondents thought CWQA had had a beneficial impact on
      planning and review processes across the institution as a whole (i.e. not only in the
      CEG and WRE curricular areas). Table 5.1 indicates a ‘notional score’ of 62% for
      this item.
520   Supplementary comments here include:-
            ‘[Has encouraged the] use of subject audit to stop topics being repeated’
            ‘Helps to focus on appropriate/relevant set of criteria’ (modest).

ASW Consulting                               38                 Final Report – January 2006
Careers Wales                                                                        Evaluation of CWQA

               ‘Issues such as assessment, recording and reporting discussed and included in
                the departmental delivery plan’ (modest).
               ‘Gives a current focus… ensures SOWs18 etc are up to date’ (modest).
521       Data from both the questionnaire and our fieldwork suggests that monitoring and
          review is one of the aspects to benefit most from institutions’ involvement with the
          CWQA. The following are taken from the notes of fieldwork visits:-
               ‘Evaluation has been the Achilles heel but the school is now becoming more
                comfortable with it. It fits in well with the Estyn self-evaluation requirements,
                so constitutes good preparation and experience.’
               ‘Has helped school to evaluate just where CEG is going: it has added some
                discipline into the whole area. An example of this is where the school’s
                evaluation of work experience showed their timing of placing pupils with
                employers around Christmas time was not optimal: employers were too busy to
                give the additional time needed by special needs pupils.’

522       Fieldwork suggested that CWQA had had a noticeable impact on the status and
          profile within the institution, but less so externally. Starting with the internal
          impact, our visits suggested that:-
               working towards the award had brought about greater awareness and
                appreciation of the work of Careers Co-ordinators;
               achievement of the award had resulted in a boost in the confidence (and in
                some instances, status) of the relevant post holders;
               the cross curricular dimension in particular helped to plant CEG and WRE
                themes in the work of colleagues from other subject areas;
               the fact that the award is externally assessed helped to reinforce its status and
523       The picture in terms of external profile was more mixed.                On the credit side, we
          noted that:-
               curriculum advisers from some unitary authorities (non-CEG/WRE specialists)
                had been impressed by the award and encouraged other institutions to
                participate. Carmarthenshire LEA, for example, had included it in their
                Education Development Plan;
               institutions were typically proud to display the Dragon’s Tail plaque;
               award ceremonies were well received. These are mainly internal but we were
                told also of some area events. The latter have a bigger impact but are also
                resource intensive and not always easy to justify on cost:benefit criteria;
               employers had a positive response to WRE developments.
524       However, other people with whom we discussed this point felt the external profile
          was marginal. Two of the schools we visited had included an article about the
          CWQA in newsletters but apparently it had not stimulated much interest among
525       The picture from the questionnaire is more positive. Table 5.1 provides data on the
          impact of the award in raising the profile of the institution as a whole (i.e. not just
          CEG and WRE) externally. The ‘notional score’ (71%) was the highest in this table.
526       Supplementary comments include the flowing:-

18   An abbreviation used to indicate Schemes [sometimes Schedules] of Work

ASW Consulting                                        39                      Final Report – January 2006
Careers Wales                                                                      Evaluation of CWQA

               ‘See website, prospectus. Excellence of delivery is an achievement’ (major).
               ‘The Headteacher is very positive about us achieving the award’ (major).
               ‘Press coverage; award on display’ (major).
               ‘Selling point for school’ (major).
               ‘The ‘all-Wales dimension’ has an impact on parents’ (major).
527       Few explanations were offered by respondents giving a ‘zero’ rating. One exception
          was a comment pointing out that the institution already enjoyed a high profile.


528       Another of the impacts assessed was around the extent to which participation in
          CWQA had increased the involvement of Senior Managers and Governors, and
          improved links with external partners, such as employers.
529       We are satisfied, on the basis of data provided in this chapter and elsewhere, that
          participation in the CWQA has had the impact of increasing senior management
          involvement in CEG and WRE. Many of the comments provided under different
          headings have already touched on this (see for example, the first bullet in paragraph
          516 above). We, therefore, do not repeat this material here.
530       Instead, we add a summary observation from discussions with CWQA Co-ordinators:
          ‘The award has required the involvement of Senior Managers [within institutions] to
          a degree not evident before. In some instances, the award has been a means for
          enthusiastic Careers Co-ordinators to approach management regarding CEG/WRE
          issues and development; but there have also been examples where Senior Managers
          have seen the award as a tool to motivate, and provide a focus for, Careers Co-
531       Turning to Governors, data from our fieldwork visits suggests that they have been
          involved in the CWQA - but only with a light touch. In one of the colleges, Governors
          had been given a presentation up front about the award, but typically the position
          was that Headteachers/Principals provided periodic updates. It was common,
          however, for newly drafted or revised policies to be approved formally by Governing
532       As regards the involvement of (external) partners in planning and delivery, Tables 5.2
          and 5.3 above showed notional ‘scores’ of 58% and 66% for the impact of CEG and
          WRE respectively. Additional comments about the impact of the modules (all CEG
          except the one marked) include:-
               ‘There has been greater involvement of employers and Careers Wales in our
                curriculum’ (major).
               ‘Better integrated programme for everybody concerned’ (major).
               ‘Careers Wales are a key partner …[and] local businesses are very supportive’
               ‘Helped us gain the QiSS19 award as well’ (major).
               (WRE) ‘Pilot of Careers Wales helping with placements for work experience’
               ‘Strong links were already in place but evaluating the links has improved’
               ‘In review stages weaker elements were highlighted’ (modest).

19   Quality in Study Support, awarded by Canterbury Christ Church University

ASW Consulting                                        40                    Final Report – January 2006
Careers Wales                                                                              Evaluation of CWQA

533       The following two comments were linked with zero ‘scores’:-
                ‘I would have appreciated more involvement with the school CA who had little
                 knowledge of the award’.
                ‘In place already e.g. mock interviews’.

534       For Senior Managers, a key determinant of the relevance and effectiveness of CWQA
          will be the extent to which it helps to deliver institutional objectives ‘in the round’
          (i.e. not necessarily associated with CEG and WRE specifically).20 Table 5.1 suggests
          that the award does indeed have such an impact. The ‘notional score’ was 70%, one
          of the highest across the 3 tables.
535       Supplementary comments include the following:-
             ‘Ensured that all departments included a statement in their SOW’ (major).
                ‘CEG and WRE central focus of FE’ (major).
                ‘Increased the learning within PSHE. Pupils take it more seriously!’ (major).
                ‘The CWQA has been used to work towards targets in SIP21 2004/05’ (major).
                ‘Good preparation for Estyn inspection’ (major).
                 ‘It is hard to break entrenched habits and CWQA is no exception’ (modest).
                 ‘[Helped to link] several school objectives’ (modest).
536       The two contributions explaining zero ‘scores’ were as follows:-
                ‘We felt the whole process was too judgemental, inspectoral and critical.’
                ‘Most of this, again, happens independent of the Award.’


537       As regards impact on meeting national standards, the award has undoubtedly raised
          awareness of – and compliance with – ACCAC frameworks. This was widely accepted
          by all those with whom we discussed the point. Indeed, the ACCAC learning
          outcomes are integral to the design of the CWQA.
538       Preparing for Estyn inspections was one of the reasons for participating in the
          CWQA given by many schools and colleges. Working towards the award was said to
          provide a very useful preparation, not least in portfolio development and collection of
539       On balance, there was a sense of disappointment that Estyn as a body had not
          formally endorsed the CWQA, although some individual inspectors were encouraging
          schools and colleges to participate. CW Co-ordinators and Consultants had been
          surprised by some of the assessments of CEG and WRE provision in local
          institutions given by Estyn inspectors, as the feedback from the latter was
          sometimes at odds with the picture revealed by the CWQA process.


540       Another of the factors explored through the questionnaire was the extent to which
          the award was helping to promote staff development. The notional ‘score’ for this
          criterion was 61%, placing it in the middle band of ratings.

20   And see also the third bullet in paragraph 207 for a view from a non-participating institution.
21   School Improvement Plan

ASW Consulting                                           41                      Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

541   Supplementary comments included the following:-
            ‘On going training for 20 staff annually’ (major).
            ‘Training of form tutors to deliver a portfolio’ (modest).
            ‘The CEG/WRE Co-ordinator has been affected but less far reaching across all
             staff’ (modest).


542   The final impact assessed through the questionnaire was ‘prompting interesting
      questions in other curriculum areas’. The ‘notional score’ here was 55%, the lowest
      rating in the tables, but still suggesting a positive picture.
543   Comments included the following:-
        ‘Greater co-ordination of WRE across curriculum is linking English to work
         experience’ (major).
        ‘All Heads of Department have identified where CEG/WRE is taught in their
         subject’ (major).
            ‘Highlights other areas within PSHE which have not been updated’ (major).
            ‘Raises some transferable points’ (modest).
            ‘Mapping of PSE across curriculum’ (modest).
544   Other positive spin offs, mentioned during fieldwork, from participation in CWQA
            networking and sharing of best practice;
            better co-ordination pre and post-16;
            focus on equal opportunities;
            greater coherence with PSE.


545   The key points from this chapter are these:-
          measuring impact of the award in a quantitative way is almost impossible in a
           small scale study such as this, owing to the interplay of other factors and
            nevertheless, there is good evidence of the award having a major impact on the
             quality of CEG and WRE programmes; the effectiveness of planning and review
             processes; raising of the profile of CEG and WRE departments; and actively
             involving Senior Managers;
            there is some evidence – but less compelling – of the award having a positive
             impact on meeting broader institutional objectives; involving governors and
             external partners; raising the profile of CEG/WRE (and the institution as
             whole) in its local community; promoting staff development; and
             networking/sharing of best practice.

ASW Consulting                                 42                 Final Report – January 2006
Careers Wales                                                                  Evaluation of CWQA



601       This chapter assesses the impact of the award on learners – the pupils and students
          in institutions participating in the award. The sections are headed:-
               methodological challenge;
               learner views;
               third party views - questionnaires;
               third party views – fieldwork.


602       The introductory comments to the previous chapter drew attention to the difficulties
          in distinguishing the impact of the award, from the impact of the CEG and WRE
          programmes it is underpinning and quality assuring. This point applies even more
          to assessing the impact of the award on learners.
603       We were unsure, when writing our proposal for this study, whether it would be
          possible to distinguish any impacts among learners which were unambiguously
          attributable to the award. Nevertheless, we raised the question in the briefing paper
          and invited respondents to indicate their own views and supporting evidence; we
          also asked CW and institution staff whom we met during fieldwork; and, of
          particular interest, we asked young people themselves through discussion groups
          kindly organised by three of the participating institutions. This brief chapter
          summarises all the comments received.


604       As mentioned above, we took the opportunity to speak with groups of learners about
          their experiences with CEG and WRE in the hope of identifying any benefits
          attributable to the award. The learners were drawn from the following year groups
          in the 3 schools22 where we were able to canvass views at first hand:-
               a mixed group from Years 9 to 13;
               two groups comprising students from Years 10 and 12;
               a group of Year 11 students.
605       The awareness of careers and work experience issues among the students we met
          was outstanding. They were able to describe their entitlements and talk through
          several examples of activities they had participated in. These included Enterprise
          Week, The Trading Game, The Real Game, Young Enterprise, tutorials, mock
          interviews, work experience placements (some had fixed their own), development of
          CVs, Progress File, careers databases and software, information on the local labour
          market, and talks by external speakers.
606       In short, there was no doubt that they were well informed and articulate. The
          difficulty lies, of course, in determining whether or not that is attributable to the fact
          that their institution is participating in the award; and, if so, whether the impact is
          considerable, modest or negligible.

22   For further information, see Appendix B.

ASW Consulting                                    43                  Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

607   What cannot be disputed is that they had benefited from a well designed and
      imaginatively delivered programme – but whether that would have been delivered in
      the absence of the award must remain a subjective judgement, in the absence of a
      much larger study. As a matter of fact they did not display much awareness of the
      award (other than a recollection of the Dragon’s Tail plaque!).


608   We turn now to ‘third party’ views (i.e. the perceptions of adults seeking to interpret
      the impact on young people). Looking first at the questionnaire, participants were
      asked to consider the following options and tick one of the adjacent boxes:-
            ‘There are no discernible impacts on learners that can be attributed to an
             institution’s participation in CWQA.’
            ‘There are impacts on learners, but they are indirect (in the sense that everyone
             benefits from an institution with high morale, good PR, clear curricular
             objectives etc).’
            ‘There are direct impacts on learners that can definitely be attributed to an
             institution’s participation in CWQA (e.g. lower drop out, better transitions,
             more appropriate destinations).’
609   Most respondents thought learners would be unaware of the award (other than
      through any publicity surrounding its achievement), but that they would
      nevertheless benefit indirectly. Although 8 respondents ticked the ‘no discernible
      benefit’ box, many of the narrative comments suggested they agreed with the
      ‘indirect benefit’ position.
610   There was one robust exception, however. This respondent added the comment:
      ‘[Participation in the award] could even be seen as detrimental to learners as it
      means time NOT spent organising new developments and activities for CEG’.
611   21 respondents ticked the middle option (indirect benefits). As many as 16 of this
      group added comments expanding on their view. The following is a representative
            ‘Students may not necessarily be aware of the award but they benefit from the
             examination and self evaluation of the process.’
            ‘Pupils rate CEG as an important part of education, helping at transition times
             and with the realisation of the importance of skills.’
            ‘If contextualised in the curriculum, [the CWQA] forms part of several
             improvements in retention and attainment.’
            ‘[Improved] resource materials KS3 and KS4.’
            ‘Everybody benefits from WRE, as a cross-curricular 'Industry Week' is
             arranged every summer for everyone in the school - but they are not aware of
             the Award.’
            ‘[There is a] learning benefit from staff having a clearer understanding of
             WRE/CEG and its importance in the curriculum when integrating it into
            ‘Generally true across college [i.e. that there are indirect benefits]. Staff at
             training event were positive and keen to use what was learned with students –
             this was my impression, confirmed by the evaluations.’
612   23 respondents ticked the third option (direct benefits). Of these, only 2 did not add
      further comments. There was, therefore, much valuable material to explore.

ASW Consulting                                44                 Final Report – January 2006
Careers Wales                                                               Evaluation of CWQA

613   Further analysis suggested two main types of response.             The first group (8
      responses) suggested benefits that might more appropriately be grouped with the
      ‘indirect benefit’ responses. These included the following, for example:
            ‘Forms part of wider support structure so could not be reliably quantified’.
            ‘New developments contribute to continuing good standards’.
            ‘Resources - improved and updated’.
            ‘Ensuring a quality programme          and   additional   feedback from     pupils,
             parents/focus groups’.
614   The second group were examples of benefits that were apparently directly
      attributable to the award (9 responses, but some with multiple benefits proposed).
      Rather than reproduce these comments verbatim, we thought it would be more
      helpful to group them by theme.
615   The product of this process is as follows:-
            more realistic choices at transition times; better transitions; transition stages
             better managed; more mature at transition stages;
            pupils better able to present themselves at work experience interviews;
            increased awareness; pupils more aware of opportunities;                  improved
             identification of destinations; better information on destinations;
            enhanced learning through Dynamo, Industry Days etc; improved teaching and
            improved motivation; pupils more focused;
            pupils know who to see for advice; more appropriate guidance to make better
             informed decisions; improved decision making skills.


616   We have extensive notes from our discussions with CW staff and school/college staff
      around the issue of the impact of the award on learners. Not surprisingly perhaps,
      the points made were similar to those already summarised in this chapter. We
      therefore provide a brief summary only of the perceptions held most widely.
617   These are as follows:-
            it is difficult to cite hard evidence of directly attributable benefits, even though
             learners undoubtedly derive indirect benefits from their institution’s
             participation. It would be possible to set up a longitudinal study to explore the
             point further but this is not a current priority;
            ultimately the client of the award is the institution, not the individual learner,
             although in practice the distinction is a fine one;
            learners in CWQA institutions enjoy well designed, integrated programmes,
             delivered by trained staff, supported by learning targets and subject plans, and
             with opportunities to provide feedback through evaluation processes;
            learners in CWQA institutions are often described by mainstream Careers
             Advisers as better prepared for interviews and more knowledgeable about the
             options open to them;
            staying on rates and/or post-16 participation rates were said to be better at
             CWQA institutions. Similar comments were also made about destinations from
             post-16 institutions.

ASW Consulting                                 45                  Final Report – January 2006
Careers Wales                                                        Evaluation of CWQA


618   Adding some headline conclusions from this chapter here would be somewhat
      repetitive. We would, therefore, refer readers seeking a quick summary to paragraph
      617 above.

ASW Consulting                             46                Final Report – January 2006
Careers Wales                                                             Evaluation of CWQA



701    This has been a long report, with brief conclusions added at the end of each chapter.
       We therefore do not repeat them here. Instead we turn to the recommendations
       arising from our evaluation.
702    The sections are headed:-
            rationale;
            shorter term recommendations;
            longer term recommendations.


703    There are three main reasons, and one subsidiary consideration, why we do not
       think that now is the time for radical changes to the design and delivery of the
       award. The main reasons are that:-
            the award in its present form has a brief history only. The CEG module dates
             from 2001 and the WRE only from 2004. It would be premature to implement
             radical change so soon, unless there were overwhelming arguments for doing
            there are no less than 178 institutions at various stages of working towards the
             award. To change the parameters in a significant way now would give rise to
             considerable confusion;
            a major redesign can confidently be anticipated for 2007/8 (launch of merged
             and revised ACCAC frameworks) and so it would make sense to use the
             intervening period to plan the new version of the award on a collaborative basis
             with teacher/lecturer colleagues, rather than engage in constant ‘tinkering’.
704    The ‘subsidiary consideration’ referred to in the previous paragraph is hinted at in
       the first bullet above. In short, if the evaluation showed that the award was widely
       perceived to be failing, it would be illogical not to implement urgent reforms. But
       this is not the case: there are aspects requiring attention, of course, but these do not
       warrant a wholesale redesign.
705    We therefore propose a series of short term recommendations, the introduction of
       which we think would improve the award over the next 2 to 3 years. However, we
       think a more radical look is required in the longer term. This may require a
       cessation of the current award and the launch of a different product in 2008.
706    Finally in this rationale section, we acknowledge that the next section contains too
       many recommendations for them all to be sensibly implemented over the next couple
       of years: some prioritisation will be required. Our suggestion is that the Quality
       Award Group should recommend to the Chief Executives a short list of actions
       which will yield a quick benefit but without giving rise to precisely the uncertainty
       and confusion in the market place which we are stressing should be avoided.


707    The CEO Group should set a target number of institutions to be supported through
       the award in the period until July 2007. The target would take into account
       resource considerations and area differences.

ASW Consulting                                47                 Final Report – January 2006
Careers Wales                                                                     Evaluation of CWQA

708    In the light of the pan-Wales and local area targets, projections should be calculated
       for the number of consultant days and assessor days needed for each area for each
       intervening year. Training and deployment issues should be discussed nationally
       but it may be necessary to continue with ad hoc arrangements if no consensus is
       forthcoming on operational aspects such as inter company charges for the ‘loan’ of
       CWQA personnel.23
709    A working definition is needed as to the boundaries around CWQA support, and
       linked activities such as curriculum development, teacher training, Progress File and
       some aspects of Careers Wales Online. This will not be straightforward, but without
       some clarity on this point, resource planning will be on shifting sands.
710    We would suggest ‘playing down’ the ‘entitlement to 6 days24 free consultancy’ point
       for public consumption. We accept that it would be ill advised to go back on this
       offer for institutions already in the system but, given the ambiguity over defining the
       boundaries around consultancy - and the pressure on CW teams, we think a rather
       more low key reference to the support available would be prudent. The alternative
       would be keeping the ‘free consultancy’ as a marketing point, but reducing the
       stated notional entitlement (e.g. up to 4 days for the first module, but with the offer
       of additional consultant days on a charged basis).
711    Serious consideration should be given to rationalising the Principles and Criteria;
       the suggestions arising from our research are presented in Chapter 3. It will be
       important to strike the right balance between being seen to respond professionally to
       feedback received, on the one hand - and causing confusion, on the other. Granting
       dispensation to some of the Principle 3 requirements to institutions with the
       Investors in People standard is just one example of a change that could be made
       without detriment to the coherence of the CWQA design.
712    As part of this exercise in ‘tweaking’ the current requirements, the particular
       contexts of post-16 institutions, special schools and other discrete units should be
       looked at again (perhaps by inviting an experienced professional from each of these
       sectors to advise the Quality Award Group). The degree of customising which will be
       possible (and prudent) to introduce over the next couple of years will be modest –
       but being seen to respond to positive suggestions will be an important tactical and
       presentational point.
713    Consideration should be given to ways of fast tracking institutions in defined
       circumstances (e.g. those working towards a second module). We recognise that this
       is possible already through mapping evidence supplied for one module against the
       requirements for the other - but here, too, there is a presentational, as well as a
       process, point. The fact that some ‘short cuts’ are perfectly OK was not widely
714    The Quality Award Group should explore the possibilities for streamlining the
       assessment process and panels.
715    ‘Mainstream’ Careers and EBL advisers should be involved in at least some of the
       CWQA discussions involving ‘their’ schools and colleges. Periodic briefings by local
       CWQA Co-ordinators at Careers Adviser team meetings is another common sense
       action (already done in many areas, of course).
716    More emphasis should be given to ‘area wide’ training days for school Co-ordinators
       (as opposed to one-to-one consultancy). Similarly, self help groups and mentoring
       arrangements (e.g. experienced Careers Co-ordinators with the award supporting

23 Longer discussions of points relating to the deployment and training of Consultants and Assessors were
provided towards the end of Chapter 4.
24 Or 8 days if both modules are being tackled.

ASW Consulting                                     48                   Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

      those new to the task) should be actively encouraged. A small investment in CWQA
      Co-ordinator time in setting up and encouraging such groups would (should!) lead to
      savings in consultancy time later on.
717   Occasional opportunities should be planned for Welsh speaking Consultants and
      Assessors to meet together and tackle Welsh medium issues jointly.
718   Other points to consider as part of the short term package would include:-
            getting a clear message through to schools and colleges about the parameters
             and timescales for future changes to ACCAC frameworks;
            developing better links with LEA advisers as regards the CWQA contribution to
             teacher development and curriculum support;
            preparing some case studies on the impact of the award on institutions and


719   Our only longer term recommendation is that planning should start as soon as
      possible on the shape of the award for 2008 onwards. It will be essential to position
      this exercise sensitively so as not to cause consternation in the market place (which
      could result in wholesale withdrawals from the current award).
720   The external factors influencing work on the new award would include:-
          the current focus on teacher/lecturer workloads, especially as regards its likely
           effect on the amount of available co-ordinator time and institutional structures;
            developments in the 14-19 curriculum, particularly the emergence of the
             Learning Core;
            the merger and rationalisation of ACCAC frameworks.
721   It would be politic to involve potential clients of the award (institution staff) and to
      consult with organisations having influence at a national strategy level (ACCAC,
      Estyn, WAG) in the planning exercise. This will help to build ownership and broader
      support; it would also identify at an early stage any sensitivities and potential ‘show
      stoppers’. There is also a broader group of organisations, including Fforwm, LEAs
      and teacher/lecturer associations, whose contribution it would be tactful to canvas.
722   We would strongly urge that the planning exercise is tackled from a ‘zero base’. It
      may be that the right approach for the next decade will be very different.
723   More radical alternatives would include ‘prizes’ and/or ‘competitions’ (e.g. as in the
      National Training Awards), and accreditation for the individuals (as well as the
      institution). It seems almost inevitable that the new award will be less resource
      intensive (perhaps along the lines of the Basic Skills Award), and perhaps broken
      down into ‘bite-sized’ chunks. However, these are precisely the points to be
      discussed; we would not want to constrain this debate before it has begun!

ASW Consulting                               49                 Final Report – January 2006
Careers Wales                                                                                        Evaluation of CWQA

                                                                                                               Appendix A
                            EVALUATION FRAMEWORK

                                                                                                                                           Strategic Players
                                                                     Careers Wales




       How has CWQA raised the status of CEG/WRE internally –
 1                                                                      *               *                *          √                         √
       How has CWQA helped in meeting national standards such
 2                                                                      *               *                *          √                         √
       as ACCAC frameworks and Estyn inspection requirements?
       How has CWQA generated a better understanding of
 3     CEG/WRE by staff, students and parents? (Examples               √                *                *          √            *
       where possible)
       How has CWQA enhanced the reputation of establishments
 4                                                                     √                *                *          √                         √
       in local communities?
       In what way has CWQA helped provide more efficient use of
 5                                                                                                       *          √
       time and resources to meet specific learning outcomes?
       What contribution (if any) has CWQA made in providing a
 6     better coordination of Careers Wales and schools/college        √               √                 *          √                         √
       How has CWQA helped to encourage a quality approach to
 7     the management of CEG/WRE in terms of planning,                  *              √                 *          √                           *
       documenting, reviewing and evaluating processes?
       How has CWQA contributed to increasing the sharing of
 8                                                                     √               √                 *          √                         √
       CEG/WRE-related best practice locally and across Wales?
       In what way (if at all) has CWQA provided a source of
 9     support for work related and Careers Co-ordinators in            *              √                √           √                           *
10     If a modest fee was introduced for the assessment and
                                                                                       √                            √
       award process, would this affect take up?

Key:     √ = key questions to be raised with these establishments/individuals
         * = if opportune, some         of   these   questions     might             be              raised                  with        these

ASW Consulting                                  50                           Final Report – January 2006
Careers Wales                                                                                                Evaluation of CWQA

                                                                                                                                              Strategic Players
                                                                          Careers Wales




     What have been the implications for your establishment in
 1                                                                                                                             √
     applying for CWQA?
     What has been the demand on senior management time? Has it
 2                                                                                                                             √
     been beneficial? Too demanding? Any alternative suggestions?
     How have you approached the ‘whole establishment’ requirement
 3                                                                                                                             √
     of CWQA?
     How have directors/boards of governors been involved/kept
 4                                                                                                                             √
     informed of CWQA developments?
     What have been the overall resource implications of
 5   achieving/working towards CWQA? (staff and infrastructure                                                                 √
     resource demands might be covered here)
     What have been the general benefits of achieving/working
 6   towards CWQA?          Have there been any difficulties?      Any                                                         √
     alternative suggestions?
     Have you conducted any cost benefit analysis of achieving the
 7   award (formally or informally)? What are the results? What are                                                            √
     your conclusions?
     Post-16 provision – coordination to show effective progression;
 8                                                                                                               *             √
     how has this worked? Any issues? Any perceived benefits?
     College – how have you involved/do you plan to involve all units
 9   across the establishment? What are the advantages of doing this?                                            *             √
     Any issues?
     College – have you extended CWQA criteria to part time students?
10   Do you see advantages in doing this? What are they? Any                                                     *             √
     How would you assess the awareness of CWQA across the whole
11                                                                                                               *             √
     of your establishment?
     What has been staff reaction to CWQA? Staff directly involved in
12                                                                                                               *             √
     CEG/WRE? Staff not directly involved?
13   Estyn inspections – has CWQA had any impact in this context?                  *                  *          *             √                         √
14   Progress file – what impact, if any, has CWQA had in this context?                               *          *             √         *               *
     In the context of CWQA, how have you involved key partners? Are
15                                                                                                    *                        √
     there any differences before and after? Benefits? Issues?
     Has CWQA had any wider impact in terms of meeting/delivering
16                                                                                                                             √                          *
     institutional objectives?
     Specialist institution – how relevant is CWQA to your
17                                                                                                                             √                          *
18   In what way is your CEG/WRE provision supported by CW staff?                                                              √
     In terms of CEG/WRE what is your assessment of the support
19                                                                                                                             √
     offered by CW support?
     Have your support requirements of CW diminished or increased
20   after achieving CWQA/Do you see your support requirements of                                                              √
     CW diminishing or increasing on achieving CQWA?
     In terms of CWQA, how clear and understandable were the
21   criteria and processes laid down by CW?            Any suggested                                                          √
     How difficult (or otherwise) was the process of preparing your
     establishment to enter and go through the application process
22                                                                                                                             √
     (and achieve the award)? How could this be improved or made
     more efficient?
     Given the ‘cost’ to the school/college in terms of resources, has
23                                                                                                                             √
     CWQA been a worthwhile exercise? Reasons

ASW Consulting                                      51                                    Final Report – January 2006
Careers Wales                                                                                    Evaluation of CWQA

                                                                                                                                  Strategic Players
                                                                 Careers Wales




     Have there been any perceived improvements in the
 1   delivery of CEG/WRE within your establishment?                                                  *             √
     Examples? Issues?
     How, if at all, has CWQA impacted on the
 2   planning/development/delivery of PSE (coherence angle in                            *           *             √                                  *
     What impact has CWQA had in terms of developing the
 3                                                                                       *           *             √         *                        √
     content of CEG/WRE programmes?
     How, if at all, has CWQA extended the range, quality and
 4                                                                                       *           *             √                                  √
     depth of CEG/WRE provision?
     What effect has CWQA had on the management and
 5                                                                                       *           *             √                                  √
     planning of CEG/WRE
     Has CWQA had any effect on the management and
 6                                                                                                                 √                                  √
     utilisation of staff involved in PSE/CEG/WRE?
     How has CWQA impacted on your evaluation and forward
     planning processes for PSE/CEG/WRE? Has it had any
 7                                                                                                                 √                                  *
     wider impact in this context across the whole
     Do the principles of CWQA have any relevance or
 8                                                                                                                 √                                  *
     appropriateness to the wider curriculum?
     Is there any evidence that CWQA has had any effect on the
 9   development/delivery of the wider curriculum within                 *               *           *             √                                  √
     In your experience, do you see CEG and WRE as being
10                                                                                       √                         √                                  √
     complementary? If so in what way? If not, why?
     Has CWQA provided any stimulus in helping to develop
11                                                                                                                 √                                  √
     cohesion between CEG/WRE and with PSE?
     On which module (CEG or WRE) have you initially
     concentrated in your application for CWQA? Do you see
12                                                                                                                 √
     yourself applying for assessment in the other module? If
     so, why? If not, why?

ASW Consulting                                52                             Final Report – January 2006
Careers Wales                                                                                          Evaluation of CWQA

                                                                                                                                        Strategic Players
                                                                    Careers Wales




     What has been the direct impact on learners of the
 1   school/college achieving/applying for CWQA in terms of                 *                  *           *             √        √                *
     achievement of CEG/WRE learning outcomes?
     Has CWQA had any spin-off impact on wider aspects such
 2   as student attendance, confidence, behaviour, attitude,                                                             √        √                *
     morale and general examination results?
     Does CWQA produce differentiated effects at each Key
     Stage, i.e. does it have more impact on one over the others,
 3                                                                                                                       √                        √
     or does it have a broadly similar impact across the Key
     What effect has CWQA had on planning the progression of
 4                                                                                                                       √         *              √
     students between each of the Key Stages?
     What effect has CWQA had in terms of Equal Opportunities
 5   within   CEG/WRE     and more      widely   within   the                                                            √         *              √
     In what way, if at all, has CWQA had an impact on teaching
 6                                                                                                                       √         *              √
     and learning processes within the school/college?
     How, if at all, has CWQA impacted on the way in which you
 7                                                                                                                       √         *              √
     undertake assessment, recording and reporting processes?
     How has CWQA helped students in making effective career
 8                                                                                                                       √         *              √
     choices and ultimate transitions?
     Is there any (anecdotal) evidence that the school’s
     achievement of CWQA has had any impact on students
 9                                                                          *                  *           *             √        √               √
     after leaving compulsory education e.g. in terms of
     seeking/finding work or ongoing learning opportunities?

ASW Consulting                                  53                                  Final Report – January 2006
Careers Wales                                                                                          Evaluation of CWQA

                                                                                                                                        Strategic Players
                                                                    Careers Wales




     What impact has the development and management of the
 1                                                                          √                  *           *                                      √
     award had on CW as a whole?
     What are the resource and logistical issues of managing
 2                                                                          √                  √           *
     What are the main management issues and considerations
 3                                                                          √
     in managing CWQA?
     What are the resource and logistical implications of helping
 4                                                                          √                  √          √
     establishments prepare for the award?
     What is your view on the amount and level of support you
 5   offer to establishments preparing for and undergoing the               √                  √          √
     assessment process?
     Do you feel that the amount of time devoted to
     teacher/curriculum support and that of promoting CWQA
 6                                                                          √                  √          √                                        *
     more widely is in appropriate proportion? How might this
     be changed or improved?
     Once an establishment achieves the award, does the
 7   support requirement placed on you by that establishment                √                  √          √                                        *
     How can resource demands placed on CW be managed
 8                                                                          √                  √          √
     Would a compliance model make a difference?        In what
 9                                                                          √                  √          √                                       √
     way? Advantages? Disadvantages?
     Are CW ‘on the right lines’ in the context of CWQA, given
10                                                                          √                  *
     the demands on resources and logistics?
     Should CW (continue to) be the body to administer and
11   manage CWQA? Who else could be involved? How would                     √                  *                                                  √
     that work?
     If CW was to rationalise some of its CWQA functions, which
12   should be retained? Could any be ‘dropped’ entirely?                   √                  √          √              √
     Which could be transferred and to which organisation?
     Should there be any incentives offered to establishments
13   for the achievement of CWQA? For example additional                    √                  √          √              √                        √
     support after achievement (a reward).
     Or should one expect those establishments that have
     achieved the award to be more self sufficient? Should
14   resources     therefore    be      directed    towards                 √                  √          √              √                        √
     encouraging/helping other establishments to achieve the
     Given the ‘cost’ to Careers Wales in terms of resources and
15                                                                          √                                                                     √
     time, has CWQA been a worthwhile exercise? Reasons

ASW Consulting                                  54                                  Final Report – January 2006
Careers Wales                                                                                             Evaluation of CWQA

                                                                                                                                           Strategic Players
                                                                       Careers Wales




     The same basic structure and process is applied to all types
 1   and levels of institutions (1-size-fits-all), do you think that           √                  √                         √
     this approach works effectively?
     The CWQA structure has 9 Principles, supported by
     between 8 and 17 Criteria – are these all appropriate?
 2                                                                             √                  √                         √
     Anything missing?  Which could be omitted without
     significant loss?
     The CWQA process stage 1 – Self Assessment & Action
     Planning – what are your experiences of this part of the
 3                                                                                                *          √              √
     process.    Comments?       Practical suggestions for
     The CWQA process stage 2 – Commitment & Development –
 4   what are your experiences of this part of the process.                                       *          √              √
     Comments? Practical suggestions for improvement
     The CWQA process stage 3 – Assessment & Moving on –
 5   what are your experiences of this part of the process.                                       *          √              √
     Comments? Practical suggestions for improvement
     How can more establishments be encouraged to apply for
 6                                                                             √                  √           *             √
     CWQA is a resource intensive model – would an alternative
 7   approach of a compliance model be acceptable? Benefits?                   √                  √           *             √
     Drawbacks? What fundamental changes would you make?

ASW Consulting                                    55                                   Final Report – January 2006
Careers Wales                                                      Evaluation of CWQA

                                                                      Appendix B

Careers Wales Cardiff and the Vale
Mary Jeans, CWQA Co-ordinator
Jenny Chambers, Curriculum Adviser
Shirley Rogers, Senior Manager for Youth Guidance
June Jensen, external Consultant (St Cyres School)
Glan Rees, external Consultant (Glyn Derw High School)
Jeff Evans, Education Business Manager

Careers Wales Gwent
Sally Farr, CWQA Co-ordinator (both National and Gwent)
Kay Mahony, Education and Guidance Manager
Margaret Noakes, Head of Client Services and Partners
John Cook, external Consultant
Jane Johns, external Consultant
Willie Stewart, external Consultant

Careers Wales Mid Glamorgan and Powys
Kay Bibby, CWQA Co-ordinator
Beth Titley, Education Business Contract Manager
Wayne Feldon, Chief Executive
Steve Duggan, external Consultant
Jeff Davies, external Consultant and Assessor
Sharon Winnard, external Consultant
Steve Hole, Business Development Manager
Ken Dicks, 14-19 Manager

Careers Wales North East
Barry Williams, CWQA Co-ordinator
Phil Enticott, Senior Manager
Joyce M’Caw, Chief Executive
Carole Garrett, internal Consultant

Careers Wales North West
Delyth Rowlands, CWQA Co-ordinator and Assessor, and Curriculum Manager
Ffiona Williams, Senior Manager
Elin Vaughan Williams, Consultant and Development Co-ordinator
Bethan Williams, Consultant and Education Business Partnership Co-ordinator

Careers Wales West
Briony Webb, Careers Education and Quality Award Co-ordinator
Graham Bowd, Director of Guidance
Angela Newscome, Consultant/Assessor and Education Team Leader, Carmarthenshire
Sue Jones, Consultant and Team Leader, Business Improvement
Michelle David, Consultant and Careers Adviser, Neath Port Talbot College

ASW Consulting                            56               Final Report – January 2006
Careers Wales                                                                    Evaluation of CWQA

Careers Wales Quality Award Group
Trina Neilson, CEO Gwent and CWQA ‘Champion’
Mary Jeans, CW Cardiff and the Vale, and Chair of CW Quality Award Group
Sally Farr, CW Gwent and National Co-ordinator
Kay Bibby, CW Mid Glamorgan and Powys
Barry Williams, CW North East
Jean Williams, CW West

Careers Wales Chief Executives Group25
Lesley Rees, Executive Director (Careers Wales Association)
Ray Collier, CEO CW West
Mark Freeman, CEO Cardiff and the Vale
John Llewellyn, CEO North West
Joyce M’Caw, CEO CW North East
Trina Neilson, CEO CW Gwent

Coleg Llysfasi
David Smith, Vice Principal
Jayne Lewis, Careers Co-ordinator

Cowbridge Comprehensive School
Margaret Evans, Head
Julian Davis, Deputy Head
Andrea Pennell, Assistant Head
Julie Thelwell, Careers Co-ordinator
Jill Jones, Secretary, Careers Education
Year 12 students (3)
Year 10 students (5)

Pembrokeshire College
Maxine Thomas, Learning Services Manager
Lesley Fudge, Senior Tutor

Ysgol Bodedern
Annwen Morgan, Deputy Head
Menai Owen, Careers Co-ordinator
Students (8) drawn from Years 9 to 13 (i.e. 1 from Year 13; 1 from Year 12; 3 from Year 11;
2 from Year 10; and 1 from Year 9)

Ysgol Bryn Castell
Nigel Griffiths, Head
Sue Martinson, Careers Co-ordinator

Ysgol Glan Taf
Haydn Pritchard, Careers Co-ordinator
Year 11 students (7)

David Kitchen

25 Wayne Feldon, CEO of CW Mid Glamorgan and Powys, was prevented by exceptional weather conditions from
attending this session. He contributed views via the fieldwork visit, however.

ASW Consulting                                    57                    Final Report – January 2006
Careers Wales                                                      Evaluation of CWQA

                                                                      Appendix C
                          WRITTEN COMMENTS RECEIVED
Aberdare Girls’ School (MG&P)                 Swansea College (W)
Amman Valley School(W)                        Thomas Richards Centre (Gw)
Argoed High School (NE)                       Tonyrefail School (MG&P)
Bishop Gore School (W)                        Tredegar Comprehensive School (Gw)
Cantonian High School (C&V)                   West Monmouth School (Gw)
Castell Alun High School (NE)                 Ysgol Crug Glas (W)
Cathays High School (C&V)                     Ysgol David Hughes (NW)
Coleg Ceredigion (W)                          Ysgol Gyfun Llanbedr-Pont-Steffan (W)
Coleg Glan Hafren (C&V)                       Ysgol Gyfun Y Strade (W)
Coleg Gwent (Gw)                              Ysgol Heol Goffa (W)
Coleg Llysfasi (NE)                           Ysgol John Bright (NW)
Coleg Menai (NW)26                            Ysgol Morgan Llwyd (NE)
Cowbridge Comprehensive School (C&V)          Ysgol Rhiwabon (NE)
Cwmcarn High School (Gw)                      Ysgol Syr Hugh Owen (NW)
Deeside College (NE)                          Ysgol Syr Thomas Jones (NW)
Denbigh High School (NE)                      Ysgol Tir Morfa (NW)
Duffryn High School (Gw)                      Ysgol Tryfan (NW)
Dwr-Y-Felin (W)                               Ysgol Y Gwendraeth (W)
Dyffryn School (W)                            Ysgol Y Moelwyn (NW)
Dyffryn Tâf (W)                               Whitemill Special Unit (W)
Fitzalan High School (C&V)                    [Unnamed institution] (W)
Glan Afan Comprehensive School (W)
Glyn Derw High School (C&V)
Gwernyfed High School (MG&P)
Howells School (NE)
John Summers High School (NE)
Llanrumney High School (C&V)
Llantarnam School (Gw)
Merthyr Tydfil College (MG&P)
Milford Haven School (W)
Mold Alun School (NE)
Neath Port Talbot College (W)
Pembrokeshire College (W)
Pen-Y-Cwm School (Gw)
Penyrheol Comprehensive School (W)
Pontarddulais Comprehensive School (W)
Porth County Community School (MG&P)
Queen Elizabeth Cambria School (W)
Rhosnesni High School (NE)
Rougemont School (Gw)
St Cyres School (C&V)
St John Baptist High School (MG&P)

26   Brief comments by phone

ASW Consulting                           58                Final Report – January 2006
Careers Wales                                                            Evaluation of CWQA

                                                    Appendix D
                CWQA COSTINGS (source: Careers Wales)
Table D.1: Consultancy costs for the award process to support an establishment to
achieve both award modules

Stage                                              Entitlement       Unit cost       Total

Self assessment and action planning                Up to 2 days      £150.00        £300.00
Development                                        Up to 8 days      £150.00       £1,200.00
Assessment                                         Up to 3 days      £200.00        £600.00
Total                                                                              £2,100.00

Table D.2:    Reassessment costs

Stage                                              Entitlement       Unit cost       Total

Self assessment, action planning and
                                                   Up to 1 day       £150.00       £150.00
Assessment                                         Up to 1 day       £200.00       £200.00
Total                                                                              £350.00

Table D.3:    Other costs

Dragons Tail Trophy                                                                 £150.00

Table D.4:    Training and development costs (funded through National Co-ordinator

Cost item                                             Co-ordinator days          Other costs

Initial training via 1 annual 2 day course                  5 days                   £750.00
Annual national update course for all Consultants:
                                                            6 days                 £1,000.00
2 x 1 day courses
3 national accreditation panels                             3 days                   £500.00
Total                                                       14 days                £2,250.00

1 Costs in Tables D1 to D3 are met by individual careers companies.
2 National Co-ordinator costs are determined annually – in 2005/06 costs are £15,500.00
which includes figures in Table D4.
3 Time costs of Consultants attending training is not included.

ASW Consulting                               59                   Final Report – January 2006

To top