Appendix –Careers Pilot Report

APPENDIX A: METHODOLOGY APPENDIX A: METHODOLOGY ERO‟s evaluation included a mix of methods (interviews, questionnaires, observation and document examination) in four key stages. Findings from each stage were discussed with the Ministry and used to identify priorities for subsequent stages. The priorities at each stage were agreed with the MoE. A review of the project documentation, research on career education and professional development and discussions with stakeholders, including staff at a small number of schools, provided contextual information and identified areas for ERO to cover in the questionnaires and interview schedules. The key stages of ERO‟s evaluation were:  an initial survey of all pilot schools;  interviews with Career Services consultants;  visits to case study schools; and  end of year postal survey of schools not included in the case studies. An initial survey of all pilot schools Because the Pilot provides a flexible model and offers a range of services for schools to select, an initial postal survey sought information about key aspects of how the Pilot was being implemented in each school. Schools were asked which services were selected to be provided by Career Services, which staff were involved, how many staff were involved, how many students at each level, how at risk was defined, and the planned timing of key events. The Education Review Office, in consultation with the Ministry of Education and Career Services, developed a questionnaire that was trialled (at a local school) and modified. The questionnaire was sent to schools in May, and after some follow up phone calls, all 75 Pilot Schools responded. ERO‟s survey showed that schools applied to be part of the Designing Careers Pilot for a variety of reasons. The most common reasons were the wish to:  develop the careers education programme; (24 schools)  acknowledge the importance of careers education; (21)  provide better for Year 10 students; (19)  access the resources provided through the Pilot; (9)  obtain Pilot funding; (9)  benefit students; (8)  help at risk students; (6)  obtain professional development. (6) Responses to the questionnaire showed that schools organised the development of the Learning and Career Plans (LCPs) through a number of curriculum areas, including social studies, health, English, and pastoral care and administration time, or in a mix of areas. Designing Careers Pilot Evaluation Appendices May 2006 Page A1 APPENDIX A: METHODOLOGY Visits to case study schools Twenty-two schools were selected for in-depth study and two reviewers visited each school. Case studies were undertaken to provide rich descriptions of various ways in which the Pilot was implemented and its impact in the early stages. The case studies also enabled ERO to identify possible success factors and barriers to effective implementation. The case-study schools were selected because they were implementing the Pilot in a number of different ways, such as in various curriculum areas, as part of pastoral care, or in a mix of areas. There was at least one school from each Career Services district. In addition, the selected schools represent a range of characteristics such as geographic location, isolation, gender (15 coeducational, 3 boys, 4 girls), decile (8 low, 11 middle, 3 high), and size (11 with a roll below 800, 6 with a roll between 800-1200, 5 with a roll above 1200). The case study visits occurred in term 3 (18 schools) and term 4 2005 (4 schools). The timing allowed time for the professional development to have occurred, for LCPs to have been completed and used, for Year 10 students to have made subject choices for 2006, and for some work to have been undertaken with at risk students. During the visits, the reviewers interviewed key personnel and examined project documentation. Reviewers interviewed the principals, careers advisers, classroom teachers, key staff, small groups of Year 10 and at risk students, and some parents. Some of the students who had been identified as being at risk could not be interviewed as they were not at school at the time of the visit. It was possible to obtain information from parents at only a small number of schools. As it was not possible to interview all those involved in the Pilot at the time reviewers were in the schools, questionnaires were developed to obtain the views of a greater number of teachers and students. One questionnaire was developed for teachers who had delivered the LCPs and a second questionnaire sought the views about career education of other teachers who were not directly involved in delivering the Pilot. Two different questionnaires were developed to obtain views of a greater number of students – one questionnaire was largely pre-coded and the other had open questions. Schools were asked to use the format that seemed most appropriate for their students. Most schools used some of each questionnaire – 358 students completed the pre-coded version and 146 completed the open-ended version. Altogether, during the case study visits, information was obtained from 18 principals, 22 careers advisers, 307 teachers, 722 Year 10 students and 48 students at risk. The intention had been to visit schools at a time parents would be available for interview, possibly at subject or career meetings. However, arranging this proved to be difficult given other constraints in fitting in visits after LCPs had been completed and in time to meet timelines agreed with the Ministry of Education. ERO attended a parent meeting at one school. The investigative questions that the review teams asked in the case-study schools were trialled with the first schools interviewed and were modified slightly for the other schools. Project documentation included guidelines for classroom teachers on delivering the programme, course handbooks for subject selection, and student workbooks. LCPs were examined to assess the extent of their completion and their likely usefulness. May 2006 Page A2 Designing Careers Pilot Evaluation Appendices APPENDIX A: METHODOLOGY Interviews with Career Services consultants In June 2005, reviewers interviewed 25 Career Services consultants and some national project managers from the eleven regional centres to obtain their views about how well various elements of the Pilot were working in schools and about any changes that could be made to improve its effectiveness. The consultants‟ experience in implementing the Pilot in several schools has enabled them to identify factors that are important for success in delivering careers education. The investigative questions that the review teams asked the consultants were trialled in two districts and modified slightly for the other interviews. The questions are based on the terms of reference for ERO‟s evaluation and on the evaluative questions presented to the Ministry of Education in the first interim report. These questions were closely linked to the responsibilities of Career Services as outlined in the Designing Careers Initiative Agreement between the Ministry of Education and Career Services, to the desired outcomes of the Pilot and to the specific objectives of Career Services within the Designing Careers project, as outlined in the Designing Careers Pilot Project Definition Document, 5 October 2004. End of year postal survey of schools not included in the case studies In the fourth term 2005, the 53 schools that were not visited were surveyed about their perceptions of the Pilot and the LCP. This survey built on the information obtained in the case studies and enabled tentative hypotheses to be tested in a larger number and variety of settings. These hypotheses related to key features of, and contributing factors for, effective delivery of career education, planning and guidance. The survey involved separate questionnaires for the principal, the careers adviser, and teachers. Questionnaires were completed by 51 schools – 50 principals, 50 careers advisers and 146 teachers. Designing Careers Pilot Evaluation Appendices May 2006 Page A3 APPENDIX B: KEY DATA FOR SECTION 2 - RESOURCES APPENDIX B: KEY DATA The information in this appendix comes from questionnaires completed by key stakeholders at each stage of the evaluation. The order of presentation matches the report. The source of information is noted for each finding. 2 2.1 Resources Professional Development How satisfied is the school with the process of discussing and agreeing the best mix of professional support Career Services will provide to the school? [Source: Initial Survey Q6] All schools but one were satisfied with the process of discussing and agreeing the best mix of professional support from Career Services. Very satisfied 74% Satisfied 24% Not satisfied 1% Which of the professional support offered by Career Services as part of the Pilot does the school plan to access during 2005? [Source: Initial Survey Q3] Professional support from Career services staff 75 responses Initial professional development activity for school staff who are / will be working with their students to develop Learning and Career Plans Regular ongoing support (formal / informal) for school staff working with students Problem solving support for both general issues and specific cases Number schools using Number of school staff involved Median Range 75 7 1 -40 75 65 Number schools using 5 3 1 -40 1 -15 Number of students involved Quality reviews of completed Learning and Career Plans Direct delivery of career planning support to selected at risk Year 10 – 13 students and students with disabilities 50 65 100 15 1 - 480 1 - 200 Designing Careers Pilot Evaluation Appendices May 2006 Page B1 APPENDIX B: KEY DATA FOR SECTION 2 - RESOURCES How useful have aspects of the Pilot been so far? [Source: Initial Survey Q9] % (of 75 schools) Initial presentation and orientation about the Pilot Initial professional development activity for school staff who are / will be working with their students to develop Learning and Career Plans; The flexible model of a Learning and Career Plan for school staff to use with their Year 10 students The flexible model of a Learning and Career Plan for school staff to use with at risk students The opportunity to network with other Pilot schools to share issues and best practice Presentations by Career Services staff to board or parent association Very useful 45 48 Useful 42 34 Not useful 11 1 Not yet/ Too soon 1 16 45 26 62 18 34 39 27 20 3 9 3 7 18 26 8 56 How useful have you found the support provided by Career Services? [Source: 50 Careers Advisers End of Year Survey Q2a] % Career Services Facilitating networking with other Pilot schools Ongoing support and advice for careers adviser Support for at risk students Professional development for careers adviser Professional development for staff on LCPs Professional development for staff on other topics, e.g. Real Game, Career Plan-It Attending parent meetings to provide course and career information In-class support during LCP completion [3 staff meetings] Very useful 67 64 63 56 49 44 30 22 Useful 31 32 35 38 32 40 26 16 Not useful 0 2 0 0 0 0 0 4 Not used 2 2 2 6 19 16 44 58 Designing Careers Pilot Evaluation Appendices May 2006 Page B2 APPENDIX B: KEY DATA FOR SECTION 2 - RESOURCES 2.2 LCPs How appropriate do you consider the content, layout and sections of the Learning and Career Plan for Year 10 students in your school or classes? [Sources: End of Year Survey Q5 (50 Careers Advisers, 145 teachers); 84 teachers involved with developing LCPs onsite Q1a] Careers advisers end of year Concept of a LCP Concepts included Process to complete a LCP Language used Three-page gatefold layout Electronic version Sections of the LCP About me My goals Reach my goals Reality check My subject choices Teachers End of year Appropriate Teachers Case study Very Appropriate appropriate Very Appropriate Very appropriate appropriate 67 32 24 15 21 17 61 54 40 36 64 33 68 76 76 38 14 36 41 49 44 31 63 43 31 37 34 25 50 45 38 32 45 35 53 62 57 49 20 50 53 54 55 49 47 26 29 19 21 58 53 39 41 47 49 69 62 72 64 38 42 53 49 46 Three-page gatefold – not appropriate 40% CA, 14% teachers Electronic version – don‟t know 49% CA, 46% teachers; not appropriate 20% CA, 9% teachers Reach my goals – not appropriate 12% CA, 9% teachers Reality check – not appropriate 20% CA, 9% teachers CA [9 format, 6 modified it, 3 language, 3 electronic, 3 didn‟t use, 3 repetitive] Teachers [11 not always appropriate/relevant for Year 10/better for Year 11, 7 takes a long time to complete, 4 some struggle to complete, 3 need lots of guidance/preteaching to complete] Students [11 students enjoy it, 8 links with subject choice, 11 useful/worthwhile, 5 students think about careers and subjects, 5 gives a sense of direction, 8 language too hard, 4 too long, 4 repetitive] Do you have any other comments about the LCPs? [Source: 84 teachers involved with developing LCPs onsite– Q5] [case study tchrs - 2 modified LCP, 4 depends on students, 4 parents not very involved, [students vary, some terms need help [4 design of LCP, 4 language, 2 parents, 8 fit with programme, 3 subject selection, 3 student ability, used, 3 not appropriate year level, ] Designing Careers Pilot Evaluation Appendices May 2006 Page B3 APPENDIX B: KEY DATA FOR SECTION 2 - RESOURCES How appropriate is developing an LCP for the types of Year 10 students listed below? Source: End of Year Survey Q6 High achievers Low achievers Students who already know what career or job they want Students who do not know what career or job they want Maori Pacific Students who lack motivation Students at risk of not completing a successful transition from school to employment or study 50 Careers advisers Very appropriate Appropriate 145 teachers Very appropriate Appropriate 79 54 53 64 63 62 62 71 21 35 45 36 35 36 23 18 64 64 38 73 55 53 57 63 34 27 56 26 45 47 36 33 Low achievers – not appropriate 10% CA, 9% teachers Students who lack motivation - not appropriate 15% CA, 7% teachers At risk - not appropriate –11% CA, 4% teachers [2 high achievers often know already, get bored;] 2.3 Other resources Which career education resources and activities were used with Year 10 students before or while they developed their LCP? [Source: 50 Careers Advisers End of Year Survey Q7] Number Career Plan-It Goal line Real Game Career Quest Jobs by Interest Yes 39 17 29 34 43 No 8 15 11 9 3 [16 Kiwi Careers, 6 Jobs Galore, 3 Courses Galore, 5 school resources, 4 Real game in Year 9] Designing Careers Pilot Evaluation Appendices May 2006 Page B4 APPENDIX B: KEY DATA FOR SECTION 2 - RESOURCES How useful have each of the following resources been with Year 10 students? [Source: 50 Careers Advisers End of Year Survey Q2b] Very useful Career Services Learning and Career Plan Jobs by Interest Jobs Galore Career Plan-It Career Quest Courses Galore The Real Game Kiwi Cards Other providers Goal line Career Compass Useful 43 57 60 42 36 57 20 26 36 0 Not useful 0 2 2 4 0 7 4 12 0 0 Not used 0 0 4 16 17 17 29 53 54 95 % 57 40 33 38 47 19 47 9 10 5 [Other: 11 Kiwi Careers, 4 Pathfinder How useful have the resources been? [Source: Initial Survey Q10] % Learning and Career Plans Career Plan-It The Real Game Options Kit Jobs by Interest Jobs Galore Courses Galore Career Quest Other information and online resources via the Career Services and Kiwi Careers websites, state (65) Other information and online resources, state (28) Very useful 44 40 47 35 46 48 38 53 65 Useful 39 41 26 50 45 41 41 29 28 Not useful 3 1 6 2 1 1 9 6 0 Not used 14 18 21 13 7 10 13 12 8 57 36 0 7 Although almost all schools indicated they accessed other resources through the Career services and Kiwi Careers websites, they generally didn‟t supply any other details about what they used. Designing Careers Pilot Evaluation Appendices May 2006 Page B5 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION 3 3.1 Implementation and effectiveness Implementation In which subject classes will students develop their Learning and Career Plans? [Source: Initial Survey Q7] % of 75 Social Studies English Health Life Skills Form time, tutor group time, home room time Other, state Year 10 students 67 21 19 7 24 27 At risk students 8 4 5 16 28 59 Students with disabilities 13 1 3 5 21 32 About 40% of schools were planning for Year 10 students to complete their LCPs in more than one of the subject classes or form class. Sometimes this was because students did part of the LCP in one class and another part at a different time. Sometimes this was because different groups of students worked on the LCP in different classes. For example, students in transition or homeroom classes worked on their LCPs there while other students worked on their LCPs during social studies. Overall, 67% planned to develop their Learning and Career Plans in social studies, 21% in English, and 24% in form time. At risk students were usually provided for in other ways. At risk students and those with disabilities usually developed their LCPs on an individual basis, often being withdrawn for a 1-1 interview. In which subject classes did Year 10 students develop their Learning and Career Plans? Please tick the appropriate boxes. [Source: 50 Careers Advisers End of Year Survey Q3] Timetable location Social Studies English Health Pastoral care time, e.g. form time, tutor group time, home room Staff involved Social studies teachers English teachers Health teachers Pastoral care teachers, e.g. form/tutor teachers Deans Career Services consultants Another outside careers consultant Yes (N) 30 10 11 13 29 8 9 17 24 26 2 No 6 17 18 17 5 19 20 15 11 9 21 [13 Careers Adviser, 7 careers teacher, 4 time taken from several subjects, Designing Careers Pilot Evaluation Appendices May 2006 Page B6 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION How well do you think developing Learning and Career Plans fits with each of the curriculum or pastoral areas below? [ie. where is the best place to cover the content?] [Source: 84 teachers involved with developing LCPs onsite Q2] % Social Studies English Health Life Skills Form time, tutor group time, home room time Very well 53 18 24 62 21 Well 31 44 36 24 39 Not at all well 5 22 19 5 32 Don‟t know 11 16 21 9 8 [5 form time too limited, not good fit with AOs for Social studies but health and life skills time limited] Do you think students should do other career education activities before developing an LCP? [Source: 84 teachers involved with developing LCPs onsite Q4] 36 Yes, Real Game 18 Yes, other, state 50 Yes, Plan-It 8 No (N) Which staff will be working with students to develop and provide advice on their Learning and Career Plans? [Source: Initial Survey Q8a] % of 75 schools Year 10 students 92 23 51 24 69 40 At risk students 88 31 44 35 16 20 Students with disabilities 52 12 16 17 11 16 Careers advisers / teachers Transition staff Deans Guidance counsellor/s Subject teachers, state which subjects: Year level, tutor group, form, home room teachers Did Year 10 students have individual interviews to complete/check their LCPs? [Source: 50 Careers Advisers End of Year Survey Q4] 36 Yes / No 8 Staff involved (number) Social studies teachers English teachers Health teachers Pastoral care teachers, e.g. form/tutor teachers Deans Career Services consultants Another outside careers consultant [5 some only, 3 after seniors have left] Yes 10 4 3 10 17 16 0 No 12 15 16 11 9 10 16 Designing Careers Pilot Evaluation Appendices May 2006 Page B7 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION Approximately how many periods were involved for Year 10 students developing the LCPs? [Source: 50 Careers Advisers End of Year Survey Q8] Related activities LCPs Total [range: 1-126, median 10] [range: 1-100, median 4] [range: 5-226, median 14] periods periods periods [wide range – some seem to have included LCP only (eg 6 classes), others the whole programme (eg 16 periods social studies + 9 periods health), hard to interpret] 3.2 Use and usefulness of LCPs How do you feel about the Learning and Career Plan? [Source: 358 student questionnaires onsite Q2] % The idea of doing a Learning and Career Plan Discussing it with your friends Discussing it with your parents Discussing it with the careers adviser Filling out the Plan The way it looks The words it uses Discussing it with someone from outside the school Positive 68 70 69 58 58 55 45 40 Don‟t know 28 21 24 34 34 37 47 44 Negative 4 9 7 8 8 8 8 15 Was the chance to talk about your LCP the first time a teacher took a real interest in you? [Source: 150 open student questionnaires onsite Q3] 34% Yes Designing Careers Pilot Evaluation Appendices May 2006 Page B8 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION What is the best thing about the Learning and Career Plan? [Source: 358 pre-coded student questionnaires onsite Q3] What job/subjects night fit/suit Ideas for the future Plans for the future/direction Variety of jobs Info about jobs and careers How to get a job Work exploration Subject selection Self knowledge Strengths Actions needed Keep options open Discussion Real Game Helpful Prepares Missing lessons 59 32 30 27 23 7 4 25 15 12 7 3 7 4 4 4 16 What do you like least about the Plan? [Source: 358 pre-coded student questionnaires onsite Q4] Nothing 59 Writing 31 Too long/repetitive 20 Work involved 10 Filling it out 9 Real Game 9 (especially not being able to choose the job) Answering questions I didn‟t know 4 Personal information 4 Thinking 4 Boring Confusing Irrelevant 9 6 3 How did your child feel about developing a Learning and Career Plan? [Source: 23 Parent Questionnaires onsite Q2] 3 Very positive 16 Positive 3 Not very positive Designing Careers Pilot Evaluation Appendices May 2006 Page B9 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION Is the information from the LCPs collated for use within the school? [Source: 50 Careers Advisers End of Year Survey Q10] 36 Yes / No 5 Has been used 15 6 15 21 13 8 Will be used 18 16 9 15 15 31 Answered Both 7 3 2 5 3 3 To target relevant career or course information to students To arrange relevant speakers or visits As a basis for deciding subject option lines To identify students at risk To enable staff to access information about students To track changes in students goals over time Which staff have access Deans Guidance staff Form teachers Subject teachers 21 16 16 8 16 18 15 11 4 4 1 1 Does the school collect and collate data on the intended careers of students? [Source: Initial Survey Q23a] 86% Yes If Yes, in what ways is this data used? [Source: Initial Survey Q23b] The data are used in a number of ways. Primarily to document student intentions and once documented, are used to target individual students when specific careers information is received by the school. It also enables the school to ensure appropriate subject choices are made and that their goals are realistic. Some schools noted they use the data to identify students at risk. To encourage students to attend related liaison events To ensure relevant information reaches these students as soon as it comes into the school To ensure that students are taking the appropriate subjects and pathways Designing Careers Pilot Evaluation Appendices May 2006 Page B10 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION 3.3 At risk students How has the school defined at risk students? [Source: Initial Survey Q11a] About one quarter (20 schools) use the MoE definition of students at risk of not making a successful transition to further education, training or work. Many schools gave several categories of risk. The most commonly used categories are:  performance/progress/lack of achievement/ low qualifications/low literacy or numeracy 32  attendance/truancy 15  behaviour 10        no career path identified lack of engagement or motivation inappropriate fit between skills and expectations risk of early leaving staff assessment or referral self assessment or referral referral to guidance/pastoral system 21 11 3 3 7 6 3 Other definitions include: poor family/whanau support (8), alcohol/drugs (2), Mäori (2), limited career information (2), bright under achievers, gifted, lack of support to connect with appropriate agencies, difficulty doing LCP, ESOL, mental health, health, social interaction, transience, happiness/wellbeing, disadvantaged groups, transition, alternative English class, multi-talented and confused. Two schools considered most of their students to be at risk: Any student we identify with behavioural/alcohol/drug/ESOL/family/lacking motivation/learning difficulties/mental health/truancy/personal/poor academic/unrealistic/undecided vocationally issues – nearly everyone. All students are potentially at risk. The underachievers, poor whanau support, poor literacy/numeracy. This school has a Year 10 class of mainly at risk students. How many of these students? [Source: careers advisers end of year survey Q13] [The number of students at risk ranged from 8 to 130, median 30. 14 schools did not give a figure.] Designing Careers Pilot Evaluation Appendices May 2006 Page B11 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION What support have you planned for these at risk students? [Source: Initial Survey Q11b] The most frequent support planned by schools was individual interviews (26 schools), or support from school careers staff (33) or Career Services staff (36). Often the students considered most at risk were to be referred to Career Services and school staff would work with others. Three schools used Career Quest. Other school staff involved were deans 9, guidance counsellors 8, form teachers 3, pastoral care staff 3. Schools also provided work experience or exploration 11, STAR 2, GATEWAY 2, mentoring 2, and outside speakers 2. In what ways has the school provided support for these at risk students? [Source: 50 Careers Advisers End of Year Survey Q14] Yes In-school support Curriculum e.g. alternative courses for English, maths etc Placement e.g. transition class or course Employment skills course e.g. STAR Work experience through Gateway Other experience in the workplace Support from the careers adviser Referral within school, e.g. guidance counsellor, dean, etc Support from outside school Support from Career Services consultant Support from another outside consultant Referral outside school [other – visits to tertiary institutions] No Median number of students 16 16 15 20 10 30 20 20 7 2 38 36 35 18 28 45 38 44 10 15 3 5 6 14 9 0 2 2 19 14 Designing Careers Pilot Evaluation Appendices May 2006 Page B12 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION 3.4 Parent involvement Please describe briefly how the school involves parents [Source: Initial Survey Q18] Subject selection by students: Course/subject selection meeting Subject/course option book Parent/report evenings Interviews (invitation, if requested, compulsory) Career evening/PACE? 32 25 17 14 7 parents sign 20 Schools indicated they involve parents in career planning: Parent evenings/meetings 16 PACE 10 Career evenings 7 Notices in newsletter 14 LCPs to sign 11 Contact by Careers Adviser/Careers Adviser available 15 Interviews with Careers Adviser 10 Careers Adviser contacts parents 9 Telephone consultations 6 Has the LCP been used with parents? [Source: 50 Careers Advisers End of Year Survey Q11] 27 Yes / No 18 [10 for individual interviews for some or all students, 5 at subject selection meetings, 11 copy sent or taken home, 2 with student permission, 2 may not come back] Have your parents given you any advice about planning for the future? [Source: 150 students open questionnaire onsite Q7] Choose something you enjoy/are interested in Work hard in school Persevere Get a good job Stay at school Go to university Do a particular job They encourage/push Set goals Aim higher Keep options open 20 20 6 7 4 3 3 3 2 2 2 Designing Careers Pilot Evaluation Appendices May 2006 Page B13 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION What involvement have you had with your child’s Learning and Career Plan? [Source: 23 Parent Questionnaires onsite Q1] Unaware of LCP Discussed with student Helped to find information Signed the Plan 3.5 Accountability 5 Yes 18 Yes 11 Yes 14 Yes Funding If possible, please indicate approximately how the total 2005 funding allocation (including Designing Careers) will be spent: [Source: Initial Survey Q14a] 2004: $ 2005: $ 2005: $ 2005 > 2004 24 schools } 2005 = 2004 19 schools } 2005 < 2004 18 schools } Excluding the specific Pilot funding Including the specific Pilot funding excluding Pilot funding The data for this question was not supplied on consistent basis: some included salaries, some didn‟t; some included GST, some didn‟t. Some schools provided estimates, some said it was too soon to say, and some said they were still planning. Of the schools that supplied information, half are allocating more funding to careers education (excluding the Pilot money), a quarter have the same funding and a quarter are receiving less funding than in 2004. If possible, please indicate approximately how the total 2005 funding allocation (including Designing Careers) will be spent: [Source: Initial Survey Q14b] N of nonzero responses 40 40 38 24 32 31 35 Range $000 0.2 – 48 0.5 – 40 0.2 – 19 0.5 – 23 0.4 – 8 0.5 - 38 0.3 - 92 Median $000 2 6 2 1 2 3 5 % based on $ for all schools 12 22 9 6 5 10 36 Professional development Teacher release Resources, including books Work exploration / work experience Outside speakers/ consultants Computers and related costs Other, state Forty-five schools provided a breakdown of their spending but some of these schools left some categories blank. As schools have not all included the same categories, these figures should be regarded as indicative only. Categories with the greatest expenditure were teacher release, professional development and „other‟. The schools with the high „other‟ values have included salaries. Designing Careers Pilot Evaluation Appendices May 2006 Page B14 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION Use of additional Designing Careers funding from initial questionnaire Please describe briefly how the school is planning to use the additional Designing Careers funding available through the Pilot. [Source: Initial Survey Q2] Schools provided an indication of how they had spent the Pilot funding. The most common uses of the additional funding is intended to be in acquiring more resources and in having a better staffing ratio to enable more individualised student guidance. Resources Teacher release PD Extra staffing for careers Travel, trips Admin, ancillary To work with at risk To do individual interviews Parent meetings Visiting speakers External consultants, programmes 36 schools 20 42 18 11 14 13 4 5 4 8 15 not stated, 5 for interviews 32 not stated, 10 from CS Not possible to distinguish between CS and outside PD providers – most do not say. Most say teacher release but don‟t say what used for eg PD, interviews, develop careers programme. Please indicate whether the school has used the Designing Careers funding available through the Pilot in each of the ways listed. If possible, please indicate the approximate amount spent. [Source: 50 Careers Advisers End of Year Survey Q1] Professional development Professional development from Career services Extra time for the careers adviser For professional development To coordinate or develop the career education programme To carry out individual interviews of Year 10 students To work with at risk students Release time for staff For professional development To carry out individual interviews of Year 10 students Other Resources including storage Administration, ancillary support Travel/trips Outside speakers Computers and related costs Parent information, meetings Yes 38 25 27 24 26 34 18 39 26 32 13 24 22 No 4 12 13 16 13 7 19 1 8 8 16 14 13 Designing Careers Pilot Evaluation Appendices May 2006 Page B15 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION Success Indicators [Source: Report on Initial Survey] The project has a specific success indicator of reducing the number of students who do not go into a job, training or education when they leave school. However, Year 10 students are unlikely to leave school for at least two years. Only half the Pilot schools provided data about the destination of leavers in 2004 and 2003. Many of these schools had no or few leavers who did not go to a job or further education. It will be difficult to show an impact on the key success indicator of the Pilot in these schools. However, some careers consultants and schools appear to have overlooked the need to establish intermediate indicators to gauge the success of the Pilot. Few schools seem to have identified intermediate indicators such as improvements in retention, attitudes, engagement. An early success indicator that has been identified is that careers education has had a raised profile. Self review [Source: Report on Initial Survey] It will be useful when visiting case study schools to explore evaluation strategies and success indicators schools have used to determine the success of the Pilot. It is likely that schools will have determined what worked (and didn‟t) related to the timing, the resources, the management, the delivery and the identification of at risk students. A key area of interest is whether it is feasible to deliver the programme through the form teachers, specific subject teachers or through an integrated school approach. Do you have or plan to collect data on any of the aspects below which could be used to evaluate the Pilot programme and to assess its outcomes? [Source: 50 Careers Advisers End of Year Survey Q21] % Attitude of teachers Attitude of students Improved attendance [3 no time or money to do this] Do you have or plan to collect data on any of the aspects below which could be used to evaluate the Pilot programme and to assess its outcomes? [Source: 50 Principals End of Year Survey Q5] Attitude of teachers Attitude of students Improved attendance Have already 24 33 12 Plan to collect 58 56 21 No 18 11 67 Have already 21 32 5 Plan to collect 60 60 26 No 19 9 69 Designing Careers Pilot Evaluation Appendices May 2006 Page B16 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION 3.6 Impact of the Pilot in the school How well is the Designing Careers Pilot going in your school? [Source: 50 Principals End of Year Survey Q1] Very well 47 Well 49 Not very well 0 Not at all well 2 Too soon to say 2 [11 positive for students, 8 positive for teachers] How effective do you consider the provision of career education in your school? [Source: Initial Survey Q19] % of 75 This year Last year Very effective 36 15 Effective 57 66 Not effective 2 15 Don‟t know 3 5 How effective do you consider the current provision of career education in your school? [Source: 50 Principals End of Year Survey Q2] (%) This year with the Pilot Last year without the Pilot Very effective 60 13 Effective 38 63 Not effective 2 25 Don‟t know 0 0 To what extent has the Pilot extended the provision of career education in the school? (%) [Source: End of Year Survey Q3] To a great extent 41 46 To some extent 53 52 A little 6 2 Not at all 0 0 Don‟t know 0 0 50 principals 50 Careers Advisers [Principals: 18 for Year 10, 10 for at risk, 7 building on existing programme, 5 more staff involved, 5 students making better informed decisions re subjects, 4 Career services important, 3 staff upskilled]. [Careers advisers: 11 built on previous programme, 9 focus on Year 10, 6 more staff involved, 5 more time for careers adviser, 5 developed overall school programme, 3 more resources, 3 links to subject selection, 3 for at risk, 3 raised awareness]. Designing Careers Pilot Evaluation Appendices May 2006 Page B17 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION Please indicate whether the Pilot extended the provision of career education in each of the ways listed below. [Source: 50 Careers Advisers End of Year Survey Q19] Accessing online careers information Posters with career information displayed in classrooms Careers evenings for students and parents Trips to careers events or tertiary providers Speakers from outside school e.g. motivational, goal setting, etc Experience in the workplace [8 already did these, 5 already did trips] Please indicate whether career education and career information is or was provided in each of the ways listed below. If appropriate, state the year level/s involved. [Source: Initial Survey Q17] Number of schools Providing Career Education in Various ways at each Year Level 2005 Year levels No. 9 10 11 12 13+ No. 9 Yes 42 33 15 26 27 23 No 6 13 28 17 19 22 2004 Year levels 10 11 12 13+ As a Social Studies unit As an English unit As a Life Skills unit As a Health unit Accessing online careers information Posters with career information displayed in classrooms Careers evenings for students and parents Careers Expo Deans assemblies Speakers from outside school eg motivational Individual interview with all students Information or interview available on request by students Mentoring programmes Work exploration Work experience STAR Gateway Programme 57 21 37 21 67 68 13 3 1 8 25 39 50 11 3 11 48 44 0 4 25 6 52 43 1 3 29 4 52 43 0 1 19 1 49 43 50 16 35 17 60 63 11 4 1 10 24 34 46 8 3 8 38 39 0 3 20 5 43 41 0 3 27 4 46 41 0 1 16 1 42 43 43 65 49 63 62 69 20 13 16 26 9 47 30 26 25 30 28 55 29 36 44 42 21 61 29 51 44 45 27 61 28 45 46 49 50 59 32 61 39 59 58 68 12 9 14 22 5 46 16 14 19 25 14 50 22 29 24 39 17 55 22 48 30 45 31 56 22 43 32 47 50 56 34 53 53 66 39 7 3 1 4 1 5 12 7 21 1 11 23 33 34 19 15 36 34 51 33 16 29 29 50 32 26 44 49 64 33 3 3 1 4 1 4 8 6 20 1 5 17 23 32 13 8 28 31 51 27 13 23 30 49 28 Designing Careers Pilot Evaluation Appendices May 2006 Page B18 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION Between 2004 and 2005, there was an increase in the number of schools providing career education in each of the ways listed. The largest increases were for career evenings for parents, deans‟ assemblies, work exploration and mentoring programmes. For Year 10, the largest increases were in the number of schools providing attendance at Careers Expos (increase of 19 schools), individual interviews with all students (14 schools), careers evenings for parents (14), and accessing online information (10). To what extent has the Pilot led to the outcomes listed below? End Of Year Survey Q4 Principals (50) % great % to some extent extent 69 29 43 47 42 43 26 20 16 14 47 51 42 31 68 53 61 45 Careers advisers (50) % great % to some extent extent 65 33 54 42 38 31 31 38 21 17 38 54 40 35 61 38 53 46 Improved the provision of career education for Year 10 students Improved support for at risk students Raised the profile of career education amongst students Increased the number of staff involved in career education Enabled the development of a team involved with career education Raised the profile of career education amongst teachers Facilitated networking with other schools Improved the provision of career education throughout the school Provided the school with information to better structure courses and option lines to meet student needs Provided an opportunity to improve the involvement of parents with the school 12 47 17 30 Principals and careers advisers consider the Pilot has resulted in various outcomes, particularly:      Improved the provision of career education for Year 10 students Improved support for at risk students Raised the profile of career education amongst students, Increasing the number of staff involved in career education, and Enabling the development of a team approach. Careers advisers also noted the increased networking with other schools. [7 Pilot fitted in/added, 5 raised profile among students]. Designing Careers Pilot Evaluation Appendices May 2006 Page B19 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION To what extent has the process of developing a Learning and Career Plan led to each of the outcomes below for Year 10 students? % 50 careers advisers end of year Q12 To a great extent 146 teachers end of year Q3 84 teachers case study Q3 To a great extent To some extent To some To a great To some extent extent extent Increased knowledge about careers Increased understanding about the relevance of school to their future goals Improved subject selection Increased knowledge about themselves Increased motivation and/or engagement Raised aspirations Improved relationships with teachers Increased involvement of parents 49 36 49 57 43 31 49 50 42 37 47 44 54 33 22 17 13 7 41 63 59 62 45 39 28 13 12 12 7 1 48 72 44 53 38 19 22 7 8 - 58 39 40 - To what extent has the support provided for at risk students led to each of the outcomes listed below? [Source: 50 Careers Advisers End of Year Survey Q15] To a great extent To some extent A little Not at all Don‟t know Increased knowledge about what they might do when they leave school Increased understanding about the relevance of school to their future goals Increased knowledge about themselves Raised aspirations Increased motivation and/or engagement Improved relationships with teachers Increased involvement of parents 33 24 24 16 14 0 2 56 64 62 52 50 42 24 11 9 13 25 26 37 32 0 2 0 0 2 2 12 0 0 0 7 7 19 29 Designing Careers Pilot Evaluation Appendices May 2006 Page B20 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION Have you decided what subjects you’ll do next year? [Source: 358 student questionnaires onsite Q6] 79% Yes How did you decide? [Source: 358 student questionnaires onsite Q7] To fit future career or job Subjects like/enjoy/find interesting What good at From the LCP/careers unit/ Plan-It Parents Teacher Careers adviser Friends Don‟t know/haven‟t decided Will decide later Keep options open 91 82 19 19 27 13 6 6 28 5 3 Do you know what job you might do when you leave school? [Source: 358 student questionnaires onsite Q8] 2+ different areas Mechanic Doctor Accountant Beauty therapist/hairdresser Builder/carpenter Clothes designer Teacher Lawyer Retail 32 13 10 10 11 11 10 10 9 6 67% Yes / No 30% (some a very wide range) Do you know what qualifications you need for that job? [Source: 358 student questionnaires onsite Q9] No Don‟t know Kind of Particular subjects given University/tertiary Particular skills NCEA level 1 100 15 14 26 17 6 2 Designing Careers Pilot Evaluation Appendices May 2006 Page B21 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION Where do you get information from about jobs and courses? [Source: 358 student questionnaires onsite Q10] Don‟t know Careers adviser Teacher School Career centre Career interview LCP Internet Kiwi Careers Websites Computer Tertiary websites Book Brochures Newspaper Parents Friends People in the job want 30 91 53 23 13 11 6 58 23 18 7 7 12 11 7 48 12 10 Who has helped you most about planning your future? [Source: 150 open student questionnaires onsite Q5] Parents Teacher Careers adviser Myself Grandparents, uncle, aunt Friends Other people 86 37 21 14 11 10 4 Do you think developing a Learning and Career Plan is useful for Year 10 students? [Source: 23 Parent Questionnaires onsite Q3] 13 Very useful 10 Useful 0 Not very useful Has developing a Learning and Career Plan helped your child to choose subjects for next year? [Source: 23 Parent Questionnaires onsite Q4] 9 Yes, a lot 11 To some extent 1 Not really Do you have any information about the outcomes for at risk students? [Source: 50 Careers Advisers End of Year Survey Q16] Un-collated information about individual students Collated information 28 Yes / No 7 13 Yes / No 16 May 2006 Page B22 Designing Careers Pilot Evaluation Appendices APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION Nature of support: Individual interview on one occasion Individual interviews on more than one occasion Helped find career or job information Helped student prepare a CV Helped student apply for job or course Took student to visit workplaces Outcome (if this information is readily available) Clearer about a possible job/career direction Remained at school – improved motivation Remained at school – no change in motivation Applied for a course Left to go to a job Left to go to a course Left with no intended destination Number of students if known 875 390 720 210 140 200 245 200 120 40 45 40 25 To what extent do you think the school is likely to continue the current career education programme in the longer term (when the Pilot funding is no longer available)? [Source: 50 Principals End of Year Survey Q6] To a great extent 54 To some extent 33 A little 7 Not at all 4 Don‟t know 2 [11 depends on funding, 9 will continue, 5 embedded in curriculum] To what extent do you think the school is likely to continue the current career education programme in the longer term (when the Pilot funding is no longer available)? [Source: 50 Careers Advisers End of Year Survey Q22] To a great extent 46 To some extent 37 A little 9 Not at all 4 Don‟t know 4 [20 depends on $, 6 Year 10 programme, 7 career education important] To what extent do you think the school is likely to continue the current career education programme in the longer term (when the Pilot funding is no longer available)? [Source: Initial Survey Q12] To a great extent 50% To some extent 31% A little 4% Not at all 4% Too soon to say 10% Designing Careers Pilot Evaluation Appendices May 2006 Page B23 APPENDIX B KEY DATA FOR SECTION 3 - IMPLEMENTATION Which aspects is the school likely to continue in the longer term? [Source: 50 Careers Advisers End of Year Survey Q23] % Career education programme for Year 10 Year 10 students developing LCPs Individual interviews for Year 10 students Career Services consultant working with at risk students Other support for at risk students Networking with other schools Yes, as is 46 43 28 50 53 45 Yes, modified 54 55 52 30 45 43 No 0 2 20 20 2 11 [11 Career services consultant only if $, 6 ongoing improvement, 4 networking thru clusters, 4 will continue LCP in Year 11] Which aspects is the school likely to continue in the longer term? [Source: 50 Principals End of Year Survey Q7] % Career education programme for Year 10 Year 10 students developing LCPs Individual interviews for Year 10 students Career Services consultant working with at risk students Other support for at risk students Networking with other schools Yes, as is 45 50 35 50 31 34 Yes, modified 55 50 50 40 67 57 No 0 0 15 10 2 9 Designing Careers Pilot Evaluation Appendices May 2006 Page B24 APPENDIX B KEY DATA FOR SECTION 4 – EFFECTIVE PRACTICES 4 School Views on Important Elements for Effective Careers Education How well does the career programme provided through the Pilot integrate with other career education activities in the school? [Source: 50 Careers Advisers End of Year Survey Q17] % Year 7-9 career education Year 11-13 career education Opportunities for experience in the workplace Provisions for at risk students STAR courses Gateway programme Very well 41 53 47 63 50 83 Well 47 42 37 37 42 6 Not very well 12 5 16 0 8 11 Important elements for effective career education % Principals end of year (50) Very important Important Careers advisers end of year (50) Very important Important Teachers case study schools (151) Very important Important Learning where to find information about careers Developing self awareness Links with other career education programmes e.g. Gateway, STAR Career education integrated into one or more curriculum area Involving parents All students developing a learning and career plan (Year level 10) Schools providing subject options that are driven by student choices Work experience/work exploration Subject choice evenings that include career information Learning about specific careers Students choosing subjects that relate to a likely future job Access to an external career consultant 73 76 70 27 24 30 84 78 73 16 22 25 68 53 39 31 44 51 61 76 67 30 24 33 64 57 59 34 43 41 35 31 40 51 63 47 60 40 51 47 28 55 65 57 23 25 27 35 39 64 61 56 61 64 40 52 43 36 36 56 45 41 34 35 43 30 26 56 57 52 57 50 Principals, careers advisers and teachers all considered the most important elements of an effective career education programme are:  developing self-awareness and  learning where to find information about careers. May 2006 Page B25 Designing Careers Pilot Evaluation Appendices APPENDIX B KEY DATA FOR SECTION 4 – EFFECTIVE PRACTICES Principals and careers advisers, and teachers to a lesser extent, also considered it important that:  there are links with other career education programmes.  career education is integrated into one or more curriculum areas  parents are involved  students should develop a learning and career plan  schools provide subject options driven by student choice  career information is included at subject choice evenings, and  include work exploration. What does a Year 10 student need to know about careers? [Source: 358 students onsite questionnaire Q1] % Jobs that fit with your interests The best subjects to take Where to get information about jobs or careers Who to ask for information about jobs or careers Strengths you have How to get a job Where you might want to work About a specific job/career Yes 89 86 82 78 72 72 68 65 Don‟t know 10 12 15 19 24 22 26 26 No 1 2 3 3 5 6 6 9 Does it matter if a Year 10 student has no plans for the future? [Source: 150 students open questionnaire onsite Q6] No: Have plenty of time to decide Too young Will decide when ready Would probably change mind Year 11 better Gives you a more open mind Planning helps choose right subjects You‟ll drift/get nowhere You get further if you have a goal You‟ll get good qualifications and skills 37 10 8 6 4 3 19 8 4 3 39%Yes / No 59% Yes: What do Year 10 students need to learn about for career education? [Source: 23 Parent Questionnaires onsite Q6] What they are good at What their interests are What their values are Some ideas about possible careers or jobs Where to get information about jobs and careers How to set goals and plan 23 Yes 23 Yes 23 Yes 21 Yes / No 2 23 Yes 23 Yes Designing Careers Pilot Evaluation Appendices May 2006 Page B26 APPENDIX B KEY DATA FOR SECTION 4 – EFFECTIVE PRACTICES How important is career education for students in your school? [Source: 151 teachers (general) Q1] % To help students choose appropriate subjects To motivate students generally To provide a way to involve parents To raise aspirations Very important Important Not important Don‟t know 64 46 22 59 34 48 53 37 2 5 16 3 0 1 7 1 What do you consider are the main barriers to providing effective career education in the school? [Source: Initial Survey Q20a] The constant response to this question is the lack of time available to meet the demands of career education. Responses consistently list: time – few hours allocated to careers education 51 timetable constraints eg “crowded curriculum” 16 funding limited eg for computers, external services 12 personnel, lack of training, other staff uninterested, low status 7 limited resources, eg language too hard, some outdated, some costly 6 physical environment – No dedicated Careers area 4 parental attitudes 9 What do you consider are the main barriers to students from your school making a successful transition from school to further education, training or work? [Source: Initial Survey Q20b] Students Lack of qualifications Poor subject choice Motivation/commitment to complete training Lack of knowledge/info re careers Unrealistic expectations Lack goal setting, planning Lack of work ethic Cost of courses Lack of confidence Absenteeism Short term focus Drugs, alcohol Family Family need for money Lack of family support Unrealistic expectation Low education/work experience of parents Parent perception Community Isolation/have to move Unskilled jobs available locally Number of schools 16 4 13 11 9 8 8 7 6 4 4 3 10 9 8 7 6 9 4 Designing Careers Pilot Evaluation Appendices May 2006 Page B27 APPENDIX B KEY DATA FOR SECTION 4 – EFFECTIVE PRACTICES What do you think your school should do to help you decide about your future? [Source: 358 student questionnaires onsite Q11] Course/topic/seminar Work experience Provide information about quals and subjects needed for jobs Information on a wider range of jobs Career programme LCP Bring in people to talk about their jobs Advice Help with subject choice Individual talks with career planner Help choose a job Encourage students Talk to us Help us More options Nothing/it‟s fine now 18 15 11 10 10 9 9 16 15 13 11 12 10 7 18 24 Would knowing your likely future career make you feel more positive about school? [Source: 150 student open questionnaires onsite Q2] 80% Yes Designing Careers Pilot Evaluation Appendices May 2006 Page B28