Evaluating and Monitoring Study Support Activities by xiaohuicaicai


									Monitoring and Evaluating Study Support Activities

One of the purposes of the government‟s drive to develop extended services in and
around schools is to raise standards in schools. As part of the core offer of extended
services, programmes of Study Support are the most readily achievable way of doing

Study Support is at the heart of school improvement. It is relevant to every child in
every school because it has three overlapping purposes, each of which has raising
students‟ achievement at its core:
     Removing obstacles and developing readiness for learning
     Increasing competence at learning
     Broadening and deepening success in learning.

Study Support can contribute to raising standards either directly (for example through
increased attainment as a result of activities such as paired reading schemes,
homework clubs or revision sessions) or indirectly (for example through increased
motivation or improved attendance in school).

School governors and the senior leadership team will want to review how their Study
Support programme is supporting the delivery of the national strategies and school
improvement plan targets, and ensuring that the five outcomes of Every Child Matters
are met.

Therefore it is important to monitor the activities provided, and to evaluate their
success. This can then be used to further develop the programme of activities.

What is monitoring?

Monitoring is an ongoing process – to look at what is happening with your study
support programme, for example who is attending and what they are doing when they
are at the club. An example of monitoring is keeping registers of attendance – you can
show that an activity is successful if pupils keep coming back.

What is evaluation?

Evaluation is a structured way of thinking about what happens during your project, and
why. It can be simple or complex, depending on the resources you have available and
what you want to find out. A key part of any evaluation process is setting objectives
and intended outcomes for the projects, and then gathering evidence to see whether
these objectives have been achieved. The diagram below shows the steps you may
take in your evaluation process.
      The Evaluation Process
      From Evidence and Evaluation, Teachernet website

                               Determining what you want to
                               find out & how you will identify

                               COLLECTING EVIDENCE
                               by using appropriate research

                                Assembling and interpreting

                                       on and sharing

                                 and moving forward using
                                findings to improve practice

Planning your evaluation

The evaluation planning template in the toolkit is a useful tool for setting out what you
want to achieve and how you will go about it.

Quality assuring study support activities

As part of the evaluation of your study support programme, you should carry out
regular quality assurance checks on the activities that you are running, particularly
those offered by external providers. It is also good practice if you are able to do this for
activities that you signpost to.
You may like to involve others in carrying out the observation visits, for example
governors, or pupils. The QA template in the toolkit can be used when you undertake a
quality assurance visit – remember to discuss your observations with the providers
concerned as a means of improving provision.
Your observation notes might include comments on safeguarding pupils, how the tutor
is evaluating their activities and measuring impact, pupil comments etc.
The evaluation schedule in the toolkit is a useful way of providing an overview of the
visits you have undertaken. You may decide that you would like to visit all clubs on a
rotational basis, or that you focus your attention on external providers or those judged
to be performing at a satisfactory level or below.

Measuring the Impact of your study support programme

One important evaluation question to ask in relation to your study support activities is
“what impact is the activity having on the young people taking part?”. This could be
done for individual activities or for the programme as a whole -what difference does
study support in general actually make? This information can then be used for a wide
variety of purposes, as summarised in the diagram below:

                                                     Are you meeting your needs and
   RECRUITMENT                                       objectives?
   of pupils                                         Funding
   of staff                                          Reporting e.g for SEF
   of organisations

                                 Why measure impact?

Assess outcomes
Effects on specific groups                IMPROVEMENT
Trends over time                          Targeting
Value for money?                          Programme details eg locations / times / staffing
                                          Inform planning and help shape future

Many schools and organisations have found that measuring the impact of study
support activities is not always an easy task, given the wide-ranging benefits that
reportedly result from such activities and the difficulty of isolating the impact of study
support. Questions of „what to measure‟ and „how to do it‟ are common, and there is no
one correct answer. It may be that you want to measure the direct impact of study
support on academic attainment, or that you want to focus on soft skills such as
emotional awareness or self-esteem. Some data may be easier to collect than others –
as shown in the diagram below:

           EASY                           DIFFICULT

                                                Attitude to school
             School attendance
                                                Attitude to learning
             Test scores
                                                Self esteem
             Skill acquisition


There are numerous different methods for collecting the data you require – these are
discussed in more detail in The Evaluation Factor (Wigan Council, 2006). Whichever
tool you use, you will need to collect either qualitative or quantitative data.

Types of Evidence: qualitative and quantitative data

Evidence based on quantitative data is numerical. Any kind of figures such as data on
participation (i.e. records of attendance), percentage of participants who are satisfied
with the service, reading ages, pupil attainment, number of reported crimes in the area
etc are all examples of quantitative evidence. Such evidence may be obtained using
research tools such as administrative records, survey questionnaires or local area

Evidence based on qualitative data is non-numerical and based upon interpretations,
observations, accounts and opinions. Examples include participant comments,
observations of activity/staff involved, interviews and focus groups and open ended
questions on questionnaires.

It is usually preferable to gather a mix of both qualitative and quantitative evidence,
and to investigate changes over time.

Where to start?

The measuring impact planning sheet in the toolkit can help you with planning a more
in depth evaluation of the impact of a particular activity.

It follows a similar structure to the Evaluation Process diagram above: you should start
with a question you would like answered, such as “Does attendance at the science
club improve attainment in science?”. You will then need to consider the data you will
need to collect. Because impact relates to improvement, you will need to consider data
for both before the activity (baseline data) and afterwards. For the above example you
would need to collect attendance data (to know who has attended the club), baseline
data such as predicted GCSE results and post activity data such as actual GCSE

             Possible Questions to ask

             To what extent did activity X achieve its objectives?
             Does attendance at activity X improve academic
             Does attendance at activity X develop personal
             What type of pupil benefits most from activity X?
Data collection resources

As mentioned above, Wigan Council has produced an informative guide to evaluating
Study Support activities which explains many of the potential data sources for your
evaluation. It also contains resources that can be photocopied or alternatively adapted
to suit your own particular needs or used as a stimulus for developing your own
activities. It can be downloaded from

The National Evaluation of the Children‟s Fund (NECF) has also produced The
Evaluator’s Cookbook – a resource book of participatory evaluation exercises for use
with children and young people. It can be downloaded from http://www.connexions-

The toolkit contains several resources that may be of use in evaluating the impact of
study support, including school-wide surveys, and an impact evaluation for younger
pupils. These can of course be adapted to suit your own needs. You might also like to
think about getting pupils involved in designing or carrying out the research.

Alternative methods of data collection

Don‟t be afraid to try out different methods of data collection to find out what works well
for your school and community.

Some schools have found that using an online survey tool has really helped increase
response rates. It is very simple to create a basic online survey using free tools such
as www.surveymonkey.com or www.surveymethods.com. These programmes can also
be used to collate results, although you will need to subscribe to be able to access the
more advanced features of the survey tools (discounts may be available however for
educational institutions).

Kent Children‟s University recommends the use of interactive voting tools as a really
effective way to engage young people in evaluating activities. There is a range of
hand-held voting pads available, from simple ones with „yes‟ and „no‟ buttons, to more
complex tools with text entry functions. The software makes it easy to create question
and answer sessions and children and young people enjoy using them. An additional
benefit is that responses are instant and can be traced back to individuals. Qwizdom is
one provider of such tools (www. Qwizdom.co.uk). Many schools may find that they
already have a set being used in classrooms, or cluster may like to share a set for use
in evaluating their extended services, including study support.

Other things to consider

As well as thinking about what data you are going to collect you should consider your
sample size and whether a control group is possible.

Sample size is only a concern when you want to draw general conclusions from your
findings. If you are measuring the impact of a single activity it is likely that you can
assess the whole group and won‟t need to consider issues of sample size and
composition. If there are too many students to record the results of the whole group
you will need to use a sample group, and it is important to ensure the sample is of
sufficient size for results to be representative. As a general rule, a sample size of 30 is
considered sufficient for statistical purposes. This issue most often arises when a
school wishes to look at the overall impact of study support, i.e. comparing the
progress of pupils who id not attend any form of study support against those who did.

Control groups are helpful in determining whether observed outcomes are the result of
the study support activity or whether other factors had an influence. The control group
should ideally be similar in every way to those who attended the activity, the only
difference being that the control group did not attend. An ideal control group is often
difficult to set up, but conclusions will always be more reliable if even an imperfect
control group is used.

Hints and Tips

        Make sure what you ask is relevant and covers the main issues. If
        information is interesting but you are not really going to use it, don‟t
        waste people‟s time by collecting it

        Make clear the aims and objectives of the project, for example to
        increase the number of girls attending after school ICT activities, and
        work out your key indicators of success, e.g. regular attendance of
        60% of the target group, more parents from vulnerable families
        contacting the school informally etc

        Try to establish baseline quantitative evidence before the
        project/service begins, for example the number of late arrivals or the
        number of unauthorized absences. Then you can collect the same
        evidence at the end of the project and use it to assess any changes.

        Try to encourage the involvement of key stakeholders in gathering
        evidence. This will promote engagement with, and ownership of, the
        project. For example, students may be involved with the questionnaire
        design and in the collating of responses; gather views from a range of
        people such as members, teaching staff, partners and parents

        Be creative in your approach to collecting evidence – for example,
        questionnaires can be created online, feedback can be collected using
        photographic or video evidence (but be aware of safeguarding issues
        around collecting evidence – you will need written permission from the
        children and their parents/carers to take photographic or video
        evidence, and data protection issues need to be considered if you are
        collecting personal information about individuals)

        Work out what you are going to do with the information – know how
        you will collect, analyse and feedback the results
        Make one person responsible for the process of collecting data and
        designate people to analyse and feedback the results

        Use your feedback to feed forward! In other words, use your data not
        only to establish the extent to which projects have met their aims and
        objectives, but also to inform future planning.

And most importantly, don’t be afraid to start!
Evaluating your study support activities does not have to be overwhelming – start
small, perhaps by looking at the attendance and enjoyment of one particular activity,
then build up to more detailed evaluations such as looking at improvement related to
one particular club, which you can use to inform targeting and planning. You may then
feel able to move on to a larger scale measurement of the impact of study support
across the school and action research into study support and its effects.

Quality in Study Support (QiSS)

The Quality in Study Support (QiSS) programme provides a range of services to
support the development of study support and other extended services.

QiSS produce the Study Support Code of Practice – the new version is titled
“Extended Learning opportunities– a framework for self-evaluation in Study Support”.
This document sets out principles of good practice in study support and provides key
indicators and a process for self review. The QiSS process can be used as a tool to
ensure the quality of Study Support in a school, leading to the award of a recognised
quality mark.

One of the main aims of QiSS is to provide support and professional development
training for schools, among other organisations, in the impact, measurement and
evaluation of study support provision. They have produced a set of leaflets to assist in
measuring impact, which are available on the QiSS website.

For more information about QiSS please contact the Cambridgeshire County Council
extended learning team or visit the QiSS website: www.canterbury.ac.uk/qiss/.

To top