BID FOR by hcj


									                                 BID FOR
                           2003-4 SEMESTER 2

    Validating the methods of assessment used during the dissertation
     phase of postgraduate management courses offered in CHPM at

Submitted by Jean Robson, Lecturer and MBA(HPN) Programme Director,
and Sarah Derrett, Lecturer and MBA(HE) Programme Director, CHPM, Keele

Problem Statement

CHPM offers a portfolio of postgraduate degree courses in a specialised field
of management in health for the UK and the international market. The
original postgraduate degree course was a Health MBA, initially established in
1986, which has since developed and specialised over the last 17 years into
seven different masters degrees. Each master’s degree requires the
completion of a dissertation/management project component worth 60 M
level credits. Students complete their dissertations over different time periods,
( 4 – 12 months) depending upon whether they are full time or part time
students. Most students are mid-career professionals with extensive work
experience and fairly specific career expectations.

While there is complete agreement on the importance of the
dissertation/project experience for the award of a Masters’ degree at every
level ( professionally, institutionally, academically), there appear to be
different perceptions at faculty level regarding the aims and outcomes of the
experience. This has emerged through a series of departmental faculty
seminars set up to review the organisation and management of dissertations
over the last two years. Differences between faculty is not a new or even a
negative phenomenon. There has been some interesting work done by
Quinlan (2002)1 exploring how academics assess scholarship. This earlier
promising initiative has now run out of steam, largely through pressures on
staff time and the need for a specific project with defined outputs and
specifically assigned responsibilities.

Student perceptions of the dissertation experience within the department
have been monitored episodically over the last five years, using quantitative
and qualitative data, and the department needs to analyse this data. This
analysis might be used to refine further data collection and analysis on the
different ways students perceive and proceed with their dissertations. Student
expectations and experience of the dissertation are clearly variable, and as a

 Quinlan,K(2002) Inside the peer review process: how academics review a colleagues teaching
portfolio, Teaching and Teacher Education, vol 18, pp 1035-1049

department we need to explore how different expectations determine
independent learning outcomes and achievement.

A further complication arises in the nature of the academic award on
completion. MBA programmes have largely developed out of a philosophy and
pedagogical practice which maximise work/learning relationships. This flows
through into the dissertation component, permitting students doing MBA
courses to undertake more practical management projects. These are often
problem-solving exercises related directly to their immediate employment in a
management capacity. However, in the department, there are several MSc
courses where employment and learning are not necessarily as closely
aligned, and which might require more ‘classical’ research skills to be
developed during the course of the dissertation. We propose to review the
work of the Society for Research in Higher Education(SRHE) on Project
Methods in Higher Education2 with regard to designing and maximising the
validity of project assessment regimes and to assess the extent to which
project and research dissertation assessments can/should be combined into a
single format.

One reason for including dissertations as a required component of the
postgraduate experience is to consolidate independent learning skills, and to
provide students with the opportunity to be more self-directed in a specific
topic of their own choice, thus meeting individual learning needs. A high
proportion of our students work independently through their dissertations for
various reasons but currently the department do not assess the process and
the degree of independent learning that has taken place, merely the product
(dissertation). Through this project we would intend to identify and compare
other department’s practices in Keele and elsewhere in assessing the
independent learning outcomes occurring during dissertations, and explore
the feasibility and validity of using some forms of self-evaluation.

Despite standard requirements for dissertations for every course within the
department, there appears to be considerable variation in the quality of
submitted dissertations within and between courses. This may be partially due
to the heterogenous nature of the student population within the department,
and the fact that there are seven different programmes using assessed
dissertations. As all programmes assign a similar critical value of 60+M level
credits, the department needs to reduce the variability between dissertations
offered through different courses, if indeed the variations are significant.

Discrepancies between markers throw up questions on the reliability of the
marking schemes used in the department. The department, in common with
university practice, uses blind double marking for all dissertations, but the
proportion of dissertations requiring to go to a further adjudication stage
suggest that there might be problems of reliability. To increase reliability we
might require either revision of the marking scheme and/or further training of

    Adderley,K et al ( 1979) Project Methods in Higher Education, SRHE, Guildford, Surrey

staff to reduce between-marker discrepancies. This problem of inter-marker
reliability is common and persistent throughout education3 and should not be
overstated, but might be amenable to further reductions within the

As reliability is partially contingent upon the validity of any marking scheme it
is proposed to simultaneously review the criteria and mechanisms adopted
within the department to assess dissertations and to explore the inclusion of
peer and/or self-evaluation mechanisms into the entire process at different
stages. Some of the department programmes are using peer evaluation at
early stages of the dissertation, and we would intend to review the
comprehensiveness, efficacy, timing, and utility of these approaches. Self-
evaluation has not been formally adopted within the department, and we
need to explore other departments’ experiences in this area before proposing
alternative practices.

Project Aim

1. To evaluate the quality of student (independent) learning in CHPM
2. to improve guidance for supervisors of MBA students
3. To develop reliable & acceptable dissertation marking schemes


Improve the fit between output and purpose for the dissertation component
in all seven CHPM master’s programmes.


   Improved supervision and assessment skills among all CHPM faculty
    members involved in supervising and/or assessing dissertations
   Increased ability among postgraduate health management students to
    accurately assess/predict own performance, and that of their peers
   A robust, high quality assessment procedure for dissertations which is
    readily accessible, reliable, valid and with a high degree of compliance
    from users. Access will be in multiple forms including on-line, CD Rom,
    and hard copy.
   A useful model/template which can adapted/adopted by other user groups
    at Keele.
   A summary evaluation report

 Klein,J(2002) The failure of a decision support system: inconsistency in test grading by teachers,
Teaching and Teacher Education, vol 18 , pp 1023-1033

Project Activities

1. Assess department performance with regard to dissertation experience:
   declared/not declared purposes; variations in student and staff
   perceptions and preparedness ; format (project vs research) ;
   enhancement of independent learning; parity between courses ; reliability
   in measurement; validity of measures used; access to learning support.
2. Review evidence from other sources and departments, within the
   university and outside, regarding current and best practice for managing
   dissertation experience.
3. Revise/Develop and test dissertation framework and mechanisms including
   new marking scheme, modes of self-assessment, modes of peer-
   assessment , contracts of expectations/independent learning.
4. Staff training for all (30) full-time and part-time faculty members who will
   supervise or assess dissertations through seminars, mentoring, briefing
   sheets, and self assessed questionnaires.
5. Evaluate and report back to CHPM and Learning and Teaching Support

Inputs Required

Support to the sum of £2,750 is requested from the Innovation Project Fund
in order to expedite the following inputs:

   Offset staff time, including research assistance   £1000
   Consultancy fees ( eg. software development)       £750
   Administrative (temp. clerical) assistance         £300
   Travel costs                                       £300
   Staff training seminars                            £200
   Materials, stationery, CD Rom, photocopies         £200

Value Added to Keele

Teaching staff feel well supported and valued, and the innovation should
improve the quality of teaching and in the long term, improve the reputation
and value added-score of the entire institution.

First draft : 23/01/04, jr.


To top