1 HIGHER EDUCATION QUALITY IN FLORENCE UNIVERSITY FIRST ANALYSIS

							HIGHER EDUCATION QUALITY IN FLORENCE UNIVERSITY: FIRST ANALYSIS OF A SELF ASSESSMENT MODEL IMPLEMENTATION Prof. Ing. Mario Tucci, Ing. Luca Cellesi Dipartimento di Energetica “Sergio Stecco” Università degli Studi di Firenze Via C. Lombroso 6/17 – 50134 Firenze mario.tucci@unifi.it luca.cellesi@unifi.it

Abstract

Quality management in higher education had a large diffusion in all Europe in last 10 years. Mostly after Bologna Declaration (1999), quality assurance became a crucial element of each country’s university structure, proposed in every document and every resolution of the UE member states governments and their educational ministries, with the consequent great development of many assessment schemas, and adaptation of ISO 9000, EFQM excellence model or even other international models inspired to these. In Italy the best established model for quality in higher education is the self assessment CRUI (Italian Universities Rector Conference) model, introduced firstly in 2001 as Campus and CampusOne projects and in the following years reviewed. This model considers a 5 dimension evaluation scheme (including management system, requirements and objectives, resources, teaching process, results analysis and improvement), to completely describe a higher education organisation. About two years ago, Florence University, following its Governing Council resolution to adopt goals and strategies emerged from European and International Knowledge Society scenario, started a process to obtain the CRUI model compliance certification of all degree courses to satisfy internal and external requirements (i.e. national or local authorities constraints related to special funds assignment). After two years, 88 degree courses, including first level degrees (3 years), 5-year specialised degrees, Masters and post-graduate schools, have been certified and now it’s time for a preliminary results analysis. During the certification process, a Self Assessment Report (RAV) was prepared by each Self Assessment Course Group (GAV): in this document the courses were described in detail according to the evaluation model’s 5 dimensions. In this paper, although data analysis is still going on, after a model fundamental characteristics brief description, investigation is conducted on those 1

reports in order to better understand how courses have faced some specific elements, for example: the importance of a clear quality policy and correlated objectives definition, unavoidable starting point for every quality management system implementation; the consistent stakeholders careful individuation, necessary for an effective implementation; an accurate graduate’s roles specification, coherent with course mission and so on. For each element the paper tries to point out differences and analogies, critical issues but even revealed good practices together with their reasons.

Introduction
Quality implementation in higher education institutions is a great challenge[1] in all Europe, especially after Bologna Declaration (1999). Each country have established his own assessment model for quality assurance in universities, generally inspired to international schemas, like ISO 9000, EFQM excellence model[2] and so on. Actually, we have also to distinguish between national quality assessment models, and ones chosen by each universities: in fact it often happens that a single institution adopts a specific reference scheme in order to better comply the imposed requirements. In Italy, although a lot of different quality initiatives were implemented, CRUI (Italian Universities Rector Conference)[9] developed a suitable model for didactic activities quality assessment. The first model draft was introduced with the Campus project, involving about 30 Italian universities in the period 1995-2000[3]. The framework already provided the self assessment methodology acceptance and was organized in four fundamental parts: management responsibilities, resources, process management, assessment and improvement. The new model version was presented with CampusOne project in 2001 and was utilized until 2005. For the first time appeared the five dimension assessment framework based on: requirements and targets, organisational system, resources, formative process, results analysis and improvement. A further upgrade (even though most of basic elements were unvaried) was introduced in 2005 and is presented in the following pages together with its implementation in University of Florence.

The CRUI quality assessment model
The CRUI assessment model [4] was from the beginning purposely defined for the degree courses quality evaluation, trying to adequate typical elements of ISO 9001 model to this particular application field. The model, even from the 2001 version, identify five principal processes (named “dimensions”) able to completely describe the course activity: 2

1. management system 2. requirements and targets 3. resources 4. Educational process 5. results, analysis and improvement. This five elements are briefly described below and they are synthesised in Figure 1

Figure 1 – CRUI quality assessment model scheme

a) Management System The degree course has to define how the processes of the 5 evaluation dimensions are managed, including: processes identification, process splitting up into basic activities, individuation of interactions between different processes, definition of responsibilities and organisational structure. In this section each course has also to define all the documents necessary for the processes management (i. e. handbook, registration, forms, …) and the way of communication with other actors involved. Moreover, the periodical review of all the system has to be described in details, in order to verify its adequacy, suitability and effectiveness. b) Requirements and Targets The definition of a complete list of all the stakeholders involved in the course activities is requested, including making the respective requirements clear. According to them, a set of general goals is created together with specific and coherent policies able to support their achievement. A very important part of the goals is represented by a careful individuation of roles for graduated students, finally the real objective of each course. Moreover, 3

a special category is represented by “learning goals”, the specific characteristics that the course want to transfer to students, generally referable to particular knowledge, ability and behaviour. c) Resources Each course have to indicate its requirements about human resources (including teaching, support and non-teaching staff) and facilities. The adequacy of available resources has also to be demonstrated or, alternatively, the course must declare how the deficiencies would be solved, remembering that the requirements must consider not only didactic activities but even all the support processes necessary for working. d) Educational process This is probably the most important part of the assessment model because the course must describe here a lot of elements: - supplying of didactic activities, including teaching contents, characteristics and their consistency with learning goals - expected activities implementation control - adequacy and effectiveness verification - supporting processes management, like admittance guidance, tutorial, vocational training support, internationalization and job placement. Sometimes, to help in this analysis, educational process can be seen as a “transformation system” or a “production process”[5] in order to better individuate all the elements listed above. e) Results, analysis and improvement First of all, the definition of typologies of data to be collected is requested, together with their acquisition modality. Then the course must al least illustrate obtained results about: - Students attraction capability - Didactics activities effectiveness - Supporting processes effectiveness - Graduate placement monitoring Finally, for each model dimension, strength, weakness and improvement opportunities have to be listed. The evaluation process is based on a two step scheme: 1. self-evaluation 2. external evaluation 4

During self evaluation, each Course Self Assessment Group (GAV) purposely constituted, draws up a Self Assessment Report (RAV) to describe in details the implementation of the five dimensions in the course organisational structure. To make the GAV work easier, each dimension is represented in the model by a set of “elements” and “factors” going thoroughly into specific aspects to be evaluated. The external evaluation is conducted by a group composed by teachers and other stakeholders representatives. The group first examines the course RAV and then carries out a visit on site among the course structures. In the end, an Evaluation Report and a Certification Report are released.

The CRUI model Implementation in University of Florence
At the beginning of 2004, University of Florence Governing Council, driven by internal and external requirements (i.e. national or local authorities constraints related to special funds assignment), decided to adopt goals and strategies emerged from European and International Knowledge Society scenario. Consequently, a group of referents from the involved structures was selected and a two years long project started to obtain the CRUI model compliance certification of the greatest part of degree courses. In the first project, this steps were scheduled: • • • first stage (June 2004 – June 2005): 37 degree courses certification second stage (January 2005 – January 2006): 17 degree courses certification third stage (May 2005 – May 2006): 23 degree courses certification

in each stage the evaluation of a certain number of Masters and post-graduate schools was included as well, for a total of a about 88 certifications for all the 12 faculties. The project activities also include GAV members specific formation and the supplying of suitable tools to help their work, for example a web application for the “publication” of RAV to facilitate collaborative development and assessment by the external auditors. After about two years, we can observe that the deadlines listed above has been substantially matched and all the involved courses have obtained the certification. In order to better understand the consequences and possible future developments of these evaluation activities, we now try to analyse some specific elements of the RAV to point out how the degree courses (78 on a total of 88 – for the others we have no data) have faced them and how they were considered by external auditors (where the external evaluation report are already available – 48 on 78). Obviously, a complete analysis is a very long activity, considering that each RAV is an almost 100 pages long document describing a great number of aspects. So it’s impossible by this time to have 5

all the model’s 5 dimensions examined in this paper and we have made a first choice, focusing our attention only on a little number of strategic elements. Their evaluation should be however significant enough for a first discussion about quality management implementation level in University of Florence degree courses.

The analysis
First of all we have selected 4 elements to limit this preliminary analysis and they are listed below. For each, we first report about RAV contents, trying to classify their answers, then we insert a reference to external evaluation (even if these information are not yet available for all courses). 1. Processes description. As the first element of “A” dimension, degree courses have to describe the main processes regulating courses functioning, including responsibilities, interactions, basic activities and so on. The description can fulfil the evaluation model’s 5 dimensions scheme or even consider other reference framework (as far as all the requested information is available). This description is very important in order to point out if the management system is really under the course control, mandatory condition for an effective TQM principles implementation 2. Stakeholders individuation together with their requirements. The degree course have to define who are the stakeholders of its activities and to make their requirements clear because these are the only references to evaluate efficiency and effectiveness of the course itself. Only stakeholders requirements clear definition allows to reach their complete satisfaction. External stakeholders are also becoming increasingly important to ensure the social engagement of institutions and external environmental scanning is recognised as an important function that requires further development[6]. 3. Graduate’s roles specification. The number of possible roles for graduated students, together with an accurate definition of them, is probably the most important parameter to evaluate the real “usefulness” of a degree course and the ultimate goal of its institution. 4. Quality policy and correlated objectives definition. This is the unavoidable starting point for every quality management system implementation and is necessary even for degree courses. The policy have to consider all the critical aspects of the management system and the objectives must be measurable, coherent with the policy and oriented to the improvement of the performance. Moreover policy is a very important element to evaluate how the management of each course have already acknowledged quality culture, remembering that “leadership” is one of the critical success factors for sustaining continuous improvement in any organisation.[7] 6

About external evaluation we just consider the number and typology of “non conformities” (NC) pointed out on the reports, trying to classify them in some main categories. Process description All the analyzed RAVs draw a process description, but there are great differences. First of all, several courses individuated different numbers of processes necessary for the activities management, from a minimum of 4 to a maximum of 16, with an average of 7,4 (see Figure 2). More than 75% defined “sub-processes” as well, organized on 1 (38,5%) or 2 (35,9%) hierarchical levels. The number of sub-processes varies from 4 to 43 with an average of 23,5 (see Figure 3) but only in a very limited number of cases, the description is accurate and complete. We can here remember that a complete process description includes at least: input and output elements definition, exact activities sequence, responsibilities and used resources.

Number of processes identified
18 16 14 12 10 8 6 4 2 0
Figure 2 – Number of processes identified by each course (the last value on the right is the average value)

7

Presence and number of sub-processes
50 40 30 20 10 0
Figure 3 - Number of sub-processes identified by each course (the last value on the right is the average value)

We can finally observe that about 72% of degree courses have inserted in their RAV a graphic representation of the processes (Figure 4), using different frameworks and instruments. Which could be the reasons of this differences? Eventually, all the degree courses have almost the same structure and the same organization chart, but the attention and capability for an exact description changes. Belonging to a specific scientific area can be an important factor: generally, degree courses in engineering or economy area can have an helpful background in their faculties about quality management, but we find interesting description even in humanities courses. This is also confirmed by Figure 4 where we obtained the same percentage for scientific and humanities courses about presence of graphic representation. A possible explanation for this difficulty in processes description can also given by the necessity to fit old behaviours and procedures to these new framework, which require a careful analysis and a strong commitment to really implement total quality management. Moreover some of this old activities haven’t any type of formalization and they aren’t registered in a specific form, so increasing the trouble connected with describing them.

8

Presence of graphic representation
100,0% 80,0% 60,0% 40,0% 20,0% 0,0% Total Scientific Humanities Yes; 71,0% Yes; 73,9% Yes; 71,8% No; 29,0% No; 26,1% No; 28,2%

Figure 4 – Presence of graphic representation to enhance processes description

Considering external evaluation we find that only 8,3% of courses (that is 4 on 48) don’t have any NC reported for this element. The main NC categories and their frequency are reported below (Figure 5).
NC typologies
improving documentation; 4,2% severe incompleteness; 20,8% incomplete process schedule; 10,4% inconsistent description; 6,3% no efficacy control; 2,1%

inaccurate processes relationship and sequence; 60,4%

incomplete description; 43,8%

Figure 5 – NC typologies

The greatest problem regards the incompleteness of description (almost 65% of degree courses, with a 20% of cases where the incompleteness is “severe”), while another 60% presents an inaccurate individuation of relationship and sequence. Undoubtedly, in the next RAV edition, a lot of degree courses must completely review their processes description, as we can also argue by the analysis results. 9

Stakeholders and requirements individuation The following figure (Figure 6) represent the most significant stakeholders pointed out in the different RAV.
Individuated stakeholders
80 70 60 50 40 30 20 10 0
st se ud co en nd ts ar y sc bu ho si ol ne ss w pr or of ld es tra si on de In st al as itu s so tio ci ns at io an n gr d ad pu ua bl ic te co d sp m ec pa ia ni lis es ed de gr U ee ni ve te ac rs ity he rs of Fl Sp or ec en ifi ce c bo ar M d IU or R ca ce m rti pa fic ot ny at he io n rd bo eg ar po re d st e co gr ad ur se ua s te sc ho if t sc s/ ol ie m s nt od ifi ul c ip an ro d f cu fa ltu m ra ilie lc s om m un Ita ity lia Er n as St m at us e pr ot og he ra ru m ni ve w rs or iti ki es ng st ud en ts ga v

Figure 6 – List of individuated stakeholders

Students, are the only stakeholder indicated in all the analysed documents and other 7 of them (University of Florence 75, Teachers 69, Institutions and public companies 67, business world 65, secondary schools 63, trade associations 55 and professionals 53) are mentioned by more than an half of courses. The total number of stakeholders for each course vary from 5 to 14 with an average of 9,7. Two main reasons can be indicated for these differences: 1. courses belonging to different areas need to have relationships with specific stakeholders 2. each degree course management group has his own inclination to evaluate who are “stakeholders” for its activities, depending on previous experiences, particular behaviour and so on. Considering students the most important stakeholder[8] for a degree course it’s almost unavoidable: in fact, for the traditional perception of their role and their relationship with the course activities, they are judged very near to the figure of “customers”. However it’s quite strange that for some courses business world or secondary schools or professors are not considered as stakeholders. 10

Following analysis could have the purpose to understand why so similar structures consider in a so different way if graduated students, institutions, student families etc. are stakeholders or not. To confirm this doubt, we find in external evaluation reports that about 69% of courses are requested to better individuate their stakeholders, including a deeper definition of the relationship management procedures. About precise requirements definition, we find that almost 13% of RAVs doesn’t include any specification, while other courses indicate an average of 11 requirements with a maximum of 39 (and other 10 courses more than 20). Even if an improvement is possible, we think that the situation is quite satisfying: the greatest part of courses have explicitly reported stakeholders requirements that represent the most important objective reference for following effectiveness evaluation. External auditors as well have a good impression about this element, as “only” 29% of courses need to better define these requirements. Graduate’s roles specification The following Figure 7 illustrate a very diversified situation
Graduated students roles
35 30 25 20 15 10 5 0

Figure 7 – Number of graduate’s roles individuated (the broken line divided scientific courses – on the4 left- and humanities)

The roles number varies from 0 to 32 with an average of 6,2. Almost 20% of courses indicated less than 2 roles. Moreover, only 34,6% of them report a description (sometimes vague and not very clear) of the specified roles. External evaluation of this element is quite encouraging: only 10,4% of degree courses must define roles in a more specific way, but it should be interesting to have the results of all the involved courses. However we think that the course cited above must reconsider the indicated roles together with the correspondent definitions. Likewise, even courses with a too long list of roles, have to evaluate the real coherence of them. Further analysis, could investigate as 11

well about possible overlapping between different courses indicating the same roles for their graduated students. Quality policy and correlated objectives definition All the involved degree courses define their own policy, but only 13% include specific references to quality implementation: this isn’t a very good starting point for an organization that have decided to adopt such an evaluation model. Moreover no more than 32% of them define quantitative (and then measurable) objectives, with an average of 3,8 objectives for each course. Consequently, every further efficiency and effectiveness estimation become difficult and probably inaccurate. Considering NC indicated by external evaluation we note that about 44% of courses have to better define policy and 77% of them need more coherence between policy and defined goals. Definitely, about these elements there’s a great necessity of improvement. A lot of degree courses have engaged quality implementation process but we must understand if that is an aware choice or just the attempt to satisfy an external requirement: in this case policy definition become unavoidably less clear and interesting and not correlated with measurable goals. However, the previous data probably point out that no less than a quarter of the involved courses have a significant awareness about quality implementation.

Conclusions
In order to better understand the first results and possible future development of CRUI model implementation and correlated internal and external evaluation activities, we have tried to analyse 4 specific elements of the RAV (processes description, stakeholders individuation together with their requirements, graduate’s roles specification, quality policy and correlated objectives definition) to point out how the degree courses (78 on a total of 88) have faced them and how they were considered by external auditors (where the external evaluation report are already available – 48 on 78). Generally, obtained data reveal a very differentiated situation between various courses and several elements have to be underlined: 1. Many courses describe in different ways the processes regulating their activities and often those descriptions are incomplete and unable to satisfy external auditors. Assuming that the organisational structure should be quite similar, the involved courses (at least those 12

belonging to same faculty) could make a common effort to build up a common framework to describe the principal processes. 2. In a lot of cases, stakeholders definition must be reviewed. We suppose that probably the RAV authors have indicated only “traditional” stakeholders, that is those directly involved by course activities, while other subjects, with more unclear but also important relationships, have been neglected. 3. The graduated students roles specification must become a fundamental element for the degree course organization (these should be obvious, but as a matter of it is not): the defined policy and correlated goals, didactic activities planning and so on, must be coherent with them. In this analysis we find that some courses have to better individuate the specific roles. 4. A true quality policy definition, also correlated with measurable goals, is present only in a little number of evaluated courses. Quality culture need to have a larger diffusion among them: the CRUI model implementation must become a management choice, not an annoying imposition.

References
[1] S. Lagrosen, R. Seyyed-Hashemi, M. Leitner - Examination of the dimensions of quality in higher education - Quality Assurance in Education, Volume 12 Number 2, 2004, pp. 61-69 [2] J. J. Tarı - An EFQM model self-assessment exercise at a Spanish university - Journal of Educational Administration, Vol. 44 No. 2, 2006, pp. 170-188 [3] D. Bassi, R. Buccianti, R. Mirandola, L. Modica, M. F. Romano, E. Stefani - An Italian quality approach in universities: Campus model and three years experience - Campus co-ordination and control unit CRUI- Roma [4] AAVV - Guida alla valutazione dei corsi di studio – Fondazione CRUI per le Università Italiane, 2003 [5] S. Sahney, D. K. Banwet, S. Karunes – Conceptualizing total quality management in higher education – TQM Magazine, Volume 16 number 2, 2004, pp 145-159 [6] AAVV - Developing an internal quality culture in European universities: Report on the Quality Culture Project 2002 – 2003 - European University Association, 2004 [7] A. E. Osseo-Asare, D. Longbottom, W. D. Murphy - Leadership best practices for sustaining quality in UK higher education from the perspective of the EFQM Excellence Model - Quality Assurance in Education, Vol. 13 No. 2, 2005, pp. 148-170

13

[8] P.B. Sakthivel, G. Rajendran - TQM implementation and students’ satisfaction of academic performance - The TQM Magazine, Vol. 17 No. 6, 2005, pp. 573-589

Web sites [9] CRUI organisation: http://www.crui.it

14