Best practices in continuous evaluation Evidence-Based Decision ...

Best practices in continuous evaluation: Evidence-Based Decision Making and Admissions Models Admissions criteria and success criteria ETS Europe January 2006 Jenny Dalalakis Gerben van Lent jdalalakis@etseurope.org gvanlent@etseurope.org Key Messages • Admissions Criteria are the outcome of a decision process which must be regularly evaluated • The process has the following components: – – – – Strategic objectives of the institution Instruments to identify student quality Student application model Admissions criteria Objective of this presentation: Introduce Best Practices in Admission Evaluations How to Admit? What is Student Quality? Why Admit? How to Decide? Admissions: Best Practices C. How to Admit? Admissions Criteria Why Admit? Consensus on Strategic Objectives A. What is Student Quality? Qualities sought in student body B. How to Decide? Evaluation Models, Fairness, Validity Outline Determining Institute’s Strategic Objectives for Admissions Deciding on Qualities of Student Body Evaluating Application Models Deciding on Key Admissions Criteria Wrap up: Conclusions and Questions Strategic Objective: Successful fit between Policy and Admissions Process College Board research identified best practices as: • • • • Consensus on admissions objectives – – – – – – – – – – Explicitly affirmed and revised periodically Quantitative and qualitative criteria of incoming student body Board, staff, alumni, current student body, labour market representatives Academic and non-academic qualities of admired students Acceptable instruments to measure these desired qualities Institutional objectives supported by individual and group qualities of student body Reflected in application components (explicit) and underlying factors (role of informal priorities) Institutional goals supported in transparent manner Institution’s Policy affects weight given to student quality measurements Profile of student balanced among different qualities sought relevance of information, equal access, transparency of admission process Periodic correlation measurement change is inevitable Consensus on the qualities sought in student body Admissions criteria Evaluations and processing models • • • Fairness – – – Validity of admissions process Future considerations Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003) Admissions: Best Practices C. How to Admit? Admissions Criteria Why Admit? Consensus on Strategic Objectives A. What is Student Quality? Qualities sought in student body B. How to Decide? Evaluation Models, Fairness, Validity Strategic Objectives: Why Admit? ACA Research identified 6 key objectives – – – – – – Policy Revenue Profile Sustainability Non-discrimination Demand (Hybrid needs typically exist) Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004) Why Admit? Key objective: Development Policy Institution Policy or Government Policy requires foreign students – France: Prestige in educating other countries’ elite students and future leaders – Netherlands: Manner of providing aid to developing countries – UK, US: Foreign students become host country’s goodwill ambassadors when they return home – EU: Support for mobility (Bologna, Lisbon) means international student exchanges are vital at Higher Education level Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004) Why Admit? Key objective: Revenue Market-driven need for funds related to foreign students • Public and private schools may charge more for all or some international students directly so the extra income is welcome – UK, US – CH (MBA programme students specifically) – NL (English-taught programme students specifically) Public university with foreign students may qualify for extra government funds – Australia Programme survival in danger and loss of staff so foreign students help numbers – SE, CH, NL Disincentives may apply if institution does not qualify for extra funds if they have foreign students – CH Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004) • • • Why Admit? Key objective: Institutional Profile International profile is of intrinsic value for Institution • • • • Foreign students enhance innovation and modernization Global pool better for choosing world-scholar quality – Minimum or maximum target % may be set: AU Global recognition of host university for its student quality is very competitive – Competition at level of Institution/ Discipline or at level of Labour Market Demands • Nature of programme requires international approach – Research literature is in English (MBA, Engineering): FR – Applied Languages, International Relations, International Law Foreign students bring the world home to remote areas – AU, SE Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004) Why Admit? Key objective: Sustainability Securing the future of the programme, staff, research • Smaller countries do not have enough local students for keeping some disciplines alive at their universities – NL, SE, CH need Science students • Students coming for Masters programmes are encouraged to stay on for PhD programmes to secure enough teaching staff assistants and research assistants Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004) Why Admit? Key objective: Non-discrimination No discrimination is tolerated between any student groups • Quality of students comes first regardless of student’s origin • Important objective for highly selective / private institutions: US • Some are more equal than others (informally or formally) – CH: evidence required that foreign student could have studied back home – FR: to protect local student quota, local students chosen with earlier deadline for public university places – SE: first choose local students and unfilled quota goes to foreign students Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004) Why Admit? Key objective: Students’ Demand Large number of foreign students want to come to the Institution • Local, regional and global mobility (economic, academic, professional) is a growing demand by students, governments, labour market so students expect international programmes in countries of interest to them • Universities receive large number of foreign student applications and react by starting international programmes and (hopefully) support programmes Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004) Admissions: Best Practices C. How to Admit? Admissions Criteria Why Admit? Consensus on Strategic Objectives A. What is Student Quality? Qualities sought in student body B. How to Decide? Evaluation Models, Fairness, Validity How to Admit? Look for Quality What is Student Quality? ACA research showed a variety of instruments for gathering evidence on student quality – – – – – – – – – – – Secondary school/ Bachelor diploma Transcript of courses List of completed courses Motivation statement/personal statement Letter of recommendation Proof of work experience Proof of previous academic work Curriculum Vita Standardized tests for subject and skill competencies Proof of knowledge of the teaching language Right to access Higher Education in home country Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004) How to Admit? Look for Quality What is Student Quality? College Board research on factors used in making admissions decisions – Academic Achievement, Quality, and Potential • Direct measures (coursework, grades, test scores) • Caliber of high school/ previous study institution (rank of class, school) • Evaluative measures (depth and breadth of interests, world awareness, communication skills) – Non-academic Characteristics and Attributes • Geographic (remote or disadvantaged region, local resident) • Personal Background (minority, socio-economic status, age) • Extra-curricular activities, service, & leadership (awards, social service, work experience) • Personal attributes (talent, character, commitment, social concern, creativity, determination) • Extenuating circumstances (family problems, health issues, language difficulties) This may differ somewhat from what institutions tell students what they are looking for Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003) Where do Student Quality Instruments fit? University policy for educating students S t u d e n t b a c k g r o u n d known Low score on TOEFL ITP or TOEIC inclusive exclusive High score on GMAT High score on TOEFL unknown Where do Student Quality Instruments fit? University policy for educating students known S t u d e n t b inclusive a c k g r o u n d •Motivation statement/personal statement •List of completed courses •Letter of recommendation •Proof of work experience exclusive •Secondary school/bachelor diploma •Right to access HE in home country •Proof of knowledge of the teaching language •Standardized tests •Proof of previous academic work unknown How to decide depends on which type of error you tolerate more capable refused admitted not capable Consider exceptional cases: (a) Lottery systems (Numerus Clausus) (b) Admissions and immigration (NL, DE) S t u d e n t b a c k g r o u n d University policy for educating students known inclusive exclusive unknown ‘ Capable’ = Successful How do we recognize success? Popular indicators of evidence: I. Student quality II. Attrition III. Time to degree • Faculty ratings • Grade point average • Results of Exams Which Student Quality? Competencies important for Student Quality = graduate school success Some qualities are also important to the labour market Student Quality? Student Quality = Competencies important for graduate school success Three research studies • Two studies asking faculty for competencies of successful students • One scheme for classifying outcome or criterion variables: “taxonomy of higher order performance components.” Types of Competencies linked to Student Quality I. II. III. IV. V. VI. VII. VIII. • • Job-specific, Study-specific task proficiency Non-job, Non-study-specific task proficiency Written & Oral Communication task proficiency Demonstration of effort Maintenance of personal Discipline Facilitation of peer and team performance Supervision/ Leadership Management/ Administration Some are present when student is admitted Some are part of the curriculum that the institution provides (educational value added) and so are expected at graduation Source: Standardized Letter of Recommendation, Educational Testing Service Competencies I. Job/study Specific Task Proficiency 1. Research/scholarly experience (amount and quality) 5. Mastery of discipline 11. Ability to read and analyze research in the field 13. Breadth of perspective C. Explanation (Enright & Gitomer) G. Synthesis (Enright & Gitomer) Competencies II. Non-Job/study specific Task Proficiency 7. Ability to teach (cross-listed in VII and VIII) 8. Independence (cross-listed in IV) 9. Creativity; ability to think “out of the box” 14. Open mindedness 15. Opportunism/Resourcefulness 16. Critical thinking ability, logic, problem solving ability 18. Computer literacy/Understand & Manage Technology 19. Skill in investigation 20. Ability to find the facts: information gathering B. Creativity (Enright & Gitomer) E. Planning (Enright & Gitomer) Competencies III. Written & Oral Communication Task Proficiency 4. Communication (professional) 6. English-as-a-Second-Language ability A. Communication (Enright & Gitomer) IV. Demonstration of Effort 2. Persistence/Tenacity* 8. Independence* 10. Enthusiasm* 17. Drive/Commitment/Motivation/Zeal* D. Motivation (Enright & Gitomer) Competencies V. Maintenance of Personal Discipline 12. Values/Character: integrity, fairness/openness/honesty, trustworthiness/ consistency – personal and professional* 21. Professional posture VI. Facilitation of Peer and Team Performance 3. Collegiality/Making professional connections/Networking (Crosslisted with VII) F. Professionalism VII. Supervision/Leadership 3. Collegiality/Making professional connections/Networking (Crosslisted with VI.) 7. Ability to teach (Cross-listed with II. and VIII.) VIII. Management/Administration 7. Ability to teach (Cross-listed with II. and VII.) Cross reference to Tuning Project How are competences and learning outcomes related? • Learning outcomes according to Tuning methodology should be formulated in terms of competences. • Learning outcomes are requirements of a unit or a programme and are expressed in terms what the learner knows and is able to do at the end of the learning experience. • Competences may be developed to a greater degree than the level required by the learning outcome. Graduates  Capacity for analysis and synthesis  Capacity to learn  Capacity for applying knowledge in practice  Elementary computing skills  Capacity to adapt to new situations Employers  Capacity to learn  Capacity for applying knowledge in practice  Capacity for analysis and synthesis  Capacity to adapt to new situations  Interpersonal skills Academics  Basic knowledge of the field of study  Capacity for analysis and synthesis  Capacity to learn  Capacity for generating new ideas (creativity)  Capacity for applying knowledge in practice Staff-centred vs. Student-centred? On completion of a first cycle degree in Mathematics, students should be able to • • • • • Show knowledge and understanding of basic concepts, principles, theories and results of Mathematics Understand and explain the meaning of complex statements using mathematical notation and language Demonstrate skill in mathematical reasoning, manipulation and calculation Construct rigorous proofs Demonstrate proficiency in different methods of mathematical proof. Admissions: Best Practices C. How to Admit? Admissions Criteria Why Admit? Consensus on Strategic Objectives A. What is Student Quality? Qualities sought in student body B. How to Decide? Evaluation Models, Fairness, Validity How to recognize competencies? The Admission Challenge What are the competencies you attribute to successful students? Which of them have to be present at the start? Is the aim Inclusive or Exclusive selection? The Admission Challenge What evidence do you consider: Is it fair, valid, reliable? Have you considered all costs? What is the legal setting? Evaluating an Application: Different models of meeting the Admissions challenge Admissions offices may be responsible for  Combining all individually-rated components and weighing – – – – Application information Transcripts Essays Recommendations  Combining academic and non-academic factors and weighing – Some factors considered separately e.g., communication – Balance of academic and non-academic – Many wide-scope factors vs. few key factors Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003) Evaluating an Application: Different model examples Example 1 – Academic – Personal characteristics – Suitability for desired field of study Example 2 – Academic – Communication – Character, leadership, initiative Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003) Evaluating an Application: Different model examples Example 3 – Exceptional academic Achievement – Academic Promise – Potential to Contribute Example 4 – – – – Academic Achievement Academic Qualities Non-academic Achievement Personal Qualities Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003) Evaluating an Application: Different model examples Example 5 – – – – – Quality of Courses Grades in Core Curriculum Test results Activities (extra-curricular or curricular) Essay Example 6 – – – – – – Academic Performance Extra-curricular activities Teacher / counselor recommendations Interview Personal Inventory Essays Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003) Evaluating an Application: Different model examples Example 8 Example 7  Academic Achievement – Application and Essay  Intellectual Curiosity – Academic Performance – Level of Challenge of Academic Record  Potential  Commitment – Recommendation and Interview  Communication – Personal Qualities  Engagement with others – Special Talents  Extra-curricular activities  Initiative Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003) Making Decisions: Making sense of what information comes in from Applications • Single readers, multiple readers, group readings, automated readings – Process x for collecting academic info (formulaic rating) into an index – Process y for collecting non-academic info (holistic rating) into an index • • • Clear minimum thresholds of entry per index or factor Clear criteria for strong, and weak candidates – Level descriptors for each level of qualification Clear guidelines for multiple entryways – Special consideration for minorities, shortage of students – Lottery for some competitive programmes with limited places – Work experience • Final review of denials – Conditional acceptance – Additional information or interview requested Quality Assurance for Application Models • Setting key current criteria into application form – Direct and indirect questions • Setting dates for re-evaluation of application form criteria – Selection of staff who represent key constituents affected by decision Adapting admissions criteria to institutional profile over time – Respecting changes in strategic objectives Adapting to external changes in education, demographics, labour market – Understanding where the student body comes from and goes to • • Best Practices in Admissions How to Select Admissions Criteria What to look for Qualities sought in student body Consensus on Admissions objectives How to Decide  Application & Processing models  Fairness  Validity Key Messages • Admissions Criteria are the outcome of a decision process which must be regularly evaluated • The process has the following components: – – – – Strategic objectives of the institution Instruments to identify student quality Student application model Admissions criteria Discussion and insights 1. Why do you admit students?  What are the 3 main Strategic Objectives of your institution?  What are your main 3 Admission objectives?  What are your 3 most common profiles of admitted students? 2. What is student quality for you?  What are the 3 main student-quality instruments that you use? Consider how your policy objectives affect the level of detail that you require from your instruments.  What are the most common instruments of your peers?  Where do your admissions instruments fall on this chart? Where do your Student Quality Instruments fit? S t u d e n t b a c k g r o u n d University policy for educating students known inclusive exclusive unknown Discussion and insights 3. What is the quality of your added value to the student’s education?  Which 2 key competencies does your institution expect to be present at admission?  Which 2 key competencies does your institution expect at the learning process? 4. How do you recognize quality?  What evidence does your institution require? Evidence from the student directly or other institutions Student portfolio High-school/ previous programme transcripts Letters of recommendation Personal answers to questions on the application Candidate’s scores on standardized admissions tests Language standardized tests   Which formal instruments do you use? How do you ensure fairness, validity, and reliability? Discussion and insights 5. What type of student application do you use?   What are the top 3 factors in your qualification evaluation model? What quality assurance principles to you use to keep your application model valid?  How frequently do you hold re-evaluation?  How do you ensure validity of criteria/ benchmarking success factors What is your process of updating criteria? 