Docstoc

QUALITY ASSURANCE

Document Sample
QUALITY ASSURANCE Powered By Docstoc
					QUALITY ASSURANCE AT
ALGONQUIN COLLEGE OF APPLIED ARTS AND TECHNOLOGY
PROGRAM QUALITY REVIEW 2008 - 2009 GUIDE FOR TEAM LEADERS FOR COLLEGE APPROVED PROGRAMS

TABLE OF CONTENTS
Getting Help ..................................................................................................... iii Introduction ......................................................................................................... 1 Welcome to Program Quality Review ................................................................ 2 2008 – 2009 Timeline ........................................................................................ 2 History of Program Review at Algonquin College .............................................. 4 Cyclical Program Quality Review Diagram ........................................................ 5 Team Leader Orientation .................................................................................... 6 Orientation Sessions for Team Leaders ............................................................ 7 Release Time .................................................................................................... 7 Team Leader Orientation Sessions Syllabus .................................................... 8 PQR – What is the Process .................................... Error! Bookmark not defined. Getting Started ................................................................................................ 10 Understanding Evaluation Criteria ................................................................... 11 Evaluating the Program Against the Criteria ................................................... 15 Reviewing Documents ..................................................................................... 27 Reviewing Survey Data ................................................................................... 30 Curriculum Mapping ......................................................................................... 35 Program Standards ......................................................................................... 36 Purposes of Curriculum Mapping .................................................................... 37 Terms and Concepts ....................................................................................... 38 Course Outline Review ...................................... Error! Bookmark not defined. Facilitating Focus Groups ................................................................................ 44 Types of Focus Groups ................................................................................... 45 Tips for Facilitating Focus Groups ................................................................... 46 Faculty Focus Groups ...................................................................................... 48 Purpose of Program Audits ............................................................................. 49 How Program Audits Support Program Quality Review .................................. 49 Planning Faculty Focus Groups ...................................................................... 51 Preparing Faculty Focus Group Reports ......................................................... 52 External Stakeholder Focus Groups ............................................................... 62 Planning External Stakeholder Focus Groups ................................................. 63 Possible Items for Discussion.......................................................................... 65 Student Focus Groups ..................................................................................... 74

PQR 2008 – 2009: Guide For Team Leaders

i

Planning Student Focus Groups...................................................................... 75 Possible Items for Discussion.......................................................................... 77 Appendix A ........................................................................................................ 87 Final Report Template ..................................................................................... 88 Appendix B ...................................................................................................... 102 Roles of Program Quality Review Participants .............................................. 103 Annual Program Audit Diagram ..................................................................... 105 Program Mix Review Diagram ....................................................................... 106

PQR 2008 – 2009: Guide For Team Leaders

ii

GETTING HELP
You will be guided through the Program Quality Review process by Susan Lomas, the College’s Program Quality Assurance Administrator.

Susan Lomas’s Contact Information

C532
(in the Office of the Vice President Academic)

Ext: 7681 Email: lomass@algonquincollege.com

Help is only a phone call away

All of your questions about the Program Quality Review process can be directed to the Program Quality Assurance Administrator, who will do her best to respond and help you.

PQR 2008 – 2009: Guide For Team Leaders

iii

INTRODUCTION

1

In this section
   Welcome to program quality review 2008 – 2009 timeline History of program review at Algonquin College

PQR 2008 – 2009: Guide For Team Leaders SPTS

1

WELCOME TO PROGRAM QUALITY REVIEW
Thank you for taking on this new role as a Program Quality Review team leader for the 2008 - 2009 academic year. The College appreciates your contribution to the quality assurance process and we hope that this experience will add considerably to your knowledge and understanding of the program in which you teach and/or coordinate. You will become more aware of the many factors which affect your program and the quality of educational services the College is able to provide to students. This guide has been developed to help you through the Program Quality Review process.

2008 – 2009 TIMELINE
May to June
  Confirm programs to be included in the pilot. Identify team leaders.

September
   Introduction for team leaders and chairs Distribution of Team Leader Guides Review of curriculum and mapping of program outcomes against provincial vocational outcomes if available. If no program outcomes exist for the program there will need to be a recommendation to create them.

October
   Team leaders and chairs meet to; analyze statistics and identify any issues to be added to focus group agendas; set dates for the three focus groups; and to identify agenda items for each of the focus group sessions. Send out invitations to members of focus groups. Completion of all mapping and confirmation of compliance with Ministry/College standard, or identification of any gaps.

November
    Confirmation of issues to be discussed at focus group sessions. Agendas set Focus groups for faculty, students, and external stakeholders. Review of input from the focus groups and recommendations discussed by team leader and chair (and possibly faculty). Identification of budget implications for recommendations.

PQR 2008 – 2009: Guide For Team Leaders SPTS

2

December/January
     Review of costing data by Chairs and Team leaders to review conclusions and develop reasonable recommendations for improvement. Implementation plan developed. Report written by team leaders and acknowledged by chair. Report submitted to the PQA Administrator. Debrief.

PQR 2008 – 2009: Guide For Team Leaders SPTS

3

HISTORY OF PROGRAM REVIEW AT ALGONQUIN COLLEGE
Program review has been an annual event at Algonquin College, probably starting as soon as the College opened. Each year the program faculty members revise and update the curriculum to maintain its currency and ensure that the graduates have the skills needed to be successful in the workplace. Formal, in depth review processes have been implemented at various times in the College’s history. At this time, program review at Algonquin College consists of three processes: 1) the Annual Curriculum Review process, 2) the Program Mix Review, and 3) the Program Quality Review. The Annual Curriculum Review process is based on feedback from students and advisory committee members when considering revisions needed each year to keep the program current. Revisions to the curriculum and the program narrative information are made in GeneSIS, the student information system. The curriculum is reviewed in the Office of the Vice President Academic, and once it is approved, the new program version is created. This information in GeneSIS then becomes the foundation for course loading, fees loading, scheduling, registration and our College publications. This process is followed by the School of Part-time Studies for its programs as well. The second annual review is called the Program Mix Review. Staff in the Academic Operations and Planning department compile data that reflect the Quality Index Measures for a program. Capstone questions from the KPI and Course Assessment surveys, registration and Program Costing data are reviewed. Programs not meeting the college benchmarks are reviewed and strategies are developed with the Chair, to improve the quality of the program and/or the financial contribution. This information is not currently readily available for College Approved programs. As of 2004, the College has again implemented a more in depth review called the Program Quality Review. This process builds on the annual review activities, mirroring the annual program audit and providing five years of data used in the program mix review process. This allows the faculty/administrative team to do some trend analysis and plan for the next five years. Each of the College’s programs will be reviewed every five years. This process was piloted in 2007 for programs in the School of Part-time Studies. This manual serves as a guide to assist with this process.

PQR 2008 – 2009: Guide For Team Leaders SPTS

4

CYCLICAL PROGRAM QUALITY REVIEW MAP

Identify Programs Orientation to Process and Templates Central Data Provided

Assign Team Leader

Team Leader and Chair Review survey results + conduct SWOT analysis

Faculty Review of Curriculum Data External Stakeholder / Advisory Committee Focus Group Student Focus Group

Review of Findings + Formulation of Recommendations

Team Leader completes Final Report

Implementation of Recommendations

Annual follow up of implementation plan

PQR 2008 – 2009: Guide For Team Leaders SPTS

5

TEAM LEADER ORIENTATION

2

In this section
   Orientation sessions for team leaders Release time Team leader orientation sessions syllabus

PQR 2008 – 2009: Guide For Team Leaders SPTS

6

ORIENTATION SESSIONS FOR TEAM LEADERS
Meetings for team leaders have been scheduled based on consultation with the dean and the academic managers. There are six formal training sessions identified. Each of these will last about 3 hours per week. The orientation sessions will not start until the second week of the term. The orientation process for team leaders involves a series of six workshops, each three hours in length. Upon completing the team leader orientation process you will be able to 1 2 3 4 5 6 7 Evaluate a program against identified criteria. Lead focus groups for internal and external stakeholders. Map curriculum to validate that the program of student meets current Ministry requirements. Analyze statistical data and make inferences based on trends. Identify quality in a program when measured against identified standards and determine how to reinforce/maintain this quality. Formulate realistic and attainable recommendations to improve quality where needed. Analyze curriculum maps to identify gaps and redundancies.

RELEASE TIME
Each academic chair will be reimbursed for the hiring of a faculty consultant to assist with the curriculum review process for the equivalent of 6 hours per week for 15 weeks, secretarial support for 3 hours per week, incremental increases in hospitality monies for focus groups, and any travel directly related to the Program Quality Review Process. All expenses except travel are charged to the program cost centre. In January, the PQAA will transfer the expenses out of the program cost centres. Travel claims require advanced authorization by the PQAA and expenses are submitted directly to her on an Employee Travel Expense form (available on my Algonquin under “Forms”). Printing of the final report for five copies will be the responsibility of the Program Quality Assurance Administrator. Additional copies can be made by the individual academic managers/chairs.

PQR 2008 – 2009: Guide For Team Leaders SPTS

7

TEAM LEADER ORIENTATION SESSIONS SYLLABUS
Schedule of Orientation Workshop for 2008 - 2009 Program Quality Review Process College Approved Programs Team Leader Orientation Sessions Session 1 Meeting Agenda     

Introduction to Curriculum Introduction to Program Review Evaluation Criteria Ministry Program Standards and Curriculum Mapping Development of Program Outcomes – setting up sessions with LTS Program Review – where to start Review curriculum mapping Course outline reviews Planning the Focus Group Sessions

Session 2

               

Session 3

Leading the Focus Group Sessions Reviewing the Survey Data Setting the agenda

Session 4

Preparing the Report Formulation of Recommendations Developing an Implementation Plan The Final Report – what should it look like, what to include, how to submit The approval process for the implementation plan Proofing and sending the report to printing. Following up with Focus Groups

Session 6

TBC

Final Report due

TBC

Debrief

PQR 2008 – 2009: Guide For Team Leaders SPTS

8

PQR – WHAT IS THE PROCESS?

3

In this section
     Getting started Understanding evaluation criteria Evaluating the program against the criteria Reviewing documents Reviewing survey data

PQR 2008 – 2009: Guide For Team Leaders SPTS

9

GETTING STARTED
The report template contains the outline of the information to be included in the report. It will not necessarily get written in the order that it is displayed. The place to start is in a section called “Findings”. As you evaluate your program against the criteria, you document your findings in this report. The document is set up in a table format so that you can add or delete information for each of the criteria, without having to worry about formatting. The evaluation criteria are identified starting on the next page. Most of these criteria are designated by the Ministry as criteria that define a quality program. Some of the criteria are not relevant for College approved programs however, given most College approved programs consist of Ministry funded courses, the criteria may be relevant in part. As well, some of our certificate programs flow into programs which result in the student receiving Ontario College credential so it is beneficial to understand what these criteria are. Following the criteria, you will find some direction about whether or not the criterion is relevant and how to evaluate your program against each of the criteria. The College already surveys students, graduates and employers and this data is available to you in the reports provided to you, however this data is limited for College Approved programs. The guide tells you to which questions to refer for each of the evaluation criteria. As you review the data provided, it will be clear which areas need further exploration. If your data provide adequate information that the criterion is met, you need only note that in the Findings document. In areas where data is not as available or the data raised some questions, you may decide to investigate further at your focus groups. Each of the focus group sections includes the criteria that are relevant to that group. As you are preparing for the sessions, you will be reminded of potential areas to be considered, keeping in mind that not all criteria need to be discussed at the focus group sessions if you are convinced you have enough data to demonstrate quality. As you review the survey data given to you and record the findings in your report, you will identify issues you need to discuss. This will help you set up the agendas for your focus group sessions. After all the focus groups and faculty consultation sessions are complete, you will then be able to review the focus group reports to identify and formulate your recommendations for your final report.

PQR 2008 – 2009: Guide For Team Leaders SPTS

10

EVALUATION CRITERIA
Criterion 1. Admission, credit for prior learning, promotion, graduation, and other related academic policies support program development and student achievement of program learning outcomes. 1. The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program. 2. Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them. 3. Students know how to get internal and external credits and recognition for prior learning. 4. Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program. 5. Students know how they will be evaluated. 6. Students indicate the learning requirements are relevant and meaningful. 7. Students indicate that assessment methods relate to the learning requirements.

Criterion 2. Programs conform to the Framework for Programs of Instruction and the Credentials Framework, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance. 1. The duration and structure of the program are consistent with the program learning outcomes and the credential offered. 2. Appropriate credits are allocated for each component of the program, and transfer and laddering options are stated. 3. Prerequisites do not unnecessarily hinder progress in the program. 4. Program learning outcomes are consistent with the credential granted, the title of the credential awarded, the provincial program standards (where these apply), and the minimum essential expectations of the workplace. 5. Program learning outcomes are reflected in course outlines.

PQR 2008 – 2009: Guide For Team Leaders SPTS

11

6. Programs learning outcomes are used in prior learning assessment. 7. Changes to courses and program outcomes are introduced on a timely basis and are designed to maintain the relevance of the program. 8. The program has established articulation agreements. 9. The program conforms to the College policy for the number of English courses. 10. All curriculum documentation is up-to-date including course outlines and the program monograph information. 11. There is congruency between the course learning requirements and the program learning outcomes. 12. There is a match between course learning requirements, course learning activities and learning resources. 13. Concepts of environmental sustainability are imbedded in the program curriculum. 14. Students have opportunities to develop the skills, knowledge and attitudes necessary to succeed in a global economy.

Criterion 3. Methods of program delivery and student evaluation are consistent with the program learning outcomes. 1. Program delivery, including that which takes place off-site, is consistent with the nature of the program, the learning outcomes, and the needs of the students. 2. There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities. 3. Learning methods are published and are matched to the learning outcomes. 4. College designated targets regarding hybrid courses are met. 5. Learners are provided the skills necessary to be successful with the learning strategies selected. 6. Evaluation criteria are published and students are aware of how and when they are going to be evaluated. 7. There is a match between course learning requirements and evaluation methods, i.e., evaluation methods allow students to demonstrate the course learning requirements 8. Evaluation methods are valid and reliable.

PQR 2008 – 2009: Guide For Team Leaders SPTS

12

9. Students indicate that feedback is timely and allows them to build on their learning.

10. Students perceive evaluation to be fair. 11. Resubmissions, supplementals, and appeals are published, appropriate, fair, valid, and consistent. 12. Student workload and assessment is balanced across the term at both the course and program level. 13. There is a range of evaluation methods used consistent with a variety of learning styles. 14. Learners can earn credit for up to 75% of the program’s courses, using the PLA process. 15. Academic policies and practices that provide for the development and continuous improvement of teaching and learning methods are valued, documented, and supported. 16. Graduate capabilities, including knowledge, skills, and attitudes are consistent with program outcomes.

Criterion 4. Human, physical, financial, and support resources to support student achievement of program learning outcomes are available and accessible. 1. The program faculty members, as a whole have adequate academic preparation and workplace experience to deliver a quality program. 2. Faculty members are evaluated every three years. 3. Faculty members engage in professional development activities that ensure they are current in their field and developing teaching expertise. 4. Students consider faculty to be available. 5. Students consider faculty to be adequately prepared for class. 6. All students are assigned an academic advisor. 7. Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is.

PQR 2008 – 2009: Guide For Team Leaders SPTS

13

8. Labs, clinical facilities and placement facilities are complementary to and integrated into the program and allow the learner to demonstrate the learning outcomes. 9. Students indicate that there are adequate and accessible learning resource materials including: textbooks in the bookstore; online materials; print resources; equipment and student support services, to allow them to be successful. 10. The program is financially viable. The demand for the program has been sustained for the last five years. There is a future demand for graduates of this program. 11. The learning environment is safe.

12. The students are provided with the information they need to know to function safely in both the College and workplace learning environments.

Criterion 5. Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.

1. Students indicate that they are satisfied with the program. 2. Issues raised at Program Councils are addressed in a timely fashion and feedback is provided to the Council. 3. Learners progress through the program, achieve program outcomes and graduate in a timely fashion. 4. Learners with a wide range of abilities demonstrate the expected learning outcomes. 5. Graduates are satisfied with the overall program experience. 6. Graduates are obtaining employment in their fields. 7. Graduates are successful in obtaining external licenses or credentials where relevant. 8. Employers are satisfied with graduate performance.

PQR 2008 – 2009: Guide For Team Leaders SPTS

14

EVALUATING THE PROGRAM AGAINST THE CRITERIA
The following guide will help you use the data provided to you to evaluate your program against the criteria. In some situations there will be adequate survey data and other reports to allow you to conclude that the criterion is met. In other areas you may have questions after reviewing the data. These issues then need to be discussed with any or all of, the chair, the faculty, the students or other external stakeholders. Guidance is given for each of the criteria to help you decide how to proceed. Criterion 1. Admission, credit for prior learning, promotion, graduation, and other related academic policies support program development and student achievement of program learning outcomes.

1. The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program. Students registering in continuing education courses are not usually required to apply to the program through OCAS. Exceptions to this policy are published in the program monograph and on Course, Algonquin College’s guide for continuing education offerings. As well, when students are required to proceed through a program in a specified sequence, this is published both in on Course and in the program monograph. Individual course information in on Course includes any specific pre requisite skill or knowledge that the student needs to be successful in the course, for example, “must be able to type at least 50 words per minute. Given the mature status of most continuing education students, the College allows students to choose to take a course and accept the accountability regarding pre requisites. Please identify what is being published and practiced for your program. 2. Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them. This will be a key question for the Student Focus Group. Also, question #5 on the Course Assessment Report can provide some insight into this element. It is essential that the information given to registrants, allows them to understand what skills they will learn, and how it will prepare them for the workplace, as well as for what jobs in the workplace. This information is published in the Program Narrative – Overview section of the Monograph. The members of the external stakeholder focus group, both employers and graduates of the program, will be able to provide feedback regarding the Program Monograph.

PQR 2008 – 2009: Guide For Team Leaders SPTS

15

3. Students know how to get internal and external credits and recognition for prior learning. This information is provided in on Course and the program monograph. A key item for discussion at your student focus group is whether or not the students know where to find this information and if they find this information useful. You may find that if students did not need to apply for exemptions, they may not have noted what the process was. It is important therefore to ask this question of those who did apply for credit to determine if they received the assistance they needed in a timely fashion. 4. Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program. Students need to know what the expectations and obligations are for them to advance in the program. This might include the necessity to submit health and immunization forms, proof of a driver’s licence, or a police records check. Identify any special requirements the program might have and how the students are informed. 5. Students know how they will be evaluated. Students need to understand what to expect regarding evaluation methods, how to receive special accommodation, how to appeal a grade, and how progression is determined. Describe how your department informs students regarding assessment and progression. Examples of unique situations where students have received special accommodation can be briefly discussed. This may include information included in letters to students/applicants, course descriptions, course outlines, etc. The course assessment report question #4 will indicate whether or not students feel the evaluation methods are clearly communicated. 6. Students indicate the learning requirements are relevant and meaningful. Student feedback is available in the Course Assessment Roll Up Report with questions #1f and #5. This may also be a question at the student focus group session. 7. Students indicate that assessment methods relate to the learning requirements. Student feedback is available in the Course Assessment Roll Up Report with question 2c. You may also ask this question at the Student Focus Group session.

Criterion 2. Programs conform to the Framework for Programs of Instruction and the Credentials Framework, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance. 1. The duration and structure of the program are consistent with the program learning outcomes and the credential offered.

PQR 2008 – 2009: Guide For Team Leaders SPTS

16

The Framework for Programs of Instruction document is included in Appendix B of this document. You can compare your program duration and outcomes to the framework to ensure your program is in the appropriate credential level. You will have received some guidance about how to do this during one of the orientation sessions. Given the quantity of information provided during these sessions, if you need a review or more information please contact the PQAA. College approved programs are not required to include General Education or Essential Employability Skills. If any of the courses in your program belong to a program which results in an Ontario College credential, you may see the link to EES outcomes. Templates are provided to map the curriculum to program outcomes. If program outcomes are not yet developed, this should be a recommendation and program outcomes will need to be developed. Assistance is available from the curriculum consultants in the Learning and Teaching Services department. 2. Appropriate credits are allocated for each component of the program, and transfer and laddering options are stated. It will be important for faculty to review that courses are organized and sequenced to support student learning, are allocated appropriate hours of study and are consistent with published requisites. There are questions on the Course Assessment Roll Up Report that may provide student input: #1e, and 2a.

3. Prerequisites do not unnecessarily hinder progress in the program. Prerequisites are not usually an issue for CE programs. If your program does have prerequisites discuss what they are and how the department enforces them. What happens if a student does not have the prerequisite? 4. Program learning outcomes are consistent with the credential granted, the title of the credential awarded, the provincial program standards (where these apply), and the minimum essential expectations of the workplace. Program outcomes need to be reviewed using the Credentials Framework. If program outcomes do not exist, they will need to be developed. This will be a recommendation for the program. You can enlist the help of the faculty in Learning and Teaching Services who have experience in curriculum development. You can arrange this by calling them directly or arranging through the academic manager or the PQAA. Consistency of the curriculum with the credential offered is especially relevant for graduate certificates. The depth and breadth of the curriculum needs to be reviewed. Curriculum maps can be produced once all course outlines have been reviewed. The maps need to be reviewed to ensure that the program meets the Program Vocational Standards, There is an orientation session and a section of this guide to help you develop and analyze curriculum maps. Again, should you require additional help, do not hesitate to contact the PQAA. Curriculum maps are included in the final report in Appendix A.

PQR 2008 – 2009: Guide For Team Leaders SPTS

17

In a situation where the curriculum does not meet the Ministry Standard in any area, the plan to ensure the curriculum will conform in the next academic year must be clearly identified in the recommendations and implementation plan . 5. Program learning outcomes are reflected in course outlines. Programs with course outlines loaded into COMMS (Course Outline Mapping and Management System) will have program outcomes reflected in course outlines. If your program course outlines are not yet loaded into COMMS identify whether or not the program learning outcomes are reflected in the course outlines and linked appropriately to the specific course learning requirements. The PQAA can again provide individual assistance if needed after the respective orientation session which includes this information. 6. Program learning outcomes are used in prior learning assessment and recognition (PLAR). PLAR options are recorded as part of the course outline review and the findings are documented on the Summary sheet. Note that you need to identify on each course outline whether or not the student can challenge the course using PLAR. As you review each course, it needs to be determined how the faculty can evaluate the outcomes. Usually a department has two weeks to set up an appropriate challenge. If is therefore necessary that the challenge be determined in advance. The two weeks lead time will then allow for the updating of the challenge. 7. Changes to courses and program outcomes are introduced on a timely basis and are designed to maintain the relevance of the program. In this section describe how your department determines what updates are necessary and how they are incorporated into the curriculum. For example, as a result of feedback from students, agency staff and advisory committee members, changes are made annually as part of the annual curriculum review process. You can refer to relevant meeting minutes to indicate how this information was gathered and used. 8. The program has established articulation agreements. Any existing articulation agreements should be reviewed to ensure that curriculum changes made over the past five years have been updated in the articulation agreement. Programs not currently partnering in an articulation agreement may wish to consider doing so, especially if there is a need to have a degree to progress in the field. College certificates that ladder into programs that offer an Ontario College credential need to be reviewed to ensure the laddering opportunities for students are maximized. Similarly, a process needs to be in place for tracking new agreements being piloted as part of the College School Work Initiative. Identify any projects occurring in partnership with your program and how they are being managed.

PQR 2008 – 2009: Guide For Team Leaders SPTS

18

9. The program conforms to the College policy for the number of English courses. The minimum English hours of instruction for two-year and three-year programs is 90 hours, normally delivered in two 45-hour courses, and in one-year programs it is 45 hours. This may not be an issue for a College certificate.

10. All curriculum documentation is up-to-date including course outlines and the program monograph information. Course outlines will be individually reviewed using the course outline review worksheet. The summary document will be used to identify any course outlines that need updating. The work needed to ensure course outlines are complete will be discussed with the Chair and allocated in the Implementation Plan. Program Monograph information is updated each year. If any areas have been designated as requiring modification as a result of this review, state what changes are being made. Otherwise indicate the changes that are being made as a part of the annual update process. 11. There is congruency between the course learning requirements and the program learning outcomes. The faculty who teach in the program will be able to provide valuable insight into the sequencing of the courses and the culmination of the various course learning requirements to the program learning outcomes. The curriculum maps will demonstrate the distribution of the program outcomes across all levels and indicate if any areas need consideration, for example for outcomes that have more than adequate coverage. The analysis of the curriculum maps is discussed in the Curriculum Mapping section of this manual. As well, question #1f in the Course Assessment Roll Up report may provide student feedback. 12. There is a match between course learning requirements, course learning activities and learning resources. This will be an important criterion for the faculty to review as part of the Course Outline Review process. As well, there are questions on the Course Assessment Roll Up Report that will provide student input: #2a, b, and c. 13. Concepts of environmental sustainability are imbedded in the program curriculum. Describe how your program has introduced or is planning to introduce concepts of environmental sustainability into the curriculum. If this is a new area for consideration in the program curriculum the faculty will need to discuss with the chair how to proceed. In this situation, the implementation plan might involve discussion at advisory committee meetings for the upcoming year with changes to the curriculum to be implemented in the following year and a plan for maintaining these concepts current.

PQR 2008 – 2009: Guide For Team Leaders SPTS

19

14. Students have opportunities to develop the skills, knowledge and attitudes necessary to succeed in a global economy. In this section you can describe how concepts of cultural diversity, team work, flexibility, transfer of skills, interpersonal skills, problem-solving and decision-making ability, adaptability, creative thinking, self-motivation, the capacity for reflection, etc. are incorporated in the program curriculum. Many of the skills are currently being imbedded through the incorporation of the Essential Employabilities Skills as mandated by the MTCU. In many cases there are no specific learning outcomes or course learning requirements that directly reflect this learning. As part of the course outline review courses can be identified as needed review and updating and the implementation plan would describe how this would happen.

Criterion 3. Methods of program delivery and student evaluation are consistent with the program learning outcomes. 1. Program delivery, including that which takes place off-site, is consistent with the nature of the program, the learning outcomes, and the needs of the students. This criterion will need to be discussed with students and faculty. With the broad mix of demographics in the classroom, curriculum design needs to be flexible. As well, students are not equally prepared to learn online, in groups, or using independent studies. It is important to note successes as well as challenges faced in the classroom. If the program is delivered on more than one campus, discuss how regional differences are accommodated, noting that the programs are reviewed in the same cycle, and that the curriculum is mapped to the same outcomes. Also outline any communication that exists between the faculty of the different campuses. If a program is delivered off-site, describe accommodations to ensure there is consistency in the ability of the students to demonstrate the program outcomes. This could include specifications as outlined in the Memorandum of Understanding and the process for ensuring that off campus graduates actually meet the learning outcomes. 2. There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities. The team leader and faculty will need to review the course outlines and the information in the course outline review form provided to evaluate this criterion for each course. 3. Learning methods are published and are matched to the learning outcomes. The Course Assessment Roll Up report has student input in question #3. The course outline review will also identify any areas where a match does not occur and changes are needed.

PQR 2008 – 2009: Guide For Team Leaders SPTS

20

4. College designated targets regarding hybrid courses are met. As of 2006-2007, the following targets are required: Level 1: Level 2: Level 3: Level 4: Levels 5 & 6: 2 hours of online activity through hybrid instructional mode 2 hours of online activity through hybrid instructional mode 3 hours of online activity through hybrid instructional mode 3 hours of online activity through hybrid instructional mode 3 hours of online activity through hybrid instructional mode

This criterion is not relevant for College approved programs. 5. Learners are provided the skills necessary to be successful with the learning strategies selected. Faculty will need to review course learning activities and ensure that relevant skills are introduced to ensure students have the necessary skills, for example for orientation to online learning, or group theory, if group projects are required. 6. Evaluation criteria are published and students are aware of how and when they are going to be evaluated. The Course Assessment Roll Up report has student input in question #4. The course outline review process will also identify any gaps. Penalties for late assignments need to be reviewed to ensure that penalties are consistent across the program or that any differences in specific courses or course sections, are justifiable, published and implemented equitably. 7. There is a match between course learning requirements and evaluation methods, i.e., evaluation methods allow students to demonstrate the course learning requirements This will be included in the course outline review process as discussed in the curriculum review section of this document. Student input for this criterion can be found in the Course Assessment Roll Up Report in question #2a. It is important to include examples in the report about the methods used and how they allow students to demonstrate the outcomes. Please highlight special accommodations made or innovative methods used by the program faculty. 8. Evaluation methods are valid and reliable. Describe the processes that the faculty use to ensure that evaluation methods are reliable and valid, for example, examining negatively discriminating questions on multiple choice examinations, prior to submitting final grades. 9. Students indicate that feedback is timely and allows them to build on their learning.

PQR 2008 – 2009: Guide For Team Leaders SPTS

21

The Course Assessment Roll Up Report data is limited but you could look at question 2a 10. Students perceive evaluation to be fair. The Course Assessment Roll Up report offers student feedback in question #1e. 11. Resubmissions, supplementals, and appeals are published, appropriate, fair, valid, and consistent. Describe your program policy and how it is published and implemented. In particular, department practices can be compared by the Chair and the faculty, to the Directive - E1 Evaluation of Student Learning and Directive - E29 Program Progression and Graduation Requirements. 12. Student workload and assessment is balanced across the term at both the course and program level. This is not usually an issue for CE courses. 13. There is a range of evaluation methods used consistent with a variety of learning styles. This criterion will need to be discussed with both faculty and students. Again, the review of the course outline review sheets will provide examples of the range of methods used. In the report, highlight methods used that demonstrate meeting the needs of various learning styles. As well, highlight any innovative evaluation strategies used. 14. Learners can earn credit for up to 75% of the program’s courses, using the PLAR process. Each course outline needs to be reviewed. PLAR challenges are identified on each course outline. This will be a simple calculation of the number of courses with documented PLAR challenges versus the total number of courses. Where less than the required number of courses have PLAR challenges available, recommendations and a plan will need to be developed to address this. PLAR challenges will need to be established using the existing College and Ministry policies and fees structures. Assistance is available from Learning and Teaching Services. It is important to note that on the course outline it is only necessary to identify that the course is challengeable and who to contact. The actual challenge format should be identified and planned although time will be needed to ensure the challenge tool is updated. Note: Once a challenge process is in place, the PLAR flag needs to be identified in the student information system. 15. Academic policies and practices that provide for the development and continuous improvement of teaching and learning methods are valued, documented, and supported.

PQR 2008 – 2009: Guide For Team Leaders SPTS

22

Describe how your faculty members have experimented with new teaching methods and how they evaluate the effect on student success. A survey of the teachers in your program will identify who has taken the TALL program and the PT teacher orientation sessions. 16. Graduate capabilities, including knowledge, skills, and attitudes are consistent with program outcomes. Given the students can usually take courses in any order, it may be difficult to determine graduate capabilities. Program Outcomes and maps will help with this. Describe how the program provides experiential learning opportunities that allow the learners to perform, with support, as a practitioner in their field. You can discuss the planned learning activities that allow the student to perform as a practitioner, for example, placement opportunities or capstone assignments.

Criterion 4. Human, physical, financial, and support resources to support student achievement of program learning outcomes are available and accessible. 1. The program faculty members, as a whole have adequate academic preparation and workplace experience to deliver a quality program. The Chair will gather data regarding the preparation and experience of the faculty team and submit a summary report as part of the Faculty Focus Group report. Student perceptions of faculty knowledge and skill are available in the Course Assessment Roll Up Report (#1 a-g) 2. Faculty members are evaluated every three years. Describe how faculty are evaluated both formally and informally in your department. 3. Faculty members engage in professional development activities that ensure they are current in their field and developing teaching expertise. This data will come from the summary report that the Chair will submit to be included in the Appendix with the Faculty Focus Group report. 4. Students consider faculty to be available. Some student feedback is available in the Course Assessment Roll Up Report in question #13a. 5. Students consider faculty to be adequately prepared for class. See questions #1a-d on the Course Assessment Roll Up report. It will be necessary here, as above, that any concerns raised by students are discussed in a confidential manner and

PQR 2008 – 2009: Guide For Team Leaders SPTS

23

are addressed in the Implementation Plan in a confidential manner. The PQAA may be of assistance when deciding how to document areas which need to be addressed. 6. All students are assigned an academic advisor. Describe how students in your program can get academic advising assistance. 7. Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is. Describe how the department informs students of the availability of academic advising. 8. Labs, clinical facilities and placement facilities are complementary to and integrated into the program and allow the learner to demonstrate the learning outcomes. In the Course Assessment Roll Up Report, student feedback is available in question #13i 9. Students indicate that there are adequate and accessible learning resource materials including: textbooks in the bookstore; online materials; print resources; equipment and student support services, to allow them to be successful. The Course Assessment Roll Up Report provides student feedback in questions #2b and #13. 10. The program is financially viable. The demand for the program has been sustained for the last five years. There is a future demand for graduates of this program. Conclusions for this criterion can be drawn from the Registration and Revenues data. . You will have been provided with 5 years of data for program contribution and registration data. Registration and withdrawal statistics from the last 5 years will help identify any significant areas that may need follow up in the focus groups. 11. The learning environment is safe. Student perception of the learning environment is found in the Course Assessment Roll Up Report as of 2008. You do not have any historical data for this. This question needs to be asked at the Student Focus Group session. If students identify areas where they feel unsafe – this needs to be explored and strategies to resolve this should be discussed and feedback given. It is also important that students feel “safe” asking questions, i.e. that the classroom environment is conducive to students being comfortable sharing their learning needs.

PQR 2008 – 2009: Guide For Team Leaders SPTS

24

12. The students are provided with the information they need to know to function safely in both the College and workplace learning environments. Describe any special orientation processes provided for learners to help them function in the classroom, lab or workplace environments. Any special precautions taken in lab settings, like WHMIS protocols, use of safety equipment, etc., can be discussed here.

Criterion 5. Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.

1. Students indicate that they are satisfied with the program. The Course Assessment Roll Up Report can also be used to identify student satisfaction with course quality, especially question #5. Any questions on this report which indicate student dissatisfaction may identify issues that need to be discussed at the student focus group session. 2. Issues raised at Program Councils are addressed in a timely fashion and feedback is provided to the Council. CE programs do not have Program Councils however you can indicate some anectodal information about how student issues are addressed and how you respond to student requests for help. You can also describe how the academic communicate with each other to ensure student complaints and requests are addressed in a timely fashion. 3. Learners progress through the program, achieve program outcomes and graduate in a timely fashion. Looking at the Registration reports for the last five years will help identify any courses in particular that appear to inhibit a student’s progress through the program. Identification of courses which have high failure rates may need to be reviewed and strategies recommended that allow the students to be successful while maintaining the academic integrity of the program. Faculty and students can provide valuable feedback for this criterion. Concerns here also include core courses that are only offered once a year. Are there strategies available to allow students to continue without having to lose a whole academic year? When courses with high failure rates are offered off-cycle to help students remain on cycle in the program, this should be discussed along with strategies developed to help the students be successful. An example of this would be choosing textbooks that include programmed learning CDs for courses that require a lot of rote learning like anatomy or geography.

PQR 2008 – 2009: Guide For Team Leaders SPTS

25

There are many other factors that impact adult learner progress in a program. It is therefore important that the College reviews troublesome courses to ensure strategies are in place to allow students to be successful.

4. Learners with a wide range of abilities demonstrate the expected learning outcomes. While no statistics are available for the number of students with differing abilities and their success in the program, faculty and students may provide some anecdotal information relevant to this criterion. This is a good place to highlight special accommodation that program staff members have made to assist students with special needs to meet the program outcomes and be successful in graduating from the program. 5. Graduates are satisfied with the overall program experience. There is no survey data available for this question so you will need to ask your graduates this question. 6. Graduates are obtaining employment in their fields. Many continuing education students are already working in the field and are upgrading or looking for a career change. This question will need to be modified when graduates are surveyed to consider how the program assisted them in either gaining employment or advancing in their field. 7. Graduates are successful in obtaining external licenses or credentials where relevant. Include this information and its relevance as applicable. For example, if the students write a licencing examination to obtain provincial or national certification and you have the results of the success of your graduates on those examinations, include them here.

8. Employers are satisfied with graduate performance. Review the five year trend for employer satisfaction in your Program Mix Review data. Please note that the response rate for the employer survey is low, given that we must first contact the graduate and receive permission and employer contact information prior to contacting employers. Due to the lower response rates, the data may not be as meaningful as with the other surveys which have a higher response rate. Any issues that arise from the survey data can be added to the agenda of the external stakeholder focus group meeting.

PQR 2008 – 2009: Guide For Team Leaders SPTS

26

REVIEWING DOCUMENTS
The program approval process requires significant documentation to ensure that the curriculum meets Ministry and College Standards. Curriculum is revised annually and it is prudent during the Program Quality Review to ensure that the curriculum currently delivered by the College still conforms to these standards. As well, our clients need information about our programs to make informed decisions. Lastly, directives and policies must be designed and administered to ensure fair and equitable treatment of all our students, to deliver the best program we can, and to ensure the integrity of our credentials. College Directives affecting Program Quality Review will be revised as directed by the Vice President Academic. Any concerns regarding any of the Directives should be brought to the attention of the Program Quality Assurance Administrator. Documenting the document review process is an important part of the Program Quality Review process. The following documents need to be reviewed by the team leader, in cooperation with the Chair and department faculty, to ensure that the information contained is complete, accurate, and current and that it conforms to Ministry and College Standards: 1 2 3 4 5 6 7 8 9 Program Monograph Course Outlines Program Council Meeting Minutes if applicable Advisory Committee Meeting Minutes if applicable Articulation Agreements if applicable Reports from External Accreditation or Licensing bodies if applicable Program of Study Program Learning Outcomes i Hybrid courses

Program Monograph
The Program Monograph contains the Admission Requirements for the program. These are documented in the Student Information System and revised each year. Admission Requirements should ensure that students are adequately prepared to enter the program to ensure that they have the foundational knowledge needed on which to base their learning without unnecessarily limiting access to a program. College Eligibility is Ministry mandated and are for the most part standard for post secondary programs: Ontario Secondary School Diploma or at least 19 years of age. Program Eligibility is determined by the department and should reflect any necessary knowledge and skills that the students should have prior to entering the program. Common program requirements can include a specific level of Mathematics or Science. Mature students may be required to meet the program eligibility even though they are 19 or older but do not have a high school diploma. To allow students adequate time to plan their course load in high school, the

PQR 2008 – 2009: Guide For Team Leaders SPTS

27

current monograph publishes admission requirements for the upcoming academic year, not the current one. The Program Description/Overview and the Employment Opportunities sections of the Calendar/Monograph need to provide the student with the information needed to allow the student to determine that this is the appropriate program to select. The Expenses section should list all relevant program expenses to allow students to plan their financial commitment. Fees and Application Information are supplied by the Registrar’s Office and are published in on Course.

Course Outlines
A course outline must be developed for each and every course offered in the program. It must be accessible to students and on file in the PLAR office. There is a now a shared drive where all course outlines are to be posted. All sections of the course outline must be completed. Please refer to Academic Directive E33 - Course Outlines for more information. Proposed for 2008 Fall, all course outlines for full-time activity will be maintained on the Course Outline Mapping and Management System (COMMS). Necessary accommodation for course outlines for the School of Parttime Studies will then be made. The maps you produce for this PQR process will be very helpful when course outlines are entered into this system. The team leader needs to ensure that there is congruency between course learning requirements, course learning activities, learning resources and evaluation. Course outlines need to be reviewed with the faculty teaching the courses to gain the information needed for the review process.

Program Council Meeting Minutes
Program Council meetings should be held at least twice each term for full-time programs. The minutes of these meetings will help identify issues that should be brought forward to any or all of the focus groups. Information about Program Councils can be found in Directive E2 – Program/Department Councils. It is also critical that issues raised by students are addressed and that feedback is provided to the students. Not every issue raised at a Program Councils can be resolved. However, students do need to understand what action, if any, will take place and the proposed timeline for any resolution. Most CE programs do not have Program Councils but should keep on file, survey data or minutes of any student meetings.

Advisory Committee Meeting Minutes
Each full-time program should have an Advisory Committee which meets four times a year. The minutes of the meetings will help identify issues that should be brought

PQR 2008 – 2009: Guide For Team Leaders SPTS

28

forward to any or all of the focus groups. Information regarding Advisory Committees can be found in Directive A1 – Advisory Committees. Some CE programs do have advisory committees or attend meetings of advisory committees for relevant full-time programs. Feedback from these meetings can be useful in the review process.

Articulation Agreements
The final PQR report should include a list and brief description of any articulation agreements that exist with other post secondary institutions/professional associations. If no articulation agreements currently exist and it appears that the graduates from the program regularly seek university education or industry certification, it may be prudent to recommend that an articulation agreement be considered.

Reports from External Accreditation or Licensing Bodies
If relevant to the program, student success on external licensing exams should be recorded in the final report.

Program of Study
When looking at the program of study, some consideration needs to be given to the following. The courses need to be sequenced appropriately. Prerequisites and co requisites must be appropriate to ensure students have an adequate foundation on which to build their learning without unnecessarily impeding progression in the program.

Program Learning Outcomes
Templates for mapping the curriculum against provincial standards for program learning outcomes, general education and essential employability skills, if applicable are provided to team leaders.

Hybrid Courses
The following criterion is also used in the review process: “The program meets the minimum number of hybrid courses as outlined in the policy.” While looking at the Program of Study, identify the number of course hours that are delivered online in each term and compare that to the College target, if applicable.

PQR 2008 – 2009: Guide For Team Leaders SPTS

29

REVIEWING SURVEY DATA
There is a limited amount of survey data already available for continuing education courses and programs. For the Program Quality Review process, as much of this data as is feasible will be provided. This will reduce the amount of data that needs to be collected and will allow team leaders to focus on analysis of the data and identification of items that can be discussed further at the focus groups. The following sources of survey data are discussed in this section: 1 2 : Review of Registration Statistics The Course Report

Review of Registration Statistics
Three years of Registration data are provided to look at trends in registrations. Items of concern may include a course where a large number of students withdraw or a course has to be repeatedly cancelled for lack of registration. As well, it will be useful to look at the number of students who are registered who complete the course successfully.

The Course Assessment Report
This report is a roll up of all course section course assessments in a program. It provides student feedback at the program level and will identify program strengths and areas of concern. This will help the team leaders identify items for discussion at the student focus group. In areas where the result seems lower than expected, the Academic Manager needs to review the individual course section forms to identify if a one or more sections are skewing the statistics.

PQR 2008 – 2009: Guide For Team Leaders SPTS

30

Continuing Education Course Assessment Questions

CONTINUING EDUCATION SURVEY - EXCERPTS
1. I have completed this survey in another course, within the past two weeks.
NO

As of January 2008

YES

Reasons for Coming and Registration 2. What was your most important reason for choosing to register in this institution?
1) 2) 3) 4) 5) 6) 7) 8) 9) Previous positive experience at this College Reputation of this College Reputation of this course/program Location This course/program is only offered here Course full at other institution Recommendation of friend/employer Cost of program/course Other

3. How did you receive the Continuing Education catalogue/calendar? 1) Mailed to me in response to my request 2) Delivered to my home 3) Picked it up at a college campus 4) Received it from my employer/co-worker 5) Web site 6) Did not receive 7) Other 4. What sources of information about Continuing Education courses did you find most valuable? 1) Newspaper 2) Program information brochure/flyer 3) Personal visit to College campus 4) Phone call to College 5) Family member or friend 6) Employer/Professional Association 7) Web site 8) Other 5. What is the main reason you are taking this Continuing Education Course? 1) Looking for a job (entry level position) 2) Improving myself in my current career 3) Preparing for a change of careers 4) Personal development and fulfillment 5) Pursuing a hobby or interest

PQR 2008 – 2009: Guide For Team Leaders SPTS

31

6) Seeking a credit towards a full-time program 6. What are you working towards? 1) Certificate (less than 20 courses)

PQR 2008 – 2009: Guide For Team Leaders SPTS

32

COURSE/WORKSHOP ASSESSMENT FORM - EXCERPTS
Continuing Education Prior to January 2008

Teaching and Learning 1. How would you rate this course/workshop?
1-Excellent 2-Good 3-Fair 4-Not Satisfactory 5--Does Not Apply

a) b) c) d) e) f) g)

Teacher’s knowledge of the subject Teacher’s presentation of course/workshop material Helpfulness of teacher Opportunity for participation and discussion in class Fairness of evaluation (e.g. tests, assignments) Course/workshop content Overall, the teacher

2. How useful were the following in learning the course/workshop material?
1-Very Useful 2-Somewhat useful 3-Not too useful 4-Not at all useful 5-Does not apply

a) Usefulness of assignments b) Usefulness of requires course/workshop textbooks c) Usefulness of handout material

3. Did you receive an outline with an explanation of the objectives at the beginning of this course/workshop? YES NO 4. Was the method of evaluation explained to you in this course/workshop?
NO YES

5. Based on your experience in this course/workshop, would you recommend this course/workshop to a friend with the same interests? YES NO NOT SURE Information and Services 12. How would you rate the following aspects of the campus facilities?
1-Excellent 2-Good 3-Fair 4-Not Satisfactory 5--Does Not Apply

a) b) c) d) e) f) g) h) i)

Cleanliness of the building (e.g. hallways) Cleanliness of the washrooms Cleanliness of the classrooms Classroom temperature and air circulation Directional signs outside the building Directional signs inside the building Library hours Bookstore hours Cafeteria hours

PQR 2008 – 2009: Guide For Team Leaders SPTS

33

13. How would you rate the following?
1-Excellent 2-Good 3-Fair 4-Not Satisfactory 5--Does Not Apply/Did Not Use

a) b) c) d) e) f) g) h) i) j) k) l) m)

Helpfulness of academic or department staff Helpfulness of registration staff Parking Quality of library resources Availability of required books at bookstore Helpfulness of bookstore staff Price of books Access to college computers outside staff time Classroom or lab equipment Special needs services Career counseling College food services Helpfulness of food service staff

PQR 2008 – 2009: Guide For Team Leaders SPTS

34

CURRICULUM MAPPING

4

In this section
    Program standards Purposes of curriculum mapping Terms and concepts Course outline review

PQR 2008 – 2009: Guide For Team Leaders SPTS

35

PROGRAM STANDARDS
Since 1993 the Government of Ontario has been developing system-wide program standards for post-secondary programs in colleges across Ontario for programs that award an Ontario College credential. You may have some courses in your program which are common to a program leading to an Ontario College Credential. Each program standard contains the following elements: 1 2 3 The Vocational Standard (Program Vocational Outcomes) The Essential Employability Skills Standard The General Education Standard

Vocational Standard
The Vocational Standard is the vocationally specific learning outcomes which apply to each diploma and certificate program. Vocational standards apply to all similar programs offered by colleges across the province. Not all programs have provincial vocational outcomes. If this is the case, and where possible, the PQA administrator will provide program descriptions to assist in the development of program level outcomes. College approved programs that do not yet have program outcomes will include the development of the new outcomes in the recommendations. These will need to be presented to the Curriculum Review Committee and they will then be entered into COMMS and subsequently transferred to COMMS

Essential Employability Skills Standard
The Essential Employability Skills Standard is the essential employability skills (previously known as Generic Skills) needed for both career and personal success (communication skills, numeracy skills, thinking and problem-solving skills, information management skills, interpersonal skills, and personal skills). They are expressed as learning outcomes. There are eleven Essential Employability Skills learning outcomes which apply to each program.

General Education Standards
The General Education Standards are the requirements for general education courses that provide all learners with choice and breadth of experience beyond the vocational areas. Note: The Essential Employability Skills and General Education Standards apply only to Ontario College certificate, diploma and advanced diploma programs. Apprenticeship, college approved programs and graduate certificate programs are not required to meet these standards. Applied Degree programs have program standards established by PEQAB.

PQR 2008 – 2009: Guide For Team Leaders SPTS

36

PURPOSES OF CURRICULUM MAPPING
College program faculty annually review and revise their curriculum to ensure that they meet programs standards. A curriculum map can be used to        document curriculum and the inclusion of standards for accrediting/credentialing groups identify opportunities in the program for learners to demonstrate learning outcomes at the required level serve as a planning tool to ensure that all program standards are developed within the program identify areas of redundancy where content is more than adequately covered identify paths that learners can follow to meet graduation requirements help faculty, learners and others “situate” courses and learning experiences within the larger curriculum provide an overview of the curriculum for the total program

PQR 2008 – 2009: Guide For Team Leaders SPTS

37

TERMS AND CONCEPTS
We plan to map how each course addresses the learning outcomes. You will be asked to indicate if a learning outcome is taught, if it is assessed and whether or not there is an opportunity for a culminating performance or demonstration of the learning outcome in your course(s). If the program course outlines are entered into COMMS, these decisions are made at the time of entering the curriculum information into COMMS. For those programs not yet a part of the COMMS project, the decisions will be taken as part of the course outline review process and the manual production of maps. To identify where to indicate if an outcome is taught, assessed or provides an opportunity to demonstrate a culminating performance you must be able to answer yes to each of the questions in that section.

Teach

Do you provide instruction/learning opportunities in this skill in your course? Is this skill identified in one or more course learning requirement(s)? Do you devote a significant amount of time to facilitating student development of the skills and knowledge embedded in the outcome? Are there assignments, tests or projects which are designed to allow you to evaluate or assess student performance of this outcome or some of its elements? Do you, in your evaluation of student performance, verify that this particular outcome (or a significant component of it) has been achieved? Is this outcome reflected in your course outline in the course learning requirements and/or embedded knowledge and skills and in the evaluation section of the course outline? Culminating performances are tasks or activities designed to assess a learner’s ability to demonstrate one or more learning outcome(s) in its totality. While they do not necessarily occur at the end of a program of study, they do evaluate whether a learner is able to integrate and apply their learning to demonstrate the performance described in the learning outcome(s) at the exit level. Is there an opportunity for you to evaluate the outcome in its totality? Does the evaluation result in a final product or performance which allows you to determine whether the learner has integrated the knowledge and skills identified in the elements of performance? Can you determine from this performance if the learner has demonstrated the outcome? Program culminating performances should require learners to demonstrate learning similar to what would be expected of new graduates in as close to a “real world” context as possible. If a culminating performance is assigned to a course, the professor teaching the course will be asked to evaluate whether or not the students successfully demonstrated the outcome(s) the culminating performance addresses

Assess

Culminating Performance

PQR 2008 – 2009: Guide For Team Leaders SPTS

38

MAPPING OF PROGRAM OUTCOMES PROGRAM NAME: PROGRAM CODE:

Program Outcomes/Course 1.

Totals
T A C

2.

T A C

3.

T A C

4.

T A C

5.

T A C

6.

T A C

7.

T A C

8.

T A CP

9.

T A C

10.

T A C

11.

T A C

Totals

T A C

PQR 2008 – 2009: Guide For Team Leaders SPTS

39

Program Vocational Outcomes—Program Totals Program Name: Date:
Program Outcomes 1. Level 1
T A C 2. T A C 3. T A C 4. T A C 5. T A C 6. T A C 7. T A C 8. T A CP 9. T A C 10. T A C 11. T A C Totals T A C C

Level 2

Level 3

Level 4

Level 5

Level 6

Totals

PQR 2008 – 2009: Guide For Team Leaders SPTS

40

COURSE OUTLINE REVIEW
Each year as part of the curriculum review process, course outlines are reviewed and updated as necessary, to reflect changes made in the course content. Course outline updates may include changes to course learning requirements, learning activities or assessment methodologies. There are five Ministry criteria about course learning requirements, program outcomes, learning activities and resources, and evaluation methods: 1 There is congruency between the course learning requirements and the program learning outcomes. There is a match between course learning requirements, course learning activities and learning resources. Learning methods are published and are matched to the learning requirements. Evaluation methods allow students to demonstrate the course learning outcomes. PLAR opportunities exist and are based on learning outcomes.

2

3 4 5

On the following page is a working document that can be used to help assess the above criteria. There is also a summary sheet that needs to be completed and included in your final report. All faculty members are expected to formally review their course outlines and submit one of the worksheets to the Team leader for each course section taught. The Team leader then will be able to identify any gaps in meeting the criteria and can prepare the summary sheet for the report. For any gaps noted, an action plan will need to be developed and included in the final recommendations. Note: Please save the working sheets as they will assist in the revision of course outlines in the Course Outline Management and Mapping System (COMMS).

PQR 2008 – 2009: Guide For Team Leaders SPTS

41

COURSE OUTLINE REVIEW CHART
The following chart will help you evaluate whether or not your curriculum meets the following criterion: 1. There is congruency between the course learning requirements and the program learning outcomes. 2. There is a match between course learning requirements, course learning activities and learning resources. 3. Learning methods are published and are matched to the learning requirements. 4. Evaluation methods allow students to demonstrate the course learning outcomes. 5. PLAR Challenges exist and are based on learning outcomes Note: You will need to complete this sheet for each one of your course sections.
DATE OF MAP:

Program Name: Course name and code: Course Learning Learning Requirements Activities 1.

Academic Level: Evaluation methods Match with Criteria: Y/N 1. 2. 3. 4. 5. 1. 2. 3. 4. 5. 1. 2. 3. 4. 5. 1. 2. 3. 4. 5. 1. 2. 3. 4. 5.

2.

3.

4.

5.

PQR 2008 – 2009: Guide For Team Leaders SPTS

42

Course Outline Review Summary Sheet Program Name: Date: This template is in the Final Report Document. It is completed by the Team leader once all course section information has been completed and submitted. Course Name Criteria (see Course Comments – particular strengths to Outline review chart) and Number be recognized or recommendations for changes needing to be made 1 2 3 4 5

PQR 2008 – 2009: Guide For Team Leaders SPTS

43

FACILITATING FOCUS GROUPS

5

In this section
  Types of focus groups Tips for facilitating focus groups

PQR 2008 – 2009: Guide For Team Leaders SPTS

44

TYPES OF FOCUS GROUPS
During the Program Quality Review process there will be three types of focus groups. The first one includes the entire faculty involved in the delivering the program if possible. The second is with external stakeholders which includes graduates. The third group involves students currently in the program. The goal of each of the focus groups is to generate feedback from the members about their perception of the program’s effectiveness. There are a variety of criteria that are relevant to each focus group. Once the team leader and the academic chair have reviewed the available survey data, they can determine the criteria for which they need more information. Some of the survey data may provide adequate information to confirm that some criteria have been met and feedback from the relevant focus group may not be necessary. On the following pages are detailed directions to assist you in planning and facilitating a focus group session and writing a report on the feedback obtained. There are agenda templates, letter of invitation templates, report templates, and checklists. These templates can be modified if the information is gathered using an email survey. Given the team leader’s involvement in a program as a member of the faculty or as the coordinator, it has been determined that it is more effective to have an external facilitator to chair the External Stakeholders’ Focus Group meeting and the Student Focus Group meeting. Each team leader will pair up with a team leader colleague from another School. Each team leader will plan his/her focus groups and the agendas. However, team leaders will facilitate the meetings for each other. This allows the team leader to participate in the discussion and follow up on issues without having to be concerned about keeping the meeting on track. More importantly, it helps to reduce the bias in the recording of the comments and report writing process. The report of the focus group meetings will be written by the facilitator with an opportunity provided to the program team leader and program chair to review the report for accuracy before it is finalized. In the event that a focus group is not deemed feasible and a survey is determined to be the most efficient way to gather data, the survey responses will be sent to the facilitator partner who will tabulate the results and write the report for the program team leader/chair.

PQR 2008 – 2009: Guide For Team Leaders SPTS

45

TIPS FOR FACILITATING FOCUS GROUPS
Here are hints from those who have facilitated external stakeholder groups in the past: 1 Smile  Welcome everyone Ensure the room is set up properly.  Prepare flip chart paper, masking tape and markers ahead of the meeting.  Ensure some one is designated to help record, add paper, etc..  Use non toxic markers. Take the time to do introductions well  This helps set up a positive environment. Review the agenda with the group.  Allow for questions. Start with an easy issue to get things going.  Save tougher agenda items for later in the session. Listen carefully. Ask for clarification if the comments or questions are not clear.  Ask the person to please repeat the question/comment.  Record all comments – no judgments. Face the group as much as possible. Pause to review the content on the flip chart paper.  Make corrections if necessary. Try to ensure everyone has a chance to comment – but don’t force it.  Monitor the time if one or two people seem to be dominating the conversation. Watch the time  Keep the agenda moving to ensure all topics are addressed. Allow time to review flip charts for issues requiring recommendations.  These should be on a separate chart.

2

3

4

5

6 7

8 9

10

11

12

PQR 2008 – 2009: Guide For Team Leaders SPTS

46

13

Gain group agreement for recommendations.  If there is a contentious issue that is taking up too much time, this then gets deferred to the Chair for further action. Thank everyone for their time and input.  Let the participants know that they will receive feedback one the review process is finished.

14

PQR 2008 – 2009: Guide For Team Leaders SPTS

47

FACULTY FOCUS GROUPS

6

In this section
     Purpose of program audits How program audits support program quality review Planning faculty focus groups Preparing faculty focus group reports Sample documents

PQR 2008 – 2009: Guide For Team Leaders SPTS

48

PURPOSE OF PROGRAM AUDITS
A program audit provides a framework for the annual review of curriculum, program organization and identification of program-related issues by faculty who teach in the program. The faculty is encouraged to examine all aspects of the program and make necessary changes on an annual basis in order to ensure that the program is meeting learner needs and is remaining current and relevant. Program audits should occur every year to ensure that program-related issues are addressed and that changes made to the program are documented in the annual program review document. The amount of faculty time required to conduct a program audit will vary from year-toyear and from program-to-program depending on a wide variety of internal and external factors. Program teams should plan at least two sessions each academic year for this task. When program learning outcomes or program organization are changed substantially, more time will be needed to revise curriculum and rewrite the program literature. An agenda should be set for each of the team meetings and notes should reflect the discussion and decisions taken. This report will then be the basis for the program audit report which should be submitted to the department chair before the end of the academic year. The tools provided for the Program Quality Review process can be used for the annual Program Audit as well as for the more formal five year Program Quality Review.

HOW PROGRAM AUDITS SUPPORT PROGRAM QUALITY REVIEW
The Program Self-Audit System is an important part of the ongoing program review process. Its purpose is to provide faculty teams an opportunity to review the events, successes, failures and issues that have arisen in the program during this and recent academic years and to document and report significant highlights in the Program SelfAudit Report. An important function of program review is to inform all program stakeholders about the diverse demands, trends and issues which affect the program and the people associated with it. It endeavours to promote positive attitudes toward necessary change in keeping with the times and societal needs. Faculty participation in the active examination of the program’s major activities and resources is critical as is their involvement in making needed amendments to courses, program learning outcomes and organizational and delivery systems. The Program Self-Audit Guide on the following pages is designed to facilitate the work of faculty in the annual examination of the program. Faculty teams may choose their

PQR 2008 – 2009: Guide For Team Leaders SPTS

49

own way of using this document as an aid in the annual audit process and in writing the report. The list of program-related activities and resources in the guide may include more factors than a team is ready to examine each year, or, it may have overlooked some areas that should be discussed. There is space included at the end of each category to allow faculty teams to add any items to the list as they see fit. One aim of the Program Self-Audit is to promote a learning culture throughout the College by facilitating regular examination and renewal of the course-related and program-related services we offer, by the professionals who are responsible for their quality and relevance. Program Self-Audit empowers the faculty to assume responsibility for program renewal within a college system that is increasingly challenged to prepare graduates for a dramatically changing world of work in Ontario. Several documents need to be sent out to the faculty members, as applicable: 1 2 Program Self-Audit (to be submitted to team leader for summarizing) Course Outlines with  Course Outline Review Document  Vocational outcomes mapping documents

The Faculty Education and Experience Survey is sent out and summarized by the Chair. Only the summary chart will be included in the final report, not the individually submitted data sheets. Sample memos are included to assist with the invitation to faculty members and explanations of what is expected. The Faculty Focus group will deal with three major agenda items: 1 2 3 Mapping Vocational Program Outcomes to courses Review of Course Outlines Discussing issues (results of Program Self-Audit)

The Program Quality Review team leader will convene a meeting of as many faculty members who teach in the program as possible, including part-time faculty and professors of service courses. The Program Self-Audit, relevant mapping and review work sheets, and the course outlines should be distributed prior to the faculty focus group. The faculty members should review their course outlines and the Program SelfAudit document prior to the meeting. As faculty members complete the Self-Audit, they can identify issues that need to be discussed at the faculty focus group session. It may be recommended to follow up with some of the issues raised at the external stakeholders focus group or the student focus group. Faculty teams are advised to use the Self-Audit document as a checklist to determine which items should be given priority during the faculty focus group. The Yes/No columns allow faculty to check off which criteria are perceived to be met. Specific recommendations should be summarized at the end of each set of items for discussion.

PQR 2008 – 2009: Guide For Team Leaders SPTS

50

PLANNING FACULTY FOCUS GROUPS
Time Frame for the Session
You will need 3-4 hours for the faculty focus group session, depending on how much individual preparation is done by the faculty members. There may need to be two sessions for faculty. The first session will be early in the term, prior to the student and external stakeholder sessions. It must be early enough to ensure that curriculum mapping is done by the end of January, to allow the identification of any curriculum updates that need to be developed prior to the deadline for the annual curriculum review process. The team leader and the chair will have reviewed the statistics and will determine if there any trends that need to be investigated. Additions to agendas for the student and external stakeholder focus group sessions will be finalized. Discussion regarding any of the documents being reviewed can be addressed at this time. The working part of the session should be about three hours in length. All faculty members who teach in the program should be invited. Part-time faculty members will be reimbursed to attend this meeting. A second faculty session may be planned after all the focus groups have been completed. The team leader and the chair can share with the faculty recommendations made by focus group members. Conclusions and final recommendations need to be formulated for the final report. The input from this session would be documented in a separate report and included as one of the appendices. The team leader can open the meeting by welcoming everyone and providing an opportunity for introductions as necessary. After a brief overview of the Program Quality Review process, the team leader can then review the Agenda, and explain the process and ground rules. The ground rules for the discussion are:      Ask that only one person speak at a time. Members are asked to be as concise and focused as possible. All comments/perceptions should be freely stated and will be recorded on flip charts by the team leader. All contributions are regarded as valid and no consensus is required with respect to the perceptions and feedback stated and recorded. The Team leader will attempt to achieve consensus only with respect to final recommendations to be made to the Program.

All elements of the discussion should be recorded on flip charts during the session. The team leader may enlist the help of an external recorder, if one is available in the department. The flip charts will serve as a permanent record of the discussion and will provide the basis for the focus group report.

PQR 2008 – 2009: Guide For Team Leaders SPTS

51

PREPARING FACULTY FOCUS GROUP REPORTS
Team leaders will facilitate the Faculty Focus Group for their own programs. The team leader will prepare the report for the session which s/he has facilitated, using the transcriptions of the flip chart notes as the basis for the report. The program chair and the team leader will need to meet to review the report before it is finalized. The Contents of the Report will be as follows: 1 2 3 4 5 6 7 8 Identification of the Faculty Members who attended the meeting Purpose and objectives of the Focus Group Process for the conduct of the Focus Group session Anticipated outcomes Agenda Context and participation Highlights of the Focus Group Session Recommendations – This would include any recommendations made after a review of the information from the various Focus Group sessions

Note: Do not include the notes taken on the flipcharts. These comments are the foundation for the highlights in the report but the actual list is not to be included in the final report.

PQR 2008 – 2009: Guide For Team Leaders SPTS

52

Memo
To: From: CC: Date: Re:

Faculty members of XXX Program; see Distribution below xxx, Chair xxxxx, Team Leader for 2008-2009 Program Quality Review

Program Quality Review

The xxx Program is part of the College’s cyclical Program Quality Review (PQR) process for the 2008-2009 cycle. Xxx has been assigned the task of reviewing the program against the Ministry and College Criteria. Part of the process is to determine that the program is adequately staffed to ensure the delivery of a quality program. To that end, please complete the attached document and submit it to me by November 30th, 2008. I will be summarizing the information and will submit cumulative totals in a report. If anyone has any concerns about how this information will be conveyed, please feel free to discuss them with me. You will also be receiving documents from xxx that will assist in reviewing the curriculum. Curriculum review is an important part of your work each and every year and this is an opportunity for us to review the cumulative effects of changes we have made over the last five years. We need to ensure that the program is compliant with the vocational outcomes and that there is congruency between the course learning outcomes, learning activities and assessment activities. Please do consider the Program Self Audit document carefully and identify any areas of concern that you might have. We need to take the time to acknowledge the areas of quality in our program, and devise strategies to ensure that the quality is enhanced. Thank you for your support for these PQR activities. Attachments: Faculty Education, Experience and Professional Development Survey

PQR 2008 – 2009: Guide For Team Leaders SPTS

53

Faculty Education, Experience and Professional Development Survey

This survey is to be submitted directly to the Chair of the program who will summarize the data. This survey is considered confidential information once completed. Name: Date: Program: Education Number of years relevant experience in the workplace Number of years teaching in this program 1. as a full-time professor 2. as a part-time professor Date of hire: Other relevant Credentials Other relevant Experience List any Professional Development activities attended in the last 5 years. Include formal courses, workshops, conferences, industry renewal opportunities, sabbatical leaves, etc. When was your last performance review?

1. 2.

Confidential once completed

PQR 2008 – 2009: Guide For Team Leaders SPTS

54

Summary of Faculty Credentials and Experience Program: Date: Total number of students in the program Total number of Faculty teaching in the program: Full-time Part-time Education: Number of Faculty with PhD Number of Faculty with DEd Number of Faculty with Master’s Degree Number of Faculty with Bachelor’s Degree Number of Faculty with Diploma Number of Faculty with special credential (please list the relevant credentials and the number of Faculty with the credential) Number of Faculty with less than Diploma preparation Teaching Experience: Total number of years of full-time teaching experience Total number of years of part-time teaching experience Special areas of strength of Faculty Any areas of concern regarding Faculty education and/or experience Workplace Experience Number of years of relevant workplace experience combined. Professional Development: Number of Faculty members who have attended professional development within the last two years. Performance Reviews Number of Faculty with a written performance review done in the last year. Number of Faculty with written performance review done more than a year ago but within the last two years. Number of Faculty with written performance review done more that two years ago but within the last three years. Number of Faculty with no written performance reviews with the last three years.

PQR 2008 – 2009: Guide For Team Leaders SPTS

55

Memo
To: From: CC: Date: Re:

Faculty members of XXX Program; see Distribution below xxxxx, Team Leader for 2008-2009 Program Quality Review xxx, Chair[Click here and type name]

Faculty Focus Group[Click here and type subject]

As you aware, the xxx Program is part of the College’s cyclical Program Quality Review Process for the 2008-2009 cycle. You are invited to a Faculty Meeting on xxx, xxx, xx, 2008 in room xxx. Please review and complete the following documents as outlined below. These documents will be discussed at the meeting and then collected by the Team leader who will compile summary reports. Faculty Self Audit: This document includes the Ministry and College criteria against which programs at the College are evaluated. Please read the document carefully and indicate whether or not you feel the program meets each of the criteria. Put a check mark in the appropriate column and identify any comments you wish to make at the meeting. You will be submitting this document at the end of the meeting so the cumulative results can be tabulated. Issues raised by the faculty members can be brought forward to the student and external stakeholder focus groups for further exploration if necessary . Course outline review document documents: Faculty members must complete and submit the following documents for each course that they teach:   Mapping document for Vocational Outcomes Course outline review document

Attachments: Vocational/Program Outcomes Mapping Document at the course level Course Outline Course Outline Review Working Document

PQR 2008 – 2009: Guide For Team Leaders SPTS

56

Program Quality Review Criteria Program Self - Audit Each faculty member completes this Audit and submits it to the team leader who summarizes the data as part of the Faculty Focus Group Report. Program Quality Review Criteria Program Self - Audit Each faculty member completes this Audit and submits it to the team leader who summarizes the data as part of the Faculty Focus Group Report. Criterion 1. Admission, credit for prior learning, promotion, graduation, and other related academic policies support program development and student achievement of program learning outcomes. Yes No Comments 1. The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program. 2. Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them. 3. Students know how to get internal and external credits and recognition for prior learning. 4. Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program. 5. Students know how they will be evaluated. 6. Students indicate the learning requirements are relevant and meaningful. 7. Students indicate that assessment methods relate to the learning requirements.

Criterion 2. Programs conform to the Framework for Programs of Instruction and the Credentials Framework, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance. Yes No Comments 1. The duration and structure of the program are consistent with the program learning outcomes and the credential offered. 2. Appropriate credits are allocated for each component of the program, and transfer and laddering options are stated. 3. Prerequisites do not unnecessarily hinder progress in the program.

PQR 2008 – 2009: Guide For Team Leaders SPTS

57

4. Program learning outcomes are consistent with the credential granted, the title of the credential awarded, the provincial program standards (where these apply), and the minimum essential expectations of the workplace. 5. Program learning outcomes are reflected in course outlines. 6. Programs learning outcomes are used in prior learning assessment. 7. Changes to courses and program outcomes are introduced on a timely basis and are designed to maintain the relevance of the program. 8. The program has established articulation agreements. 9. The program conforms to the College policy for the number of English courses 10. All curriculum documentation is up-to-date including course outlines and the program monograph information. 11. There is congruency between the course learning requirements and the program learning outcomes. 12. There is a match between course learning requirements, course learning activities and learning resources. 13. Concepts of environmental sustainability are imbedded in the program curriculum. 14. Students have opportunities to develop the skills, knowledge and attitudes necessary to succeed in a global economy. Criterion 3. Methods of program delivery and student evaluation are consistent with the program learning outcomes. Yes No Comments 1. Program delivery, including that which takes place off-site, is consistent with the nature of the program, the learning outcomes, and the needs of the students. 2. There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities. 3. Learning methods are published and are matched to the learning outcomes. 4. College designated targets regarding hybrid courses are met. 5. Learners are provided the skills necessary to be successful with the learning strategies selected. 6. Evaluation criteria are published and students are aware of how and when they are going to be evaluated. 7. There is a match between course learning

PQR 2008 – 2009: Guide For Team Leaders SPTS

58

requirements and evaluation methods, i.e., evaluation methods allow students to demonstrate the course learning requirements 8. Evaluation methods are valid and reliable. 9. Students indicate that feedback is timely and allows them to build on their learning. 10. Students perceive evaluation to be fair. 11. Resubmissions, supplementals, and appeals are published, appropriate, fair, valid, and consistent. 12. Student workload and assessment is balanced across the term at both the course and program level. 13. There is a range of evaluation methods used consistent with a variety of learning styles. 14. Learners can earn credit for up to 75% of the program’s courses, using the PLA process. 15. Academic policies and practices that provide for the development and continuous improvement of teaching and learning methods are valued, documented, and supported. 16. Graduate capabilities, including knowledge, skills, and attitudes are consistent with program outcomes. Criterion 4. Human, physical, financial, and support resources to support student achievement of program learning outcomes are available and accessible. Yes 1. The program faculty members, as a whole have adequate academic preparation and workplace experience to deliver a quality program. 2. Faculty members are evaluated every three years. 3. Faculty members engage in professional development activities that ensure they are current in their field and developing teaching expertise. 4. Students consider faculty to be available. 5. Students consider faculty to be adequately prepared for class. 6. All students are assigned an academic advisor. 7. Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is. 8. Labs, clinical facilities and placement facilities are complementary to and integrated into the program and allow the learner to demonstrate the learning outcomes. 9. Students indicate that there are adequate and accessible learning resource materials including: textbooks in the bookstore; online materials; print No Comments

PQR 2008 – 2009: Guide For Team Leaders SPTS

59

resources; equipment and student support services, to allow them to be successful. 10. The program is financially viable. The demand for the program has been sustained for the last five years. There is a future demand for graduates of this program. 11. The learning environment is safe. 12. The students are provided with the information they need to know to function safely in both the College and workplace learning environments.

Criterion 5. Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens. Yes No Comments 1. Students indicate that they are satisfied with the program. 2. Issues raised at Program Councils are addressed in a timely fashion and feedback is provided to the Council. 3. Learners progress through the program, achieve program outcomes and graduate in a timely fashion. 4. Learners with a wide range of abilities demonstrate the expected learning outcomes. 5. Graduates are satisfied with the overall program experience. 6. Graduates are obtaining employment in their fields. 7. Graduates are successful in obtaining external licenses or credentials where relevant. 8. Employers are satisfied with graduate performance.

PQR 2008 – 2009: Guide For Team Leaders SPTS

60

Faculty Focus Group AGENDA Conduct of the Faculty Focus Group sessions should be as standardized as much as possible for all programs. The Agenda for each session will closely resemble the following outline.

Introductions      Team leader and faculty team Objectives of the session Process/ground rules/groups Tasks/activities Anticipated outcomes

Tasks      Mapping of Vocational Learning Outcomes Review of Course Outlines and Syllabus’ Identification of Issues arising from Program Self-Audit Identification of agenda items for student and external stakeholder focus groups Identification of Recommendations for consideration

PQR 2008 – 2009: Guide For Team Leaders SPTS

61

EXTERNAL STAKEHOLDER FOCUS GROUPS

7

In this section
   Planning external stakeholder focus groups Possible items for discussion Sample documents

PQR 2008 – 2009: Guide For Team Leaders SPTS

62

PLANNING EXTERNAL STAKEHOLDER FOCUS GROUPS
As discussed earlier facilitating a focus group requires objectivity. For this reason, the focus group leader for the external stakeholder focus group will be from another School. The focus group leader will facilitate the external stakeholder focus group and will write the report on the session. The program team leader and the program chair will meet to determine who to invite to the external stakeholder focus group. The chair will send out invitations to relevant Advisory Committee Members and additional employers, and other guests as determined by the chair and team leader. Faculty will be invited to the meeting as well. A draft invitation is included in this package. Conduct of the external stakeholder focus group sessions should be as standardized as much as possible for all programs. The Agenda for each session will closely resemble the outline included in the following pages. A letter is sent to each participant after the focus group session to thank them for participating. A brief summary of the recommendations can be included. It should also be noted that the may not be included in the final implementation plan and how the department will inform the participations of the availability of the final report.

Time Frame for the Session
The working part of the session should be about two hours in length. This does not include a brief social time at the beginning of the session or any socializing at the end. Refreshments should be available before the session begins so that participants can help themselves before the group starts. They can obtain refills on an informal basis throughout the discussions. It is better to not have a formal break in the middle of the session.

Leadership of the Session
The chair of the department should state the purpose of the focus group meeting and then introduce the focus group leader at the beginning of the session and invite the participants to introduce themselves. The focus group leader should then review the Agenda, and explain the process, and the ground rules. The ground rules for the discussion are    Ask that only one person speak at a time. Members are asked to be as concise and focused as possible. All comments/perceptions should be freely stated and will be recorded on flip charts by the team leader.

PQR 2008 – 2009: Guide For Team Leaders SPTS

63

 

All contributions are regarded as valid and no consensus is required with respect to the perceptions and feedback stated and recorded. The team leader will attempt to achieve consensus only with respect to final recommendations to be made to the program.

All elements of the discussion should be recorded on flip charts during the session. The focus group leader may enlist the help of an external recorder, if one is available in the department. The flip charts will serve as a permanent record of the discussion and will provide the basis for the focus group report. Again, note that the actual lists of comments are not to be included in the report. The focus group leader will prepare the report for the session which s/he has facilitated, using the transcriptions of the flip chart notes as the basis for the report. A template for this report has also been included. The program team leader and the focus group Leader will need to meet to review the transcription of the notes, before the report is written. A draft of the external stakeholders focus group report should be verified by the program team leader and the chair of the department before the report is finalized. Both the chair and the program team leader should receive a copy of the final report by the report date deadline.

PQR 2008 – 2009: Guide For Team Leaders SPTS

64

POSSIBLE ITEMS FOR DISCUSSION
Below are the criteria that may be relevant for discussion at the External Stakeholders Focus Group. Some of these criteria may not need to be addressed at the group session, unless the survey data or statistics indicate further information is required. Lists of comments regarding these criteria should be analyzed and the final report will contain the highlights and relevant recommendations, not the contents on the flip chart pages.

Students and Graduates
The qualifications and prerequisites required of the applicant are appropriate to allow the student to be successful without limiting access to the program. Students are obtaining employment in their field. What is the perspective of the group with respect to future employment opportunities for the next 5-10 years? Processes for granting of internal and external credits are clearly defined and available to students. Does the group feel that current employees have opportunities to upgrade/receive credit for experience/have access to the courses needed to obtain a college credential? Has anyone at the meeting used the advanced standing, external credit or Prior Learning Assessment and Recognition process? Do people know where to look? Do they have suggestions about where they would expect to find it? Employers are satisfied with graduate performance. Does the group feel that the graduates have adequate practical experience? Do the graduates understand the reality of the workforce? Are they able to relate the skills they have learned to the workforce? Graduates are successful in obtaining external licenses or credentials where relevant. Do the graduates understand the legal responsibilities of their job? Are they aware of the accreditations required?

Curriculum
The curriculum conforms to Ministry Standards with respect to Vocational Learning Outcomes, Essential Employability Skills and General Education. Does the group have any feedback on the relevancy of any of these standards? The curriculum conforms to the College Policy for the number of English Courses. What English skills would employers look for in our graduates? Any suggestions for applications in the workplace that could be adapted to learning activities I the classroom?

PQR 2008 – 2009: Guide For Team Leaders SPTS

65

The program curriculum is consistent the credential offered. Does the group feel that the program curriculum needs to change in the next 5-10 years in anticipation of a change in the knowledge base needed to work in the field? Students perceive the curriculum to be relevant. Does the group perceive the curriculum to be relevant?

Resources
Labs, clinical facilities and placement facilities are complementary to and integrated into the program to allow the learner to demonstrate the learning outcomes. Does the group have any recommendations about the labs, equipment and placement facilities? The students are provided with the information they need to know, to function safely in both the College and workplace learning environments. Any new information the College should know about? For example, WHMIS changes, new safety regulations.

PQR 2008 – 2009: Guide For Team Leaders SPTS

66

External Stakeholder Focus Group AGENDA

Introductions       Focus Group Leader Participants including faculty and staff Objectives of the session Process/ground rules/groups Tasks/activities Anticipated outcomes

The Past   Events/highlights/issues Patterns and trends that have influenced the program

The Present    Environmental influences o From the outside environment o From the inside environment Perceived strengths and weaknesses of the program Present trends in the vocation/workplace

The Future   Anticipated future trends in the vocation/workplace Actions needed to address trends

Recommendations for the Program

PQR 2008 – 2009: Guide For Team Leaders SPTS

67

External Stakeholder Focus Group Report 1. Purpose and Objectives of the Focus Group The External Stakeholder Focus Group sessions have been convened to gather feedback from persons who interface with our program on a regular basis and/or have unique perspectives to contribute to the Program Quality Review process. These sessions are an important component of the College’s Program Quality Review process. The xxx Program is one of several programs participating in the review process during this academic year. The Focus Group Sessions are intended to:     Generate feedback from members about their perceptions of the program’s effectiveness in a global sense; Probe for the environmental influences on the program from inside and outside the College; Tap the perceptions of members with respect to future trends in the workplace/vocation/occupational group that may signal some shift in the program’s training emphasis or some potential new directions to pursue. Follow up any questions that arise from survey review.

2. Process for the Conduct of the Focus Group session An active process was used to generate as much feedback about a range of program-related issues as possible within the time allowed. A full two hours will be required. The session will be facilitated by one the Program Quality Review Focus Group Leaders. 3. Anticipated outcomes     Evaluative feedback about the Program as it is currently delivered; A record of the Focus Group members’ perceptions of the present status of the Program and future trends that may affect the Program; Recommendations for Program updates/revision; Feedback on the Focus Group process followed.

4. Agenda 5. Context and Participation 6. Highlights of the Focus Group Session a. Environmental Influences on the Program i. External ii. Internal b. Perceived Strengths of the Program

PQR 2008 – 2009: Guide For Team Leaders SPTS

68

c. Perceived Weaknesses of the Program d. Trends affecting the future of the Program 7. Recommendations

PQR 2008 – 2009: Guide For Team Leaders SPTS

69

Sample letter from Program Chairs to Focus Group Members Address Date Dear xxxx Algonquin College has implemented a cyclical Program Quality Review process. This year the xxxxxxxxxx Program is engaging in the Program Quality Review process. The College is seeking your active participation in the External Stakeholder Focus Group session which will be held on xxxx, xxxx, xx, 2008, in room xxx. Other participants will include members of the Program Advisory Committee, graduates and employers. I have enclosed three documents with this letter. One document outlines the Purposes, Objectives, Format and Anticipated Outcomes of the Focus Group session. A diagram shows the components of the Program Quality Review process and the relationship of the External Stakeholders Focus Group. The agenda is also enclosed. During the review process the College is soliciting perceptions of the Program held by various stakeholders who interact regularly with program faculty, students and graduates. A list of issues and topics which may be addressed is also included. The Program Quality Review process seeks input from as broad a base as possible and from a wide range of perspectives held by various sectors linked to the Program. The data collected will contribute valuable information and guidance to ensure that the Program remains current and meets the changing needs of our students and the workplace. I anticipate that the Focus Group session will require two hours of your time, not including travel and a period of light refreshments prior to the start of the session. To ensure full participation for this special meeting, we ask that you respond to this invitation on or before xxx, to allow us to invite an alternate in the event you are unable to attend. The format for the Focus Group will be discussion-based and the session will be facilitated by one of the College’s Program Quality Review Team leaders. Your perceptions are important to the Program and we hope that you will be able to attend. Thank you for your ongoing support of our Program. Sincerely,

Program Chair

PQR 2008 – 2009: Guide For Team Leaders SPTS

70

Purpose and Objectives Purpose of the Focus Group The Focus Group sessions have been convened to gather feedback from our clients. The sessions are an important component of the College’s Program Quality Review process. Objective of the Focus Group Sessions The Focus Group Sessions are intended to:   Generate feedback from students about their experience with the College and their perception about the education they are receiving; Identify factors students feel impact the learning environment;

Format for the Session An active process will be used to generate as much feedback abut a range of program-related issues as possible within the time allowed. A full two hours will be required. The session will be facilitated by the Program Quality Review Team leader. Anticipated Outcomes of the Session The Focus Group Session is expected to provide:     Evaluative feedback about the Program as it is currently delivered; A record of the Focus Group members’ perceptions of the present status of the Program and future trends that may affect the program; Recommendations for Program updates/revision; Feedback on the Focus Group process followed.

Recommendations for the Program

PQR 2008 – 2009: Guide For Team Leaders SPTS

71

FOCUS GROUP MEETING CHECKLIST Name of Focus Group: EXTERNAL STAKEHOLDERS List of Participants Name and Organization

Contact Information (telephone no./ Email / address)

RSVP (attending Y/N) Alternate required?

PQR 2008 – 2009: Guide For Team Leaders SPTS

72

FOCUS GROUP MEETING CHECKLIST Name of Focus Group: Date of Focus Group Meeting: Timeline 1 month before 1 month before 3 weeks before Actual Date Task TL and Chair to meet to establish guest list Book room for session Send out invitations to participants Order Refreshments Follow up with those who have not responded. Arrange for alternates if necessary. Meet with Focus Group Leader to plan agenda items Gather items for meeting: Name tags Masking tape Flip chart paper Markers Agenda Any other handouts Paper and pen for recorder Meet with Focus Group Leader to review draft report Final report received by Program Team leader and Program Chair.

3 weeks before 2 weeks before

1 week before

1 week after 2 weeks after

PQR 2008 – 2009: Guide For Team Leaders SPTS

73

STUDENT FOCUS GROUPS

8

In this section
   Planning external stakeholder focus groups Possible items for discussion Sample documents

PQR 2008 – 2009: Guide For Team Leaders SPTS

74

PLANNING STUDENT FOCUS GROUPS
Team leaders will facilitate the student focus group for another program other than their own. The focus group leader will prepare the report for the session which s/he has facilitated, using the transcriptions of the flip chart notes as the basis for the report. The focus group leader, the program chair and the team leader will need to meet to review the report before it is finalized.

Time Frame for the Session
The working part of the session should be about two hours in length. Refreshments should be available at the beginning of the session to minimize interruptions once the session gets going.

Leadership of the Session
The chair of the department should introduce the focus group leader at the beginning of the session, explain the purpose of the session and invite the participants to introduce themselves. The focus group leader should then review the Agenda, and explain the process and ground rules.

Who to Invite
It is recommended that 12-20 students be invited to the student focus group session, depending on the size of the program. There should be fairly equal representation from all years and all sections/groups. Students can be invited by telephone, email or letter, ensuring that they understand that it is expected that they respond to the invitation and that an alternate will be found if the student is unable to attend. Students can be selected randomly from a class list – i.e. every fourth or fifth student, depending on the size of the class. The ground rules for the discussion are:       Ask that only one person speak at a time. Those attending are asked to be as concise and focused as possible. All comments/perceptions should be freely stated and will be recorded on flip charts by the focus group leader. All contributions are regarded as valid and no consensus is required with respect to the perceptions and feedback stated and recorded. The focus group leader will attempt to achieve consensus only with respect to final recommendations to be made to the program. This is an opportunity to discuss global issues regarding the program. Any students who have individual issues regarding the faculty are invited to make an appointment to discuss them with the program chair.

PQR 2008 – 2009: Guide For Team Leaders SPTS

75

All elements of the discussion should be recorded on flip charts during the session. The focus group leader may enlist the help of an external recorder, if one is available in the department. The flip charts will serve as a permanent record of the discussion and will provide the basis for the Focus Group Report.

PQR 2008 – 2009: Guide For Team Leaders SPTS

76

POSSIBLE ITEMS FOR DISCUSSION
List of potential topics and Issues for discussion with some suggested questions.

Students and Graduates
The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program. Did you know what you needed to know to get into the program? Did you have enough of a Mathematics foundation to succeed in level one? Did you know anyone who wanted to get into the same program as you but didn’t – and why? Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them. Does the course content match what you expected to learn? Do you expect these studies will lead to the type of career you are seeking? Were you prepared for the financial and workload commitments needed? Processes for granting of internal and external credits are clearly defined and available to students. Did you know how to get any exemptions for courses taken at another post secondary institution or here at Algonquin College in another program? Students indicate that they are satisfied with the program. What parts of the program provide you with the most satisfaction? What parts of the program provide the least satisfaction? Students indicate that they have input into decisions regarding their academic life. Do you feel that you have input into decisions regarding your studies? Issues raised at Program Councils are addressed in a timely fashion and feedback is provided to the council. If students keep the minutes of the program council meetings, often feedback is not distributed. Students also need to understand that some items may take longer to address than others but that the item has been forwarded for action - or not. Learners progress through the program, achieve program outcomes and graduate in a timely fashion. Do you feel that the learning outcomes are achievable in the designated time frame for the program? Have you been successful in all your courses so far? If not, why not? Learners with a wide range of abilities demonstrate the expected learning outcomes. Do any students present have any input they would like to offer regarding the learning activities and how students with different learning abilities and learning styles are able to demonstrate their learning?

PQR 2008 – 2009: Guide For Team Leaders SPTS

77

Curriculum
There is congruency between the course learning requirements and the program learning outcomes. Do you think that each of the courses contribute to the overall learning outcomes? There is a match between the course learning outcomes, course learning activities and learning resources. Do the learning activities and learning resources support your success in demonstrating the course learning outcomes? Courses are organized and sequenced to support student learning, are allocated appropriate hours of study and are consistent with published prerequisites. Does the sequencing of the courses make sense? Do you have the foundational knowledge needed as you progress in the program? Prerequisites do not unnecessarily hinder progress in the program. Are there any courses that seem to obstruct student progress in the program? If so, do you have any suggestions about how this could be improved? Curriculum design maximizes flexibility of student learning. Do you feel that you have adequate flexibility to accommodate your learning? Do you have any suggestions regarding how the program delivery could be more flexible to support your learning? (What do the students consider to be needed flexibility – courses available at different times to allow them flexible schedules; different learning offerings, i.e. online as well as in class; multiple resources available, a variety of learning activities; opportunities for PLA challenges; laddering of curriculum).

Instruction and Evaluation
The program provides experiential learning opportunities that allow the learners to perform, with support, as a practitioner in their field. Do you feel you are provided with adequate opportunities to perform as a practitioner in your field? There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities. Do the instruction methods support your learning style and your needs and abilities? Do you have any suggestions for improving this aspect of the program delivery? Learning methods are published and are matched to the learning outcomes. Did you know what learning methods to expect and do you feel they match the learning outcomes? Learners are provided the skills necessary to be successful with the learning strategies selected. Do you receive adequate preparation to be successful with the learning strategies being used, for example, online learning or group presentations?

PQR 2008 – 2009: Guide For Team Leaders SPTS

78

Evaluation criteria are published and students are aware of how and when they are going to be evaluated. (This criterion may be adequately answered in the student surveys.) Students indicate that feedback is timely and allows them to build on their learning. (This criterion may be adequately answered in the student surveys.) Students perceive evaluation to be fair. (This criterion may be adequately answered in the student surveys.) Evaluation methods allow students to demonstrate the course learning outcomes. Do the ways the professors evaluate you allow you to demonstrate the course learning requirements? Are there other ways that may better allow you to demonstrate your learning? Student workload and assessment is balanced across the term at both the course and program level. Do you find the workload to be balanced across the term for all your courses? There is a range of evaluation methods used consistent with a variety of learning styles. Do the evaluation methods used match your learning style? Can you offer suggestions about other evaluation methods that would suit you better?

Resources
Students consider faculty to be available. (There may be adequate data in the student surveys.) Students consider faculty to be adequately prepared. (This criterion may be adequately answered in the student surveys.) Students are assigned an academic advisor. Do you know who your academic advisor is? Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is. Have you two met this term or connected in another way? Do you have any suggestions about how your academic advisor can better support your learning? Labs, clinical facilities and placement facilities are complementary to and integrated into the program and allow the learner to demonstrate the learning outcomes. Do the labs, clinical and placement facilities allow you to demonstrate the learning outcomes? Students indicate that there are adequate and accessible learning resource materials including: textbooks in the bookstore; online materials; print resources; equipment and student support services, to allow them to be successful.

PQR 2008 – 2009: Guide For Team Leaders SPTS

79

(This criterion may be adequately answered in the student surveys.) The learning environment is safe. Do you feel safe in all learning environments? Do you have adequate supervision when demonstrating new skills in a placement environment? If there is an area where students do not feel safe, do they have suggestions about how this could be resolved? The students are provided with the information they need to know how to function safely both in the College and workplace learning environments. Do you feel that you are well orientated to practice lab and workplace learning environments? Are you offered specialized safety training? Can you identify any situations where you felt you were not able to practice your skills in a safe manner?

PQR 2008 – 2009: Guide For Team Leaders SPTS

80

Student Focus Group AGENDA Conduct of the student focus group sessions should be standardized as much as possible for all programs. The Agenda for each session will closely resemble the following outline.

Introduction       Focus Group Leader Participants Objectives of the session Process/ground rules Tasks/activities Anticipated outcomes

Topics for discussion are listed below under the following headings     Students/Graduates Curriculum Instruction/Evaluation Resources

Recommendations for the Program

PQR 2008 – 2009: Guide For Team Leaders SPTS

81

Student Focus Group Report

1.

Purpose and Objectives of the Focus Group The Focus Group sessions have been convened to gather feedback from our clients. The sessions are an important component of the College’s Program Quality Review process. The Focus Group Sessions are intended to:   Generate feedback from students about their experience with the College and their perception about the education they are receiving; Identify factors students feel impact the learning environment; Format for the session An active process was used to generate as much feedback abut a range of program-related issues as possible within the time allowed. A full two hours was allowed. The session was facilitated by one the Program Quality Review focus group leaders.

2.

3.     4. 5. 6.

Anticipated outcomes Evaluative feedback about the program as it is currently delivered; A record of the focus group members’ perceptions of the present status of the program and future trends that may affect the program; Recommendations for program updates/revision; Feedback on the focus group process followed. Agenda Highlights of the Focus Group Session. Recommendations

PQR 2008 – 2009: Guide For Team Leaders SPTS

82

Purpose and Objectives The focus group sessions have been convened to gather feedback from our clients. The sessions are an important component of the College’s Program Quality Review process. Objective of the Student Focus Group Sessions Focus group sessions are intended to:   Generate feedback from students about their experience with the College and their perception about the education they are receiving; Identify factors students feel impact the learning environment;

Format for the Session An active process will be used to generate as much feedback about a range of program-related issues as possible within the time allowed. A full two hours will be required. The session will be facilitated by the focus group leader. Anticipated Outcomes of the Session The Focus Group Session is expected to provide:     Evaluative feedback about the program as it is currently delivered; A record of the focus group members’ perceptions of the present status of the program and future trends that may affect the program; Recommendations for program updates/revision; Feedback on the focus group process followed.

Recommendations for the Program

PQR 2008 – 2009: Guide For Team Leaders SPTS

83

FOCUS GROUP MEETING PARTICIPANTS CHECKLIST Name of Focus Group: STUDENT FOCUS GROUP List of Participants Year or Academic Level

Contact Information (telephone no./ Email / address)

RSVP (attending Y/N) Alternate required?

PQR 2008 – 2009: Guide For Team Leaders SPTS

84

FOCUS GROUP MEETING CHECKLIST Name of Focus Group: STUDENT FOCUS Date of Focus Group Meeting Timeline 1 month before 1 month before 3 weeks before Actual Date Task TL and Chair to meet to establish guest list Book room for session Send out invitations to participants Order Refreshments Follow up with those who have not responded. Arrange for alternates if necessary. Meet with Focus Group Leader to plan agenda items Gather items for meeting: Name tags Masking tape Flip chart paper Markers Agenda Any other handouts Paper and pen for recorder Meet with Focus Group Leader to review draft report. Final report to Program Team leader and Chair

3 weeks before 2 weeks before

1 week before

1 week after 2 weeks after

PQR 2008 – 2009: Guide For Team Leaders SPTS

85

Sample Thank You Letter from Program Chairs to Focus Group Members

Address Date Dear xxxx I am happy to take this opportunity to thank you for your recent participation in our focus group session as part of our Program Quality Review process. Your contribution to our process for improving this program is appreciated. I have included a copy of the Focus Group Report for your convenience. A copy of the final report will be available at our first meeting in the Fall Term.

Thank you for your ongoing support of our Program. Sincerely,

Program Chair

PQR 2008 – 2009: Guide For Team Leaders SPTS

86

APPENDIX A

A

In this appendix
 Final report template

PQR 2008 – 2009: Guide For Team Leaders SPTS

87

FINAL REPORT TEMPLATE

PROGRAM QUALITY REVIEW 2008-2009 Insert Program Name and Number

PQR 2008 – 2009: Guide For Team Leaders SPTS

88

Table of Contents

Executive Summary Introduction Conclusions Recommendations Implementation Plan Findings Appendices: Appendix A: Program Monograph Program Outcomes Program Delivery Information (if applicable) Curriculum Maps for Program Standards Course Outline Review Summary Appendix B: Faculty Focus Group Report Summary Faculty Self Audit Faculty Credential Summary Appendix C: External Stakeholders Focus Group Report Appendix D: Student Focus Group Report Appendix E: Statistical Data and Program Costing Data

PQR 2008 – 2009: Guide For Team Leaders SPTS

89

Program Quality Review 2008-2009 Name of Program Executive Summary The Executive Summary is a one-page, single-spaced review of the highlights of the Program Quality Review report. The focus of this summary is to highlight the overall status of the program, the special strengths and achievements of the program, as well as any challenges the program is facing and the strategies recommended to address these challenges. Four paragraphs can cover the information needed: a brief history of the program and delivery information; strengths and highlights; challenges; and future. For example (this is fictitious but uses some real situations from recent reports.)
The Housemaking Program was first offered at Algonquin College in 1981 as a one-year certificate. In 1992, a second year was added to the program to meet the demands for more highly skilled and knowledgeable workers, and in 1997, the delivery mode was compressed into a non-semestered diploma program format. The program prepares graduates to enter the housebuilding industry in the Ottawa and surrounding area. The program is also offered at the Perth Campus. There has been a consistently high demand for the program and its graduates except for a minor slump in the housing depression in the early 90’s. There are a number of local builders who have supported the program over the last 30 years and our graduate hiring rate is consistently above 90%. The KPI and Course Assessment Surveys note high satisfaction rates with respect to the dedication and expertise of the program faculty. This commitment from both full-time and part-time faculty members has ensured the program has maintained a high standard of delivery. Two major concerns will be addressed in the implementation plan. The first is the need for the students to be ready to enter the job market in the spring rather than the fall. The faculty will review the delivery mode over the next year and determine ways to deliver the curriculum to meet the demands of the students to move quickly through the program yet be available at the start of the construction season. The other most significant concern is the expectation to lose the coordinator and the other most senior full-time professor in the next year as both of these faculty members are eligible to retire. Succession planning will be a high priority to ensure we have excellent teaching and curriculum development skills in the department along with our current practical expertise. A review of course outlines indicates that some outlines are not as complete as would be desired. A plan is in place to ensure this will be resolved in the May-June planning session. This program has enjoyed a high demand, and high employment rates. Industry projections in the Ottawa area suggest that this demand will continue. Advice from our industry partners on an annual basis will assist us in keeping this program strong over the next few years.

PQR 2008 – 2009: Guide For Team Leaders SPTS

90

Introduction The introduction section describes the program as it existed at the beginning of the review process. This part of the report gives more information about the program: its purpose, its evolution from its first inception to today, relevant external accreditation organizations linked to the program, number of faculty, both full-time and part-time. You can also include the nature of the student body and special contributions that the program offers in the community. Also discuss variations in delivery like co-op options, campus options, Winter intakes, multiple intakes, etc. Finally include a discussion of the use of classroom, labs, computer labs, placement opportunities, field trips, etc.

PQR 2008 – 2009: Guide For Team Leaders SPTS

91

Conclusions This part of the report addresses the analysis of the data. Discuss the review of the statistics and the identification of agenda items for the focus groups. A brief paragraph can be dedicated to highlights of each of the three focus groups. The last half or third of this section speaks to the evaluation of the various recommendations from each of the focus groups and the decisions taken for the formulation of the implementation plan. For example, the students may suggest that the computer applications course is irrelevant and they don’t like the hybrid courses. The external stakeholders provide feedback that the graduates are not computer literate. The final recommendation would probably indicate the need to make the computer applications course relevant to the industry by adding inventory control and invoicing software exercises, and communication to the students that computer skills are desirable in the workplace. To address the students’ dislike of hybrid courses, a recommendation could identify the development of better orientation to Blackboard for both faculty and students. (A reminder of the importance of the feedback that should be provided to the Focus Group members, letting them know the final resolution.) Some recommendations may arise that will not be feasible to implement and therefore will not be addressed in the final recommendations. These need to be discussed here. For example, one group suggested that students have a university level course as a perquisite for admission to the program, given the students’ need for a strong science background. Given that this is not allowable by Ministry standards and not feasible, the recommendation might be modified to ensure that highest Grade 12 College course is listed as a required course, or if this is already the case, the recommendation may not warrant any action. If the students have the high school subject at the grade twelve level, there may need to be a college level course in level one to help the students with the material. This should be explained here, i.e. recommendations from the Focus Groups should not just disappear without explanation and feedback. Some recommendations will be desirable to implement but may have budget constraints. These recommendations can be maintained but identified as having fiscal constraints. Sometimes recommendations may be carried forward until such time that the funds are available or it is determined that a less costly solution is needed and implemented.

PQR 2008 – 2009: Guide For Team Leaders SPTS

92

Recommendations and Implementation Plan Recommendations must be measurable and should have start and end dates for completion. Each recommendation needs to be broken down to specific tasks and each task needs to be assigned. Reasonable completion dates need to be established. This plan will be reviewed annually until all items are addressed. Recommendation Action to be taken (Identify who is responsible for each action) Level 01 – professor Jones Level 02 – professor Smith Start Date Completion Date

All course outlines are completed and learning outcomes matched to learning outcomes and evaluation methodologies.

May 2009

June 2009

Equipment in the lab needs repairs and updating

All deficient/outdated or broken equipment will be reported to the Coordinator for development of a list of need equipment for next budget year. Repairs will be done as required Chair to consider budget implications for hiring fulltime staff or allocating planning and advising hours for part-time staff.

Immediate

List submitted to Chair in December 2008, ongoing as well.

Additional faculty are allocated to be available for academic advising and curriculum revision.

Fall 2008

PQR 2008 – 2009: Guide For Team Leaders SPTS

93

Findings
EVALUATION OF THE PROGRAM AGAINST THE CRITERIA Program Quality Review Criteria

YES

NO

Criterion 1. Admission, credit for prior learning, promotion, graduation, and other related academic policies support program development and student achievement of program learning outcomes.

1. The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program.

2. Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them.

3. Students know how to get internal and external credits and recognition for prior learning.

4. Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.

5. Students know how they will be evaluated.

6. Students indicate the learning requirements are relevant and meaningful.

PQR 2008 – 2009: Guide For Team Leaders SPTS

94

YES 7. Students indicate that assessment methods relate to the learning requirements.

NO

Criterion 2. Programs conform to the Framework for Programs of Instruction and the Credentials Framework, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance.

1. The duration and structure of the program are consistent with the program learning outcomes and the credential offered.

2. Appropriate credits are allocated for each component of the program, and transfer and laddering options are stated.

3. Prerequisites do not unnecessarily hinder progress in the program.

4. Program learning outcomes are consistent with the credential granted, the title of the credential awarded, the provincial program standards (where these apply), and the minimum essential expectations of the workplace.

5. Program learning outcomes are reflected in course outlines.

6. Programs learning outcomes are used in prior learning assessment.

7. Changes to courses and program outcomes are introduced on a timely basis and are designed to maintain the relevance of the program.

8. The program has established articulation agreements.

PQR 2008 – 2009: Guide For Team Leaders SPTS

95

YES 9. The program conforms to the College policy for the number of English courses.

NO

10. All curriculum documentation is up-to-date including course outlines and the program monograph information.

11. There is congruency between the course learning requirements and the program learning outcomes.

12. There is a match between course learning requirements, course learning activities and learning resources.

13. Concepts of environmental sustainability are embedded in the program curriculum.

14. Students have opportunities to develop the skills, knowledge and attitudes necessary to succeed in the global economy.

Criterion 3. Methods of program delivery and student evaluation are consistent with the program learning outcomes.

1. Program delivery, including that which takes place off-site, is consistent with the nature of the program, the learning outcomes, and the needs of the students.

2. There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities.

PQR 2008 – 2009: Guide For Team Leaders SPTS

96

YES 3. Learning methods are published and are matched to the learning outcomes.

NO

4. College designated targets regarding hybrid courses are met.

5. Learners are provided the skills necessary to be successful with the learning strategies selected.

6. Evaluation criteria are published and students are aware of how and when they are going to be evaluated.

7. There is a match between course learning requirements and evaluation methods, i.e., evaluation methods allow students to demonstrate the course learning requirements

8. Evaluation methods are valid and reliable.

9. Students indicate that feedback is timely and allows them to build on their learning.

10. Students perceive evaluation to be fair.

11. Resubmissions, supplementals, and appeals are published, appropriate, fair, valid, and consistent.

12. Student workload and assessment is balanced across the term at both the course and program level.

13. There is a range of evaluation methods used consistent with a variety of learning styles.

PQR 2008 – 2009: Guide For Team Leaders SPTS

97

YES

NO

14. Learners can earn credit for up to 75% of the program’s courses, using the PLA process.

15. Academic policies and practices that provide for the development and continuous improvement of teaching and learning methods are valued, documented, and supported.

16. Graduate capabilities, including knowledge, skills, and attitudes are consistent with program outcomes.

Criterion 4. Human, physical, financial, and support resources to support student achievement of program learning outcomes are available and accessible.

1. The program faculty members, as a whole have adequate academic preparation and workplace experience to deliver a quality program.

2. Faculty members are evaluated every three years.

3. Faculty members engage in professional development activities that ensure they are current in their field and developing teaching expertise.

4. Students consider faculty to be available.

5. Students consider faculty to be adequately prepared for class.

6. All students are assigned an academic advisor

PQR 2008 – 2009: Guide For Team Leaders SPTS

98

YES

NO

7. Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is.

8. Labs, clinical facilities and placement facilities are complementary to and integrated into the program and allow the learner to demonstrate the learning outcomes.

9. Students indicate that there are adequate and accessible learning resource materials including: textbooks in the bookstore; online materials; print resources; equipment and student support services, to allow them to be successful.

10. The program is financially viable. The demand for the program has been sustained for the last five years. There is a future demand for graduates of this program.

11. The learning environment is safe.

12. The students are provided with the information they need to know to function safely in both the College and workplace learning environments.

Criterion 5. Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.

1. Students indicate that they are satisfied with the program.

2. Issues raised at Program Councils are addressed in a timely fashion and

PQR 2008 – 2009: Guide For Team Leaders SPTS

99

YES feedback is provided to the Council.

NO

3. Learners progress through the program, achieve program outcomes and graduate in a timely fashion.

4. Learners with a wide range of abilities demonstrate the expected learning outcomes.

5. Graduates are satisfied with the overall program experience.

6. Graduates are obtaining employment in their fields.

7. Graduates are successful in obtaining external licenses or credentials where relevant.

8. Employers are satisfied with graduate performance.

PQR 2008 – 2009: Guide For Team Leaders SPTS

100

Appendices: The following Appendices are added to the final report:

Appendix A: Program Monograph Program Outcomes Program Delivery Information (if applicable) Curriculum Maps for Program Standards Course Outline Review Summary Appendix B: Faculty Focus Group Report Summary Faculty Self Audit Faculty Credential Summary Appendix C: External Stakeholders Focus Group Report Appendix D: Student Focus Group Report Appendix E: Statistical Data and Program Costing Data

PQR 2008 – 2009: Guide For Team Leaders SPTS

101

APPENDIX B

B

In this appendix
   Roles of program quality review participants Annual program audit diagram Program mix review diagram

PQR 2008 – 2009: Guide For Team Leaders SPTS

102

ROLES OF PROGRAM QUALITY REVIEW PARTICIPANTS
Program Quality Assurance Administrator
Program Quality Review is coordinated by the Program Quality Assurance Administrator, who reports to the Executive Dean, Academic Development, who in turn is responsible to the Vice President Academic. The responsibilities of the Program Quality Assurance Administrator include        Consult with major academic constituencies College-wide regarding Program Quality Review. Prepare documents related to Program Quality Review. Plan and organize training programs for team leaders. Support the Program Quality Review Process in programs. Meet regularly with the team leaders. Facilitate evaluation of the Program Quality Review process. Ensure the College is compliant with Ministry requirements regarding Quality Assurance.

Program Team Leader
The program team leader is a faculty member who is assigned to conduct Program Quality Review for the program in which he/she teaches. The responsibilities of the team leader include           Attend all meetings called by the Program Quality Assurance Administrator. Attend all Training Sessions related to the team leader role. Convene meetings of the program faculty team. Convene a Student Focus Group. Convene an External Stakeholders Focus Group. Facilitate Student and External Stakeholder Focus Groups for a program from another School and submit reports for these sessions. Report to the program chair frequently on the progress of the Program Quality Review. Analyze data and incorporate the results of the analysis in the final report.Evaluate the Program Quality Review process. Submit the final report to the program chair, dean, PQA Administrator and the Vice President Academic.

PQR 2008 – 2009: Guide For Team Leaders SPTS

103

Deans / Executive Deans
The responsibilities of the Deans/Executive Deans include      Identify programs for Program Quality Review. Support the Program Quality Review process. Review the Program Quality Review Report. Follow up regarding results and recommendations included in the report. Report to the Vice President Academic on the progress of Program Quality Review and of follow-up on the recommendations.

Chairs / Deans
The responsibilities of the Chairs include            Identify team leaders (TL) each spring for programs identified to be in the next cycle. Meet with each TL in May and discuss the upcoming process, encourage and facilitate attendance at the Aligning and Building Curriculum Conference in the upcoming Fall and Spring. Attend the information session for Chairs. Provide release time as outlined in the PQR process Meet with the TL in mid January to review the stats and original assessment of the criteria. Assist with agenda setting for focus groups and to create list of those to invite. Provide secretarial assistance for sending out invitations and documents for the focus groups and for putting the report together, using the templates in the TL Guide. Meet regularly with the TL and monitor progress. Attend focus group sessions and introduce the group members to the process. Provide summary report for faculty credentials and experience. Provide feedback regarding the process. Collaborate in the development of the final recommendations and implementation plan, ensuring adequate budgetary support for those recommendations that are to be implemented immediately (i.e. within one year) and consideration of those recommendations for which funds are not currently available. Include recommendations in Performance Contract. Facilitate the implementation of the implementation plan. Ensure feedback is provided to all those who participated in the focus groups. Meet with TL after one year to review progress and assist in preparing the submission of the Follow - up Report.

   

PQR 2008 – 2009: Guide For Team Leaders SPTS

104

ANNUAL CURRICULUM REVIEW PROCESS

Curriculum Rollover by the Curriculum Administrator

Input from Advisory Committee

Review of Curriculum by Program Faculty

Input from Program Council o Program Council o Review Surveys

Updates to Curriculum Identified

Updates to Program of Study and Program Narrative entered into GeneSIS

Changes reviewed by Curriculum Administrator and approved

o Program Description o Admission Requirement o Fees/Expenses o POS o Course Description

o o o o

New program version Activated for: course loading fees loading registration monograph production

PQR 2008 – 2009: Guide For Team Leaders SPTS

105

PROGRAM MIX REVIEW DIAGRAM

Quality Index Measures

Review of Program Mix Data

Program Costing

Review by VPA, Executive Deans and Chairs

Programs with <25% contribution or <70% Quality Index Identified

Strategies to Improve Quality Index / Program Costing Deficiencies

PQR 2008 – 2009: Guide For Team Leaders SPTS

106


				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:501
posted:11/28/2009
language:English
pages:110