Using Institutional Effectiveness in Business/Administrative Areas by HC120913232548


									    Using Program Review to Facilitate

Mississippi Association for Institutional Research
                March 31, 2005
             Terri M. Manning, EdD
     Central Piedmont Community College
Some history….
 CPCC is the largest community college in
  both the Carolinas with approximately 60,000
 We have over 100 instructional program
  areas including for-credit, continuing
  education and literacy
 We have almost 50 administrative and
  student affairs units
Some history….
 When we started this process (1998), the College
  had done nothing truly “IE related” since the last
  SACS visit in 1992 when they received a
  recommendation for every must statement in section
 The IR office staff had rolled over and all institutional
  memory was gone.
 We had less than 5 years to put every program and
  unit through a review process.
Under the New Core Requirements
and Comprehensive Standards
 Core Requirement 2.5 states “The institution engages
  in ongoing, integrated, and institution-wide research-
  based planning and evaluation processes that
  incorporate a systematic review of programs and
  services that (a) results in continuing improvement, and
  (b) demonstrates that the institution is effectively
  accomplishing its mission.”
 Comprehensive Standard 3.3.1 states “The institution
  identifies expected outcomes for its educational
  programs and its administrative and educational support
  services; assesses whether it achieves these outcomes;
  and provides evidence of improvement based on
  analysis of those results.”
 IE at Central Piedmont

 CPCC established an IE committee in 1998
  and an IE website in 2000
 Committee members were from various
  instructional departments, student services
  and administrative/business offices
 The committee established an IE plan for the
Visual of the IE Plan
 Four Major Overlapping Pieces of the
 IE Plan
                               Annual goal setting and
                                follow-up required by all units
                               Program/unit review on a 3-5
                 Program/       year cycle
                Unit Review
Assessment                     College assessment plan
                                involving surveys,
                                assessment and data
                  College      The evaluation of general
Annual Goal
Setting Cycle
                Assessment      education
Developing the Process
 We wanted to create a meaningful
  program/unit review process
 We wanted programs to complete the
  process having learned something valuable
  (not a document to set on a shelf)
 We wanted the process to be “outcome-
  based” to stand the test of time
Developing the Process
 The first review process was created for
  instructional programs (main focus at most
 During the 1998-99 year, 18 programs were
  reviewed (approximately 100 programs over
  five years)
 The perception at the beginning was that “this
  is just another academic exercise”
 But the results were very different than any
  review process previously completed
    Organizational Stages of
    Outcome Evaluation                                  Stage 5
                                                 Acceptance & adaptation
                                                 Challenge & competition
                                    Stage 4      Catalyst - Proactive

                                  Depression - exhaustion
                                  Compliance - Passive reactive
                        Stage 3
                     Bargaining - no time/no money
                     Seek outside sources

          Stage 2
       Anger and antagonism
       Resistant & Reactive

  Stage 1
Disbelief & Denial
Paralysis - Passive resistance
 Instructional Program Review

 The review process contained five sections:
     I.     The Program Profile
     II.    Program Content
     III.   Student Learning Outcomes
     IV.    Need for Change
     V.     Future Issues
  Instructional Program Review
 Tasks accomplished through review:
     Defined their program
    Detailed faculty and staff (credentials, prof. dev,
    Unit mission or purpose
    Link unit mission to the college mission
    Defined the content of their program
    Defined the population served
    Determined student learning outcomes specific to their program
    Set outcome objectives
    Performed some means of assessment
    Analyzed results
    Determined strengths and weaknesses
    Created strategies for improvement
    Determined future needs (including curricular change, needed
      resources, staff and space)
 All units had to complete a one-year follow-up
Identifying Outcomes
 We focused on two types of measures:

     Instructional Outcomes
          Program Outcomes
          Student Learning Outcomes
     Administrative Outcomes (outcomes for
      administrative programs)
 Administrative Outcomes

 Many units do not directly serve students or they
  want results within their units that are not truly
  student outcomes.
 They want to improve services or approach an
  old problem in a new way.
 They want to become more efficient and
 They will set administrative outcomes.
My Administrative Outcome Objectives

1. 80% of faculty/staff responding to the faculty/staff survey
   will perceive that Planning and Research responds
   quickly to their requests for data.
2. 80% of faculty/staff responding to the survey will perceive
   that Planning and Research makes a significant
   contribution to the College.
3. 80% of faculty/staff responding to the survey will perceive
   that Planning and Research contributes to the
   effectiveness of CPCC.
4. 80% of faculty/staff responding to the survey will indicate
   that Planning and Research produces enough reports to
   meet the planning and information needs of faculty and
Administrative Outcomes for
 Objectives set for the program (has nothing to
  do with students outcomes)
     Example
         to recruit one new faculty member
         to seek and gain accreditation
         to increase retention by 10%
         to send each faculty member to at least once
          professional conference per year
         to gain funding for an innovative program through
          a grant proposal

 "Outcomes" are benefits for people: changes in
  knowledge, values, position, skills, behavior or status.
  More simply stated, outcomes are typically what service
  providers hope recipients achieve once they complete a
  program or receive services. This is not the “what” but the
  “why” of education.
Types of Outcomes
 Student learning outcomes are outcomes
  related to the learning that takes place in the
  classroom. We measure improvements in
  writing, speaking, understanding the scientific
  method, etc.
 Program outcomes are the benefits to a
  student who receives an associate degree in
  Nursing or completes a certificate in Network
  Administration? Typical outcomes might be
  licensure exam scores, job placement, etc.
 Outcome objectives are just objectives that
  relate to the identified outcomes.
  Program Outcome Model

Resources      Services              Products or Results of

Staff          Education (classes)   Numbers served
Buildings      Services              FTE (input next year)
Facilities     Counseling            Classes taught
State funds    Student activities    Students recruited

State regulations
Program Outcomes Model


                                Benefits for People

                                *New knowledge
                                *Increased skills
                                *Changes in values
                                *Modified behavior
                                *Improved condition
                                *Altered status
 Instructional Program Review

 Rules for the process included:
    Involvement on the part of all faculty in the
     department under review (not just one person). It is
     recommended that the program begin with a brief
     faculty retreat to discuss and divide tasks.
    All programs must use at least one external
     committee (advisory groups are fine) to provide
     feedback to programs.
    All programs must utilize feedback from students.
During the first few years…
 Planning and Research conducted training on
  “how to conduct program review” with each
  unit scheduled to go up for review that year.
 A staff member from Planning and Research
  served as a liaison to each program being
  reviewed to help them with the process.
 What We Learned….

 The first group was dragged kicking and
  screaming through the process
 Faculty had very little time for the details
 Faculty had trouble identifying “student
  learning outcomes”
 The results produced great marketing
 Once they saw the results, faculty embraced
  the process
What Happened

 Deans used the results to make a case for
 The administration became interested in what
  was learned through the review process
 Instruction created a position to deal mainly
  with program review and IE issues within
 Faculty knew their programs were working
IE Committee Decision

 The instructional program review process was
 The review of instructional units was important –
  but administrative/student services units helped
  create an environment conducive to learning at
  the college and supported the learning process
 Administrative units should go through a similar
IE Committee Decision, cont.
 Create a committee to draft a similar process for
  the administrative and students services areas of
  the college
 Administrative units were spread across three VP
 Two representative from each VP area
  participated in drafting the new review process
 We spent approximately six months creating a
  workable process
The Administrative Unit Review Design

I. The Unit/Program Profile
   A. The Mission/Purpose
       1.    Role unit plays in the college mission
       2.    Unit/program goals as they relate to the
             college’s mission
   B. The Staff
       1.    Professional and administrative staff (b-e
                    since the last review)
             a.     Position description/duties
             b.     Credentials (full and part-time, if any)
             c.     Accomplishments (if applicable)
             d.     Service to college, community and
The Administrative Unit Review
B. The Staff (continued)
         e.      Professional development activities
   2.    Classified Staff
         a.      List of names and positions
         b.      List of required credentials (if any)
C. The Customer/Client Served
   1.    Breakdown of students/faculty or staff by
         type or demographic information
         (thorough explanation of who is served)
  The Administrative Unit Review
II.   Definition of Services or Program
      A.      Definition of day-to-day duties of the unit
      B.      Innovations, new projects, new initiatives,
              local, state-wide or national efforts
      C.      Required functions of unit (description and
                      status of compliance)
              1.      SACS requirements
              2.      State mandates
              3.      Federal mandates
              4.      Other
 The Administrative Unit Review
III.Administrative Objectives and Student Outcomes
    (where appropriate)
       A.      Administrative objectives (2-3 objectives)
       B.      Outcomes (or status if incomplete) of
               innovations, new projects, new initiatives,
               local, state or national efforts
       C.      Assessment explanation (what was
               assessed, who, when, how many)
       D.      Results of administrative objectives
               based on assessment
Assessment of Administrative Units

 During the year of program review, the
  “Annual Faculty/Staff Survey” contained
  questions from those units being reviewed.
 Results were given to each unit and broken
  out by campus and job type
    Click on “survey results”
  The Administrative Unit Review
IV.   Need for Change
      A.    Strengths identified by external sources,
            faculty, staff and students
      B.    Weaknesses identified by external
            sources, faculty, staff and students
      C.    Recommendations by faculty, staff,
            external sources and students to improve
            the unit's services and programs
      D.    Strategies for change (based on input
            from A, B & C above) - closing the loop
      E.    A one-year follow-up brief report to the
            Unit VP reporting on the progress of D
            above (due April 15, the year following
 The Administrative Unit Review
V.   Future Issues - Resources needed for future efforts
     A.    Market trends within the broad service unit or
           program area (based on best-practices, the
           literature or training received)
     B.    Anticipated future changes and needs
           (based on market trends)
     C.    Resources, equipment, space, staffing and
           work load changes needs for future growth or
     D.    Future plans of unit
Assistance from the Website

 The IE website:
     Explanation of IE process
     Templates and forms for review
     “Solid” examples for clarification
     Schedule for review
    Unit Review – Lessons Learned

    To do this well you need several critical pieces:
    1. Support from the top (President or Chancellor,
       Vice Presidents or Vice Chancellors)
    2. Buy in from the grassroots level (participation in
       the development of the process)
    3. Across-the-college participation (no one is
    4. Technology to make it easy (web page and
       review templates)
    5. Technical support from institutional research
Administrative Response
 Units spend a lot of time working on the
  program review.
 If we want them to take it seriously, we have
  to take it seriously.
 Once reviews are reported at the end of the
  year, their Dean or Vice President/Chancellor
  need to:
     1. Read them
     2. Respond to them
Appropriate Administrative Response

 Sit down with the unit (group meeting)
 Give them your attention
 Discuss what was learned
 Discuss their major issues
 Discuss what they need to make
 Actually attempt to direct resources to them to
  make those improvements
 Then they do not see it as an academic
Overall Benefits
 #1 No recommendations in the area of Institutional
  Effectiveness from SACS during the October 2002 re-
  accreditation visit
 The college became change-oriented
      Units had to define strategies for change
      Didn’t have to be perfect but rather making continuous
      Strategies for change helped identify needed resources for
      Units had to “close the loop” with the one-year follow-up
       (couldn’t promise and not deliver)
      Once they started, we couldn’t get them to stop doing it.
 Overall Benefits

 Units became empowered to perform their
  functions in an optimal manner and to ask for
  what they needed (no one noticed them before,
  now they do)
     Created data to support needs
     Accomplishments were reported to major
      administrative groups across the college
      (President’s Cabinet, Planning Council, IE
      Committee, etc.)
 Overall Benefits

 The college community understood the
  purpose and function of every unit
 Senior administration realized the benefits and
  became strong supporters
     Data are reported annually from program/unit
     VPs became stronger advocates for their units
      making changes to improve services
 Administrative Units Reviewed

Security                         Health and Safety Office
Human Resources                  Professional Development
Resource Development             The Foundation
EEO Office                       President’s Office
Facilities Management            Distribution Services
Facilities Design/Construction    Planning and Research
Financial Services               Payroll
Budgeting                        Financial Reporting
Purchasing                       Accounts Payable
Cashiering                       Audit and Compliance
Basic Skills Reporting           Inventory Control
Bookstore                        Campus Printing
Vending                          Information Technology
Admin. Computing Services         Marketing and Community
Students Affairs Units Reviewed
   Counseling and Advising
   Financial Aid/Veteran’s Affairs
   Career Services
   Disability Services
   “Campus-based” student services (6 campuses)
   Admissions and Records
   Registration Services
   The Library
   The Academic Learning Center (tutoring)
   Testing Center
   Graduation Office
   Student Life
Five Units

    Some Brief Results
An Instructional Program
 Workplace Basic Skills
     This program is a literacy initiative that goes
      directly into the worksite and teaches ESL
      classes, GED prep and GED classes.
 During their review, they surveyed both
  employers and students.
 This was the first time they had ever done
What They Learned

 Employers said:
     43.8% of employers reported increases in
      employee performance as a result of
      participation in the program.
     31.3% reported a reduction in absenteeism by
     87.5% said classes improved the morale of
      their employees
     37.5% said participants received raises
     50% said communication had improved.
What Students Said

  70.2% reported being able to fill out job forms better
  35.5% said they could now help their children with
   their homework
  91.1% said they felt better about themselves
  44.4% said they had received a raise, promotion or
   opportunity as a result of the courses
  86.3% said their ability to communicate in the
   workplace had improved
What Has Happened Since
 Their assessment data has shown up in their
  marketing brochures to employers.
 Their enrollment has grown dramatically.
 They have received funding and marketing
  support from Charlotte Reads (considered a
  model adult literacy program).
   Planning and Research

 Weaknesses Identified:
    Data/information needs to be disseminated to the grass roots
     level of the College (not just administration)
    Data needs to be made more accessible and easier to
 Strategy for Change:
    Make improvements in the department’s webpage
    Place more information on the webpage
 Result:
    Units are using data to make decisions
    Planning and Research has a heavily used website with user-
     friendly spreadsheets, tables and an online Fact Book. In-
     person requests for data have dramatically declined because
     faculty/staff are using the webpage
 Professional Development
 Weaknesses Identified:
    Instructors could not attend training sessions due to the
     times they were scheduled (top of the hour when
     classes were scheduled on the half hour)
 Strategy for Change:
    Starting times for all training sessions will be adjusted
     to begin on the half hour rather than the hour
 Results:
    More faculty were able to attend trainings and numbers
     increased in some areas
 The Bookstore

 Weaknesses Identified:
    Communication between faculty and staff and the
     bookstore management needs improvement
 Strategies for Change:
    A bookstore advisory committee has been formed. Action
     items identified will be followed up to insure that the
     bookstore is meeting the needs of faculty/staff
 Results:
      Decided to outsource the bookstore
      Worked with faculty to ensure an adequate number of
       books were available at the beginning of the term
      Created a process to move books from campus to campus
       to accommodate student needs
      Created more buy-back centers on the campuses to
       improve services
 Information Technology Services

 Weaknesses Identified:
    Better follow-up is needed on Help Desk requests
    Requests need to be answered faster
    Have knowledgeable people at the Help Desk
 Strategies for Change:
    Employ students with pertinent certification to man the Help
    Empower Help Desk staff with decision-making ability that
     helps provide the required level of service
 Results:
    Help Desk created a tracking system that has radically
     improved time from request to completion
Major Benefits for IR Office

 We created work for ourselves with great payoff.
 Each unit being reviewed had a liaison from IR to
  help them with surveys, objectives, etc.
 Now, everyone knows us.
 They think we are very helpful and a great asset to
  the institution.
 They call us for information and then listen to what
  we say.
 “Best” Results of “Best” Practice

 Better use of data across the college
 We have become more student/customer focused
 Review gives direction for goals and needed changes
 Departments are empowered to do their jobs (can’t slip
  through the cracks and be unnoticed)
 Problems must be resolved (there is no hiding and no
 Surveys provide needs assessment data as well as
  evaluation which gives departments direction
For a copy of this presentation
 Contact Terri Manning
 Download or print presentation:
     Click on “studies and reports”
     Listed as MAIR Program Review Presentation

To top