Project Report Effective Salesmanship - PowerPoint by lth18577


More Info
									The Nuts and Bolts of
Outcome Assessment

   Terri Manning, Ed.D.
   Center for Applied Research
   Central Piedmont Community College
Stages of Grief for Outcome Measurement
                                                   Stage 5
                                             Acceptance & adaptation
                                             Challenge & competition
                                  Stage 4    Catalyst - Proactive
                                 Compliance - Passive reactive
                    Stage 3
                  Bargaining - no time/no money
                  Seek outside sources
       Stage 2
      Anger and antagonism
      Resistant & Reactive
 Stage 1
Disbelief & Denial
Paralysis - Passive resistance
In Education…
   We’ve learned that things come and they go
   Most of these trends are purely academic
   The national educational climate is skeptical
    about accountability!
   They want us to prove that students are
    learning, that their lives are improving and
    that we are good stewards of funds!
The Spellings Commission
    Some things the draft report called for:
        The creation of an overall measurement of an institution's
         "bottom line," including measures of institutional costs and
         performance that let parents and policy makers view
         institutional results;
        A mandate that institutions measure student learning
         outcomes, disseminate the results to students, and report
         them publicly in the aggregate;
        The development of a national student unit-record database to
         follow the progress of each student;
        The establishment a national accreditation framework that
         includes comparable performance measures, and making the
         findings of reviews easily accessible to the public;
According to SACS, an Institution
Needs to Have in Place:
   An ongoing, integrated, and institution-wide research-based planning
    and evaluation processes which includes:
       a systematic review of institutional mission, goals and outcomes
       results in continuing improvement in institutional quality
       demonstrates that the institution is effectively accomplishing its mission
   A list of expected outcomes, the assessment of those outcomes and
    evidence of improvement based on analysis of those outcome results
    in each of the following areas:
       educational programs (student learning outcomes at the program and
        individual level)
       administrative support services
       educational support services
   Identified college-level general education competencies (based on
    best practices in assessment) and provide evidence that graduates
    have attained them
But Why?
   SACS Core Requirements and Comprehensive
    Standards are based on best practices of effective
    institutions (a peer developed and peer reviewed
   SACS staff members enforce the standards.
   But shouldn’t we periodically take a serious look at our
       Are they learning?
       Who is learning best?
       Are they achieving the outcomes we expected?
       Should we make changes in programs and services?
       Do we need more in-depth services?
       Do we need a new curriculum or a change in methodology?
The Great Fallacy
   Grades
       In this day of social promotion, grade inflation and
        different teaching/learning philosophies, grades
        tell you virtually nothing.
       They are not a measure of outcome achievement.
       Two teachers will grade a student differently for
        the exact same work.
       They cannot be used!
Why are We Moving from
Goals to Outcomes?
   Outcomes are program-specific
   They measure the effect of classroom
    activities and services provided.
   Outcomes represent a new way of thinking
   Outcomes have become widely accepted by
    our various publics
   They are here to stay
   We used to measure ourselves by our
Program Outcome Model
  INPUTS         ACTIVITIES            OUTPUTS

 Resources      Services               Products or Results of

 Staff          Education (classes)    Numbers served
 Buildings      Services               FTE (input next year)
 Facilities     Counseling             # Classes taught
 State funds    Student activities     # Students recruited

 State regulations                    United Way model
 Program Outcomes Model


                                    Benefits for People

                                    *New knowledge
                                    *Increased skills
(Outcomes answer the “so what”      *Changes in values
                                    *Modified behavior
                                    *Improved condition
                                    *Altered status
                                    *New opportunities
Which Is It?

              An input
              An activity
              An output
              An outcome
GED Preparation

    College provides well trained faculty.
    200 students complete their GED.
    Students move from public housing.
    30 courses are offered each semester.
    150 FTE are generated.
    Students’ reading level improves.
    Students are gainfully employed.
Student Services

    Four counselors are hired.
    Students successfully transfer courses
     and enter the university system.
    Students receive financial aid at the
    Students attend campus activities.
    Students are able to pay back their
     student loans.
General Education Courses

   Students receive creative classroom
   Faculty members receive a grant to offer
    multimedia opportunities to students.
   Students become active participants in
    County civic activities.
   700 FTE are generated in core courses.
   Students’ math skills improve.
Outcomes are ……...

   Driven by the mission.
   Related to overall program goals.
   Specific to the teachings/activities of your
   Determined by faculty and front-line staff.
   Measured carefully and specifically.
Inputs through Outcomes:
The Conceptual Chain



                                Initial                   OUTCOMES



Different Types of Outcomes

   Learning Outcomes (can be at course,
    program or institutional level)
   Program Outcomes
   Administrative Outcomes
Definitions and Examples
   Learning Outcomes:
       What changes in knowledge, skills, attitude, awareness,
        condition, position (etc.) occur as a result of the learning
        that takes place in the classroom. These are direct benefits
        to students.
       Examples:         general learning skills (e.g. improved
        writing and speaking abilities), ability to apply learning to
        the work environment (e.g. demonstrate skills in co-op),
        program-specific skills developed or enhanced (e.g. take
        blood pressure.)
Definitions and Examples
   Program Outcomes:
       The benefits that results from the completion of
        an entire program or series of courses. Are there
        benefits for students who get the AAS in welding
        versus those who take a few courses? If so what
        are they?
       Typical examples are: licensure pass rates,
        employment rates, acceptance into 4-year
        schools, lifelong learning issues, contributions to
        society, the profession, etc.)
Definitions and Examples
   Administrative Outcomes
       Units/programs want to improve services or
        approach an old problem in a new way.
       They want to become more efficient and effective.
   Typical examples are:
       All faculty will attend one professional meeting annually
        so they can stay up-to-date in their field, or:
       Counseling wants to recruit a new counselor with
        expertise in working with first-generation students, or:
       Facilities services wants all college units to feel that they
        respond quickly to security issues.
What is an Outcome

    A short-term, measurable, specific activity
     having a time limit or timeline for completion
     around a specific outcome
    They measure outcomes and are used to show
     progress toward goals
    They specify who, will do what, under what
     condition, by what standard and within what
     time period
How to Set Outcome
   There’s no magic number
       e.g. 80% or 90%
       What is reasonable?
       What can you afford?
       What realistically can your staff accomplish?
       What percent shows you’re not committed and
        what percent shows you’re naïve?
How to Set Outcome
   Examples:
       Fifty percent of students will be able to
        communicate effectively in writing (complete the
        writing exam with a grade of 60 [D] or better)
       By the end of the spring term, 95% of faculty and
        staff will have completed 20 contact hours of
        professional development (workshops, college
        courses, conferences, onsite trainings, etc.)
More Realistic
    Seventy percent of students will be able to
     communicate effectively in writing (complete the
     writing exam with a grade of 75 [C+] or better)
    By the end of the spring term, the professional
     development office will increase their offerings for
     faculty and staff by 10% over what was offered
     last year (workshops, college courses,
     conferences, onsite trainings, etc.)
How to Set Objectives
   The first time you set objectives, be
   Allow yourself a pilot semester or year to
    determine the appropriate levels of change
    that can be expected
   Don’t pull a rabbit out of your hat (e.g. let’s
    grow enrollment by 10%)
   May need to benchmark (what does it mean)
  Fall Curriculum Enrollment at ABC
 17,500                                                                            17,188
 17,000                                                          16,660               (1.2%)
                                                        16,245               (1.5%)
                                              15,997                (2.6%)
 16,000                              15,724
                            15,488                        (1.6%)
 15,500            15,208                        (1.7%)
          14,975                        (1.5%)
 15,000                      (1.8%)
 14,500                                                             Headcount

          1997      1998     1999      2000      2001     2002      2003      2004     2005

Should they set a goal for a 5% enrollment growth for next fall?
Why is This Hard?
    Because it is education
    Because the best results may not
     happen for years
    Because we are so busy doing what we
     are doing…. we forget why we are doing
Let’s Look at Healthcare
   When you have strep throat and go to the doctor for
    your antibiotics (your intervention)
   What are your intended outcomes?
   Would the doctor ever tell you:
       We believe curing disease is a developmental process
       We believe there is value in the activity of taking pills and
        receiving shots
       We’re not sure if you’ll get better, it is how all doctors have
        treated the disease since we learned about it
   What would you say to that?
 How to Measure
Identifying Outcomes
   Faculty/staff in an area are often the worst in defining
    outcomes because they are too close to the subject.
New Teachers         2nd- 3rd      Middle Years of     Older
1st Year             Year Teachers Teaching            Teachers

Unconsciously        Consciously       Consciously     Unconsciously
Unskilled            Unskilled         Skilled         Skilled

Don’t realize they   Realize they      Know what works Can’t remember
don’t know what      don’t know        and why         why they do
they are doing       what they are                     what they do –
                     doing – begin                     just know it
                     to ask for help                   works
Sources of Ideas for Outcomes
    *   program documents
    *   program faculty and staff
    *   national associations/credentialing boards
    *   key volunteers
    *   former students
    *   parents of students
    *   records of complaints
    *   programs/agencies/employers that are the next
               step for your students
    *   other colleges with similar programs, services
               and students as yours
    *   outside observers of your program in action
How Often
   Should we measure objectives or student
    learning outcomes every year?
   When does measurement become too time
   Units need time to put into effect the
    changes made as a result of outcome
    assessment before they are thrown back into
    another cycle. They need time to reflect on
    changes and results.
Disappointing Outcome Findings:
Why Didn’t We Meet Our Objectives?

    Internal Factors:

    *       Sudden staff turnover
    *       New teaching philosophy/strategy
    *       Curricular change (campus move)
    *       Unrealistic outcome targets
    *       Measurement problems (lack of follow-
            through, no effective tracking)
Disappointing Outcome

  External Factors:

  *       Community unemployment increases
  *       State funding changes
  *       Related programs (BS or MS programs) close
  *       Public transportation increases fares or shuts
          down some routes serving your campus or
          time slot
  *       Employment trends change
Use Your Findings
Internal Uses for Outcome Findings

    Provide direction for curricular changes
    Improve educational and support programs
    Identify training needs for staff and students
    Support annual and long-range planning
    Guide budgets and justify resource allocations
    Suggest outcome targets (expected change)
    Focus board members’ attention on
     programmatic issues
    Help the college expand its most effective
    Facilitates an atmosphere of change within the
An Example from Instruction
   Workplace Basic Skills
       This program is a literacy initiative that goes
        directly into the worksite and teaches ESL
        classes, GED prep and GED classes.
   During their review, they surveyed both
    employers and students.
   This was the first time they had ever done
What They Learned
    Employers said:
        43.8% of employers reported increases in
         employee performance as a result of
         participation in the program.
        31.3% reported a reduction in absenteeism by
        87.5% said classes improved the morale of
         their employees
        37.5% said participants received raises
        50% said communication had improved.
What Students Said
      70.2% reported being able to fill out job forms
      35.5% said they could now help their children
       with their homework
      91.1% said they felt better about themselves
      44.4% said they had received a raise,
       promotion or opportunity as a result of the
      86.3% said their ability to communicate in the
       workplace had improved
What Has Happened Since
   Their assessment data has shown up in their
    marketing brochures to employers.
   Their enrollment has grown dramatically.
   They have received funding and marketing
    support from Charlotte Reads (considered a
    model adult literacy program).
External Uses of Outcome Findings

     Recruit talented faculty and staff
     Promote college programs to potential students
     Identify partners for collaboration (hospitals, businesses,
     Enhance the college’s public image
     Retain and increase funding
     Garner support for innovative efforts
     Win designation as a model or demonstration site
So will someone help me do
this… help me select and
measure outcomes for my
    Ideas for Outcomes
   Typical general education goals:
       Students will demonstrate the ability to obtain meaning from printed,
        electronic, and graphical resources
       Students will effectively communicate both orally and in writing.
        Students will demonstrate the ability to locate, critically evaluate, and
        present information.
       Students will apply mathematical concepts and skills to analyze,
        manipulate, and interpret quantitative data.
       Students will demonstrate the basic computer skills necessary to
        function in a technological world.
       Students will demonstrate the ability to identify, analyze, question, and
        evaluate content as a guide to understanding and action.
       Students will demonstrate knowledge of cultural differences.
       Students will demonstrate an understanding of the influence of the
        individual on group behavior and conversely, the influence of the group
        on the individual.
       Students will demonstrate comprehension of the major steps of the
        scientific method.
       Students will demonstrate knowledge of the humanities and critical skills
        in assessing cultural/artistic merit and significance.
    Let’s work through a few.
   How does the welding program improve math skills in
       Welding students will be proficient at weights and measures,
        calculating angles and predicting poundage of resistance.
   How does the culinary arts program teach students to
    write effectively?
       Culinary students will be able to successfully write menu
        descriptions, restaurant advertisements and job descriptions.
   How does the Nursing program teach students to
    think critically?
       Through case scenarios, nursing students will identify the
        correct course of treatment for diabetic patients based on
        descriptions of symptoms, behaviors and patient knowledge.
     Top Ten Skills for the Future
     Work ethic, including self-motivation and time management.
     Physical skills, e.g., maintaining one's health and good
     Verbal (oral) communication, including one-on-one and in a group
     Written communication, including editing and proofing one's work.
     Working directly with people, relationship building, and team work.
     Influencing people, including effective salesmanship and
     Gathering information through various media and keeping it
     Using quantitative tools, e.g., statistics, graphs, or spreadsheets.
     Asking and answering the right questions, evaluating information,
      and applying knowledge.
     Solving problems, including identifying problems, developing
      possible solutions, and launching solutions.
    The Futurist Update (Vol. 5, No. 2), an e-newsletter from the World Future Society, quotes
    Bill Coplin on the “ten things employers want [young people] to learn in college”
Let’s work through a few?
   How does the early childhood program improve the
    work ethic of childcare workers or the children in
    childcare centers?
   How do IT programs improve teamwork skills?
   How does Engineering improve students’ ability to
    use spreadsheets and read tables/graphs?
   How do you improve a student’s ability to:
       Influence people
       Have time management skills
       Ask the right questions
       Solve problems in the work environment
  Learning Outcomes for the 21st
Students in the 21st Century will need to be proficient in:
 Reading, writing, speaking and listening
 Applying concepts and reasoning
 Analyzing and using numerical data
 Citizenship, diversity/pluralism
 Local, community, global, environmental awareness
 Analysis, synthesis, evaluation, decision-making, creative thinking
 Collecting, analyzing and organizing information
 Teamwork, relationship management, conflict resolution and workplace
 Learning to learn, understand and manage self, management of
   change, personal responsibility, aesthetic responsiveness and wellness
 Computer literacy, internet skills, information retrieval and information
(The League for Innovation’s 21st Century Learning Outcomes Project.)
Let’s work through a few.
   How does any given program or course improve:
       Listening skills
       Environmental awareness
       Creative thinking
       Relationship management
       Conflict resolution
       Self-management
       Wellness
       Information management
   How can we measure it and use the results.
   Identifying and defining outcomes is the easy
   The devil is in the details.
   How do we track it, where does it all go, how
    do we score it, compile it, turn it into a
    comprehensive report.
   How do we “demonstrate improvement in
    institutional quality.”
Things to Remember
   Outcome measurement must be initiated from the
    unit/department level (promotes ownership of process).
   Measure only what you are teaching or facilitating.
   Measure what is “important” to you or your program.
   Be selective (2-3 outcomes only for a course, a select list
    for programs and institutional outcomes).
   Put as much time in to “thinking through” the tracking
    process as you do into the definition of outcomes.
   Spend the time up front in planning and the process will
    flow smoothly.
   It will prove to be energy well spent.
Look at Early Childhood Examples
   We do not do outcome evaluation so we can
    say we did it.

   We do it only for one reason:
       To Improve Programs and Services
Where Colleges Get In Trouble
   Overkill – they evaluate everything that walks
    and breaths every semester in every area.
   No time to “reflect” before they enter back
    into another assessment cycle.
   No focus on “use of results.”
   No ability to track results and tally them
    across the College.
The Insanity Principle

   Doing the same thing we have always
    done but expecting different results.

   Terri M. Manning
   (704) 330-6592

To top