Portfolio evaluation

Document Sample
Portfolio evaluation Powered By Docstoc
					Portfolio Evaluation: Assessing
  CCPTP conference: 2010 Joint Conference
         Michael J. Scheel, Ph.D.
      University of Nebraska-Lincoln
Portfolio: The Concept
 “A purposeful collection of student work that tells the story
  of the student’s efforts, progress, or achievement in a given
  area(s)” (p. 36; Arter & Spandel, 1992).
 Portfolio assessment is used for faculty tenure evaluations;
  for some license or diplomate evaluations
 In counseling, beyond performance on a standardized test or
  completion of courses, case examples serve to demonstrate
  competence (Coleman, 1996).
 “An artist’s portfolio shows not only what the artist knows
  but how he or she can put that knowledge into practice over
  a range of situations” (p. 219; Coleman, 1996).
Portfolio: The Advantages
 Can demonstrate what one knows that reflects the
    complexity of the topic and the integration of skills with
    knowledge (Collins, 1992).
   Integration of information collected over several contexts
   Provides stimulus for discussion in which the student can
    present the meaning of an activity or a competency.
   Can demonstrate increasing sophistication over time
   Stimulates self-reflection because the learner is responsible
    for deciding the goal and the content of the assessment
The portfolio process
1)   Determine the goal
2)   Identify what evidence will demonstrate acquistion of the
3)   Submit a caption (competence statement) about the
     importance of the evidence
4)   Prepare a statement (narrative) about how the portfolio as
     a whole reflects learning
5)   Fosters a self-reflective process by the learner (Collins,
Examples of evidence/artifacts
 Client rating of counselor effectiveness; supervisor evaluations
 Self-evaluations; integrative narratives of a content area
 Videotapes of therapy sessions that demonstrate an intervention
   Case presentations; Ethical dilemmas presented through a case
   A personal ethical decision-making model
   Personal theory of change
   One page essays on theory orientation, multicultural competence, use of
    assessment in practice etc. similar to an APPIC app.
   Case presentations and write-ups
   Outreach programs presented with powerpoints; presentations at prof.
    conferences; publications
   Psych. assessments conducted; written reviews of assessment
    instruments; reviews of literature
UNL Copsych Portfolio
 Only the counseling psychology program within the Dept. of
  Ed. Psych. uses portfolio evaluation with an oral
  examination; satisfies the Graduate College comprehensive
  exam requirement
 Continuance refining process since 2004; Copsych program
  approved latest version in the Fall of 2009 as the only
  comprehensive exam method for our program
UNL Portfolio Procedure
1)   meet with your academic advisor to discuss the portfolio
     procedure and plans for completing comprehensive examinations
2)   develop a plan during Program of Study committee
     meeting in consultation with your doctoral committee regarding the
     portfolio procedure; doctoral committee members are invited to
     participate; all program core faculty are required to participate
3)   work closely with your academic advisor to assemble portfolio;
     submit a completed version to your advisor; portfolio is
     comprised of a narrative summary for each competency area; written
     narratives are approx. 7 to 10 pp. for each area; the bulk of the
     portfolio is comprised of these narrative summaries; a limited number
     of accompanying documents/artifacts may be included as
     supplementary materials to enhance or further illustrate information
     contained in the narrative summaries; E.g., theoretical orientation
     essay; mc self-exploration paper, integrated assessment report.
UNL Portfolio Procedure (cont.)
4) Advisor distributes portfolio to each committee member and each
   committee member reviews
5) Meeting is held with the committee and student to discuss each
   area of the portfolio; Committee members bring questions and
   needs for clarification to the meeting after individual reviews of
   the portfolio
6) Committee arrives at a consensus decision of ‘pass’ or ‘no pass’
     after the conclusion of the meeting;
7) If ‘no pass’ is assigned to a portion of the exam, the student will
     be given instructions concerning changes and additions to be
     required to bring the area or areas up to a passing level.
Contextual factors
1) The majority of coursework that fulfills the Counseling
   Psychology Core Curriculum should be completed at the
   time of Comprehensive Exam (most students complete
   portfolio exam in 5th or 6th semester).
2) Doctoral students applying for internship require
   notification from the Director of Training that they have
   completed their comprehensive exam at the time of their
   application. Thus, successful completion of comprehensive
   examinations (as well as successful proposal of dissertation)
   must be achieved by October 15th of the year in which the
   student is applying for pre-doctoral internship.
Portfolio Contents
 Scoring rubrics are provided to assist students in considering
   what material to include in their doctoral portfolios.
   Specifically, students are expected to demonstrate
   competence in the following six areas:
1. Measurement, Assessment, and Psychological Testing
2. Counseling Theories and Practice
3. Career Development
4. Multicultural Issues
5. Ethical, Legal and Professional Issues
6. Area of Emphasis
Scoring Rubric
Counseling Theory and Practice
 Portfolio: Examples of performance indicators
1) Knows constructs and major theories and systems of thought
2) Personal theory of change articulated
3) Counseling interventions
4) Evaluate effectiveness of treatment
 Portfolio Area:
                               Benchmarks of
Counseling Theory
  and Practice

    Knows constructs and
     major theories and
     systems of thought         Foundational
                                • Scientific Knowledge/Methods
                                • Reflective Practice
                                • Ind. & Cultural Diversity
                                • Interdisciplinary Systems

   Personal theory of change

   Counseling interventions     Functional Competencies
                                • Assessment, Diagnosis, and
                                  Case Conceptualization
                                • Intervention
                                  •Knowledge of Interventions
                                  •Intervention Planning
                                  •Intervention Implementation

   Evaluate effectiveness of
Foundational competencies that line up with portfolio
performance indicators
Foundational Comp Benchmarks:
-   Scientific knowledge/methods: applies evidence in
    practice; compares and contrasts theoretical perspectives;
    scientific mindedness – articulates issues derived from lit. in
    supervision and case conf; reviews scholarly work related to
    clinical practice; willingness to present work for scrutiny by
-   Reflective practice self-assessment: awareness of personal
    strengths; summarizes lit relevant to client care
-   Individual and cultural diversity: monitors and applies
    knowledge of self as a cultural being in assessment, treatment,
    and consultation
-   Interdisciplinary Systems: demonstrates ability to articulate
    the role that others provide in service to clients
Functional Competencies
 Assessment, Diagnosis, and Case Conceptualization:
  identifies and conceptualizes symptoms through a
  developmental context; formulates treatment plans
 Intervention: Knowledge of interventions
 Intervention: planning
 Intervention: implementation - applies evidence-based
  interventions that take into account empirical support,
  clinical judgment, and client diversity; clinical skills used in
  development of therapeutic relationships; uses clinical
  judgment effectively
Portfolio area: Multicultural Counseling
 Examples of performance indicators
1)   Integrated conceptualization of mc competencies
2)   Awareness of personal values  other cultural groups
3)   Knowledge and application of mc theories of psychology
4)   Demonstrates mc skills
5)   Demonstrated plan for continued mc competence
Foundational Competencies:
Multicultural Counseling
- Reflective practice self assessment: reflective practice self-identifies cultural
  identities; articulates attitudes, values, and beliefs toward diverse others
- Relationships: interpersonal relationships: negotiates relationships with ind and
  groups that significantly differ from oneself
- Individual Cultural Diversity: Self-awareness (e.g., knowledge of self as a
  cultural being; understands cultural identities; critically evaluates feedback and
  initiates consultation/supervision over diversity) Applied knowledge (e.g.,
  knowledge of cultural diversity science, theory, and contextual issues;
  sensitivity to treatment applications with diverse others; understands individual
  cultural diversity and APA policies; conceptualizes diversity in appropriate and
  effective ways; considers diversity in assessment, diagnosis and case
  conceptualization; uses culturally sensitive instruments; seeks consultation;
  knowledge of multiple identities in case presentations and impact on these
Functional Competencies
 Assessment, diagnosis and case conceptualization:
  assessment (e.g., aware and use culturally sensitive
  instruments); Integration (e.g., displays know. of assessment
  tools specific to a client pop. & specific site)
 Intervention: Planning (e.g., contexually oriented
  including diversity); Implementation (e.g., applies evidence
  interventions while considering cultural diversity)
 Supervision and teaching: Awareness of factors affecting
  quality (e.g., impact of diversity on prof. settings and
  supervision; awareness of APA and other policies as
  interpreted through cultural and other contexts)
Future directions for portfolio
evaluation development
 Further lining up of portfolio criteria with the competency
 A need for research to validate portfolio assessment as a
  method to be used to measure competence (e.g., concurrent
  validity studies)
 Need to develop reliable methods of scoring and evaluation
 Any scoring method must recognize and value the holistic
  and integrated nature of the performance (Collins, 1992).
  Resist the temptation to be reductionistic.
 Baltimore, M. L., Hickson, J., George, J. D., Crutchfield, L. B. (1996).
    Portfolio assessment: A model for counselor education. Counselor Education and
    Supervision, 36(2), 113-121.
   Coleman, H. L. K. (1996). Portfolio assessment of multicultural counseling
    competency. The Counseling Psychologist, 24(2), 216-229.
   Coleman, H. L. K. (1997). Multicultural counseling competencies: Assessment,
    Education, and Training, and Supervision. In Pope-Davis, D. B. & Hardin, L. K.
    (Eds.) Multicultural aspects of counseling series,Vol. 7. pp. 43-59. Thousand Oaks,
    CA: Sage.
    Collins, A. (1992). Portfolios for science education: Issues in purpose,
    structure, and authenticity. Science Education, 76(4), 451-463.
   Van der Schaaf, M. F., & Stokking, K. M. (2008). Developing and validating a
    design for teacher portfolio assessment. Assessment and Education in Higher
    Education, 33(3), 245-262.
Email address for UNL portfolio
For requests of copies of the portfolio

Shared By: