Challenges Related to Performance Assessments A Case Study Based

							   Challenges Related to
Performance Assessments: A
Case Study Based on Teacher
 Work Sample Methodology

      Mark Girod and Hilda Rosselli
       Western Oregon University
TNE Learning Network Annual Meeting 2006
        The WOU Context
• Approximately 5000 students (2 colleges)
• Small rural community with placements in
  over a dozen communities
• 33 tenured faculty 26 w/Teacher Ed focus
• Long history of Teacher Education
• Four initial prep programs (1 advanced)
 Undergraduate program
• Two cohorts a year (N = 75)
• Team planned
• Dual authorizations
  • Early Childhood, Elementary, Middle, High
– Contextual teaching and learning
  • Community Mapping
  • Service Learning
  • Integrated field experiences
Master of Arts in Teaching
• Designed for second career or those with
  a bachelor’s degree
• Preparing middle and high school
  teachers
• Both full-time and part-time offerings
• Part time enhanced with online and
  Saturday classes
• Four new cohorts a year
    Master of Science in
        Education
• Graduate Initial licensure programs in
  Special Education and Deaf Education
• Advanced program designed for those
  seeking additional professional
  development (can add endorsements and
  authorizations)
• Revised set of proficiencies
• Exit requirement (APD, theses, comps,
  project)
         The WOU Context
•   Unionized campus
•   Lowest tuition = slim infrastructure
•   Quarter system/36 hour load
•   NCATE accredited/TSPC proficiencies
•   Home to Teaching Research Institute
       1 + 1 = SYMBIOSIS




Continuous               Research backed results
improvement
  History and                Advanced
  experience in              research skills
  teacher                    and experience
  preparation
              Research meets Practice!
    CHALLENGE 1
How can programs provide
students with safe low stakes
structured practice on
performance assessments
used in high stakes decisions?
  Practicing Connecting
  Teaching and Learning
 Cook School District Simulation

• Web-based, simulated classroom(s)
• Based on work sample framework
• Seven threads: context, content,
  assessment, adaptation, alignment,
  reflection, and instruction
       What does pilot
      testing suggest?
Primacy of alignment
Necessity for individualization
Role of assessment in supporting learning
Data-driven decision making
Better work sample scores
Better real-world teaching scores
Increased perceptions of skillfulness and
value for seven conceptual threads
    CHALLENGE 2
How can institutions
incorporate assessment
fidelity assurances into the
culture of faculty workloads
and college infrastructures,
particularly at smaller
institutions?
    Resource Implications
•   Design of assessments
•   Piloting of assessments
•   Data storage and retrieval
•   Analysis (disaggregated and aggregated)
•   Systematic review and use of results
•   Sustainability
•   Refinement and redesign of assessments
         Design Reality
 • In house expertise vs. external
  • Measurement fidelity
  • Faculty FTE
• Faculty input and buy in
  • Endogenous vs. exogenous motivation
  • Accountability meets academic freedom
• Standardized vs. customized
  • Proficiency based
  • Program specificity
             Flowchart

NCATE     Program      Element   Assessment
Domain   Proficiency



                           Yes
Data      Rule
                           No     Flag
                   Undergraduate Program Gate 4
                Recommendation for Initial Licensure

                                                                                                                                                              GATE 4
    N C ATE                    WOU                                    A SSESSM ENT ELEM EN TS                                                      Recommendation for Inital Lic ensure
   D OMA IN S             PR OFIC IEN C IES                       R ef lects stro ng link with impa ct o n
                                                               st udent lea rning a s ref lect ed in TWSM.

                                                          Depth & Breadth*


                             1 .Co n ten t                  Integration 1b
 C ONTEN T
KN OWLED GE                 K n o w led g e                 Alignment 4b
                            & P ed ag o g y
                                                       Curriculum M odific.-3f, 3a
                                                                                                                                                                                                                      Y
                                                               Benchmarks & Ass es sments
                                                                                                                                                                Req u ir ed                         A p p ro v al /
                           6 . Liter acy D ev .                                                          Cultural Inflluence                                   Co u r sew o r k    Reg istr ar
                                    &                                 Literature                                                                                                                     main tain
                           Co mmu n icatio n                                                                                                                                                          G PA
                                                                                                         Facets of Literacy                                                                                           N
                                                               Language Development

                                                                   2nd Language* -2b, 2e                 Literacy Programs

                                                                                                                                                                                                                      Y
                                                                                         Disabilities* - 2c, 2e, 2g
                                                       Ass es sment Tools 4a,b
                          4 . A ssessmen t
                          & I n stru ctio n              Goals & Standards
                                                                                                                                                                 Tests               Prax is         Passin g
                                                        Planning--1a-g, 2j, 3c,e,
                                                                5a, c, d                                                                                                                                              N

                                                          Implementation*
                                                                                                  Phys ical Environment                                                                                                   Graduation and
PED A GOGIC A L
                                                             Time M anagement 1g,2h,i                                                                                                                                     Recommendation
 KN OWLED GE                                                                                         Student Behavior*-2b,d,e                                                                                                for Initial
                            5 . Classr o o m                   Clas sroom As sistants *2k
                                Climate                                                              Clas sroom Climate 5c
                                                                                                                                                                                                                      Y      Licensure
                                                                Human Dignity*-2a,e
                                                                                                                                                                2 W o rk          Field Bo o k      Sco re o f
                                                                                                                                                                samp les          Ev alu atio n s   p ro ficien t
                                                         Operations & Concepts 1e                          Ass es sment Evaluation
                                                                                                                                                                                                                      N
                                                        Des igning Learning Environments                    Professional Practice

                            7 .Tech n o lo g y            Evaluating/Selecting Software*                  Social & Ethical Is sues*




                                                                 Human Development M odels 4a                                                                                                                         Y
                                                                                                                                                               Co mp letio n      Su mmativ e         ap p ro v al
                                                                        Learning Theories*                                                                     o f Stu d en t     Ev alu atio n       b y b o th
                             2 . H u man                                                                                                                        Teach in g
                            D ev elo p men t                            Applications 3a, 3d
                            & Lear n in g
                                                                                                                                                                                                                      N
PR OFESSION A L                                                                     Awarenes s
                                                                                                                          Integration*-1f,2c, 4a
 KN OWLED GE
                                                                              Inclusion*-1f,2c, 4a

                           3 . D iv er sity &                                       Advocacy
                           Ex cep tio n ality
                                                                    Philosophy of Education

                                                                   Reflective Practitioner* 3.2
                             8 .Ph ilo so p h y ,
                             Best Pr actice,                                                                          Educational Res earch* 3d
                             & Reflectio n
                                                                       Res ponsibility* 3.2

                                                                             Professional B ehavior 5a
D ISPOSITIONS
                               9 .Pro f essio n al                           Active Participation 4.3
                               Co mmitmen t &
                                D isp o sitio n s                            Support Learning                  Human Dignity*-2a,e
                                                                              Community 5b
                                                                       Professional C ommunication 2g

                                                                              Self-direction ?2.4?, 4a


                           1 0 . Co llab o ratio n ,            M ulti-Disciplinary Teaming 4.3
                            Par tn ersh ip s &
                               Lead er sh ip                   Constructive Communication 2.1                 Clas sroom As sistants *2k

                                                                     Ass istance & Support

                                                                     Learning Community
        Piloting Reality
•   FTE for leadership/oversight
•   Faculty buy in and involvement
•   Timelines for implementation
•   Fidelity check points
•   Ease of administration/analysis
•   Timely return on results for review
•   Inclusive decision making process
•   Communication to users
     Data System Reality
• System design/ development/maintenance
  • In-house vs. commercial
  • Master design, prototypes, server infrastructure,
    system integrity
  • Expertise, time, dedicated FTE
• User friendliness
  • Parallel look
  • On the spot assistance
• Access/confidentiality/IRBs
• Sustainability
  • Alignment with regional accreditation
   Analysis & Review Reality
• Gatekeeping function
  • Alerts and timely gatekeeping
  • Student access
• Mining and reporting functions
  • Linkage to SIS
  • Queries: The right questions
• COE Data Analysis Review Events
• Stakeholders (LAS, Consortium, school
  partners)
• Timely follow up and revisions
   Sustainability Reality
• Dedicated FTE for staff and faculty
  • TRI/IR
• Technology upkeep and upgrades (in-house)
• Centralized printing costs
• Regular training/professional development
  • Faculty/adjuncts/mentors/staff (new &
    veteran)
• Support for faculty scholarship
• Keeping administration primed
• Sustaining motivation and excitement
     CHALLENGE 3
 How can the field move
 performance assessments
 from generic to more content
 specific applications?

Contact: Dr. Adele Schepige
   schepia@wou.edu
     TWSM Basic Elements
Contextual information,
Pre assessment
Instructional plans and
rationale for the unit,
Assessment instruments,
Assessment data and analysis
of learning, and
Reflection about the teaching
experience and unit
                 Process
• Set of guidelines developed with specific
  science requirements for the work sample
  (secondary level)
• Piloted in a quasi experimental design
• Results showed more depth and breadth in
  planning section, more careful analysis of
  student prior knowledge, and stronger
  connections between content and
  instructional decisions
      Next Stage of Testing
• Each component of TWS examined for
  alignment with NSTA standards
• Descriptions and explanations developed to
  make requirements more content specific.
• Scoring guide modified to include specific
  references to science outcomes aligned to
  the NSTA standards within each component
  of the work sample.
• Comparative analysis of modified TWSM
  with earlier prototype currently underway.
General Requirements for      NSTA Standards
Rationale
Provide a conceptual overview NSTA Knowledge of Conceptual
                              Science Content 1a,1b,1c,
                              NSTA Nature of Science 2a,2b,
                              NSTA Inquiry 3a,
                              NSTA Issues 4a

Explain the relationship to        NSTA Science in the
state goals/standards and the      Community7a,7b
community.
Provide a brief summary of the     NSTA Assessment 8b
pre-assessment analysis
Explain instructional decisions,   NSTA 5: General Skills of
including the integration of       Teaching
literacy and the relationship to   NSTA 6: Curriculum
what is known about student
prior knowledge
    CHALLENGE 4
How can universities
effectively use data gathered
from performance
assessments across
institutions to answer
questions about teacher
quality in larger scale studies?
Oregon Collaborative
 Research Initiative
•Cross institutional set of studies
 on teacher work samples
•Open invitation to all 18
 preparation institutions
•Includes 40 researchers and
 teacher educators from 12
 institutions
        Evolution study…
• Examination of the state mandated teacher
  work sample and how its uses and
  implementation have evolved across contexts
• Key questions include:
   • How and when was the work sample first
     used across institutions?
   • What prompted particular elaborations or
     evolutions of the methodology?
   • How well has the original intent of the
     work sample been achieved?
      Experience study…
•Examining perceptions of authenticity
 among faculty and candidates
•Key questions include:
  •Do candidates view the work sample
   as meaningful and valuable?
   Components? Change?
  •What aspects do faculty value?
  •How does the work sample get
   taught? - seven threads?
           Effect study…
• Examine the effect of preparation using the
  work sample framework on assessment
  practices, instructional practices, and,
  ultimately, P-12 student learning
• Key questions include:
   • What effect do candidates have on the
     kind and complexity of P-12 learning
     pursued and attained?
   • Are there variances by kind of student,
     kind of candidate, and kind of context?
   Ongoing challenges…
•Ownership of data
•IRB approval
•Communication across sites
•Institutional integrity
•Congruity of definitions
•Buy-in
•Time, money, and resources
•Differences in perspective…
       Along the way…

  Need for a logic of inquiry for
establishing conceptually coherent
     and empirically verifiable
   connections among teaching,
   teacher preparation and K-12
 learning within today’s schools.
       Creation of CTPL
     Coalition for Connecting
       Teaching, Teacher
      Preparation and K-12
             Learning

A collaboration involving individuals
    from over 25 institutions and
              agencies