Managing Data Collection: Where we were, where we are and where we

							 Managing Data Collection
      for High Stakes
Performance Assessments:
Where we were, where we
are, and where we want to
            be!
          Dr. Gwen Brockman
      California State University,
            Dominguez Hills
   Brief History of CSUDH
  Program Implementation
• SB 2042 Credential Program
  – Candidates enrolled after July 1, 2004
  – Basic skills requirement
  – Subject matter competency
  – Reading Instruction Competency
    Assessment requirement
  – Incorporated AB 1059 to include
    instruction for ELL
  – Summative Assessment – high stakes
     Incorporating the
   Summative Assessment
• 2 Models available in California
  – California Teacher Performance
    Assessment (Cal-TPA)
  – Performance Assessment for California
    Teachers (PACT)
• Pilot year 2007-2008
• Full implementation for 2008-2009
  academic year
What is a Teaching Event?
The Teaching Event (TE) is a summative
performance assessment that meets the
requirements for a California Teaching
Credential. The TE is intended to be a
capstone assessment that requires novice
teachers to plan a series of integrated
lessons, video tape their instruction,
analyze student work, and reflect on their
teaching (based on National Board
Certification).
     Incorporating the
   Summative Assessment
• Mapping backwards
  – Accreditation concerns
  – Curriculum adjustments
  – Allowing for multiple opportunities for
    practice
• Getting faculty buy-in
• Managing the rising cost of data
  collection
  Preparing for the Roll-Out
• Backwards Mapping for Accreditation
  – National Council for Accreditation of Teacher
    Education (NCATE)
  – Commission on Teacher Credentialing (CTC)
  – Western Association of Schools and Colleges
• Backwards Mapping for Summative
  Assessment
  – Choosing what data to collect
  – PACT Teaching Event
     • Includes Planning, Instruction, Assessment,
       Reflection, and Academic Language
  Moving Toward the Next
 Generation of Assessment
• Reducing assessment and reporting
  costs
• Using an electronic data collection
  system – searching for the right fit
• Identifying key artifacts
• Measuring student learning outcomes
  for teacher candidate’s students
    Getting Faculty Buy-In
• Teaching Event as summative assessment
  – Adjusting curriculum
  – Development of rubrics
• Utilizing TaskStream
  – Setting up the system to begin data collection
Collecting Evidence
Collecting Evidence (cont.)
    Getting Faculty Buy-In
• Teaching Event as summative assessment
  – Adjusting curriculum
  – Development of rubrics
• Utilizing TaskStream
  – Setting up the system to begin data collection
  – Training of faculty and university supervisors
  – Changing culture of electronic data submission
    collection (students and faculty)
  – Reliability in scoring teacher candidates
Reliable Scoring - Rubrics
Multiple Evaluator Report
         Where we were…
• Hand data collection
  – Teaching event data (full implementation
    pilot year 2007-2008)
    • Demographic and scoring data (quantitative)
      were entered by hand (2 to 4 weeks)
    • Scoring sheets were kept to be scanned
    • Sample teaching events were kept as
      exemplars and scanned for electronic report
      submission
        Where we were…
• Other data necessary for
  accreditation and approval by CTC
  – Program requirements (standards
    based)
  – Surveys – Exit (program improvement),
    master teacher, and university
    supervisor
  – Signature Assignments – key
    assessments
  – End of program requirements
         Where we were…
• Hand data collection and reporting (6
  Master’s & 7 credential programs serving
  2600 candidates)
  – Excel files were sorted by content area, intern
    vs. student teachers, and grant programs, etc.
  – Scores and demographic data were entered by
    student workers (6 student workers, part-time,
    all year)
  – Scoring sheets and other data were kept
    hardcopy to be scanned
Managing the Rising Cost of
     Data Collection
• Hand data collection: approximately
  $160, 000
• Current spending: approx. $80,000
  – About 50% is now collected
    electronically by TaskStream
  – Includes faculty coordinator (part-time
    year-round) and 3 student assistants
    who work 20 hours each week
        Where we are…
• Setting up TaskStream’s Directed
  Response Folios
• Using Help!
• TaskStream meets our needs for
  coursework and performance data
  collection
• Options, options, options
           Where we are…
• Data collection in TaskStream
  – Demographics – 15 descriptors (ethnicity,
    gender, native English speaker, primary
    language, program, credential area, etc.)
  – Signature Assignments (SA)
  – Teaching Event
  – Surveys – program exit, MT, Supervisor, course
    reviews (coordinators and supervisors
    encourage candidates to participate.)
  – Forms – supervision fieldwork/student teaching,
    context for learning, lesson plans, lesson
    rubrics, parent conferences, bulletin boards,
    resumes, case studies, portfolios
  – Fieldwork – candidate performance in K-12
    classrooms
         Where we are…
• TaskStream reporting system options
  – Direct reporting based on standards
    (CTC, NCATE, WASC)
  – Direct reporting of signature
    assignments for accreditation and
    program improvement
  – Comparisons between programs,
    standards, students, subgroups
    (demographic descriptors)
         Where we are…
• Data Storage on TaskStream
  – Previously run reports
  – Web page option
    • Storing exit survey, performance data,
      coursework, Masters exam completion rates,
      for all programs
    • Available for program coordinators
    • Manager account to run additional reports
  – Goals and areas of improvement related
    to rubrics
           Where we are…
• Data reporting
  – Intern and student teachers (ST)
    • Compare proficiency between ST and Interns
    • Compare trend data
    • Reports for areas of improvement
  – Program feedback –what’s working & what’s
    not working
    • Drives professional development
       – Supervisor support with academic language
       – Workshops – differentiation, scaffolding, academic
         language
       – Curriculum changes
       – Increased practice for teacher candidates in areas of
         need
    Where we want to be!
• All data collection to be electronic
  through one system and include all
  programs
• Data analysis between and in
  programs (e.g., tied to conceptual
  framework, vision, mission, school of
  education goals, and university
  mission and vision)
    Where we want to be!
• Continuous improvement with data
  – Focus groups mapping professional
    practices (rubric components), rich
    conversations
  – Candidate profiles to ensure coherent
    progress and analyze with faculty teams
  – Improved collection of student learning
    outcomes
  – Improved collection of student
    dispositions
 Benefits and Values of Electronic
  Data Collection, Accountability,
     Reliability, and Portfolios
• Value
  – Decreased assessment spending by half
  – Lowered costs related to accountability and
    reliability of scoring
  – second option for data storage
• Benefits
  – Access for coordinators and program chairs
  – Real-time data analysis and faster turn-around
    on reports
  – Accountability for faculty and students
  – “Pack it up” option for candidate portfolio
    development and employment
   Questions?



Gwen Brockman
CSU Dominguez Hills
gbrockman@csudh.edu
310.243.2332