Measurement

Document Sample
Measurement Powered By Docstoc
					The Evolution of
Accountability Reporting:
Lessons Learned Over Nine Years
By a State Education Agency

       Mindy Crain-Dorough, PhD
    Southeastern Louisiana University
         mindy.dorough@selu.edu
      Robyn Madison-Harris, EdD
   Louisiana Department of Education
           robyn.madison@la.gov

                                        1
Overview of presentation
 Background on report development and what
  Louisiana does
 History of Accountability Changes
 Problems/Solutions
 Recommendations for future directions




                                              2
Purpose of this study
 To compile a list of lessons learned as a
  reference for other states
 To document the history of accountability
  reporting in Louisiana
 To explain the complexities of the report
  development process
 To provide recommendations for
  improvements accountability reporting in
  Louisiana and other states

                                              3
Report development –
Not an easy thing to do
 Present complicated information and data in a format
    that is aesthetically pleasing and understandable
   Negotiate many conflicting opinions about content
   Turn complicated and detailed policy into computer
    programs that calculate scores
   Coordinating many large data files
   Extensive checking of all aspects of calculations and
    report generation has to be conducted.
   All of these must be done in a very rigid timeframe.


                                                            4
Why do we go to all this trouble?
 Not just to meet federal and state mandates
 Parents make decisions about which schools
  to which they will send their children and
  even where to live.
 Parents are also driven to take action at their
  children’s schools.
 Teachers, principals, and district staff use the
  results to make decisions about school
  improvement efforts and funding priorities

                                                     5
Louisiana’s choices
 Reports produced at the state level for
  districts and schools
 School Report Card for Parents in paper
  format and sent home with every child
 Electronic PDFs of reports are available on
  LDE website
 Assessment and accountability reporting are
  separate
 LDE works with a contractor to produce
  reports
                                                6
Back to the beginning
 Louisiana had report cards as far back as
  1996-1997 as part of a legislative mandate
 Transformed report cards into accountability
  reports in 1998-1999




                                                 7
School Report Card for Parents




                                 8
Inside




         9
Back




       10
Changes to reports over the years
     Year                  Change Incurred                         Impact on Reports
SY 1998-1999   First year of school accountability           Complete overhaul
                                                             2 new reports
SY 1999-2000   Interim year – no new labels reported         Addition of new tables and text
               Science and Social Studies data reported

SY 2000-2001   Grades 9-12 entered accountability            Complete overhaul
               First year growth was evaluated
SY 2001-2002   District Accountability data first reported   New district reports
               (1 year lag)
SY 2002-2003   Incorporated NCLB mandates                    Complete overhaul
                                                             1 new report
SY 2003-2004   Graphic design overhaul of all reports        Complete overhaul
SY 2004-2005   No major changes
SY 2005-2006   Adjustments for hurricanes                    Complete overhaul
               Incorporation of iLEAP
SY 2006-2007   Incorporation of graduation rate and index    Addition of new tables and text
               Revised school sanction classifications
                                                                                               11
Along comes NCLB
 First reports containing NCLB mandates in 2002-
    2003
   Louisiana had to incorporate NCLB into policy and
    thus the existing reports
   A dual system created – SPS Component and
    Subgroup Component
   Conducted an evaluation of existing reports for what
    was and was not working
   Prioritized contents
   Put highlights in report cards and details in a new
    report
                                                           12
 Incorporation of NCLB
   2001-2002             Added in           Intended                NCLB
    Reports             2002-2003           Audience             Requirements
                                                                   Included
School Report Card                      Parents                General information
for Parents                                                    on dual system, table
                                                               with Subgroup AYP
                                                               information
School Report Card                      Principals, faculty,   More details than
for Principals                          and district staff     Parent report card on
                                                               dual system and
                                                               Subgroup AYP
School                                  Media,                 Major changes to
Accountability                          researchers, and       format
Report (“Press                          the general public
Report”)
                     Subgroup           Principals, faculty,   All other NCLB
                     Component Report   and district staff     requirements
                                                                                  13
School Report Card for Parents




                             14
Inside




         15
Back




       16
Methodology
 Qualitative – authoethnographic, case study,
  and phenomenological approaches
 Data Collection
     Authoethnographic recount by authors
     Interviews of current and former accountability
      staff
     Document analysis
 Data Analysis – cross-case examination and
  content analysis strategies

                                                    17
Problems/Solutions
 Accountability system changing every year
     Be organized
     Planning meetings with key parties
     Preparation for planning meetings
      “Regrouping” at the end of each year




                                              18
Problems/Solutions
 Errors in the reports
       Improve the checking process every year using a log of
        problems found in previous year(s)
       Include cross-report consistency checking in the process
       Create thorough processing rules, “data maps,” and record
        layouts that work together
       Communication between report developers, data calculation
        staff, and the contractor on these issues is key
       Allow districts to review data in aggregated form before
        inclusion in reports, especially for new indicators
       Have emergency plans in place should errors occur
       Use everyone’s time effectively – hire temps if necessary to
        do more tedious checking


                                                                   19
Problems/Solutions
 Too many conflicting opinions about content
     Use focus groups to obtain evidence of what
      works best for consumers
     Develop a protocol for making decisions
      regarding conflicting viewpoints




                                                    20
Problems/Solutions
 Miscommunication within a state agency
   Is the agency structure conducive to functional report
     development?
   Develop an issues database to track issues and their
     resolutions as well as a process for maintaining the
     database
   Develop a thorough timeline with a process for
     ensuring everyone is on task and a protocol for
     revising the timeline should unexpected events arise
   Find a method of meeting regularly to discuss issues
     and timelines that is most effective for all involved – we
     are currently having weekly conference calls with the
     contractor and impromptu meetings in-between

                                                              21
Problems/Solutions
 Vendor relations
    Good vendor qualities include flexibility, availability to
     devote extensive time to the project, advanced
     technical skills, and good quality assurance practices
    Balance of working for us and working with us
    Establish protocol for staff turnovers and management
     changes
    Set standards and roles for quality assurance
     procedures and processes
    Regular communication and established
     communication procedures

                                                              22
Problems/Solutions
 New mandates (internal and external to state)
     Evaluate the current reports for what works
      and what doesn’t
     Prioritize contents of the report (current and
      anticipated)
     Take advantage of the opportunity to make
      enhancements to the reports that are
      undergoing major revision anyway
     Look at what other states are doing to get
      ideas
                                                       23
Problems/Solutions
 Disasters occur
     Meet with those most impacted by the events
      to obtain all information before making
      decisions
     Consider all aspects of the accountability
      system and brainstorm possible impacts




                                                    24
Problems/Solutions
 Giving one individual too much responsibility
     Have a team of report developers
     Rotate lead responsibility for individual reports
      every two or three years




                                                      25
Problems/Solutions
 Promoting and supporting report release
     Use Press Conferences, written press
      releases, and radio announcements
     Create a “Help Desk” to answer questions
      about accountability and other agency
      programs
     Provide accountability trainings on policy
      changes, the importance of data quality, and
      description of report contents to school and
      district staff
                                                     26
Problems/Solutions
 Making reports useful
     Brainstorm areas in the agency that can be
      highlighted in the report cards
     Provide a message to parents in the report
      card from the superintendent to inspire
      parents to become more involved in their
      child’s education




                                                   27
Validity and reliability
 Validity
      Confirm valid use of reports through focus
       groups and training
 Reliability
      Thorough and detailed checking process




                                                    28
The current “Parent Report Card”




                                   29
Recommendations
 Transition to interactive accountability
  reporting and tie to school improvement
  efforts
 Combine assessment and accountability data
 Reduce the number of releases and release
  the Parent Report Cards earlier
 Re-evaluate the content and purposes of the
  current reports
 Have an “Accountability Czar”

                                             30