Scaling Progress in Early Childhood Settings (SPECS) STEPHEN J. BAGNATO, Ed.D., NCSP Professor of Pediatrics & Psychology Director, Early Childhood Partnerships Director, SPECS (PAPREKA/PEIOS) Children’s Hospital of Pittsburgh/UCLID Center University of Pittsburgh School of Medicine firstname.lastname@example.org www.uclid.org SPECS Program Evaluation Research Team Leaders: PAPREKA and PEIOS Stephen J. Bagnato, Ed.D., NCSP, Director Candace Hawthorne, Ph.D., OTR/L, Coordinator Ilene Greenstone, MA, Coordinator Pip Campbell, Ph.D., OTR/L, Coordinator Assisted by Western and Eastern PA Research Teams What is the Authentic Assessment Alternative to Conventional Testing in Early Childhood? John T. Neisworth Stephen J. Bagnato John Salvia Frances M. Hunt Inauthentic Measurement in Early Childhood “Much of developmental psychology [early childhood assessment] as it now exists is the science of the strange behavior of children with strange adults in strange settings for the briefest possible periods of time.” (Bronfenbrenner, 1979, p. 19) Authentic Assessment in Early Childhood Natural observations of ongoing child behavior in everyday settings and routines vs. contrived arrangements; Reliance on informed caregivers (teachers, parents, team) to collect convergent, multi- source data across settings; Curriculum-based measures linked to program goals, content, standards, & expected outcomes; Universal design; equitable assessment content and methods; Intra-individual child progress supplemented by inter-individual normative comparisons; NAEYC/DEC/HS & PA DAP Assessment Standards & Practices Are There Professional and Pennsylvania Standards for Authentic Assessment in Early Childhood/Early Intervention? Selected Professional Standards for Early Childhood Assessment (DEC, 2004; NAEYC, 1997, HS, 2000) Reliance on developmental observations-ongoing observational assessments overtime Performance on “authentic, not contrived, activities” Integration of assessment and curriculum Child progress on past performances as the reference, not group norms Choose materials that accommodate the child’s special functional needs Use only measures that have high treatment validity Rely on curriculum-based measures as the foundation or “mutual language” for team assessments Defer a diagnosis until evaluation of a child’s response to a tailored set of interventions Use scales with sufficient item density to detect even small increments of progress 6 “Best Practice” Criteria for Authentic Assessment in Pennsylvania’s ECE Programs 1. Purpose: Assess for program planning not diagnosis or exclusion; eliminate “readiness” testing practices 2. Method: “No tabletop testing”; Deemphasize scores; observe and assess functional skills that link to the curriculum standards 3. Context: Observe and record evidence of natural, ongoing child development and behavior in typical, everyday routines not contrived settings 4. Process: Rely on teachers/caregivers to observe and record child progress 2-3 times each year 5. Standards-based: Align all assessments and their item content with ELS and curricula; link assessment with expected outcomes 6. Parent Partnerships: Enable parents to have central role in providing observational data on progress Measures for PEIOS/PAPREKA Research: Balance of Attributes Tension to balance research rigor with utility; Choice of measures based on following elements: 1. Simplicity 2. Authenticity 3. Utility 4. Evidence-base 5. Standards-referenced 6. Functional content 7. Sensitivity The Pennsylvania Early Intervention Outcomes Study (PEIOS) Documenting the Benefits of Early Intervention Supports in PA to Fulfill State and Federal Mandates What are the missions, research questions and authentic measurement design for PEIOS? PEIOS “Fast Facts” Aim: Document early intervention outcomes for state & federal mandates County agencies and MAWAs in 6 PA Regions mapped to PQP Random selection Collect data on entry-level functioning compared to performance at followup Use both/either an existing measure and a common functional measure across programs: ABAS II File reviews to code program and service intensity Multiple research strategies to analyze interrelationships among program intensity and child/family outcomes Classify outcomes by OSEP/ECO categories PEIOS Outcome and Research Measures Child Measure [Caregiver]: Adaptive Behavior Assessment System II (ABAS; Harrison & Oakland, 2004) Child Measure [Caregiver]: Program-Identified Measure (BDI; DC; COR) Program Measure [PEIOS Team]: Program Specs (Bagnato, 2005) Program Measure [PEIOS Team]: Developmental Specs (Bagnato, 2005) Adaptive Behavior Assessment System II (ABAS, 2003) Multi-dimensional observational & judgment- based rating scale (3-point) of functional competencies Ages: 0-89years Early childhood forms: Parent & Teacher/Provider Nationally standardized: Ages 0-5=2100; all forms and all ages=5270 Norm-referenced scores: General Adaptive Composite (100,15); Subskills (10,3) Excellent technical research base; disability studies; Aligns with DSM IV; AAMR categories Psychological Corporation ABAS Rating Format 0……..Is not able [can’t; too young; physical limits] 1……..Never/almost never when needed [prompts] 2……..Sometimes when needed [with/out help] 3……..Always/almost always when needed [before] G……..Check if you guessed ABAS Domains: Parent Form (241); Teacher (216) Communication Community Use School Living/Home Living Functional Pre-Academics Health & Safety Leisure Self-Care Self-Direction Social Motor ABAS Disability Research: Clinical and Matched Control Samples Mental Retardation Developmental Delay Biological Risk Factors (e.g., prematurity; drugs) Motor and Physical Impairments Language Disorders Autism and PDD Learning Disability ADHD Alzheimer’s Disease Neuropsychological Disorders Behavior and Emotional Disorders Deaf and Hard of Hearing PEIOS Program Evaluation Research & Measurement Model January 2006: Identify and train PEIOS program evaluation liaisons January-February 2006: Random selection of children January-February 2006: Train on use of the ABAS II (if chosen) with liaisons, teachers, others January-April 2006: Collect EI-entry child data January-April 2006: Conduct file reviews to document program intensity May 2006: Collect child progress data using ABAS II or program-chosen measure (each September and May) TIME IN INTERVENTION CHILD & FAMILY PROGRESS OUTCOMES PROGRAM INTENSITY PEIOS LONGITUDINAL REPEATED MEASURES REGRESSION RESEARCH DESIGN AND TIMELINE REGION 1 September May REGION 2 REGION 3 ABAS II ABAS II Program Scale Program Scale D-SPECS D-SPECS P-SPECS P-SPECS What are the federal OSEP/ECO outcome indicators and reporting timelines for PEIOS? OSEP/ECO Child Outcome Indicator Domains 1. Positive social-emotional skills (including social relationships) 2. Acquisition and use of knowledge and skills (including early language/communication; early literacy) 3. Use of appropriate behaviors to meet one’s needs Domains, sub-domains, and item content of measures are mapped to these integrated functional areas Sample LaRosa Database OSEP/ECO Child Outcome Indicator Metrics 1. % of children who reach or maintain functioning level comparable to same-age peers 2. % of children who improve functioning toward same-age levels 3. % of children who did not improve functioning 4. % of children maintaining own rate and preventing regression 5. % of children showing specific curricular skill improvements compared to own previous skill levels 6. % of children whose developmental progress profiles exceed own pre-intervention (maturational) expectations and those of their local EI peer group (IEI; CEI; PCI, HLM- EAPS) Statistical Impact of ECI on Child Progress Exceeding Maturation after 31 Months of Programming [Pooled HR/DD groups; n = 104; p<.000; 48th%ile>68th%ile; 95% confidence interval] 850 832 825 DOCS total score 800 790 800 ECI 775 761 776 777 750 757 expected 725 735 729 700 683 713 upper margin of error of 675 684 expectation 650 2.30 3.00 3.20 3.40 3.60 3.80 4.30 5.00 Age (in years) OSEP/ECO Family Outcome Indicators [Draft Reconciliation] Based on ratings on the Family Outcomes Survey (in development) 1. Understand their child’s strengths, abilities, and special needs 2. Know their rights and advocate effectively for their children 3. Help their children develop and learn 4. Have support systems 5. Access desired services, programs, activities in their community Timelines for State Reporting of Child Outcome Data to OSEP (Recent Report from OSEP/ECO National Meeting , Washington, DC, 1/12-13/06 December 2005 SPP: Measurement plan submitted February 2007 APR: Report on EI-entry child data only—no progress data wanted February 2008 APR: 1st progress report February 2009 APR: 2nd progress report February 2010 APR: 3rd progress report What is the collaborative model for training and implementation in PEIOS and PAPREKA? DIRECTOR PAPREKA COORDINATORS: Western PA Eastern PA RESEARCH SYSTEMS EVALUATION DATA MANAGER ASSISTANTS STATISTICIAN PROGRAMMER REGIONAL PA CONSULTANT COORDINATORS LIAISON COUNTY MAWA REGIONAL SPECS TEAM SCHOOL DISTRICT CONSULTANT PARTNERSHIP- Administrators TEACHERS CAREGIVERS EVALUATION ASSISTANT The Pennsylvania Pre-Kindergarten Analysis (PAPREKA) A 4-Year Independent Program Evaluation Research Collaborative to Document the Impact and Outcomes of Partnership for Quality Pre- Kindergarten (PQP) SPECS Pennsylvania Early Childhood Intervention Outcome Studies www.uclid.org; Early Childhood Partnerships: SPECS Heinz Pennsylvania Early Childhood Initiatives (ECI) [1997-present) (Bagnato, etal., 2002; 2005) Pennsylvania Pre-Kindergarten Analysis (PAPREKA; 2005-2009] (Bagnato etal. 2005) Pennsylvania Early Intervention Outcomes Study (PEIOS; 2005-2008) (Bagnato etal., 2005) TRACE Center of Excellence for Early Childhood Assessment (2002-2007; Dunst, Trivette, Bagnato) The Efficacy of a Direct Instruction Add-On to a DAP Curriculum in 4KIDS at Braddock (2005- 2007) Pennsylvania Preschool Integration Initiative (PAPII; 1989-1993) (Bagnato & Neisworth, 1993) SPECS Evaluation of Heinz Pennsylvania ECI Outcomes (1997-2004) Nearly 4000 at-risk children Urban and rural communities: Pittsburgh, Erie, Central PA, York, Lancaster, Beaver School district collaborations Steady developmental gains Delayed group gains Social-behavioral gains, even for disorders Early school success for >400 children: K-5 Reduced grade retention and special education Program quality improvements Improvements in parenting skills Evolving community networking Analysis of time-in-intervention effects (“dosage”) What are the missions, research questions and measurement design for PEIOS? PAPREKA Missions for PQP Document child and program outcomes attributable to PQP Analyze the comparative impact of various school district/community ECE partnership models on child and program outcomes Analyze the early school success of PQP children during their K year Field-validate the PA Early Learning Standards and develop an ELS assessment rubric Set the stage for future followup of PQP children into early grades and linkage with PSSA results PAPREKA Research Questions for PQP 1. PQP children will show a pattern of actual progress which will outpace their maturational expectancies. 2. Program quality and interactive teaching styles will predict child progress and outcomes. 3. Certain types of partnership models will better predict child outcomes and program quality. 4. For both children at-risk and with delays, those who participate and remain engaged in the PQP programs for the longest periods of time (“dosage”) will show the most significant progress. 5. PQP children will demonstrate early school success in preschool and kindergarten based upon three criteria: blind teacher assessments on the BSSI; greater % of ELS attainments; and reduced grade retention and special education placement rates compared to historical district benchmarks. PARTNERSHIP MODEL TIME IN INTERVENTION CHILDREN’S EARLY SCHOOL SUCCESS OUTCOMES PROGRAM ASPECTS What is the authentic measurement approach and timeline for PAPREKA? Natural learning environment “...the ongoing/routine/typical circumstances and contexts of a child”…the child’s natural “developmental ecology” that forms the environmental basis for real-life early learning and adaptation (OSEP, 1998). SPECS Approach for PAPREKA Design research model through collaboration with community partners (“participatory action research”) Train teachers for ongoing child assessments Conduct “authentic assessments” in everyday preschool settings using natural observations Provide feedback to teachers & parents for individualized early learning plans Evaluate progress on “Key Performance Indicators (KPI): child, program quality, teacher- child interactions, & early school success outcomes Classify type of partnership model in each site Assess K achievements of children Research impact of PQP PAPREKA Measures 1. Basic School Skills Inventory 3 (BSSI-3) (Hammill, etal, 1998) 2. Preschool and Kindergarten Behavior Scale (PKBS) (Merrell, 2003) 3. Early Childhood Environment Rating Scale (ECERS-R) Screener (Cassidy etal, 2005) 4. Teaching Styles Rating Scale (TSRS) (McWilliam, etal, 1996) 5. PA Early Learning Standards (ELS) Rubric (in development) Sample MIS Database A Sample of Some Things I’m Really Good At Mary’s Skills put 2 words together (examples: 'Want more,' 'Get down') refer to myself by name participate in and enjoy nursery rhymes and 22 months old finger games like 'Humpty Dumpty' or 'Itsy Bitsy Spider' Some Things I’m Working On copy sounds made by others drink from a regular cup without help answer simple questions Some Things I Want to Learn Next dress myself, but I may need some help or supervision share toys with other children when asked by an adult count two or three objects (saying the number as I touch the object) Database Merged Document Class Snapshot #2 Student Progress on Selected Assessment Items January 2003 DOCS Assessments Sunshine Center Young Toddlers 1 2 3 4 5 6 7 8 9 Item Briana Keisha Mark Akeem Felix Katy Carlos # Item Description J. Ned T. S. Ali M. S. A. J. B. S. 24 to 35 months draw a straight line from top to 295 bottom gt n gt Highlighted use one hand to open and close & Bolded 296 scissors gt gt gt n gt gt Item #’s usually use plurals and past 297 tense correctly when speaking gt gt gt n n gt gt Areas of use some verb endings ('-s' for plurals, '-ed' for past, '-ing' for More Program 298 present) n n n n n gt gt Support 299 tell if self is a boy or a girl gt gt bend at the waist to pick up something off the floor (does not 300 squat) match to the colors red, blue, 301 green, and yellow gt n gt gt n n gt gt pull down my pants to use the 302 bathroom Basic School Skills Inventory (BSSI) Learning readiness skills for children Authentic teacher observational ratings Ages: 48-107 months (Pre-3rd grade) 6 Domains: Spoken language; Reading; Writing; Math; Behavior; Daily living Standard and T-Scores (100/15; 50/10) Functional skills/benchmarks for learning Graduated scoring: 0, 1, 2, 3 (mastery) Norms = 757 children; 5 states PRO-ED BSSI Subscale Samples Spoken Language Uses complete sentences when talking Listens to and retells a story in sequence Initiates and maintains conversations with others Reading Recognizes upper/lower case letters Names letters when sounds are spoken Has basic site vocabulary of 5 words BSSI Subscale Samples Writing Writes from left to right Writes first name without a model Writes single letters when asked (b, h, m, t, a, e) Mathematics Counts objects in set of fewer than 10 Counts aloud from 1-20 Understands concepts of 1st, 2nd, 3rd BSSI Subscale Samples Classroom Behavior Makes friends easily Takes turns Uses teacher feedback to improve learning Can attend to activity for 5 minutes Daily Living Skills Enters and exits school by self Assumes responsibility for own belongings BSSI Rating Scale When completing the BSSI, a four-point rating scale is used to rate each behavior or skill: 0 (Does not perform) 1 (Beginning to perform) 2 (Performs most of the time) 3 (Performance indicates mastery) National Normative Comparisons for ECI Childrens’ Early Learning Abilities: Second Semester Grades K-1 Basic School Skills Inventory-Revised 150 140 Mean Standard Score 130 120 110 100 90 80 70 60 Spoken Reading Writing Math Classroom Daily Overall Social Problem Academic language behavior living skills behaviors competence skills Domain Acquisition of Kindergarten Precursor Skills Matching PA Early Learning Standards: Achievement in Final 6 Months of ECI 100 Time 1 % Learning Skill s Achieved 90 Time 2 80 70 60 50 40 30 20 10 0 Asks Understands Counts Names Writes Names Knows/ Names Retells Tells "Reads" Total letters letters printed "reads" items meanings letters signs Statistical Impact of ECI on Child Progress Exceeding Maturation after 31 Months of Programming [Pooled HR/DD groups; n = 104; p<.000; 48th%ile>68th%ile; 95% confidence interval] 850 832 825 DOCS total score 800 790 800 ECI 775 761 776 777 750 757 expected 725 735 729 700 683 713 upper margin of error of 675 684 expectation 650 2.30 3.00 3.20 3.40 3.60 3.80 4.30 5.00 Age (in years) Preschool and Kindergarten Behavior Scales (2003) 36-72 months Dual focus on social skills and problem behaviors Competencies/concerns with treatment validity Cooperation, interaction, independence, and behavior domains (attention, withdrawal) Parent and professional ratings Norms = 2,855 Graduated scoring and SS PRO-ED Preschool and Kindergarten Behavior Scales (2003) SOCIAL SKILLS Never Rarely Sometimes Often Works or plays independently 0 1 2 3 Follows instructions from adults 0 1 2 3 Shows self-control 0 1 2 3 Participates in family or classroom discussions 0 1 2 3 Follows rules 0 1 2 3 Takes turns with toys and other objects 0 1 2 3 Preschool and Kindergarten Behavior Scales (2003) PROBLEM BEHAVIOR Never Rarely Sometimes Often Acts impulsively without thinking 0 1 2 3 Must have his/her own way 0 1 2 3 Is restless and “fidgety” 0 1 2 3 Withdraws from the company of others 0 1 2 3 Is overly sensitive to criticism or scolding 0 1 2 3 Disrupts ongoing activities 0 1 2 3 Social-Behavioral Progress Pattern for ECI Children: Behavior Disorder Group (18%) T1-T5 150 Time 1 140 Time 3 Time 5 130 Mean Standard Score 120 110 100 90 80 70 60 50 Social Skills Problem Behavior Domain Roles for Authentic Assessment in PAPREKA Teachers PAPREKA Team [September and May] [September and May] (Note: January for 2006) (Note: January for 2006) BSSI-3 ECERS-R Screen PKBS TSRS Linking Assessment to State Early Childhood Outcomes: PA Early Learning Standards (ELS) Approaches to Learning Demonstrate initiative & curiosity Creative Arts Expresses self through Language & Literacy movement & music Understands language Logical-Mathematical sounds Personal-Social Develops space-shape concepts Program Partnerships Develops self-control skills Science Form school partnerships with ECE Social Studies Acquire knowledge about matter and living things Understand role of self in community Linking BSSI Items to the PA Early Learning Standards BSSI Area or BSSI Item PA EL Standard Domain (for 4- & 5-year-olds) Spoken Answers questions Develop and expand Language correctly after listening to a listening and story [SL 11] understanding skills [RL 1] Reading Follows printed instructions (reads and follows directions) [R 20] Classroom Demonstrates a readiness Behavior to respond when his or her turn comes during group activities [CB 11] Daily Living Follows directions relating Skills to paper-and-pencil tasks [DL 10] Linking BSSI Items to the PA Early Learning Standards BSSI Area or BSSI Item PA EL Standard Domain (for 4- & 5-year-olds) Mathematics Assigns the correct Learn about numbers, numeral to a set of objects numerical representation, [M 7] and simple numerical operations [LM 1] Understands the concepts of “first”, “second”, and “third” [M8] Daily Living Tells time within 5 minutes Skills from analog watch or clock face [DL 19] Overview of Programmatic Measures: ECERS & TSRS ECERS-R Screener TSRS 16 core items 20 items Two factors: Focus on teacher- 1. Activities- child interaction Materials (9) during instruction 2. Language- 8 teaching strategy Interaction (7) variables: Redirects, Introduces, Elaborates, Follows, Informs, Acknowledges, Praises, Affect PAPREKA LONGITUDINAL REPEATED MEASURES REGRESSION RESEARCH DESIGN AND TIMELINE PARTNERSHIP MODEL 1 September May PARTNERSHIP MODEL 2 PARTNERSHIP MODEL 3 BSSI-3 BSSI-3 PKBS PKBS ECERS-R ECERS-R TSRS TSRS Roles of Implementation Sites in the Partnership: Implement Assessment & Gain Incentive Funds Designate PQP Evaluation Liaison Train with PAPREKA team to learn assessments Train teachers in assessments Gain signed parent consent Ensure completion of assessments by teachers 2x/year Collect completed assessment forms and transfer hard copies to PAPREKA Coordinator Show teachers how to use feedback letters Roles of PAPREKA Team for Implementation Sites Meet with partnership to form trusting collaboration Maintain confidentiality Train liaison and teachers in authentic assessment “best practices” Complete programmatic observations 2x/year Collaborate to classify type of partnership in each program Produce child feedback letters Issue 1 year-end “report card” Analyze aggregate data on child outcomes PAPREKA Team Roles with The PQP Planning Sites Schedule collaborative meetings to develop our partnership early as a basis for future work Offer early training sessions on assessment purposes and eventual evaluation model and process Demonstrate computer child feedback options for programs to guide teaching and quality improvements Consult with partnerships to facilitate their eventual implementation proposal Collect extant information from sites on various programmatic features Begin process of classifying sites by type of partnership model Not everything that can be measured counts, and not everything that counts can be measured (Einstein, 1951) The MisMeasure of Man (Stephen J. Gould, 1981) “We pass through this world but once. Few tragedies can be more extensive than the stunting of life, few injustices deeper than the denial of an opportunity to strive or ever hope, by a limit imposed from without, but falsely identified as lying within” (p.28).
Pages to are hidden for
"SPECS"Please download to view full document