Get documents about "DARA Research and Next Steps
Document Sample
DARA
Research and Next Steps
Cara Cahalan-Laitusis & Linda Cook
Educational Testing Service
National Accessible Reading Assessment Projects
Presentation
• Experimental Study of Read Aloud
• Psychometric Research
• Research Plans for Year 3
– Psychometric analysis of experimental data
– Tailored Test Design
– Cognitive labs
– IEP Decision Making for read aloud
National Accessible Reading Assessment Projects
Differential Boost from Read Aloud
(Non-disabled vs. RLD)
1. Is there a Differential Boost from read aloud?
2. How well do test scores (standard, audio, and
fluency) predict variance in teacher ratings of
reading comprehension?
3. Are teachers’ able to predict which students
will benefit from read aloud?
National Accessible Reading Assessment Projects
Prior Research
• No Differential Boost
– Kosciokek & Ysseldyke (2000)- Small sample size (n=31)
– Meloy, Deville, and Frisbie (2002) – Between subjects design
(n=260, 76% non-disabled, randomly assigned to audio or
standard)
– McKevitt & Elliott (2003)-Small sample size (n=39)
• Differential Boost
– Crawford and Tindal (2004)-(n=338, 78% non-disabled)
– Fletcher, et. al (2006)-Between subjects design (randomly
assigned to audio or standard). Sample included 91 Dyslexic (poor
decoder) and 91 average decoders
National Accessible Reading Assessment Projects
Data Collected
• GMRT Forms S and T
– Extra Time
– Extra Time with Read Aloud via CD
• 2 Fluency Measures
– WJ Reading Fluency
– Test of Silent Word Reading Fluency
• 2 Decoding Measures (4th grade only)
– WJ Letter Word ID
– WJ Word Recognition
• Demographic and Survey Data
National Accessible Reading Assessment Projects
Sample
• 1170 4th Graders
– 522 Students with RLD
– 648 Students without a disability
• 855 8th Graders
– 394 Students with RLD
– 461 Students without a disability
National Accessible Reading Assessment Projects
Design
Session 1 Session 2
Group Form Accommodation Form Accommodation
1 S Standard T Audio
2 S Audio T Standard
3 T Standard S Audio
4 T Audio S Standard
National Accessible Reading Assessment Projects
Means for Grade 4
Non-LD RLD
Test/Condition N Mean SD N Mean SD
WJ Letter Word ID 604 504 21 469 473 29
WJ Word Attack 604 504 15 469 484 20
TOSWRF 604 102 10 469 89 12
WJ Fluency 604 501 24 469 474 21
Audio 604 502 32 469 477 30
Standard 604 497 37 469 457 31
Boost 604 5 24 469 19 27
National Accessible Reading Assessment Projects
Means for Grade 8
Non-LD RLD
Test/Condition N Mean SD N Mean SD
TOSWRF 463 103 13 373 90 12
WJ Fluency 463 560 42 373 514 34
Audio 463 555 31 373 521 27
Standard 463 553 3 373 511 28
Boost 463 2 21 373 10 23
National Accessible Reading Assessment Projects
Scores by RLD and Grade
600
550
500
Standard
450
Audio
400
350
300
RLD Not LD RLD Not LD
Grade 4 Grade 4 Grade 8 Grade 8
National Accessible Reading Assessment Projects
1. Is there a Differential Boost from
read aloud?
Repeated Measures ANOVA and ANCOVA
• Dependent Variables:
– GMRT “Standard”
– GMRT Audio
• Independent Variables:
– Disability Status (RLD vs. NLD)
– Form/Order (STSA, STAS, TSSA, TSAS)
• Covariate: Decoding and Fluency
Measures
National Accessible Reading Assessment Projects
ANOVA Findings
• Yes, students with reading-based learning
disabilities have larger gains (on average) from
read aloud than students without disabilities
– Finding consistent at both grades 4 and 8, but
boost is larger at grade 4
– Controlling for Decoding and/or Fluency as a
covariate did not alter findings
National Accessible Reading Assessment Projects
2. How well do test scores predict
reading comprehension?
• Multiple regression analyses to
determine how much variance in
teachers’ rating of reading
comprehension (5-point scale) were
predicted by three test scores:
– Standard
– Audio
– Fluency
National Accessible Reading Assessment Projects
Regression Findings
• Audio score does not significantly predict
variance in Teacher Ratings of Reading
Comprehension (beyond standard and
fluency) for Grade 8 RLD
• Audio score adds to prediction of reading
comprehension (beyond standard and fluency
scores) for three groups (NLD grade 4, NLD
grade 8, and RLD grade 4), but incremental
change is small
National Accessible Reading Assessment Projects
3. Are teachers’ able to predict which
students will benefit from read aloud?
Analyses:
• Analysis of variance in boost by teacher
predictions
• Cross-tabulations of teacher ratings by
degree of boost (more than on SEM, less
than one SEM, neither)
National Accessible Reading Assessment Projects
Accuracy of Teacher Prediction
For this study each student took a reading
comprehension test that was read aloud by a CD
player and another reading comprehension test
that they read to themselves. Which test do you
predict the student did better on?
Ⓐ Test read aloud by CD player
Ⓑ Test the student read to themselves
Ⓒ No difference
National Accessible Reading Assessment Projects
Findings from Teacher Predictions
• ANOVA indicated that on average teachers were
able to predict score gain from audio at grade 4
but not grade 8
• At the individual level teachers accurately
predicted if a student would benefit from the
audio version about 35% of the time and were
completely wrong about 5% of the time
National Accessible Reading Assessment Projects
DARA Psychometric Research
• Purpose of psychometric research: To help us
understand how an examinee's disability or the
accommodations he or she receives impacts the
psychometric properties of a reading test
National Accessible Reading Assessment Projects
Results of This Year’s Psychometric Analyses
• Psychometric Analyses
– Factor analyses
– Differential item functioning analyses
• Populations
– Students with learning disabilities who took the test with and
without accommodations
• Test
– Grade 4 and grade 8 English-language arts (ELA) assessment
• Focus
– Determine if the test measures the same constructs for
• Examinees without disabilities
• Examinees with disabilities who took the test with and without
accommodations
National Accessible Reading Assessment Projects
Number of Items for English-Language Arts Assessment
Test Content No. of Items
Reading Word Analysis, Fluency, and 18
Systematic Vocabulary
Development
Reading Comprehension 15
Literary Response and 9
Analysis
Total—Reading 42
Writing Writing Strategies 15
Writing Applications 1*
(Genres and Their
Characteristics)
Written and Oral English 18
Language Conventions
Total—Writing 34
*Essay item (all others are multiple-choice). The essay item was not used
in the study
National Accessible Reading Assessment Projects
Number of Items for Grade 8 English-Language Arts Assessment
Test Content No. of Items
Reading Word Analysis, Fluency, and 9
Systematic Vocabulary
Development
Reading Comprehension 18
Literary Response and 15
Analysis
Total—Reading 42
Writing Writing Strategies 17
Written and Oral English 16
Language Conventions
Total—Writing 33
National Accessible Reading Assessment Projects
STAR ELA Grade 4 and Grade 8 Summary Statistics
Grade 4 Total Groups Grade 8 Total Groups
Group N Mean SD N Mean SD
(1) Students 298,622 48 14 357,374 46 12
without
disabilities
(2) LD, without 9,045 29 12 18,512 29 10
accommodations
(3) LD, 504/IEP 4,724 27 10 4,325 27 9
accommodations
(4) LD, read- 1,367 29 11 874 27 9
aloud
accommodation
National Accessible Reading Assessment Projects
Factor Analyses of ELA Assessment
• Exploratory analyses (separately in each group)
– how many factors
• Confirmatory (multi-group)
– Establish base-line model
– Confirm number of factors needed to describe data
across all groups
National Accessible Reading Assessment Projects
Differential Item Functioning (DIF)
Analyses
• The purpose of this study was to examine
differential item functioning on the same English-
Language Arts assessment that was used for the
factor analyses
• “DIF is a statistical observation that involves
matching test takers from different groups on the
characteristic measured [by the test] and then
looking at performance differences on an item.”
(Sireci, 2006)
National Accessible Reading Assessment Projects
Method
• Mantel-Haenszel Categorization—3 Levels
– A Negligible DIF
– B Slight to Moderate DIF
– C Moderate to Large DIF
• Directions of DIF Flags
- Favors reference group
+ Favors focal group
National Accessible Reading Assessment Projects
Comparisons Made in the Study
Comparison
Number Reference Group Focal Group
1 Without disabilities LD
no accommodations
2 “ LD
IEP/504 accommodations
3 “ LD
read-aloud accommodation
(& IEP/504 accommodations)
National Accessible Reading Assessment Projects
DIF Categories ELA Grade 4 LD Without Accommodations
Easy 1.00
Percent Correct
.75
A
.50
B
.25
.00
-3.00 -2.00 -1.00 0.00 1.00 2.00 3.00
Difficult MH D-DIF
Favors Students Favors
Without LD Students
Disabilities
National Accessible Reading Assessment Projects
DIF Categories ELA Grade 4 LD With Accommodations
(IEP/504)
1.00
Easy
Percent Correct
.75
A
.50
B
.25
.00
-3.00 -2.00 -1.00 0.00 1.00 2.00 3.00
Difficult MH D-DIF
Favors Students Favors
Without LD (IEP/504)
Disabilities
National Accessible Reading Assessment Projects
DIF Categories ELA Grade 4 LD With Accommodations
(Read-Aloud)
Easy 1.00
Percent Correct
.75
A
.50 B
C
.25
.00
-3.00 -2.00 -1.00 0.00 1.00 2.00 3.00
Difficult MH D-DIF
Favors Students Favors
Without LD (Read-Aloud)
Disabilities
National Accessible Reading Assessment Projects
DIF Categories ELA Grade 8 LD Without Accommodations
Easy 1.00
Percent Correct
.75
.50 A
.25
.00
-3.00 -2.00 -1.00 0.00 1.00 2.00 3.00
Difficult MH D-DIF
Favors Students Favors
Without LD Students
Disabilities
National Accessible Reading Assessment Projects
DIF Categories ELA Grade 8 LD With Accommodations
(IEP/504)
1.00
Easy
Percent Correct
.75
.50 A
.25
.00
-3.00 -2.00 -1.00 0.00 1.00 2.00 3.00
MH D-DIF
Difficult
Favors Students Favors
Without LD (IEP/504)
Disabilities
National Accessible Reading Assessment Projects
DIF Categories ELA Grade 8 LD With Accommodations
(Read-Aloud)
Easy 1.00
Percent Correct
.75
A
.50 B
C
.25
.00
-3.00 -2.00 -1.00 0.00 1.00 2.00 3.00
Difficult MH D-DIF
Favors Students Favors
Without LD (Read-Aloud)
Disabilities
National Accessible Reading Assessment Projects
Interpreting the Results of the DIF Study
• Grade 4
– 1 C DIF item, 8 B DIF items
• Grade 8
– 1 C DIF item, 6 B DIF items
• Majority of flagged items were reading items that
favored students who took test with read-aloud
accommodation
• Consistent with Factor Analysis Results
National Accessible Reading Assessment Projects
Next Steps
• Psychometric Research
• Examination of Tailored Testing
• Cognitive Labs
• IEP decision making
National Accessible Reading Assessment Projects
Psychometric Research
National Accessible Reading Assessment Projects
Plans for Next Year’s Psychometric Analyses
• Psychometric analyses
– Factor Analyses
• Differential item functioning analyses
• Populations
– Students with learning disabilities who took the test with and
without an audio accommodation
• Test
– Gates-McGinitie Reading Test
• Focus
– Aid in interpretation of results of differential boost study
– Increase understanding of impact of disability and audio
accommodation on reading test scores
National Accessible Reading Assessment Projects
Factor Analyses We Plan to Carry Out
• Aid in interpretation of results of differential
boost study
– Compare factor structures for students without
disabilities who took test with and without
accommodation
– Compare factor structures for students with
disabilities who took test with and without
accommodation
National Accessible Reading Assessment Projects
Factor Analyses We Plan to Carry Out
• Increase understanding of impact of disability
and accommodation on reading test scores
– Compare factor structures of test given to examinees with
and without disabilities under standard conditions
– Compare factor structure of test given to examinees with
disabilities who take test with accommodations and
examinees without disabilities who take test without
accommodations
National Accessible Reading Assessment Projects
Purpose of Doing DIF and DDF Analyses
on Data From the Differential Boost Study
• Aid in interpretation of results of differential boost
study
• Increase understanding of impact of disability and
accommodation on reading test scores
National Accessible Reading Assessment Projects
Possible Comparisons for DIF Analyses
Summary of Possible Reference Group/Focal Group Combinations
Comparison Reference Group Focus Group
1 RLD Standard RLD Audio
2 NLD Standard NLD Audio
3 NLD Standard RLD Standard
4 NLD Standard RLD Audio
National Accessible Reading Assessment Projects
Procedures for Analyzing Data
• Differential Item Functioning: Mantel-
Haenszel
• Differential Distractor Analysis:
Standardized Distractor Analysis
National Accessible Reading Assessment Projects
Two Staged Tailored Testing
National Accessible Reading Assessment Projects
Operational Data
Percent Below Chance
Grade Disability Format Test Takers Items
4 LD Audio 17.4 21.3
Other 23.3 30.7
Standard 21.2 21.3
None Standard 2.2 1.3
8 LD Audio 20.0 32.0
Other 19.7 32.0
Standard 16.6 24.0
None Standard 1.5 0.0
National Accessible Reading Assessment Projects
GMRT Data
Percent Below Chance
Grade Disability Format Test Takers Items
4 RLD Audio 4.4 5.2
Standard 20.5 23.0
None Audio 1.5 1.1
Standard 2.6 2.1
8 RLD Audio 4.8 11.5
Standard 12.2 15.7
None Audio 0.2 1.1
Standard 1.1 1.1
National Accessible Reading Assessment Projects
DARA Tailored Testing Model
• Two (or three) stages of testing
• Students subtests on stage 2 are determined by performance
on routing test administered in stage 1
• Ideally computer administered but can be paper administered
• Some parts could be individually administered (e.g.,
decoding) if only a few students are routed into a decoding
measure and this format reduces the number of students
receiving individualize testing accommodations (e.g., read
aloud by human)
National Accessible Reading Assessment Projects
Reading
Comprehension
Routing Test
Extended Reading
Reading Fluency Comprehension
Test
Decoding and Extended Extended
Comprehension Comprehension
Test with Audio Test with Audio
National Accessible Reading Assessment Projects
Advantages of Model
• Score is more reliable estimate since items are targeted to
students ability level
• Students may feel less frustrated if they can do some of the
items on the routing test
• Teacher receives more information on low performing
students strengths and weaknesses
• Fundamental Skills and Comprehension are not confounded
for students with poor fundamental skills (some LD) or
poor comprehension (some LD and ELL)
• Growth can be more accurately measured in students
working significantly below (or above) grade level
National Accessible Reading Assessment Projects
Disadvantages of Model
• Requires computer administration or
teacher scoring of items after stage 1
• Students who are routed to fluency test
may be embarrassed
• Routing decision is made before test is
scaled or standard setting is completed
• Design could route more that 2% of
students to modified test
National Accessible Reading Assessment Projects
Questions for Year 3
• How many items (and of what difficulty)
are needed for an accurate routing test?
• Can we equate the audio extended and
standard extended using the routing test?
• What portion of students would be routed
to fluency measure and what portion
would be routed to decoding?
National Accessible Reading Assessment Projects
Questions for Year 3 (continued)
• Are the 2 alternate routes highly correlated with
the standard administration?
• What is the impact audio, fluency, and decoding
scores on total test score.
– If student is not a fluent reader should the total test score
be non-proficient?
• Is the routing test accurate for all students?
– Do some students do better on hard items?
– Do some students having trouble with the first few items
on the test?
National Accessible Reading Assessment Projects
Questions for Year 3 (continued)
• How should we weight different measures and
what impact will this have on subpopulations?
• Could we compose a tailored test from a state’s
current operational item pool?
– If not how many additional items would be required
and at what difficulty level?
National Accessible Reading Assessment Projects
Cognitive Labs
National Accessible Reading Assessment Projects
Background
• Cognitive labs using the think aloud method
on reading comprehension questions
• Build off the findings of last year’s large
scale differential boost study
– Gates MacGinitie Reading Comprehension Test
– Use items found in preliminary findings of the
DIF analysis of the GMRT data
National Accessible Reading Assessment Projects
Cognitive Labs Advantages
• Beneficial to learn about components of mental
processes of reading (Afflerbach & Johnston, 1984, Alavi,
2005; Pressley & Afflerbach, 1995)
• Beneficial in the development of assessments
(Caspar, Lessler, & Willis, 1999; Desimone & LeFloch, 2004; Willis,
2005)
• Open flexible procedure can be catered to the
specific situation and activity (Davison, Vogel &
Coffman, 1997)
• May use a small sample size
• Procedure has been successfully conducted
with children as young as 3rd grade (Laing & Kamhi,
2002; Paulsen & Levine, 1999; Trambasso & Magliano, 1996)
National Accessible Reading Assessment Projects
Cognitive Labs Disadvantages
• Thinking aloud is an unnatural step which may affect
or interfere one’s normal mental processes
• Students with disabilities may have difficulty with the
procedure (Johnstone, Miller, & Thompson, in press)
• Responses have the potential to be incomplete or
incorrect
– Lack of desire/motivation
– Embarrassment
– Inability to understand the task
National Accessible Reading Assessment Projects
Purpose of Study
This study is being conducted to serve the following
purposes:
1. How do students with and without reading-based
learning disabilities differ as they approach a
reading comprehension assessment?
2. Is this type of information gathering and data
quality worthwhile to conduct in future large scale
studies considering:
– Age of students
– Students with disabilities
National Accessible Reading Assessment Projects
Research Questions
1. In what way do students with reading-
based disabilities respond differently to
reading comprehension questions
compared to students without
disabilities?
2. What errors occur while reading the
passage/reading the items?
National Accessible Reading Assessment Projects
IEP Decision Making
National Accessible Reading Assessment Projects
IEP Decision Making
• What factors contribute to boost?
– Low standard score
– WJ Reading Measures
– Teacher Predictions
– Student Preference
National Accessible Reading Assessment Projects
Analyses planned
• Regression analyses to predict boost for
RLD students using
– WJ scores
– Standard score
– Use of read aloud in class or on tests
– Teacher predictions
– NJ ASK from prior year
National Accessible Reading Assessment Projects
