Get documents about "Redesign of Alternate Assessments for Students with Significant
W
Shared by: drr10525
Categories
Tags
alternate assessment, cognitive disabilities, alternate assessments, students with disabilities, achievement standards, grade level, special education, content standards, assessment design, universal design, new mexico, academic achievement standards, student performance, english language arts, iep team
-
Stats
- views:
- 5
- posted:
- 8/29/2010
- language:
- English
- pages:
- 49
Document Sample
Redesign of Alternate
Assessments for Students with
Significant Disabilities: National
and State Perspectives
CCSSO
June 16, 2008
1 Redesign of Alternate Assessments
Presentation Team
Moderator: Patty McDivitt, Data Recognition
Corporation
Aran Felix, Alaska Department of Education
and Early Development
Melissa Fincher, Georgia Department of
Education
Claudia Davis, Louisiana Department of
Education
Discussant: Rachael Quenemoen, National
Center for Educational Outcomes
2 Redesign of Alternate Assessments
Presentation Focus
Purpose and Rationale
Goals
Development
Challenges
Lessons Learned
3 Redesign of Alternate Assessments
Alaska
Aran Felix
Alaska Department of Education and
Early Development
4 Redesign of Alternate Assessments
Purpose and Rationale
Alaska’s previous Alternate Assessment
Portfolio assessment
Housed in the Special Education Unit
Assessed English/Language Arts, Math, and Skills for Healthy
Living (only E/LA and Math used for AYP)
Intended use for grades 3, 6, 8, 11 only
Stressed academic content
Alternate Performance Standards (content standards)
Used as Goals on student IEPs
Stressed inclusion (Generalization dimension of scoring)
Department of Education organized scoring sessions
Single set of achievement standards
Scoring dimensions: Skill (student achievement),
Generalization, Appropriateness
5 Redesign of Alternate Assessments
Purpose and Rationale
Steps to explore a solution:
Alternate Assessment moved to Assessment Unit
Teacher and Parent Survey conducted
Reliability-Validity Study conducted
Considered removing Generalization & Appropriateness
dimensions from scoring for AYP; report only to districts.
Needed a new standard setting and better overall technical quality
OSEP Condition drove development of a Data-folio assessment for
grades 4, 5, 7, 9, 10, Portfolio for all grades overwhelming
teachers.
Accomplished the original intent of inclusion and
teaching/assessing content standards
Moved scoring to test vendor
Passed Peer Review requirements
6 Redesign of Alternate Assessments
Goals
Assess a broader range of content standards
Manage test size and testing window
Increase standardization
content assessed (apples to apples)
training of teachers to administer the
assessment
scoring of the assessment
training of the protégés
Establish a higher technical quality
Create an online assessment (training,
scoring, reporting)
7 Redesign of Alternate Assessments
Development
1) Funding Issues
Small population of alternate-eligible
students
Request for Information
2) What type of assessment to develop?
Research other states’ approaches
Research using another state’s item bank
Decision needed: Retrofit portfolio with
performance tasks or move to a performance
task system?
8 Redesign of Alternate Assessments
Development
Look of new assessment
Used another state’s item bank while examining
alignment the first year
Performance Tasks in content areas
4-8 items per task
Paper/pencil materials allowing for some adaptation
Assessors administer one-on-one to student
Online test materials (scoring protocols, student
materials, training manuals)
Online training for test administrators plus proficiency
modules to maintain reliability
Online scoring, reporting, unofficial report
Original 3-year plan compressed to 2 years (peer review)
9 Redesign of Alternate Assessments
Development
Year 1: 2005-2006 (Portfolio still in place)
Developed Request for Proposals
Contracted with new vendor
Convened committees of educators
• Developed Proficiency level descriptors and
• Developed Extended Grade Level Expectations
Piloted online system with technology coordinators
• Created security access levels
• Developed online security rules
Trained Qualified Assessors for Pilot
Pilot tested the new assessment
Created a crosswalk of test items to Alaska content
standards (phase 1 blueprint)
10 Redesign of Alternate Assessments
Development
Year 2: 2006-2007
Developed cousin items
Conducted bias review
Conducted statewide trainings and orientations
• Developed a qualified assessor and qualified mentor-
trainer path and materials
• Included mentor pre-test and debrief audios
Administered assessment (one test)
Piloted science assessment
Conducted standard setting for RWM
Received Technical Report
Submitted new assessment to Peer Review (and received
approval)
AYP and student reports to parents/districts/website
11 Redesign of Alternate Assessments
Development
Year 3: 2007-2008
Developed cousin items and conducted bias review
Repackage all content tests into grade clusters vs. grade
level within content areas
New look for website
Trained Mentors on science assessment
Administered all assessments including science
Teacher Survey of Consequential Validity, Mentor Audios
Conducted standard setting for science
Conducted standard validation for
Received Technical Report
Submitted new assessment to Peer Review (and received
approval)
Piloted science assessment
AYP and student reports to parents/districts/website
12 Redesign of Alternate Assessments
Development
Future Plans for years 2008-2011
Develop larger item bank, conduct reviews
Field test new items
Construct new operational test forms, A & B
Conduct a new Standard Setting on Forms A and B of
each content-area assessment
Consider a program review by external evaluator
13 Redesign of Alternate Assessments
Challenges
Compressing the schedule from 3-4
years to 2 years
Possibility of administering two
Alternates simultaneously
Using another state’s items
―Alaskanizing‖ the assessment
Moving from a one-size model to a
grade-cluster model
14 Redesign of Alternate Assessments
Lessons Learned
The importance of flexibility and
responsiveness
Understanding capacity when huge
demands (such as compressed time
schedule) are imposed by second peer
review deadlines
The role of an outside consultant as EED
advisor for the first years of the project
Involvement of TAC prior to issuing RFP
15 Redesign of Alternate Assessments
Recommendations
RFP: get help from other states or hire a consultant; include any
customizing details; require experienced project manager; include
right to own test items
Link pattern of test development to general education cycle and
include content experts in all aspects of test development (standards
to test items)
Communicate with the field regularly through the process of
developing extended content standards. Provide these standards to
the field early.
If using any online pieces conduct pilot of online system to explore
platform issues and get sign-off from someone in charge at district
level
Exercise caution when updating online system during test window
Include an errata page on online system for notifying field of
updates
Have a backup server if you have online components
Remain cheerful!
16 Redesign of Alternate Assessments
Georgia
Melissa Fincher
Georgia Department of Education
17 Redesign of Alternate Assessments
Purpose and Rationale
Why did Georgia redesign our
alternate?
In a word: COMPLIANCE
In a few words: OPPORTUNITY TO
LEARN
18 Redesign of Alternate Assessments
Goals
The GAA is designed to ensure that
students with significant cognitive
disabilities are:
Provided access to the state-mandated
curriculum.
Given the opportunity to demonstrate
progress toward achievement of
curriculum knowledge, concepts, and
skills.
19 Redesign of Alternate Assessments
Development
The GAA is a portfolio of student work
provided as evidence that a student is
making progress toward grade-level
academic standards.
Evidence provided must show
instructional activities and student work
that is aligned to specific grade-level
standards.
20 Redesign of Alternate Assessments
Development
Georgia elected to go with a portfolio
format to allow flexibility for a very
diverse group of students.
The portfolio format allows the teacher,
who knows the student best, to design
and document instructional tasks that
are meaningful and purposeful for the
individual student.
21 Redesign of Alternate Assessments
Development
Collaboration between Assessment, Special
Education, and Curriculum
Significant investment in training of educators
surrounding curriculum access
Ongoing documentation of decisions
surrounding development and implementation
Involvement of Georgia’s Technical Advisory
Committee
Augmented with an AA-AAS expert
22 Redesign of Alternate Assessments
Challenges
Documenting technical quality
Traditional indices do not lend
themselves easily to alternate
assessments
Federal Peer Review
Complicated assessment program
Teacher buy-in
23 Redesign of Alternate Assessments
Lessons Learned
Investing in teacher training pays off, but it
needs to be continual as the assessment
evolves.
Technical documentation of alternate
assessments looks different, but it is worth
pursuing.
Validity is an ongoing journey.
Students are doing things we never thought
possible.
24 Redesign of Alternate Assessments
Louisiana
Claudia Davis
Louisiana Department of Education
25 Redesign of Alternate Assessments
Purpose and Rationale
Purpose: Louisiana’s alternate
assessment did not receive approval in
the USDOE peer review process.
26 Redesign of Alternate Assessments
Goals
Louisiana’s alternate assessment (LEAP
Alternate Assessment or LAA) lacked:
Academic focus (although linked to state
standards)
Alignment with grades or grade spans
USDOE advised a redesign of the LAA to
be implemented in spring of 2008
LDE began the redesign in July of 2007
27 Redesign of Alternate Assessments
Development
First, Louisiana developed Extended Standards (ES), i.e.,
extensions of state content standards.
What should students with significant cognitive
disabilities know and be able to do?
How do we identify expectations regarding the
breadth and depth of the standards, benchmarks, and
grade-level expectations (GLEs)?
How should students demonstrate knowledge and
skills based on the GLEs?
What is the appropriate range of tasks to be used to
measure this knowledge and these skills?
28 Redesign of Alternate Assessments
Development
LDE contracted with the testing vendor to develop Extended
Standards for ELA, Mathematics, and Science.
Recommendations from an initial committee of state special
educators regarding selected standards and a template for the
ESs, including Complexity Levels, were used to guide the
development.
Committees of Louisiana educators (general and special
educators) reviewed the proposed ESs with Complexity Levels.
Content-area groups across four grade spans (grades 3-4, 5-6, 7-8,
and 9-11)
Review was time intensive -- one intensive week
29 Redesign of Alternate Assessments
Development
Complexity Levels of the Extended Standards:
Each ES has 3 levels of complexity (1 being least
difficult) that provide access to general education
concepts and skills.
They serve as guidelines for the development of
assessment tasks at 3 levels of complexity.
Math Example:
3. Add and/or subtract to solve simple problems.
2. Identify simple addition and subtraction concepts within
daily living problems.
1. Count to solve simple problems.
30 Redesign of Alternate Assessments
SAMPLE PAGE AND KEY FOR ENGLISH LANGUAGE ARTS
Standard One: Students read, comprehend, and respond to a range of materials, using a variety of
strategies for different purposes.
Benchmarks Grade-Level Expectations Extended Standards Complexity Levels
ELA-1-E4: recognizing 8. Identify story elements, ES-8-5: Identify story 3. Identify the main character in a
story elements (e.g., including: elements, including: story
setting, plot, character, theme character 2. Identify two characters in a story
theme) and literary conflict 1. Identify one character in a story
devices (e.g., simile,
character traits, feelings, and
dialogue, personification)
within a selection motivation (ELA-1-E4)
5. Identify a variety of story 3 is most complex
elements, including:
the impact of setting on
character
multiple conflicts
first- and third-person points
of view
Extended Standard:
development of theme
(ELA-1-E4) O
ES-8 /5 refers to GLE 8 from grade 3
ES-8/O refers to GLE from grade 4
5
GLE from grade 3 (top) and grade 4 (bottom)
Development
Extended standards and complexity
levels were finalized in late
September.
Extended Standards Handbook (draft)
disseminated to school districts in
October.
Assessment development began.
32 Redesign of Alternate Assessments
Alternate Assessments
Grade Span Content Area
3 and 4 ELA, Math, Science
5 and 6 ELA, Math
7 and 8 ELA, Math, Science
High school
9 ELA, Math
10 ELA, Math
11 Science
33 Redesign of Alternate Assessments
Alternate Assessments
25 performance tasks per content and
grade span
Each task administered and scored by
the teacher
Tasks scored on a 0-1 point or 0-2
point scale using a rubric
34 Redesign of Alternate Assessments
Development
Alignment Study
Alignment of performance tasks to the
Extended Standards
Panel of eight expert independent reviewers
(4 state reviewers/4 national reviewers)
Based on Webb’s Depth-of-Knowledge
alignment process for use in aligning ES to
performance tasks (2007)
• Level 1: Recall of Information
• Level 2: Basic Reasoning
• Level 3: Complex Reasoning
35 Redesign of Alternate Assessments
Challenges
Time was the biggest challenge.
Short time frame: 8 months between first
development activity (July, 07) and administration
window for the assessment (February–March, 08)
Lack of adequate time for development activities
Lack of preparation time for teachers to provide
instruction and to prepare themselves and their
students for the assessment
Lack of time for adequate delivery of professional
development to district personnel
Lack of time to field test the items (2008 scores will
not be included in state accountability results for
schools and districts)
36 Redesign of Alternate Assessments
Challenges
Other challenges
Scheduling collaborative meetings with colleagues in
Special Populations on a short timeframe
Student IEPs completed based on former alternate
assessment
Necessary changes in Participation Criteria for AA
Decisions regarding
• Test format
• Accommodations for students
• Manipulatives for tasks
• Assistive technology
37 Redesign of Alternate Assessments
Lessons Learned
Adequate preparation time for students and teachers is a
MUST!
Adequate professional development is a MUST!
The assessment is not appropriate for all students with
significant cognitive disabilities (SCD).
Teachers of students with SCD are not so familiar with
standardized testing and need more training on
administration procedures and security.
Opinions from the field of the new assessment ranged
from ―loved it‖ to ―hated it‖ based primarily on their
students’ abilities to access the test.
Assistive technology personnel need to be included at the
beginning of the development phase.
38 Redesign of Alternate Assessments
To End on a Positive
Some Teacher Comments:
―I’m so pleased to have a curriculum (Extended
Standards) to guide my instruction.‖
―The new test is easier to administer.‖
― A good test for content.‖
―IEP goals will have to be rewritten to address
the new assessment.‖
―Great improvement.‖
―Great design!‖
―Pleasantly positively surprised.‖
―The test reflected what I teach.‖
39 Redesign of Alternate Assessments
National and State
Perspectives
…. And Next Steps
Rachel F. Quenemoen, Senior
Research Fellow, NCEO
National Center on Educational
Outcomes
40 Redesign of Alternate Assessments
NCEO STATE SURVEY REPORTS
2005 State Special Education Outcomes: Steps
Forward in a Decade of Change
2003 State Special Education Outcomes: Marching
On
2001 State Special Education Outcomes: A Report
on State Activities at the Beginning of a New
Decade
1999 State Special Education Outcomes: A Report
on State Activities at the End of the Century
Thompson & Thurlow (1999, 2001, 2003)
Thompson, Johnstone, Thurlow, & Altman (2005)
41 Redesign of Alternate Assessments
Content Addressed by Alternate Assessments:
Change Over Time
Year Fnct. Fnct. Stnds Exp/ Grade IEP Other Revising
skill skill Plus ext St level team
No Link Fnct. stnd* stnd** deter
link Stnds skills cntnt
Stnds
1999 16 --- 1 19 --- --- 24 ---
2000 9 3 7 28 --- --- 3 ---
2001 4 15 9 19 --- --- 3 ---
2003 2 --- 4 36 --- 3 3 2
2005 --- --- 1 21 10 1 7 10
*Category possibly included grade level standards prior to 2005
** Category introduced in 2005
2005 - Outcomes Measured by Rubrics on
Alternate Assessments
Number of Regular States
0 5 10 15 20 25 30 35 40 45 50
Skill/Competence_ 25 (40)
Level of Assistance 25 (32 independence)
Degree of Progress 23 (23)
Number/Variety of Settings 20 (21)
Alignment with Academic Content Standards 18
Ability to Generalize 15 (18)
Appropriateness 13 (20)
Staff Support 10 (20)
Social Relationships 10
Self Determination 9
Participation in General Education Settings 7 (12)
Support 7
(Numbers in parentheses from 2001)
Alternate Assessment Approaches 2000-2005
(from 2005 Survey)
Year Portfolio Rating IEP Other In Develop-
or Body of Scale or Analysis ment/
Evidence Checklist Revision
Regular States
1999 28 (56%) 4 (8%) 5 (10%) 6 (12%) 7 (14%)
2001 24 (48%) 9 (18%) 3 (6%) 12 (24%) 2 (4%)
2003 23 (46%) 15 (30%) 4 (8%) 5 (10%) 3 (6%)
2005* 25 (50%)** 7(14%)*** 2 (4%) 7 (14%) 8 (16%)
Unique States
2003 4 (44%) 0 (0%) 1 (11%) 1 (11%) 3 (33%)
2005 1 (11%) 1 (11%) 1 (11%) 0 (0%) 1 (11%)
**Of these 25 states, 13 use a standardized set of performance/events/tasks/skills.
***Of these 7 states, three require the submission of student work.
Flexibility and
Standardization
Nominal categories are NOT often useful for
characterizing the technical aspects of the assessment
(see Gong & Marion, 2006).
The evaluation of technical adequacy interacts with the
types of alternate assessments (i.e., choices/ degree
of flexibility-standardization) being employed.
This does NOT mean that standardization is good and
flexibility is bad—it all depends on purposes!
45 Redesign of Alternate Assessments
Where Are We Now?
Content coverage – National Alternate Assessment Center
work – University of Kentucky: Is it reading? Is it math? Is it
science?; University of North Carolina: Links for Academic
Learning; other methodologies for alignment.
Peer Review suggests great variability, near and far
linkages, but a steady trend is toward academic content.
Key questions: is it measureable bits or big ideas and
concepts or both? What is grade level for these students?
Scoring criteria and procedures – What does student
performance look like? Student vs. system? How do we
measure ―independence?‖ Who scores? Who checks? Trust but
verify? Flexibility vs. standardization issue.
Peer Review suggests great variability on this. Concerns
related to support, prompts, communications issues,
academics interaction.
46 Redesign of Alternate Assessments
Where Are We Now?
Part 2
Performance/achievement descriptors and standard setting
Is there achievement on the content?
Is the content clearly referenced?
How have accessibility issues been factored in?
What does independence mean? How good is good enough?
What should these students know and be able to do? How
well? This needs careful monitoring over time,
consequential validity studies.
Approach
Degree and logic of flexibility and standardization choices
Nominal categories are not particularly useful descriptors.
Unfortunately, “…the naked eye is drawn to test format” not
educational soundness (Baker, 2007)
47 Redesign of Alternate Assessments
More or Less Than Meets the
Eye?
BECAUSE of the number of uncertainties
still in play, we need:
Transparency
Integrity
Consequential validity studies
Planned improvement over time
48 Redesign of Alternate Assessments
Questions
49 Redesign of Alternate Assessments
Related docs
