Piedmont Elementary Evaluation of PLATO Focus

CREP Center for Research in Educational Policy An Evaluation of the Effects of FOCUS on First Grade Reading Achievement in a Title I Elementary School Anna Grehan Steven M. Ross Center for Research in Educational Policy University of Memphis 325 Browning Hall Memphis, TN 38152 Toll Free: 1-866-670-6147 August 2004 _____________________________________________________________________ EXECUTIVE SUMMARY Purpose The purpose of this study was to evaluate the implementation and effects on student achievement of the PLATO FOCUS Reading and Language Program in the Piedmont City (AL) School District first grade classes. FOCUS was used by the school to align their core reading program to Reading First guidelines. The evaluation examined (a) student reading achievement on the Dynamic Indicators of Basic Early Literacy Skills (DIBELS), (b) classroom teaching strategies using FOCUS relative to Reading First elements and literacy best practices, and (c) teacher, principal, and administrator attitudes and experiences with FOCUS. Research Questions The primary research questions were: 1. 2. 3. 4. What are the influences of FOCUS on student literacy achievement? What literacy teaching practices are used in FOCUS classrooms? What are teacher reactions to FOCUS? What are principal and administrator reactions to FOCUS? Method Participants The participants in this study were drawn from three first grade classes and one pre-first class during the 2002-2003 and 2003-2004 academic years from Piedmont Elementary in Piedmont Alabama. In 2003-2004, Piedmont was a K-5 school serving approximately 485 students. Instruments Dynamic Indicators of Basic Early Literacy Skills (DIBELS). DIBELS is a tool for early identification of children with potential literacy problems and an assessment of response to instruction. The DIBELS assessment is designed to enable educators to modify their approach if a student is not on course to achieve reading goals (Good & Kaminski, 2003). Literacy Observation Tool (LOT©). The LOT is an instrument for observing in elementary classrooms where teachers are engaged in teaching reading and other reading practices (Smith, Ross, & Grehan, 2002). The LOT was designed to assist schools in evaluating the effectiveness of teacher implementation of research-based reading strategies. Focus Groups. The research team constructed a structured focus group protocol to be used with all participating teachers, administrators, and reading coach. Design The research design included both qualitative and quantitative methods. Through the use of observations, focus group, and interview protocols, data were collected to examine those research questions that addressed issues related to the reactions of teachers, the principal, Evaluation of Focus 1 and other staff to the FOCUS program and to the effects the program had on teaching practices in the classroom. A quasi-experimental design was utilized in which DIBELS scores on the Phoneme Segmentation Fluency (PSF) and the Nonsense Word Fluency (NWF) subtests from first graders participating in the FOCUS program in 2003-2004 were compared to subtest scores of first graders in 2002-2003 (Control cohort), before FOCUS implementation. Results of Quantitative Analysis The results from the repeated measures mixed design ANOVA indicate that the change in pretest and posttest scores for both PSF and NWF was significant. The highly significant results suggest that the change in scores between the fall and the spring testing periods was large and meaningful. Significant interactions between the two subtests and control and treatment groups were also found. Additional follow-up independent t-tests showed significant differences between the Control and FOCUS treatment group posttest scores. Significant differences were not found between control and treatment pretest scores. The follow-up paired t-tests ES’s for PSF (+1.51) and NWF (+2.18) were significantly larger for FOCUS and reflect impressive growth in posttest scores. Additionally, the mean fall scores for both subtests for FOCUS were slightly lower than the mean fall scores for the Control cohort. In other words, the first graders with FOCUS began the year with slightly lower average scores, yet ended the year with higher scores than those of the Control cohort. Results of Qualitative Analysis Teacher directed FOCUS lessons were primarily coded as phonemic awareness instruction during the three days of LOT observations. All FOCUS activities were oral and the only materials utilized were keyword pictures and card scripts. Every first grade teacher supported FOCUS classroom instruction with the computer software application during daily learning center activities. The four teachers participating in the focus group agreed that the FOCUS program was an important supplement to the primary core reading program. The teachers reported that they were able to use FOCUS to reinforce concepts of phonics and phonemic awareness. The teachers reported that the computer application had proven simple to use and fairly stable. The teachers believed that their students enjoyed working on the computer and the program was flexible enough to allow each teacher to schedule student access differently. The teachers appeared to be satisfied with the professional development they had received. The principal, superintendent, and other administrators all viewed FOCUS as an important tool for improving reading instruction at Piedmont and as a natural addition to the district’s current PLATO Learning initiative. All of the administrators expressed satisfaction with the training provided to the district and believed that the implementation of the program into the broader curriculum had been effective. Conclusion Given that 2003-2004 was the implementation year of FOCUS, this suggests that the students’ interactions with the FOCUS curriculum, supplementing the core reading program, may have had a positive impact on the student’s two DIBELS subtest scores. Evaluation of Focus 2 _____________________________________________________________________________________ An Evaluation of the Effects of FOCUS on First Grade Reading Achievement in a Title I Elementary Schooll Overview The purpose of this study was to evaluate the implementation and effects on student achievement of the PLATO FOCUS Reading and Language Program in the Piedmont City (AL) School District first grade classes. The evaluation examined (a) student reading achievement on the Dynamic Indicators of Basic Early Literacy Skills (DIBELS), (b) classroom teaching strategies using FOCUS relative to Reading First elements and literacy best practices, and (c) teacher, principal, and administrator attitudes and experiences with FOCUS. FOCUS Reading and Language Program The FOCUS program is an integrated reading and language curriculum that provides a balanced approach to literacy learning. The program provides a combination of teacher directed lessons, interactive computer sessions, and daily practice to teach students phonemic awareness, the alphabetic principle, sound and spelling relationships, vocabulary instruction, and oral reading fluency in clearly defined sequences. FOCUS consists of 45 units/lessons, one each for the 45 fundamental sounds (phonemes) of the English language. Lessons are presented in a series of seven Cards that progress from basic reading to complex skills. Suggested teacher scripts are provided for each task on the Cards. All components within each lesson help students focus on the unit’s target sound, reinforcing phonemic awareness of that sound. The Evaluation of Focus 3 program can be tailored to grade level by adjustments in the sequence and amount of material covered and the level of instructional support provided. This flexibility allows FOCUS to supplement other reading and language programs if needed. The FOCUS program is designed to be presented during a typical 180-day school year with each of the 45 units able to be incorporated into an average reading block. Teachers may choose one or more of the individual activities as appropriate. This is supplemented by software activities which can be done 2-4 times a week. The developer describes FOCUS as based on a research foundation grounded in reading research literature, having been examined through the use of control group field studies, and having been reviewed by reading and language experts. The developer states that the program meets the Reading First and No Child Left Behind definition of “scientifically-based” reading programs (PLATO Learning, Inc., 2002). Study Rationale With the passage in 2002 of the federal legislation, No Child Left Behind (U.S. Congress, 2001), increased focus has developed for identifying “proven” practices that demonstrably raise student achievement. The U.S. Department of Education’s Reading First program provides the necessary assistance to states to establish research-based reading programs for students in kindergarten through third grade. Scientifically-based reading research has identified five essential components of reading instruction. These components are: (1) Phonemic Awareness – The ability to hear, identify, and manipulate phonemes (individual sounds) in spoken words, (2) Phonics – The understanding that there is a predictable relationship between the sounds of spoken language and the letters and spellings that represent those sounds, Evaluation of Focus 4 (3) Vocabulary Development – The development of stored information about the meanings and pronunciation of words, (4) Reading fluency, including oral reading skills – The ability to read text accurately and quickly, and (5) Reading comprehension strategies – The strategies for understanding, remembering, and communicating with others about what has been read. Research demonstrates that children need to master skills in these five interrelated areas in order to become proficient, successful readers. The Reading First program focuses instructional methods and materials, assessments, and professional development on these key areas. The program provides assistance to states and districts in selecting and developing effective instructional materials, programs, learning systems, and strategies to implement methods that have been proven to teach reading. Reading First also provides assistance for the selection and administration of screening, diagnostic, and classroom based instructional reading assessments with a proven validity and reliability in order to measure where students are and monitor their progress (U.S. Department of Education, 2002). Research Questions The primary research questions were: 1. What are the influences of FOCUS on student literacy achievement? 2. What literacy teaching practices are used in FOCUS classrooms? 3. What are teacher reactions to FOCUS? 4. What are principal and administrator reactions to FOCUS? Evaluation of Focus 5 Method School Description Piedmont Elementary is located in a small town in rural east central Alabama. The school traces its history back to 1905 when a local mill established a private, one teacher school for mill employees’ children. A public school, named Southside Elementary, was established in 1958 and provided grades 1-6 to the community. In 2001-2002, in response to continued district growth, students, faculty, and staff transferred to a new school building renamed Piedmont Elementary. In 2003-2004, Piedmont was a K-5 school serving approximately 485 students. While the principal reported that teacher and student mobility was generally low, due to local industrial closures, enrollment has declined from the highs of previous years as residents left to seek new employment opportunities. Approximately 18% of the school’s population are minorities with black students accounting for 15% of the total. Over one half of the students qualify for free or reduced price lunch, and the school operates a school wide Title I program. The school follows a traditional school calendar and it has 26 certified teachers. The Stanford 9 results from 2003 indicated that over 70% of the students scored as “Proficient.” Piedmont is part of the Alabama Reading Initiative and receives funding for a reading coach through this grant. Participants The participants in this study were drawn from three first grade classes and one pre-first class during the 2002-2003 and 2003-2004 academic years. The three regular first grade teachers were the same in both years; however, there was a different pre- Evaluation of Focus 6 first teacher in 2003-2004. In 2002-2003 there were 61 students in these 4 classes, while in 2003-2004 the number of students had grown to 75. Demographic data on the 2003-2004 classes were available from the DIBELS testing as required by No Child Left Behind. In 2002-2003, whites made up 78% of the first grade class, African-Americans were 19% of the students, and 3% were other minorities. Of the combined pre-first and first grades 8% were listed as special education. During 2003-2004, 78% of the first grade students were white, while 17% were African-American, and the remaining 5% of the students were classified as other minorities. Approximately 13% of the students were listed as having learning disabilities or speech/language impairments. As the demographic data indicates, both years’ classes were very comparable in composition. In addition, as presented in Tables 3 and 4, the mean fall semester scores on the DIBELS Nonsense Word Fluency subtest were very similar with the 2002-2003 class scoring 25.23 while the 2003-2004 class achieved 24.8. Instruments Dynamic Indicators of Basic Early Literacy Skills (DIBELS). DIBELS is a tool for early identification of children with potential literacy problems and an assessment of response to instruction. The DIBELS assessment is designed to enable educators to modify their approach if a student is not on course to achieve reading goals. The Early Childhood Research Institute on Measuring Growth and Development at the University of Oregon constructed DIBELS. The Institute reports it has validated the instrument’s ability to predict outcomes, and has tested its reliability with young children across the country. Evaluation of Focus 7 DIBELS was designed to fulfill two purposes: monitoring. screening and progress- The screening function begins in kindergarten, before students have learned to read words. It occurs in the fall, mid-winter and spring of each year through the third grade. The progress-monitoring components are used selectively with at-risk children on a week-to-week basis, if necessary, to determine how well they are progressing toward a goal. DIBELS consists of seven subtests and most can be used for either benchmark or progress-monitoring assessments. Different subtests are administered depending the time of year and the grade. Benchmark assessments are given to all children in a grade three times a year, while progress-monitoring assessments are used electively. The seven subtests are: 1. Initial Sound Fluency (Kindergarten) – This subtest measures the child’s ability to identify, isolate, and pronounce the first sound of an orally presented word. 2. Letter Naming Fluency (Kindergarten to Grade 1) – This subtest asks students to name as many letters, both uppercase and lowercase randomly mixed, as they can in one minute. 3. Phoneme Segmentation Fluency (Mid-Kindergarten to end of Grade 1) – This subtest is a direct measure of phoneme awareness. Students are asked to say the individual sounds that make up a word or syllable containing three or four phonemes. 4. Nonsense Word Fluency (Mid-Kindergarten through beginning Grade 2) – This subtest measures a student’s ability to link letters with sounds and use that knowledge to decode three-letter syllables that alone are nonsense words. 5. Oral Reading Fluency (Mid-Grade 1 to Grade 3) – This subtest includes benchmark passages at each grade level that are used to measure accuracy and speed in reading graded passages. 6. Oral Retelling Fluency (Mid-Grade 1 to Grade 3) – This assessment asks the student to tell as much as they can about a passage that they are asked to read. Evaluation of Focus 8 7. Word Use Fluency (Fall of Kindergarten through Grade 3) – This subtest is designed to assess vocabulary knowledge and expressive language for students in each grade level. Scores provided by the subtests can provide teachers and administrators with indicators of whether the student is at a low risk, at risk, or some risk of achieving subsequent outcomes (administered prior to the benchmark goal). If the measure is administered at the time or after the benchmark is set, the level of performance may be either established, deficit, or emerging. Educators can use these indicators as guides for developing tailored intervention strategies to assist individual students to reach their reading goals (Good & Kaminski, 2003). Literacy Observation Tool (LOT©). The LOT was developed by researchers at the Center for Research in Educational Policy, College of Education, The University of Memphis, and is an instrument for observing in elementary classrooms where teachers are engaged in teaching reading and other reading practices (Smith, Ross, & Grehan, 2002). The LOT was designed to assist schools in evaluating the effectiveness of The LOT has been teacher implementation of research-based reading strategies. aligned to the National Reading Panel and National Research Council findings and captures all five essential components of the Reading First program. Individual observations and the summary are organized around the following six categories: Instructional Orientation, Instructional Strategies, Assessment, Learning Environment, Visible Print Environment, and Materials Used. The subcategories of Instructional Strategies include the five essential components of reading identified by the National Reading Panel as important in achieving effective reading instruction: Phonemic Awareness, Phonics, Fluency, Text Comprehension, and Vocabulary. The LOT Evaluation of Focus 9 captures explicit instruction in these five component areas. Quantitative data yielded by the observations reflect the degree to which targeted components are occurring in classrooms. The rubric ratings range from 0 = Not observed, to 1 = Rarely, 2 = Occasionally, 3 = Frequently, and 4 = Extensively observed. The components of the LOT were specifically matched to the topic areas identified by the National Reading Panel as derived from “scientifically-based” reading research. The Crosswalk- on the following page- provides detailed information on the LOT’s alignment to the appropriate topic areas. Evaluation of Focus 10 Crosswalk Between Major Reading Initiatives and LOT Observations National Research Council (1998). Oral Language and Reading Concepts of Print, Letter Naming, and Phonemic Awareness. National Reading Panel (1999). Alphabetics Phonemic Awareness Instruction. Reading First (2001). Literacy Observation Tool – LOT (2002) Phonemic Awareness Instruction. Phonemic Awareness Instruction. Letter Knowledge. Rhyming Concepts of Print. Explicit Phonics Instruction. Fluency Instruction. Vocabulary Instruction. Text Comprehension Instruction. Phonics and Word Recognition Accuracy. High-Frequency Words and Fluency. Strategic Comprehension. Phonics Instruction. Fluency – Guided Oral Reading. Text Comprehension. Vocabulary Instruction. Teacher Preparation and Comprehension Strategies Instruction. Teacher Education and Reading Instruction. Computer Technology and Reading Instruction. Phonics Instruction. Fluency Instruction. Vocabulary Instruction. Text Comprehension Instruction. Writing and Reading. Writing, Spelling, and Handwriting. Language Mechanics. Independent Silent Reading Independent Silent Reading. Engagement and Interest in Reading. School-Wide Reading Programs. Instructional Orientation. Learning Environment. Visible Print Environment. Materials Used. Screening, Diagnostic, Monitoring, and Outcome Formal and Informal Reading Assessment. Evaluation of Focus 11 There are two forms for recording observations. The Literacy Observation Tool Notes form is completed for each teacher observed. At the conclusion of the observation period, the individual notes forms are then synthesized and summarized on the Literacy Observation Tool Data Summary form. The standard LOT procedure involves a trained observer visiting 7-9 randomly selected classrooms, for 10 minutes each, during the typical 1½ to 2 hour reading/ literacy block. A recent reliability study using Generalizability Theory (Sterbinsky & Ross, 2003) indicated that LOT observations have a reliability of .75 when at least five LOTs are completed in a school. Additionally, the validity of the LOT was assessed via a content validation process in the development phase, including the use of a panel of subject matter experts, both researchers and practitioners, from areas such as the University of Memphis, the Memphis City Schools, and the state departments of education in Tennessee, Louisiana, and Illinois. To ensure the reliability of data, observers receive a manual that describes and operationally defines reading variables that comprise the LOT, and participate in formal training to ensure that the identifying and coding of reading variables occurs in a consistent manner to guarantee the reporting of reliable and valid data. After receiving the manual and instruction in a group training session, each observer participates in sufficient practice exercises and an inter-rater reliability consensus rating process to ensure that his/her data are comparable with those of experienced observers. The LOT was piloted in the Memphis City Schools and is used in multiple schools across the United States, including 70 Reading Excellence Act (REA) and 56 Reading First (RF) schools in the state of Tennessee. The REA and RF evaluations and Evaluation of Focus 12 research studies included rigorous matched control quasi-experimental designs for which the LOT was the primary outcome measure. Teacher Focus Group. The research team constructed a structured focus group protocol to be used with all participating teachers. A copy of the protocol appears in Appendix A. It was designed to encourage the teachers to address topics related to (a) their experiences with the FOCUS program; (b) the nature and adequacy of professional development and resources available; (c) implementation demands; (d) perceived advantages and disadvantages relative to formerly used programs; (e) perceived impacts on children with regard to motivation, interest, and learning; and (f) suggestions for improvement. Administrator/Counselor/Reading Coach Focus Group. A focus group protocol was constructed to capture perceptions additional stakeholders. The protocol was similar to that used with the teachers and asked individuals to provide their feedback on topics related to implementation and the advantages and disadvantages of the FOCUS program. Principal Interview. formal. The interview process used with the principal was less This interview was designed to collect the principal’s perceptions of the implementation of FOCUS and its effects on both faculty and students. Design The research design included both qualitative and quantitative methods. Through the use of observations, focus group, and interview protocols, data were collected to examine those research questions that addressed issues related to the Evaluation of Focus 13 reactions of teachers, the principal, and other staff to the FOCUS program and to the effects the program had on teaching practices in the classroom. To collect the qualitative data, a team of two University of Memphis researchers visited the school in January 2004 for three days. The researchers were expert in conducting LOT observations and in the application of Reading First and evidencebased literacy practices. Each researcher observed teachers during the reading block each day. Visits to First Grade teachers’ classrooms were made at random and Researchers used teachers were asked to use FOCUS as they would normally. “targeted” LOT observations that allowed them to remain in the classroom for the entire reading period. The use of a “targeted” process allowed the researchers to observe how the FOCUS computer application was integrated into the reading lesson. During the afternoons of the site visit, the researchers conducted the focus groups and the principal interview. In addition, the researchers reviewed the FOCUS program General researcher observations are presented components and related materials. before detailed LOT results. A quasi-experimental design was utilized in which DIBELS scores on the Phoneme Segmentation Fluency (PSF) and the Nonsense Word Fluency (NWF) subtests from first graders participating in the FOCUS program in 2003-2004 were compared to subtest scores of first graders in 2002-2003 (Control cohort), before FOCUS implementation. The PSF and NWF subtests were chosen for analysis because only these two subtests are given in the fall and repeated in the spring during the First Grade. The PSF and NWF subtests have similar structures. The score is the number of correct letter-sounds or phonemes produced in one minute. The measures take about two minutes to administer Evaluation of Focus 14 and have 20 alternate forms for monitoring progress (Sopris West Educational Services, 2003). First grade teachers began DIBELS progress monitoring in the 2003-2004 school year. Repeated-measures mixed design analysis of variance (ANOVA) were conducted using 2002-2003 and 2003-2004 DIBELS subtest scores on the PSF and NWF to determine if the scores were significantly different in the spring (posttest) from the scores recorded in the fall (pretest) for each academic year cohort of first graders. In addition, follow-up tests were performed to evaluate the relationships of significant interactions at a .05 level between the subtest scores and the year the subtests were taken. A repeatedmeasures analysis was selected because scores were available for the two subtests from two different times periods for both academic year cohorts. A summary of the instrumentation relative to the research questions is provided in Table 1. Table 1 Data Instruments Applicable to Each Research Question Research Question 1. What are the influences of FOCUS on student literacy achievement? 2. What literacy teaching practices are used in FOCUS classrooms? • • • • • • • • • • Data Instrument(s) DIBELS 2003-2004 (FOCUS) and 20022003 (control) student-level test scores LOT Observations. Teacher focus group. FOCUS student data. Administrator focus group. Teacher focus group. Principal interview. Administrator focus group. LOT observations. Principal interview. 3. What are teacher reactions to FOCUS? 4. What is the principal’s reaction to FOCUS? Evaluation of Focus 15 Results The results of the study are presented below by instrument. General Researcher Observations First grade teachers at Piedmont Elementary were observed during their reading block on three consecutive days in January 2004. Five teachers were observed: three regular first grade classrooms and one pre-first and special education classroom. Students in these classes (with the exception of the pre-first class) were regrouped for reading based upon reading ability. The first grade-reading block was from 11:30 a.m. to 1:00 p.m. The pre-first reading block was divided in half, with approximately one hour in the morning and one hour in the afternoon devoted to literacy centers. Classrooms were attractive, clean, and print rich. There was extensive evidence of the school’s first grade core reading program, Action Reading, posted in the classroom. All first grade classrooms had libraries, multiple trade books, and basal textbooks available for use. Classes also had access to a new computer lab in the building. Refer to Appendix B for first grade classroom photographs. The three regular first grade classrooms followed the same general format for the 90-minutes of reading instruction. Teachers spent approximately 60-minutes in a whole-group, direct instruction format followed by 30-minutes of center activities. Whole-group activities included a teacher read aloud, explicit phonics instruction utilizing the Action Reading program, and FOCUS lessons. It was evident that teachers attempted to align their Action Reading phonics skills with FOCUS lessons; however teachers did not have authentic literature for students to support concepts as they were being taught. Teacher-created reading materials aligned to direct instruction concepts Evaluation of Focus 16 were observed in every classroom. Teachers utilized reading selections written on large chart tablets and teacher-made stories on handouts to reinforce concepts and provide practice. Basal texts, decodable, and leveled books were not observed in use during the direct instruction portion of the reading class. The three first grade and pre-first teachers utilized the teacher directed FOCUS lessons daily during the three-day observation period. Time devoted to lessons was approximately 15-minutes and all were exclusively oral. FOCUS lessons appeared to supplement the existing core-reading program, Action Reading, in regular first grade classrooms. During observations, teachers utilized Cards 1 (Get in FOCUS), 2 (FOCUS on Words), and 3 (FOCUS on Sentences) of lesson packets. Lesson emphasis appeared to be weighted toward phonological awareness activities (Card 2). However, teachers tended to utilize scripts and follow the sequence of activities as presented on lesson cards. The Action Reading program, which has a strong phonics emphasis, was often presented first to students followed by the FOCUS lesson. All teachers were organized, well prepared, and utilized FOCUS lesson materials such as picture cue cards. Children were actively engaged in lessons and appropriate pacing of the instruction was observed. During the learning center portion of the reading block, students rotated between four to five literacy activities. Time devoted to each center activity varied from class to class. In one first grade classroom students rotated centers every 15-minutes, while in another classroom, they stayed in their assigned center for the entire 30-minutes. The pre-first classroom had center time in the afternoon for one hour. Every first grade classroom designated computers as a center and utilized the FOCUS computer Evaluation of Focus 17 application exclusively. All classrooms had three new networked Dell computers and one or two older computers. classrooms computer centers. Three to four students utilized the FOCUS computer application during center time. Teachers had the program turned on and children easily “clicked” on their name with the mouse to begin the program. Students used the mouse to operate the program and appeared comfortable and at ease in navigating the FOCUS program. No students were observed asking for assistance with the program or computer. However, teachers were rarely observed monitoring student activity at the computer. Most often teachers were observed in small group instruction or assisting students in other centers. During the classroom observations, the computer lessons on which students worked varied by student and did not necessarily align with classroom instruction. For example, even though a teacher introduced the “v” sound in the FOCUS lesson, students were observed to be on a variety of other computer sound activities. Some students also appeared to have learned how to manipulate the program by repeatedly clicking until the correct response was found. This allowed the students to spend Refer to Appendix B to view the four first grade extended time on the “paint” page. Teachers were observed prompting students to “move off” the paint page. Literacy Observation Tool (LOT) As indicated in the description of the LOT, the observation procedure primarily focuses on six categories of the basic elements of literacy instruction: Instructional Orientation, Instructional Components, Assessment, Learning Environment, Visible Print Environment, and Materials Used. Ratings of observed activities are made using a five Evaluation of Focus 18 point rubric (0 = not observed, 1 = rarely, 2 = occasionally, 3 = frequently, and 4 = extensively). In the following descriptive analysis, percent frequently and extensively were combined and reported. More precise data regarding observed literacy practices measured by the five-point rubric for the LOT observations are presented in Table 2. Evaluation of Focus 19 Table 2 Literacy Observation Tool (LOT©) Data Summary Piedmont Elementary School Number of Observation Visits for 2003-2004 N= 8 Note: One school observation visit equals approximately 7-10 individual classroom visits. The extent to which each of the following is present in the school: Percent Not Observed Percent Rarely Percent Occasionally Percent Frequently Percent Extensively INSTRUCTIONAL ORIENTATION Small group Whole class Learning centers Cooperative/collaborative learning 37.5 25.0 37.5 75.0 25.0 0.0 0.0 25.0 12.5 0.0 12.5 0.0 0.0 50.0 37.5 0.0 25.0 25.0 12.5 0.0 INSTRUCTIONAL COMPONENTS Reading - The Teacher: Concepts of Print Book/print conventions Alphabetics Letter naming/knowledge Phonemic awareness instruction Rhyming Explicit phonics instruction Fluency Models fluent oral reading Has students read/reread orally together Vocabulary Introduces/reviews key vocabulary Explicit vocabulary instruction Text Comprehension Explicit comprehension strategy instruction Makes connection to prior knowledge Asks students for predictions Uses higher level questioning Guides visual imaging Guides interactive discussion Reads self-selected materials 62.5 0.0 75.0 0.0 50.0 12.5 100.0 100.0 87.5 50.0 75.0 75.0 87.5 75.0 75.0 37.5 50.0 12.5 25.0 12.5 25.0 0.0 0.0 12.5 50.0 25.0 25.0 12.5 25.0 12.5 0.0 0.0 12.5 50.0 25.0 37.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 37.5 0.0 25.0 12.5 25.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 12.5 0.0 12.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 0.0 0.0 0.0 Independent Reading - The Student: Evaluation of Focus 20 Table 2 (continued) Literacy Observation Tool (LOT©) Data Summary Piedmont Elementary School The extent to which each of the following is present in the school: Percent Not Observed Percent Rarely Percent Occasionally Percent Frequently Percent Extensively INSTRUCTIONAL COMPONENTS, continued Letter formation/handwriting Writing process Language mechanics lessons Conferences with students Provides for students' sharing Writes independently Response writing Writing - The Teacher: 100.0 0.0 100.0 100.0 100.0 100.0 87.5 62.5 87.5 100.0 87.5 12.5 0.0 0.0 12.5 0.0 0.0 37.5 0.0 37.5 75.0 87.5 75.0 25.0 50.0 75.0 100.0 100.0 12.5 75.0 50.0 0.0 0.0 0.0 0.0 0.0 25.0 12.5 0.0 0.0 37.5 0.0 0.0 37.5 0.0 0.0 12.5 0.0 25.0 0.0 12.5 12.5 0.0 12.5 12.5 0.0 0.0 0.0 0.0 50.0 0.0 0.0 0.0 0.0 0.0 12.5 12.5 0.0 0.0 0.0 12.5 0.0 12.5 0.0 0.0 0.0 37.5 0.0 0.0 25.0 0.0 12.5 25.0 0.0 12.5 0.0 0.0 12.5 12.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 12.5 12.5 87.5 25.0 12.5 0.0 0.0 12.5 12.5 12.5 0.0 0.0 0.0 37.5 12.5 0.0 0.0 0.0 37.5 12.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 25.0 12.5 62.5 37.5 100.0 100.0 0.0 87.5 25.0 0.0 0.0 0.0 12.5 25.0 0.0 0.0 0.0 25.0 0.0 0.0 Writing - The Student: ASSESSMENT Formal testing Portfolios IRI, running records LEARNING ENVIRONMENT Conducive to cooperative interactions Students actively engaged Effective classroom management Teacher actively monitors VISIBLE PRINT ENVIRONMENT Alphabet Word wall Labeling (names, objects, areas) Classroom library Evidence of student writing/work products MATERIALS USED Basal texts Big books Books on tape Computers Fiction books Non-fiction books Poetry Newspaper/magazines Word/vocabulary materials Worksheets/workbooks Other Evaluation of Focus 21 Instructional Orientation. The primary Instructional Orientation utilized in first grade classrooms was direct instruction with the whole class (75%). Learning centers were noted in 50% of classroom observations, whereas small group instruction and cooperative/collaborative learning were utilized significantly less often at 25% and 0%, respectively. Instructional Components. The beginning reading and readiness activities most often observed were phonemic awareness instruction (50%) and explicit phonics instruction (25%). Concepts of print, letter naming/knowledge, and rhyming activities were rarely observed (0% frequently and extensively) during classroom observations. Teacher modeling of fluent oral reading was observed in 12.5% of classroom observations and students’ reading/rereading together was observed 25% of the time. No vocabulary or text comprehension activities were observed occasionally, frequently, or extensively. Students reading self-selected materials and writing activities were also rarely observed, with 12.5% and 0%, respectively, of classroom observations characterized as frequently and extensively. Assessment. block. (12.5%). Learning Environment. The LOT observations suggest a positive learning Assessment activities were rarely observed during the reading The only assessment activity observed was DIBELS progress monitoring environment in first-grade classrooms. Students were rated as being actively engaged during 100% of classroom observations and effective classroom management was observed in 87.5% of classroom visits. Evaluation of Focus 22 Visible Print Environment. Print-rich environments were observed in all first grade classrooms. All (100%) of classrooms had classroom libraries and displayed the alphabet and words walls. Student work was displayed in 37.5% of classrooms. Materials Used. Materials most often utilized during reading classroom observations were word/vocabulary materials (62.5%), computers (50%), fiction books (37.5%), and worksheets/workbooks (12.5%). Materials rarely used were basal texts, big books, books on tape, non-fiction books, poetry, and newspaper/magazines. Principal Interview General Information. The principal reported that Plato Learning was first established at the high school seven years ago, then the middle school, and finally, last year at the elementary school. He noted that FOCUS is a new program provided by PLATO Learning and teachers received training for the program in summer 2003. He also stated that kindergarten, pre-first, first, and second grade teachers use the program. According to the principal, the district purchased the program, provides technical support, and the district curriculum coordinator coordinated training for the FOCUS program. The principal felt his proudest accomplishment, as principal at Piedmont, is the transformation of teachers as intelligent consumers of data. “Teachers know where their children are.” He viewed technology as a means to provide individualized instruction and monitor progress for every child. Integration of FOCUS into school’s literacy program. The principal described FOCUS as another “tool in our tool box.” He reported that Piedmont has been a part of the Alabama Reading Initiative (ARI) for the past five years and that the school serves Evaluation of Focus 23 as a literacy demonstration site. He noted that ARI funds a reading coach for the school, who has helped integrate and implement the program in the school. He also indicated that the district has had a strong emphasis on technology and feels that it is important to expose students to technology. He believed that FOCUS provided the “answer they were looking for” and that the prior school curriculum was missing elements for younger children. Most and Least Effective Elements. The principal reported that integration of FOCUS into small group instruction and centers has been effective and that teachers have embraced the program. He believed that anytime students are exposed to computers and game-like activities that are interactive “they are always focused.” Additionally, FOCUS and PLATO will also be utilized in the new computer lab. The principal stated that FOCUS provides another way to monitor individual student progress; however, the program does not provide the same reports as PLATO Learning. He felt that accountability was important and indicated reports that provide time on task or actual time students utilize the FOCUS program are not available at this time. “These reports are important to board members who allocated the money.” He also reported that computer down time has been “upsetting to teachers.” He expressed concern that due to frustration with computer problems teachers may not continue to use the program. “It is critical that the program be user friendly and teachers perceive it as a useful tool that is researched-based.” Role in program implementation. The principal believed that his role in the FOCUS implementation was to ensure teacher buy-in. He wanted teachers to perceive the program “as a positive and to be supportive of technology.” He described himself as Evaluation of Focus 24 the “encourager” of new initiatives in the school. He also reported that he meets with teachers regularly in weekly grade level meetings. Additional resources needed. According to the principal, no additional resources were needed because of the district emphasis on technology and PLATO Learning. Results. The principal reported that the reading coach believes that “DIBELS data look better this year.” He stated that DIBELS will be used to measure FOCUS program effectiveness and that all students K-3 are assessed. He also noted that FOCUS data are not currently utilized for grading purposes, however the school reports the Accelerated Reader STAR scores to parents. Superintendent, Curriculum Coordinator, and Reading Coach Focus Group General Information. The superintendent reported that he came to the district in 1999 as the technology coordinator and federal programs coordinator. He was also the principal at the high school for three years and became superintendent this year. He related that while he was principal he continued to serve as the technology coordinator and “in essence, he still is” due to his expertise. According to the superintendent, he had prior experience in technology in another district and was knowledgeable about computer-aided instruction and brought the PLATO Learning program to the district. He stated that initially the main focus was high school exit exam remediation. He indicated that PLATO was the only program aligned to the Alabama exit exam. The superintendent also reported that he recently made a presentation at the national PLATO conference and is very supportive of the PLATO program. He related that PLATO was implemented at the elementary school in the 2002-2003 school year and Evaluation of Focus 25 the PLATO Learning program is utilized at the elementary school mainly as remediation and enrichment and not necessarily as part of the day-to-day curriculum. The superintendent reported that all elementary students would begin to have additional access to computers at least once a week in a new lab. Additionally, Piedmont’s number of Internet connected computers to student ratio is in the top 10 statewide out of 130 school systems. The reading coach stated that Piedmont has the lowest student to computer ratio in the county even though “they are the poorest of the school districts in Calhoun County.” She reported that a significant amount of federal dollars have been received to fund technology efforts and that technology had always been a priority in the district. The curriculum coordinator remarked that the district was “very lucky to have so much technology.” The superintendent stated that, in the elementary school, he wanted to make computers part of the instructional day, where computers could be utilized during small group instruction. The curriculum coordinator reported that three new Dell computers were installed in every elementary classroom in the 2002-2003 school year and classrooms should now have at least four computers. She also reported that every classroom has Internet access and that computers are networked. She noted that in addition to the FOCUS program the elementary school uses the Accelerated Reader and Star Math programs. Integration of FOCUS into school’s literacy program. According to the curriculum coordinator, she became aware of the FOCUS program at the PLATO conference in October 2002. She reported that she attended FOCUS sessions and observed the software in a computer lab. She felt that FOCUS “fit the needs of all their students Evaluation of Focus 26 including special education – I thought that FOCUS would also be another way to have skills introduced and reviewed.” The superintendent stated that he felt that there was a gap in the K-2 program. The reading coach noted that FOCUS “picks up where we have weaknesses.” She stated that programs like Sing, Spell, Read, and Write and Action Reading do not have a phonemic awareness program but rather “they jump right into phonics.” She indicated that FOCUS is a supplement to their current reading program and that there are no conflicts between programs, such as the teaching of letters and sounds. The reading coach mentioned that first grade teachers do not use PLATO as much as the other grades. She also indicated that the school does not distinguish between FOCUS and PLATO, “We call FOCUS ‘PLATO’ and this confuses the PLATO people.” She reported that the first grade teachers use the FOCUS program daily as designed. She believed that their strong implementation was due primarily to the excellent training the teachers received. Most and Least Effective Elements. The superintendent stated that FOCUS is a strong enhancement to their current reading program. He indicated that while he was pleased with their current reading program, FOCUS had increased usage of computers in the reading block. The curriculum coordinator also expressed that she was She indicated that the speech impressed with the program in K-2 classrooms. pathologist and special education teachers also utilized the program and reported student improvement. The school also has plans for using FOCUS for remediation and tutoring in the after school program. Evaluation of Focus 27 The reading coach reported, “FOCUS has a strong phonemic awareness component, unlike other reading programs, which omit this area of instruction.” She noted that Action Reading is considered to be the first grade phonics program and FOCUS the phonemic awareness program. She stated that a tremendous advantage of the FOCUS program was the ability to customize the letter/lesson sequence to their Action Reading program. First grade teachers have aligned their sequence to Action Reading and kindergarten teachers have customized their sequence to Sing, Spell, Read, and Write. The reading coach, in one-on-one observation of students, noted the transfer of skills learned in FOCUS lessons to other reading areas, such as counting of syllables in words. All administrators reported positive impressions of the FOCUS training and ease of program implementation. The reading coach felt that the FOCUS trainer had a strong background in reading and technology. She reported that teachers were given the opportunity to practice FOCUS on the computer and teacher directed lessons during the two day training session. She also reported that the trainer has been responsive to questions and problems with follow-up phone calls and a site visit. The superintendent agreed that training and “follow-up are the key.” expectation that teachers utilize the program. Most negative comments related to FOCUS report generation. superintendent expressed disappointment in the reporting features of FOCUS. The He He also stated that there is an expected and was led to believe that all the reporting features available in PLATO would be available in FOCUS. Unlike PLATO, FOCUS reports do not provide the number of minutes students utilize the computer application. Additionally, since students have Evaluation of Focus 28 learned how to manipulate the program (i.e., speedily access the paint page), scores that students receive on the computer-based assessments appear unreliable and cannot be used for grading and assessment purposes. He noted that he had addressed the reporting issues with their FOCUS trainer and had requested guidance in how to include FOCUS performance in student grades. The curriculum coordinator indicated that available reporting is not user friendly. She noted there is a screen that provides student level scores by skill, however when the report is printed “that is not what you get.” She also stated that there are three levels to FOCUS and some children could be utilizing inappropriate levels that could be too easy or difficult. She also reported that the lack of available FOCUS reporting had caused communication problems in the district. Software glitches were also reported. The superintendent indicated that the animated character on the screen uses her thumb to begin counting and her voice is sometimes not in sync with the animation. He also expressed concern that while FOCUS was “a pretty good program” he felt that the school “was doing the Beta testing to find out what the problems are.” Role in program implementation. The superintendent reported his role in program implementation to have been the securing of funding and technology support for FOCUS. “PLATO and FOCUS have top priority with technology.” The curriculum coordinator took the lead in the initial implementation and planning. The reading coach stated that as a half-time split between the middle and elementary school, her time this year has been devoted to basic program implementation. She reported having an opportunity to assist teachers in weekly grade level meetings. She also noted that all Evaluation of Focus 29 teachers were trained in August but did not begin implementing the program until the first of September. Additional resources needed. The superintendent reported that 12 FOCUS kits were ordered at a cost of approximately $750 each. He stated that K-2 classrooms already had updated computers to support program implementation. However, technology support continues to be a major issue in the district. The district does not have a full-time technology technician. He noted that teachers have an email address to report problems directly to technical support. He also stated that during the fall a network line problem caused work to be delayed on the elementary computer lab and first grade teachers may have lost access to FOCUS some in December. Results. The reading coach reported that she was expecting an increase in student performance on DIBELS for two reasons: “We are now doing DIBELS progress monitoring and FOCUS.” She also reported that FOCUS had helped teachers align their reading curriculum. She noted that FOCUS data have not been discussed and analyzed at grade level meetings and that she had not correlated FOCUS data with DIBELS results. In the first year of FOCUS implementation “we have focused on implementation and how is it is going.” Teacher Focus Group General Information. Teachers reported that FOCUS implementation had gone well and the program was easy to use, although one teacher noted that the program was not available in December due to computer problems. The teachers stated that they had implemented PLATO the year before and felt comfortable with technology. All teachers use FOCUS during the reading block and PLATO with their homeroom as Evaluation of Focus 30 enrichment for math or in the computer lab. Teachers also reported that because firstgraders are regrouped for reading based upon ability it would be difficult to use FOCUS with their homeroom students. In order to use the program effectively teachers would need to access another teacher’s reading class to pull up their homeroom student information. Integration of FOCUS into school’s literacy program. The teachers believed that the FOCUS program addressed phonemic awareness, phonics, vocabulary, and comprehension skills. They also perceived that the program reflected the findings of the National Reading Panel in these areas. Three teachers stated that Card 6 (reading) was too difficult for most students. All regular first-grade teachers agreed that FOCUS helps to reinforce phonemic awareness and phonics skills introduced in Action Reading. Teachers also indicated that FOCUS helps students with writing and listening skills. The pre-first teacher stated that she uses FOCUS mainly as a phonemic awareness program along with other materials. Teachers reported using a variety of program reading materials. Regular firstgrade teachers noted that Action Reading was their main program. They also reported using other workbooks/worksheets, decoding sheets, another phonemic awareness program, old basal sets, leveled readers, decodable books, Scholastic books, and teacher created materials. The teachers expressed a need for additional decodable books for practice and take home. The pre-first teacher reported using Pannel phonics, Scholastic Literacy Place, and teacher-created materials. A teacher related that in the last reading adoption, first-grade teachers chose Action Reader instead of the basal text Evaluation of Focus 31 everyone else in the school uses. The school also implements the Accelerated Reader program and reports Star scores to parents on report cards. Teachers reported utilizing the FOCUS program four days a week. indicated that on Friday first-grade classes have a different schedule. They Although all teachers indicated 15- to 20-minute teacher-directed lessons four days a week, their computer application time varied by classroom. In the pre-first classroom every student has access to the FOCUS computer application daily for 15-minutes. The largest firstgrade class provided student access to computers for 15-minutes once a week. The other two teachers reported once a week access for 30-minutes and twice a week access for 30-minutes. Regular first-grade teachers have aligned the FOCUS sequence of lessons to Action Reading. sequence. The pre-first teacher reported utilizing the default FOCUS lesson All teachers agreed that FOCUS was a supplementary and not a comprehensive reading program. Their reasons included the lack of decodable text and letter introduction. Three teachers agreed that they were only using lesson Cards 1, 2 and 3; however the teacher with the highest reading ability group reports occasionally using additional FOCUS lesson cards. The teachers detected no gender differences in the use or response to the FOCUS program. One teacher did comment however, that girls seemed to participate more in the direct instruction portion of the FOCUS lesson. Most and Least Effective Elements. All teachers agreed that FOCUS is an easy program to manage and implement, and is “user friendly for young children.” They stated that PLATO was more difficult for young children to use. Teachers indicated that Evaluation of Focus 32 their students’ loved working on the computer and could utilize the program with limited teacher assistance. Teachers believed that the extra FOCUS reinforcement of letters and sounds was beneficial for students. Teachers also reported that professional development had been excellent and that grade level meetings and support from the reading coach had helped solve any issues or concerns. Teachers stated that they followed FOCUS direct instruction with the computer application during learning center time daily; however, they frequently voiced concerns about the direct instruction not matching the student computer practice. One teacher said, “I would like the computer practice to match my instruction as a direct reinforcement of what I was teaching.” Although teachers have the capability to align the sequence, students can progress through the program at their own pace and could easily be behind or ahead of the FOCUS direct instruction. Teachers also stated that students had learned how to manipulate the computer program. Every teacher reported that students had figured out how to click their way through to get to the “paint” page “without having to get anything right.” They indicated that students love the “paint” page and that there was no time limit on how long they could spend on this aspect of the program. Teachers suggested that a time limit be incorporated into the program. Student manipulation of the program and boredom were seen as major impediments to utilizing FOCUS data reports. Teachers did not believe that the scores were a true reflection of student achievement. Reports were not perceived as teacher friendly and teachers stated they are not utilizing them and do not understand how they could be a helpful tool. One teacher commented, “We can not use the reports for Evaluation of Focus 33 ongoing student assessment grades, or even to modify instruction”. They suggested that additional professional development was needed to address effective use of FOCUS reports. Teachers also expressed concern that students might become bored with the FOCUS program, “especially if students used it in kindergarten and first grade.” Teachers were unsure if activities varied in the different levels of the program. They also suggested that additional student activities should be provided to reinforce skills taught. Teachers agreed that worksheets and decodable text that reinforce skills currently had to be created by the teacher. Teachers expressed concern that some of the vocabulary and picture key cards were difficult for “at-risk” students with limited background knowledge. A teacher commented, “My students had no idea what an omelet was.” They also stated that the reading selection and activities on Card 6 were too difficult for most of their students. The teachers reported some problems with the software program. They indicated that it was difficult for students “to count along with Emma” and that she counts with her thumb. They believed the difficulty in counting had affected student performance. Additional resources needed. Teacher responses were similar to administrator comments relative to the need for additional resources. Teachers believed they had adequate computer resources to support the FOCUS program. They also noted that the technical support had been responsive “even during the major problem in December.” They stated that PLATO and FOCUS have priority over other technology Although the teachers believed that the reading coach was issues in the district. Evaluation of Focus 34 supportive, they indicated that she was “spread pretty thin” since she supports both elementary and middle schools. Results. Teachers indicated that there has been little change in core reading instruction due to implementation of the program. “FOCUS enhances and supports what we are already doing with our Action Reading program. It is an additional tool.” Teachers believed that FOCUS might help improve DIBELS results, especially on the Phoneme Segmentation sub-test. However, they reported little connection with the other subtests administered to K-2 students; Letter Naming Fluency, Nonsense Word Fluency, and Oral Reading Fluency. They stated, “Action Reading does a better job of helping prepare for DIBELS benchmark assessments.” Descriptive Analyses Tables 3 and 4 present the mean scores of first graders on the Phoneme Segmentation Fluency (PSF) and Nonsense Word Fluency (NWF) subtests for the fall and spring for the Control cohort and the FOCUS treatment group. The data are presented by class and includes the number of students, by class, and number identified as “established” in the benchmark goal by the subtest. In the table containing the Fall FOCUS data, the number of students defined as “low risk” of failing to achieve the benchmark goal is presented for the NWF subtest. First grade classes were assigned T-1 (highest performing) through T-3 (lowest performing) -based upon regrouping by ability. T-4 represents the pre-first grade class. Table 3 presents the fall and spring subtest scores for the Control students. In the fall, the mean scores were highest for the PSF subtest when compared to the NWF. In addition, more of the students scored as “established” in the PSF than the NWF. The Evaluation of Focus 35 T-1 and T-2 classes had the highest percentage of students identified as “established” in the PSF, while the T-3 and T-4 classes, which had the lowest mean scores of the four classes, had the fewest students identified as “established.” At 45.11, T-1 had the highest NWF mean score; however, T-2 had a slightly higher percentage of its students identified as “established.” The Control spring scores for the four classes indicated substantial improvement. The mean scores for both the PSF and NWF subtests were higher in the spring when compared to the fall, for all four classes. For the PSF subtest, the greatest improvement between fall and spring was registered by T-4, moving from a fall mean of 17.64 to 55.57. In addition, 100% of the students in T-4 taking the PSF in the spring qualified as “established”. However, it must be noted that T-4, the pre-first class, had a much smaller class size than the other classes. T-4 also showed the most improvement in NWF scores from the fall to the spring, moving from a 8.91 mean score to 58.43. Evaluation of Focus 36 Table 3 Mean, Standard Deviation, and Number Established by Class for PSF and NWF Scores Fall and Spring Control Cohort (2002-2003) Fall Scores Class T-1 T-2 T-3 T-4 Total Spring Scores Class T-1 T-2 T-3 T-4 Total N 17 17 15 7 56 Subtest PSF NWF PSF NWF PSF NWF PSF NWF M 63.47 91.82 56.82 41.76 53.46 60.93 55.57 58.43 SD 10.71 31.39 15.11 14.45 13.63 22.93 13.20 21.03 13.49 30.46 Number Established 17 15 16 6 14 10 7 5 54 36 N 18 19 13 11 61 Subtest PSF NWF PSF NWF PSF NWF PSF NWF PSF NWF M 55.67 45.11 52.63 22.21 27.15 15.92 17.64 8.91 41.79 25.23 SD 10.16 20.84 14.47 12.16 20.82 12.70 16.35 9.60 21.72 20.10 Number Established/Low Risk* 18 7* 18 9* 6 0* 3 0* 45 16 PSF 57.79 NWF 64.18 PSF = Phoneme Segmentation Fluency, NWF = Nonsense Word Fluency. Evaluation of Focus 37 Table 4 Mean, Standard Deviation, and Number Established or Low Risk by Class for PSF and NWF Scores Fall and Spring FOCUS Treatment Group (2003-2004) Fall Scores Class T-1 T-2 T-3 T-4 Total Spring Scores Class T-1 T-2 T-3 T-4 Total N 19 22 18 13 72 Subtest PSF NWF PSF NWF PSF NWF PSF NWF M 76.89 102.05 70.72 77.5 61.39 68.72 67.54 65.0 SD 16.80 41.79 16.98 17.51 20.70 25.24 19.07 19.34 18.80 30.90 Number Established 19 19 21 21 17 14 12 11 69 65 N 20 23 19 13 75 Subtest PSF NWF PSF NWF PSF NWF PSF NWF PSF NWF M 45.5 36.7 34.0 26.30 34.26 18.32 40.69 13.31 38.29 24.80 SD 13.92 14.77 13.74 11.91 15.39 14.02 18.42 9.28 15.60 15.26 Number Established/LowRisk* 17 16* 14 13* 10 7* 9 2* 50 38 PSF 69.44 NWF 79.53 PSF = Phoneme Segmentation Fluency, NWF = Nonsense Word Fluency Evaluation of Focus 38 Table 4 includes the information for the FOCUS first and pre-first grade classes. The same classes were designated as T-1 through T-4 in both years examined to account for regrouping. In the FOCUS treatment group, T-1 again had the highest mean scores on the fall PSF test at 45.5 and 17 of the 20 students in the class were characterized as “established.” Students in T-4 averaged a 40.69 score in the fall, followed by T-3 at 34.26 and T-2 at 34.0. T-1 also had the highest mean on the NWF subtest during the fall achieving a 36.7 score. Sixteen of the 20 students assessed in T-1 were identified as “low risk” of failing to achieve planned outcomes. Just as the Control cohort data indicated, the four classes in FOCUS made substantial improvement in their mean scores on both subtests during the spring. T-1 had the highest mean scores of the four classes on both the PSF and NWF subtests at 76.89 and 102.05 respectively. T-1 also had 100% of its 19 students classified as “established” by the assessments. While T-1 had the highest mean scores, the other 3 classes also experienced marked improvement. T-2 had the next highest mean scores for both subtests at 70.72 for the PSF and 77.5 for the NWF. T-4 had the third highest PSF score at 67.54 and T3, with a score of 68.72, the third highest NWF score. The majority of the students in the four classes were categorized as “established” on the two subtests. ANOVA Repeated Measures Analysis Repeated measures mixed design ANOVA is reported for the two DIBELS subtests for first graders from both the 2002-2003 (Control cohort) and 2003-2004 (FOCUS) academic years. This analysis was performed to determine if there was a Evaluation of Focus 39 significant difference in the students’ spring scores on the Phoneme Segmentation Fluency (PSF) and the Nonsense Word Fluency (NSF) subtests when compared to the students’ fall scores for both groups (treatment and control). Follow-up analyses are also reported whenever interactions were significant at a .05 level. Only those first graders having both fall and spring scores on the PSF and NWF subtests were included in the analysis. This criterion allowed the use of the repeated measures design and ensured that the students examined attended Piedmont for the entire school year. Table 5 presents the mean and standard deviation of scores on the PSF and NWF subtests for the students in the two groups studied. As the data in the table indicates, the mean scores on both the PSF and NWF subtests increased between the fall and the spring for the first graders in both academic years. For the Control cohort, the mean PSF fall score was 43.61, while the spring, score on the PSF was 59.06, a difference of 15.45. The NWF fall score for this academic year was 26.76 and the spring score was 64.53. Table 5 Mean and Standard Deviations of Fall and Spring PSF and NWF Scores by Cohort Cohort Control N = 51 Subtest PSF Fall PSF Spring NWF Fall M 43.61 59.06 26.76 SD 19.99 12.52 19.93 FOCUS N = 65 M 39.43 71.48 24.85 SD 14.32 18.19 15.54 31.73 NWF Spring 64.53 31.63 80.77 PSF = Phoneme Segmentation Fluency, NWF = Nonsense Word Fluency. Evaluation of Focus 40 The mean PSF fall score for the FOCUS first graders was 39.43. In the spring, the mean PSF score improved by 32.05 points to 71.48. During the same time period, the NWF mean score increased from 24.85 in the fall to 80.77 in the spring. Mean scores for each test by academic year is presented graphically in Figure 1. Figure 1 Mean Scores by Test and Group 90 80.77 80 71.48 70 64.53 60 59.06 Mean Score 50 43.61 39.43 40 30 26.76 24.85 20 10 0 PSF PRE PSF POST Tests NWF PRE NWF POST Control Focus Evaluation of Focus 41 Table 6 summarizes the mean gains on both subtests in each academic year. The average gain in scores in the Spring when compared to the Fall for the PSF subtest was 15.45 for 2002-2003 first grade students and 32.05 for students in 2003-2004. The gain in NWF scores for the 2002-2003 and 2003-2004 cohorts was 37.76 and 55.92 respectively. Table 6 Mean Gain in Scores for PSF and NWF Subtests in 2003-2004 Academic Years Academic Year 2002-2003 Subtest PSF M 15.45 SD 20.49 M 32.05 2003-2004 SD 21.26 25.66 NWF 37.76 28.14 55.92 PSF = Phoneme Segmentation Fluency, NWF = Nonsense Word Fluency. A 2 (Pretest and Posttest) x 2 (Group: Treatment and Control) repeated measure mixed design ANOVA was performed on the PSF and NWF scores. Table 7 provides ANOVA summary tables for the two analyses. As shown in Table 7, the main effect for Testing was highly significant F (1,114) = 147. 19, p < .0001. A significant interaction between PSF (pretest and posttest) and group (treatment and control) was also found F (1,114) =17.97, p < .0001. The Group main effect was not significant. Follow-up paired t-tests analysis indicated that there were significant differences between the PSF pretest and posttest scores for both groups, control [t (50) = 5.38, p < .0001), ES = +0.75]. and treatment [t (64) = 12.15, p < .0001, ES = +1.51]. Independent Evaluation of Focus 42 t-tests showed that there was no statistical significant difference between treatment and control on the PSF pretest, t (114) = 1.31, p = .193, ES = -0.24. However, a significant difference was found between the treatment (M = 71.48) and control (M = 59.06) on the posttest, t (114) = 4.16, p < .0001, ES = +0.73. The larger effect size for treatment (ES = +1.51) and posttesting (ES = +0 .73) suggests a strong FOCUS program effect. The main effect for NWF testing was also significant F (1,114) = 349.90, p < .0001. A significant interaction between NWF (pretest and posttest) and group (treatment and control) was also found F (1,114) = 13.14, p < .001. The Group main effect was not significant. Follow-up paired t-tests analysis indicated that there were significant differences between the NWF pretest and posttest scores for both groups, control [t (50) = 9.58, p < .0001, ES = +1.34] and treatment [t (64) = 17.57, p < .0001, ES = +2.18]. Independent t-tests showed that there was no statistical significant difference between treatment and control on the NWF pretest, t (114) = 0.583, p = .561, ES = -0.11. However, a significant difference was found between the treatment (M = 80.77) and control (M = 64.53) on the posttest, t (114) = 2.74, p < .01, ES = +0.50. The larger effect size for treatment (ES = +2.18) and posttesting (ES = +0.50) suggests a strong FOCUS program effect. Evaluation of Focus 43 Table 7 ANOVA Repeated Measures Mixed Design for Significance by Subtest Subtest Within Subjects PSF PSF*GROUP Between Subjects GROUP Within Subjects NWF NWF*GROUP 349.903 13.114 .000 .000 .754 .103 1 1 1 Between Subjects GROUP 3.068 .083 .026 PSF = Phoneme Segmentation Fluency, NWF = Nonsense Word Fluency. p < .05 Level of Significance. 2.977 .087 .025 1 147.188 17.968 .000 .000 .567 .136 1 1 F p Eta Squared df Discussion Literacy Teaching Practices Used in FOCUS Classrooms. Teacher directed FOCUS lessons were primarily coded as phonemic awareness instruction during the three days of LOT observations. First grade teachers devoted most of their approximately 15-minutes of FOCUS instruction on Card 2 (FOCUS on Words). Cards 1 and 3 (Get in FOCUS and FOCUS on Sentences) were given less emphasis. FOCUS on Words (Card 2) encompasses over 10 phonological tasks and teachers appeared to utilize scripts and follow the recommended sequence of the lesson. All FOCUS activities were oral and the only materials utilized were keyword pictures and card scripts. Every first grade teacher supported FOCUS classroom instruction with the computer software application during daily learning center activities. Evaluation of Focus 44 However, computer practice did not necessarily align with classroom instruction. Students were observed to be working on a variety of different lessons and skills. Action Reading is considered the core reading program and the regular first grade teachers have aligned the FOCUS sequence to the Action Reading program. In LOT observations Action Reading activities were primarily coded as phonics instructions. Teachers devoted approximately 30 minutes to Action Reading activities and often incorporated a teacher read aloud. After whole class instruction, students rotated to a variety of center activities. Although all classes had alphabets, word walls, and classroom libraries, students were rarely observed using authentic reading materials and basal texts were utilized only in the pre-first classroom. Of concern was the lack of fluency, vocabulary, and text comprehension activities observed during the first grade-reading block. Teachers expressed their concern regarding the lack of availability of decodable text to support skills introduced in Action Reading and FOCUS lessons. Teaching activities traditionally emphasized in first grade such as; introduction of vocabulary, asking students for predictions, making connections to prior knowledge, using higher level questioning, and guiding visual imaging and interactive discussion were rarely observed. Teacher Reactions to FOCUS The four teachers participating in the focus group agreed that the FOCUS program was an important supplement to the primary reading program, Action Reading. The teachers reported that they were able to use FOCUS to reinforce concepts of phonics and phonemic awareness. Classroom observations indicated that teachers primarily utilized lesson cards that related to phonological skills. Evaluation of Focus 45 The teachers reported that the computer application had proven simple to use and fairly stable except for a period in December when the system was offline for an extended period. The teachers believed that their students enjoyed working on the computer and the program was flexible enough to allow each teacher to schedule student access differently. Two teachers allowed students to use the computer in centers between 15 and 30 minutes once a week, one teacher made the program available for 30 minutes twice a week, and the “pre-first” teacher allowed her students access for 15 minutes daily in a self-contained situation. While the teachers used the computer portion of the program differently, all four teachers reported using the teacher directed lessons four days a week. The teachers appeared to be satisfied with the professional development they had received. The focus group participants reported that they received two days of training before school started and had one day of support after the beginning of the school year. The teachers described the training as good and suggested that the trainer had done an excellent job of modeling how the program should be used in class. While the teachers were generally positive about FOCUS, they did report some concerns. Chief among them was their concern that the students too easily manipulated the software. The teachers reported frequent instances in which students simply continued clicking on a response until they were able to get to the “paint” activity. Teachers believed that the system should have a built-in timer that limited a student’s time in the paint area and would force the student to move to a new lesson. The first grade teachers expressed their concern regarding the inability to align skills introduced in daily whole class direct instruction with student computer practice. Evaluation of Focus 46 First grade teachers had a strong background in reading and realized the importance of following teacher modeling and direct instruction with aligned student practice and reinforcement activities. Teachers also viewed vocabulary and reading used on the lesson Cards as difficult for some students. They reported that their students did not always have the prior knowledge or background to make connections. Teachers also expressed frustration with the reporting function of the software and expressed a need for additional professional development. They were skeptical that the reports were very meaningful given that some students discovered how to manipulate the computer program. Teachers weren’t sure whether a student was struggling with a particular lesson or was just bored and “clicking out of it” to get to the “paint” feature. Refer to class reports in Appendix C. Finally, the teachers reported that they saw little connection between FOCUS and DIBELS. The teachers’ perception was that FOCUS didn’t help with nonsense word fluency although they believed that it did help some with phoneme segmentation. They believed that the Action Reading program did a better job of preparing their students for the DIBELS assessment. Principal’s and Administrator’s reactions to FOCUS. The principal, superintendent, and other administrators all viewed FOCUS as an important tool for improving reading instruction at Piedmont and as a natural addition to the district’s current PLATO Learning initiative. Each of the administrators reported having a positive perception of PLATO Learning products after witnessing their use in other schools within the district. Evaluation of Focus 47 The administrators believed that it is important to expose students at an early age to technology and therefore felt that FOCUS is a useful mechanism for expanding the use of computers throughout the instructional day. All of the administrators expressed satisfaction with the training provided to the district and believed that the implementation of the program into the broader curriculum had been effective. The administrators believed that FOCUS filled a gap in the K-2 reading curriculum. The reading coach referred to FOCUS as the phonemic awareness program and Action Reading as the phonics program for first grade. Most of the negative comments from the administrators mirrored those expressed by the teachers. Principally, the administrators were dissatisfied with the reporting options available to the district through the software. The district had reported its concerns to PLATO. The superintendent also expressed his belief that PLATO was using Piedmont to Beta test the software. As an example, the superintendent mentioned software glitches that caused the narration to be out of sync with the animation. The principal was concerned that problems with the software might cause teachers to abandon the daily use of the program. Influences of FOCUS on Student Literacy Achievement. As presented in Tables 3 and 4, both the Control and FOCUS first grade classes saw improvements in their scores on the PSF and NWF DIBELS subtests as the academic years progressed. In spring 2003-2004 all first grade classes reported higher scores for both the PSF and NWF over the prior year. Interestingly, while the pre-first class reported largest increase in scores for both tests in the Control group, the two Evaluation of Focus 48 highest reading ability groups had the largest increase for the PSF and NWF in the FOCUS group. Since first graders at Piedmont are regrouped during reading class based on ability level, variation was expected. However, since the increase in scores for all reading groups were similar in the FOCUS group, this suggests that FOCUS may have assisted students at all ability levels to improve their skills. Regardless of the variation in scores across classes, the results from the repeated measures mixed design ANOVA presented in Table 6 indicate that the change in pre and post test scores for both PSF and NWF was significant. The highly significant results suggest that the change in scores between the fall and the spring testing periods was large and meaningful. Significant interactions between the two Additional follow-up subtests and control and treatment groups were also found. independent t-tests showed significant differences between the Control and FOCUS treatment group posttest scores. It is not surprising that the effect size for PSF is higher, since the skill assessed was closely aligned to observed FOCUS instruction (PSF: ES = + 0.73, NWF: ES = +0.50). Significant differences were not found between control and treatment pretest scores. The follow-up paired t-tests ES’s for PSF and NWF were significantly larger for FOCUS and reflect impressive growth in posttest scores. Additionally, the mean fall scores for both subtests for FOCUS were slightly lower than the mean fall scores for the Control cohort (PSF: ES = +1.51, NWF: ES = +2.18). In other words, the first graders with FOCUS began the year with slightly lower average scores, yet ended the year with higher scores than those of the Control cohort. Given that 2003-2004 was the implementation year of FOCUS, this suggests that the students’ interactions with the Evaluation of Focus 49 FOCUS curriculum, supplementing the Action Reading program, may have had a positive impact on the students’ two DIBELS subtest scores. Conclusions and Recommendations • Piedmont’s technology emphasis and district support played a major role in positive teacher attitudes and ease of FOCUS implementation. • FOCUS is currently used at Piedmont as a supplement to the comprehensive reading program in the area of phonemic awareness. First grade teachers were consistently integrating this aspect of the program into whole group direct instruction and small group/center activities during the 90-minute reading block. • Teachers utilized lesson Cards 1, 2, and 3. They primarily emphasized Card 2, which focuses on phonological skills. The teachers considered the program lacking in fluency and vocabulary activities. A review of Cards 1 and 3 indicate some vocabulary development, higher-level questioning, connections to prior knowledge, and visual imaging. As verified in the LOT data, these activities were rarely coded as observed during the three days of classroom observations. Teachers may not have used all of the available activities on each card due to time constraints and their emphasis on FOCUS as primarily a phonemic awareness supplement. • The capability to align FOCUS lessons to the core reading program provides teachers the ability to integrate lesson components that are needed for reinforcement and enrichment. Traditionally, basal reading programs have been lacking in explicit phonemic awareness instruction. The reading coach and district administrators recognized this need to supplement their core program in this Evaluation of Focus 50 “essential component of reading instruction” and felt FOCUS provided a meaningful supplement, which can be easily aligned to the core program. From observations at the school, it appears that FOCUS would meet the intent of the national Reading First grant requirement of explicit phonemic awareness instruction. • There was a point of difference between the district administrators and teachers regarding the relationship of the FOCUS program and DIBELS assessment improvement. Teachers believed that increased scores were due primarily to the Action Reading Program while the reading coach reported that FOCUS implementation was a primary factor in the increased scores this year. This discrepancy may reflect the teachers’ strong commitment to the Action Reading program. Additionally, the skills that are assessed in first grade on DIBELS are not “directly” included in the FOCUS program; i.e., segmenting three- and four-phoneme words into their individual phonemes and the ability to produce verbally nonsense words. Teachers reported the importance of skill practice in these areas. The phonemic awareness activities, phoneme segmentation and blending, could be more explicitly incorporated into teacher directed FOCUS lessons. • FOCUS students scored significantly higher than control students on; the PSF and NWF posttests. The respective effects sizes of +0.73 and +0.50 indicate educationally meaningful differences, especially in the absence of pretest differences on the same tests. • Because FOCUS is used as a supplement to the core program, Action Reading, it is difficult to distinguish how much of the gain in mean scores should be attributed to FOCUS implementation. Additionally, increased DIBELS scores could partially be Evaluation of Focus 51 due to testing effect. By May 2004, students had multiple test exposures to DIBELS through progress monitoring and benchmark assessments. Accordingly, the advantages demonstrated by the FOCUS group should be interpreted cautiously as suggestive of possible program effects but requiring validation in more controlled experimentation. • The true impact of FOCUS may not be known until the 2003-2004 first graders are tested in the second and third grade for oral reading fluency to determine if gains in first grade DIBELS scores can be sustained over time. For example, at the end of 2003-2004, 31% of the second and 27% of third graders were assessed as “at risk” on the Oral Reading Fluency subtest of DIBELS. • FOCUS should improve its reporting function for teachers and administrators. Reports should become more user friendly and provide cumulative scores for skills such as phonemic awareness and phonics. These data could be used to monitor and predict DIBELS benchmark testing. In addition, reports should provide data on the time on task by individual students. • Additional flexibility should be part of the FOCUS computer application. For example, following direct instruction, teachers need the capability to ensure that students are all working on or being reinforced in the same skill. Teacher frustration must be addressed by limiting the opportunity for students to work on a variety of lessons at the same time. Reading First supports instructional content based on the five essential components of reading instruction integrated into a coherent instructional design. A coherent design includes explicit instructional strategies that address students’ Evaluation of Focus 52 specific strengths and weaknesses, coordinated instructional sequences, ample practice opportunities, and aligned student materials. First-grade teachers realized the importance of providing direct instruction and students then having the opportunity to practice the same skill on the computer. Allowing students to work on skills not related to the direct instruction during the reading block would likely be considered “layering” and not scientifically based reading instruction. Reading First also suggests that in-class grouping strategies be used, including small group instruction as appropriate to meet student needs. The Piedmont teachers stated that the FOCUS computer application could provide students with meaningful reinforcement during learning center time while they were conducting small group sessions. District administrators also planned to utilize FOCUS for early intervention and reading remediation. • Fluency, vocabulary, and comprehension activities were rarely observed. It will be difficult to incorporate these activities into FOCUS if teachers continue to use it as a supplemental program that reinforces explicit phonemic awareness instruction. Generally, basal reading series are the core program in most schools and teachers will need to be provided additional training and support in how to use these components of the FOCUS program. Evaluation of Focus 53 References Good, R. H. & Kaminski, R. A. (2003) DIBELS Indicator of Basic Early Literacy Skills. Longmont: CO. Sopris West Educational Services. PLATO Learning, Inc. (2002). Bloomington, MN. Smith, L. J., Ross, S. M. & Grehan, A. W. (2002). Literacy Observation Tool (LOT©). Memphis, TN: Memphis. Sterbinsky, A., & Ross, S. M. (2003). Memphis, TN: Memphis. U.S. Congress (2001). No Child Left Behind Act of 2001. Washington, DC: Author. U.S. Department of Education (2002). Washington, DC: Author. Guidance for the Reading First Program. Literacy Observation Tool Reliability Study. Center for Research in Educational Policy, The University of FOCUSTM: Reading and Language Program. Center for Research in Educational Policy, The University of Evaluation of Focus 54 Appendices Appendix A Appendix B Appendix C Photographs of PLATO Classrooms Class Reports Teacher Focus Group Evaluation of Focus 55 Appendix A Evaluation of Focus 56 Appendix A (continued) Evaluation of Focus 57 Appendix A (continued) Evaluation of Focus 58 Appendix A (continued) Evaluation of Focus 59 Appendix A (continued) Evaluation of Focus 60 Appendix B T1 T2 Evaluation of Focus 61 Appendix B (continued) T3 T4 Evaluation of Focus 62 Appendix C Teacher Focus Group Questions Piedmont Elementary I. General Information How is the implementation of your FOCUS Reading and Language school program going? Compare and contrast this year with last year’s reading/literacy program. A criterion for the Reading First (National Reading Panel Findings) is to use a balanced literacy program that employs proven strategies and methods for curriculum, and instruction that are based on scientifically based research and effective practices. Discuss how your literacy program meets this criterion. Describe how FOCUS has been integrated into your literacy program. How are teacher directed lessons utilized? How often? Which cards? How are interactive computer sessions utilized? How often? Describe how your school chose the FOCUS program. (*Note – Typically only asked during the first year of program implementation). How would you describe teacher support for the FOCUS program? Would you say support for the FOCUS program is increasing or decreasing? What evidence is there of support or opposition? Can you think of specific positive or negative comments made by teachers about your school’s FOCUS program? Evaluation of Focus 63 Appendix C (continued) What additional resources have been needed to support the FOCUS program? (Note: Resources include time, space, personnel, and materials in addition to money; e.g., a reading coach and planning time) How long have you been using computers in your instruction? II. Classroom Level Changes If I were to visit classrooms during reading, what would I see that would represent your school’s FOCUS program? How is this different from the way reading instruction used to be? Specifically, what contributions has the FOCUS program made in terms of: • • • • • • Reading instruction? Teacher professional development? Extended learning opportunities for students (tutoring and after-school programs)? Computer utilization/usage? Transition programs for kindergarten students? Monitoring of student progress and achievement? How does your reading program address special needs children? Evaluation of Focus 64 Appendix C (continued) III. Results How is the FOCUS program impacting students? How are the FOCUS Student and Class reports utilized by teachers and the administration? Assessment? Monitoring? Diagnostic? Placement? Have measurable student performance goals and implementation benchmarks been developed for First Grade students? What processes did you use? Are there differences in achievement (DIBELS or FOCUS reports)? Can you describe any differences in student motivation or enthusiasm? For learning? For reading? Conduct? Relationships between students? Between students and teachers? How has the FOCUS program impacted teachers? How has the FOCUS program impacted relationships between teachers? Discuss differences in teacher collegiality and teamwork, motivation and enthusiasm. How has the FOCUS program created shared leadership and a broad base of responsibility for literacy reform efforts? IV. Professional Development How would you describe training sessions for the FOCUS program? How have new faculty been brought into the program? What is your assessment of how successful this has been? Any suggestions for how to continue the professional development process at your school? Evaluation of Focus 65 Appendix C (continued) Tell me about training and support from the reading coach or district office. What kind of support do they provide? How effective has the support been? How does your school handle technical support for the computers? V. Parent and Community Involvement How would you describe parent and community support for your school’s FOCUS program? How has the level of parent involvement in the school been impacted by your FOCUS program? Describe school efforts to inform and involve parents. Are parents and other community members more involved in literacy efforts now than in the past? What is the evidence of increased involvement? VI. Closure: Do you have any suggestions to improve the FOCUS program? Are there any important aspects of the FOCUS program implementation that have not been mentioned today? Any additional comments you would like to make? Evaluation of Focus 66