Technology in Schools

2000 Research Report on the Effectiveness of Technology in Schools 7th Edition 1730 M Street NW, Suite 700 Washington, DC 20036 +1 (202) 452-1600 orders@siia.net http://www.siia.net ABOUT THE AUTHORS: Jay Sivin-Kachala and Ellen R. Bialo are Vice President and President of Interactive Educational Systems Design (IESD), Inc., an educational technology consulting firm in New York City. IESD provides a variety of services related to the development and evaluation of educational software, multimedia products and websites, conducts research in the field of educational technology, and develops print materials that supplement educational software. Services include: • • • • • • • • • Market research, analysis, and strategic planning -- activities include surveys, focus group research, user testing, interviews, demographic and financial data analysis, and literature reviews Development of product specifications, design documents, storyboards, and scripts Development of instructional content Product evaluation -- assessing the match between product content and presentation and the needs of specific audiences Creation of supplemental print materials, including teacher’s guides, teacher training guides, user and reference guides, and student activities Research and writing on educational technology, curriculum integration, and instructional innovation Field testing Planning and implementation of technology-related school staff development Project direction and administration -- assembling exemplary personnel to meet the needs of our clients on time and within budget. IESD's clients include software developers and publishers, technology hardware manufacturers, government agencies, non-profit institutions and school districts. Jonathan Langford is a free lance educational writer who frequently works with IESD. Sivin-Kachala and Bialo can be contacted at (212) 769-1715 or at iesdinc@aol.com. 2000 Research Report on the Effectiveness of Technology in Schools 2 Table of Contents EXECUTIVE SUMMARY..................................................................................................... 7 THE P ROMISE AND CHALLENGE OF EDUCATIONAL TECHNOLOGY............................................ 7 AN AREA OF SIGNIFICANT CHANGE AND GROWTH ................................................................. 7 ABOUT THIS REPORT ............................................................................................................ 8 CONCLUSIONS ...................................................................................................................... 9 General............................................................................................................................ 9 Positive Effects of Technology on Student Achievement..................................................... 10 Positive Effects of Technology on Student Motivation and Self-Concept.............................. 11 Effects of the Teacher’s Role and Instructional Decisions.................................................. 11 Effects of Specific Software Design Features .................................................................... 11 Effectiveness of Interactions Involving Educators and Students in the Learning Environment................................................................................................................... 13 SECTION 1: EFFECTS OF TECHNOLOGY ON STUDENT ACHIEVEMENT................ 15 COMPARISON STUDIES .........................................................................................................15 LARGE-SCALE TECHNOLOGY IMPLEMENTATIONS..................................................................17 CURRICULUM AREAS AND STUDENT ACHIEVEMENT ..............................................................18 Language Arts and Reading............................................................................................. 18 Language Development ................................................................................................18 Reading .......................................................................................................................19 Reading and Spelling....................................................................................................20 Spelling .......................................................................................................................20 Reading and Writing.....................................................................................................20 Writing........................................................................................................................20 Mathematics................................................................................................................... 22 Science and Medicine...................................................................................................... 27 Social Studies ................................................................................................................. 29 Foreign Language .......................................................................................................... 30 English as a Second Language (ESL)............................................................................... 30 Logo and Other Programming Languages........................................................................ 31 Meta-Analyses .............................................................................................................31 Logo and Creativity ......................................................................................................31 Logo and Problem-solving ............................................................................................32 Logo and Geometry Concepts .......................................................................................32 Technical Training.......................................................................................................... 32 Career Education ........................................................................................................... 33 LEARNER CHARACTERISTICS AND STUDENT ACHIEVEMENT ...................................................33 SPECIAL P OPULATIONS AND STUDENT ACHIEVEMENT ...........................................................35 Early Childhood Education ............................................................................................. 35 Special Needs Students.................................................................................................... 37 Learning Disabled Students...........................................................................................37 Low-Achieving Students...............................................................................................39 Special Education .........................................................................................................40 SOFTWARE DESIGN CHARACTERISTICS AND STUDENT ACHIEVEMENT ....................................40 Type of Software ............................................................................................................. 41 2000 Research Report on the Effectiveness of Technology in Schools 3 Instructional Control....................................................................................................... 42 Amount of Practice ......................................................................................................... 45 Feedback ....................................................................................................................... 46 Objectives and Advance Organizers ................................................................................. 48 Cognitive Strategies ........................................................................................................ 49 Pedagogical Agents ........................................................................................................ 52 Conceptual Change Strategies ......................................................................................... 52 Instructional Scaffolding ................................................................................................. 53 Learner Support in Foreign Language Learning ............................................................... 55 Still Graphics in Vocabulary Development....................................................................... 55 Graphs and Tables in Mathematics Instruction ................................................................. 56 Animated Graphics ......................................................................................................... 56 Spoken Narration, Text, and Sounds with Graphics........................................................... 59 Multiple Representations of Arithmetic Problems.............................................................. 61 Dynamic Visualization .................................................................................................... 62 Video ............................................................................................................................. 62 Navigational Technique................................................................................................... 63 Motivational Contexts for Learning.................................................................................. 66 Window Presentation Styles............................................................................................. 67 Ergonomic Combination of Design Features .................................................................... 67 RECENT TECHNOLOGIES AND STUDENT ACHIEVEMENT .........................................................67 Telecommunication......................................................................................................... 67 Collaboration and Writing to a Distant Audience............................................................68 Access to Online Resources ..........................................................................................69 Document Exchange and Discussion..............................................................................71 Professional Development.............................................................................................71 Distance Learning.........................................................................................................71 Video ............................................................................................................................. 73 Mathematics ................................................................................................................73 Science........................................................................................................................73 Social Problem-Solving ................................................................................................74 Video Instruction with College Students ........................................................................75 One Caveat..................................................................................................................76 Hypermedia.................................................................................................................... 76 Adaptive Testing ............................................................................................................. 78 INSTRUCTIONAL DECISIONS AND STUDENT ACHIEVEMENT ....................................................78 Student Grouping............................................................................................................ 78 Time Spent on Computer ................................................................................................. 82 Self-Regulatory Teaching Strategies................................................................................. 83 Exploratory vs. Confirmatory Approach to Computer Simulation....................................... 83 Grading and Incentive..................................................................................................... 84 Preparatory Training in the Use of Tool Software............................................................. 84 Task Structure when Students Search the Internet............................................................. 84 Medium for Administration of Assessment........................................................................ 84 CONCLUSION.......................................................................................................................85 SECTION 2: EFFECTS OF TECHNOLOGY ON STUDENT SELF-CONCEPT AND ATTITUDE ABOUT LEARNING....................................................................................... 87 EFFECTS ON STUDENT SELF -CONCEPT ...................................................................................87 CURRICULUM AREAS AND STUDENT ATTITUDES ...................................................................88 2000 Research Report on the Effectiveness of Technology in Schools 4 Language Arts ................................................................................................................ 88 Mathematics................................................................................................................... 88 Science........................................................................................................................... 89 Social Studies ................................................................................................................. 91 SOFTWARE DESIGN CHARACTERISTICS AND STUDENT ATTITUDES..........................................91 Learner Control.............................................................................................................. 92 Static and Animated Graphics.......................................................................................... 92 Text in Foreign Language Software.................................................................................. 93 Designing for Gifted Students.......................................................................................... 93 Feedback Variations in Computer-Based Testing.............................................................. 94 RECENT TECHNOLOGIES AND STUDENT ATTITUDES...............................................................94 Telecommunication......................................................................................................... 94 Video ............................................................................................................................. 96 Adaptive Testing ............................................................................................................. 97 Virtual Reality ................................................................................................................ 97 SPECIAL P OPULATIONS AND STUDENT ATTITUDES.................................................................98 Early Childhood Education ............................................................................................. 98 Students with Limited English Proficiency........................................................................ 98 Special Needs Students.................................................................................................... 98 INSTRUCTIONAL DECISIONS AND STUDENT ATTITUDES..........................................................99 Student Grouping............................................................................................................ 99 Providing a Learner-as-Multimedia-Designer Environment............................................. 100 CONCLUSION.....................................................................................................................101 SECTION 3: EFFECTS OF TECHNOLOGY ON INTERACTIONS INVOLVING EDUCATORS AND STUDENTS IN THE LEARNING ENVIRONMENT ....................... 102 EFFECTS OF TECHNOLOGY ON TEACHER-STUDENT INTERACTIONS AND EDUCATORS' INSTRUCTIONAL BEHAVIOR................................................................................................102 Differences in Classroom Interaction Patterns................................................................ 102 Technology in a Project-Based Learning Environment.................................................... 105 Technology in an Inquiry-Based Learning Environment.................................................. 107 Technology-Rich Learning Environments....................................................................... 107 Networked Learning Environments ................................................................................ 108 Internet Use.................................................................................................................. 108 EFFECTS OF TECHNOLOGY ON STUDENT INTERACTIONS.......................................................109 Computer Work vs. Seatwork ......................................................................................... 109 The Learning Task ........................................................................................................ 109 Writing with Word Processors....................................................................................... 109 Networked Learning Environments ................................................................................ 110 Software Characteristics ............................................................................................... 111 Learner Characteristics................................................................................................. 112 Instructional Decisions.................................................................................................. 112 CHARACTERISTICS OF A DESIRABLE TECHNOLOGY-BASED LEARNING ENVIRONMENT ...........113 TEACHER P ROFESSIONAL DEVELOPMENT AND P ROFESSIONAL COMMUNICATION ..................115 CONCLUSION.....................................................................................................................117 BIBLIOGRAPHY .............................................................................................................. 118 2000 Research Report on the Effectiveness of Technology in Schools 5 2000 Research Report on the Effectiveness of Technology in Schools 6 2000 Research Report on the Effectiveness of Technology in Schools: Executive Summary The Promise and Challenge of Educational Technology Two decades of practical experience with educational technology and a decade of research on it permit us to make several assertions with a fair degree of certainty: • Technology can improve teaching and learning, but just having technology doesn’t automatically translate to better instructional outcomes. Whether a given school experiences the potential benefits of technology depends on the software it chooses, what students actually do with the software and computer hardware, how educators structure and support technology-based learning and whether there is sufficient access to the technology. In order to promote desired instructional outcomes, software must be effectively designed. What we know about effective software design has grown to the point that electronic publishers can create and educators can choose quality software. There isn’t one "right" type of software nor one "right" way to use technology. Rather, the software and the way it is used instructionally must match the school’s learning and teaching goals and must be appropriate for the specific students who will use it. If we want students to engage in appropriate technology-based learning experiences and if we want educators to successfully structure and support these experiences, then teacher professional development and support is essential. Making appropriate school- and district-level decisions about desired uses of educational technology, software selection, professional development and support, hardware and technology infrastructure, and overall implementation requires a wellconceived planning process. Computer and communication technologies continue to change—and educational technology tools and their implementation must continue to evolve as well. With the ever-accelerating rate of technological change, new opportunities are continually arising for positive, innovative uses of technology in education. This underscores the need for ongoing educational technology planning, evaluation and refinement of the current implementation, and professional development and support. • • • • • An Area of Significant Change and Growth As we move into the 21st century—and our third decade of using microcomputers and related information technologies for instructional purposes—the education market has transformed from a stand-alone desktop world to a Web-connected world. In this new age of computing, the opportunities and challenges are great for educators, publishers, and hardware vendors alike. During the last decade, U.S. K-12 schools have approximately tripled their spending for instructional technology, from $2.1 billion in 1991-1992 to a projected $6.2 billion in 1999-2000, 2000 Research Report on the Effectiveness of Technology in Schools 7 not including E-rate funding (Quality Education Data, 1999). Similar increases have occurred in higher education, with estimated total technology expenditures of $2.7 billion in 1999-2000 (Market Data Retrieval, 1999). About This Report In 1990, the Software Publishers Association (SPA) published its first "Report on the Effectiveness of Microcomputers in Schools." (SPA and IIA merged in 1999 to become the Software & Information Industry Association.) In that report, numerous research studies supporting the use of technology as a valuable tool for learning were described. These studies indicated that the use of technology as a learning tool could make a measurable difference in student (1) achievement, (2) attitudes and (3) interaction with educators and other students. The evidence suggested that positive effects of technology were dependent upon the subject area, characteristics of the student population, the teacher’s role, student grouping decisions, the design of the software and the level of access to technology. Since then, research documenting the effectiveness of educational technology has continued to grow and become more detailed. This seventh edition of the report continues the tradition of summarizing leading research on the effectiveness of technology in K-12 and higher education and on the effectiveness of specific software design characteristics. This report, commissioned by the Software & Information Industry Association (SIIA) and conducted by an independent educational technology consulting firm, Interactive Educational Systems Design Inc. (IESD), summarizes educational technology research from the late 1980s through 2000. This report is based on 311 research reviews and reports on original research projects, from both published and unpublished sources. Of these 311 studies, 135 were published in professional journals and 56 were doctoral dissertations. The 311 studies were chosen from an original set of more than 3,500. Research studies summarized in this report feature one or more of the following as part of their methodology: • • • • • • • A technique for synthesizing and analyzing data from many different studies (meta-analysis) Comparison of the use of technology to traditional instructional methods Comparison of different software designs Comparison of the use of technology by student populations with different learning characteristics Comparison of the use of technology under different learning environment conditions Analysis of student performance data (e.g., test results, performance assessments) Analysis of classroom observation and surveys of educators and students Studies were not included for a variety of reasons. Some were weak in methodology (e.g., comparisons of a computer-based instructional treatment to no alternative treatment) and others addressed topics not of concern in this report (e.g., critiques of typical research methods; research on the attitudes of student teachers; research on the design of the physical layout of technology-rich classrooms). 2000 Research Report on the Effectiveness of Technology in Schools 8 The report is divided into three sections: • • • Effects of technology on student achievement Effects of technology on student self-concept and attitude about learning Effects of technology on interactions involving educators and students in the learning environment Subsections that are new to this seventh edition of the report include the following: • Effects of technology on student achievement  Large-scale technology implementations  Curriculum areas and student achievement o Technical training  Software design characteristics and student achievement o Amount of practice o Pedagogical agents o Learner support in foreign language learning o Graphs in mathematics instruction o Spoken narration, text, and sounds with graphics o Multiple representations of arithmetic problems  Instructional decisions and student achievement o Exploratory vs. confirmatory approach to computer simulation o Task structure when students search the Internet o Medium for administration of assessment Effects of technology on interactions involving educators and students in the learning environment  Effects of technology on teacher-student interactions and educators’ instructional behavior o Technology in a project-based learning environment o Internet use • A comprehensive bibliography of the research cited is also included. Conclusions Technology is making a significant positive impact on education. The findings of greatest importance in the research reviewed for this report include the following. General • The specific student population, the software design, the educator's role, how the students are grouped, the preparedness of the educator and the level of student access to the technology influence the level of effectiveness of educational technology. Educators are an essential element in the effectiveness of technology. • 2000 Research Report on the Effectiveness of Technology in Schools 9 • • • Software is effective because it allows individual learner traits and multiple pathways to learning (e.g., text, graphics, speech) to be taken into consideration when software is being designed and used. Effectiveness of educational technology depends on a match between the goals of instruction, characteristics of the learners, the design of the software and technology implementation decisions made by educators. Students of teachers with more than 10 hours of training significantly outperformed students of teachers with 5 or fewer training hours. Positive Effects of Technology on Student Achievement Educational technology has demonstrated a significant positive effect on achievement. Positive effects have been found for all major subject areas, in preschool through higher education and for both regular education and special needs students. More specifically: • • Large-scale, statewide implementations of educational technology (in West Virginia and Idaho) have been correlated to gains in standardized test scores. In studies focusing on reading and language arts, technology has been shown to provide a learning advantage in the areas of phonological awareness (awareness of the structure of sounds in a language), vocabulary development, reading comprehension and spelling. Furthermore, there is evidence that students who use word processing software in combination with carefully sequenced instruction in the writing process or writing tools with built-in guidance in the writing process improve their writing significantly more than students without access to such tools, as do students who write to a real audience via the Internet or e-mail. Technology has been used effectively to support mathematics curricula that focus on problem solving and hands-on, constructivist, experiential activities. Students participating in such technology-supported learning experiences have demonstrated superior conceptual understanding of targeted math topics than students receiving traditional instruction. Studies focusing on science education suggest the benefits of simulations, microcomputerbased laboratories, video to anchor instruction to real-world problems, and software that targets students’ misconceptions. A learning advantage has been found when students have developed multimedia presentations on social studies topics. Kindergartners who have used technology have benefited in areas such as improved conceptual knowledge, reading vocabulary, reading comprehension, and creativity. Educational technology has significant positive effects on achievement for special needs populations. Speech recognition is an especially valuable compensatory tool for the learning disabled. Interactive video is especially effective when the skills and concepts to be learned have a visual component and when the software incorporates a research-based instructional design. Use of online telecommunication for collaboration across classrooms in different geographic locations can improve academic skills. Using the Internet to provide college students with supplementary instructional resources can benefit their academic performance. Use of distance learning has been shown to be as effective as instruction that takes place locally. • • • • • • • • • 2000 Research Report on the Effectiveness of Technology in Schools 10 Positive Effects of Technology on Student Motivation and Self-Concept • Educational technology has been found to have positive effects on student attitudes toward learning and on student self-concept. Students felt more successful in school, were more motivated to learn and had increased self-confidence and self-esteem when using computer-based instruction. The evidence of these effects is the strongest for:  Language arts and writing instruction  Mathematics instruction  Science instruction  Telecommunication technology, including the Internet  Video technology Educational technology has significant positive effects on student attitudes for special need populations. • Effects of the Teacher’s Role and Instructional Decisions • The teacher's role is of primary importance in creating an effective, technology-based learning environment—an environment that is characterized by careful planning and frequent interaction among students and the teacher. Teacher professional development and decisions about how computers are to be used in instruction may matter more than how often technology is used. Students trained in collaborative learning on computer in small groups had higher student achievement, higher self-esteem and better attitudes toward learning than students working individually. The positive effects of collaborative learning were especially pronounced for low ability students and for female students. Expanding student responsibilities through a learner-as-multimedia-designer environment can positively impact student attitudes. Decisions about the choice of medium when assessing student performance should reflect students’ experience writing on the computer. Testing students by having them write by hand has been shown to result in an underestimation of their writing abilities if they are accustomed to writing on computer. However, multiple choice tests administered via computer and using pencil and paper have been shown to yield similar results. • • • • Effects of Specific Software Design Features Specific software design elements have been shown to have a positive impact on student achievement and on student motivation and self-concept. Several research-based suggestions follow. • Offering students some control over the amount and sequence of instruction, including options for student review of material, can result in higher achievement and better student attitudes toward learning than having the software control all instructional decisions. However, low-achieving students and students with little prior content knowledge are likely to require more structure and instructional guidance than other students are. When students have a high need to learn, this may nullify the impact of the level of learner control. 2000 Research Report on the Effectiveness of Technology in Schools 11 • • In tutorial and practice software, programs with feedback providing knowledge of correct responses were found to be superior to programs that require students to keep answering until they achieve a correct response. Furthermore, feedback that identifies why a response is wrong was found to be more effective than feedback that only identifies what was wrong. Software that includes embedded cognitive strategies provides students with a learning advantage. Helpful cognitive strategies include:  Repetition and rehearsal of content  Specific note-taking techniques  Paraphrasing  Outlining  Cognitive mapping or diagramming  Drawing analogies and inferences  Generating illustrative examples  Having students explain their steps in solving problems  Specific techniques for reading in the content areas  Using pictorial information Students can benefit academically from software with embedded conceptual change strategies  sequences of instruction that move students from their faulty preconceptions to a more accurate understanding of the concepts involved. Instructional scaffolding—gradually decreasing the level of help available and/or gradually increasing the complexity of the task—can be effective in improving student achievement. Animation and video can enhance learning when the skills or concepts to be learned involve motion or action. Animation accompanied by spoken narration is generally superior to animation accompanied by explanatory text. When including narration, additional extraneous audio (e.g., music, sound effects) should be avoided. Still graphics can enhance learning when the concepts or skills to be learned have a visual component but do not involve motion or action. Content-related graphics (both static and animated) and video can help improve student attitudes and motivation in mathematics and science. Students using hypermedia software can benefit from an interface that includes a navigation map that shows the links among the various screens of information and the hierarchical structure of the information. It is also advisable to make the entire hyperspace to which students will eventually have access fully transparent while limiting their access to what is currently instructionally appropriate. Foreign language students can benefit from presentation of video segments with captioning (i.e., subtitles in the target language) and from access to native language translation when reading text-and-graphics dialogues in the target language. These design features are likely to make a difference for ESL students as well. Recent research additionally suggests possible benefits from inclusion of the following software design characteristics (subject to further exploration and confirmation): • • • • • • • • • 2000 Research Report on the Effectiveness of Technology in Schools 12          Providing sufficient practice Stating objectives Advanced organizers in simulations Pedagogical agents that communicate with a human voice in a personalized dialogue with the student Graphs in mathematics instruction Multiple representations of concepts Dynamic visualization of abstract concepts Motivational contexts, such as story, game, and fantasy elements Multiple window presentation options (overlapping vs. tiled windows) Effectiveness of Interactions Involving Educators and Students in the Learning Environment • Introducing technology into the learning environment has been shown to make learning more individualized and student-centered, to encourage cooperative learning and to stimulate increased teacher-student interaction. Technology has been used successfully to support constructivist, inquiry-based and project-based instructional methods. Specific characteristics of the learning environment help to maximize the benefits of educational technology: District-level involvement and the leadership of a school-level computer coordinator are key factors in developing a school environment conducive to effective use of technology.  Educators are more effective after receiving extensive training in the integration of technology with the curriculum.  Exemplary computer-using educators benefit from a social network of other computer-using educators at their school.  Exemplary computer-using educators typically have smaller class sizes and more funds available for software acquisition.  Educators should carefully plan, and actively participate in, learning activities that incorporate tool software. Before students use database software independently, they should be given search strategy training.  Educators should offer students self-directed learning experiences and activities that encourage self-expression.  Students benefit from personal interaction among class members. Courses for which computer-based networks were used increased student-student and student-teacher interaction increased student-teacher interaction with lower-performing students, and did not decrease the traditional forms of communication used. Many students who seldom participated in face-to-face class discussions become more active participants online. Classroom connectivity to the Internet was found to be the best predictor of teachers’ professional use of the Internet. Furthermore, classroom connectivity in general and, more specifically, connectivity with four or more computers were found to be important factors in predicting whether teachers directed student research involving the Internet.  • • • 2000 Research Report on the Effectiveness of Technology in Schools 13 • When upper elementary students use the Internet to conduct research, they tend to spend more of their time browsing rather than conducting carefully planned searches. Teachers are advised to provide a variety of support, possibly including "natural language" search engines, guided practice in conducting searches, broadly defined research tasks, and instruction in identifying and using relevant source material. These findings may apply to older students with limited Internet research experience as well. • Small group collaboration on the computer is especially effective when students have received training in the collaborative process. However, there are trade-offs in deciding whether students should work individually or collaboratively:  Students who worked in groups were found to interact more with their fellow students (including cognitive and positive social interactions), to use more appropriate learning strategies, and to persevere more on assigned instructional tasks.  Students who worked individually at the computer were found to spend more time actually engaged with the software and to complete their tasks more quickly, but they needed more help from the teacher. Greater student cooperation, sharing and helping behaviors has been shown to occur when students use computer-based learning in which students compete against the computer rather than against each other. • University and inservice teacher training provides educators with greater comfort in using computers, an increased desire to use computers and an understanding of how to integrate software into the classroom curriculum. • Preservice teachers have successfully used communication technologies such as email, news groups, and listserv mailing lists to exchange ideas on instructional issues. Preservice teachers who do this over several weeks have been shown to make greater gains in self-efficacy and confidence in their teaching abilities than teachers without access to such tools. • Positive changes in the learning environment brought about by technology are more evolutionary than revolutionary. These changes occur over a period of years, as educators become more experienced with technology. Long-time computer-using teachers tend to make changes in the learning environment generally related to a constructivist teaching approach. • This report provides software developers and publishers with research that will enable them to improve educational technology so that it continues to have a significant positive impact on student achievement, self-concept and motivation and on the interactions in the learning environment for students of all ages, capabilities, socio-economic backgrounds and areas of interest. The research reviewed in this report aims to help educators make effectual educational decisions as they incorporate technology-based learning experiences into the curriculum, increase student achievement and motivation in a variety of subject areas and consider advantageous software design characteristics when selecting software. 2000 Research Report on the Effectiveness of Technology in Schools 14 Section 1: Effects of Technology on Student Achievement Original research reports and reviews of educational research published between 1990 and 1998 confirm that microcomputers and other educational technologies have beneficial effects on student achievement. Research indicates the effectiveness of using technology to support instruction in a wide variety of curriculum areas, including: language development, reading, writing, spelling, mathematics, science and medicine, social studies, foreign languages, English as a second language, computer programming, psychology, accounting, music appreciation, technical training, career education and cognitive skill development (for example, Ferretti and Okolo, 1997; Katz and Hall, 1997; Miech, Nave and Mosteller, 1997; Farnsworth, l996; Luzzo and Pierce, 1996; Okolo and Ferretti, 1996; Schultz, 1995; Mayfield-Stewart, Moore, Sharp, Hasselbring, Goldman and Bransford, 1994; Woehler, 1994; Bangert-Drowns, 1993; Liu, 1993; Beyer, 1992; Yang, 19911992; Kulik and Kulik, 1991; Liao and Bright, 1991; Anderson-Inman, 1990; Land and Haney, 1990; Liao, 1990; Woodruff and Heeler, 1990). A growing body of research shows, however, that the effectiveness of educational technology depends on a match between the goals of instruction, characteristics of the learners, the design of the software and technology implementation decisions made by educators. In this section, we set out to accomplish three things: 1. Report on recent studies in which technology-based instruction is compared to other instructional methods 2. Explore several well-researched curriculum areas in depth 3. Review research that relates student achievement to learner characteristics, special populations, software design, more recent technologies and instructional decisions concerning technology implementation Comparison Studies Several recent studies present meta-analyses of the effects of technology-based instruction in comparison with other instructional treatments (usually traditional methods). Meta-analysis is a method of assessing the effects of technology-based instruction across many different studies, using a common measurement scale, called effect size (ES). ES can be interpreted as "the proportion of the experimental scores that are greater than the average score in the control group." An ES of 0.30 means that "62 [percent] of the experimental group scored higher than the average student in the [control] group." An ES of 0.70 means that 75 percent of the experimental group scored higher than the average student in the control group. An ES of 1.00 means that 84 percent of the experimental group scored higher than the average student in the control group (Ryan, 1991). 2000 Research Report on the Effectiveness of Technology in Schools 15 According to Kulik and Kulik (1991), an ES of 0.3 is considered to be a "moderate but significant effect." In their meta-analysis of 254 controlled evaluation studies covering students from kindergarten through higher education, they found that computer-based instruction (CBI) had an average ES of 0.3. Where differences in achievement were statistically significant, the difference favored CBI in 94 percent of the cases. In a follow-up meta-analysis of 97 studies, Kulik (1994) found an average ES of 0.38 for CBI involving software classified as drill-and-practice and tutorial. He also compared this type of software to other instructional innovations, adjusting for similarity of study conditions. Kulik found that Schools can dramatically improve the achievement of their high-aptitude learners by giving them school programs that provide greater challenge. The next most potent innovations involve individual tutoring by computers or by other students ...computer tutoring seems to be slightly more effective... Instructional technologies that rely on paper and pencil are at the bottom of the scale of effectiveness. Kulik found methodological problems with studies about Logo: higher ESs (several above 1.00) for studies in which the criterion test was individually administered yet lower ESs (several below 0.16) for studies in which the criterion test was group administered. Kulik found average ESs below 0.15 for other categories of software. Keep in mind, however, that the vast majority of the studies included for these analyses were published prior to 1986, when software quality was dramatically lower and the importance of curriculum integration was not as well understood. A meta-analysis by Fletcher-Flinn and Gravatt (1995) found that 120 studies conducted between 1987 and 1992 showed an average ES of .24 for computer-assisted instruction (CAI), with an ES of .33 for studies between 1989 and 1992. In looking at a range of factors including educational level, course content, publication year, duration of study, the same or different teacher for the control group, type of CAI and method of assigning study subjects, they found no significant differences in study results for any of these factors. They concluded that gains in proficiency resulted from "the superior quality of CAI materials, rather than some intrinsic aspect of computers...as vehicles of instruction." In other words, the quality of the instructional design of CAI is critical. It is noteworthy that new multimedia technology was not included in the analysis, because so few multimedia studies had been conducted in this area. In a meta-analysis by Fazal (1996) of 40 studies on higher education, adult education and military training conducted since 1990, computer-based instruction (CBI) yielded an average ES of .35. Breaking down the results by type of software, Fazal found that simulations had the greatest effect (ES = .69), followed by tutorial plus drill and practice (ES = .49), tutorial plus simulation (ES = .40), drill and practice (ES = .34) and tutorials (ES = .07, a very small effect). Of the 40 studies, 10 compared time required with CBI to time required with more conventional instructional methods. Based on these studies, Fazal found that CBI required, on average, only 70 percent of the time required for more conventional instruction. Because none of the studies used a systematic cost model, however, it is impossible to say to what degree these savings in time translate into overall cost-effectiveness. Fazal further determined, based on a limited number of studies, that effect sizes were greater when students worked in pairs and groups than when they worked individually, a finding that deserves further research. Ryan (1991) conducted a meta-analysis of 40 comparative studies focusing on the use of computers in elementary schools. She calculated an average ES of 0.309. Ryan found that the amount of 2000 Research Report on the Effectiveness of Technology in Schools 16 technology-related teacher training was significantly related to the achievement of students receiving CBI. Students of teachers with more than 10 hours of training significantly outperformed students of teachers with 5 or fewer training hours. (Survey research by Cates, McNaull and Gardner (1993) lends support for the importance of teacher training. Compared to educators with less training, educators with more than 3 credit hours of university coursework and more than 3 contact hours of inservice training rated themselves significantly higher on scales of computer expertise and computer comfort, had "significantly higher opinions of the usefulness of software," and reported significantly greater use of computers in the classroom.) In his meta-analysis of the effects of using a word processor as an instructional tool, BangertDrowns (1993) found an average ES of 0.27 for improvement of writing quality. He characterized this effect, based on 20 studies, as small but statistically significant. However, nine studies that focused on word processing in the context of remedial writing instruction yielded an average ES of 0.49, a significantly larger effect. Two meta-analyses suggest the instructional value of interactive video (IV). In Fletcher's (1990) meta-analysis of 47 studies of higher education and military and industrial training, it was found that students receiving instruction via IV had achievement scores that averaged 0.50 standard deviations above the scores of students receiving conventional instructional. McNeil and Nelson (1991) completed a meta-analysis of research on the cognitive achievement effects of IV, resulting in an overall ES of 0.502. They found significantly higher results when teachers used IV as a supplement to traditional instruction than when IV was used to replace traditional instruction. A meta-analysis by Cavanaugh (1998) summarized the results of 19 studies of distance education in grades 3 through 12 that used videoconferencing or on-line telecommunication. Fourteen of the 19 studies (74%) had a positive effect size for distance education. Based on a small but positive overall effect size of 0.147, Cavanaugh found that "when distance education is necessary in order to provide needed educational options to students, it can be as effective as traditional instruction." After eliminating three foreign language studies from the analysis, the overall effect size was 0.344, a moderate effect. The author concluded that distance education "cannot be considered to be more or less effective than traditional instruction." Similarly, another metaanalysis summarizing 19 studies of distance video telecourses for adults found that "there does not appear to be a difference in student achievement between distance learners and traditional learners" (Machtmes, 1998). These meta-analyses confirm that in general, technology has a positive effect on student achievement, and that distance learning is equivalent to traditional instruction in its effects on student achievement. The studies also underscore the importance of the teacher's role in the effective use of educational technology. Large-Scale Technology Implementations Two studies have examined the impact of large-scale investments in educational technology. Research sponsored by the Milken Exchange on Education Technology (Mann, Shakeshaft, Becker, and Kottkamp, 1999) reported on eight years' worth of statewide funding ($7 million a year) of West Virginia school districts to purchase approved hardware and instructional software for K-6 students as part of a Basic Skills/Computer Education (BS/CE) program. Implementation started at the K-1 level in 1990-91 and moved up the grade range in successive years. The program 2000 Research Report on the Effectiveness of Technology in Schools 17 featured a combination of engaging software with a basic skills focus, enough computers to provide regular and equitable access to all students, and professional development for all K-6 teachers. As part of their analysis, researchers looked at a representative sample of students from 18 schools that took the Stanford-9 achievement test as fifth graders in 1996-97 and again as sixth graders in 1997-98. Regression analysis showed that 11% of the variance in basic skills gain scores was accounted for by factors in the BS/CE program. The more access students had to computers, the more positive student and teacher attitudes were toward computers and technology, and the more training teachers received, the better the performance of the students. In a follow-up analysis by Solmon (1999), software access and time students spent with computers were calculated as having a combined effect size of .456. Solmon also estimated that a one-month gain in test scores using the BS/CE model might cost a little less than $200 per student. In comparison, he estimated that a one-month gain in test scores from reducing class size from 21 to 15 would cost over $650 per student. Solmon concluded that the technology-related costs are "several times less than the costs of achieving the same test score growth by class size reduction." Researchers from the Idaho Council for Technology in Learning (1998) reported on effects from the Idaho Technology Initiative, which provided $10.4 million a year for K-12 technology purchases and integration from 1994 to 1998—$12.55 per student per year, or $212.75 per student over the five-year period. School districts were allowed a great deal of flexibility in how and where they spent these funds. Based on a correlation of standardized test gains over several years to students' level of technology exposure, researchers found "a positive relationship between academic performance in...language, math, and reading and the integration of technology in Idaho's K-12 schools." Comparing the performance of all Idaho eighth graders taking the Iowa Test of Basic Skills (ITBS) with their scores on the same test four years earlier, students with a high level of exposure to technology experienced statistically significantly greater academic gains in reading, math, language, and a test category the researchers identified as "core studies" than students with low exposure to technology. Similarly, eleventh graders taking the Test of Achievement and Proficiency who had a high level of exposure to technology gained significantly more than low-technology exposure students in all four test areas, compared to their ITBS scores of three years earlier. These studies provide evidence that large-scale technology implementations can make a difference in the academic performance of students across a range of subject areas. Curriculum Areas and Student Achievement Language Arts and Reading Language Arts has been a frequent focus of educational technology research. Highlights are presented as follows. Language Development A Harvard University researcher examined the effectiveness of a beginning literacy system that combines interactive storybooks with activities to support reading, writing, speaking and listening (Schultz, 1995). Compared to first graders who received only traditional instruction, students who used the technology-based literacy system regularly over a period of three months demonstrated significantly greater gains in basic language skills, such as understanding the relationships among the parts of the English language, classification and reading comprehension. 2000 Research Report on the Effectiveness of Technology in Schools 18 A team of Vanderbilt University researchers found that "Multimedia Environments that Organize and Support Text" (MOST) have a positive impact on the language development of at-risk, innercity kindergartners (Mayfield-Stewart et al., 1994). The MOST environment includes animated video versions of children's stories, daily sessions with story sequencing and multimedia bookmaker software, story-writing activities, decoding software, "Little Read-Along" books and folktales and trade books that are thematically related to the video stories (Sharp, Kinzer, Risko, 1994). After experiencing the MOST environment for three months, a group of students was compared to students in a conventional kindergarten classroom. The MOST students showed significantly superior gains in auditory skills (sound discrimination and sound-symbol correspondence) and language skills (basic cognitive concepts and listening comprehension). The MOST students constructed significantly higher quality oral stories in response to "picture sequences of familiar events." Their stories included more actions, obstacles and propositions; evidenced better use of tense; and were judged at a higher narrative level. The MOST students also scored significantly higher on a test of decoding in context. Reading Foster, Erickson, Foster and Torgeson (unpublished) developed then tested a tutorial and practice program called Daisy Quest, which they designed to increase young children's phonological awareness. Phonological awareness is defined as "one's sensitivity to or explicit awareness of" the structure of sounds in one's language. The developers view the value of the program as helping "children understand the alphabetic principle during the early stages of reading instruction." The majority of practice in Daisy Quest is in recognizing identical sounds or sound combinations across words (e.g., the initial sound in "fish" and "fat;" the final sound combination in "dog" and "log"). The program uses high-quality digitized speech. In two separate studies and five different measures of phonological awareness, the computer-based approach was found to be significantly more effective than regular instruction. The average ES of 1.05 is considered significant. Stone (1996) found that second grade students who had received computer-assisted instruction (CAI) in reading and other areas since kindergarten scored significantly higher in both reading comprehension and vocabulary than students in a nearby school with no CAI. Both groups were approximately the same age and socioeconomic status. In addition, both groups followed the identical Board of Education-approved curriculum and spent the same amount of time in reading instruction. Students used several different software programs focusing on reading, vocabulary development and mathematics development. Stine (1993) compared the effectiveness of whole language reading instruction with and without interactive CD-ROM, computer-based books with second graders eligible for Chapter 1 remedial programs. Students who used the CD-ROM books demonstrated significantly greater gains in vocabulary and reading comprehension than students who did not. In a study by Woehler (1994), at-risk middle school students were shown to benefit from the basic reading skill software included as part of the Computer Curriculum Corporation (CCC) comprehensive courseware system. The CCC reading curriculum includes drill-and-practice software that addresses interpretive comprehension, literal comprehension, word meaning, word analysis and reference skills. The management system enables students "to work at varying levels dependent on [their individual] success and speed." Students who used this technology-based reading curriculum made significantly greater gains in reading achievement than students who did not have access to the system. This positive impact is consistent with results of a 2000 Research Report on the Effectiveness of Technology in Schools 19 meta-analysis by Kulik (1994) of over 20 studies of the CCC system. Kulik found an average ES of 0.40 for the system. Reading and Spelling In a study in Britain, Scrase (1998) reported on the effects of a multi-sensory system for teaching literacy skills that feature a speaking computer. Children and adults using the Starcross Indirect Learning (IDL) system listened to sentences dictated by the computer, then typed the sentences as the program read out the letters. Correct typing was displayed, while asterisks replaced incorrect typing. Research findings demonstrated that students improved both their reading age and their spelling age by six months or more for each month in the program. This represented 3.7 times their previous rate of progress for reading and 4.4 times their previous rate for spelling, a "highly significant" difference. Further analysis showed that the system is most effective for children with problems in their visual processing (dyseidetic dyslexics). Spelling Stone (1996) found that students who had used reading and language arts software from kindergarten through second grade scored significantly higher in spelling than students in schools who had not used the software. Students used several software programs, including a story-based reading development program, a spelling and vocabulary development program and a program in which students were taught to read by writing whatever they were able to say. Two studies identified by Anderson-Inman (1990) suggest that keyboarding and keyboarding software can be used to improve spelling for low performing students. In one study, (Hollen, 1987) students worked with keyboarding software that allowed educators to enter the words to be used in typing drills; students used it to practice spelling words and take spelling tests. In the other study, (McClendon, 1989) students practiced spelling using laminated keyboards not attached to computers; McClendon found that significant improvement in spelling scores resulted from a combination of keyboard-based practice and direct spelling instruction. Reading and Writing Green (1991) explored the effects of three different approaches to writing instruction on achievement in reading for inner city, Mexican-American third-graders. A writing process approach with word processing resulted in significantly higher reading achievement than a similar approach without word processing or a grammar-oriented approach to writing. Writing Researchers have found that student use of word processing software results in higher quality writing. Such results were reported for regular education students, (Owston, Murphy, Wideman, 1991). English as a Second Language (ESL) students (Silver and Repa, 1993; Li, 1990) and for special needs students (Bair, 1990). Bair found that emotionally disturbed students in grades 6-12 who used computers demonstrated superior performance in writing maturity, reduction in spelling errors and length of writing. Silver and Repa found that ESL high school students who used word processing software in combination with a developmental learning sequence on the writing process evidenced higher quality writing than students receiving equivalent instructional without word processing. Owston, Murphy and Wideman found that regular eighth graders' computer-written papers were rated significantly higher in quality than handwritten papers for overall writing competence, focus/organization, support and mechanics. They note, however, that 2000 Research Report on the Effectiveness of Technology in Schools 20 ...students appear to bring their own personal style of working to the word processing environment; computers do not of themselves dramatically change student writing style. Instead, they appear to facilitate whatever level of editing the user wishes to engage in. Nix (1998) found that fourth grade students who had been exchanging e-mail regularly with partners at another school performed better in persuasive writing tasks both on and off the computer than students who had not been using e-mail. In the first comparison, students who used e-mail to write a persuasive essay to their partners scored significantly higher in audience awareness and argumentation than students at the control school who composed handwritten essays. In the second comparison, after three months of using e-mail, students scored significantly higher overall, were more aware of their audience, were more organized and produced lengthier texts than students who had not used e-mail, even though both groups wrote their test essays by hand. It was determined in a recent study by Zellermayer, Salomon, Globerson and Givon (1991) that a computer writing tool that automatically provides ongoing, unsolicited guidance throughout the stages of the writing process improved the quality of student writing even after the support structure was removed. Students who had used this tool expended more mental effort and demonstrated significantly greater writing improvement than students who had used a tool that provided guidance only upon request or students who had practiced writing using only word processing software. Research by March (1993) suggests that the type of writing guidance provided in software makes a difference. He compared the effects of several variations of guided writing software with junior high students. March found that combining two forms of guidance had a positive impact on writing skill. One form of guidance is called structural chunking. Computer-guided structural chunking must...do two specific things. First, it must break a writing task into intelligible components. Second...the software must prompt the writer into the type of thinking an expert would perform faced with the same task. The other form of guidance is referred to as online guides. These include: • • • Ideas for generating writing, in the form of writing starters (e.g., "The place looked like..." or "One thing everyone should know about this subject is...") Evaluative strategies, in the form of questions that expert writers might ask themselves (e.g., "How does this go along with my first idea?") Examples In the study, structural chunking was presented automatically. Online guides were available to students on demand. Students using writing software that combined structural chunking and an online guide demonstrated superior writing ability than students who used writing software without such guidance. A study by Rockman et al. suggests that providing students with home computers and integrating their use with classroom instruction can have a positive impact on writing skill development. The Buddy System Project places computers in the homes of upper elementary school students in school districts throughout Indiana. It also provides support for educators' attempts to employ 2000 Research Report on the Effectiveness of Technology in Schools 21 innovative instructional strategies that take advantage of the increased access to technology. Rockman et al compared the gains in writing skill of Buddy and non-Buddy students in five fourthgrade and two fifth-grade classrooms (Rockman et al., 1995). They found that ...the Buddy program's efforts in writing resulted in gains more than three times higher than those in comparison schools. Several studies support the notion that teacher decisions are critical in effective writing instruction using a word processor. For example, Beyer (1992) compared a process approach to writing with regular word processing to writing instruction with only limited access to computers. The process approach with word processing included daily teacher use of the process approach to writing; student use of computers on a rotating basis to work on writing assignments; daily teacher use of a short writing lesson; teacher-student writing conferences; and ongoing teacher monitoring of student computer use and progress with writing assignments. Students using this approach significantly out-performed the group who had limited computer access on organization, development, standard writing conventions and overall writing quality. A study by Dailey (1991) suggests that assigning high school students to small cooperative groups when writing with a word processor is superior to having them write on computers individually. The students in small groups of three were preassigned roles of recorder, keyboarder or checker. Students working independently received whole class writing instruction before proceeding to the computer. Results indicated significant differences in favor of the small groups across the stages of the writing process. Valeri-Gold and Deming (1991), after reviewing research on computers and basic writing instruction, aptly conclude: ...the most effective utilization of computer software in the basic writing classroom combines the best of writing instruction theory with a creative use of computer technology. Only well-informed, trained and caring composition instructors will help to bridge the gap between technology and humanity. In her review of research on writing with word processors, Snyder (1993) found that when a sound model of teaching writing is implemented, students using word processing have demonstrated higher levels of achievement than equivalent students writing without word processing. Sormunen and Ray (1996) found greater improvement among university business writing students working in teams who used group systems software for collaborative analysis of problems and document writing and revision. Students in teams that used the software scored significantly higher on the message content and message organization components of the posttest and achieved significantly higher total posttest scores, than students who completed the same assignments without using the software. The researchers concluded: "The group support systems software lends itself to rapid and easy reorganization of message content." As a result, "use of the software may represent a better method of teaching collaborative writing in the classroom or practicing collaborative writing in the business environment." Mathematics Several studies compared instruction in mathematics using technology to instruction using traditional methods. 2000 Research Report on the Effectiveness of Technology in Schools 22 Elliott and Hall (1997) found that the use of computer-based mathematics activities enhanced mathematical achievement among at-risk four-year-olds. Children were placed into one of three groups, two of which used computer-based mathematics software. Children in the third group participated in a range of typical discovery-oriented preschool mathematics activities off-computer, together with computer activities in other areas. Students in both groups that used computerbased math activities had significantly higher posttest scores on the Test of Early Mathematical Ability—TEMA 2. Raghavan, Sartoris and Glaser (1997) reported positive impacts on student learning from the first two units of the Model-based Analysis of Reasoning in Science (MARS) middle school curriculum. These units, which focus on area and volume, include models (area tiles and volume cubes) that students manipulate on the computer screen to replicate, compare and construct twoand three-dimensional figures. Performance of sixth grade students who completed the units was compared with that of eighth grade students who had studied perimeter, area and volume in both sixth and seventh grade and had reviewed geometry concepts at the beginning of eighth grade Algebra I. The researchers found that although both groups performed about the same in simple application questions, the sixth grade students (who had completed the MARS units) significantly out-performed the eighth graders on questions requiring problem, data and concept analysis, resulting in a significantly better overall score. To evaluate long-term retention, some of the MARS students were also retested in December of their eighth grade year. Even after two years, the MARS students significantly out-performed other eighth grade students in Algebra I, with a mean score of 13.4 out of 25 compared with 7.9 out of 25 for the non-MARS students. Researchers from Leicester University (Underwood, Cavendish, Dowling, Fogelman and Lawson, 1996) found that students using mathematics software in an Integrated Learning System (ILS) at schools throughout the United Kingdom showed significant learning gains, compared to students not using the software. The study included students between the ages of eight and thirteen. Those in primary schools performed significantly better in the areas of addition, subtraction, multiplication and extensions. Students at the secondary level showed significant gains in the areas of operations and diagrams. Three studies were designed so that instructional time for the experimental and control groups was approximately the same. Stone (1996) compared second grade students who had used several mathematics and reading software programs since kindergarten with students in a nearby school who did not use the software. Both schools followed the same Board of Education-approved course of study. However, students who had used the software scored significantly higher in mathematics problem-solving on a standardized test. In another study by Fletcher, Hawley and Piele (1990), third and fifth graders received either CAI (Milliken Math Sequences) or traditional math instruction for 71 days. At both grade levels, students receiving CAI scored significantly higher than students receiving traditional instruction on a test of basic math skills, as well as on an assessment of computer literacy. CAI was rated as having greater cost utility. Reglin (1989-1990) focused on a seminar on mathematics skills, in preparation for an exam required for admission to teacher education programs. One group of students received nine hours of classroom instruction and nine hours of computer-assisted instruction (CAI); the other group received 18 hours of classroom instruction. Students in the classroom instruction-plus-CAI group showed significantly higher achievement gains than the students in the classroom instruction-only group. Since these studies were controlled for time spent on instruction, their results suggest that the computer experience, not merely time-on-task, account for the differences in student achievement. 2000 Research Report on the Effectiveness of Technology in Schools 23 Two studies by researchers at the Stevens Institute of Technology (Jurkat, Skov, Friedman, Pinkham and McGinley, unpublished paper) demonstrated the positive effects of commercially-available high school mathematics software on retention (i.e., performance on a delayed posttest). In one study, each student received instruction for two geometry topics, one with supplemental software and one without. One group used software for the first topic and the other group used software for the second topic. For retention, student performance was significantly better when instruction included software. In the other study, two groups of students were compared. One received instruction that included supplemental software and the other group did not use software. Once again, the group using software demonstrated significantly better retention (70 percent better) than the group who did not use software. The researchers point to the technology experience of the teachers in the computer-mediated treatments as a critical factor. They note that the educators "had been part of a computer-mediated education development project for more than two years." Funkhouser and Djang (1993) found that high school algebra and geometry students who used commercially available problem-solving software scored significantly higher on tests of mathematics content than a comparable group of students who did not use the software. The students using the software also made significant gains in problem-solving ability. Mac Iver, Balfanz and Plank (1999) found significant improvement in knowledge and application of math procedures by Philadelphia seventh graders who participated in CATAMA, "a combination of computer-assisted instruction and structured cooperative learning." In place of an elective class, students spent one class period a day for 10 weeks working in pairs on a tailored lesson sequence covering "a broad spectrum of proficiencies in math computation, math concepts, word problems, geometry, algebra, and thinking skills." Performance of CATAMA students on the end-of-year Stanford 9 Mathematics Procedures test was compared to that of non-CATAMA students whose performance had been similar on the previous year's test. Researchers found that "mathematics procedures achievement scale scores were much higher for typical CATAMA participants than for students in the comparison sample." This translated into an 11-point difference in the end-of-year national percentile rankings: 54th percentile for CATAMA students, 43rd percentile for the comparison group. Mac Iver, Balfanz and Plank also noted one particularly practical benefit of these results for students from a school that comprises over 85% low-income families and over 50% non-native English speakers: Many of the special admission high schools in Philadelphia...require a student to score at [85th local] percentile or above at the end of the seventh-grade year in order to be considered for admission as a ninth-grader. Eighteen of 48 CATAMA participants reached the 85th citywide percentile on their mathematics procedures test as opposed to only 7 of the 48 matched students from the comparison sample. Tucson Unified School District #1 (1994) reported positive results for "an integrated curriculum revolving around the use of a networked business computer system." The computer network gave teachers and their seventh grade students access to a variety of commercially available tool and reference software products. For two consecutive years, students involved in this innovative curriculum demonstrated significantly greater improvement in mathematics basic skills than comparison students receiving traditional instruction did. Furthermore, the computer-using students had a lower average number of absences and a lower rate of withdrawal from school. 2000 Research Report on the Effectiveness of Technology in Schools 24 Koedinger and Sueker (1996) found that college students using the Practical Algebra Tutor (PAT), an intelligent computer tutor that presents skills "in the context of authentic, realistic problem-solving tasks," scored significantly higher in a performance assessment of algebraic problem-solving, qualitative reasoning and the ability to communicate effectively about mathematics than students who did not use PAT. Scores on the departmental final exam did not differ significantly for the two groups, demonstrating that time spent using PAT "did not detract from the...acquisition of the algebraic manipulation skills targeted" by the final examination. This is good news for schools seeking to incorporate both performance-based assessment and traditional testing in their student evaluation program. In a related study, Koedinger, Anderson, Hadley and Mark (1997) compared performance of ninth grade students using PAT in 20 nontraditional classrooms with that of two other groups using a more traditional curriculum: five standard ninth grade algebra classes, and two "Scholar" academic track algebra classes. Students using PAT worked in cooperative teams on a National Council of Teachers of Mathematics (NCTM) Standards-oriented problem-solving curriculum, where teachers were freed to provide more individualized help to students. On two standardized tests with basic skills objectives, PAT students scored as well or better than the standard algebra classes (significantly higher on one test), even though this was not a primary focus of the curriculum. On two NCTM-oriented tests, they "significantly and substantially" out-performed the standard classes and equaled or exceeded the Scholars classes. This included one test of students' abilities to translate between verbal descriptions, graphs, and symbolic equations, where students using PAT scored significantly higher than the Scholars students did. This set of findings attests to the potential for using technology in a problem-solving curriculum without sacrificing basic skills. In a study by Alexander (1993), college algebra students who used software designed "to aid in the instruction of functions using concrete visualization" plus a graphing calculator significantly outperformed students receiving conventional instruction in their "understanding of the concept of functions...and in mathematical modeling." The results in this study suggest the power of the computer to provide concrete visual support for the learning of abstract concepts. O'Callaghan (1998) found that students using technology in a constructivist college algebra class "achieved a better overall understanding of functions and were better at the components of modeling, interpreting, and translating" among symbols, tables and graphs than students in two traditional algebra classes. Students in a Computer-Intensive Algebra (CIA) course completed activities that typically required them "to solve problems, often with the help of computer tools (e.g., symbol-manipulation programs), and to describe their methods of solution." Results demonstrated that CIA students had a significantly deeper conceptual understanding of algebra. Gerretson (1998) compared the geometry performance of two groups of preservice elementary teachers enrolled in a mathematics methods course. Both groups completed hands-on, experiential activities designed to help them develop a deeper understanding of the concept of similar geometric figures. The control group used manipulatives and traditional tools such as a protractor and ruler, together with a mechanical device for drawing similar figures. The experimental group used Geometer's Sketchpad software. Results showed that students using the software performed significantly better on a posttest measuring understanding of the concept of similarity. The researcher attributed this difference to the fact that students using the software were able to generate multiple examples of similar polygons quickly and easily, while students in the control group were limited to a smaller number of examples they could construct using handheld tools. 2000 Research Report on the Effectiveness of Technology in Schools 25 Two studies by the Cognition and Technology Group at Vanderbilt University (1991) explored the effectiveness of The Adventures of Jasper Woodbury, a video series focusing on mathematical problem-solving. [The video series is] designed to promote problem posing, problem-solving, reasoning and effective communication. Each adventure is a 15-20 minute story. At the end of each story, the major character (or group of characters) is faced with a challenge that the students in the classroom must solve before they are allowed to see how the movie characters solved the challenge. Students in Jasper classes out-performed students receiving traditional math instruction in solving one-step, two-step and multi-step word problems. Jasper students demonstrated greater skill in planning for problem-solving and generating "the subgoals necessary to achieve a larger goal." Jasper is a good example of software that engages students in the construction of their own knowledge and that anchors instruction in the context of meaningful, real-world problems. Fortunately, the results demonstrated that "spending time on Jasper did not negatively affect performance on standardized tests." (Jasper students performed marginally better than the control students, but this difference was not statistically different.) A subsequent Vanderbilt study (Barron, Vye, Zech, Schwartz, Bransford, Goldman, Pellegrino, Morris, Garrison, and Kantor, 1995) suggests that supplementing Jasper videos with technologybased thinking tools and follow-up activities help students to apply what they've learned to new problem situations. The supplementary software focuses on four different parts of the problemsolving process. Smart Lab...provides feedback to students about decisions they have made in the course of their work... Roving Reporter...[provides interviews with] various students in the learning community [who speak] about the problem-solving they have been doing...to showcase student reasoning and provide an opportunity...to react to various ideas. Toolbox...provides ideas for visual representations (e.g., timelines, graphs) that can be conceived of as "tools" for thinking, problem-solving and communicating. The Challenge...[gives] students a new problem-solving challenge...[and prompts them to] revise their work based on feedback they had just received. On a new problem scenario that required application of the mathematical skills they used during the original Jasper problem, students who used these supplementary tools demonstrated significantly greater problem-solving skill than Jasper-only students did. A study by Shyu (1997) found that use of the Encore Vacation videodisc as part of an anchored instructional program led to similar improvement in problem-solving skills among Taiwanese fifth grade students. Students watched the story segments linearly, then took a pretest designed to test their abilities in problem identification, problem formulation, generation of subgoals and execution of an appropriate solution. Students then watched the individual video segments again and worked cooperatively over eight class periods to solve mostly mathematical problems based on information from the videodisc. After completing the instruction, students performed significantly better on 2000 Research Report on the Effectiveness of Technology in Schools 26 a posttest of problem-solving skills. These results were consistent across high-, mid- and lowability groups in mathematics and science, demonstrating that videodisc-based anchored instruction can be an effective tool for teaching problem-solving across both cultures and ability levels. Researchers at the Educational Testing Service Policy Information Center (1998) raised an important caveat with respect to student achievement and the use of educational technology in mathematics: teacher professional development and decisions about how computers are used in instruction may matter more than how often technology is used. In their analysis of National Assessment of Educational Progress of 1996 (NAEP) data for 6,227 fourth-graders, they found that mathematics achievement was related to using computers for "learning games" (a category that likely includes activities that promote higher-order thinking skills) and to teacher professional development in the use of computers. In their analysis of NAEP data for 7,146 eighthgraders, they found that mathematics achievement was related to teacher professional development in the use of computers in general and in their use to teach higher-order thinking skills. However, at this grade level, "the use of computers to teach lower-order thinking skills was negatively related to academic achievement." At both grade levels, sheer frequency of use of computers in schools was negatively related to mathematics achievement. Because there was no detailed correlation between type of software and frequency of use, it is impossible to say whether this finding applies to use of computers generally or if it simply reflects ineffective, time-consuming use of computers to teach lower-level skills. It should be noted that the NAEP mathematics assessment was designed to align with the NCTM Curriculum and Evaluation Standards for School Mathematics (1996) and therefore focused on striking a balance among conceptual understanding, procedural knowledge and problem-solving. Given the extensive changes in mathematics instruction over the past several years, it is possible that much of the software used by the students in the study was directed toward more traditional goals and methods and therefore may not correlate well to current NCTM Standards. Science and Medicine Researchers in Turkey (Yalçinalp, Geban and Ozkan, 1995) compared the performance of students using a CAI chemistry program with students receiving the same content through lecture and discussion. Eighth grade students received supplementary instruction on chemical formulas and the mole concept through eight hours of CAI or teacher recitation over a four-week period. The CAI program incorporated presentation of concepts, examples and problem-solving exercises. At the end of the period, students who used the CAI scored significantly higher in their understanding of topics related to the mole concept than those who had received similar instruction through recitation only. A McGill University researcher (Wilkie, 1994) found that software that provides auditory narration and animation to visually represent biology concepts is an effective aid to learning. Sixty junior high students, of which half were judged learning disabled (LD), participated in the study. Two groups used two variations of the computer program, called INFECTRON, designed to teach "how the body defends itself against infection." A third group did not use the software during the study. The students who completed INFECTRON significantly outgained non-INFECTRON students on a test of declarative knowledge and on reasoning tasks that required application of this knowledge in real life situations. This advantage was demonstrated for LD and non-LD students alike. Two recent studies demonstrated the advantages of combining hands-on science instruction with tool and simulation software. 2000 Research Report on the Effectiveness of Technology in Schools 27 Gardner, Simmons and Simpson (1992) found evidence of the benefits of hands-on meteorology activities combined with content-specific tool software. Three groups of third-graders were compared: one group receiving hands-on activities with software; one receiving hands-on activities without software; and one receiving traditional classroom instruction. The hands-on activities with software group significantly out-performed the hands-on activities only group on a test of meteorology knowledge. Both of these groups scored significantly higher than the students receiving traditional instruction did. Lazarowitz and Huppert (1993) had similar results with high school biology students. One group received classroom-laboratory instruction that included use of a software program that combined simulated experiments and laboratory analysis tools. The other group received classroomlaboratory instruction only. The group using the software demonstrated significantly higher achievement in content knowledge and in the science process skills of graph communication, data interpretation and controlling variables. Research by Svec (1994) suggests that college physics students can benefit from a microcomputerbased laboratory (MBL). In this study, the MBL consisted of ...motion activities that use sonic rangers and computers to gather and display data graphically...options [for testing and learning] include rapid and frequent graphing of motion, prediction and a test of prediction with data and a variety of activities. The MBL was used to supplement lecture-based instruction. Students were shown a type of motion and were prompted to make predictions about the corresponding graph. Then they would carry out the motion and watch as the MBL generated the graph, allowing them to compare their predictions to the actual results. Students who experienced this MBL-and-lecture approach demonstrated a 15 percent improvement in a test of graphing interpretation skills, whereas students who received traditional lab-and-lecture instruction actually showed a decline. And where the traditional lab-andlecture students demonstrated a 25 percent improvement in a test of motion content graphing, MBL-and-lecture students showed a 112 percent gain. A group of Virginia researchers (Kinzie, Strauss, Foss, 1993) found evidence of the benefits of an interactive videodisc-based simulation of frog dissection. On a test of anatomical identification, there was no statistical difference in the achievement gains of high school biology students who used the simulation only and students who completed a dissection without completing the simulation. What is more important is that students who used the simulation as preparation before completing an actual dissection significantly outgained the dissection-only students. Furthermore, the simulation-then-dissection students "were found to be more effective in conducting the dissection than those who had received no preparation." Research at Vanderbilt University (Goldman, Petrosino, Sherwood, Garrison, Hickey, Bransford and Pellegrino, 1996) suggests that using video to anchor instruction "in interesting and realistic problem-solving environments" is an aid to chemistry learning. Students were divided into two groups. One group was shown network news coverage of a train derailment and chemical spill near a lake. The news segment covered how the spill spread throughout the local region, a biologist's examination of the damage and the negative impacts on "fish, vegetation and recreational life of the river system." The second group watched the same news coverage plus an "anchoring" video about an incident involving an overturned tanker in a "day-in-the-life" format. In the anchoring video, a hydrologist explains her job and then is faced with a tanker spill of an unidentified dangerous chemical onto a highway and into an adjacent river. After more information about the properties of 2000 Research Report on the Effectiveness of Technology in Schools 28 the chemical is presented, the video fades to black and students are asked to identify the chemical and to answer questions about the spill's consequences. Students are given access to authentic data resources with which to solve the problems. Then the video continues and the hydrologist, her team and a chemist demonstrate how they solve the same problems. Students experience two additional sequences following the same pattern: video depiction of problems and presentation of data, the opportunity for students to solve the problems using authentic materials, then video presentation of "experts" showing how they actually solve the problems. Students who experienced the news coverage plus the anchoring video learned significantly more chemistry content than students who only watched the news coverage. A study conducted at the University of Minnesota explored the use of computer-assisted instruction units on diffusion and osmosis in a university-level biology survey class (Jensen, Wilcox, Hatch and Somdahl, 1996). Program modules targeted students' misconceptions with specific questions and graphic illustrations of possible answers, including wrong answers. Students then made choices, frequently producing incorrect answers and "providing the instructor with a series of 'teachable moments' to explain the topics further." Posttesting showed that the understanding of diffusion and osmosis concepts among students in classes using the program was significantly greater than among students in classes where it was not used. Clark (1996) found that innovative use of technology could reduce time needed for a laborintensive tutoring system without reducing its effectiveness. Students in a college course focusing on thermodynamics and light were required to submit four portfolio assignments, which were reviewed by a coach via e-mail and sent back to the student for revision before final submission. Coaches for one group of students generated original, "custom" comments in response to assignments. Coaches for the other group read the students' draft assignments, then selected pregenerated responses from a "cyber-coach," editing them as needed to respond to unique features of students' work. Assignments were rated both before and after coaching on the basis of integration of knowledge, a key goal for the portfolios, and both groups showed significant improvement of 0.5 on an eight-point scale. However, cyber-coaching shifted the ratio of coach to student from 1 to 21 to 1 to 80, with the average interaction reduced from 15-20 minutes to 2-3 minutes per student assignment, while achieving equivalent results. Farnsworth (1996) studied the effectiveness of a set of four simulations that were developed for students in a veterinary program. "These simulations were developed to be delivered via computer because there was not enough room in the class schedule or faculty-time available to hold tutor-led small group discussions," thus providing a resource that otherwise would have been unavailable for students. Within the simulations, students were presented with specific problems and encouraged to emulate clinical reasoning by asking physical examination questions. Corrective feedback was provided after students had identified solutions. Over the course of the simulations, student diagnostic ability improved, showing "a significant relationship between repeated use of case simulations...and increase in diagnostic efficiency," as well as significant savings in time. Social Studies In a study by Ferretti and Okolo (1997; also described in Okolo & Ferretti, 1996) middle schoolaged students, most of them (17 out of 21) diagnosed as learning disabled, showed significant gains in knowledge about the American Revolutionary War after developing collaborative projects on related topics using word processing or multimedia software. Before taking part in the study, students had completed four textbook-based lessons on the Revolutionary War, but a knowledge 2000 Research Report on the Effectiveness of Technology in Schools 29 pretest showed they had learned very little during those lessons. After completing collaborative projects, however, students more than doubled their scores on the knowledge test, resulting in a large educational effect size of 1.9. This finding suggests "project-based inquiry in the social studies may substantially augment the effects of textbook-based instruction." Another study by the same researchers (Ferretti & Okolo, 1997) compared two classes of sixth grade students who created multimedia presentations on the Spanish colonization of Latin America with another class that completed a textbook-based unit on the same topic. They found that compared to pretest scores, on a knowledge posttest the two experimental classes ...significantly outperformed the control class. While scores in both experimental classes increased, scores in the control class decreased slightly. Subsequent analysis showed that although students diagnosed with learning disabilities scored lower on both the pretest and posttest, those in the experimental classes made approximately the same gains on the knowledge test as other students in those classes. Foreign Language Miech, Nave and Mosteller (1997) surveyed 22 empirical studies published between 1989 and 1994 and 13 reviews and syntheses published between 1987 and 1992, all on computer-assisted language learning (CALL) in higher education in the United States. They found that The most consistent finding reported in the research on CAI [computer-assisted instruction] and CALL up to the early 1990s is that, although CAI and CALL are effective in improving student achievement when used as a supplement to traditional classroom instruction, neither is apparently effective as a replacement for traditional classroom instruction. These researchers described a study by Avent (1993) in which students in beginning German classes who completed CALL courseware units created by the researcher instead of going to the language laboratory scored significantly better on grammar and vocabulary. The mean score on the final examination for students who completed the CALL units was 82.2 for grammar and 79.7 for vocabulary (out of 100), compared to 73.4 and 70.0 for the control group. Based on their review of the literature, Miech, Nave and Mosteller also identified several general principles of effective CALL design. They reported that programs requiring extended responses and greater interaction result in higher student achievement than those involving more limited responses. Similarly, traditional drill and practice CALL programs are more effective if students are required to understand the meaning of sentences in which corrections are to be made, instead of simply making the corrections mechanically. CALL is also more effective if students share in the control of program features such as the level of help information that is accessed at any one time. English as a Second Language (ESL) Tozcu (1998) reported that college-level English as a second language (ESL) students who used software for focused practice with frequent vocabulary words showed significantly greater improvement in both vocabulary knowledge and reading comprehension than students who did not use the software. Three hours a week for eight weeks, students completed activities on the 2000 Research Report on the Effectiveness of Technology in Schools 30 computer designed to build knowledge of the words most frequently used in English, in addition to their regular ESL class sessions. Computer activities included choosing words to match meanings, choosing meanings to match words, choosing the missing word from a sentence, and spelling missing words or words that matched a definition. Students in the control group spent a similar amount of time reading articles, writing summaries, and answering comprehension questions— activities similar to regular class instruction. Students who used the software not only scored significantly higher on a vocabulary posttest, but also improved significantly more in their reading comprehension than students in the control group. Liu (1993) found that hypermedia software designed to permit exploration of semantic networks can help international college and graduate students (non-native English speakers) in English vocabulary development. A semantic network is the web of interrelated concepts that represents one's depth of understanding of any given concept. For example, a semantic network for the concept lawyer could include trial, judge, defendant, jury, etc. Students who used the software demonstrated significant gains in vocabulary knowledge. Logo and Other Programming Languages The effects of Logo and other computer programming languages on student cognitive abilities continue to be a focus of research. Meta-Analyses In two meta-analyses of the effects of computer programming on cognitive performance (e.g., "reasoning skills, logical thinking and planning skills and general problem-solving skills"), programming had an average ES of 0.41 when compared with non-computer instructional methods (Liao & Bright, 1991, Liao, 1990). This suggests that the "learning of programming can positively enhance students' cognitive performance." Logo and Creativity Clements (1991) compared Logo's effect on the creativity of eight-year-olds with the effects of other "creative" uses of computers and a scheduled school "activity period" for special interest activities. As part of the Logo treatment, students were introduced to basic Logo commands and presented with simple "challenges." Later the concept of "procedural thinking" was introduced and encouraged as students engaged in programming. Component processes of procedural thinking were given human-like identities, so that the students could better relate to them and remember them. Students chose their own projects and were prompted to "reflect on their use of componential processes." The other computer group wrote compositions (using an integrated prewriting, word processing and editing package) and created drawings (using graphics software). As with the Logo group, there was a focus on processes, student selection of writing and drawing projects and "interpersonal interaction with peers and (the same) teacher." After 25 weeks, students in both computer groups significantly outscored the non-computer group in verbal creativity. The Logo group demonstrated significantly higher performance in figural creativity than either of the other groups. It is important to note that these positive effects were the results of both a computer environment and a teacher-dependent instructional environment. 2000 Research Report on the Effectiveness of Technology in Schools 31 Logo and Problem-solving A group of researchers found that pairs of students engaging in Logo activities achieved significantly higher gains in metacognitive processing than pairs of students who used CAI problem-solving programs (Nastasi, Clements, and Battista, 1990). Metacognitive processes are those related to planning and evaluating one's thinking while solving problems. The research suggests two characteristics of Logo that may account for its effectiveness: (1) it stimulates students to make their own rules; and (2) its open-ended structure allows students to successfully resolve their cognitive conflicts. Logo's ability to stimulate resolution of cognitive conflicts based on "ideas relevant to the problem's solution" has been confirmed in subsequent research (Nastasi & Clements, 1993). Researchers found evidence that cognitively based resolution of cognitive conflicts helps promote higher-order thinking. Logo and Geometry Concepts Three recent studies demonstrated that the use of Logo had a significant effect on students' understanding of basic geometry concepts. In one study (Clements & Battista, 1990), fourth graders who used Logo developed "mathematically sophisticated and elaborate ideas of angle, angle size and rotation." In the second study (Johnson-Gentile, Clements, and Battista, 1994), fifth and sixth graders who experienced the Motions strand of the Logo Geometry curriculum (focusing on symmetry, geometric motions and convergence) significantly outscored students receiving regular mathematics instruction on a test of motion concepts. Furthermore, students who received the Logo Geometry curriculum including experience with Logo software significantly outperformed students who received the Logo Geometry curriculum without Logo software on a delayed test of motion concepts. In the third study (Yusuf, 1995; Yusuf, 1991), seventh and eighth graders who used Logo significantly outscored students who received traditional math instruction on a test of the concepts of point, ray, line and line segment. The Logo students also demonstrated superior conceptualization of these geometry concepts in interviews. Keep in mind that the Logo-based instruction students received in all of these studies required the active participation of Logo teachers. The achievement effects are the result of an interaction between the software's characteristics and the classroom environment educators create. Technical Training Katz and Hall (1997) explored use of a computer-based tutoring system for training air force maintenance technicians in advanced troubleshooting. Software developers created the Sherlock tutor by engaging experts in structured interviews in which "pairs of experts interact during a verbal simulation of a troubleshooting scenario, with one expert acting as the problem solver, and the second expert essentially simulating the equipment." The tutor provided novice technicians with coached practice in troubleshooting on an equipment simulation, together with focused postperformance feedback on "reasoning errors and violations of good troubleshooting practice." The researchers found that technicians who completed Sherlock significantly outperformed a comparable group of technicians who had not used the software on both a field test involving equipment that technicians used on their jobs and a test of generalizability on an unfamiliar system. Further, the novices' scores on both tests were comparable to those of a group of master technicians with "over four times the job-related experience." These findings suggest that a similar development model could be effective in creating intelligent tutoring systems for other technical fields with a significant troubleshooting component. 2000 Research Report on the Effectiveness of Technology in Schools 32 Career Education Luzzo and Pierce (1996) determined that the use of the DISCOVER computer-assisted career guidance system significantly increased career maturity scores among middle school students. Students in the treatment group attended the school's computer lab for an hour each day over a twoweek period, completing all three of the DISCOVER modules. Students in the control group were taught a unit on oral and written business communication. At the end of the two-week period, students who had completed the DISCOVER modules scored significantly higher on an attitude scale posttest, although both groups had performed similarly on the pretest. The researchers noted: Such a result highlights one of the potential benefits of [computer-assisted career guidance], namely that significant changes in a student's readiness to make realistic educational and vocational decisions can be realized in a relatively brief period of time. When considered together, the studies described above suggest that the use of software and technology help students to achieve in a variety of curriculum areas. Learner Characteristics and Student Achievement Several studies support the conclusion that different types of learners respond differently to specific programs and software features. Weller, Repman, Lan and Rooze (1995) compared learning among users of hypermedia-based instruction (HBI) with different cognitive styles. Eighth grade students used the Computer Ethics Stack, an HBI program presenting specific scenarios dealing with computer ethics. Different versions of the program included advance organizers (e.g., a screen explaining four types of computer crime) and/or structural organizers (e.g., screen location titles). The researchers found that with all versions of the program, students who had been identified as field-independent learners performed better than field-dependent learners did. (According to Chou and Lin [1998], field-independent learners tend "to experience the parts, as distinct from the field, as an organized whole," whereas field-dependent learners tend "to experience the parts as fused into the whole.") Performance of field-dependent learners varied on different versions of the program, with the highest scores occurring when they used the version that included neither advance organizers nor structural organizers. They concluded that ...designers of HBI should look for ways to take into account the users' cognitive styles. Perhaps hypermedia software should initially diagnose a user's cognitive style and then provide optional modes of interactivity depending on this learner characteristic. Lin and Davidson-Shivers (1996) correlated the performance of students with more field-dependent or field-independent learning styles to different linkage structures for a hypertext lesson. Students in an undergraduate computer literacy course were first rated on a 19-point scale indicating their relative degree of field-independence. Students then explored one of five versions of a hypertext lesson on the Tiananmen Square events in Beijing and completed a posttest on the lesson content. Overall, students with a more field-independent learning style tended to out-perform students with a more field-dependent learning style. No significant differences in performance were found for different lesson linkage structures, a finding that runs counter to the suggestion that more structured hypertext designs are more effective for field-dependent learners. However, a 2000 Research Report on the Effectiveness of Technology in Schools 33 survey of student attitudes found that students tended to like the hierarchical structure (access to content nodes that are hierarchically directly above or below the current node) and hierarchicalassociative structure (similar to the hierarchical, but with additional links to nodes that are related by reference to the same term or concept) more than a strictly linear structure. Niederhauser, Salmen, Skolmoski and Reynolds (1998) correlated learning from a hypertext program among undergraduate students in an educational computing class to their prior use of computer applications to seek information (knowledge-seeking) or for entertainment (featureexploring). They found a significant positive correlation between knowledge-seeking and performance, but a negative correlation between feature-exploring and performance. This finding is consistent with the hypothesis that knowledge seekers, those who "strategically access logical sequences of screens and acquire information in a systematic manner," learn more from hypertext than feature explorers, who "spend their time trying to [understand] how the program works and what kinds of screens it contains." Hsu (1996) found that patterns in the use of hypermedia programs are strongly affected by psychological type and learning style. In a neuroscience class at Stanford Medical School, students used an interactive hypermedia program to view neuroanatomy text-based information, line diagrams, color cross-sections and dissection images. Patterns in the use of hypertext links to move among different sections of the program varied widely, reflecting both psychological type and learning style. For example, Intuitive perceivers, or those who rely heavily on intuition in processing information about the world around them, used hypertext links to make nonlinear transitions among different parts of the program. These included jumps between different program resources, from text to pictures and from one section of the brain and spinal cord to other, nonadjacent sections. On the other hand, Sensing perceivers, or those who rely heavily on their senses to process information, used more linear transitions. These effects were reduced in cases where the learning style of the students did not closely match their psychological type. For example, Intuitive students whose learning style included a preference for arranging information sequentially were more linear in their use of the program than Intuitive students with a random learning style. In general, pictorial information was used much more extensively by students than text information. This is not surprising, given the type of learning involved in the study of neuroanatomy. The results of this study suggest the value of hypermedia learning programs that offer both textbased and graphic-based information sources and that provide multiple pathways to information in order to meet the individual needs of different students. In a study of dental hygiene students with different levels of spatial ability, Bastecki and Berry (1996) found that students with high spatial ability learned better when information was presented in an overlapping window format, while those with low spatial ability learned better from a tiled window format. The researchers concluded that ...learners prefer to impose their own spatial schemata on the information presented... Compatibility of the learner's spatial ability level with the appropriate window environment has been shown to benefit the educational outcomes. Moreno and Mayer (1999b) analyzed the learning of sixth grade students who completed a multimedia program that taught addition and subtraction of signed (positive and negative) numbers. The program included not only symbolic representations of number sentences (e.g., 2 – -3 = __), but also animation featuring movement of a bunny along a number line to represent the operations. 2000 Research Report on the Effectiveness of Technology in Schools 34 They found that students with high spatial ability experienced significantly higher gains from pretest to posttest than students with low spatial ability. In another experiment, the same researchers compared the learning of high-achieving and low-achieving sixth grade students (based on pretest scores) who used two versions of the program: one with and one without the animation and number line representation. They found that high-achieving students who used the version with the number line and animation (referred to as the "multiple representation" or MR version) performed significantly better on the posttest than high-achieving students who used the single-representation (SR) version; however, no such difference was found for the lowachieving students. The researchers concluded: When using multimedia instructional methods we are faced with a trade-off problem. On one hand, multimedia can facilitate learning by representing concepts in more than one modality. On the other hand...the more representations offered for the same procedure, the heavier the cognitive load and the harder the learning. Our study provided evidence that learning in a multimedia MR environment benefits higher achieving and high spatial ability students the most, although lower achieving and low spatial ability students do not learn significantly more from MRs than from a SR. Special Populations and Student Achievement Early Childhood Education Carlson and White (1998) found that use of a commercially available software program significantly improved kindergarten students' understanding of the concepts of left and right. Students spent 10 minutes a day for two weeks working with the "Jellybean Hunt" activity from Edmark Corporation's Trudy's Time and Place House. In the program, students had to use combinations of left, right, and forward directions to navigate a hungry ant to colored jellybeans on a napkin. These afternoon kindergarten students scored significantly better on the posttest than another group of students who attended the same kindergarten class in the morning. There was no significant difference between the two groups before the treatment. The authors concluded, This research confirms that it is possible to provide students, as early as kindergarten, with the opportunity to have a favorable experience with a computer while enhancing their understanding of a particular educational concept. Two studies support the conclusion that well-designed computer-based activities, when presented with the active participation of a professional teacher or trained tutor, can increase young children's cognitive abilities. Goldmacher and Lawrence (1992) compared two groups of Head Start preschoolers; one group received a computer enrichment program (Computertots) and the other group engaged in standard Head Start activities. The computer-based activities involved a wide variety of software, a professional teacher and monthly themes that integrated important skills that are prerequisites for kindergarten attendance. Students in the computer group demonstrated improvements in all academic skills tested and their growth in memory and visual perception was significantly greater than that of the non-computer group. Chang and Osguthorpe (1990) investigated the achievement effects on kindergartners of reading instruction involving picture-word processing software and help from specially trained older students (fourth- and fifth-graders). 2000 Research Report on the Effectiveness of Technology in Schools 35 The picture-word processor...allows users to write messages on a computer by simply pressing squares of picture-words on an electronic tablet without having to spell words or use extensive hand-eye coordination. The computer-using students performed significantly better than students receiving regular classroom instruction on tests of word identification, picture-word identification and passage comprehension did. Research by Haugland (1992) suggests that the type of software young children are exposed to makes a difference in their cognitive development. She identified nine software programs as "developmental," according to a scale developed by Haugland and Shade (1988). Some of the characteristics of developmental software are as follows. • • • AGE APPROPRIATE...reflects realistic expectations of young children. CHILD CONTROL...Children decide the flow and direction of an activity, not the computer. EXPANDING COMPLEXITY...Software [is] an exciting new world...which is easy to enter. But once the child is inside, it continually expands to teach the child powerful ideas. INDEPENDENCE...The software facilitates...independence by enabling children to master at least the initial components of the program quickly, with minimal instruction or prompting... PROCESS ORIENTATION...The process of exploring the software, of finding out about the world, engages the child. Children often become totally immersed in the joy of discovery ...Their motivation is intrinsic. REAL-WORLD MODEL...The software is a simple reliable model for children of some aspect of the world. Vividly the software portrays...realistic objects...in meaningful situations or settings. TRIAL AND ERROR...software gives children unlimited opportunities for creative problem-solving ...Children test alternatives over and over again until they are satisfied. TRANSFORMATIONS...allow[s] children to change objects and situations, which would be much more difficult in daily life...children view hidden processes and learn the nature of cause and effect relationship (Haugland & Shade, 1990). • • • • • Haugland identified nine other software programs as "non-developmental." Four preschool classes were exposed to four different treatments throughout most of one school year: developmental software plus corresponding off-computer activities; developmental software only; nondevelopmental software only; and no exposure to computer software. Children's play with the computer software took place in the context of on-going preschool activities as one of many available activity options. The two classes that had experience with developmental software demonstrated "significant gains in intelligence, non-verbal skills, structural knowledge, long-term memory and complex manual dexterity." Furthermore, the class that was exposed to developmental software plus corresponding off-computer activities also showed significant improvement in "verbal skills, problem-solving, abstraction and conceptual skills." The class that had access to nondevelopmental software demonstrated significant gains in concentration and short-term memory but significant losses in creativity. 2000 Research Report on the Effectiveness of Technology in Schools 36 Snider (1996) found that open-ended software can have a significant positive impact on kindergarten students' creativity. Snider's criteria for rating software as open-ended included the following: 1. Open-ended in design to support divergent thinking, originality, fluency, and elaboration 2. Child is in control of the program 3. Non-judgmental in design (i.e., no right or wrong answers) 4. Multimedia including color graphics and sound. 5. Immediate feedback that is intrinsically oriented 6. Illustration of cause and effect relationships 7. Adaptable system requirements Eighty-six kindergarten students from six classrooms were assigned to one of three treatment conditions. One group worked cooperatively on the computer (six students working with two computers) with the open-ended software for 25 minutes, three days a week, for 15 weeks. Another group spent the same amount of time working in groups with more structured software that did not encourage open-ended responses from the child. The only time spent on the computer by the third (control) group was 30 minutes a week in the school's computer lab, where the only available software was drill and practice. Snider reported that students who used the open-ended software made significantly greater gains in both figural creativity (creativity on tasks using pictures or figures) and verbal creativity (creativity on word-based tasks) than both the other groups. Gains in verbal creativity were also significantly greater for students using structured software than for the control group. Special Needs Students Several researchers have recently examined the effectiveness of technology-based instruction for students with special needs, including students identified as learning disabled, low achieving or in need of special education. Learning Disabled Students Seven studies provide evidence of the potential of writing software, expert systems, videodisc, hypermedia, optical character recognition, speech synthesis and speech recognition as educational and compensatory tools for use with learning disabled (LD) students. MacArthur, Graham, Schwartz and Schafer (1995) studied the effects of a writing instruction program for LD elementary students that integrated word processing, instruction in writing skills and strategies and a process approach to writing. At the end of the year, students who had participated in the program showed significantly greater gains in the quality of their narrative and information writing than students who had not taken part in the study. Nebraska researchers (Zhang, Brooks, Frields, and Redelfs, 1995; Zhang, 1993) compared the effects on LD elementary students of writing instruction incorporating a writing tool designed especially for LD students, writing instruction incorporating a commercially-available word processing package and writing instruction without computers. Special features of the LD writing tool include teachers' and students' word lists, synthesized speech, an icon-driven interface with aural help and positive reinforcement through attractive printed certificates. The study found that 2000 Research Report on the Effectiveness of Technology in Schools 37 the LD-specific writing tool "enhance[d] significantly the holistic quality" of students' writing when compared with the other two programs. Garzella (1991) compared the effectiveness of an expert system for reading diagnosis and prescription (CAPER) to traditional methods of diagnosis and prescription. Elementary-level LD students of teachers who used the expert system significantly outgained students of teachers using traditional methods in word identification skills. Woodward and Gersten (1992) report that when teachers of LD high school students used a videodisc program to teach fractions, "almost two-thirds of the students reached or exceeded criterion performance." Xin (1993) found that video technology gave learning disabled elementary students a significant advantage in vocabulary development and achievement in reading comprehension compared to students not exposed to video. Video was used to provide meaningful visual contexts when learning new words. A fifth study compared the effectiveness of two versions of the computer-based Student Assistant for Learning from Text (SALT) in helping LD students "compensate for their reading difficulties (MacAuthur & Haynes, unpublished)." The "basic" version provided information elements found in a basic biology textbook (text passages, illustrations, outline and comprehension questions) plus a computerized "notebook." The enhanced version added speech synthesis, an on-line glossary, links between questions and text, highlighting of main ideas and supplementary explanations that summarized important ideas. The enhanced version was designed to take more fully advantage of the computer. Students scored significantly higher on a comprehension test when using the enhanced software version with hypermedia features. Researchers for California State University, Northridge (Higgins & Raskind, 1997; Higgins & Raskind, 1995; Raskind & Higgins, 1995; Murphy & Higgins, 1994) tracked the impact of a variety of compensatory technology tools that had been introduced for use with LD students at the university. They found that LD students who used speech recognition software to compose 500word essays scored significantly higher than those who used no assistance in composing their essays did. They found further that although LD students using no assistance or dictating their essays to a transcriber scored significantly lower than non-disabled students, students using the speech recognition software did not differ significantly in their performance from their nondisabled peers. As reported in Murphy and Higgins (1994), the researchers concluded that ...the technology "encouraged" the use of students' stronger oral vocabulary by circumventing spelling and punctuation difficulties and, because the technology recognized longer words better than short ones, counteracted a tendency to choose immature vocabulary. This group of researchers also determined that LD students located significantly more errors when they used a speech synthesis system to review and proofread their own essays than when their essays were read aloud to them by a human reader or when they received no assistance. Analysis of type of errors detected showed that students found significantly more errors 2000 Research Report on the Effectiveness of Technology in Schools 38 with speech synthesis in the areas of capitalization, spelling, usage and typographical errors. Additionally, on a test of comprehension of written passages, Higgins and Raskind (1997) found that when students used optical scanning, character recognition and speech synthesis, "the greater the disability, the more the technology elevated comprehension," although these technologies also interfered with the performance of more proficient readers. A cost analysis reported in Murphy and Higgins (1994) showed that use of these compensatory technologies saved money for the university. Based on a "bare bones" estimate comparing cost of purchase, training and maintenance of the technologies with the cost of recruiting, employing and training personnel to provide comparable services (e.g., transcription), they determined that use of the technologies saved the university $310 per student, per semester. Based on an enhanced estimate that included cost of outreach, needs assessment and provision for further training, the cost savings were $234 per student, per semester. These findings indicate that compensatory technology represents a cost-effective alternative that can significantly enhance LD student performance in a variety of areas. Low-Achieving Students Six studies suggest that low-achieving students can benefit from a variety of educational technology resources, including computer-based training systems, software tutorials, carefully designed basic skills drill and tutorial software and integrated learning systems (ILSs). Skinner (1990) investigated the effects of a sophisticated computer-based training course on classroom management for undergraduate physical education majors. The course components included the following: (1) "students progressed through [tutorial and practice] material at a selfdetermined pace"; (2) "a mastery criteria of 70% was set for each unit quiz"; (3) "demonstrations and lectures were presented to extend knowledge mastered via the text and CBI"; (4) "communication between instructors and students was conducted via a computer bulletin board"; and (5) "proctors and tutors were available as needed by students." Results on unit quizzes and the final exam showed the superiority of the computer-based training to text-based instruction without computer (but with proctors and tutors available as needed). The benefit of the computer-based approach was markedly greater for low-achieving students than for high achievers. Smith (1996) reported that computer-aided instruction could have a significant impact on the learning of developmental math students at a community college. Students who had been assigned to the Department of Learning Assistance based on math placement scores spent a hour learning about matrix algebra either by reading from a commercial text or by completing one of two computerized tutorial lessons written by the researcher. One tutorial used interactive animation of objects and symbols; the other was static. Two weeks after completing the tutorial, students who had used the animated version performed significantly better on a delayed posttest than students who had read the text or completed the static tutorial. Researchers from the University of California Santa Barbara (Semmel, Gerber and Semmel, unpublished) studied the effects of specially designed software on students in grades 4-6 identified as needing "extended practice of basic math facts on the computer to increase their speed." The software tests for the time it takes each student to find and press a key and then factors this "response latency" into timed drills. The software also includes diagnostic testing to determine an appropriate starting level for practice. After using the program to practice on single-digit 2000 Research Report on the Effectiveness of Technology in Schools 39 multiplication problems, students took a timed paper-and-pencil test. Results indicate that elementary age students needing extended math practice achieved automaticity for singledigit problems and for double-digit problems that required no added operations. A study by Carter suggests that supplementing classroom instruction with tutorial and practice software had a positive impact on mathematics and reading achievement for low-performing ninth graders (1994). A group of students received computer-based instruction for one 50-minute period per week in both their math and English classes for most of one school year; for the remaining time, these students took part in regular classroom instruction. They demonstrated significantly greater gains in both mathematics and reading skills than another group of low-performing students who received traditional instruction without access to computers. Both groups had the same amount of total instructional time. Swan, Guerrero, Mitrani and Schoener (1990) reported the results of New York City's Computer Pilot Program, which was targeted to educationally disadvantaged students across the elementarythrough-high school grade span. The results indicate the effectiveness of 13 comprehensive computer-based instruction programs, many of which are ILS-based. Overall ESs were 0.8 for reading and 0.9 for mathematics. In another study, Schmidt (1991) found that a distributed ILS resulted in significantly higher achievement gains in reading, math and language arts for lowachieving students compared to traditional instruction. Swan et al. (1990) cite four features of computer-based instruction that makes a difference for educationally disadvantaged students: (1) it is "perceived by students as less threatening than traditional instruction"; (2) it provides "extensive drill and practice with immediate feedback"; (3) it offers individualized diagnosis of student strengths and weaknesses; and (4) implementation of such instructional systems "provides students with greater academic support." Special Education Sartorio (1993) compared preschool special educational instruction that incorporated commercially available language development software to instruction without computers. The software programs could be used with "appropriate adaptive computer devices." Special education preschoolers in the class that used the software showed significantly greater improvement in language development as measured by a standardized test. Software Design Characteristics and Student Achievement Research supports what experienced, computer-using educators observe every day: specific design characteristics of software make a difference. Studies were identified underscoring the importance of a variety of design characteristics, such as: • • • • • • • • Type of software Instructional control (learner control versus program control) Amount of practice provided Type of feedback Use of objectives and advance organizers Embedding of cognitive strategies Use of voice and personalized dialogues by pedagogical agents Embedding of strategies for conceptual change 2000 Research Report on the Effectiveness of Technology in Schools 40 • • • • • • • • • • • • • • Instructional scaffolding of learner support Text display of spoken commands, story contexts, and English-language translations in foreign language learning Inclusion of still graphics Inclusion of both tables and graphs in math instruction Inclusion of animated graphics Use of narration with animated graphics Limitation of extraneous sounds and music Use of multiple representations of concepts Inclusion of dynamic visualization Use of video Navigational techniques Motivational contexts Window presentation styles Ergonomic combination of design features Type of Software Wood (1991) explored the effects on mathematics achievement of two different types of software: a tutorial program and a tool program. The subjects were high school students studying algebra. The students using the tutorial demonstrated higher achievement in computational skills; however, the students using the tool program evidenced higher achievement in their understanding of algebra concepts. The study suggests that the best choice of software type may depend on the instructional goal. Since success in mathematics requires both computational and conceptual skills, students are likely to benefit from both types of software. A study by Niederhauser (1998) further suggests that selection of mathematics software for student use should take into account the overall instructional orientation of the class. Niederhauser compared student pretest and posttest performance on the Stanford Achievement Test (SAT) of fifth graders from two traditional and two reform-oriented classes. (Both reform-oriented classes were with the same teacher.) Students in the traditional classes used a drill-and-practice based integrated learning system (ILS) for two hours (four half-hour sessions) a week, while students in the reform classes used the HyperCard program approximately 1 hour and 40 minutes per week (20 minutes a day) to model and represent mathematical rules they had developed through other classroom activities. Niederhauser found that students in the reform-oriented classes started with lower pretest scores but ended with higher posttest scores, with significantly greater gains in all three test sections (Concepts of Number, Mathematics Computation and Mathematics Application). While effects of the software cannot be separated out from other elements of classroom instruction, Niederhauser suggests that the ILS software may have provided less benefit for students in the traditional classes because it was so similar to what they were already doing in the classroom. By way of contrast, student use of HyperCard in the reform-oriented class "extended the classroom activities, lending coherence and depth to the overall mathematics learning experience." Rieber and Parmley (1995) analyzed the use of structured and unstructured physics simulations with university students, with and without an introductory tutorial. They found that students who completed either version of the simulation after the introductory tutorial performed significantly better on a posttest than students who experienced neither the simulation nor the tutorial. Students 2000 Research Report on the Effectiveness of Technology in Schools 41 who had completed the structured simulation, in which subskills were introduced in a controlled sequence and students were given increasing levels of control over a freefloating on-screen object, did equally well without the tutorial. However, students who did not receive the tutorial and who completed the unstructured simulation, "in which subjects assumed full control over the animated [object] from the very beginning," showed no significant improvement over students who had completed neither the tutorial nor the simulation. The researchers concluded that ...adult learners are able to learn inductively from structured computer-based simulations... However, subjects were not able to learn inductively from completely open-ended simulations. Some software is developed and assigned to students with the intent of improving general problemsolving abilities. McClurg (1992) found that different genres of problem-solving software have different effects on student abilities. One group of third and fourth graders used software programs that focused on spatial patterning tasks (e.g., detect the pattern, continue a pattern). The other group worked on software challenges involving spatial rotation. The spatial rotation group experienced a significantly greater gain in a test of figural classification than the spatial patterning group. The author attributed the different gains to the specific conceptual skills required to solve the problems presented by each software program. This research suggests the need for educators to closely inspect the challenges provided in problem-solving programs. Instructional Control Several recent studies suggest the importance of instructional control (learner control versus program control) as a software design issue. Dalton (1990) explored the effects of an interactive video on comets using two different instructional pacing strategies. One group of fifth and sixth graders worked through a fixedsequence presentation in which text screens were presented "for a fixed length of time, based on the average reading rate of a group of similar learners." The other group worked through the same fixed-sequence presentation, except that the students controlled the amount of time spent on each text screen. For both groups, follow-up questions were presented untimed. Students who used the learner-paced version significantly outscored those using the program-paced version on a test of basic facts and definitions. A study by Arizona State University researchers (Hannafin and Sullivan, 1994) suggests that high school geometry students can benefit from software that provides learner control over content presentation. The software program used in this study covers basic geometry concepts. Students who used versions of the software in which they could add or bypass example, practice and review screens demonstrated significantly greater achievement than students who used versions offering no such control over presentation. A study by Miller (1996) focused on the interaction between the level of learner control over the presentation sequence and another instructional design variable: the level of the learner's need for the information. Four versions of a multimedia presentation on a specialized area of accounting practice (ISO 9000) were developed. The presentation versions differed in the instructional control strategy used. • The "learner controlled sequencing strategy gave the users complete navigational control over the sequence in which they viewed the presentation." 2000 Research Report on the Effectiveness of Technology in Schools 42 • The "program controlled sequencing strategy only allowed users to control their pace through the presentation." In all the software versions, university accounting students were directed to view the presentation as if they were staff accountants for a CPA firm. However, the versions differed in the level of information need induced by the on-screen instructions: • Low need instructions provided a scenario in which the motivation for viewing the presentation was that the student had been "wondering what [ISO 9000] is about." High need instructions provided a scenario in which the motivation for viewing the presentation was that the head of the CPA firm wanted the student to brief him on the presentation and that the student would be graded according to his/her answers to questions specified in the instructions. • In Miller's experiment, students were divided into four groups, with one group using each of the four software versions: learner controlled-low need instructions, learner controlled-high need instructions, program controlled-low need instructions, and program controlled-high need instructions. Among students completing versions with high need instructions, there was no significant difference in achievement between the group using the learner controlled version and the group using the program controlled version. However, among students working through versions with low need instructions, the group using the program controlled version significantly out-performed the group using the learner controlled version. The results suggest that "the sequencing strategy of the multimedia presentation only...influence[s] learning when the subject's need for the information is low." It should be noted that the students participating in this study had little or no prior knowledge of the content. The effects of learner and program controlled software might be different for more knowledgeable students. Shyu and Brown (1993) compared the effects on college students of watching an interactive video under two different learner control conditions. The video demonstrated and explained, step-by-step, how to make an origami crane. One group proceeded in the fixed sequence of steps, with the option to replay the current step as many times as desired but with no opportunity to return to previous steps. The other group had total menu-based control over the sequence of steps, including replay of any step, stopping in the middle of a step and returning to any previous step; this version of the program also included advisement in the form of a suggested sequence of steps to view. Students assigned to the total learner control with advisement condition performed significantly better at folding the paper into a crane. Lee (1990) studied the effects of computer-based instruction on Logo commands using two different strategies for controlling instructional review. Each lesson included a tutorial presentation and follow-up questions. After responding incorrectly to a follow-up question, one group of third graders had the option of reviewing instructional components or continuing on to the next question; students in this group were also regularly encouraged to make a self-assessment of their understanding of the concepts presented. The other group automatically received a review presentation after each wrong answer. Students in the learner-control review group significantly out-performed the program-controlled review group in measures of metacognitive monitoring. The learner-control review group needed fewer prompts to identify errors in presented information and fewer prompts to answer questions correctly. 2000 Research Report on the Effectiveness of Technology in Schools 43 Crooks, Klein, Dwyer and Jones (1995) found that the way in which learner control over content pacing and sequence is provided can have an impact on learning. College education majors enrolled in a course on learning and motivation experienced one of two versions of a software lesson on writing assessment items. ... LeanPlus...learners receive a basic instructional program and are given the option of requesting additional instruction. ...FullMinus...learners are given a full instructional program and given the option of bypassing instruction. Students who received the FullMinus version significantly out-achieved those who used the LeanPlus version. Five recent studies underscore the complexity of the phenomenon of learner control, especially how it interacts with teaching style and learner characteristics. Rowley, Miller, Armstrong Lab/Intelligent Training Branch and Carlson (1997) investigated the interaction between teaching style and learner control of software using two versions of the RWISE automated composition training tool. Following an initial semester during which students were required to use the software linearly and the level of instructional support was preset within the software (guided mode), ninth graders from 114 classes were divided into two groups. One group continued to work in guided mode, while the other group was able to access instructional screens and manipulate tools without restrictions (open mode). Student performance gains on a written essay test administered before and after the second semester were correlated both to use of guided or open mode during the second semester and to the instructional style of teachers, as measured by the Canfield Instructional Styles Inventory. Overall, students who had worked in the open mode during the second semester improved their performance significantly more than students who had worked in the guided mode (6 percent versus 3 percent) did. However, specific results varied according to the instructional style of the teacher. For some teaching styles, students working in the open mode performed better; for other styles, students working in the guided mode performed better. According to the researchers, This finding suggests that it may be reasonable to increase the adaptability of supportive software environments such as R-WISE, in order to accommodate both a variable level of learner-control and variations in the preferred instructional styles of the teachers. In research by Kinzie, Sullivan and Berdel (1992), two groups of ninth-graders received CAI covering science topics under different control-of-review conditions. One group automatically received appropriate review presentations following practice items and feedback (program control). The other group had the option of choosing less review (learner control). Unlike the experiment reported by Lee, students in the learner-control group were not explicitly directed to make selfassessments of their understanding of the material. Male students using the program-control version significantly out-achieved male students using the learner-control version. No significant difference was found for female students. The results indicate the importance of encouraging student self-assessment when learner control over instructional review is provided. 2000 Research Report on the Effectiveness of Technology in Schools 44 Shin, Schallert and Savenye (1994) studied the impacts of different versions of a hypermedia lesson presented to second graders with high and low prior content knowledge. The versions of the lesson on food groups varied according to the degree of learner control over access to other screens of information. In the free-access condition...a network structure [allowed students to] choose as many options as desired. The information was linked to every possible topic in the lesson and each [of four] group[s] could be accessed in any order. ...In the limited-access condition, [which] represents hierarchical structure, the student could only choose a topic that was directly related to the information just presented. Each food group could be tried in any order but the balance meal [summary] section was available only when all food groups had been tried. Students with low prior content knowledge that used the limited-access version of the lesson significantly outscored similar students who experienced the free-access version on a test of food groups content. However, there was no significant difference in test performance for students with high prior content knowledge. Research by Simsek (1993) suggests that while providing a high degree of instructional control is an effective software design strategy, care must be taken regarding grouping of low-ability students. One version of a tutorial lesson on solar energy "allowed a group of fifth and sixth graders to exercise control over the amount, review and sequence of instruction." A second version of the software offered students no such control. Overall, students who received the learner-control version demonstrated significantly higher achievement on a test of solar energy content recall and comprehension and on a test of retention taken two weeks after instruction. However, homogeneous small groups of low-ability students exposed to the learner-control version performed significantly worse than low-ability students exposed to the same software version but placed in groups of mixed ability. Presumably, the low-ability students in the mixed-ability groups benefited from the guidance provided by their higher-ability partners. Temiyakarn and Hooper (1993) found that learner control, student grouping and prior student achievement interact in their effects on subsequent achievement in biology. Standardized achievement test scores were used to determine the prior achievement of sixth grade students, who used one of two versions of a software lesson on "Relationships among Organisms." Highachieving students who used the learner-controlled version of the software in heterogeneous small groups outscored all other high-achieving students on both immediate and delayed posttests. However, low-achieving students who used the program-controlled version of the software in heterogeneous small groups out-performed all other low-achieving students on both tests. The results of the last three studies suggest caution before exposing low-achieving students or students with little prior content knowledge to software with a high degree of learner control. Either the software design or the teacher should provide sufficient structure for such students to succeed. Amount of Practice Schnackenberg, Sullivan, Leader and Jones (1998) compared the instructional effectiveness of two versions of tutorial-and- practice software: one with substantially more practice items than the 2000 Research Report on the Effectiveness of Technology in Schools 45 other. Both versions were adapted from a college textbook and focused on concept and rule learning. For practice...on concept-learning objectives, learners were given the name of the concept...and were asked to select the best exemplar of it from among four choices. For rule learning objectives, learners were given a situation in which a particular rule taught in the program would be applicable and were asked to select the most appropriate of the four given response choices for that situation. The full version of the program included a total of eight practice items per objective, whereas the lean version of the program contained only two practice items per objective. Both versions covered the same nine objectives and were identical in all respects except for the amount of practice provided. The posttest performance of university students using the full version was significantly better than that of students using the lean version. Additional analysis indicated that students did not take significantly longer to complete the full version (2 hours and 23 minutes versus 2 hours and 9 minutes for the lean version). Feedback Recent research suggests that the kind of feedback provided in technology-based instruction can have different effects on learning. One study (Clariana, 1990) compared the effectiveness of answer-until-correct (AUC) feedback and knowledge-of-correct-response (KCR) feedback ...when a student misses a question, the typical AUC feedback is "NO, TRY AGAIN." ...KCR feedback provides the learner with the correct answer to a question after one attempt. Low-ability students in grade 11 receiving KCR feedback during social studies reading comprehension practice significantly out-achieved students receiving AUC feedback. The KCR feedback provided information that students could use to clarify misunderstandings of what they had read, whereas the AUC feedback provided no such information. Researchers from the University of Memphis (Tennessee), who found similar results with college students taking education courses, confirmed the value of KCR feedback (Morrison, Ross, Gapalakrishnan and Casey, 1995). In this study, five different versions of a software lesson on educational technology were compared: • • • • • KCR feedback AUC feedback Initial AUC feedback followed by delayed KCR feedback No feedback Content presentation without interspersed questions The KCR version of the software was found to be superior to the other versions. Miech, Nave and Mosteller (1997) described a study by Nagata (1993) in which the type of feedback in computer-assisted language learning (CALL) software was shown to have a significant impact on student achievement. Nagata found that college students from an intermediate Japanese 2000 Research Report on the Effectiveness of Technology in Schools 46 course using a CALL program with "intelligent" feedback in the form of detailed error analysis performed significantly better than students who received more conventional CALL feedback that only identified what was wrong instead of explaining why it was wrong. Two studies explored the impact of advisement as a form of feedback in hypermedia exploration programs. Klayder (1993) studied the impact of two levels of "time-and-scope" advisement when using a hypertext program. The hypertext program covered more content topics than was used in the classroom instruction focus. Static advisement consisted of a single printed page explaining expectations about content on which to focus. Dynamic advisement added: (1) a constant display of time expectations for the current section [of the program], (2) a continually updated display of the current amount of time spent in the current section and (3) a display of times remaining in advised sections which appear[ed] when the recommended time for a section had been exceeded. Students receiving hypertext with dynamic advisement performed significantly higher on factual questions on an achievement posttest than did students using hypertext with static advisement. Lee and Lehman (1993) investigated the effects of instructional cueing, another form of advisement feedback embedded in a hypermedia exploration program. College students were categorized as having an active, neutral or passive learning style, based on their performance on a learning style test. They were randomly assigned to one of two versions of a hypermedia "stack." One version monitored student exploration of screens that elaborated on basic information. If a student tried to advance without accessing the elaborative information, a message would appear advising to probe further; however, the student could choose to ignore the advice. The other version of the program did not provide advice about available elaborative information. Active learners achieved about the same regardless of the hypermedia version used. However, passive and neutral learners demonstrated significantly higher levels of achievement using the hypermedia version with instructional cueing as feedback. Santiago and Okey (1990) reported on the effects of two different types of advisement feedback: adaptive advisement and evaluative advisement. Adaptive advisement gives information related to the amount/sequence of instruction the learners need to do based on their initial or current performance level. An example of adaptive advisement would be guidance such as "Review the Cell Division Tutorial. Then choose Cell Division Practice on the Main Menu and complete at least three problems." Evaluative advisement [informs students] on current learning level in relation to required mastery level. An example of evaluative advisement would be information such as "You answered 6 of 10 questions correctly. You must answer at least 8 of 10 questions correctly to achieve mastery." The subjects of the study were university students in a pre-service curriculum for prospective teachers. Students receiving adaptive advisement achieved significantly higher than students receiving evaluative advisement. The results also indicate "the effectiveness of adaptive advisement did not depend on the learners' locus of control orientation." Learners with an internal locus of control believe that their circumstances are based on their own behavior. Learners with an 2000 Research Report on the Effectiveness of Technology in Schools 47 external locus of control believe that what happens to them is due to factors outside their control. (External locus of control is frequently cited as a characteristic of low-achieving students.) Clariana (1993) studied the achievement effects of advisement in the form of progress reports generated by an integrated learning system (ILS). The progress reports showed each activity completed, the percentage correct for each activity, the amount of time spent on each activity and the date the activity was completed. Students receiving ILS progress reports demonstrated significantly higher mathematics achievement than students who used the ILS without receiving such reports. Buzhardt and Semb (1998) compared college student performance on unit quizzes featuring different types of feedback in an Introduction to Child Development and Behavior course. Three types of feedback were provided: end-of-test, item-by-item with the option to skip items and return to them and item-by-item without the ability to skip test items. The researchers found that students performed significantly worse when they received feedback after each item but were not allowed to skip items compared to the other two feedback formats. No significant differences in performance were evident on an end-of-semester final exam for content covered under the three conditions. However, since unit quizzes are typically considered in college grading schemes, these findings suggest that design decisions about quiz feedback can influence student grades, even though differences in student learning may level out by the end of the course. Objectives and Advance Organizers Cavalier and Klein (1998) reported on the performance of fifth and sixth grade students completing a HyperCard earth science tutorial on modern-day prospecting featuring different types of orienting activities. Students who previewed a list of instructional objectives performed significantly better than those who read an advance organizer paragraph or who completed no orienting activity. Analysis of posttest scores showed that viewing the objectives improved student performance on intentional learning (items directly aligned to the outcomes of the lesson, which were similar to practice items found in the tutorial). However, viewing the objectives had no significant impact on incidental learning (items testing information provided in the program but not directly aligned with lesson outcomes). These results held true both for individual students and for student dyads working cooperatively. Cavalier and Klein speculated "objectives enhanced performance on intentional scores because the objectives provided a clear link between expectancies for learning and incentive for learning." On the other hand, the advance organizer may have been less effective because "[the tutorial] was designed following a systems approach and included most of the elements of effective instruction proposed by instructional design theorists." This explanation is consistent with the conjecture that advance organizers may add the most value when content is not otherwise well organized or presented effectively. Kang (1996) found that fifth, sixth and seventh grade students who read an advance organizer passage outlining the main points of a Wilderness Survival simulation performed significantly better on the posttest than students who read an introductory paragraph with no specific or useful information for completing the simulation. Scores on the posttest, which involved having students reflect on their simulation experiences and choices, were higher across all three grade levels for students who completed the advance organizer version. It is likely that the advance organizer provided students with a structured way of proceeding through the relatively open-ended simulation. 2000 Research Report on the Effectiveness of Technology in Schools 48 Cognitive Strategies Several recent studies suggest the learning effectiveness of software with embedded cognitive strategies. Armel and Shrock (1996) described the benefits of embedding note taking, an established cognitive strategy of successful learners, as part of computer-based instruction. University education majors were directed to complete a HyperCard lesson on the human heart. One version of the lesson regularly prompted students to take notes on the computer and required them to input at least five words before proceeding with the lesson. A second version provided a space for note taking and informed students that they could take notes, but did not prompt or require note taking during the lesson. A third (control) version did not include on-computer note-taking capability. Students who took notes were always able to see all their notes and were given a five-minute review period before advancing to the posttest; however, they did not have access to their notes during the test. Students who were required to take notes scored significantly higher on the posttest than both other groups, while students with optional note-taking scored significantly higher than the control group. Similar results were observed for time to complete the lesson, with the required note-taking group taking significantly longer than both other groups and the optional note-taking group taking significantly longer than the control group. Commenting on their findings, the researchers underscored the unique value of computer-based instruction as a tool for directed note taking: Forcing an individual to take notes is a thoroughly new instructional intervention, which can only be implemented practically with the aid of computer-based instruction. The computer environment allows users to be monitored, measuring whether or not they take notes and if so, how much they have entered in the note-taking area. Katayama, Crooks and Nelson (1999) compared the instructional effectiveness of three different approaches to prompting student note taking. ...skeletal [prompting] only provided the students with the headings and categories for which they were expected to complete all the relevant notes... For example, column headings consisted of "definition" and "purpose" and row labels consisted of topics related to the content of the text passage. ...partial [prompting] provided students...with basic headings for conceptually organizing their notes...[and] with approximately half of the notes and required the students to key in the missing notes... ...[no prompting] consisted of a blank screen for each text passage in which students could take whatever notes they wished... In one study by Katayama et al., university students were divided into three groups, with each group completing a different version of the software program. Each group read, on computer, a 3,500-word text "covering the basics of educational research" and had several opportunities to take notes. The software versions differed only in the approach to prompting note taking. After reading the text and taking their study notes, each group took three tests. No significant differences were found among the groups on a multiple-choice test of facts based on information stated in the text or on a test in which "students had to recall the hierarchical structure of the text." However, on a transfer test in which "students had to apply their understanding of educational research to 2000 Research Report on the Effectiveness of Technology in Schools 49 novel situations," the group that completed the software version with the partial prompting approach to note taking significantly out-performed the group that experienced the no prompting version. There was no significant difference between the transfer test scores of the partial prompting group and the skeletal prompting group. A follow-up experiment confirmed the advantage of the partial prompting approach over the no prompting approach on a transfer test but the lack of an advantage on a structure recall test. Barba and Merchant (1990) explored the effectiveness of science software with embedded cognitive strategies such as repetition and rehearsal of content, paraphrasing, outlining and cognitive mapping, drawing analogies and inferences, specific techniques for reading in the content areas and using pictorial information. These researchers compared the achievement of 10th grade biology students using two different versions of a HyperCard stack on insect classification: one with embedded cognitive strategies and one without. The students using the version with embedded cognitive strategies significantly out-performed the other students on a subsequent insect classification task. The benefits of embedded cognitive strategies were more pronounced for low verbal learners than for high verbal learners. Garrison (1996) found that college students who were prompted to use metacognitive strategies while working in tutorials on biological statistics scored significantly better on a posttest than students who completed the same tutorials, but were not prompted to use the strategies. Students in the first group began with metacognitive training in which students were taught to visualize the entire problem, check their progress and estimate the answer. As they were working through the problem, they received simulated "e-mail" messages prompting them to use the metacognitive strategies and to write their responses in a return e-mail message. They were then asked early in the second session to write about the current problem in an electronic notebook. Students in the second group completed the same tutorials, but did not receive the metacognitive training, did not write or receive e-mail and were not asked to write in the notebook. The study found that although the first group spent considerably more time completing the tutorials, time was not by itself a predictor of success, suggesting that students' improved learning came from more consistent use of the metacognitive strategies and not simply from spending more time with the material. Grossen and Carnine (1990) compared the effects on learning disabled high school students of two versions of computer-assisted instruction (CAI) on the logic skills. One version embedded the cognitive strategy of having students generate a diagram as a response to a logic problem before choosing the correct diagram from multiple choices. In the other version, students were presented with multiple choice options immediately, without having to generate a diagram. On a simple transfer task as a measure of achievement, students receiving the program with the embedded strategy of first generating a response significantly out-performed students who used the plain CAI version. Furthermore, the group that used the version with the embedded strategy significantly outscored the comparison group on the more difficult logic tasks. Students using the embedded strategy version required fewer questions to reach mastery criteria throughout the program without requiring more total instructional time. Aleven, Koedinger and Cross (1999) tested the hypothesis that a cognitive tutor for geometry could be "more effective if it requires students to provide explanations for their solution steps." The PACT Geometry tutor software was "designed to be an integrated part of a new [NCTM Standardsoriented] high-school geometry course," featuring problems that "often involve a real-world problem situation that calls for geometric reasoning." The researchers developed two versions of the software: 2000 Research Report on the Effectiveness of Technology in Schools 50 In the "answer-only" version of the tutor, students were required only to calculate unknown quantities in each geometry problem. In the "reason" version, students also needed to provide correct explanations for their solution steps, by citing geometry theorems and definitions. They did so by selecting from the tutor's Glossary of geometry knowledge, presented on the screen... ...Students who are stuck on a problem can search the Glossary for a rule that applies. In addition, the tutor provides hints on request. A group of high school geometry students who used the reason version of the PACT Geometry tutor lesson on angles demonstrated significantly higher achievement in this conceptual area than a group who used the answer-only version. While the group using the reason version spent approximately 14 percent more time using the tutor, the difference was not found to be statistically significant. The researchers explained that students completing the reason version "have more work to do per problem, since they have to provide reasons for their solutions steps." Johnsey, Morrison and Ross (1992) investigated the achievement effects of embedded and detached training in generating elaborations (i.e., illustrative examples) of concepts introduced during computer-based instruction. The students were adult employees taking a computer-based professional development course. Students were divided into four groups, each of which received a different version of the software: major content information only (the control group); major content with experimenter-generated elaborations; preliminary tasks instructing students to generate their own elaborations followed by major content information only (detached training); and major content information followed by an instructional unit on elaboration strategies, their value and features and specific techniques for applying the strategies (embedded training). Students receiving embedded or detached elaborations training significantly out-performed the control group on recall of information. However, only the group receiving software with embedded elaborations training showed significantly higher achievement on application of the content to new situations. Chyung (1996) compared the development of self-regulated learning (SRL) skills among undergraduate college students using two types of computer-assisted instruction programs, one "intelligently" controlled (INC CAI) by the computer and the other totally learner controlled (TOLC CAI). The INC CAI programs included a cognitive apprenticeship tutoring strategy, in which students were provided direct coaching and guidance during the first lesson based on an evaluation of a model of the student's performance. In the second INC CAI lesson, coaching was less direct and students had more responsibility for their learning--an instructional practice known as "fading." The TOLC CAI lessons did not employ any cognitive apprenticeship or coaching. Although academic achievement was not significantly different between INC CAI and TOLC CAI, students in INC CAI showed significantly greater improvement of SRL skills, such as deliberately choosing presentations of key concepts and examples and electing to take selftests. These students completed more lessons and spent more time learning than the TOLC CAI group. As students advanced from the first to the second INC CAI lesson, they were able to become involved in highly active learning processes without the help of the coach. 2000 Research Report on the Effectiveness of Technology in Schools 51 Pedagogical Agents Moreno and Mayer (unpublished) conducted a series of studies to better understand how pedagogical agents should "communicate with students to promote constructivist learning." A pedagogical agent is any modality or combination of modalities used to convey instructional information that guides students' thinking and behavior as they work through an interactive learning experience. In Moreno and Mayer's research, the learning environment was a multimedia program, entitled "Design-a-Plant," that includes a "lifelike pedagogical agent who provides advice to learners as they graphically assemble plants from a library of plant structures such as roots, stems and leaves." In four studies, Morena and Mayer compared versions of Design-a-Plant with pedagogical agents that • • • Communicated via human voice or via on-screen text Had a human image (animated or video of a human face) or no image Communicated via personalized, self-referenced dialogues or via "depersonalized generic monologues" In the studies, university students worked through the software program, then completed a content recall test, a problem-solving test, and a "program-ratings sheet." Students who used a version of the program in which they heard the voice of the pedagogical agent significantly outperformed students who experienced the on-screen text agent on the recall and problemsolving tests, and rated the lesson significantly higher. Furthermore, students who used a version of the program in which the agent communicated via personalized dialogues significantly out-scored students who interacted with an agent that communicated via depersonalized monologues on the recall and problem-solving tests, but there were no significant differences in their program ratings. Also no significant differences were found when comparing students who saw a human image of the agent to students who completed a version of the program with no human image. The researchers concluded "in a constructivist science lesson... the [pedagogical] agents' voice and conversational style play a fundamental role in the promotion of meaningful learning." Conceptual Change Strategies Two recent studies point out the benefits and identify effective design features of programs designed to challenge students' misconceptions of content topics and help them develop a correct understanding of the concepts involved. Biemans and Simons (1996) studied two versions of a program designed to promote conceptual change among middle school geography students. In both versions, students read text sections on the computer and then completed a sequence of directed activities designed to move them from their faulty preconceptions to a more accurate understanding of physical geography concepts. In the first version of the program, students read the text and completed the sequence of activities for each concept twice. In the other version, students were required to complete the second sequence of activities only if their understanding of the concept was still flawed after completing the first sequence of activities. Unlike the first version, the second version also represented information graphically, highlighted key text points in different colors, prompted students to record information on a worksheet and made reading about conceptual change strategy optional. Students using the second version of the program performed significantly better than both students using the first version and students who read the text but did not complete either version of the program. The authors theorized that the reduction of required reading about conceptual change strategy in the 2000 Research Report on the Effectiveness of Technology in Schools 52 second version allowed students to pay more attention to the text itself while still gaining the benefits of completing the conceptual change process. As reported above in the section on Curriculum Areas and Student Achievement, researchers at the University of Minnesota (Jensen et al., 1996) studied computer-assisted instruction in a universitylevel biology survey class that targeted students' misconceptions on diffusion and osmosis. Program modules included specific questions and graphic illustrations of possible answers, including wrong answers. Students then made choices, frequently producing incorrect answers and "providing the instructor with a series of 'teachable moments' to explain the topics further." In some course sections, students were also required to write justifications for their answers. Students in classes where the program was used demonstrated significantly greater understanding of osmosis concepts than students in classes where the program was not used. Surprisingly, students who were not also required to write justifications of their answers experienced significantly greater gains than those who were required to write did. The study authors speculated that this was because ...when justifying their choices, students in the writing sections simply write down and thus reinforced, their misconceptions...students who were not using the writing spent more time in discussion of the "new" information they gleaned from the exercises and this aided in effecting conceptual change. Instructional Scaffolding Researchers at Purdue University (Kao and Lehman, 1997; Kao, Lehman and Cennamo, 1996) demonstrated the usefulness of scaffolding, a multi-tiered support structure for helping students build new skills, in a program introducing applied statistics to students in a university educational psychology course. The program consisted of a set of similar statistics problems. For each problem, support was provided at four levels. • Level one, the fullest level of support, demonstrated "the steps to solve the problem in detail with visual, verbal and symbolic instruction." This level provided the answer to students. Level two provided "the visual and verbal hints to the current problem steps," but did not provide the answer. Level three provided the verbal hints, but did not include the graphic or visual hints. Level four provided no hints at all. • • • Students completed one of three versions of the program. In all three versions, students went through the first problem at level one, the complete demonstration. In the "least support" version, students then completed all later problems at level four. In the "full support" version, participants also started the second problem at level four, but returned to level one if they made mistakes during the problem. This process continued for later problems. In the "scaffolded" version, students began the second problem at level two and progressed through levels three and four on subsequent problems, returning to more fully supported levels if they made mistakes. In all three versions, students continued until they had correctly completed two problems at level one. Students in the "scaffolded" version performed significantly better on a posttest than students in the "least support" and "full support" versions, suggesting that scaffolding can be an effective software design feature for improving student achievement. 2000 Research Report on the Effectiveness of Technology in Schools 53 Gradually increasing the complexity of the model used in an instructional simulation, or model progression, can be viewed as another type of scaffolding of learner support. Swaak, Joolingen and DeJong (1998) explored the learning effects of model progression and practice exercises when incorporated into SETCOM, a learning environment that includes a simulation model of oscillatory motion, a conceptual area within physics. [In SETCOM] subjects can control a number of input variables and watch the behaviour of the oscillating system (a mass suspended from a spring) as it is expressed in a graph… in numerical output, and in an animation of the system... Three levels of model progression are present: free oscillation, damped oscillation, and forced motion. The number and kind of input variables that can be controlled and the output variables that can be observed increase with each level. At the simplest level the learner can control only two input variables: the mass and the force constant of the spring. At the most complex level, the learner can control five input variables: the mass, the force constant, the damping constant, the amplitude of the external force, and the frequency of the external force. In all cases the learner can also have the simulation return to its initial state (i.e., the state of the simulation with which the learner started to work). At each level of model progression, a number of assignments guide the learner in the exploration of the model behind the progression level. There is a text-and-graphics explanation of each variable included in each level of the model progression. In addition, explanatory feedback is provided in response to student answers to questions within assignments. Swaak et al. compared three versions of the simulation environment. The full version incorporated model progression and assignments, including the corresponding feedback explanations. The second version included model progression only; there were no assignments or feedback explanations. In the third version, only the most complex level of simulation model was available (i.e., no model progression), and no assignments or feedback explanations were provided. College physics students who used either the full version of the simulation environment or the model progression-only version demonstrated significantly greater achievement on a test of intuitive knowledge than students who used the no model progression-no assignment version. (This WHAT-IF test required students to view a graphic-and-text representation of the system's current condition, read "the change of a variable within the system," and then choose the correct resulting predicted state.) No significant difference on intuitive knowledge achievement was found between the full version group and the model progression-only version, which may be due to the inclusion of one student who actually scored lower on the posttest than on the pretest. When this student was removed from the analysis, the remaining full version students were found to have significantly out-gained the model progression-only version students. There were no significant differences among the three groups in achievement gains on a test of the student's ability to explain the relationships between pairs of variables. The students also completed multiple choice pre- and posttests of "definitional knowledge about the facts and concepts of the domain." A low, statistically insignificant relationship was found between definitional test gain scores and intuitive knowledge test gain scores, suggesting, "a gain in intuitive knowledge does not automatically mean a gain in definitional knowledge." 2000 Research Report on the Effectiveness of Technology in Schools 54 Based on this finding, the researchers advised, "in the context of simulation based learning, it makes sense to introduce new ways of measuring knowledge in addition to [the] traditional "definitional" type of knowledge tests." Learner Support in Foreign Language Learning Jakobsdóttir and Hooper (1995) measured the effects of providing text together with spoken words and including story context in a computer-based foreign language lesson. Fifth grade students completed a beginning lesson in Norwegian, clicking on icons to select nouns and verbs in response to short spoken commands in Norwegian. As each command was spoken, text of the command was also displayed in some versions of the lesson. Additionally, in some versions, the commands were presented in the context of a short fairy tale. The researchers found that students who both saw the text and heard it spoken made fewer errors during the lesson and did better on both the immediate and delayed posttests. Benefits from the story context were less clear, possibly because the experimental treatment in the story was relatively brief (less than 30 minutes) and benefits of context often emerge gradually. However, girls seemed to show a greater benefit than boys do from the use of context. Generally speaking, "girls demonstrated higher achievement than did boys and rated the lesson higher..." Grace (1998) explored the effectiveness of another design feature in computer-assisted language learning (CALL): providing "first-language sentence-level translations as a means for verifying meaning." Two versions of the CD-ROM, Le Fils d'AstŽrix (The Son of Asterix), based on a comic book series, were used in the experiment. The two versions were identical in most respects: Each lesson consisted of...comic book screens with dialogues in which...French vocabulary items to be tested in the project were embedded... Each graphic depiction served as an advance organizer for its corresponding options. The subjects in both groups were able to select a French written text of the dialogue. To further promote correct guessing of words, they were also able to access...French definitional sentences of the target words in the French text. In addition, the subjects could call up the corresponding audio track with supporting background audio and voices of native-speaking actors. One version of the program also included the "option of accessing the English textual translation of the French dialogue in each screen." The other version did not offer the English translation option. University students who used the English translation version of the software scored significantly higher than students who used the version with no English translation on both a vocabulary test given immediately after completing the program and a follow-up vocabulary test two weeks later. The advantage of the English translation version was found to be consistent across several student personality types. Still Graphics in Vocabulary Development Chun and Plass (1996) found that university students in a German language class learned vocabulary better from still picture illustrations than from video segment illustrations. Students reading a story in German on the computer could click on difficult vocabulary words, then choose to view one or more of the annotations that were available in the program. Annotations included a text definition of the word (available for all the vocabulary words) and, in some cases, either a still picture or a short video. All the annotations also included an audio recording of the word spoken in 2000 Research Report on the Effectiveness of Technology in Schools 55 German. Students then took an unannounced vocabulary test that included 12 words for which only text definitions had been available, 12 words for which text + picture annotations had been available and 12 words for which text + video annotations had been available. Students performed significantly better on the words for which text + picture annotations had been available. When asked which of the annotations had helped them identify the vocabulary word, "the learners also showed a tendency to report the picture[s]...more often as the retrieval cues." The authors speculated that this was because still pictures "have a constant, fixed quality and can be looked at for as long as the learner wishes, which allows for the development of a mental model of the information." They cautioned, however, that this pattern may not be general for all types of learners, but may vary with different cognitive styles and abilities. (For example, see the section on Animated Graphics, below.) The relative effectiveness of still and animated graphics is likely to depend on the learning task, as well. Graphs and Tables in Mathematics Instruction Johari (1998) compared two versions of a multimedia tutorial-and-practice program designed to help students learn to "conceptualize variables from [math] word problems" and to create corresponding linear functions. In one version, data on the variables in the word problems were presented in tables and graphs. In the other version, the data were presented only in tables. Both programs, InductiveThinker Table and InductiveThinker Table & Graph, had two lessons. Lesson one contained information about the input, output, and independent and dependent variables. Lesson two included information about the rate of change (or the slope of the function) and linear function creation. Some screens were added to InductiveThinker Table to construct InductiveThinker Table & Graph. The pace was user controlled and subjects had the ability to navigate between pages, sections, or lessons and could exit the program at any time. University students using both versions of the program scored significantly higher on the posttest than on the pretest in both function construction and variable conceptualization. In the area of function construction, students using the table-and-graph version of the program demonstrated significantly greater achievement than students using the table-only version. However, no significant difference in their performance was found on test items focusing on variable conceptualization. Animated Graphics Several studies found evidence for the benefits of animated graphics. Calvert, Watson, Brinkley and Penny (1990) experimented with different versions of a graphic "microworld" designed for young children's language development. ...a computer screen depicted a park scene that had a green grassy area, a blue lake, a blue sky, a black train track and a brown road. Twenty-four...objects...could appear by...typing...the word for the...object. In one version, a still-frame object would appear accompanied by a spoken verbal label (using synthesized speech). In another version, an animated object would appear without a spoken label. Poor-reading second graders who used the version with animated objects recalled significantly 2000 Research Report on the Effectiveness of Technology in Schools 56 more words than similar students who used the version with spoken labels did. The animated version "increased the poor readers' verbal recall to the level of their better reading peers." Szabo and Poohkay (1996) similarly found that university students in a math education class learned better from animated illustrations than from static graphics or text-only descriptions. On the posttest, which included a hands-on triangle construction problem and multiple choice questions, students who read text and viewed animated graphics showing how to construct a triangle using a compass performed better than students who read a text explanation only or students who read a text explanation accompanied by static graphic illustrations. Smith (1996) reported that developmental math students at a community college learned significantly more from a one-hour animated software tutorial on matrix algebra than from a static one. Animation was used mostly to highlight symbols, objects, and "morphing of addition elements and multiplication factors into sums and products." Students could stop or repeat animation or alter the variables from a menu. A third (control) group read for an hour from a commercial algebra text. Smith found that both the static-CAI and the animated-CAI students performed significantly better than the text group on the immediate posttest. However, on the two-week delayed posttest, students who had used the animated tutorial performed significantly better than both other groups , and there was no significant difference between the static-CAI and text groups. Animation seems to have had a positive impact on content retention for students in this study. Hays (1996) correlated sixth, seventh and eighth grade students' spatial ability to the relative effectiveness of different versions of a computer-based presentation program focusing on the concept of diffusion. One version of the program included only textual material; another version included static illustrations; and a third version featured computer-generated animations. Based on the results of a spatial ability test, students were divided into low-spatial-ability and high-spatialability groups. Analysis of pretest to posttest gains (using a Concept Evaluation Statement test, in which students read a statement and wrote their own paragraph explaining what they know about what they read) found that gains for both high- and low-spatial-ability students were greatest with animation and lowest with text only and that differences among the program versions were significant. Hays noted, "animation appears to be best utilized in instructional settings that have some need for visualizations on the part of the learner." Researchers at Texas A & M University (Rieber, Boyce and Assad, 1990) studied the effects of animated, computer-based instruction on college students' learning and retrieval in physics. Groups of students were assigned versions of a lesson on Newton's first law of motion that differed according to the level of "visual elaboration" (animated graphics, still-frame graphics or no graphics). While there was no significant difference in achievement among the groups, students who had interacted with animated graphics required significantly less time to answer posttest questions. [This difference]...indicates that although animation did not affect learning, it helped to decrease the time necessary to retrieve information from long-term memory and then subsequently reconstruct it in short-term memory. There are many skills in which it is advantageous to minimize one's information retrieval time (e.g., diagnosis of medical problems or learning deficiencies). In another study, Rieber and others at the University of Georgia (Rieber, Tzeng, Tribble and Chu, 1996) found that university students performed better and more quickly and experienced less 2000 Research Report on the Effectiveness of Technology in Schools 57 frustration when feedback on a physics simulation was provided in the form of an animated graphic than when it was provided as a numerical text readout. Students clicked on-screen buttons to move an on-screen "ball" toward a "target" through a series of up, down, left or right "kicks." Each kick applied an impulse force to the ball, changing its speed and direction accordingly. The position of the ball and the target were reported continuously, either through a numerical display giving a horizontal and vertical value for both the ball and the target or through an animated display of the ball and target showing the ongoing motion of the ball. Students who received the animated feedback scored better in a posttest on Newtonian motion principles, completed the simulation more quickly (though using the same number of screen button "kicks") and reported significantly less frustration than those who received the numerical display feedback. Justice (1999) compared the effectiveness of still graphics and animated graphics versions of two computer-based learning modules designed to teach people how to use a cell phone. Each module addressed a specific cell phone operation. In one module, the task was "to recall, change and store a new number in the cell phone memory." In the other module, the task was ...to unlock and change the lock code on the cell phone... Both tasks required several steps and numbers to be entered correctly for the task to be completed. After completing the learning modules, university students answered questions about numbers referred to in the presentations. Then the students completed the two tasks with an actual cell phone as a performance assessment. The rate of successful task completion was dramatically and significantly higher for the students who completed the animated graphics version of the modules than for the students who used the still graphics version. For example, for the first assessment task each student attempted, 61 percent of the animated graphics group successfully completed the task, while only 11 percent of the static graphics group successfully completed the task. For the second assessment task each student attempted, 61 percent of the animated graphics group successfully completed the task, but only 17 percent of the static graphics group successfully completed it. Furthermore, the group using the animated graphics version consistently took significantly less time to complete each assessment task. Rieber (1991) conducted a study examining the effects of animated and still-frame graphics on the intentional and incidental learning of fourth graders. "Intentional learning" refers to the skills and concepts directly taught; "incidental learning" refers to "those objectives that are not directly taught but only implied through contextual cues provided in [the graphic] displays." Two groups of students worked through different versions of a lesson on Newton's laws of motion developed specifically for elementary grade students. One version included animated graphics and the other version included still-frame graphics. Using the animated graphics version resulted in significantly higher achievement with respect to both intentional and incidental learning. ChanLin and Chan (1996) compared different versions of a computer-assisted lesson on recombinant DNA technology that incorporated use of metaphors as well as animation. Six different versions of the lesson were developed, with: • • • • Animated graphics and metaphors Static graphics and metaphors No graphics but with metaphors Animated graphics but without metaphors 2000 Research Report on the Effectiveness of Technology in Schools 58 • • Static graphics but without metaphors No graphics or metaphors The researchers found that students using the version that included both animated graphics and metaphors significantly outscored students using all other versions of the program. The studies described above all suggest learning benefits when incorporating animated graphics rather than static graphics in instructional software designs. However, Shu-Ling (1998) found a learning advantage to using still graphics when presenting analogies on basic computer programming concepts and functions within a computer-based learning environment (CBL). This program consisted of two parts: an introduction section for teaching the basic concepts, structures, and functions of the programming language...and the analogy section... Shu-Ling compared three versions of the lesson: The CBL lesson differed in the way that the analogy was displayed on the screen. [The text-only version] contained verbal descriptions of the analogy for each programming concept. For example, the INPUT statement is like a robot that asks you to give it data. You can give him data through typing on the keyboard. A robot will then put the data into storage. [The text-and-still graphics version] contained the verbal descriptions supplemented by still graphics that illustrated the ideas presented in the text. For example, a still graphic displays the robot that turns its palm up and holds the data give[n] to it, standing by the storage place. [The text-and-animation version] contained the verbal descriptions supplemented by animated graphics that demonstrate the internal process of the computer for the INPUT statement. For example, the robot is given a number from the learner. The number "flies" into the hand of the robot. The robot then puts the number into storage. On a test of programming concepts and functions, college students who used the text-and-still graphics version of the CBL significantly outscored students who experienced the text-andanimation and text-only versions, "suggesting that analogy with still graphics produced higher comprehension." One possible explanation for the superiority of the text-and-still graphics version over the text-and-animation version is that the nature of the analogies may not have required animation for initial comprehension and that the combination of text and still graphics was easier to store in and retrieve from memory. Spoken Narration, Text, and Sounds with Graphics Researchers from the University of California, Santa Barbara completed several studies to compare a variety of design options involving animation in combination with on-screen text, spoken narration, music, and/or ambient sound effects (Moreno and Mayer, 2000; Moreno and Mayer, 1999a; Moreno and Mayer, unpublished). In one experiment, Moreno and Mayer (1999a) developed three versions of a computer animation demonstrating the process of lightning. One group of college students viewed the animation while concurrently listening to corresponding narration. A second group watched a version of the animation with "integrated" explanatory on-screen text (within the graphic frame, next to the animation). The third group saw a version with the "on-screen text separated...from the animation" 2000 Research Report on the Effectiveness of Technology in Schools 59 (at the bottom of the screen). After experiencing their assigned version of the computer animation, the students completed three tests: • "The matching test presented...frames from the animation" and several terms related to the formation of lightning, along with instructions to label each frame with the appropriate terms. The retention test required students to "write down an explanation of how lightning works." In the transfer test, students were asked four problem-solving questions in which they had to apply what they learned (e.g., "What could you do to decrease the intensity of lightning?"). • • Students who viewed the narrated animation versions scored significantly higher on all three tests than students who experienced the versions with on-screen text. In addition, the group who watched the animation with integrated explanatory text significantly out-performed the group who saw the animation with separated text on the retention and transfer tests, but the two groups were statistically equivalent on the matching test. In another experiment, Moreno and Mayer (1999a) used the same computer animation of the process of lightning as in the previous experiment, but this time they compared six versions: One group of students viewed concurrently on-screen text while viewing the animation..., and a second group listened concurrently to the narration while viewing the animation... In addition to the concurrent groups, four groups of sequential presentations were included. Students listened to a narration preceding the corresponding portion of the animation..., listened to the narration following the animation..., read the on-screen text preceding the animation..., or read the on-screen text following the animation. In each of the versions with on-screen text, the text appeared in the same location on the screen. As in the previous experiment, the students who experienced the narrated animation versions scored significantly higher on all three tests than the students who watched the versions with on-screen text. On the retention and transfer tests, no significant difference was found between students who received the animation and explanation (text or narration) simultaneously and students who received the explanation before or after the animation. However, the results on the matching test were more complicated: "although the text groups performed significantly better when the presentation was sequential than when it was simultaneous..., the sequential and simultaneous narration groups did not differ from each other." Another series of studies by Moreno and Mayer (unpublished) involving a multimedia plant biology program confirmed the superiority of an animation-plus-narration software design over an animation-plus-explanatory text design in terms of students' content retention and problem-solving transfer performance. (For further detail, see Pedagogical Agents above.) In yet another pair of experiments, Moreno and Mayer (2000) explored the impact of including background music and/or ambient sound effects in instructional multimedia. The first experiment involved four animation-with-simultaneous narration demonstrations of lightning: • Only the animation with concurrent narration 2000 Research Report on the Effectiveness of Technology in Schools 60 • • • The animation, the narration, "plus [a]...synthesized and bland...instrumental music loop" The animation, the narration, "plus...environmental sounds [related to each] respective event... [For example,] a gentle wind, for the portion of the animation depicting air moving from the ocean to the land" The animation, the narration, the music loop, and the environmental sounds In general, students who experienced animation-with-narration versions that also included music scored significantly lower on retention and transfer tests (described above) than students who completed versions without music. The inclusion of music had no impact on matching test performance (a relatively low-cognitive level task). Furthermore, no significant differences on any of the tests were found between students who did and did not experience versions with environmental sounds. Additional analysis revealed that on the retention test, students who heard both music and sounds with the narration "recalled significantly less information than each of the other groups." Students who heard music but no sounds recalled significantly less than the narration-plus-sounds and narration-only groups, which did not differ from each other. In a follow-up experiment involving four animation-plus-simultaneous narration representations of "the operation of hydraulic braking systems," the disadvantage of including music in the presentation was confirmed with respect to retention and transfer test performance. Once again, the inclusion of music had no impact on matching test performance. However, in this case, students who experienced animation-with-narration versions that also included environmental sounds scored significantly lower on retention and transfer tests than students who completed versions without environmental sounds. The researchers surmised that the two mechanical sounds "repeated several times at different points throughout" the braking system’s animation might have been "too intrusive, arbitrary, and ambiguous." By contrast, the seven natural sounds used in the lightning animation (each of which was played only once) might have been more meaningful and less distracting. Based on the results of these last two experiments, Moreno and Mayer (2000) recommended: When presenting a multimedia explanation, only include complementary stimuli that are relevant to the content of the lesson. The most straightforward practical implication is that instructional software designers should carefully limit the amount of auditory material in multimedia lessons rather than add auditory materials for reasons of appeal or entertainment. Multiple Representations of Arithmetic Problems In another study, Moreno and Mayer (1999b) compared the performance of sixth grade students using single-representation (SR) and multiple-representation (MR) versions of a computer-based multimedia program on addition and subtraction of signed (positive and negative) numbers. The SR version included only symbolic number sentences (e.g., 2 – -3 = __), while the MR version also featured ...a visual representation that uses a number line and a computer animation to show how the symbolic number sentence relates to a bunny's movements along the number line, and... a verbal representation [using] written explanatory text that has the bunny describe in words how the symbols relate to its movements along the number line. 2000 Research Report on the Effectiveness of Technology in Schools 61 The researchers found that students who had used the MR version performed significantly better than the SR students on "difficult" problems (types that received more wrong answers on the pretest), but not on "easy" problems. Students using the MR version also learned significantly faster (showed more improvement on earlier sessions with the program) and made significantly fewer "negative bias" mistakes (confusing the symbol for subtraction with the symbol for a negative number) than students using the SR version. Additionally, the researchers found that higher-achieving students who used the MR version performed significantly better on the posttest than higher-achieving students who used the SR version. However, no such result was found for lower-achieving students. They concluded that ...the benefits of using MRs for example problems are strongest on difficult problems and for students who already have a good knowledge of the basic arithmetic of natural numbers. This suggests that in the teaching of mathematics, the ideal learning situation might be to first bring the lower achieving students to a higher level of proficiency on the symbolic representations so that they can benefit more fully from the multimedia, multirepresentational environment. Dynamic Visualization Foley (1998) studied the impact of incorporating "dot-density" computer visualizations of heat flow into an eighth grade science course using the Computer as Learning Partner (CLP) curriculum. Four visualization activities were spread out over the course of the semester. For each activity, students worked in pairs to complete a worksheet during a single class period. Foley found that on the final exam, students who had used the visualizations scored significantly higher on three of five questions requiring an internalized understanding of heat flow, compared to the previous semester's students, who had completed the same class, with the same teacher and curriculum but without the visualizations. Foley noted: This finding is interesting because the CLP curriculum was already a constructivist and technology rich environment that has been very successful at helping students understand these concepts. And yet the use of visualization tools for a very short intervention led to a significant improvement in students' scores. It seems unlikely that the visualizations alone would produce the effects shown here, but rather the visualizations fill a complementary role to the knowledge integration activities of the CLP curriculum. Video Researchers at the Learning Technology Center (O'Banion, Goldman, Sharp, Bransford, Vye, Beaty and Saul, 1993) at Vanderbilt University found evidence for the advantages of video as an instructional design element. They studied the effects of dynamic computer-controlled video on the story comprehension abilities of kindergarten students. They compared a video version of a story to an audio-only version (enhanced to communicate information presented in the video but not mentioned in its accompanying narration). When retelling the story, children who received the video version made significantly more summary and inference statements (representing a meaningful translation of the gist rather than a literal retelling in the story's exact words) and a wider variety of information (e.g., character and setting descriptions; internal states of characters) than children who listened to the audio-only version. The video group was also more likely to include the story's key components (the beginning, the problem, the attempt to resolve the problem and the final resolution). These results were similar for at-risk and non-at-risk students. The 2000 Research Report on the Effectiveness of Technology in Schools 62 researchers concluded, "video facilitates the formation of mental representations for stories" and suggested that it may help children develop their general sense of story structure, an important prereading skill. Two studies by Swan (1996) compared student learning from two versions of a hypermedia program on the civil rights movement, one featuring video commentary, the other replacing the video with a transcribed text of the commentary. In the first study, students who had viewed the video showed greater knowledge on a posttest than students who had read the transcribed text. In the second study, students' prior knowledge was tested on a pretest and compared with posttest performance. The results confirmed that students had gained more new knowledge from the version including video than from the version that included transcribed text. The authors conclude by suggesting ...that video adds an affective dimension to hypermedia materials, making them more meaningful, hence more memorable; and that video is represented differently from text in memory and that such representations are more complex and provide more links to...knowledge Researchers at the French Educational Project at Louisiana State University (Borras and Lafayette, 1994) found support for adding subtitles to video in foreign language software. College students taking French language courses who experienced video-with-subtitles software were judged to have given a significantly better overall "oral communicative performance" than students who used the same software without subtitles did. The video-with-subtitles group was judged superior in the effectiveness, accuracy, organization and fluency of their attempts at spoken French. Miech, Nave and Mosteller (1997) described a study by Garza (1991) that confirms the effectiveness of captioning for both foreign language and English as a Second Language (ESL) students. Students in an advanced Russian class who watched videos with captions in Russian averaged a 90 percent gain in correct responses over students who watched without captions, while ESL students who watched videos with captions in English averaged a 60 percent gain. In summarizing the results of their survey of studies on computer-assisted language learning (CALL), Miech, Nave and Mosteller concluded that "captioning video segments used in foreignlanguage instruction may be the most cost-effective measure a college or university can take to improve student learning." Navigational Technique Schroeder (1994) compared the effects of three hypermedia navigation techniques on students' content recall and comprehension and on their acquisition of "structural knowledge" (the hierarchical relationships among content elements). College undergraduates were divided into three groups, each of which was exposed to a different version of a hypermedia database on the human circulatory system. In one version, students navigated via "hotwords": words or phrases that, when chosen, brought a corresponding screen of information without a description of the relationship between the screens. The second version provided a graphical browser or navigation map that used connecting lines to show the links among the screens without text descriptions of the relationships. The third version provided a similar graphic browser but included text descriptions of the relationships among the screens along the connecting lines of the map. Students who used the graphical browser-without-descriptions version scored the highest on a test of content recall 2000 Research Report on the Effectiveness of Technology in Schools 63 and comprehension. They also demonstrated the greatest gain on a test of structural knowledge. Niederhauser, Salmen, Skolmoski and Reynolds (1998) correlated the performance of undergraduate students using hypertext to learn about behaviorism and constructivism to their use of several navigational features in the hypertext program. They found that student accessing of the topics map, a graphic representation of the hierarchical structure of the hypertext, contributed significantly to positive posttest performance. However, use of the compare and contrast links, buttons linking comparable screens in the behaviorist and constructivist sections of the hypertext, related negatively to student performance, raising questions about the value of this type of link in hypertext programs. Chou and Lin (1998) compared the impact of several different types of navigation maps in a hypertext learning system. Taiwan university students who were enrolled in a required Introduction to Information Technology course completed one of five versions of a hypertext program that presented basic information about computer networks. One version presented a global map with a hierarchical listing of all 94 hypertext nodes in the program. By scrolling up and down in the map, students could see where they were and where they had already been (indicated by different colors). Another version presented a local map showing only the local area where students were working. These local maps were not updated until students left the local area. A third version presented a local tracking map, similar to the local map but continuously updated and with the current node always shown in the center of the map. A fourth version of the program allowed students to choose which of the three map versions was currently displayed. The fifth version included no maps. Except for the no-map version, a map was displayed continuously on the left side of the screen. Students were assigned to search for 10 specific pages within the hypertext program, then spent 30 minutes browsing freely. Students were assessed on their understanding of the relationships among the concepts (nodes) presented in the program, and on their performance on the search tasks. The researchers found that students in the all-maps and global map groups performed significantly better than students in the local tracking map group on the conceptual relationships test. The researchers also found that students in the all-maps and global map groups required significantly fewer steps to find specific pages in the hypertext program than students in the other three groups. Taken together, these results suggest that despite the size and complexity of global maps, access to global maps may be more beneficial than access to local and local tracking maps. It is noteworthy that local maps, and especially local tracking maps, were found to provide no significant advantage to students over no map at all. Paolucci (1998) reported that fifth grade students performed significantly better with a branching hypermedia document structure than with either a hierarchical or a conventional structure. With the conventional approach...most topics are linked referentially on the assumption that students can begin learning some subjects when requisite knowledge is only partially grasped. Students can freely access any of its nodes... With the branching approach...the learner is presented with choice points or nodes at which different responses will result in different alternative paths through the instruction. In some cases, access to lower level nodes might be blocked until all nodes at higher levels are viewed... 2000 Research Report on the Effectiveness of Technology in Schools 64 Finally, with the hierarchy approach...links are only established between nodes containing information prerequisite to others. Users do not access nodes teaching higher level skills until those at lower levels are accessed. In the case of the specific hyperdocument on ecology that was tested in this study, "the hierarchical hyperdocument [was] comprised of 39 nodes and 37 links, the branching of 39 nodes and 50 links, [and] the conventional of 39 nodes and 173 links." Paolucci found that students using the branching version scored significantly higher on a posttest than those who used either the hierarchical or the conventional version. Further analysis showed that this difference related specifically to posttest items that had been identified as testing higher level structural knowledge, as opposed to information recall. The researcher concluded: It seems evident from the findings that when "too much" freedom (as in the case of the conventional [approach]) or "not enough" freedom (as in the case of the hierarchical [approach]) is provided to early adolescent students in the form of a computer learning system, performance may suffer. Specht (1998) tested the effects of two techniques for making hypertext easier for learners to use: adaptive annotation, which uses a system of colored icons to "[supply] the user with additional information about the content behind a hyperlink," and incremental linking, which hides links until students have viewed the appropriate prerequisite information. These techniques were incorporated into different versions of a hypertext document that presented information about prionic diseases. Neither technique by itself led to significantly greater gains on subjects' posttest scores than the standard hypertext version (i.e., no additional annotations and all links visible). However, a combination of the adaptive annotation and incremental linking techniques produced significantly greater knowledge gains than the standard hypertext. (This combination of techniques provided a presentation environment in which learners, although restricted in their ability to navigate, "had full transparency of the whole hyperspace because they could see...where they will be allowed to hyperjump later on.") Additionally, Specht reported that subjects required significantly less time to complete their viewing of the hypertext with either the adaptive annotation or the incremental linking than with the standard version, and that the version combining both techniques required the least time of all. Ritchie and Gimenez (1995) studied the effects of graphic organizers in a computer-based fourth grade science program for both native English and native Spanish speakers. Four versions of the program were developed, two in English and two in Spanish. The two versions in each language were identical, except that one version included graphic organizer screens showing connections among topics, while in the other version, topics were simply presented in a list at the same points in the program. Students in both languages showed significantly increased learning from the version that included graphic organizers, both on an immediate posttest and on a delayed posttest, with no significant differences between English- and Spanish-speakers. The results suggest that use of graphic organizers may promote learning gains in students across major cultural groups, regardless of primary language. Eliasen, McKinstry, Fraser and Babbitt (1997) compared three possible interface designs for the Session Manager software used to access electronic catalog and database searching at the University of Washington libraries. One version, matching the existing Session Manager interface, maintained separate buttons for the library's catalogs of its holdings but grouped the bibliographic databases by subject. A second version preserved the same button layout, but with additional explanatory text for each button. A third version abandoned the grouping by subject and 2000 Research Report on the Effectiveness of Technology in Schools 65 consolidated the first screen to four buttons describing type of resource: Catalogs, Indexes, Reference and All Databases. The researchers found that on a test involving appropriate selection of databases for specific purposes, undergraduate students performed significantly better on both revised versions of the interface, with no significant difference between the two versions. These results support the common-sense design principle that screen interfaces are easier to use when they clearly communicate the functionality of each navigation option. Motivational Contexts for Learning Three studies shed light on the learning effects of incorporating motivational contexts into instructional software. A study by Parker and Lepper suggests the value of embedding computer-based instruction in fantasy contexts. (1992) Three groups of third graders were exposed to different versions of a Logo program designed to teach basic geometry concepts. Each student in the first group was allowed to choose one of three fantasy contexts: • • • A pirate in search of buried treasure A detective hunting down criminals An astronaut seeking out new planets in space Students in the second group were each assigned one of the three fantasy contexts. The third group used the software without a fantasy context. Students who received a fantasy context version (whether self-selected or assigned) significantly outscored the no-fantasy group on immediate and delayed tests of their understanding of Logo procedures. Furthermore, the students exposed to fantasy contexts significantly out-performed the no-fantasy students on a delayed test of geometry concepts. Schofield (1997) described a study by Light and Littleton (1997) in which the particular scenario theme was found to have a significant impact on student performance. According to Schofield, ...changing the scenario used in software from one involving pirates to one involving teddy bears had a marked effect on girls' reactions to it. Specifically, pairs of girls performed markedly better with the latter scenario than they did with the former one even though the cognitive tasks involved were identical. Rieber and Noah (1997) studied the effects of adding a game context and a visual metaphor to "a simple computer simulation that modeled the relationship between acceleration and velocity using a discovery-based approach." University students clicked on left or right on-screen arrows to change the acceleration of an animated ball moving along a number line on the computer screen. One version of the simulation also included a game context, in which students were directed to change the direction of the ball's motion within a certain space on the number line, making it do a "flip-flop," as many times as possible in two minutes, with a resulting score. In another version, students were given a visual metaphor; they were told to pretend that the ball was rolling on a table top that could be tilted in either direction and were given a side view illustration of this visual metaphor that indicated the ball's current direction and speed by showing how the "table top" was currently tilted. Finally, one version included both the game context and the visual metaphor. 2000 Research Report on the Effectiveness of Technology in Schools 66 The researchers found that although enjoyment was significantly higher with the game context, participants who used the game versions scored significantly lower on a posttest measuring direct understanding of the relationship between acceleration and velocity. The visual metaphor, by itself, seemed to have little effect on student learning as measured by the posttest. However, in a followup simulation designed to measure tacit understanding of the science concepts, participants who had completed the version that included both the game context and the visual metaphor significantly out-performed all the other groups. Window Presentation Styles As reported under Learner Characteristics and Student Achievement, style of window presentation can also have an affect on student learning. Bastecki and Berry (1996) found that dental hygiene students with high spatial ability learned better when information was presented in an overlapping window format, while those with low spatial ability learned better from a tiled window format. Ergonomic Combination of Design Features Osler (1996) developed a mathematical model for the design of instructional software incorporating ergonomic elements and demonstrated that software created according to this model could be used effectively in a fourth grade classroom. Osler designed two HyperCard tutorials presenting information about seven African American inventors and their inventions. One version presented traditional text linearly sequenced. The other version incorporated nine ergonomic factors, including story boarding, a guide, ease of navigation tools, non-sequential branching, humor, graphics and sound. Osler found that students who used the ergonomically developed tutorial experienced gains of 39.78 points (out of 100) from pretest to posttest, compared to a gain of 5.28 points for students using the linear text-based version--a statistically significant difference. The 71 studies focusing on software design characteristics underscore the fact that decisions about specific design features do have an impact on student achievement. This substantial body of design research gives software designers the opportunity to make informed decisions about how best to meet the learning needs of students. The most effective designs described in this section reflect both best practices and tested theories about how students learn. Recent Technologies and Student Achievement Several studies focused on four of the more recent applications of technology to education: telecommunication; videodisc; hypermedia; and adaptive testing. Telecommunication A variety of recent studies illustrate the merit of using online telecommunication for a wide range of educational purposes. Taken together, these studies indicate the benefits of computer-based telecommunication to collaborate and communicate with other learners, access online resources and conduct research, exchange documents and conduct discussions, facilitate professional development and conduct distance learning. 2000 Research Report on the Effectiveness of Technology in Schools 67 Collaboration and Writing to a Distant Audience In Riel's (1992) review of research on the use of networking for collaboration across classrooms in different geographic locations, she found evidence of improved academic skills. Riel (1992) described a study by Spaulding and Lake (1992), in which low-achieving remedial writers in New York collaborated on writing projects with students from four other states, France and Germany via a telecommunication network. According to Riel: The network activity was effective in increasing the writing skills of students who were less socially oriented and less academically skilled, perhaps because of the increased opportunities to interact with and learn from their teachers and peers. Gallini and Helman (1993) studied the impact of writing for a distant audience of peers on the quality of fifth graders' compositions. The students, from a predominantly Hispanic school in New Mexico, were linked via a world-wide communications network to a "learning circle" of classes in several U.S. elementary schools and one in the United Kingdom. For six weeks, students within the learning circle exchanged questions and answers on topics related to school and their home communities. Then students were assigned to one of three groups. ...the students were instructed to prepare a written essay that would be read by (1) their teacher, (2) a self-selected classmate or (3) the learning circle members. Essays written to learning circle members were holistically rated as significantly more effective and interesting than those written to other audiences. More specifically, essays to a learning circle audience were found to be significantly superior with respect to English mechanics and organization. Similarly, Iowa State University researchers found that fifth grade students writing to distant readers via a telecommunication network wrote better quality compositions than students writing for an audience of just their educators (Allen and Thompson, 1994). Each student in the distant audience group was partnered with a college student of the same gender who "read the fifthgrader's writing and made supportive, nonjudgmental comments that were then e-mailed back." All students in both the distant audience and teacher audience groups used word processing software as a writing tool. The writing samples of the distant audience group received statistically higher holistic ratings of writing quality and had statistically higher word counts than the writing samples of the teacher audience group. Nix (1998) reported that positive effects of using e-mail to write to a distant peer audience extended beyond writing on the computer to affect handwritten classroom compositions. Fourth grade students were assigned specific topics and content for e-mail sent every other week to a "keypal" at another school. For one of their assignments, they were instructed to write a persuasive essay to their keypal on whether families watch too much television. Students scored significantly higher in audience awareness and argumentation than a control group at another school who did not use e-mail and wrote their compositions by hand for a teacher audience. Subsequently, after three months of using e-mail, students were assigned to write a persuasive essay by hand to an imagined parent audience about bicycling with one friend versus hiking with another. Even without the stimulus of writing to a real audience, students who had used e-mail still scored significantly higher overall, were more aware of their audience, were more organized and produced lengthier texts than students who had not used e-mail. The researcher noted, 2000 Research Report on the Effectiveness of Technology in Schools 68 Given the limited exposure of the experimental school to e-mail, these results are remarkably strong. The effects of e-mail use, especially those from interacting with an authentic audience and using technology in an educational context, appear to transfer to subsequent handwritten essays. An evaluation of National Geographic (NGS) Kids Network telecommunication-based science activities Weir, (1992) found that fourth and fifth graders made significant achievement gains as a result of their involvement. Students in the NGS Kids Network group: • • • "...demonstrated significant increases in the use of...graphs for organizing...observations, while [a] control group did not" Showed significant improvement in data interpretation skills Demonstrated significant improvement in place knowledge, in the ability to identify map locations using longitude and latitude, in their "understanding of factors contributing to acid rain and [in] their ability to reason about the impact of these factors..." Access to Online Resources Researchers at the Center for Special Applied Technology (Follansbee, Gilsdorf, Stahl, Dunfey, Cohen, Pisha, and Hughes, 1996) compared student performance in a project-based unit on civil rights with and without telecommunication access. Fourth and sixth grade students from schools in seven districts across the United States completed the same unit of study, including curricular materials, computers and other technology. Students in one class at each school also used online resources and activities. Of the nine key learning measures that were used to assess student projects, students with telecommunication access scored significantly higher in four areas: effectiveness of presentation, presentation of a full picture, effectiveness of bringing together different points of view and completeness of the project. Overall, ...students who had online access learned more in the Civil Rights Unit than students who did not go online... Their final projects were rated as stronger overall and stronger in most of the specific competencies measured. Blanton, Moorman, Hayes and Warner (1997) analyzed the effects of the Fifth Dimension, an afterschool activities program incorporating a variety of interactive technologies such as computers, telecommunication and multimedia, on standardized reading and mathematics test scores of elementary students. Students in the program completed at least 30 one-hour visits to the Fifth Dimension, engaging in a variety of games and activities, including telecommunication activities for searching the Internet, tools for computer-mediated and video-mediated conferencing and multiple-user electronic games. Although the development of standard academic skills was not a major focus of the program, students who had participated in the Fifth Dimension showed significantly greater gains in mathematics and reading scores than students who had not participated in the program. This suggests that programs of mixed activities incorporating telecommunication and other technology resources can have a significant effect on academic learning, even when this is not a primary goal. Research by Schacter, Chung and Dorr (1998) provides an important qualification to the benefits of having upper-elementary and lower-middle school students search the Internet. They found that 2000 Research Report on the Effectiveness of Technology in Schools 69 fifth and sixth grade students who had already been using the Internet as an educational resource for five months experienced difficulty in locating appropriate documents to carry out specific assigned tasks. An analysis of students' information-seeking patterns showed that students "browsed" by clicking on hypertext links significantly more often than they conducted searches for information. Sixty-six percent of students used only one search engine, and a review of the search logs showed that 20 of 32 students entered full sentence requests when they were conducting searches. The researchers commented, This study confirmed that children are reactive searchers who do not systematically plan or employ elaborated analytic search strategies. Regardless of whether the information-seeking task was highly specific or vague, children overwhelmingly sought information by employing browsing strategies. Students were assigned two search tasks, one narrowly defined ("Find information about the three types of crime that happen most in California"), the other broadly defined ("Find at least three pieces of information on the Internet that will help you develop a plan to reduce crime in California"). They performed significantly worse in conducting searches with the narrowly defined goal than with the more general goal. Additionally, student ratings of the quality and usefulness of the documents they had bookmarked were significantly higher than those of experts in the more narrowly defined task. The researchers also noted "few students bookmarked counter arguments to the solutions they found, and even less pursued multiple solutions." Taken together, these findings suggest the need for a variety of supportive strategies for fifth and sixth graders who are assigned Internet searching tasks, possibly including "natural language" search engines, guided practice in conducting Internet searches, broadly defined Internet search tasks, and more rigorous instruction in how to identify and use relevant sources. Agarwal and Day (1998) reported on the impact of Internet use on college students enrolled in a graduate microeconomics course and an undergraduate macroeconomics course. Students in both Internet-using classes were required to complete web-based projects that involved accessing and downloading information from the Internet. Additionally, answers to student questions about course material in the Internet classes were posted to class e-mail lists. Students also used e-mail and the class mailing lists to interact among themselves and answer each other's questions. Students in control group classes taught by the same professors were assigned similar learning projects, but were not encouraged to use the Internet and for the most part did not do so. The researchers found that students who used the Internet performed significantly better on the Test of Understanding College Economics III (TUCE) and had significantly higher final course grades than those who did not use the Internet. Gilliver, Randall and Pak (1998) reported significant learning gains from making available extensive supplemental materials via an Internet site to undergraduate students enrolled in a financial accounting course in Singapore. One hundred and eleven students out of 444 total were assigned to the experimental treatment. Materials at the site included lecture notes and examples, tutorial questions and solutions, frequently asked questions, advanced readings, advanced questions, and a variety of other resources. Students had the freedom to view whatever resources they chose, but the site programming tracked student choices and subtly encouraged students to access other resources at their level through "judicious use of the web page buttons at the bottom of each page on the site." The researchers found that students who had access to the Internet site scored significantly better on the comprehensive departmental final exam than students who did not have access to the site. 2000 Research Report on the Effectiveness of Technology in Schools 70 Document Exchange and Discussion Smith (1992) studied the achievement effects of integrating university courses in education, sociology, geology and business with a computer-based telecommunication network. Students connected to the network via personal computers to submit assignments and to interact with the instructor and other students. Students using the network received significantly higher course grades than equivalent students who did not use the network. In another study of university students, Marttunen (1997) found that e-mail was "a feasible study tool in practicing academic argumentation." Volunteers in a sociology of education course took part in six-week e-mail discussions of the assigned texts, writing at least two messages a week. Two discussion groups were in seminar mode, with a tutor selecting discussion topics and giving regular feedback. Members of two other groups selected the discussion topics themselves and received only occasional feedback from the tutor. An analysis of e-mail messages showed that the "level of argumentation improved during the experiment" for students practicing both modes of discussion. Additionally, the researcher found that members of the student-led groups engaged in "more and higher level" argumentation directed against the claims of other students or tutors, compared to students in the tutor-led groups. Professional Development Effective instructional use of computer-based telecommunication depends on teachers' ability to use it and to integrate it within the curriculum. Ivers and Barron (1994) compared two ways to provide preservice teachers with instruction in the use of an electronic mail system: video and computer-based tutorial/simulation. Students viewed the video in-groups of three and four were encouraged to ask questions and were allowed to review any information on the videotape. Students assigned to the computer-based tutorial/simulation worked individually... Both [instructional materials] were critiqued and validated by experienced...users [of the email system and the system's] support personnel. The preservice teachers who received the computer-based instruction were significantly more successful at applying what they learned to actual e-mail tasks. This is likely due to the simulation experiences provided in the software. Distance Learning Another form of telecommunication involves real-time video and audio communication, used to make learning at a distance possible. As reported previously, two meta-analyses confirm the viability of such interactive distance education as a flexible alternative to more traditional instruction. Cavanaugh (1998) reported that 14 out of 19 studies of distance education using videoconferencing or on-line telecommunication in grades 3 through 12 had a positive effect size for distance education. Cavanaugh found that "when distance education is necessary in order to provide needed educational options to students, it can be as effective as traditional instruction." Similarly, Machtmes (1998) found in 19 studies of distance video telecourses for adults that "there does not appear to be a difference in student achievement between distance learners and traditional learners." Swan and Jackman (1998) compared the performance of high school students enrolled in distance learning courses from a remote site to that of students enrolled in the same classes at the host site 2000 Research Report on the Effectiveness of Technology in Schools 71 where the instructor was located. Classes were conducted using real-time live audio and video interaction between teacher and students and included courses in foreign language, agricultural business management, vocational marketing, natural resources, calculus, chemistry, art, statistics and animal science. Swan and Jackman found no significant differences in course grades between host site students and remote students, either globally or on a course-by-course basis. This finding suggests that remote students are not at a disadvantage in distance learning courses, making such courses a more viable option for providing otherwise unavailable educational opportunities to students--a primary reason for introducing distance learning. As the researchers noted, Students enrolled in distance education courses were primarily located at remote sites. Remote site students were provided the opportunity to take these courses because courses were being offered via distance education technology. Without this opportunity, most of these students...would not have been able to enroll in these courses. Martin and Rainey (1993) compared the effectiveness of satellite-delivered anatomy and physiology instruction to the same instruction provided face to face. Students from seven high schools received instruction via satellite and students from another seven high schools received instruction in class. The schools were matched according to community population, geographical characteristics, student enrollment, race, gender and socioeconomic status. Students in both groups were equivalent in science achievement prior to the start of instruction and all schools involved in the study used the same instructional materials. In all but two of the satellite schools, a local science teacher served as a class facilitator. Students who received satellite-delivered instruction achieved at a significantly higher level than students experiencing face-to-face instruction. Contributing factors may have included the local science teachers serving as facilitators and the particular abilities of the distance learning teacher. Schutte (1996) identified similar effects among students in a university social statistics course who were randomly divided into two groups, one that met with the instructor each Saturday and another that completed the course through telecommunication as a distance learning experience. Students in the distance learning group were assigned to collaborate via e-mail on a weekly basis to generate statistical reports, to conduct a weekly hypernews discussion, to input forms through the World Wide Web and to engage in weekly real-time Internet "chats" with other students and the professor. The only time they met was to complete the midterm and the final. Students in the distance learning group expressed frustration at not being able to ask questions of the professor in a face-toface environment, but averaged 20 points higher on the 100-point midterm and final, a highly significant difference. The researcher speculated that the frustration students felt might have led to greater involvement among members of the class as a form of compensation, leading to better performance. This example presents evidence that under appropriate conditions, virtual instruction via telecommunication can be more effective than traditional instruction. Navarro and Shoemaker (1999) compared the performance of undergraduate students who were given a choice of whether to take an introductory macroeconomics course by cyberspace or in the traditional classroom (both taught by the same professor). In looking at reasons why students chose the web-based version of the course, Navarro and Shoemaker found that 2000 Research Report on the Effectiveness of Technology in Schools 72 ...44% of the Cyberlearners cited "convenience" as their primary reason... In addition, 28% indicated a desire to learn at their own pace, [and] 20% opted for the cyber option because they didn't have to attend class. Students who chose the cyberspace option did not meet with the professor after the beginning of the course. Instead, they received lectures with review questions on a CD-ROM, together with copies of the standard textbook. They also took part in threaded electronic bulletin board discussions guided by a weekly opinion poll and by questions posted by the professor with links to online articles from publications such as Business Week and the Wall Street Journal. Students were required to post at least two responses per week. A weekly chat session was also conducted by either the professor or the teaching assistant plus the students had e-mail access to the professor and TA. The researchers found that the "cyberlearners" performed significantly better on the final exam, even after correcting for overall academic ability based on students' grade point average. They concluded, "The results strongly suggest that Cyberlearners can learn as well or better than Traditional Learners." Jiang and Ting (1998) reported survey results for students enrolled in 17 web-based distance learning courses delivered through the State University of New York Learning Network. Courses included a variety of subject areas, such as American literature, biology, algebra, English composition, information science and psychology. They found that the percentage of grade weight assigned to discussion and the instructor's specification of discussion requirements (e.g., number of contributions, correct use of grammar) correlated positively and significantly to students' perceived amount of learning. Class size did not correlate significantly to students' perception of learning; however, none of the classes included more than 25 students, a relatively small class size for higher education. Video Several research efforts found advantages to choosing videodisc and other video-based technologies as instructional media for both pre-college and college students. Mathematics Kitz and Thorpe (1992) compared the effectiveness of videodisc-based algebra instruction to conventional instruction using a textbook with learning disabled adults preparing for college. Students using the videodisc significantly outscored the students using the textbook on two different tests of algebra achievement. Science Five studies showed the benefits of videodisc-based instruction in science. McWhirter (1991) investigated the achievement effects of a science videodisc on sixth graders studying weather topics. When the same teachers each taught one group of students using the videodisc and an equivalent group using textbooks, the videodisc group significantly outperformed the textbook group on a test of weather concepts. Niedelman (1991) compared the effectiveness of an earth science videodisc to textbook-based instruction plus hands-on laboratory experience with 8th graders. The videodisc stressed "core concepts," focused on "causal relationships instead of topics," and used charts as "graphic organizers" to synthesize and organize chunks of interrelated information. Students using the 2000 Research Report on the Effectiveness of Technology in Schools 73 videodisc achieved at significantly higher levels than students receiving textbook-plus-handson instruction on a test of content knowledge and a test of science problem-solving skills. A study by Grossen and Lee (1994) confirmed the advantage of this core concept earth science videodisc over textbook-based instruction for middle school students. The students were divided into three groups, each of which experienced a different instructional approach. One group used the videodisc and engaged in hands-on laboratory activities. The second group used the videodisc and participated in a problem-solving strategy ...designed to focus students' attention on representing the problem in terms of the underlying causal principles. This was done by...presenting...problems that did not have enough information to allow the students to solve them. In response...students were expected to write questions to elicit the relevant information for solving the problem. The third group used an earth science textbook and engaged in the problem-solving strategy described above. Students in the two-videodisc groups demonstrated significantly greater gains on a test of earth science concepts than students in the textbook group did. Furthermore, the video-plus-problem-solving group showed significantly higher achievement than the textbook-plus-problem-solving group on a test requiring application of earth science concepts to problem-solving situations. Researchers from the University of Oregon (Muthukrishna, Carnine, Grossen and Miller, 1993) explored the effectiveness of videodisc instruction in eliminating common science misconceptions of 8th graders studying earth science. The earth science videodisc was designed to provide "indepth, conceptually integrated instruction." The researchers referred to misconceptions as alternative frameworks (AFs)--common sense ideas and viewpoints about natural phenomena that do not conform to a sound scientific understanding. Of 78 AFs held by students "prior to instruction, only 7 persisted," a statistically significant conceptual change. Only 4% of the high ability [students] showed evidence of possessing AFs on the posttest, compared to 72% on the pretest... Only 6% of the [low ability] students showed evidence of AFs on the posttest, while 88% had AFs on the pretest... On the posttest, the students' responses were nonarbitrary, nonverbatim and substantive, indicating that meaningful learning...had occurred. Rowry (1994) found that a self-paced interactive videodisc covering a full semester of chemistry curriculum was superior to conventional instruction for urban high school students. The software combined presentation of concepts with problem-solving and simulation activities. After four months of instruction (two 45-minute periods per week), students who used the videodisc demonstrated significantly greater chemistry achievement than those receiving traditional instruction. Social Problem-Solving Bain, Houghton, Sah and Carroll (1992) compared the effectiveness of three methods for teaching social problem-solving to elementary and junior high school students: teacher-led interactive videobased instruction; teacher-led linear video-based instruction; and teacher-led instruction without video support. Students receiving the interactive video-based instruction achieved at a significantly higher level than students experiencing either of the other approaches did. 2000 Research Report on the Effectiveness of Technology in Schools 74 Video Instruction with College Students Six studies indicate the potential of videodisc-based instruction with college students. Viesulas (1994) compared the impact of demonstrations of standard chemistry experiments via video and via live lecture presentation. Students who watched the video demonstrations received significantly higher grades on their lab reports, as well as on the midterm and final exams. They also completed experiments in less time, with fewer equipment breakages. Video offers the advantages of allowing each student a closer view of the presentation and permitting replay of any segment as needed. A study by Ziegler (1990) examined the effectiveness of three different methods of introducing university students to the academic library: interactive video with learner control, linear video and traditional guided tours. Students who had used the learner-controlled interactive video scored significantly higher than other students on measures of recall learning and self-perceived effectiveness at using the library. Woodruff and Heeler (1990) reported on a unique application of interactive videodisc technology to administer aural tests to university students taking a music appreciation course. [Two]...groups were given study guides that identified aural objectives and specified the location of the musical examples for study... [One group] was required to take aural tests over each unit in a supervised computer laboratory... The [other] group did not take [the tests]. The testing followed a competency-based method developed by Keller (1968), which has proved successful in science education (Kulik, Kulik, and Carmichael, 1974). Students who took the aural tests received significantly higher grades on unit exams than the other students did. Johnson (1993) compared the effects of three instructional approaches for teaching college-level lessons in human resource development: conventional lecture-demonstration; interactive video with students handling the computer controls; and interactive video with the instructor handling the computer controls. Both videodisc approaches resulted in significantly higher levels of student achievement than conventional instruction in a test of initial learning and a delayed test to measure retention. There were no significant achievement differences between students who directly controlled the computer and those who did not. This is good news for instructors who wish to use interactive video as a whole class activity. Two studies provide evidence of the effectiveness of videodisc-based instruction in teacher education. Bitter and Hatfield (1993) compared the effects of two methods for teaching about the use of geoboards to elementary education majors, as part of mathematics methods courses. Both approaches began with instructor-led lessons. For one group of students, this was followed by videodisc-based instruction. For the other group, the follow-up consisted of cooperative, hands-on experiences with geoboards and discussion of the application of geoboards to classroom teaching. While both groups showed achievement gains in knowledge of geoboards as an educational tool, students receiving videodisc-based instruction demonstrated significantly higher gains. Vitale and Romance (1992) examined the effectiveness of videodisc instruction plus supplementary activities focusing on core science concepts with female elementary education majors. One group of students received conventional science methods instruction. The other group followed the same 2000 Research Report on the Effectiveness of Technology in Schools 75 syllabus but also received videodisc-based lessons, completed corresponding workbook activities and prepared and presented model science lessons. The students who used the videodisc and participated in the supplementary activities demonstrated significantly higher achievement on a test of application of science concepts. These six studies suggest the effectiveness of interactive videodisc and other multimedia technology when the skills and concepts to be learned have a motion-visual or aural component. Note that the videodiscs used in these studies incorporated carefully considered, research-based instructional designs. One Caveat Research from Indiana University (Billings and Cobb, 1991) provides an important caveat that the effectiveness of interactive video may depend on students' comfort level with technology. After receiving computer-assisted interactive videodisc instruction, juniors in a baccalaureate nursing program who were more comfortable with the computer significantly out-achieved students who were less comfortable. Based on this finding, the researchers recommended ...efforts to create an atmosphere for acquiring positive attitudes such as orientation programs, frequent and repeated use and availability of faculty and support personnel... They also found that students with a high need for mobility while learning tended to be less comfortable using interactive video. The researchers suggested those students with "high mobility needs should be encouraged to take frequent breaks when using" computers. Hypermedia Brantmayer compared the effectiveness of hypermedia to traditional lecture for graduate students studying safety and industrial hygiene (1994). The topic of instruction was noise and hearing conservation. Students who used the hypermedia program demonstrated significantly superior achievement on a test of hearing conservation concepts and principles. Montazemi and Wang (1995) studied the impact of a hypermedia tutoring program that had been implemented to help students in a management information systems course prepare for weekly lab tests through questions and feedback. Comparison of final examination scores with scores for the previous year showed a significant gain from 71.43 percent the previous year to 81.81 using the tutoring system. Analysis showed that students who spent more time with the program generally did better on the lab tests and the final examination. Research by Barnes (1994) suggests the value of supplementing lectures with hypermedia in college literature courses. The hypermedia software was developed to teach interpretation of Hamlet as part of a course on Shakespeare for English majors. The software features four concept maps covering "characterization, plot, literary analysis and perspectives on revenge tragedy and Oedipaloedipal interpretations." Students cannot enter a new concept map until the current one is completed. They can, however, pursue higher level links within the current concept map. Content selections include segments of two film versions of Hamlet, video commentary, written commentary, the text of the play and text screens of additional information and literary criticism. Students who used the hypermedia program and attended class lectures significantly outscored lecture-only students on an objective test that covered Hamlet. 2000 Research Report on the Effectiveness of Technology in Schools 76 Reed and Rosenbluth (1992) found support for having students become creators of content-based hypermedia. High school seniors at a summer honors academy spent a month learning HyperCard and researching the humanities of a period of four decades (including art, history, science, music, literature and technology). Students were divided into four teams, with each team focusing on a different decade. Students who developed hypermedia humanities presentations demonstrated significant increases in their perceptions of how different cultural factors influenced their decade of study and how each factor influenced other factors. Students also showed a significant increase in knowledge of cultural values, historical events and social reforms. Liu (1998) reported that having fourth grade students create hypermedia programs significantly improved their creativity scores in several areas. Students in two fourth grade classes created two HyperStudio stacks on science topics (plants and ocean). Pretest to posttest comparison on the Torrance Tests of Creativity Thinking (TTCT Figural) found that students significantly increased their creative thinking scores in the areas of fluency (how many relevant responses a child gives), elaboration (how many details are added to the main idea), and resistance to premature closure (how well a child "can make the mental leap that makes original ideas possible"). Further analysis showed that students working collaboratively increased their elaboration scores while students working individually showed no gains in this area, and that improvement in resistance to premature closure was significantly greater for students working collaboratively. Wiley and Voss (1999) compared the performance of undergraduate students constructing narratives, summaries, explanations, or arguments based on eight historical source documents in a web-like environment to that of students who received the same content presented in print format as a textbook-like chapter. They found that although students who read the textbook chapter performed significantly better on recognizing whether specific ideas appeared in the material they had read, students who read the web documents were significantly better at determining whether statements were true based on what they had read. Analyses of students' writing revealed that those who read the printed chapter used a significantly greater proportion of borrowed sentences (quoted or paraphrased from the source material), while those who wrote from the web sources used a significantly greater proportion of transformed sentences (information from the source material combined in new ways or with added facts or claims). The researchers also found that students writing from the web sources used significantly "more connections and more causal connections in particular." Taken together, these findings suggest that accessing multiple source documents in a web-like environment may promote a deeper and more sophisticated understanding than presentation in a single printed source document. Additionally, the researchers noted that ...writing from multiple sources, in this case presented in a web-like environment, was especially beneficial to students who wrote arguments... This result suggests that in order for students to gain a deeper understanding of the subject matter, writing tasks must require knowledge-transforming and not just knowledge-telling. One way to achieve this...is to give students access to a variety of sources, and a specific argument writing task, that requires them to construct their own take on the information they read. A follow-up experiment using a single source document found "little difference" in student performance and writing based on "whether students read [the source document] off of a computer or paper." This result supports the conclusion that use of multiple sources is an important component in producing more in-depth learning. 2000 Research Report on the Effectiveness of Technology in Schools 77 Adaptive Testing Three studies indicate the potential of computer-based, adaptive testing in helping students to learn. Adaptive testing presents test items based on the individual student's prior achievement level and/or his or her ongoing performance during testing. Dalton and Goodrum (1991) compared the learning effects of three pre-testing conditions: adaptive testing, traditional full-length testing and no testing. The adaptive test would stop the student from continuing as soon as it determined that the student would not reach mastery on the test. The researchers reported significantly higher posttest scores for students who had received adaptive pre-testing. A yearlong project, Microcomputer Adaptive Testing High-Risk Urban Students (MATH-R-US) resulted in "consistent improvement" of students' math scores after completing weekly adaptive tests (Signer, 1992). The project was used...by a class in an urban high school with an at-risk predominantly black population and a high rate of absenteeism. The tests were designed to accept generative rather than multiple choice responses. The testing software was able to generate practice worksheets tailored to each student's diagnosed weaknesses. Powell (1992) studied the effects of three different methods of computerized testing: selecting items to match students' prior achievement level (i.e., adaptive); selecting items at random; or selecting them based on student choice of difficulty level. It was found that adaptive testing "required significantly fewer items to reach decisions than did the random-selection tests." Furthermore, students who measured high in anxiety during testing performed significantly better when taking the adaptive test. These studies suggest that adaptive testing is more efficient than traditional testing (requiring exposure to fewer items to reach decisions), is highly appropriate for students who suffer from test anxiety and may result in greater student achievement. Instructional Decisions and Student Achievement Research cited thus far strongly suggests that differences in teachers and differences in the learning environments they establish impact highly on the effectiveness of educational technology. Several recent studies have highlighted the importance of teacher decisions about student grouping and providing guidance on working collaboratively; the amount of time spent on the computer; the use of self-regulatory teaching strategies; the choice of an exploratory or confirmatory instructional approach with computer simulations; the use of grades as incentives; preparatory training prior to student use of software tools; task structure when students search the Internet; and the choice of computer or paper and pencil medium for administering assessment. Student Grouping In a meta-analysis of 103 studies, Lou, Abrami and Muni (1998) compared achievement of students using computers individually and in groups of two to five. (The studies included both 2000 Research Report on the Effectiveness of Technology in Schools 78 school-based instruction and professional training.) The research team reported a range of results depending on a variety of study factors. They found that Group learning effects on individual achievement tend to be larger: • When group size is as small as two members [effect size of 0.18], • When students have group work experience or are provided with training or other support [effect size of 0.31], • When working with drill-and-practice or tutorial programs [effect size of 0.19], • For relatively high or low ability students [effect sizes of 0.22 and 0.3, respectively] but not medium ability students, and • For female students [effect size of 0.50]. They concluded: Under these conditions, group members may have a higher frequency of peer interaction, appropriate strategy use, and more positive attitudes toward group work and toward learning with computer technology. Together, these processes and events may lead to higher motivation and better learning. Baron and Abrami (1992) compared the effects of three grouping strategies when using language arts tutorial software. They found no significant differences in achievement among upper elementary students working individually, in pairs or in groups of four. Similarly, Aguirre (1997) found no significant differences in achievement among fourth, fifth, and sixth grade students who were learning the location of states based on whether they practiced with game-type drill and practice software individually, cooperatively in groups of four that did not compete against each other, or in groups of four that did compete against each other. However, Hooper (1992) had different results when exploring the effects of computer-based mathematics instruction using various grouping strategies. One group of fifth- and sixth-graders received training in how to learn cooperatively and then worked in pairs and groups of four at the computer. Another group worked individually on computers. The students who had worked cooperatively scored significantly higher on a posttest that included factual, application, generalization and problem-solving questions. Mevarech had similar results in a comparison of pairs and individual, low-achieving third graders who used an integrated learning system as part of their mathematics instruction (1993). The pairs, who were instructed on how to collaborate at the computer, progressed at the same rate as high achievers in their class. However, low achievers who worked alone progressed at a slower rate than the high achievers. This learning advantage for cooperative groups trained in methods of cooperation was confirmed in subsequent research by Hooper, Temiyakarn and Williams (1993), who also found that cooperative groups were significantly more efficient learners. The cooperation-training students received in the latter three studies may explain the difference in results. Repman (1993) provided support for this in a study in which she compared the effectiveness of three approaches to collaborative, computer-based instruction with seventh grade social studies students. In one group, students worked without any guidance about collaborative learning process. In another group, students were given a sheet to guide their collaboration (structured condition); they were encouraged to assign roles while working together and to take turns at each role. In the third group, students received the same guidance sheet but also underwent training in collaborative learning before beginning computer-based instruction (training condition). Students in the 2000 Research Report on the Effectiveness of Technology in Schools 79 structured collaboration and collaboration after training conditions achieved at a significantly higher level than students who collaborated on computer without guidance did. Brush (1997) studied the effects of cooperative pairing on the performance of fifth grade students working in mathematics software in an Integrated Learning System (ILS) lab setting. He found that students working on the computer in randomly assigned pairs performed significantly better on a standardized posttest than students working individually. Cooperative grouping was also associated with improved student attitudes toward mathematics and toward the software and more positive behaviors while working on the computer. As in the other studies cited above, student training in the use of collaborative groups was an important element in the program implementation, with activities and materials developed to help students start their cooperative groups and understand their roles within those groups. Similarly, Shen (1997) reported positive results when undergraduate students in an introductory computer course were given guidelines for working cooperatively in a computer-based tutorial. Students completing the tutorial in groups of two or three were instructed to make sure they were in agreement before entering commands, to ask each other for help as needed, to summarize their learning to each other at the end of each lesson, to make sure they all understood the material before the end-of-lesson quizzes and to discuss questions until they agreed on the answers. Shen found that students using the tutorial cooperatively scored significantly higher on a test of computer science achievement than students using the tutorial individually. This result was attributed to the positive effects of peer interaction. Another approach to guiding student collaboration involves the use of embedded cues within the software program. Sherman and Klein (1994) compared two versions of a science software program for eighth grade students. One version included embedded cues ...designed to facilitate verbal interaction between two learners sharing one computer...[it] prompted individuals within each dyad to verbally interact by directing them to summarize, explain or listen to the other member... The other software version did not include the embedded cues. Student pairs who used the version with collaboration cues performed significantly better on a test of content knowledge and application. When working with the software, these students spent more time within the summary and explanation portion of the program and were observed to have "exhibited more summarizing behavior" than student pairs who used the non-cued version. The researchers suggest this as a likely explanation for the results. Researchers from Bar-Ilan University also found advantages to collaboration among sixth-graders that were studying math using computers in an Israeli school (Mevarech, Silber, and Fine, 1991). Students working in pairs demonstrated higher achievement than students working individually, as measured by both immediate and delayed posttests. Both high- and low-achieving sixth graders benefited from collaborating on the computer in heterogeneous pairs, as reported in a study by Temiyakarn and Hooper (1993). Students used either a learner-controlled or program-controlled version of tutorial software on ecology. On a delayed test requiring generalization of the content presented in the tutorial, high achievers who worked in pairs significantly outscored high achievers who worked by themselves. Low achievers who worked in pairs also demonstrated superior achievement to low achievers who worked alone. These results were confirmed in another study reported by Singhayanok and Hooper (1998) of sixth grade 2000 Research Report on the Effectiveness of Technology in Schools 80 students using the same software. They found that "both high and low achievers in the cooperative treatment performed better...than did students working individually, on both program-controlled and learner-controlled computer lessons." Additionally, they reported: The learner-controlled/cooperative learning group selected more options during the lesson, and spent more time interacting with the tutorial than did the high and low achievers in the learner-controlled/individual learning groups. Researchers at the Ontario Institute for Studies in Education (Scardamalia, Bereiter, Brett, Burtis, Calhoun, and Lea, 1992) compared independent and collaborative use of a networked communal database by upper elementary school students. ...[in the] Independent Research model...students primarily [conduct] independent research projects, but [do] so within the context of a community of researchers working on related problems...The main focus of classroom activity is on individual work. ...[in the] Collaborative Knowledge Building...model, the communal features of the database are the very basis upon which classroom activities are built. ...different students [are] assigned to contribute notes that will be linked into a common structure serving some broader goal than any single student is working toward. The collaborative model resulted in significantly higher achievement in the language and mathematics sections of a standardized test of basic skills. On end-of-unit reports, collaborative students were rated as demonstrating significantly greater depth of explanation and higher quality content knowledge. Collaborative students were also judged to ask more complex questions about content-related topics. In a study of first graders writing narratives on a word processor, Witherspoon (1996) found that working in dyads increased students' writing fluency. Both single-gender and mixed-gender dyads included a higher number of words and main clauses in their narratives than students writing individually, although spelling and narrative complexity remained the same. In his analysis of existing research on computer-based instruction (CBI) in small groups, Shlechter (1991) found no consistent achievement advantage for small group learning. However, he did find that small group CBI was significantly more instructionally efficient. In other words, less instructional time per student was required for small groups than for individuals. A study by Cockayne (1991) involving university students in a biology class, confirms this finding. Shlechter's analysis also indicated an advantage for lower-ability students working in heterogeneous groups without any disadvantage to higher-ability students. Simsek (1993), in a study focusing on homogeneous and heterogeneous pairs of upper elementary students using tutorial software, confirmed this finding. Shlechter warned, however, that heterogeneous grouping might only be best "for students with extreme differences in their abilities." Researchers at the Center for Children & Technology (Honey, McMillan, Tsikalas, and Griswald, 1995) offered evidence regarding optimum group size for collaborative learning with technology. They studied the Adventures in Supercomputing program, which 2000 Research Report on the Effectiveness of Technology in Schools 81 ...offers an opportunity for female students, students of color and economically disadvantaged students to engage in independent research and pursue novel problems of their own invention. The students learned FORTRAN and parallel programming techniques and were given Internet access to supercomputers, other software and data resources. As part of the program, professional scientists and engineers mentored students. Students worked alone or in project groups, but the size and composition of the groups varied. Student achievement was evaluated via a performance assessment. Videotapes of project presentations were rated according to students' demonstrated knowledge of their topic of inquiry, critical thinking, "clarity and coherence of presentation," teamwork and ability to "apply programming skills to analyze or investigate their area of interest." A cluster analysis of student scores resulted in three quality-of-performance clusters: the Integrated Knowledge cluster (high scores across all dimensions); the Procedural Knowledge cluster (average scores across all dimensions); and the Fragmented Knowledge cluster (below average scores across all dimensions). Students in the Integrated Knowledge cluster were most likely to belong to mixed-gender groups of three. Students in the Procedural Knowledge cluster were most likely to have worked alone. Students in the Fragmented Knowledge cluster were most likely to belong to mixed-gender groups of four or five. Quinn, Pena and McCune (1996) found significant differences in the performance of university students completing different versions of a simulation of an influenza epidemic--but found that the differences were impacted by decisions about grouping. One group of students worked individually while another group worked in pairs. The assigned task was to use the simulation to determine which combination of values for four variables (number of contacts per person per week, time to illness, duration of illness and length of the immune period) minimized the number of people who were ill during the week when the epidemic was at its peak. Students assigned to one version started with a simplified simulation (two variables) and determined what they felt were the optimal values for those variables, then repeated the process with three and four variables. Students assigned to the other version worked with all four variables from the beginning. Among students working individually, the researchers found that a small but significantly greater number reached the correct conclusion when the task was initially simplified (44 percent versus 40 percent). However, among students working in pairs, more than twice as many reached the correct conclusion when the simulation was presented with full initial complexity (79 percent versus 28 percent). The researchers concluded: This finding suggests that when a task is complex, individuals may benefit from working with a partner, but when a task is simple there may be no advantage to working with a partner. On the other hand, subjects who worked alone performed better when the task was presented in parts rather than at once. Time Spent on Computer Behar-Horenstein (1996) correlated the performance gains of Chapter 1 elementary students working in an Integrated Learning System (ILS) to a variety of factors, including gender, socioeconomic status, race/ethnicity, teacher longevity and teacher experience in the program. Performance gains were measured by comparing standardized test scores in reading and mathematics with those from the previous year. The study found that reading gains were correlated to time on the ILS, teacher experience in the program and race/ethnicity, while mathematics gains were correlated only to time on the ILS. It is noteworthy that minority 2000 Research Report on the Effectiveness of Technology in Schools 82 students, who achieved lower learning gains in reading, also spent less time on the ILS and had teachers who were less experienced in the Chapter 1 program. The researcher concluded that Computer time in reading and mathematics was found to be significantly correlated with posttest achievement scores, suggesting that the more time students spent learning by the ILS, the better they performed on the CAT posttests. Self-Regulatory Teaching Strategies Elliott and Hall (1997) discovered an advantage in using a self-regulatory teaching approach with at-risk preschoolers completing computer-based math activities. Teachers working with one group of students were directed to use the teaching strategies they felt were most effective with at-risk children, typically involving "direct guidance with minimal teacher involvement other than that of an encouraging, managerial or confirmatory nature." Another group used a self-regulatory approach, including "strategies such as goal identification, active monitoring, [modeling], questioning, reflecting, peer tutoring, discussion and reasoning." This self-regulatory approach also featured greater use of cues and questions, together with modeling and demonstration of skills and strategies. The researchers found that the self-regulatory group achieved significantly better scores on the mathematical achievement posttest. Exploratory vs. Confirmatory Approach to Computer Simulation Windschitl and Andre (1998) compared two instructional approaches to use of a computer simulation: a confirmatory approach, in which students use the simulation "in a prescribed fashion to simply confirm information as directed," and an exploratory approach, in which students must develop their own strategies for using the simulation to solve problems. Non-biology majors enrolled in a human anatomy and physiology survey course spent about seven hours using a simulation that "modeled the functioning of the human cardiovascular system" to "resolve 12 cases that were designed to generate...evidence [for refuting]...several commonly held misconceptions." Students in the confirmatory group followed detailed step-by-step directions in their use of the simulation, while students in the exploratory group were shown an example of how to use the simulation to test a simple hypothesis and given a general set of guidelines, but otherwise left on their own to resolve the assigned cases. The researchers identified six questions on the pretest that were missed by at least 70 out of 221 students and that reflected a basic misconception regarding the functioning of the human cardiovascular system. Looking only at students who had missed these questions on the pretest, they found that students in the exploratory group performed significantly better on two of the six corresponding posttest questions. This suggests that in some cases, "an exploratory (constructivist) simulation experience could be more effective in altering learners' misconceptions than a confirmatory simulation experience." Windschitl and Andre also conducted a survey to determine students' "beliefs about the nature of knowledge and learning." They found a significant interaction between the relative sophistication of students' beliefs and which instructional approach led to the best results. In the exploratory group, more sophisticated ideas about learning and knowledge (such as a belief that learning may require multiple exposures to information and that problems have more than one answer) were related to higher posttest performance. In the confirmatory group, students with more sophisticated beliefs performed more poorly than those with less sophisticated beliefs. These findings suggest that for best instructional results, choice of instructional approach in using simulations 2000 Research Report on the Effectiveness of Technology in Schools 83 should take into consideration both learner characteristics and the type of learning outcomes desired. Grading and Incentive Research at the University of Memphis (Morrison et al., 1995) suggests that student performance on computer-based instruction (CBI) tasks may be related to how the performance is figured into course grade calculations. One group of undergraduate teacher education students worked on the CBI as a required activity under "performance incentives "where their level of performance would be considered in the calculation of course grades. The other group worked on the CBI as an extra credit assignment under "task incentives" where mere participation would earn the extra credit, but "their level of performance would have no bearing on the amount of credit received." Students working under performance incentives significantly outscored those working under task incentives on a test of the concepts and principles include in the CBI. Preparatory Training in the Use of Tool Software Ennis found an advantage in preparing students for the use of database software with search strategy training (Ennis, 1993). One group of fourth-graders was given search strategy instruction "how to use Boolean operators (AND, OR, NOT) in an organized way to manipulate information in the database." Another group of students received no such training. Students in both groups were given "real world" problems to solve using the database. While students in both groups gave correct answers to the problems, students who had received the preparatory search strategy training took less time to complete the problems. Task Structure when Students Search the Internet Schacter, Chung and Dorr (1998) reported on the effects of assigning different types of Internet search tasks to fifth and sixth grade students. Students were assigned two search tasks, one narrowly defined ("Find information about the three types of crime that happen most in California"), the other broadly defined ("Find at least three pieces of information on the Internet that will help you develop a plan to reduce crime in California"). They found that students performed significantly better in identifying web sites that fulfilled the broad goal than finding sites that helped meet the narrow goal, even though they conducted more searches for the narrowly defined task. Additionally, students' self-ratings of the quality and usefulness of the documents they had bookmarked were significantly higher than experts' ratings of those documents for the narrower task, but there was no significant difference in the ratings for the broader task--suggesting that students were better able to evaluate the usefulness of web sites they had located for the more broadly defined task. These findings suggest that students in the upper elementary grades are likely to be more successful in conducting Internet searches when the goal is broadly defined. Medium for Administration of Assessment Russell and Haney (1997) investigated the impact of computer versus paper and pencil test administration on student performance at a school where students were used to writing on the computer. Sixth, seventh, and eighth graders completed a test comprised of National Assessment of Educational Progress (NAEP) multiple choice and brief response items covering language arts, science, and math, and "a performance writing assessment which required an extended written 2000 Research Report on the Effectiveness of Technology in Schools 84 response" to a prompt. Both tests were administered on computer to one group of students and using paper and pencil to another group. The researchers found that there was no significant difference between the two groups in their performance on the NAEP multiple choice items. However, on both the performance writing assessment and the NAEP short answer items for language arts and science, students who wrote on the computer scored significantly higher, with an effect size of .94 on the performance writing assessment. Taken together, these results suggest that while multiple choice examinations may yield the same results whether administered on computer or using paper and pencil, for students accustomed to writing on computer for only a year or two...estimates of student writing abilities based on responses written by hand may be substantial underestimates of their abilities to write when using a computer. Conclusion Recent research consistently demonstrates the value of technology in enhancing student achievement. While several researchers have attempted to quantify its achievement effects in isolation, actual use of educational technology does not and should not occur in isolation. It is the decisions made by well-trained, professional educators that will determine the computer's ultimate instructional effectiveness. The research community has begun to provide information useful in addressing many of these issues. The research reviewed in this section can inform software developers and publishers as they design software and can help educators as they attempt to incorporate technology-based learning experiences into the curriculum. Developers and publishers of software can use the software design characteristics findings to make careful decisions on how these characteristics all contribute to student achievement. The software design characteristics include: • • • • • • • • • • • • • • • Type of software Level of learner control Number of practice items Informativeness of instructional feedback Use of objectives and advance organizers as orienting features; embedding instructional and conceptual change strategies Ways in which pedagogical agents communicate Instructional scaffolding of learner support Support structures and contexts used for foreign language learning Inclusion of still and/or animated graphics Combining of animation, narration and other sounds Multiple representations of concepts Incorporation of dynamic visualization models Relating video directly to the concepts being taught Navigational techniques Motivational contexts for instruction 2000 Research Report on the Effectiveness of Technology in Schools 85 • • Window presentation styles Ergonomic combination of design features Educators can learn how technology helps increase student achievement in a variety of subject areas and which design characteristics to consider when selecting software. Both educators and publishers can also benefit from the research on: • • • • • • • • The positive effects of grouping students for cooperative learning with technology Providing adequate time on task Incorporating performance incentives Adding self-regulatory teaching strategies Choosing appropriate instructional strategies with simulations Adequately preparing students prior to their use of tool software Structuring Internet search tasks to match the level of students Using the computer for writing assessments when students are used to writing on the computer 2000 Research Report on the Effectiveness of Technology in Schools 86 Section 2: Effects of Technology on Student Selfconcept and Attitude About Learning Research confirms the potential of educational technology to improve students' attitudes about themselves and about learning. The results of several studies indicate that technology has beneficial effects on student self-concept (DeGraw, 1992). In addition, a review of the literature finds positive effects on attitudes toward language arts, mathematics, science and social studies (Beyer, April 1992). This section of the report features studies of educational technology that address student selfconcept and student attitudes toward specific curriculum areas and reviews research that relates changes in student attitudes to software design characteristics, specific technologies and specific learner characteristics. Effects on Student Self-concept Three studies provide evidence of the positive impact of educational technology on student selfconcept. Rhyser (1990) examined the effects of integrating computer-based instruction (CBI) in an urban elementary school. Students receiving CBI expressed stronger "feelings of success in school" than students in an equivalent school without CBI did. Such feelings are an important component of a positive self-concept. DeGraw (1990) found that fourth graders grew in self-esteem and self-confidence when computers were placed in their homes and their school, as part of the Buddy System Project. Group of students received classroom instruction plus computer-assisted instruction (CAI); the other group received classroom instruction only. (Both groups received the same total amount of instructional time.) The classroom instruction-plus-CAI approach resulted in significantly higher gains on a measure of self-concept of academic ability than the classroom instruction-only method. The difference in improvement of self-concept was especially significant for students of low socioeconomic status. Previous research suggests that interaction with educational technology may lead to improved selfconcept because: (1) successful experiences with technology give students a feeling of control over their own learning (Fisher, 1986); and (2) such experiences may increase students' sense of confidence in their abilities to perform in specific learning situations (Graham and MacArthur, 1988). 2000 Research Report on the Effectiveness of Technology in Schools 87 All of these studies point to the potential of educational technology to help develop student self-confidence. Students who view themselves as successful are more likely to enjoy school and to put forward their best efforts. Curriculum Areas and Student Attitudes Several research efforts have provided support for the effects of technology on positive student attitudes in a variety of curriculum areas. Language Arts Six research reports documented improvements in students' attitudes when they used technology as part of the language arts curriculum. In their review of computers and basic writing instruction, Valeri-Gold and Deming (1991) found that "numerous studies have reported improved student attitudes toward writing while composing on the computer." In a study by Green, (1991) inner-city third graders demonstrated significantly greater improvement in attitude toward writing after experiencing a writing process approach with word processing, compared with a similar approach without word processing or a grammar-oriented approach to writing. Owston and others (1990) compared eighth graders writing on and off computer. When working on the computer, students wrote significantly longer pieces and their attitudes toward writing and editing were significantly more positive. Beyer (1992) found a similar improvement in attitude toward writing among middle school students who experienced a process approach to writing that included regular word processing. Newbold (1993) compared the effects of CD-ROM and print encyclopedias on the attitudes of sixth graders. She found those students using the CD-ROM encyclopedia were significantly more positive toward writing and toward using the library. After students were given experience with both types of encyclopedias, all students were "more positive toward using electronic encyclopedias as compared to print encyclopedias." Anderson-Inman (1990) reported on two studies in which keyboarding was found to be a highly motivating method of practicing spelling for low performing students (McClendon, 1989; Hollen, 1987). As a result, students developed "a positive attitude toward spelling practice." These six studies strongly suggest that integrating computers into the language arts curriculum will help improve student attitudes toward writing and toward spelling practice. Mathematics Four studies suggest that technology can have a positive impact on students' attitude toward mathematics. Webster (1990) found that black fifth graders in rural Mississippi who received supplemental CAI evidenced significantly more positive attitudes toward math than similar students who did not receive CAI. A study by the Cognition and Technology Group at Vanderbilt University (1992) assessed improvement in elementary school students' attitudes toward mathematics as a result of 2000 Research Report on the Effectiveness of Technology in Schools 88 experiencing The Adventures of Jasper Woodbury, a series of video vignettes designed to stimulate mathematical problem-solving. Compared with a group of students who received traditional classroom instruction, the Jasper students ...showed [significantly] less anxiety toward mathematics, were more likely to see mathematics as relevant to everyday life, more likely to see it as useful, and more likely to appreciate complex challenges. A follow-up study at Vanderbilt (Barron et al., 1995) suggests that supplementing Jasper videos with technology-based thinking tools and follow-up activities further improve students' attitudes toward learning. After viewing a Jasper episode on business planning, students who used the supplementary tools demonstrated significantly greater positive changes in their interest and confidence about business planning than Jasper-only students did. The Jasper-plus-supplements students also felt significantly more confident about applying what they learned to business planning problems they had never encountered before. Yusuf (1995; 1991) compared the effects of Logo-based mathematics instruction with traditional mathematics instruction for students in a predominantly black, urban, middle school. Students receiving the Logo-based instruction demonstrated significantly more positive attitudes toward geometry and toward mathematics in general. Science Seven studies suggest the beneficial effects of technology on student attitudes toward the study of science. Koszalska (1999) reported that use of Internet resources had a significant positive impact on middle school students' perception of science. Teachers were asked how often they had their students use the Internet to contact or receive information from "scientists, guests, or other teachers" during science lessons; use Internet-based science equipment or tools; or work with "Internet resources such as text and pictures." Students (both boys and girls) in classrooms that used Internet resources at least five times a year scored significantly higher on perception of science. Additionally, girls in classes where the Internet was used at least five times a year had significantly greater interest in a science career. Koszalska also found that "in science classrooms where teachers did not have science degrees, the use of web resources was predictive of higher student science career interest." Speculating on the significance of this finding, the researcher suggested: Teachers who did not have science teaching degrees, may not have been as knowledgeable of complex science content areas or as prepared to teach science topics as teachers with science teaching degrees. Using web resources might have provided these teachers with ready-made and tested science lesson plans, on-line and off-line activities, simulations and models to explain complex science concepts, and access to more supportive information through thousands of sites that could be used to teach science and promote student interest in science. Yalçinalp et al. (1995) compared the attitude of students using a computer-assisted instruction (CAI) program with students receiving the same content through lecture and discussion. Eighth grade students received supplementary instruction on chemical formulas and the mole concept 2000 Research Report on the Effectiveness of Technology in Schools 89 through eight hours of CAI or teacher recitation over a four-week period. The CAI program incorporated presentation of concepts, examples and problem-solving exercises. Researchers found that the "classroom instruction supplemented with CAI produced significantly more positive attitudes toward chemistry." Bissell and Simpson (1993) found that a "thematic, inquiry-oriented," computer- and videodisc-enhanced science curriculum had positive effects on middle school students' attitudes about science. The curriculum software included the Rediscover Science integrated learning system software (Edunetics Corporation) and the life, physical and earth science videodiscs in Windows on Science and Living Textbook (Optical Data). The curriculum was designed to enable "teachers to be mediators rather than givers of information." Compared with students who received traditional science instruction, those who experienced this experimental approach more strongly agreed with the following statements. We do fun things in science. We learn important things in science. I would like being a scientist. I would like to take more science classes. The differences between the two groups' responses were statistically significant. Geban et al. (1992) compared the effects of computer-simulated experiments and two other approaches with laboratory work on high school students' school attitudes toward chemistry. One group of students engaged in computer-simulated experiments in which they were guided through the process of hypothesis development, data collection, data analysis and drawing conclusions about their original hypotheses. Another group of students participated in conventional laboratory activities. A third group participated in hands-on laboratory activities that stressed hypothesis testing and problem-solving. The computer-simulated method resulted in "significantly more positive attitudes toward chemistry than the other two methods, with the conventional approach being the least effective." MacArthur and Haynes (unpublished) found that learning disabled students preferred studying science topics using a computer-based Student Assistant for Learning from Text (SALT) that included synthesized speech and that took advantage of the computer's hypermedia capabilities (e.g., an online glossary, links between questions and text, supplementary explanations). The students favored this "enhanced" version of SALT over a simpler version that more closely resembled a basic biology textbook and "thought that [the enhanced version] helped them learn the material better." An attitude that science educators typically seek to foster in students is curiosity. Brusic (1991) explored the effect of incorporating technology-based science activities on the curiosity of fifth grade students. Students receiving instruction that included technology-based activities measured higher in curiosity than students who received only traditional science instruction ("primarily teacher demonstrations of science experiments"). The TERC Star Schools science and math project (Weir, 1992) has been shown to positively affect student attitudes about tackling unknown questions—one aspect of curiosity. The Star Schools project combined network communication among teachers, innovative curriculum units, specially designed software and teacher training. Over 900 teachers were involved in the project. A sample of 80 students was interviewed before and after involvement in the Start Schools curriculum regarding attitudes about tackling known and unknown questions. Before beginning the Star Schools curriculum, about 80 percent of the students favored known questions, but after the Star 2000 Research Report on the Effectiveness of Technology in Schools 90 Schools treatment, only 64 percent favored such questions. This change in attitude was found to be statistically significant. Although it is possible to develop interactive science instruction without technology, programs that make use of technology resources such as the Internet and computersimulated experiments have improved students' attitudes toward and increase student curiosity about science. Social Studies A study by Yang (1991-1992) suggests that a social studies computer simulation of presidential decision making can have a positive effect on student motivation. Eleventh graders who had experienced the simulation scored significantly higher on a measure of motivation than students receiving traditional, print-based instruction. Ferretti and Okolo (1997) found that sixth grade students who created multimedia presentations on the Spanish colonization of Latin America scored significantly higher in selfefficacy (i.e., self-confidence in their ability to achieve in "social studies in general and the particular topic that they were studying") and academic intrinsic motivation than those who completed a traditional textbook-based unit on the same topic. Comparison of pretest and posttest scores on an attitude scale showed that self-efficacy increased and academic intrinsic motivation stayed about the same for students who created multimedia presentations. In contrast, both self-efficacy and academic intrinsic motivation actually decreased for students who completed the textbook-based unit. These studies in social studies, plus those cited above for language arts, mathematics and science provide a sense of the wide variety of software types that can positively affect student attitudes. Well-designed tutorial and practice and "enhanced" hyper-textbooks can make challenging concepts and principles easier to understand. For example, students who are visual learners can benefit from still and motion graphics and video presentations included in instructional software. Tool software—software that makes it possible to accomplish a task more easily or effectively (e.g., a word processor or spreadsheet package)—can foster creativity and curiosity and can make the task easier to accomplish. For instance, revising an essay on computer means working on just the parts the student wants to change, whereas revising without a computer requires rewriting the entire essay. Simulation software can offer students highly interactive, intrinsically rewarding experiences that textbooks cannot provide. For example, technology can allow students to role-play the president of the United States, an 18th century American pioneer or an international detective. Software Design Characteristics and Student Attitudes Several studies focused on software design characteristics and their impact on student attitudes, especially the issue of learner control—the degree to which software lets students determine how they will learn—and the inclusion of static or animated graphics. Two studies focused on the design of foreign language software, another study addressed software design characteristics found to be most motivating by gifted students and another study compared different types of feedback in computer-based testing. 2000 Research Report on the Effectiveness of Technology in Schools 91 Learner Control Four studies found support for greater student control over the learning environment. Kinzie, Sullivan and Berdel (1992) compared two groups of ninth graders who received different versions of computer-assisted instruction in science. One group worked on a version in which the program automatically determined when to review instructional material. The other group used a CAI version in which students had control over review materials. The students' decisions about which version to use during a subsequent session indicated significantly higher continuing motivation for the learner-controlled version. Hannafin and Sullivan (1995) found a more positive attitude among high school students who used learner-controlled versions of an introductory geometry software program than students who used program-controlled versions of the same program. The learner-controlled versions allowed students to add or bypass example, practice and review screens. In the program-controlled versions, the software and not the student made these decisions. Shyu and Brown (1993) considered the attitudinal effects of an interactive video under two different learner control conditions. The video demonstrated and explained, step-by-step, how to make an origami crane. One group of college students went through the steps in a fixed sequence, with the option to replay the current step as many times as desired before going on to the next step. The other group had total control over instructional sequence but was advised as to the preferred sequence of steps. Students using the version with total learner control plus advisement had significantly more positive attitudes toward instruction and felt more confident. Tool software typically provides a dramatically greater degree of learner control than does tutorial software. Wood (1991) compared the effects of tool and tutorial software on high school students' attitudes toward mathematics. Each software application was integrated with classroom instruction. Students using the tool software scored significantly higher on a measure of positive mathematics attitudes. In these research efforts, computer programs offering students greater control over their learning environments were found to have beneficial effects on student attitudes. Static and Animated Graphics Three studies found evidence for the importance of static and animated graphics as a factor in improving student attitudes. Szabo and Poohkay (1996) researched math education students' attitudes toward instructional presentations consisting of text plus static or animated graphic illustrations, as compared to textonly presentations. Students learned how to construct a triangle using a compass by reading text, reading text and viewing static illustrations or reading text and viewing animated graphics. Students using either the text-plus-static graphics or the text-plus-animated graphics version reported a more positive attitude than those who viewed the text-only version. According to research by Rieber (1994), a computer simulation of motion concepts is less frustrating when feedback is visual rather than text-based. College students used a simulation of the relationship between acceleration and velocity, in which they had direct control over the acceleration of an object. The goal was to have the object land in a certain position, within a 2000 Research Report on the Effectiveness of Technology in Schools 92 specified range. Each student experienced six trials of a version of the simulation that represented the object's current position visually—as moving along a number line. Each student also experienced six trials in which the object's position was reported as a number – corresponding to a location on the number line. The trials alternated between the two versions of the simulation. After each trial, the students rated their frustration level. Students indicated a significantly lower level of frustration when using the visual feedback version. Almost 85 percent of the students "reported that they preferred the simulation with visual feedback." Researchers in Taiwan (ChanLin and Chan, 1996; ChanLin, 1996) compared student motivation using different versions of a computer-assisted lesson on recombinant DNA technology. Versions of the lesson included animated graphics, static graphics or no graphics at all, with or without the use of metaphors to enhance graphical information. The researchers found that students using the version including both metaphors and animated graphics were significantly more motivated than students using all other versions of the program. Text in Foreign Language Software Jakobsdóttir and Hooper (1995) found that providing text together with spoken words improved student ratings of a computer-based foreign language lesson. Fifth grade students responding to spoken commands in Norwegian felt significantly more confident in their answers and rated the instruction as more relevant when the text of the commands was displayed simultaneously with the spoken commands. Borras and Lafayette found that adding subtitles to video in foreign language software had a positive impact on student motivation (1994). College students taking French language courses who experienced video-with-subtitles software had significantly more positive attitudes toward French speaking practice than students who used the same software without subtitles did. These findings may be applicable to software designed for English as a second language students, as well. Designing for Gifted Students Burt (1993) completed survey research on the software design characteristics preferred by gifted elementary school students. Her findings suggest that gifted students are motivated by computer programs that: • • • • • • Offer students a sense of control over the instructional activity Arouse curiosity Use multiple, appropriate levels of difficulty to provide a sense of challenge Provide feedback that builds the user's self-esteem Include an element of fantasy (i.e., make-believe situations that offer the "potential for taking on roles impossible in actual reality") Have learning activities that incorporate game formats Experience suggests that these design characteristics have motivational appeal for students across the ability range. 2000 Research Report on the Effectiveness of Technology in Schools 93 Feedback Variations in Computer-Based Testing Buzhardt and Semb (1998) found a clear preference among university students for unit quizzes with one of three variations in feedback design. After completing an Introduction to Child Development and Behavior course during which they had taken quizzes with all three designs, 61 percent preferred feedback administered item-by-item with the option to skip items and return to them, 23 percent preferred end-of-test feedback and only 11 percent preferred item-by-item feedback without the ability to skip test items. However, no significant differences in final exam performance were related to the three quiz variants. Recent Technologies and Student Attitudes Research indicates that computer-based telecommunication, videodisc, CD-ROM and adaptive testing can positively affect student attitudes. There is also preliminary research suggesting the motivational appeal of virtual reality. Telecommunication Spaulding and Lake (1992) studied the impact on remedial writers of a telecommunication network project in which they wrote collaboratively with students across the United States and Europe. The researchers found that "students were more motivated to write when the projects were designed by other students than when similar projects were designed by the classroom teacher." A telecommunication project featuring collaboration between students in New York State and students in Moscow city schools (MAGI Educational Services, Inc., 1992) had a positive effect on student interest in international issues and current events. After participating in "studentgenerated projects such as surveys, polls, articles, newspapers, research, analysis and creative writing," students involved in the telecommunication project were found to spend significantly more time than non-project students discussing political or social issues, discussing international events, reading news magazines at home and reading books by foreign authors not assigned by their teachers. As described under Curriculum Areas and Student Attitudes, Koszalska (1999) reported that middle school students' perception of science was significantly more positive in classrooms that used Internet resources at least five times a year in their science teaching. Use of Internet resources was also a significant predictor of interest in science careers for girls. Additionally, the researcher found that use of human resources in science teaching (contact with "scientists, guests, or other teachers" at least five times a year, either through classroom visits or via the Internet) significantly correlated to interest in a science career for both boys and girls. Girls, in particular, had "much higher scores" for science career interest in classrooms where both human and Internet resources were used than classrooms where either but not both were used. These findings suggest that use of the Internet may have especially positive effects on the attitudes of middle school students-especially girls--toward science when teachers take advantage of its capacity for social contacts with members of the scientific community. Chiu (1996) found that tenth grade students in Taiwan utilizing network resources to complete a written science report demonstrated significantly more positive attitudes toward both school science and the use of computers than students given the same assignment but who did not use the network. One group of students writing reports on weather were encouraged 2000 Research Report on the Effectiveness of Technology in Schools 94 to use online resources as background information for reports and to distribute questions, interviews, surveys or "looking for help" messages through a local area network, which also provided access to the Internet. Additionally, report topics, investigating plans, report drafts and final reports were submitted to the teacher via e-mail and were recommended to be posted to related electronic bulletin boards on earth science topics. By contrast, students in the comparison group did not use the network as a resource in creating their reports. According to the researcher, Apparently, networks provided the ability for students in this study to participate in a real functional environment. The students could get outside the school boundaries and shared data and discussed real-life issues with others around the world simply with the help of the network technology. The learning environment changed and so did the students' impression of learning. Agarwal and Day (1998) found that students who used the Internet as a resource in a graduate microeconomics course improved their attitude toward economics significantly more than students in another course section taught by the same professor who did not use the Internet. Students in the Internet class completed web-based projects and received answers to questions about course material via a class e-mail list, while students in the control class completed similar project assignments without reference to the Internet. On pretest and posttest administrations of the Attitudes toward Economics questionnaire developed by the National Council of Economic Education, students in the Internet course "expressed a significantly higher likelihood of attending a lecture given by an economist, were more likely to consider economics as their favorite subject, used economic concepts to analyze situations more frequently, and disagreed about finding economics dull." They also rated the instructor significantly higher on the standard university evaluation form than students in the control class. Smith (1992) investigated university students' attitudes after completing courses that took advantage of a computer-based telecommunication network. The purpose of the network was to make it easy for students to submit assignments and to communicate with their instructors and other students. A comparison group of students completed equivalent courses without the benefits of the network. Courses that incorporated the network ...received higher overall evaluations and higher instructor ratings than did their traditional course counterparts. Nearly three out of every four students in the [network] group rated the response time, ease in doing assignments, quantity of feedback, quality of the course and their overall experience as better or much better than past courses [without] online computer interaction. Matthew, Barufaldi and Bethel (1998) reported on the impact of electronic networking during preservice elementary teachers' three-week science teaching methods class and subsequent student teaching practicum. During the methods class, students were required to post reflections twice a week on a class news group, respond once to another student's message, access materials on science teaching over the World Wide Web, and "post regularly over a listserv/mailing list set up specifically for preservice teachers called PreSTO (Preservice and Student Teachers Online)." Additionally, three science experts interacted with preservice teachers electronically. During the 12-week practicum, students were required to "make five reflective postings based on their experiences teaching science in their classrooms" and respond at least twice to other students' posts. In contrast, students in the control group 2000 Research Report on the Effectiveness of Technology in Schools 95 ...could interact with each other during the course of the class time. They were required to reflect each day as well, but the reflection was in the traditional paper and pencil format, and was due at the end of each week. In addition, they used mostly the library to gather information... Not all students got a chance to see everybody else's work. Mathew et al. found that the preservice teachers who used networking increased significantly more than the control group in both their "belief...in their ability to affect student performance in science" and their "confidence...in their own abilities to teach science effectively." LaMaster (1996) studied e-mail communications among three groups of preservice physical education teachers. These university students used e-mail to post questions relating to their teaching and to respond to each other's questions. Typically, students described how they had taught a lesson and asked for alternative suggestions from their peers. Other queries related to student behavior, discipline and management. As a result of peer feedback, these teachers-intraining were able to make changes to their upcoming teaching plans. They also experienced a significant increase in their perceived self-efficacy toward using e-mail and "indicated that they would continue to use e-mail...throughout their professional lives." Schutte (1996) studied the effects on students in a university social statistics course who were randomly divided into two groups, one that met with the instructor each Saturday and another that completed the course through telecommunication as a distance learning experience. Students in the distance learning group were assigned to engage in a variety of collaborative assignments and electronic discussions. The only time they met was to complete the midterm and the final. Students in the distance learning group expressed frustration at not being able to ask questions of the professor in a face-to-face environment, but in a posttest questionnaire, indicated that they had communicated more with fellow students. And, although they perceived they spent significantly more time on class work, they were also more likely to think they had more flexibility, a greater understanding of the materials and more positive affect toward math, in the end, than did the traditional class. These nine studies indicate that the use of computer-based telecommunication helps to improve students' attitudes toward learning. Telecommunication can serve as a motivating focal point of instruction (e.g., as a vehicle for student writing), as an additional resource for student assignments, as a support system for traditional instruction or as a medium for developing learning communities and encouraging interaction among students. Video Six studies suggest the positive effect of videodiscs and other video-based technology on student attitudes. Thorkildsen and Lowry (1990) found that elementary students using a videodisc on mathematics core concepts had significantly higher gains in math self-concept than students receiving conventional instructional. In a study by Castelluccio (1994), traditional lecture-based instruction was compared with a HyperCard stack that combined an interactive videodisc on plate tectonics with teacher-prepared 2000 Research Report on the Effectiveness of Technology in Schools 96 text. Seventh graders who received the videodisc-plus-text instruction demonstrated significantly more positive attitudes toward learning the science content. Niedelman (1991) compared an earth science videodisc with textbook-based instruction plus hands-on laboratory activities. Junior high students receiving the videodisc instruction were significantly more confident in their science problem-solving skills, more interested in taking science classes and more positive about their learning experience overall. Bain, Houghton, Sah and Carroll (1992) found that elementary and junior high school students receiving teacher-led, video-based instructions were significantly more positive about their instruction than students receiving teacher-led lessons without video. Viesulas (1994) compared college students' attitudes when viewing demonstrations of standard chemistry experiments via video and via live lecture presentation. Students received either the video-based presentation or the live lecture before performing the experiments themselves. Although 85 percent of the students who watched the video demonstrations indicated that they "enjoyed the lab and felt they learned a lot," only 14 percent of the students who received the lecture felt that way. Vitale and Romance (1992) found that videodisc instruction plus supplementary activities can have a positive effect on the attitudes of college students preparing to become teachers. One group of students received videodisc-based lessons, completed corresponding workbook activities and prepared and presented model science lessons. The other group followed the same syllabus but received only conventional science methods instruction. The students who worked with the videodisc and participated in the supplementary activities demonstrated a significantly higher degree of confidence in their knowledge of science concepts. These studies suggest that videodisc can have an advantage over conventional teaching methods in improving student attitudes toward instruction and toward the subject being taught. Adaptive Testing Dalton and Goodrum (1991) found that adaptive pre-testing on a computer (that halted testing as soon as "nonmastery was indicated") resulted in higher levels of motivation to continue instruction among fifth and sixth graders than traditional, full-length pre-testing. Students were more motivated by computer-based, adaptive testing than traditional testing because adaptive testing did not subject them to unnecessarily long periods of failure. Virtual Reality Preliminary survey research by Bricken and Byrne (1992) suggests the power of virtual reality (VR) to motivate students. Children of middle school age attending a technology summer camp had the opportunity to use, design and implement VR environments. When given a choice between a VR experience and other familiar technology-based activities, students almost always preferred VR to using a favorite computer program on screen, to watching television and to playing video games. 2000 Research Report on the Effectiveness of Technology in Schools 97 Special Populations and Student Attitudes Early Childhood Education Two studies suggest that computers can have a positive effect on young children's attitudes toward basic academic skills and on developing their self-concept. Chang and Osguthorpe (1990) found evidence that kindergartners who used a picture-word processor while working with trained older students (fourth- and fifth-graders) improved their attitudes toward reading. Based on a survey, it was found that these young children had significantly better attitudes about reading than an equivalent group of children who had not used computers. Goldmacher and Lawrence (1992) studied two groups of preschool children attending a Head Start program. One group followed the standard Head Start program. The other participated in computer enrichment activities, in addition to standard Head Start activities. The computer-based activities were theme-based and were built around an assortment of software. Students in the computer group exhibited significantly more behaviors indicative of positive self-concept than did students in the non-computer group. In both of these studies, students using computers in early childhood education programs demonstrated more positive attitudes than children who did not have access to computers. Students with Limited English Proficiency Researchers from the University of California at Irvine (Barrutia, Bissell, Rodriguez and Scarcella, 1993) found that science videodiscs with both English and Spanish soundtracks can have a beneficial effect on the attitudes of Spanish-speaking middle school students with limited English proficiency (LEP). Compared with students in conventional classrooms, those who used the bilingual videodiscs as part of instruction over a three-year period indicated significantly stronger agreement with statements such as: "I like my science class"; "We do fun things in science"; "My friends like science"; and "I would like to take more science." Special Needs Students Several researchers have explored the effects of technology-based instruction on the attitudes of students with special needs. Motivating emotionally disturbed students to remain on task is typically a challenge for special educators. However, Bair (1990) found that emotionally disturbed students in grades 6-12 spent more time writing and wrote longer compositions using a word processor than using pencil and paper. As part of a comprehensive evaluation of computer-based instruction (mostly integrated learning systems) in New York City, Swan et al. (1990) determined that educationally disadvantaged students were more motivated and less threatened when learning on computers than when learning in regular classroom settings. Mevarech, Silber and Fine (1991) compared the math anxiety of a group of low-ability sixthgraders working with computers in pairs with an equivalent group working on computers 2000 Research Report on the Effectiveness of Technology in Schools 98 individually. The student pairs showed significantly lower levels of math anxiety than the students working individually did. Wepner (1991) reported the effects of different kinds of reading software on the attitudes of at-risk, inner city, middle school students. One group of students used drill software that focused on discrete reading skills and testing preparation software. The other group used software that practiced a complex of reading skills in the context of stories built around themes of likely interest to adolescents and young adults. The attitudes of the latter group toward themselves as readers and writers showed significantly greater improvement than the attitudes of the group receiving discrete skill practice. Researchers for California State University at Northridge (Murphy and Higgins, 1994) tracked the impact of a variety of compensatory technology tools used at the university, including optical scanning and character recognition, speech synthesis and speech recognition systems. They found that learning disabled (LD) students who had trained on the use of these technology tools had a dropout rate of only 1.4 percent over three years, compared to dropout rates of 34 percent during the same period for a matched group of LD students who had not received the training. Additionally, students who had trained on the technology tools showed significant improvements in self-concept, as determined by their scores in three out of five scales on the Dimensions of Self Concept: Identification vs. Alienation, Leadership and Initiative and Academic Interest and Satisfaction. From these studies, we can generalize that technology can be an effective motivation and can improve the attitudes of special needs students toward learning. Instructional Decisions and Student Attitudes Student Grouping Research suggests that decisions about how students are grouped when engaged in technologybased instruction can affect student attitudes. Researchers from the University of Minnesota (Hooper et al., 1993) found that elementary students using tutorial and drill software in cooperative groups had significantly better attitudes toward their computer lessons than students working independently. However, Arizona State University researchers (Crooks et al., 1995) found those college students using a computer-based tutorial in cooperative groups had significantly less positive attitudes toward their computer lessons than students working alone. Research by Repman (1993) may help to clear up the apparent discrepancy between these studies. She found that not all computer-based collaborative learning experiences are alike. Middle school students who received training in collaboration and then engaged in computer-based problemsolving demonstrated significantly higher self-esteem than students who received no collaboration training but solved problems on computer with the aid of printed guidelines on collaboration. Training in effective collaborative strategies was also an important factor in a study by Brush (1997; 1996). Activities and materials were developed to help fifth grade students form cooperative 2000 Research Report on the Effectiveness of Technology in Schools 99 pairs and understand their roles in working together on mathematics software. These students were much more positive in their attitudes toward both mathematics and the software: 91 percent of the students working in pairs reported that they enjoyed math and 94 percent said they liked the computer math lessons, compared to 57 percent and 43 percent, respectively, for students working individually. From direct observation of student work in the lab, the researcher noted further that Student behaviors appeared to be markedly different, and the enjoyment and excitement observed in the experimental classes was not as noticeable in the control class. These behavior differences should encourage educators to look beyond simple test scores in order to determine the full merits of this strategy. Yu (1998) compared the attitudes of fifth grade Taiwanese students completing computer-assisted science tutorials cooperatively in pairs with and without inter-group competition. During an initial session, students were trained on "essential cooperative learning techniques...appropriate for elementary school students..., [such as] (a) staying with your teammate, (b) sharing materials, (c) taking turns, (d) encouraging one another, and (e) speaking in quiet voices." Students in one group were encouraged to be "among the top three highest scoring teams in the class"; students in the other group were encouraged to reaching a "fixed standard of performance for excellence on the posttests" of 70%. Yu found that students who had completed the tutorials without competition among pairs had a significantly more positive attitude toward science. The researcher found this outcome particularly noteworthy since the study was conducted in Taiwan, "which [is] so accustomed to a competitive goal structuring method." Repman and Lan (1996) compared the level of risk-taking by third and fourth grade students working individually or in collaborative groups with a mathematics software program. The collaborative groups were directed to select difficulty levels and to solve the problems together. They found that students in the collaborative groups chose problems at a significantly higher difficulty level. Students in the collaborative context selected more challenging tasks because they reported feeling less threatened by the consequences of failure, a finding consistent with research showing that students take greater risks when external constraints are reduced. Taken together, these studies suggest that collaboration may enhance the positive effects that technology has on student attitudes, but only when students are adequately directed or trained in the process of collaboration. Collaboration to achieve a fixed standard may yield more positive attitudes than collaboration within a larger competitive setting. Furthermore, improvement in student attitudes based on collaboration can lead to academic risk-taking, which some educators view as a key to enhancing student learning. Providing a Learner-as-Multimedia-Designer Environment Instructional decisions in the implementation of technology extend not only to student grouping, but also to student roles and responsibilities in using technology. Results reported from one high school (Liu, 1998, August; Liu and Rutledge, 1997) suggest that expanding student responsibilities through a learner-as-multimedia-designer environment can significantly affect student attitudes. Liu and Rutledge (1997) reported a positive impact on inner-city high school students' attitudes from their involvement in "a learner-as-designer environment simulating a real-world multimedia 2000 Research Report on the Effectiveness of Technology in Schools 100 production house." Student teams chose their own projects and worked directly with both clients and multimedia experts. One team decided to create a demonstration of an electronic yearbook for the school, while the other three teams worked on three different topics (physics, dinosaurs and history) for a virtual museum at the local Children's Museum. Student experiences included visits to the Children's Museum and a trip to a local multimedia development company to share their work. Following students' involvement in the program, the researchers reported significant gains in intrinsic goal orientation (finding tasks rewarding for their own sake, rather than because of an external reward or pressure); task value (interest in the tasks students were performing); and self-efficacy for learning (students' confidence in their own ability to accomplish learning tasks). In contrast, scores for the control group—a computer class at the same high school in which students were taught the use of various computing tools— remained about the same over the course of the semester and in the case of task value, actually decreased. Liu (1998, August) reported similar results for the following year, during which students completed a course that was similar in design but expanded from a single semester to a full year to allow for more complete development of student projects. Conclusion The research cited in this section provides ample evidence of the power of technology to motivate students and to improve their attitudes about themselves and about learning. These positive effects were found for special needs students and regular education students alike, for native English speakers and for students with limited English proficiency, for students in early childhood education as well as higher education. As one researcher (Schofield, 1997) noted, one of the most common and consistent findings of studies on the affective impact of computer use in classrooms has been ...an increase in motivation and closely related constructs such as interest and enjoyment of schoolwork, task involvement, persistence, time on task, and retention in school. Perhaps the most important determinant of student attitudes when using technology is the teacher. Only the teacher can create a friendly, caring environment in which students feel secure and willing to accept the many learning challenges they will face. However, technology will enhance the environment created by the teacher. Carefully designed software products will afford students appropriate control over their learning environment, will excite students and hold their interest and will provide engaging learning experiences that are unavailable in the traditional classroom. Quality teaching and quality software, together, can improve student self-concepts and student attitudes toward learning. 2000 Research Report on the Effectiveness of Technology in Schools 101 Section 3: Effects of Technology on Interactions Involving Educators and Students in the Learning Environment Across the United States, educators are addressing the need for restructuring the school learning environment to make it more student-centered, more interactive and more of a stimulus for cooperative problem-solving. Research is beginning to provide information about the positive effects of technology on schools, as a social phenomenon and on teacher-student and student-student interactions (e.g., Becker, 1998; Ruberg, in press; Schofield, 1997; Miech, Nave and Mosteller, 1997; Smith 1992; Woodward and Gersten, 1992; Garzella, 1991; Hartman, Neuwirth, Kiesler, Sproull, Cochran, PalmQuist, and Zubrow, 1991; Ringstaff, Sandholtz, and Dwyer, 1991; Sandholtz, Ringstaff, and Dwyer, 1991; Nastasi et al., 1990). These interactions were found to make a difference in student academic achievement (Ryan, 1990). In this section, we explore studies concerning the effects of technology on teacher-student interactions and teachers' instructional behavior, review research on technology's effects on student interactions, report on recent research suggesting characteristics of a desirable technology-based learning environment and describe the use of technology as a professional development and professional communication tool for teachers. Effects of Technology on Teacher-Student Interactions and Educators' Instructional Behavior Several studies explored the effects of technology on the interaction patterns of a classroom and on the learning environments educators create. Other research efforts focused specifically on the effects of introducing computer networks into the learning environment and on factors that predict teachers' use of the Internet as a professional and instructional tool. Differences in Classroom Interaction Patterns Becker (1998) conducted a survey of 441 teachers at 151 U.S. elementary and secondary schools participating in the National School Network, a confederation of organizations involved in fostering school Internet use. Becker found a variety of significant differences in self-reported instructional practices between teachers whose students had used computers at least weekly for three or more years and teachers whose students seldom or never used computers in their classes. For example, 71 percent of long-time computer users reported changes in the direction of being willing to be taught by students, compared to 29 percent of non-users, a difference of 42 percent. Long-time computer users also reported significantly more change in the direction of skill in conducting multiple parallel activities; assignment of long projects; encouraging students to choose 2000 Research Report on the Effectiveness of Technology in Schools 102 their tasks and work independently; incorporation of interdisciplinary content; and other similar changes generally related to a constructivist teaching approach. Results were particularly strong and consistent at the high school level. Among long-time computer users, generally between 40 and 60 percent attributed individual changes partly or primarily to the impact of computer use. Similar differences were reported between teachers who directed their students in using the Internet and those students did not use the Internet. Taken together, these results suggest a clear pattern of change related to long-time, frequent computer use and Internet use with students, particularly in the areas of: (1) teachers being more willing to discuss a subject about which they lack expertise and allowing themselves to be taught by students; (2) orchestrating multiple simultaneous activities occurring during class time; (3) defining long and complex projects for students to undertake; and (4) giving students greater choice in their tasks and the materials and resources they can use to complete them. All of these changes can be seen as a "letting go," a willingness to cede more authority to students instead of directing all of their learning from the front of the class. Rockman Et Al (1997) reported effects of a pilot program placing laptop computers into 53 public and private schools, including both elementary and secondary sites. The researchers found noteworthy differences in the instructional method teachers reported using most often, based on surveys taken before and approximately six months after the program was started. Among teachers who completed both the before-placement and after-placement surveys, "43 percent of those who initially indicated they teach in large groups moved to small group or individualized learning." Additionally, 61 percent reported use of a project-based teaching style after the laptop placement, compared to 35 percent prior to placement. These findings suggest that even in the short term, placement of computers can have a significant impact on many teachers' instructional behavior. Results of a study of Indiana's Model Applications of Technology project (1989-90) indicated that, after using computers for two years, "...most of the teachers added cooperative learning techniques to their teaching methods." However, changes in teachers' instructional and management styles depended on teacher attitudes about technology and the hardware configuration at the school. One group treated the computer project as an "extra"...and they used [the computers] to present students with isolated, fragmented activities using selected software ...computer use was routine, sometimes boring, [and] remotely related to the curriculum ...[This] group was often found in...computer lab sites or sites that had turned [a mobile] mini-lab concept into a [stationary] computer lab. The mini-labs were each equipped with eight computers and were intended to rotate from class to class, for purposes of computer-assisted instruction or student multimedia development. ...the other group made great strides in finding ways to meaningfully integrate computer use with their curriculum and their daily instruction... They looked for new ways to teach... They used more careful planning ...this more often occurred in sites where there was a "computer presence" in the classroom and the restraints of [too few computers per student] demanded that the teacher find creative ways to integrate computer use. 2000 Research Report on the Effectiveness of Technology in Schools 103 Bradley and Morrison (1991) studied the differences between teacher-student interaction patterns in computer labs and in classrooms with fewer than five computers. They discovered that in classrooms, 70 percent or more of the teachers' time was spent with non-computing students. In computer labs, over 40 percent of the teachers' time is spent academically monitoring or providing explanations to computer-using students, whereas less than 5 percent of the teachers' time was spent on such activities in classrooms. More questioning of computer-using students occurred in labs. Different levels of teacher experience and/or different levels of teacher training may explain the different findings for classroom-based computer use in these two studies. Yet another study found that use of collaborative technology such as interactive brainstorming and writing programs, telecommunication links and Internet access contributed significantly to the development of collaborative classroom communities. Researchers with the University of Texas at Austin (1996) found that in a project to increase high school student collaboration, transform teacher roles and help students direct their own learning, students whose teachers used technology more extensively reported greater peer collaboration and less teacher direction of classroom learning. The researchers found that use of the technology contributed significantly to creating collaborative knowledge-building communities, both among teachers who had implemented the project model more fully and among those who had made relatively little use of the model. "The results...suggest an additional effect for technology use above and beyond that attributable to integration of knowledge building teaching practices." Two studies focused on teachers of learning disabled (LD) students. Woodward and Gersten (1992) examined the effects of a videodisc-based fractions program on the subsequent instructional styles of LD high school teachers. After teachers and students completed the videodisc, teaching styles "became more interactive." Garzella (1991) investigated the effects of an expert system for reading diagnosis and prescription (CAPER). LD elementary school teachers who used the system reported that it helped guide them in grouping students more effectively. Schofield (1997) conducted a review of research related to the impact of computer use on the affective and social dimensions of the classroom. She found that use of computers influenced teachers' roles in a variety of unanticipated ways. One commonly reported outcome was "a shift from whole group instruction to teachers interacting more with individual students or small groups of students," accompanied by less emphasis on lecture and use of "a more interactive, individualized and student-centered approach." Based on her review, Schofield included the following observations and recommendations: First, the training of teachers to prepare them for computer use [should be designed] to help them think through what other kinds of changes they will need or want to make in their classrooms. Second, studies of the consequences of computer use need always to take account of the fact that such use rarely occurs without coincident changes that may well be as important as the computer use itself in influencing students' social and academic outcomes. Third, close attention should be paid to the actual changes in social functioning that occur in classrooms using computers to try to ensure that they are those intended or at least that they are constructive. These studies, when considered together, suggest that technology can have a beneficial effect on classroom interaction patterns toward greater interaction with teachers and among class members and toward more collaborative learning experiences. The research indicates, 2000 Research Report on the Effectiveness of Technology in Schools 104 however, that teacher attitudes and the amount and arrangement of computer hardware, are important factors in determining how teachers and students interact. Technology in a Project-Based Learning Environment Researchers from SRI International's Center for Technology and Learning (Penuel and Means, 1999) analyzed the effects of the Challenge 2000: Multimedia Project, a program funded by a Technology Innovation Challenge Grant that used multimedia as a key component of project-based learning. Projects were expected to • • • • • • • Be anchored in core curriculum; multidisciplinary Involve students in sustained effort over time Involve student decision-making Be collaborative Have a clear real-world connection Use systematic assessment: both along the way and end product Take advantage of multimedia as a communication tool During the first year of classroom observations, the researchers compared classrooms across grade levels where teachers "participate[d] in the project or otherwise engage[d] in technology use" with non-technology using classrooms. They found "significant changes in classroom practices from fall to spring, with differences between technology-using and non-technology-using classrooms." • The level of engagement in long-term projects was greater in technology-using classrooms than in non-technology classrooms. ...in the fall students in technology-using classrooms were only slightly more likely than students in comparison classes to be engaged in long-term projects... By spring, that gap was very wide, with 67% of technology-using classrooms versus 14% of non-technology using classrooms involved in extended projects at the time of observation. • Students in technology-using classrooms tended to construct products as part of their learning experiences more than students in non-technology-using classrooms did. ...the differences were much greater in the spring than in the fall. In the fall, 56% of technology-using classrooms involved students in constructing products compared to 39% of non-technology-using classrooms. By the spring, that gap had widened: 73% of technology-using classrooms engaged students in constructing products versus 38% of non-technology-using classrooms. • Over time, students in technology-using classrooms became more likely to work in small groups on collaborative activities than students in non-technology-using classrooms. While in the fall, few classrooms from either sample engaged students in small-group collaboration, nearly a quarter of technology-using classrooms 2000 Research Report on the Effectiveness of Technology in Schools 105 involved small-group collaborative activity in the spring (compared to 0% of non-technology-using classrooms). • Over time, differences between the two groups emerged with respect to teacher role patterns. ...teachers from both sets of classrooms were equally likely to be engaged primarily in questioning students, a traditional role for teachers, in the fall. In the spring, far fewer technology-using teachers used questioning as their dominant way of relating to students (7% versus 29% for non-technologyusing teachers). Instead, technology-using teachers were much more likely to be in a helping or monitoring role within the classroom (43% in the spring versus 18% of non-technology-using classrooms). During the next year, researchers focused their observations on sixth and seventh grade classrooms where teachers had received project mini-grants, compared to non-project classrooms usually in the same school. They found: • "Students in Multimedia Project classrooms engaged in significantly longer activities...than students in comparison classrooms... Moreover, both in the fall and the spring, students spent more time in project classrooms engaged in longterm activities that lasted a week or more (an average of 84% of the time in project classrooms versus 49% of the time in comparison classrooms)." "Students in Multimedia Project classrooms were more likely than comparison students to spend time engaged in small group collaboration." In the fall, students in project classrooms spent 35% of their time in small group collaboration, compared to 7% in non-project classrooms. By spring, percent of time spent in small group collaboration had risen to 52% in project classrooms but only 16% in non-project classrooms. Over time, students in Multimedia Project classrooms became more likely to engage in content-related, small group discussions than students in comparison classrooms. "While in the fall, students...in project and comparison classrooms...spent roughly the same amount of time engaged in small-group discussion, by the spring, project classrooms devoted much more time to this form of discourse"--72% in project classrooms versus 30% in non-project classrooms, as opposed to 41% and 34% respectively in the fall. Small group discussion was judged by the researchers to support collaboration among student peers. There was a significant difference between the two groups in "dominant activities" observed. In the comparison classrooms, students were far more likely to be engaged in "teacher-directed solo activities" such as reading silently and listening to the teacher. In the project classrooms, students were just as likely to be engaged in "higher-level cognitive activities characteristic of multimedia design" such as "deciding on the structure of a presentation; creating multiple representations, models, and analogies; arguing about or evaluating information; thinking about one's • • • 2000 Research Report on the Effectiveness of Technology in Schools 106 audience; and revising or editing work" as they were to be taking part in teacherled activities. • Teachers in project classrooms were significantly more likely than teachers in non-project classrooms to be filling a "facilitative role (e.g., assisting or helping, managing the organization of the task, monitoring as students work on their own)" in comparison to a "directive role (e.g., explaining concepts, providing information, questioning students)." Taken together, these findings of Penuel and Means suggest that involvement in multimedia projects can significantly change classroom instruction and patterns of classroom interaction, promoting longer-term projects, collaborative activity, small group discussion, and a more studentcentered instructional approach. Technology in an Inquiry-Based Learning Environment A study by Maor and Fraser (1996) reported that after secondary students used an observational database of Antarctic bird sightings to conduct investigations, they rated their classroom environment higher on values related to inquiry-based learning. As part of their investigations, students completed a booklet structured to guide them in using the database and acquiring inquiryrelated skills, culminating in an opportunity to design and conduct their own research. After completing the program, students rated their classrooms significantly higher in Investigation, measuring the extent to which students were encouraged to engage in inquiry learning and Openendedness, measuring the extent to which students determined how they would proceed in completing the computer-based activities. Technology-Rich Learning Environments A series of reports by the Apple Classrooms of Tomorrow (ACOT) Advanced Technology Group addressed the changes that teachers underwent as a result of long-term immersion (three to five years) in a technology-rich teaching and learning environment (Ringstaff et al., 1991; Sandholtz et al., 1991; Dwyer, Ringstaff, and Sandholtz, 1990; Sandholtz, Ringstaff, and Dwyer, 1990). The technology-rich environment included computers, printers, scanners, videodisc and videotape players, modems, CD-ROM drives and "hundreds of software titles." The ACOT researchers have identified five stages of instructional change that occur gradually as a result of radically transforming the technological aspects of the learning environment: • • Entry: Teachers struggled to cope with the changes learning environment. Adoption: Teachers moved from the initial struggles to successful use of technology on a basic level (e.g., correlation of drill and practice software to classroom instruction). Adaptation: Teachers moved from basic use to discovery of its potential for increased productivity (e.g., use of word processors for student writing). Appropriation: Having achieved complete mastery over the technology, teachers used it "effortlessly" as a tool to accomplish a variety of instructional and management goals. • • 2000 Research Report on the Effectiveness of Technology in Schools 107 • Invention: Teachers are prepared to develop "entirely new learning environments that utilize technology as a flexible tool." The researchers found that mirroring teachers' growth and development over time in their uses of technology were positive changes in teachers' beliefs about the instructional enterprise (toward a willingness to experiment), shifts in the roles they and students play (toward more student-centered Instruction) and shifts in the relationships among teaching colleagues (toward team teaching). Networked Learning Environments Two studies indicate the potential of computer-based networks in improving teacher-student interaction on college campuses. According to research by Smith (1992), university courses that took advantage of a computer-based network resulted in an increase in "the amount and quality of interaction outside of class," including teacher-student interaction. Hartman et al. (1991) found that over time, instructors in networked courses substantially increased their use of electronic communication without decreasing their use of standard forms of communication. In comparing networked and non-networked sections of the same course, these researchers observed more teacher interaction with lower-performing students in networked sections and more teacher interaction with higher-performing students in non-networked sections. Internet Use Becker (1999) reported on a national survey of fourth through twelfth grade teachers in which he considered various factors that might relate to (a) their own "professional use" of the Internet, (b) their assignment of Internet use by students for research and (c) their assignment of Internet use by students for projects and publishing (e.g., cross-school collaborative projects, lessons "where students became expert in a topic and put their information on the Web"). Survey results showed that ...by far the most important variable in predicting teachers' [professional] Internet use is the teacher's level of classroom connectivity... ...the biggest effect comes from having any classroom connection at all, rather than having to connect elsewhere in the school building or at home. However, both direct connectivity and direct connectivity with multiple (4+) computers contributed to explaining variation in teacher-directed Student Research use... In addition, direct connectivity had an independent effect on teachers' professional use of the Internet...and direct connectivity with four or more computers had a small effect on...Student Projects and Publication. Other variables that helped explain teachers' professional use of the Internet included level of computer expertise, constructivist attitudes and practices, staff development on Internet use, level of informal contacts with other teachers at school, age of the teacher (with younger teachers more likely to use the Internet professionally), and home Internet access. Factors that predicted teacher assignment of the Internet as a student research tool and for student projects 2000 Research Report on the Effectiveness of Technology in Schools 108 and publications included the teacher's level of computer expertise, constructivist attitudes and practices, staff development on Internet use, and the teacher's level of involvement in professional leadership activities (e.g., serving as a formal or informal mentor for other teachers, giving a workshop or conference talk). Effects of Technology on Student Interactions Recent research on the effects of educational technology on student interactions includes a comparison between work on computer and seatwork. According to Schofield's (1997) review of research on affective and social aspects of classroom computer use, many studies suggest, "that the use of computers typically fosters relatively high levels of collaboration and interaction among students." Research has also focused on variables such as the learning task, writing with word processors, networked learning environments, software characteristics, learner characteristics and instructional decisions regarding the use of technology. Again according to Schofield, specific effects of computer use on classroom social processes vary widely based on the type of software, developers' assumptions regarding individual or group use, embedded software design features such as feedback and teachers' perceptions of the software. Computer Work vs. Seatwork Din (1996) compared interaction patterns in two high school business education classes while students were working on and off the computer. Since there were not enough computers for all students to work simultaneously, part of the class would work on the computer while the rest of the class completed activities of similar content and difficulty level as seatwork. The groups would then switch partway through the class period. In one class, students working on the computer "interacted academically" approximately twice as often during computer work than seatwork. In the other class, students using the computers "interacted academically" almost three times more often. Students also showed less time off-task while working on the computer and scored significantly better on activities they completed on the computer than on seatwork activities. The Learning Task In his review of research on small group computer-based instruction, Shlechter (1991) concluded that the demands of the computer-based learning task might help determine whether student cooperation occurs. As an example, he cited a study in which assigning learners the task of playing ...against the computer promoted sharing and helping behaviors while having dyads play against each other fostered competitive behaviors. Writing with Word Processors Kumpulainen (1996) reported on two studies of 11- and 12-year-old students in Finland and the United Kingdom who worked in pairs at the computer to complete a variety of writing tasks using a word processor. Oral language interactions of each pair were audiotaped for approximately 30 minutes and analyzed on the basis of language function. Kumpulainen found that students often used conversation to exchange knowledge, to question each other and to express agreement and disagreement. The researcher concluded that collaboration while using the computer provided opportunities for students to recall and share their own knowledge, to apply it to the task at hand and to develop awareness of their own thinking and writing processes and those of their 2000 Research Report on the Effectiveness of Technology in Schools 109 peers. However, students rarely gave justifications for their views and easily accepted each other's judgments, suggesting the need for teacher guidance and monitoring to insure appropriate interaction in tasks involving potential argumentation, disagreement and criticism. Networked Learning Environments Several studies shed light on the potential of computer-based networks for improving studentstudent interaction in a variety of instructional settings. Research by Ruberg (in press) suggests that network-based communication activities can positively affect the class participation level of students who seldom participate in face-to-face classroom activities. In a college introductory writing course, "collaborative peer review and analysis" activities took place over the campus computer network, which supported synchronous online discussions. The activities were designed to "stimulate critical thinking and share personal opinions within the context of interactive, group writing." More than half of the students described as "seldom" participating in face-to-face class discussions participated online at about the same rate as students described as "frequent" participants in face-to-face discussions. As a group, the "seldom" participants contributed an average of more than two messages per student per networked-based discussion. Riel (1992) reviewed research on the use of networking for collaboration across classrooms in different geographic locations and found evidence of improved social skills and a lessening of ethnic tensions. In their survey of studies on computer-assisted language learning (CALL), Miech, Nave and Mosteller (1997) determined that one of the chief uses of CALL at colleges and universities is to enhance authentic communication through strategies such as "[engaging] students in conversations in the target language with other people both inside and outside of the classroom." They described studies by Chun (1994) Cononelos and Oliva (1993) and Hermann (1992) in which use of CALL contributed to more effective language instruction through increased authentic communication. Chun found that students in a beginning German course who engaged in real-time conversations on a local area network started to interact with each other, rather than primarily with the teacher. Cononelos and Oliva found that students who used Internet-based newsgroups and e-mail to communicate with native Italian speakers felt that the experience improved both their confidence and the quality of their writing. Hermann's study suggested that students of French who used CALL to create a classroom newspaper (which involved accessing each other's versions of newspaper articles on the computer and communicating with each other by e-mail) did as well as students who performed more traditional fill-in-the-missing-word (cloze) CALL activities. Miech, Nave and Mosteller concluded: Considered together, these studies indicate a promising direction for the future of CALL: educators can use computers as vehicles both to support new and different interaction among students and teachers in the target language and to create opportunities for students to converse with native speakers and others outside of the classroom and the university. Based on their review of network learning research in a variety of educational settings, Riel and Harasim (1994) conclude that research should focus on how networks are used for educational purposes. They cite a study by Riel and Levin (1990) that yielded several recommendations for the design of networks: 2000 Research Report on the Effectiveness of Technology in Schools 110 Network communities can be created with a group of people with established relationships seeking new ways to coordinate their collective work or with a group ...with no prior interactions who share a strong commitment to a specific task. Easy and reliable access to the network is required for all group members or there needs to be some external motivation for using the system (e.g., low cost, job, class requirement). Groups need some form of leadership. A group needs one or more people who take on the responsibility of monitoring and facilitating group interaction. Jiang and Ting (1998) conducted a survey of the perceptions of university students enrolled in 17 Web-based distance learning courses across several subject areas and correlated the results to several instructional variables. The researchers found that the number of instructor's responses per student was statistically and positively related to perceived student-student interaction and to the actual number of responses produced per student. Additionally, perceived and actual studentstudent interaction was positively correlated to the percentage of grade weight assigned to discussion. Taken together, these results suggest that teacher activity and involvement play an important role in promoting student-student interaction in distance learning environments. Software Characteristics Three studies suggest that different software characteristics may have different effects on student interaction patterns. Cavalier and Klein (1998) noted that providing instructional objectives significantly increased helping and on-task behaviors among dyads of fifth and sixth grade students completing a HyperCard earth science tutorial on modern-day prospecting. Conversely, students who received no orienting activities engaged in significantly more off-task behaviors than students who either previewed objectives or read an advance organizer paragraph. According to Cavalier and Klein, these results support the suggestion that "providing objectives to cooperative groups may be an effective method for increasing student interactions." Nastasi, Clements and Battista (1990) explored the effects of the Logo programming language and of more narrowly focused, problem-solving software on elementary school students' problemsolving ability and their interactions. Logo afforded students far greater control over their learning environment than did the other software included in this study. Students using Logo demonstrated significantly greater growth in problem-solving ability, accompanied by higher frequencies of: cognitive conflict [i.e., conflict about the learning task], attempt at resolution of cognitive conflict,...successful resolution of conflict, rule making...[and] seeking approval... The researchers suggested that it is Logo's ability to stimulate successful resolution of cognitive conflicts that may explain its superior impact on students' problem-solving skills. Dalton (1990) found that a learner-controlled version of an interactive video science lesson resulted in significantly higher rates of interaction among students than a lesson-controlled version. These two studies suggest that software offering a high degree of learner control may encourage positive interaction among students. 2000 Research Report on the Effectiveness of Technology in Schools 111 Learner Characteristics Two recent studies addressed the effects of computer play on the social interactions of young children. Villarruel (1990) found that both special education and regular education preschoolers demonstrated more social discourse when playing on computers than when playing with other ageappropriate toys. Children using problem-solving software evidenced significantly higher proportions of elaboration and word play than children using "discovery-based" software. Rhee and Chavnagri (1991) also documented "extensive social interactions" when preschoolers play with computers, most of which is based on the child's own initiation. Another research effort focused on students with special needs. Hine, Goldman and Cosden (1990) found that learning disabled or "handicapped" students who wrote on the computer in pairs had lower error rates than students working alone. They found evidence for three different explanations of this phenomenon: One member of the dyad [i.e., the less error-prone member] would dominate text entry with a product closely resembling the error-rate characteristics of that individual. ...when students were in control of the keyboard and were entering text, they would use their partner as a source of information. ...the non-typing member would remain active in the writing process by adopting the role of monitor and offering correction information to the typing member... Instructional Decisions Yu (1996-97) reported that Taiwanese fifth grade students working in pairs whose performance was measured against a fixed standard had significantly more positive perceptions both of their own partners and of other students than did members of pairs whose performance was measured competitively against that of other students. All students were assigned to complete three computer-assisted tutorial lessons and take a posttest. Partners' scores on the posttest were averaged for both treatment groups. Students in the first group were encouraged to "reach the excellence level of performance" and received a prize if the average of their score and their partner's score was above a preset level. Students in the second group were encouraged to "be the best" and received a prize if the average of their score and their partner's score was among the three highest for the class. Yu concluded, "cooperation without inter-group competition promoted more positive inter-personal relationships both within and among the learning groups than did cooperation with inter-group competition." Walther (1997) studied the impact of long-term versus short-term interaction and emphasis on group versus individual identity on members of e-mail distribution lists including American and English university students. Ten groups of five to six students were given two consecutive group assignments to read, review and write a common document on a set of five related articles. Some students were told they would remain in the same groups for both assignments, while others were told they would be reassigned to new groups for the second assignment. Instructions for members of some groups emphasized the group nature of the task, while instructions for members of other groups emphasized diversity, individuality and the opportunity to encounter a variety of different personalities and perspectives while working on the assignment. Based on results of a relational communication questionnaire, the researcher found significant differences with respect to intimacy and affection among group members, perceived social and physical attractiveness of 2000 Research Report on the Effectiveness of Technology in Schools 112 other group members and study effort expended in completing the group assignments. In all these areas, students working in long-term groups with a group emphasis on identity scored highest, while students working in short-term groups with a group emphasis scored lowest. Students in both long-term and short-term groups with an individual emphasis on identity scored in between. These findings underscore the importance of decisions regarding the duration of collaborative groups and how the collaboration is presented to group members. As Walther states, This research finds that certain social conditions and technology lead people from different places, who have never and will never see each other, to communicate more affection, to like each other more, to think they look better and to work harder than people working together under other conditions in [computer-mediated communication] or ...working together face-to-face. Witherspoon (1996) analyzed the effects of working together in mixed-gender pairs on gender characteristics in the writing of first graders. During the first week, students worked individually on the word processor to produce a narrative. The second week, students worked in same-gender pairs and the third week, they worked together in mixed-gender pairs. Characteristics of narratives produced by girls included a greater emphasis on community life and norms, less physically demanding activities, understanding nature to solve challenges, protagonists helping each other and protagonists suffering when they act alone. In contrast, narratives produced by boys showed more physical and social contests, aggressive or destructive activities, contests with nature, individual opposition to others and happy consequences when protagonists acted alone. Narratives produced by students working in mixed-gender pairs during the third week showed significantly more female characteristics. Characteristics of a Desirable Technology-based Learning Environment Several researchers have identified characteristics of the learning environment that make for effective use of technology, both at the school level and in individual classrooms. In her analysis of the effects of computer-based instruction on elementary school achievement, Ryan (1990) identified three characteristics of the most effective learning environments: (1) personal interaction among the members of the class (i.e., teacher-student and studentstudent interaction); (2) "curriculum integration by the teacher"; and (3) inclusion of activities in which students can direct their own learning or express themselves. These characteristics were likely to result in effective learning regardless of grade level. Ehman, Glenn, Johnson and White (1991) synthesized the results of eight case studies on the use of database software in social studies. They concluded that it is essential that the teacher provide structure when students engage in complex problem-solving using computers. Their recommendations to educators are applicable to problem-solving with a variety of tool software programs: • • • Begin with a unit introduction Provide "clear expectations with a sequence of activities" Explain and model essential elements of the problem-solving process and offer students opportunities to practice these elements 2000 Research Report on the Effectiveness of Technology in Schools 113 • Provide for "regular checking of student progress in accomplishing the milestone tasks of problem-solving" Ehman et al. offer several examples of appropriate structuring by the teacher: Using examples, modeling steps and processes, providing for student practice; debriefing student learning; and sharing outcomes are all essential elements of effective instructional structuring. These studies underscore the importance of the teacher's role in creating an effective, technologybased learning environment, an environment that is characterized by careful planning and frequent interaction among students and the teacher. In a meta-analysis of 103 studies, Lou, Abrami and Muni (1998) compared the task behaviors of students in school or in professional training who used computers individually and in groups of two to five. The researchers reported: Those working individually • Engaged more in the computer programs... • Requested more help from the instructor... • Needed less time in completing their tasks... Those working in groups • Engaged more in cognitive or positive social interactions between students... • Used more appropriate strategies... • Are more perseverant on task... These findings suggest a need for tailoring learning environments and expectations based on whether students will be using technology individually or in groups. Given the importance of the teacher's role in computer-based learning, Becker's (1994) survey research on the differences between exemplary computer-using teachers and other teachers seems especially important. He found that the best computer-using teachers are more likely to add new curriculum topics to their courses and eliminate or de-emphasize some existing topics as a result of using computers. They also tend to stress more classwork in small groups, to assign software on the basis of group needs and to include students in the software selection process. Becker identified personal characteristics that differentiate the best computer-using teachers from other teachers. First, exemplary computer-using teachers spend more than twice as many hours personally working on computers at school as do other computer-using teachers... The second largest difference...is that exemplary teachers have had more formal training in using and teaching with computers. Research at the Center for Children & Technology (Honey et al., 1995) suggests that exemplary use of educational technology may also be related to teacher experience. In a study of the Adventures in Supercomputing project, students completed independent research either individually or in small groups. Students who were rated the highest with respect to the demonstrated knowledge of their topic of inquiry, critical thinking, "clarity and coherence of 2000 Research Report on the Effectiveness of Technology in Schools 114 presentation," teamwork and ability to "apply programming skills to analyze or investigate their area of interest" were most likely to have had a teacher with ten or more years of computer experience. Students who were rated the lowest were mostly likely to have had a teacher with nine or fewer years of computer experience. According to Becker (1994), the following aspects of the instructional environment increase the likelihood of finding an exemplary computer-using teacher at a school: • • • • The existence of a social network of computer-using teachers at the same school Sustained use of computers at the school...to accomplish a goal other than learning: e.g., writing and publishing, industrial arts or business applications Organized support for computer-using teachers in the form of staff development activities and a full-time computer coordinator... Acknowledgment of the resource requirements for effectively using computers smaller class sizes and funds for software acquisition. Becker (1993) also identified factors related to developing a school environment in which desirable computer outcomes occur (e.g., many teachers using computers, a variety of staff development activities occurring, more curricular than recreational use of computers, stressing computers as academic tools rather than as a delivery system for basic skills practice). His statistical analysis of survey data suggests that "substantial district-level involvement in schoollevel decision-making" and "the active...leadership of a school-level computer coordinator" are key factors. Becker's (1994, 1993) findings suggest the importance of district- and school-level administration in developing a positive environment for educational technology and in improving the quality of computer-using teachers and, thus, computer-based instruction. Teacher Professional Development and Professional Communication Just as technology can transform the roles of teachers in the classroom, the use of technology is also becoming an important part of teachers' professional development. Several studies suggest the potential impact of technology on both preservice and inservice development. Pastore (1994) studied the affects of preservice teachers using systematic observation software as an aid to analyzing their own teaching behavior. Such analysis is believed to help teachers improve their teaching style. As the subjects listened to an audio tape of their lesson, they classified each verbal statement into one of the eight defined behavior categories by "clicking" on the appropriate "button" of a computer ...[which then] calculated and printed out results. Preservice teachers who used the software spent significantly more time listening for their students' responses during their analyses than preservice teachers who relied on paper, pencil and a hand calculator did. The teachers who used the software also indicated significantly more positive attitudes toward systematic observation. 2000 Research Report on the Effectiveness of Technology in Schools 115 A recent study by LaMaster (1996) suggests the usefulness of e-mail as a tool for stimulating communication among preservice teachers. Small groups of physical education students used email to ask and answer questions about instruction, student behavior, management and other topics. The vast majority of student-to-student exchanges focused on instructional issues. As a result of peer feedback, preservice teachers reported being able to make changes in their lesson plans and upcoming teaching sessions. The author suggests that this application of e-mail may reduce the sense of isolation many novice teachers feel. Matthew, Barufaldi and Bethel (1998) reported on electronic networking among preservice elementary teachers that started during a three-week science teaching methods class and extended into the subsequent 12-week student teaching practicum. During the methods class, students interacted regularly over a class news group and "post[ed] regularly over a listserv/mailing list set up specifically for preservice teachers called PreSTO (Preservice and Student Teachers Online)." During the teaching practicum, students were required to "make five reflective postings [on the class news group] based on their experiences teaching science in their classrooms" and respond at least twice to other students' posts. Students also used the news group to exchange "lesson planning ideas, teaching ideas and class management issues." In contrast, preservice teachers in the control group had no formal mechanism set up for interacting after the methods class was over. The researchers found that the preservice teachers who used networking increased significantly more than the control group in both their "belief...in their ability to affect student performance in science" and their "confidence...in their own abilities to teach science effectively." McMahon (1997) analyzed the effects of a variety of factors on the participation of teachers in Mathematical Learning Forums—on-line professional development groups with a faculty advisor, a small number of teacher participants and designated discussion topics. Participants were expected to log on at least twice a week to contribute to discussions. McMahon found that "participants with home access posted significantly more messages than those who had access only at school" and that an analysis of the messages they sent showed a significantly higher level of professional interaction and reflection. McMahon's findings also supported the common-sense supposition that teachers who feel a lack of professional interaction at their local school sites tend to be more active in network-based professional interaction. For example, being likely to share a good lesson idea with local colleagues was inversely related to the average length of messages posted by participants and reported satisfaction with the quality of local discussions about math teaching was inversely related to the number of messages posted. Conversely, The more strongly a participant agreed that his or her principal had little idea of what happened in the classroom, the more likely [the participant was] to post a greater number of messages...to post longer messages...and to have a higher Both this inverse relation of local interaction to distance participation and the finding that participation increases with home access should be taken into account in designing distance professional interaction forums for teachers. 2000 Research Report on the Effectiveness of Technology in Schools 116 Conclusion The research cited above strongly supports the use of technology as a catalyst for changing the learning environment. Educational technology has been used to: • • • • Stimulate more interactive teaching Inquiry-based learning Project-based learning Effective grouping of students and cooperative learning However, technology as a catalyst is not sufficient by itself. Also essential are teachers who are well prepared to function in a more open, flexible, student-centered environment. Meaningful change will occur over a period of time. At first, teachers can be expected to struggle with the change brought about by technology. However, they will adopt, adapt and eventually learn to use technology effortlessly and creatively. 2000 Research Report on the Effectiveness of Technology in Schools 117 Bibliography Agarwal, R. and Day, A.E. (1998, Spring). The impact of the Internet on economic education. Journal of Economic Education, 99-110. Aguirre, A., Jr. (1997). The effects of cooperative, competitive, and individual computer-assisted instruction on the achievement of middle school students. Dissertation Abstracts International, 58/08-A (Order No. AAD98-03562). Aleven, V., Koedinger, K.R., and Cross, K. (1999). Tutoring answer explanation fosters learning with understanding. In Lajoie, S.P., and Vivet, M. (Eds.), Artificial Intelligence in Education, Open Learning Environments: New Computational Technologies to Support Learning, Exploration, and Collaboration, Proceedings of AIED-99. Amsterdam: IOS Press. Alexander, M.P. (1993). The effective use of computers and graphing calculators in college algebra. Dissertation Abstracts International, 54/06-A (Order No. AAD93-28010). Allen, G. and Thompson, A. (1994, April). Analysis of the effects of networking on computer-assisted collaborative writing in a fifth-grade classroom. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA (ERIC Document Reproduction Service No. ED373777). Anderson-Inman, L. (1990, May). Keyboarding across the curriculum. The Computing Teacher, 36. Armel, D. and Shrock, S.A. (1996). The effects of required and optional computer-based note taking on achievement and instructional completion time. Journal of Educational Computing Research, 14(4), 329-344. Avent, J. (1993). A study of language learning achievement differences between students using the traditional language laboratory and students using computer-assisted language learning courseware. Dissertation Abstracts International, 54/09-A (Order No. AAD94-04633). Bain, A., Houghton, S., Sah, F.B., and Carroll, A. (1992, Summer). An evaluation of the application of interactive video for teaching problem-solving to early adolescents. Journal of Computer-Based Instruction, 19 (3), 92-99. Bair, R.J. (1990). The effects of word processing on the writing output of emotionally disturbed students. Dissertation Abstracts International, 51/08-A (Order No. AAD91-01600). Bangert-Drowns, R.L. (1993, Spring). The word processor as an instructional tool: A meta-analysis of word processing in writing instruction. Review of Educational Research, 63(1), 69-93. Barba, R.H. and Merchant, L.J. (Fall 1990). The effects of embedding generative cognitive strategies in science software. Journal of Computers in Mathematics and Science Teaching, 10(1), 59-65. Barnes, W.G.W. (June 1994). Cognitive flexibility lessons: Using a concept map interface to traverse multiple perspectives in hypermedia. Paper presented at the Annual Meeting of ED-MEDIA, World Conference on Educational Multimedia and Hypermedia, Vancouver, BC, Canada. Baron, L.J. and Abrami, P.C. (1992). Microcomputer learning with a tutorial program-manipulating group size and exposure time. Journal of Computing in Childhood Education, 3(3/4), 231-245. Barron, B., Vye, N., Zech, L., Schwartz, D., Bransford, J., Goldman, S., Pellegrino, J., Morris, J., Garrison, S., and Kantor, R. (1995). Creating Contexts for Community-Based Problem Solving: The Jasper Challenge Series. In Hedley, C.N., Antonacci, P., and Rabinowitz M. (Eds.), Thinking and Literacy: The Mind at Work. Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers. 2000 Research Report on the Effectiveness of Technology in Schools 118 Barrutia, R., Bissell, J., Rodriguez, E., and Scarcella, R. (1993). The language and cognitive development of Hispanic middle School Students. Irvine, CA: Department of Education, University of California, Irvine: Final Report Submitted to the University of California Linguistic Minority Research Institute. Bastecki, V.L. and Berry, L. H. (1996). Window presentation styles and user's spatial ability. In Proceedings of Selected Research and Development Presentations at the 1996 National Convention of the Association for Educational Communications and Technology, Indianapolis, IN (ERIC Document Reproduction Service No. ED397776). Becker, H.J. (1999, February). Internet Use by Teachers: Conditions of Professional Use and Teacher-Directed Student Use. Teaching, Learning, and Computing: 1998 National Survey, Report #1. Center for Research on Information Technology and Organizations, The University of California, Irvine and The University of Minnesota. Becker, H.J. (1998, April). The influence of computer and Internet use on teachers' pedagogical practices and perceptions. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA. Becker, H.J. (1994, March). How our best computer-using teachers differ from other teachers: Implications for realizing the potential of computers in schools. Journal of Research on Computing in Education, 26(2). Becker, H.J. (1993). Decision making about computer acquisition and use in American schools. Computers and Education, 20(4), 341-352. Behar-Horenstein, L.S. (1996, April). Influencing the academic achievement of Chapter 1 students: Does computer technology facilitate student outcomes? Paper presented at the Annual Meeting of the American Educational Research Association, New York. Berman, M.R. (1990). The word processor as an aid in the composing process of twelfth-grade learning-disabled students. Dissertation Abstracts International, 51/03-A (Order No. AAD90-22331). Beyer, F.S. (1992, March). Impact of computers on middle-level student writing skills. Paper presented at the Annual Meeting of the American Educational Research Association. Philadelphia: Research for Better Schools, Inc. Biemans, H.J.A. and Simons, P.R.J. (1996). Contact-2: A computer-assisted instructional strategy for promoting conceptual change. Instructional Science, 24, 157-176. Billings, D.M. and Cobb, K.L. (1991, Winter). Effects on learning style preferences, attitude and GPA on learner achievement using computer assisted interactive videodisc instruction. Journal of Computer-Based Instruction, 19(1), 1216. Bissell, J. and Simpson, S. (1993). The effectiveness of an innovative, technology-enhanced and inquiry oriented middle school science program. Irvine, CA: Department of Education, University of California, Irvine: Final Report, Dwight D. Eisenhower Mathematics and Science Program, U.S. Department of Education. Bitter, G.G. and Hatfield, M.M. (1993). Teaching mathematics methods using interactive multimedia: The TMMUIV research results. Monograph 5 in the Monograph Series of Technology Based Learning and Research, Arizona State University, Study 3. Blanton, W.B., Moorman, G.B., Hayes, B.A., and Warner, M.L. (1997). Effects of participation in the Fifth Dimension on far transfer. Journal of Educational Computing Research, 16(4), 371-396. Borras, I. and Lafayette, R.C. (1994). Effects of multimedia courseware subtitling on the speaking performance of college students of French. The Modern Language Journal, 78 (1), 61-75. Bradley, M.J. and Morrison, G.R. (1991). Student-teacher interactions in computer settings: A naturalistic inquiry (ERIC Document Reproduction Service No. 343565). Brantmayer, M. (1994). The effects of computer aided instruction on prospective industrial hygiene and safety professionals' achievement on safety training programs. Dissertation Abstracts International , 55/06-A (Order No. AAD94-27952). Bricken, M. and Byrne, C.M. (1992, Summer). Students in virtual reality: A pilot study on educational applications of virtual reality technology. (ERIC Document Reproduction Services No. ED358853). 2000 Research Report on the Effectiveness of Technology in Schools 119 Brush, T.A. (1997). The effects on student achievement and attitudes when using integrated learning systems with cooperative pairs. Educational Technology Research and Development, 45(1), 51-64. Brush, T.A. (1996). The effectiveness of cooperative learning for low- and high- achieving students using an integrated learning system. In Proceedings of Selected Research and Development Presentations at the 1996 National Convention of the Association for Educational Communications and Technology, Indianapolis, IN (ERIC Document Reproduction Service No. ED397780). Brusic, S.A. (1991). Determining effects of fifth-grade students' achievement and curiosity when a technology education activity is integrated with a unit in science. Dissertation Abstracts International, 52/09-A (Order No. AAD92-01134). Burt, B.J. (1993). Elementary gifted students' perceptions of motivating factors in computer software environments. Dissertation Abstracts International, 54/07-A (Order No. AAD93-33078). Buzhardt, J. and Semb, G. (1998, April). Item-by-item versus end-of-test feedback in a computer-based testing lab. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA. Calvert, S.L, Watson, J.A., Brinkley, V., and Penny, J. (1990). Computer presentational features for poor readers' recall of information. Journal of Educational Computing Research, 6(3), 287-298. Carlson, S.L. and White, S.H. (1998). The effectiveness of a computer program in helping kindergarten students learn the concepts of left and right. Journal of Computing in Childhood Education, 9(2), 133-147. Carter, D. (1994). Effects of computer technology on student outcomes for low-performing ninth-grade students. Dissertation Abstracts International, 55/08-A (Order No. AAD95-01185). Castelluccio, R. (1994). The effects of image processing on retention and attitude of seventh grade science students during interactive video instruction. Dissertation Abstracts International, 55/09-A (Order No. AAD95-02496). Cates, W.M. and McNaull, P.A. (1993, Summer). Inservice training and university coursework: Its influence on computer use and attitudes among teachers of learning disabled students. Journal of Research on Computing in Education, 25(4), 447-63. Cavalier, J.C. and Klein, J.D. (1998). Effects of cooperative versus individual learning and orienting activities during computer-based instruction. Educational Technology Research and Development, 46(1), 5-17. Cavanaugh, C. (1998). The effectiveness of interactive distance education technologies in K-12 learning: A metaanalysis. Dissertation Abstracts International, 59/07-A (Order No. AAD98-42140). Chang, L.L. and Osguthorpe, R.T. (1990). The effects of computerized picture-word processing on kindergartners' language development. Journal of Research in Childhood Education, 5(1), 73-83. ChanLin, L.J. (1996). Enhancing computer graphics through metaphorical elaboration. Journal of Instructional Psychology, 23(3), 196-203. ChanLin, L.J. and Chan, K.C. (1996, February). Computer graphics and metaphorical elaboration for learning science concepts. Paper presented at the Annual Meeting of the Association for Educational Communication and Technology, Indianapolis, IN (ERIC Document Reproduction Service No. ED392390). Chiu, C.H. (1996). The effects of computer networks and collaboration on the development of science skills and attitudes among secondary science students in Taiwan, R.O.C. Dissertation Abstracts International, 57/06-A (Order No. AAD9633125). Chou, C. and Lin, H. (1998). The effect of navigation map types and cognitive styles on learners’ performance in a computer-networked hypertext learning system. Journal of Educational Multimedia and Hypermedia, 7(2/3), 151-176. Chun, D.M. (1994). Using computer networking to facilitate the acquisition of interactive competence. System, 22(1), 1731. Chun, D.M. and Plass, J.L. (1996). Effects of multimedia annotations on vocabulary acquisition. The Modern Language Journal, 80(ii), 183-198. 2000 Research Report on the Effectiveness of Technology in Schools 120 Chyung, S.Y. (l996). The effects of conceptual tempo and computer-assisted instructional environment on college students' self-regulated learning and academic achievement. Dissertation Abstracts International, 57/07-A (Order No. AAD96-38396). Clariana, R.B. (1993, Spring). The motivational effect of advisement on attendance and achievement in computer-based instruction. Journal of Computer-Based Instruction, 20(2), 47-51. Clariana, R.B. (1990, Autumn). A comparison of answer until correct feedback and knowledge of correct response feedback under two conditions of contextualization. Journal of Computer-Based Instruction, 17(4), 125-129. Clark, H.C. (1996). Cyber-coaching in computer as learning partner. Paper presented at the Annual Meeting of the American Educational Research Association, New York. Clements, D.H. (1991, Spring). Enhancement of creativity in computer environments. American Educational Research Journal, 28(1), 173-187. Clements, D.H. and Battista, M.T. (1990). The effects of Logo on children's conceptualizations of angle and polygons. Journal for Research in Mathematics Education, 21(5), 356-371. Cockayne, S. (1991, February). Effects of small group sizes on learning with interactive videodisc. Educational Technology, 43-45. Cognition and Technology Group at Vanderbilt University. (in press). The Jasper series: A design experiment in complex, mathematical problem solving. To be published in Hawkins, J. and Collins, A. (Eds.), Design Experiments: Integrating Technologies into Schools. New York: Cambridge University Press. Cognition and Technology Group at Vanderbilt University. (Fall 1992). The Jasper series as an example of anchored instruction: Theory, program description and assessment data. Educational Psychologist, 27(3), 291-316. Cognition and Technology Group at Vanderbilt University. (1991). The Jasper series: A generative approach to improving mathematical thinking. This Year in School Science. Washington, DC: American Association for the Advancement of Science. Cononelos, T. and Oliva, M. (1993). Using computer networks to enhance foreign language/culture education. Foreign Language Annals, 26(4), 527-34. Crooks, S., Klein, J., Dwyer, H., and Jones, E. (1995, February). The effects of cooperative learning and learner control in computer-assisted instruction. Paper presented at the Association for Educational Communications and Technology, Anaheim, CA. Dailey, E.M. (1991). The relative efficacy of cooperative learning versus individualized learning on the written performance of adolescent students with writing problems. Dissertation Abstracts International, 52/07-A (Order No. AAD91-32758). Dalton, D.W. (1990, Winter). The effects of cooperative learning strategies on achievement and attitudes during interactive video. Journal of Computer-Based Instruction, 17(1), 8-16. Dalton, D.W. and Goodrum, D.A. (1991, Winter). The effects of computer-based pretesting strategies on learning and continuing motivation. Journal of Research on Computing in Education, 24(2), 204-213. DeGraw, B.C. (1990). A study to determine changes in social interaction among students, teachers, and parents influenced by the placement of microcomputers in the homes and school of fourth-grade Fuqua students during 19881989. Dissertation Abstracts International, 51/05-A (Order No. AAD90-26713). Din, F.S. (1996, May). Computer-assisted instruction, students' off-task behavior and their achievement. Education and Treatment of Children, 19(2), 170-182. Dwyer, D.C., Ringstaff, C., and Sandholtz, J.H. (1990). Teacher Beliefs and Practices, Part I: Patterns of change. ACOT Report #8. Cupertino, CA: Apple Classrooms of Tomorrow Advanced Technology Group, Apple Computer, Inc. 2000 Research Report on the Effectiveness of Technology in Schools 121 Ehman, L.H., Glenn, A.D., Johnson, V., and White, C.S. (1992, Spring). Using computer databases in student problem solving: A study of eight social studies teachers' classrooms. Theory and Research in Social Education, 20(2), 179-206. Eliasen, K., McKinstry, J., Fraser, B.M., and Babbitt, E.P. (1997, November). Navigating online menus: A quantitative experiment. College and Research Libraries, 58(6), 509-516. Elliott, A. and Hall, N. (1997). The impact of self-regulatory teaching strategies on "at-risk" preschoolers' mathematical learning in a computer-mediated environment. Journal of Computing in Childhood Education, 8(2/3), 187-198. Ennis, D.L. (1993). A transfer of database skills from the classroom to the real world. Computers in the Schools, 9(2/3), 55-63. Farnsworth, C.C. (l996). Tracking the development of clinical expertise in veterinary students: Measuring the effect of problem-based learning. Dissertation Abstracts International, 57/08-B (Order No. AAD97-01621). Fazal, M.B. (1996, August). Effectiveness and costs of computer-based instruction in higher education and defense training: A meta-analysis. Dissertation Abstracts International, 57/09-A (Order No. AAD97-05095). Ferretti, R.P. and Okolo, C.M. (1997, March). Designing multimedia projects in the social studies: Effects on students' content knowledge and attitudes. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL. Fisher, C., (1989). The Influence of High Computer Access on Student Empowerment. ACOT Report #1. Cupertino, CA: Apple Classrooms of Tomorrow Advanced Technology Group, Apple Computer, Inc. Fletcher, J.D. (1990). Effectiveness and cost of interactive videodisc instruction in defense training and education. Alexandria, VA: Institute for Defense Analysis. Fletcher, J.D., Hawley, D.E., and Piele, P.K. (1990, March). Costs, effects, and utility of microcomputer-assisted instruction in the classroom. Paper presented at the Seventh International Conference on Technology and Education in Brussels, Belgium (ERIC Document Reproduction Service No. ED325068). Fletcher-Flinn, C.M. and Gravatt, B. (1995). The efficacy of computer assisted instruction (CAI): A meta-analysis. Journal of Educational Computing Research, 12(3), 219-242. Foley, B.J. (1998, April). Designing visualization tools for learning. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA. Follansbee, S., Gilsdorf, N., Stahl, S., Dunfey, J. Cohen, S., Pisha, B., and Hughes, B. (1996, October). The role of online communications in schools: A national study. Peabody, MA: Center for Applied Special Technology. Foster, K., Erickson, G., Foster, D., and Torgeson, J.K. (unpublished). Computer administered instruction in phonological awareness: Evaluation of the Daisy Quest program. Funkhouser, C. and Djang, P. (1993, Spring). The influence of problem-solving software on student attitudes about mathematics. Journal of Research on Computing in Education, 25(3), 339-346. Gallini J.K. and Helman, N. (1993). Audience awareness in technology-mediated environments. Paper presented at the Annual Meeting of the American Educational Research Association, Atlanta, GA. Gardner, C.M., Simmons, P.E., and Simpson, R.D. (1992, October). The effects of CAI and hands-on activities on elementary students' attitudes and weather knowledge. School Science and Mathematics, 92(6), 334-336. Garrison, S.J. (l996). Influence of metacognitive prompting on learning within computer mediated problem sets. Dissertation Abstracts International, 57/08-A (Order No. AAD97-00440). Garza, T. (1991). Evaluating the use of captioned video materials in advanced foreign language learning. Foreign Language Annals, 24(3), 239-57. Garzella, M.F. (1991). Using an expert system to diagnose weaknesses and prescribe remedial reading strategies among elementary learning-disabled students. Dissertation Abstracts International, 52/09-A (Order No. AAD92-07011). 2000 Research Report on the Effectiveness of Technology in Schools 122 Geban, O., Askar, P., and Ilker, O. (1992, September/October). Effects of computer simulations and problem-solving approaches on high school students. Journal of Educational Research, 86(1), 5-10. Gerretson, H. (1998). The effect of a dynamic geometry learning environment on preservice elementary teachers' performance on similarity tasks. Dissertation Abstracts International, 59/09-A (Order No. AAD99-05948). Gilliver, R.S., Randall, B., and Pak, Y.M. (1998). Learning in cyberspace: Shaping the future. Journal of Computer Assisted Learning, 14, 212-222. Goldmacher, H.K. and Lawrence, R.L. (1992). An experiment: Computer literacy and self esteem for Head Start preschoolers—Can we leapfrog? Paper presented at the Annual Conference of National Association for the Education of Young Children. Goldman, S.R., Petrosino, A.J., Sherwood, R.D., Garrison, S., Hickey, D., Bransford, J.D., and Pellegrino, J. (1996). Anchoring science in multimedia learning environments. In Vosniadou, S., De Corte, E., Glaser, R., and Mandl, H. (Eds.), International Perspectives on the Design of Technology-Supported Learning Environments. Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers. Grace, C. (1998). Personality type, tolerance of ambiguity, and vocabulary retention in CALL. CALICO Journal, 15(1-3), 19-45. Graham, S. and MacArthur, C. (1988). Improving learning disabled students' skills at revising essays produced on a word processor: Self-instructional strategy training. Journal of Special Education, 22(2), 133-152. Green, L.C. (1991). The effects of word processing and a process approach to writing on the reading and writing achievement, revising and editing strategies, and attitudes towards writing of third-grade Mexican-American students. Dissertation Abstracts International, 52/12-A (Order No. AAD92-12535). Grossen, B. and Carnine, D. (1990, Summer). Diagramming a logic strategy: effects on difficult problem types and transfer. Learning Disability Quarterly, 13, 168-182. Grossen, B. and Lee, C. (1994). The effects of videodisc instruction and textbook-oriented instruction in combination with problem-solving. Unpublished paper. University of Oregon. Hannafin R. and Sullivan, H. (1995). Learner control in full and lean CAI programs. Educational Technology Research and Development, 43(1), 19-30. Hartman, K., Neuwirth, C.M., Kiesler, S., Sproull, L., Cochran, C., PalmQuist, M., and Zubrow, D. (1991, January). Patterns of social interaction and learning to write. Written Communication, 8(1), 79-113. Haugland, S.W. (1992). The effect of computer software on preschool children's developmental gains. Journal of Computing in Childhood Education, 3(1), 14-30. Haugland, S.W. and Shade, D.D. (1990). Developmental Evaluations of Software for Young Children, 1990 Edition. Albany, NY: Delmar Publishers, Inc. Haugland, S.W. and Shade, D.D. (1988). Developmentally appropriate software. Young Children, 43(4), 37-43. Hays, T.A. (1996). Spatial abilities and the effects of computer animation on short-term and long-term comprehension. Journal of Educational Computing Research, 14(2), 139-155. Hermann, F. (1992). Instrumental and Agentive Uses of the Computer: Their Role in Learning French as a Foreign Language. San Francisco, CA: Mellen Research University Press. Higgins, E.L. and Raskind, M.H. (1997, Spring). The compensatory effectiveness of optical character recognition/speech synthesis on reading comprehension of postsecondary students with learning disabilities. Learning Disabilities, 8(2), 7587. Higgins, E.L. and Raskind, M.H. (1995, Spring). Compensatory effectiveness of speech recognition on the written composition performance of postsecondary students with learning disabilities. Learning Disability Quarterly, 18, 159174. 2000 Research Report on the Effectiveness of Technology in Schools 123 Hine, M.S., Goldman, S.R., and Cosden, M.A. (1990). Error monitoring by learning handicapped students engaged in collaborative microcomputer-based writing. Journal of Special Education, 23(4), 407-422. Hollen, D. (1987). The effects of keyboarding on the spelling accuracy of fifth grade students. Unpublished master's paper. Eugene, OR: University of Oregon. Honey, M., K. McMillan, K., Tsikalas, K., and Griswald, C. (1995, January). Adventures in Supercomputing, 1993-1994 Evaluation (Final Report) CCT Reports, Issue No. 1. Hooper, S. (1992, January-February). Effect of peer interaction during computer-based mathematics instruction. Journal of Educational Research, 85(3), 180-189 Hooper, S., Temiyakarn, C., and Williams, M.D. (1993). The effects of cooperative learning and learner control on highand average-ability students. Educational Technology, Research, and Development, 41(2), 5-18. Hsu, H.L. (l996). Interactivity of human-computer interaction and personal characteristic in a hypermedia learning environment. Dissertation Abstracts International, 57/05-A (Order No. AAD96-30322). Idaho Council for Technology in Learning. (1998). Idaho technology initiative: An accountability report to the Idaho Legislature. Boise, ID: Bureau of Technology Services, Idaho State Department of Education. Indiana's Fourth Grade Project: Model applications of technology. Second Year (1989-1990) Report #TAC-B-154, 155, and 156. Indianapolis, IN: Advanced Technology, Inc., and Indiana State Department of Education (ERIC Document Reproduction Service No. ED343550). Ivers, K.S. and Barron, A.E. (1994). Teaching telecommunications: A comparison between video and computer-based instruction (ERIC Document Reproduction Service No. ED378963). Jakobsdóttir, S., and Hooper, S. (l995). Computer-assisted foreign language learning: Effects of text, context, and gender on listening comprehension and motivation. Educational Technology Research and Development, 43(4), 43-59. Jensen, M.S., Wilcox, K.J., Hatch, J.T., and Somdahl, C. (1996). A computer-assisted instruction unit on diffusion and osmosis with a conceptual change design. Journal of Computers in Mathematics and Science Teaching, 15(1/2), 49-64. Jiang, M. and Ting, E. (1998, April). Course design, instruction, and students' online behaviors: A study of instructional variables and students' perceptions of online learning. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA. Johari, A. (1998). Effects of inductive multimedia programs including graphs on creation of linear function and variable conceptualization. In Proceedings of Selected Research and Development Presentations at the National Convention of the Association for Educational Communications and Technology (AECT) (ERIC Document Reproduction Service No. 423841). Johnsey, A., Morrison, G.R., and Ross, S.M. (1992). Using elaboration strategies training in computer-based instruction to promote generative learning. Contemporary Educational Psychology, 17, 125-135. Johnson, M.L. (1993). The effects of instructional strategies on student performance (interactive video, lecture method, demonstration method). Dissertation Abstracts International, 54/08-A (Order No. AAD93-34096). Johnson-Gentile, K., Clements, D.H., and Battista, M.T. (1994). Effects of Computer and noncomputer environments on students' conceptualizations of geometric motions. Journal of Educational Computing Research, 11(2), 121-140. Jurkat, M.P., Skov, R.B., Friedman, E.A., Pinkham, R.S., and McGinley, J.J. (unpublished). A field study of student performance: evaluating a model for introducing computer use in high school mathematics. Justice, L.M. (1999). The effect and affect of animated visual cues within a computer-based learning program. Dissertation Abstracts International, 60/05-A (Order No. AAD99-31625). Kang, S. H. (1996). The effects of using an advance organizer on students' learning in a computer simulation environment. Journal of Educational Technology Systems, 25(1), 57-65. 2000 Research Report on the Effectiveness of Technology in Schools 124 Kao, M.T. and Lehman, J.D. (1997, March). Scaffolding in a computer-based constructivist environment for teaching statistics to college learners. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL (ERIC Document Reproduction Service No. ED408317). Kao, M.T., Lehman, J.D., and Cennamo, K.S. (1996). Scaffolding in hypermedia assisted instruction: An example of integration. In Proceedings of Selected Research and Development Presentations at the 1996 National Convention of the Association for Educational Communications and Technology, Indianapolis, IN (ERIC Document Reproduction Service No. ED397803). Katayama, A.D., Crooks, S.M., and Nelson, C.E. (1999, April). Effects of notetaking format on achievement when studying electronic text. Paper presented at the Annual Meeting of the American Educational Research Association, Montreal, Canada (ERIC Document Reproduction Service No. ED431034). Katz, S.N. and Hall, E.M. (1997, March). Cognitive task analysis and computer-based training systems: Lessons learned from coached practice environments in Air Force avionics. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL. Keller, F.S. (1968). Good-bye teacher. Journal of Applied Behavior Analysis, 1(1), 79-89. Kinzie, M.B., Strauss, R., and Foss, J. (1993). The Effects of an interactive dissection simulation on the performance and achievement of high school biology students. Journal of Research in Science Teaching. 30(8), 989-1000. Kinzie, M.B., Sullivan, H.J., and Berdel, R.L. (1992). Motivational and achievement effects of learner control over content review within CAI. Journal of Educational Computing Research, 8(1), 101-114. Kitz, W. and Thorpe, H. (1992, Summer). The effectiveness of videodisc and traditional algebra instruction with collegeaged remedial students. Unpublished manuscript, University of Wisconsin, Oshkosh. Klayder, J.R. (1993). The effects of time and scope advisement within a hypertext environment. Dissertation Abstracts International, 54/09-A (Order No. AAD94-05754). Koedinger, K.R., Anderson, J.R., Hadley, W.H., and Mark, M.A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8(1), 30-43. Koedinger, K.R. and Sueker, E.L.F. (1996). PAT goes to college: Evaluating a cognitive tutor for developmental mathematics. In Proceedings of the Second Annual Conference of the Learning Sciences. Charlottesville, VA: AACE. Koszalska, T.A. (1999). The relationship between the types of resources used in science classrooms and middle school students' interest in science careers: An exploratory analysis. Thesis submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at The Pennsylvania State University. Kulik, C. and Kulik, J.A. (1991). Effectiveness of Computer-Based Instruction: An Updated Analysis. Computers in Human Behavior, 7, 75-94. Kulik, J.A. (1994). Meta-analytic studies of findings on computer-based instruction. In Baker, E., and O'Neil, H. (Eds.), Technology Assessment in Education and Training. Hillsdale, NJ and Hove, UK: Lawrence Erlbaum Associates, Publishers. Kulik, J.A., Kulik, C.L., and Carmichael, K. (1974). The Keller plan in science teaching. Science, 183, 379-383. Kumpulainen, K. (1996). The nature of peer interaction in the social context created by the use of word processors. Learning and Instruction, 6, 243-261. LaMaster, K J. (l996). Preservice teachers as mentors during an early field experience through electronic communication (e-mail). Dissertation Abstracts International, 57/05-A (Order No. AAD96-30917). Land, W.A. and Haney, J.J. (1990, November). The effect of computer assisted instruction and the academic achievement of junior college students. Mississippi State, MI: Mississippi State University, College of Education (ERIC Document Reproduction Service No. ED327223). 2000 Research Report on the Effectiveness of Technology in Schools 125 Lazarowitz, R. and Huppert, J. (1993, Spring). Science process skills of 10th grade biology students in a computer assisted learning setting. Journal of Research on Computing in Education, 25(3), 366-382. Lee, M.J. (1990, February). Effects of different loci of instructional control on students' metacognition and cognition: Learner vs. program control. In Proceedings of Selected Paper Presentations at the Convention of the Association for Educational Communications and Technology (ERIC Document Reproduction Service No. 323938). Lee, Y.B. and Lehman, J.D. (1993). Instructional cueing in hypermedia: a study with active and passive learners. Journal of Educational Multimedia and Hypermedia, 2(1), 25-37. Li, N.K. (1990, July). Writing with pen or computer? A study on ESL secondary school learners. Paper presented at the Annual World Conference on Computers in Education (ERIC Document Reproduction Service No. ED322720). Liao, Y.C. (1990). Effects of computer-assisted instruction and computer programming on students' cognitive performance: A quantitative synthesis. Dissertation Abstracts International, 51/10-A (Order No. AAD91-07337). Liao, Y.C. and Bright, G.W. (1991). Effects of computer programming on cognitive outcomes: A meta-analysis. Journal of Educational Computing Research, 7(3), 251-268. Light, P. and Littleton, K. (1997). Situational effects in computer-based problem solving. In Resnick, L.B., Saljo, R., Potecorvo, C., and Burge, B. (Eds.), Discourse, Tools and Reasoning: Situated Cognition and Technologically Supported Environments. NATO ASI Series F: Computer and Systems Sciences, vol. 160. New York: Springer-Verlag, 224-242 Lin, C. H. and Davidson-Shivers, G.V. (1996). Effects of linking structure and cognitive style on students' performance and attitude in a computer-based hypertext environment. Journal of Educational Computing Research, 15(4), 317-329. Liu, M. (1998). The effect of hypermedia authoring on elementary school students' creative thinking. Journal of Educational Computing Research, 19(1), 27-51. Liu, M. (1998, August). A study of engaging high-school students as multimedia designers in a cognitive apprenticeshipstyle learning environment. Computers in Human Behavior, 14(3), 387-415. Liu, M. (1993, February). The effect of hypermedia assisted instruction on second-language learning through a semanticnetwork-based approach. Paper presented at the Annual Conference of the Eastern Educational Research Association. Liu, M. and Rutledge, K. (1997). The effect of a "learner as multimedia designer" environment on at-risk high school students' motivation and learning of design knowledge. Journal of Educational Computing Research, 16(2), 145-177. Lou, Y., Abrami, P.C., and Muni, S. (1998, April). Effects of social context when learning with computer technology: A series of meta-analyses. Poster presented at the Annual Meeting of the American Educational Research Association, San Diego, CA. Luzzo, D.A. and Pierce, G. (1996, December). Effects of DISCOVER on the career maturity of middle school students. The Career Development Quarterly, 45(2), 170-172. MacArthur, C.A., Graham, S., Schwartz, S.S., and Schafer, W.D. (1995, Fall). Evaluation of a writing instruction model that integrated a process approach, strategy instruction, and word processing. Learning Disability Quarterly, 18, 278-291. MacArthur, C.A. and Haynes, J. (unpublished). The Student Assistant for Learning from Text (SALT): A hypermedia reading aide. (Research supported by a grant from the U.S. Department of Education, Special Education Programs). Machtmes, K.L. (1998). A meta-analysis of the effectiveness of telecourses in adult distance education. Dissertation Abstracts International, 60/03-A (Order No. AAD99-21107). Mac Iver, D.J., Balfanz, R., and Plank, S.B. (1999, Winter). An "elective replacement" approach to providing extra help in math: The talent development middle schools' computer- and team-assisted mathematics acceleration (CATAMA) program. Research in Middle Level Education Quarterly, 22(2), 1-23. MAGI Educational Services, Inc. (1992, February). Evaluation of the New York State/Moscow Schools telecommunications project. Albany, NY: New York State Department of Education. 2000 Research Report on the Effectiveness of Technology in Schools 126 Mann, D., Shakeshaft, C., Becker, J., and Kottkamp, R. (1999). West Virginia Story: Achievement Gains from a Statewide Comprehensive Instructional Technology Program. Santa Monica, CA: Milken Exchange on Education Technology. Maor, D. and Fraser, B.J. (1996). Use of classroom environment perceptions in evaluating inquiry-based computerassisted learning. International Journal of Science Education, 18(4), 401-421. March, T.A. (1993). Computer-guided writing: Developing expert characteristics in novice writers. Unpublished master's thesis. San Diego University. Market Data Retrieva1's 1999 Technology Survey. (1999). Shelton, CT: Market Data Retrieval. Market Data Retrieva1's Higher Education Technology Survey. (1998). Shelton, CT: Market Data Retrieval. Martin, E.D. and Rainey, L. (1993). Student achievement and attitude in a satellite-delivered high school science course. The American Journal of Distance Education, 7(1), 54-61. Marttunen, M. (1997, June). Electronic mail as a pedagogical delivery system: An analysis of the learning of argumentation. Research in Higher Education, 38(3), 345-363. Mathew, N.M., Barufaldi, J.P., and Bethel, L.J. (1998). The effect of electronic networking on preservice elementary teachers' science teaching self-efficacy. Paper presented at the Annual Meeting of the National Association of Research in Science Teaching, San Diego, CA (ERIC Document Reproduction Service No. ED424106). Mayfield-Stewart, C., Moore, P., Sharp, D., Hasselbring, T., Goldman, S.R., and Bransford, J. (1994, April). Evaluation of multimedia instruction on learning and transfer. Paper presented at the Annual Conference of the American Education Research Association, New Orleans, LA. McClendon, S.L. (1989). First grade spelling success with keyboarding. The Computing Teacher, 17(2), 35-36. McClurg, P.A. (1992). Investigating the development of spatial cognition in problem-solving microworlds. Journal of Computing in Childhood Education, 3(2), 111-126. McMahon, T.A. (1997, March). From isolation to interaction? Network-based professional development and teacher professional communication. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL. McNeil, B.J. and Nelson, K.R. (1991). Meta-analysis of interactive video instruction: A 10 year review of achievement effects. Journal of Computer-Based Instruction 18(1), 1-6. McWhirter, M.E. (1991). The effect of Level One videodisc technology on sixth-grade student achievement in science. Dissertation Abstracts International, 52/05-A (Order No. AAD91-29725). Mevarech, Z.R. (1993). Who benefits from cooperative computer-assisted instruction? Educational Computing Research, 9 (4), 451-464. Mevarech, Z.R., Silber, O., and Fine, D. (1991). Learning with computers in small groups: Cognitive and affective outcomes. Journal of Educational Computing Research, 7(2), 233-243. Miech, E.J., Nave, B., and Mosteller, F. (1997). On CALL: A review of computer-assisted language learning in U.S. colleges and universities. In Branch, R.M., and Minor, B.B. (Eds.), Educational Media and Technology Yearbook, 22, 61-84. Miller, L.C.H. (1996). Understanding the effects of a multimedia presentation's sequencing strategy on learning: An experimental investigation. Dissertation Abstracts International, 58/01-A (Order No. AAD97-18414). Montazemi, A.R. and Wang, F. (1995). An empirical investigation of CBI in support of mastery learning. Journal of Educational Computing Research, 13(2), 185-205. 2000 Research Report on the Effectiveness of Technology in Schools 127 Moreno, R. and Mayer, R.E. (2000). A coherence effect in multimedia learning: The case for minimizing irrelevant sounds in the design of multimedia instructional messages. Journal of Educational Psychology, 92(1), 117-125. Moreno, R. and Mayer, R.E. (1999a). Cognitive principles of multimedia learning: The role of modality and contiguity. Journal of Educational Psychology, 91(2), 358-368. Moreno, R. and Mayer, R.E. (1999b). Multimedia-supported metaphors for meaning making in mathematics. Cognition and Instruction, 17(3), 215-248. Moreno, R. and Mayer, R.E. (unpublished). Pedagogical agents in constructivist multimedia environments: The role of image and language in the instructional communication. (Research supported by a grant from the National Science Foundation). University of California, Santa Barbara. Morrison, R., Ross, S., Gopalakrishnan, M., and Casey, J. (1995). The effects of feedback and incentives on achievement and computer-based instruction. Contemporary Educational Psychology, 20, 32-50. Murphy, H. and Higgins, E. (1994). An investigation of the compensatory effectiveness of assistive technology on postsecondary students with learning disabilities. Final report. Submitted to the Fund for the Improvement of Postsecondary Education (FIPSE) (ERIC Document Reproduction Service No. ED414681). Muthukrishna, N., Carnine, D., Grossen, B., and Miller, S. (1993). Children's alternative frameworks: Should they be directly addressed in science instruction? Journal of Research in Science Teaching, 30(3), 233-248. NAEP Mathematics Consensus Project, National Assessment Governing Board, U.S. Department of Education. (No date). Mathematics Framework for the 1996 National Assessment of Educational Progress. Washington, DC: U.S. Department of Education. Nagata, N. (1993). Intelligent computer feedback for second language instruction. The Modern Language Journal, 77(3), 330-39. Nastasi, B.K. and Clements, D.H. (1993). Motivational and social outcomes of cooperative computer education environments. Journal of Computing in Childhood Education, 4(1), 15-43. Nastasi, B.K., Clements, D.H., and Battista, M.T. (1990). Social-cognitive interactions, motivation, and cognitive growth in Logo programming and CAI problem-solving environments. Journal of Educational Psychology, 82(1), 150-158. Navarro, P. and Shoemaker, J. (1999, March). Economics in cyberspace: A comparison study. University of California— Irvine Graduate School of Management Discussion Paper. Newbold, M.C. (1993). A comparison of sixth-grade students' access, retrieval, and utilization of information obtained from CD-ROM and print sources. Dissertation Abstracts International, 54/03-A (Order No. AAD93-20111). Niedelman, M. (1991). Problem solving and transfer. Journal of Learning Disabilities, 24(6). Niederhauser, D.S. (1998, April). Learning and computer use in traditional and reform-oriented mathematics classrooms. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA. Niederhauser, D.S., Salmen, D., Skolmoski, P., and Reynolds, R.E. (1998, April). The influence of navigational style on learning from hypertext. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA. Nix, C.A.G. (1998). The impact of e-mail use on fourth graders' writing skills. Dissertation Abstracts International, 60/03-A (Order No. AAD99-21889). O'Banion, K.M., Goldman, S.R., Sharp, D.L.M., Bransford, J.D., Vye, N.J., Beaty, J., and Saul, E. (1993). Multimedia support for language comprehension skills in at-risk kindergarten students. Paper presented at the Annual Meeting of the American Educational Research Association in Atlanta, GA. O'Callaghan, B.R. (1998). Computer-intensive algebra and students' conceptual knowledge of functions. Journal for Research in Mathematics Education, 29(1), 21-40. 2000 Research Report on the Effectiveness of Technology in Schools 128 Okolo, C.M. and Ferretti, R.P. (1996, Fall). Knowledge acquisition and technology-supported projects in the social studies for students with learning disabilities. Journal of Special Education Technology, 13(2), 91-103. Osler, J.E. (1996). The effects of an ergonomically designed computer-based tutorial on elementary students' recall. Dissertation Abstracts International, 57/10-A (Order No. AAD97-08338). Owston, R.D., Murphy, S., and Wideman, H.H. (1991, June). Effects of word processing on student writing in a high computer access environment (Technical Report 91-3). North York, ON, Canada: York University, Centre for the Study of Computers in Education (ERIC Document Reproduction Service No. ED341365). Owston, R.D. and others. (1990, May). On and off computer writing of eighth grade students experienced in word processing (Technical Report 90-1). North York, ON, Canada: York University, Centre for the Study of Computers in Education (ERIC Document Reproduction Service No. ED319053). Paolucci, R. (1998). The effects of cognitive style and knowledge structure on performance using a hypermedia learning system. Journal of Educational Multimedia and Hypermedia, 7(2/3), 123-150. Parker, L. and Lepper, M. (1992). Effects of fantasy contexts on children's learning and motivation: Making learning more fun. Journal of Personality and Social Psychology, 62 (4), 625-633. Pastore, R. (1994). The effects of computer-assisted systematic observation and goal setting on the performance of and attitudes of preservice teachers. Dissertation Abstracts International, 55/09-A (Order No. AAD95-04270). Penuel, W.R. and Means, B. (1999). Observing classroom processes in project-based learning using multimedia: A tool for educators. Unpublished paper. Center for Technology in Learning, SRI International. Policy Information Center, Research Division, Educational Testing Service. (September 1998). Does It Compute? The Relationship Between Educational Technology and Student Achievement in Mathematics. Policy Information Report. Princeton, NJ: Educational Testing Service. Powell, Z.E. (1992, April). Test anxiety and test performance under computerized adaptive testing methods. Paper presented at the Annual Meeting of the American Educational Research Association in San Francisco, CA (ERIC Document Reproduction Service No. ED344910). Quality Education Data's Internet Usage in Schools 2000, 5th Edition. (2000). Denver: Quality Education Data. Quality Education Data's Technology Purchasing Forecast 1999-00, 5th Edition. (2000). Denver: Quality Education Data. Quinn, J., Pena, C., and McCune, L. (1996). The effects of group and task structure in an instructional simulation. In Proceedings of Selected Research and Development Presentations at the 1996 National Convention of the Association for Educational Communications and Technology, Indianapolis, IN (ERIC Document Reproduction Service No. ED397826). Raghavan, K., Sartoris, M.L., and Glaser, R. (1997). The impact of model-centered instruction on student learning: The area and volume units. Journal of Computers in Mathematics and Science Teaching, 16(2/3), 363-404. Raskind, M.H., and Higgins, E. (1995, Spring). Effects of speech synthesis on the proofreading efficiency of postsecondary students with learning disabilities. Learning Disability Quarterly, 18, 141-158. Reed, W.M. and Rosenbluth, G.S. (1992). The effect of HyperCard programming on knowledge construction and interrelatedness of humanities-based information. (ERIC Document Reproduction Services No. ED355908). Reglin, G.L. (1989-1990). CAI Effects on Mathematics Achievement and Academic Self-Concept Seminar. Journal of Educational Technology Systems, 18(1), 43-48. Repman, J. (1993). Collaborative, computer-based learning: Cognitive and affective outcomes. Journal of Educational Computing Research, 9(2), 149-163. 2000 Research Report on the Effectiveness of Technology in Schools 129 Repman, J. and Lan, W. (1996, April). The effects of collaborative, computer-based drill and practice activities on academic risk taking. Paper presented at the Annual Meeting of the American Educational Research Association, New York. Rhee, M.C. and Chavnagri, N. (1991). 4-year-old children's peer interactions when playing with a computer. (ERIC Document Reproduction Service No. 342466). Rhyser, G.R. (1990). Effects of computer education on students' achievement, attitudes, and self-esteem. Dissertation Abstracts International, 52/01-A (Order No. AAD91-05649). Rieber, L.P. (1994, April). Animation as real-time feedback during a computer-based simulation. Paper presented at the annual conference of the American Educational Research Association, New Orleans, LA. Rieber, L.P. (1991). Animation, incidental learning, and continuing motivation. Journal of Educational Psychology, 83(3), 318-328. Rieber, L.P., Boyce, M.J., and Assad, C. (1990). The effects of computer animation on adult learning and retrieval tasks. Journal of Computer-Based Instruction, 17(2), 46-52. Rieber, L.P. and Noah, D. (1997). Effect of gaming and visual metaphors on reflective cognition within computer-based simulations. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL. Rieber, L.P. and Parmley, M.W. (1995). To teach or not to teach? Comparing the use of computer-based simulations in deductive versus inductive approaches to learning with adults in science. Journal of Educational Computing Research, 13(4), 359-374. Rieber, L.P., Tzeng, S.C., Tribble, K., and Chu, G. (1996). Feedback and elaboration within a computer-based simulation: A dual coding perspective. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA. Riel, M. (1992). Making connections from urban schools. Education and Urban Society, 24(4), 485-496. Riel, M. (1992, April). Educational change in a technology-rich environment. Paper presented at the annual meeting of the American Educational Research Association in San Francisco, CA. Riel, M. and Harasim, L. (1994). Research perspectives on network learning. Machine Mediated Learning, 4(2-3), 91113. Riel, M. and Levin, J. (1990). Building electronic communities: Successes and failures in computer networking. Instructional Science, 19, 145-169. Ringstaff, C., Sandholtz, J.H., and Dwyer, D.C. (1991). Trading places: When teachers utilize student expertise in technology-intensive classrooms. ACOT Report #15. Cupertino, CA: Apple Classrooms of Tomorrow Advanced Technology Group, Apple Computer, Inc. Ritchie, D. and Gimenez, F. (1995). The influence of dominant languages on the effectiveness of graphic organizers in computer-based instruction. In Proceedings of the 17th Annual National Convention of the Association for Educational Communications and Technology (AECT), Anaheim, CA (ERIC Document Reproduction Services No. ED383333). Rockman Et Al. (1997, June). Report of a laptop program pilot. A project for Anytime Anywhere Learning by Microsoft Corporation and Notebook for Schools by Toshiba America Information Systems. Rockman Et Al. (1995). Assessing the growth: The Buddy Project evaluation, 1994-5: Final report. Paper submitted to the Corporation for Educational Technology. Rowley, K., Miller, T., Armstrong Lab/Intelligent Training Branch, and Carlson, P. (1997, March). The influence of learner control and instructional styles on student writing in a supportive environment. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL (ERIC Document Reproduction Service No. ED404668). 2000 Research Report on the Effectiveness of Technology in Schools 130 Rowry, E. (1994). The effects of computer-controlled interactive videodiscs in teaching high school chemistry. Dissertation Abstracts International, 55/07-A (Order No. AAD94-33150). Ruberg, L. (in press). Facilitating student participation and interaction through an electronic peer review activity: A case study of two classes—Freshmen writing and plant science lab. To be published in Wood, S., and Edgard, C. (Eds.), Telecommunications Is Not About Computers Russell, M. and Haney, W. (1997). Testing writing on computers: An experiment comparing student performance on tests conducted via computer and via paper-and-pencil. Education Policy Analysis Archives, 5(3) [online]. Available: http://olam.ed.asu.edu/epaa/v5n3.html (Jan. 15, 2000). Ryan, A.W. (1991). Meta-analysis of achievement effects of microcomputer applications in elementary schools. Educational Administration Quarterly, 27(2), 161-184. Ryan, A.W. (1990). Meta-analysis of achievement effects of microcomputer applications in elementary schools. Dissertation Abstracts International, 51/12-A (Order No. AAD91-13106). Sandholtz, J.H., Ringstaff, C., and Dwyer, D.C. (1991). The relationship between technological innovation and collegial interaction. ACOT Report #13. Cupertino, CA: Apple Classrooms of Tomorrow Advanced Technology Group, Apple Computer, Inc. Sandholtz, J.H., Ringstaff, C., and Dwyer, D.C. (1990). Classroom management: Teaching in high-tech environments: Classroom management revisited. First-Fourth Year Findings. ACOT Report #10. Cupertino, CA: Apple Classrooms of Tomorrow Advanced Technology Group, Apple Computer, Inc. Santiago, R.S. and Okey, J.R. (1990, October-November). The effects of advisement and locus of control on achievement in learner-controlled instruction. Paper presented at the 32nd International Conference of the Association for the Development of Computer-Based Instructional Systems in San Diego, CA (ERIC Document Reproduction Service No. ED. 330332). Sartorio, V.J. (1993). Effects on computer-based learning on the language development of preschoolers in special education classrooms. Dissertation Abstracts International, 54/06-A (Order No. AAD93-31511). Scardamalia, M., Bereiter, C., Brett, C., Burtis, P., Calhoun, C., and Lea, N.S. (1992). Educational applications of a networked communal database. Interactive Learning Environments, 2(1), 45-71. Schacter, J., Chung, G.K.W.K., and Dorr, A. (1998). Children's Internet searching on complex problems: Performance and process analyses. Journal of the American Society for Information Science, 49(9), 840-849. Schmidt, S.C. (1991). Technology for the 21st century: The effects of an integrated distributive computer network system on student achievement. Dissertation Abstracts International, 52/07-A. Schnackenberg, H.L., Sullivan, H.J., Leader, L.F., and Jones, E.E.K. (1998). Learner preferences and achievement under differing amounts of learner practice. Educational Technology Research and Development, 46(2), 5-15. Schofield, J.W. (1997, Spring). Computers and classroom social processes—A review of the literature. Social Science Computer Review, 15(1), 27-39. Schroeder, E.E. (1994, February). Navigating through the hypertext: Navigational technique, individual differences, and learning. In Proceedings of the National Convention of the Association for Educational Communications and Technology and Sponsored by the Research and Theory, Nashville, TN. Schultz, L.H. (1995, February). Pilot validation study of the Scholastic Beginning Literacy System (WiggleWorks) 199495 mid-year report. Unpublished paper. Schutte, J. (1996). Virtual teaching in higher education: The new intellectual superhighway or just another traffic jam? [e-mail document] jschutte@csun.edu. Scrase, R. (1998). An evaluation of a multi-sensory speaking-computer based system (Starcross–IDL) designed to teach the literacy skills of reading and spelling. British Journal of Educational Technology, 29(3), 211-224. 2000 Research Report on the Effectiveness of Technology in Schools 131 Semmel, M.I., Gerber, M.M., and Semmel, D.S. (unpublished). Speed of retrieval of multiplication facts and the development of automaticity. Santa Barbara, CA: University of California Santa Barbara, Graduate School of Education. Sharp, D., Kinzer, C., Risko, V., and the Cognition and Technology Group at Vanderbilt. (1994, December). The Young Children's Literacy Project: Video and software tools for accelerating literacy in at-risk children. Paper presented at the Annual Meeting of the National Reading Conference, San Diego, CA. Shell, D.F., Turner, J.E., Husman, J., Droesch-Cliffel, D.M., Nath, I., and Sweany, N. (l996, April). Effects of collaborative, computer-supported, knowledge-building communities on high school students' knowledge building and intentional learning. Poster presented at the American Educational Research Association Annual Meeting, New York (ERIC Document Reproduction Service No. ED395570). Shen, C.H. (1997). The effects of gender and cooperative learning with CAI on college students' computer science achievement and attitudes toward computers. Dissertation Abstracts International, 58/11-A (Order No. AAD98-14771). Sherman, G.P. (1994). The effects of cued interaction and ability grouping during cooperative computer-based science instruction. Dissertation Abstracts International, 55/08-A (Order No. AAD95-00740). Sherman, G.P. and Klein, J.D. (1995). The effects of cued interaction and ability grouping during cooperative computerbased science instruction. Educational Technology Research and Development, 43(4), 5-24. Shin, E.C., Schallert, D.L., and Savenye, W.C. (1994). Effects of learner control, advisement, and prior knowledge on young students' learning in a hypertext environment. Educational Technology, Research and Development, 42(1), 33-46. Shlechter, T.M. (1991, November). What do we really know about small group CBT? Paper presented at the 33rd annual conference of the Association for the Development of Computer-Based Instructional Systems, St. Louis, MO (ERIC Document Reproduction Service No. ED342381). Shu-Ling, L. (1998, Spring). The effect of visual display on analogies using computer-based learning. International Journal of Instructional Media, 25(2), 151-160. Shyu, H. (1997, February). Effects of anchored instruction on enhancing Chinese students' problem-solving skills. Paper presented at the Annual Meeting of the Association for Educational Communication and Technology, Albuquerque, NM (ERIC Document Reproduction Service No. ED 405841). Shyu, H.Y. and Brown, S.W. (1993, April). Learner control: The effects during computer-based videodisc instruction. Paper presented at the 1993 Annual Meeting of American Educational Research, Atlanta, Georgia. Signer, B.R. (1992, February). A study of black at-risk urban youth using computer-assisted testing. In Proceedings of Selected Research and Development Presentations at the Convention of the Association for Educational Communications and Technology and Sponsored by the Research and Theory Division (ERIC Document Reproduction Service No. ED348024). SIIA's Education Market Report: K-12. (2000). Washington, DC: Software and Information Industry Association. SIIA's Education Market Report: Post-secondary. (1999). Washington, DC: Software and Information Industry Association. Silver, N.W. and Repa, J.T. (1993). The effect of word processing on the quality of writing and self-esteem of secondary school English-as-second-language students: Writing without censure. Journal of Educational Computing Research, 9(2), 265-283. Simsek, A. (1993, January). The effects of learner control and group composition in computer-based cooperative learning. In Proceedings of Selected Research and Development Presentations at the Convention of the Association for Educational Communications and Technology and Sponsored by the Research and Theory Division (ERIC Document Reproduction Service No. ED362205). Singhanayok, C. and Hooper, S. (1998). The effects of cooperative learning and learner control on students' achievement, option selections, and attitudes. Educational Technology Research and Development, 46(2), 17-33. 2000 Research Report on the Effectiveness of Technology in Schools 132 Skinner, M.E. (1990). The effects of computer-based instruction on the achievement of college students as a function of achievement status and mode of presentation. Computers in Human Behavior, 6, 351-360. Smith, L. (1992, August). New online education system nets better grades for college students and instructors. Paper submitted to the Association for College and University Telecommunications Administrators, ACUTA News. Smith, R.R. (1996). Effects of textbook, static CAI, and animated CAI, and learning styles on the achievement of college students in symbolic matrix algebra. Dissertation Abstracts International, 57/07-A (Order No. AAD96-37049). Snider, S. (1996). Effects of alternative software in development of creativity in at-risk and non at-risk young children. Dissertation Abstracts International, 57/12-A (Order No. AAD97-16592). Snyder, I. (1993, Spring). Writing with word processors: A research overview. Educational Research, 35(1), 61. Solmon, L.C. (1999). Afterword. In Mann, D., Shakeshaft, C., Becker, J., and Kottkamp, R., West Virginia Story: Achievement Gains from a Statewide Comprehensive Instructional Technology Program. Santa Monica, CA: Milken Exchange on Education Technology. Sormunen, C. and Ray, C.M. (1996, Summer). Teaching collaborative writing with group support systems software—An experiment. The Delta Pi Epsilon Journal, 38(3), 125-138. SPA K-12 Market Study Report. (1991). Washington, DC: Software Publishers Association. Spaulding, C. and Lake, D. (1991, April). Interactive effects of computer network and student characteristics on students' writing and collaborating. Paper presented at the Annual Meeting of the American Educational Research Association in Chicago. Specht, M. (1998, June). Empirical evaluation of adaptive annotation in hypermedia. In ED-MEDIA/ED-TELECOM 98 World Conference on Educational Multimedia and Hypermedia & World Conference on Educational Telecommunications. Proceedings (ERIC Document Reproduction Service No. ED428725). Stine, H.A. (1993). The effects of CD-ROM interactive software in reading skills instruction with second-grade Chapter 1 students. Dissertation Abstracts International, 54/09-A (Order No. AAD94-00115). Stone, T. T., III (1996). The academic impact of classroom computer usage upon middle-class primary grade level elementary school children. Dissertation Abstracts International, 57/06-A (Order No. AAD96-33809). Svec, M.T. (1994). "Effect of microcomputer-based laboratory on students' graphing interpretation skills and conceptual understanding of motion. Dissertation Abstracts International, 55/08-A (Order No. AAD95-00449). Swaak, J., Van Joolingen, W.R., and De Jong, T. (1998). Supporting simulation-based learning: The effects of model progression and assignments on definitional and intuitive knowledge. Learning and Instruction, 8(3), 235-252. Swan, K., et al. (1990, April). Perceived locus of control and computer-based instruction. Paper presented at the Annual Meeting of the American Educational Research Association in Boston, MA (ERIC Document Reproduction Service No. ED327140). Swan, K., Guerrero, F., Mitrani, M., and Schoener, J. (1990, Summer). Honing in on the target: Who among the educationally disadvantaged benefits most from what CBI? Journal of Research on Computing in Education, 381-403. Swan, K. (1996). Exploring the role of video in enhancing learning from hypermedia. Journal of Educational Technology Systems, 25(2), 179-188. Swan, M.K. and Jackman, D.H. (1998, April). Vocational student successes in distance education courses. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA. Szabo, M. and Poohkay, B. (1996). An experimental study of animation, mathematics achievement, and attitude toward computer-assisted instruction. Journal of Research on Computing in Education, 28(3), 390-402. 2000 Research Report on the Effectiveness of Technology in Schools 133 Temiyakarn, C. and Hooper, S. (1993, January). The effects of cooperative learning and learner control on high and low achievers. In Proceedings of Selected Research and Development Presentations at the Convention of the Association for Educational Communications and Technology and Sponsored by the Research and Theory Division (ERIC Document Reproduction Service No. ED362208). Thorkildsen, R. and Lowry, W. (1990). The effects of a videodisc program on mathematics self-concept. Unpublished manuscript, Logan, UT, Utah State University. Tozcu, A. (1998). The effect of teaching sight vocabulary with computer assisted instruction on vocabulary gain, decrease in reaction time for frequent word recognition, and reading comprehension. Dissertation Abstracts International, 59/04-A (Order No. AAD98-29340). Tucson Unified School District #1. (1994, April). Maxwell Middle School, School of the Future: Computer Classroom Pilot, Two-Year Findings. Tucson, AR: Tucson Unified School District #1, Department of Information Technologies. Underwood, J., Cavendish, S., Dowling, S., Fogelman, K., and Lawson, T. (l996). Are integrated learning systems effective learning support tools? Computers & Education, 26(1-3), 33-40. U.S. Congress, Office of Technology Assessment. (1988, September). Power On! New Tools for Teaching and Learning, OTA-SET-379. Washington, DC: U.S. Government Printing Office. Valeri-Gold, M. and Deming, M.P. (1991, Spring). Computers and basic writers: A research update. Journal of Developmental Education, 14(3), 10-14. Viesulas, J.N. (1994). The effect of videotape demonstrations on students learning in the undergraduate general chemistry laboratory. Dissertation Abstracts International, 55/05-A (Order No. AAD94-34754). Villarruel, F. (1990). Talking and playing: An examination of the effects of computers on the special interactions of handicapped and nonhandicapped preschoolers. Dissertation Abstracts International, 51/11-A (Order No. AAD9108303). Vitale, M.R. and Romance, N.R. (1992). Using videodisk instruction in an elementary science methods course: Premeditating science knowledge deficiencies and facilitating science teaching attitudes. Journal of Research in Science Teaching, 29(9), 915-928. Walther, J.B. (1997, March). Group and interpersonal effects in international computer-mediated collaboration. Human Communication Research, 23(3), 342-369. Webster, A.H. (1990). The relationship of computer-assisted instruction to mathematics achievement, student cognitive styles, and student and teacher attitudes. Dissertation Abstracts International, 51/10-A (Order No. AAD91-03410). Weir, S. (1992, January). Electronic communities of learners: Facts or fiction. Cambridge, MA: TERC Communications (ERIC Document Reproduction Service No. 348990). Weller, H.G., Repman, J., Lan, W., and Rooze, G. (1995). Improving the effectiveness of learning through hypermediabased instruction: The importance of learner characteristics. Computers in Human Behavior, 11(3-4), 451-465. Wepner, S.B. (1991, October-November). The effects of a computerized reading program on "at-risk" secondary students. Paper presented at the Annual Meeting of the College Reading Association in Crystal City, VA (ERIC Document Reproduction Service No. ED340006). Wiley, J. and Voss, J.F. (1999). Constructing arguments from multiple sources: Tasks that promote understanding and not just memory for text. Journal of Educational Psychology, 91(2), 1-11. Wilkie, T. (1994). Examining the effectiveness of instructive animation: A computer learning environment for teaching learning disabled students biology. Unpublished master's paper. McGill University. Windschitl, M. and Andre, T. (1998). Using computer simulations to enhance conceptual change: The roles of constructivist instruction and student epistemological beliefs. Journal of Research in Science Teaching, 35(2), 145-160. 2000 Research Report on the Effectiveness of Technology in Schools 134 Witherspoon, D.N. (1996). A gender analysis of the characteristics and content of written narratives produced by first graders at the word processor in three social groupings. Dissertation Abstracts International, 57/07-A (Order No. AAD96-38582). Woehler, C. (1994). The evaluation of a computer-assisted reading program for at-risk students. Dissertation Abstracts International, 55/07-A (Order No. AAD94-32792). Wood, J.B. (1991). An investigation of the effects of tutorial and tool applications of computer-based education on achievement and attitude in secondary mathematics. Dissertation Abstracts International, 52/06-A (Order No. AAD9132517). Woodruff, E. and Heeler, P. (1990, June). A study of the use of interactive videodisc technology to present aural tests to college music appreciation students. In Ellis, Edwin (Ed.), National Educational Computing Conference Proceedings (11th, Nashville, TN) (ERIC Document Reproduction Service No. ED329232), 306-309. Woodward, J. and Gersten, R. (1992, March-April). Innovative technology for secondary students with learning disabilities. Exceptional Children, 58(5), 407-421. Wright, J.L. and Samaras, A.S. (1986). Play worlds and microworlds. In P.F. Campbell and G.G. Fein (Eds.), Young children and microcomputers. A Reston Book. Englewood Cliffs, NJ: Prentice-Hall, Inc., 73-86. Xin, F. (1993). The effects of video-based macro-contexts in vocabulary learning and reading comprehension for students with learning disabilities. Dissertation Abstracts International, 54/04-A (Order No. AAD93-24263). Yalçinalp, S., Geban, O., and Ozkan, I. (1995). Effectiveness of using computer-assisted supplementary instruction for teaching the mole concept. Journal of Research in Science Teaching, 32(10), 1083-1095. Yang, Y. (1991-1992). The effects of media on motivation and content recall: Comparison of computer- and print-based instruction. Journal of Educational Technology Systems, 20(2), 95-105. Yu, F.Y. (1998). The effects of cooperation with inter-group competition on performance and attitudes in a computerassisted science instruction. Journal of Computers in Mathematics and Science Teaching, 17(4), 381-395. Yu, F. (1996-97). Competition or noncompetition: Its impact on interpersonal relationships in a computer-assisted learning environment. Journal of Educational Technology Systems, 25(1), 13-24. Yusuf, M.M. (1995). The effects of Logo-based instruction. Journal of Educational Computing Research, 12(4), 335362. Yusuf, M.M. (1991, October). Logo based instruction in geometry. Paper presented at the Annual Meeting of the MidWestern Educational Research Association in Chicago, IL (ERIC Document Reproduction Service No. ED348218). Zellermayer, M., Salomon, G., Globerson, T., and Givon, H. (1991). Enhancing writing-related metacognitions through a computerized writing partner. American Educational Research Journal, 28(2), 373-391. Zhang, Y. (1993). Robo-Writer and Microsoft Word: Comparative effects on improvement of writing skills of primary grade LD students. Dissertation Abstracts International, 54/07-A (Order No. AAD93-33991). Zhang, Y., Brooks, D.W., Frields, T., and Redelfs, M. (1995). Quality of writing by elementary students with learning disabilities. Journal of Research on Computing in Education, 27(4), 483-499. Ziegler, J.H., Jr. (1990). The effect of interactive video on learning, perceived effectiveness, and user attitudes in academic library orientation programs. Dissertation Abstracts International, 51/09-A (Order No. AAD91-04797). 2000 Research Report on the Effectiveness of Technology in Schools 135 Education Division Publications Quantity Publication 2000 Research Report on the Effectiveness of Technology in Schools* ($40 members & educators/ $99 non-members) __ 3-5 copies $40 each __ 6-30 copies $35 each __ 31-100 copies $25 each __ 101 or more copies $15 each This research document covers the effectiveness of technology on student achievement, student selfconcept, and teacher-student interaction. (Available on web site for members) ______ ______ 2000 Education Market Report: K-12* ($40 members/$350 non-members/$175 educator) This document summarizes current education technology market research. (Available on web site) ______ 1999 Education Market Report: Post-secondary* ($40 members/$150 non-members/$100 educators) This document summarizes current education technology market research. (Available on web site) ______ State Technology Initiatives Report (Free members/$750 non-members) This report is sent to all members interested in state policy, legislation and education programs. (Also available on web site) ______ Ed*Tech Alerts (Free members/not available to non-members) These alerts contain legislative information and regulatory initiatives affecting the education software market (sent by e-mail). ______ Education Software Management Guide with "Don't Copy That Floppy" ($35 all) This Guide enables educators to maintain software compliance with the law; includes the video. This Guide may be purchased for $20 without the video. ______ Don't Copy That Floppy Video ($20 all) This is for ordering additional copies. "Don't Copy That Floppy" comes with the Education Software Management Guide. ______ Shared Set of Values Video ($20 all) This video is created for high school and college use in orientation and ethics classes. It discusses copyright protection issues in software and printed materials. _______________________________________________________________________________________________________________ To order, please indicate which publications and method of payment below. Name: ___________________________________________________________________________________ Company: ________________________________________________________________________________ Mailing Address: __________________________________________________________________________ City, State, Zip, Country: ____________________________________________________________________ Phone Number: ___________________________ Fax: ____________________________________________ E-mail address: ____________________________________________________________________________ Method of Payment (Check One): ____ Check ____ American Express ____ MasterCard ____ VISA Card Number: ________________________________________________ Expiration Date: ______________ Account Holder's Name: ___________________________ Signature _________________________________ Please send order(s) along with check to: SIIA Customer Service Department, 1730 M Street NW, Suite 700, Washington, DC 20036-4510; fax, phone or e-mail credit card payment to the SIIA Fulfillment Department at +1 (202) 223-8756 (F), +1 (202) 452-1600 (P) or orders@siia.net (E). * Education Division Primary Contact may have 1 printed copy free upon request. 2000 Research Report on the Effectiveness of Technology in Schools 136