Cerficaton of CPS for NCLB

EDU, CPS 1 Educational Underwriters Incorporated Research Portfolio Review Classroom Performance Systems eInstruction Corporation June 2007 Valid for One year Educational Underwriters, Incorporated 131 South Main, Geary, OK 73040. 1-800-753-5496 info@educationalunderwriters.org ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 2 Educational Underwriters, Incorporated 131 West Main Geary, Oklahoma, 73040 405-884-2400 Fax: 405-884-2288 www.educationalunderwriters.org NOTIFICATION OF EdU RESARCH CERTIFICATION June 25, 2007 To Whom It May Concern: Educational Underwriters, Incorporated has fully completed the Research Portfolio Review as requested by eInstruction regarding its Classroom Performance Systems (CPS). Based on the information and research submitted to EdU by your company, we have determined that the research supporting eInstruction’s Classroom Performance Systems does indeed provide strong evidence for improving student achievement. EdU is pleased to inform you that the CPS research portfolio does contain Gold Standard research which meets all requirements currently outlined by the No Child Left Behind Act. In addition to this research, eInstruction has included a substantial body of supporting research and scholarly articles which further support the efficacy of CPS. The EdU Research Review Instrument rates a company’s research based on a 100 point scale. The research portfolio supporting Classroom Performance Systems earned a 92 – a Superior rating. Please review the enclosed report and documentation. Please submit any concerns, needs for clarification, or requests for further documentation within 90 days to Educational Underwriters, Incorporated. Thank you, Tom Deighan President, Educational Underwriters, Inc. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 3 EDUCATIONAL UNDERWRITERS PRODUCT RESEARCH REVIEW INSTRUMENT eInstruction Classroom Performance Systems Component Application Submission ……………………………….…Possible Points: 30 __3___(5 Possible) -- Researchers’ Credentials/ Professional Affiliations __3___(5 Possible) -- Research Publication __5__(5 Possible) -- Product/Company History __3__(5 Possible) -- Company Mission Statement/Educational Philosophy __8__(10 Possible) -- Assurances ______ Total Research Requirements ……………………………………Possible Points: 70 __70__ __50__ __20__ __NA_ (70 Possible) Direct Experimental or Quasi-Experimental (50 Possible) Applied Experimental or Quasi-Experimental (30 Possible) Direct Qualitative or Theoretical Support (20 Possible) Applied Qualitative or Theoretical Support Points Awarded ___22 To qualify for full possible points, all research must satisfy AERA Standards for Reporting on Research Methods: 1. Problem Formulation 2. Design and Logic 3. Sources of Evidence 4. Measurement and Classification 5. Analysis and Interpretation 6. Extrapolation 7. Ethics in Reporting 8. Title, Abstract, and Headings Total Points ___70__ __92___ ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 4 Educational Underwriters Incorporated Research Portfolio Review Classroom Performance Systems eInstruction Corporation June 2007 Educational Underwriters, Incorporated 131 South Main, Geary, OK 73040. 1-800-753-5496 info@educationalunderwriters.org ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS RESEARCH PORTFOLIO REVIEW 5 CLASSROOM PERFORMANCE SYSTEMS, eINSTRUCTION CORPORATION Abstract This white paper was produced by Educational Underwriters, Incorporated (EdU) at the request of eInstruction Corporation for the purpose of documenting and evaluating its research portfolio supporting the effectiveness of Classroom Performance Systems (CPS) as educational tools in classroom settings. Specifically, this paper evaluates whether or not the research behind CPS conforms to No Child Left Behind (NCLB) guidelines, identifies research strengths/weaknesses, and suggests future courses of action regarding the CPS research portfolio. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS A Note to the Reader This white paper is not intended to be an evaluation of CPS or eInstruction Corporation but rather an evaluation of the research specifically cited by eInstruction Corporation as supporting CPS’ effectiveness as an instructional tool. Not intended as a scholarly 6 article, the format of this white paper was chosen to present the research portfolio in such a way that would be easily understood by readers not familiar with research methodology. Background Educational Underwriters, Incorporated was formed through the cooperation of educators, researchers, and vendors as an effort to relieve the burden of NCLB’s research-based requirements on schools and other consumers of educational products. School leaders are not always trained in research methodologies and can often be frustrated when trying to decide if products are supported by scientifically based research (SBR). Educational Underwriters provides a simple ―seal of approval‖ to educational products which meet NCLB standards. How the EdU Process Works Conscientious vendors who submit to the EdU process agree to allow Educational Underwriters to judge the value of the research which supports their products. All membership fees, research review fees, and contracts must be submitted before the process begins – with no guarantee of approval from EdU. EdU then reviews all ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS research, researchers, and claims made by the company as compared to the research presented. 7 The Gold Standard for Research The first priority for Educational Underwriters is that the research complies with federal requirements as outlined in the No Child Left Behind Act. According to NLCB, all research must adhere to the following standards: SCIENTIFICALLY BASED RESEARCH- The term scientifically based research' — (A) means research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs; and (B) includes research that — (i) employs systematic, empirical methods that draw on observation or experiment; (ii) involves rigorous data analyses that are adequate to test the stated hypotheses and justify the general conclusions drawn; (iii) relies on measurements or observational methods that provide reliable and valid data across evaluators and observers, across multiple measurements and observations, and across studies by the same or different investigators; (iv) is evaluated using experimental or quasi-experimental designs in which individuals, entities, programs, or activities are assigned to different conditions and with appropriate controls to evaluate the effects of the condition of interest, with a preference for random-assignment experiments, or other designs to the extent that those designs contain within-condition or across-condition controls; (v) ensures that experimental studies are presented in sufficient detail and clarity to allow for replication or, at a minimum, offer the opportunity to build systematically on their findings; and (vi) has been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review. (United States Department of Ed, 2002) ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS It is the opinion of Educational Underwriters, Incorporated and its members that SBR is a good thing for education, in principle. Unfortunately, interpretation of the 8 NCLB requirements from state-to-state and from program-to-program has varied widely. In complying with NCLB, vendors found themselves in a difficult situation. The burden of proof relating to the research fell on their customers. Schools, specifically administrators faced with NCLB compliance, were not trained in research evaluation. Consequently, good research from conscientious companies was often held suspect, and bad research was often accepted as Gold Standard. Vendors needed to even the field of competition and consumers needed help in clarifying this problem. Educational Underwriters, Incorporated was founded specifically to facilitate consumers’ (public schools) compliance with NCLB’s scientifically research based requirement as outlined in federal legislation. ―Research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs‖ (US Department of Education, 2002) remains the foremost requirement of the law. Other requirements as outlined above further define ideal research – ―systematic . . . rigorous . . . experimental or quasi-experimental . . . rigorous . . . scientific review‖ – certainly provide stringent minimum standards for quality research. NCLB does not, however, devalue the inclusion of qualitative research, literature reviews, or scholarly reports as valuable components of a products research portfolio. Nevertheless, to meet NCLB standard, it is the opinion of Educational Underwriters that a company must include research in its portfolio that relies on ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS experimental or quasi-experimental methods that has been subjected to a peer-reviewed process: The Gold Standard. eInstruction Corporation Founded in 1980 by a pioneer researcher in artificial intelligence and instructional technology, eInstruction Corporation has established itself as a leader in classroom response technology. The company’s products include hardware, software, and internet services. eInstruction recently purchased Exam view, whose services have long been closely connected to eInstruction. 9 Classroom Performance Systems (CPS) CPS systems currently stand as the leading product in student response technology, currently in use in classrooms around the world at all levels of education, from early childhood to graduate school. Military, business, and industry applications also form a substantial portion of CPS sales. Individual CPS pads, referred to as ―clickers‖ by students and teachers, allow students to respond to teacher prompts, assignments, and tests by simply utilizing the pad which resembles a remote control. The basic pad allows students to choose among eight possible choices. These choices correspond with the assignment, test, or discussion determined by the instructor, who pre-programs the CPS for the appropriate number of responses, question type, and other factors. eInstruction has added a line of numeric pads recently, but the basic function of the CPS is not altered. CPS promotes the product aiding instruction in two major areas. First, the CPS system is designed to reduce student anxiety, engage more introverted students, and to provide anonymity for class participation. Secondly, CPS promises to streamline administrative ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 10 duties, facilitate grading, and to increase student feedback (eInstruction, 2007). Together, these functions are designed to increase student learning. The research presented by eInstruction is intended to support this assertion: proper use of CPS in a classroom setting increases student academic achievement. EXAMINATION OF THE RESEARCH The Educational Underwriters, Incorporated research review process considers the evidence presented by a product’s entire Research Portfolio. EdU verifies the researchers’ credentials, reviews the research, and requires full disclosure from the company as is necessary to earn the EdU seal of approval. Ideally, the process determines that a company earns the EdU Seal based on a scale of 0 to 100; the score of 70 being the minimum required score for EdU approval. The Research Review Instrument is included at the beginning of this Research Review; an expanded version with reviewer notes is included in the appendices. Application Submission While this section of the research review would seem most inconsequential to the efficacy of CPS units, it is a vital part of the portfolio review process. Timely, accurate, and detailed submission of applications, contracts, and documentation better enable Educational Underwriters to evaluate the state of a product’s research. This portion of the EdU process documents the researcher’s credentials, publication venues, and overall company educational philosophy. Combined with company assurances, Educational Underwriters can consider a company earnest in their desire to comply with NCLB and, most importantly, serve schools best interests. Application submission enables a ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS company to receive credit for mission statements, company philosophy, and other elements that should not be ignored in such an evaluation. While a company could technically stand solely on its research for EdU certification, member companies value 11 full disclosure of its research portfolio, including the supporting documentation required for complete view of a company’s efforts to support best practices in education. eInstruction fulfilled application submission requirements adequately, receiving 25 out of a possible 30 points. Deductions arose from two areas: Publication and Researcher Credentials. In both instances, the research portfolio presented by eInstruction contained elements which would have garnered them full credit in both areas, but consideration of their Gold Standard study led primarily to this decision. Publication Although one scholarly report submitted by eInstruction was published in Phi Delta Kappan (Black & Wiliam, 1998), this study was not of experimental design. Being based on experimental studies, however, it was considered a positive contribution to the overall legitimacy of the CPS portfolio. Another study (Matassa, 2006) was published in a trade journal and could not be considered peer-reviewed as was a eInstruction commissioned report which detailed many benefits of CPS as well as suggestions for implementation (Thalheimer, 2007). Another article not included for consideration in the application but discovered on eInstruction.com, was presented at a national conference and contained strong experimental elements (Horowitz, 1988). The age of this document and the applied nature does not, however, qualify it for Gold Standard research. Although the Navy Junior ROTC study contained very convincing evidence of CPS effectiveness ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 12 (Lewellyn, 2006), it again was not peer-reviewed. Of great hope was the East Tennessee study (Everett & Ranker, 2002) which presented compelling evidence for CPS efficacy but no independent verification of the article could be discovered. Finally, the Gold Standard study supporting the effectiveness of CPS was an unpublished master’s thesis(Eagle, 2006), and although it was peer-reviewed as an internal operation of Virginia Commonwealth University, it was determined that full credit could not be granted for ―publication‖ at this time. Suggestions for improvement upon the next review cycle have been submitted to eInstruction. Researcher’s Credentials A situation similar to the concerns with publication applied to the researcher’s credentials as well. While no doubt exists as to the high level of professionalism of the cited researchers, the Gold Standard study (Eagle, 2006) was performed by a master’s degree candidate. Compelling evidence aside, the portfolio as a whole could not be considered as strong had a doctorate performed the research and it had been published. The other researchers’ credentials – Ward, Black, Everett, Wiliam, Horowitz, Thalheimer, and Lewellyn – would have weighed more heavily had their studies met the Gold Standard. Everett’s study could possibly have met this standard, but EdU could not find documentation of it outside the eInstruction website. Product/Company History All evidence both submitted by eInstruction and researched by Educational Underwriters supported the assertion that eInstruction Corporation has a solid history and that CPS is a time-tested classroom product. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 13 Company Mission Statement/Educational Philosophy Pending further documentation. Credit awarded based on investigation of eInstruction promotional materials. Assurances eInstruction corporation has complied with all Educational Underwriters, Incorporated documentation and disclosure requirements. Review of the Literature This section discusses each element of the research portfolio individually in order of importance as related to its status as Gold Standard. Educational Underwriters does not intend to assign a hierarchy of value to the research submitted; instead, EdU purposes to present the Gold Standard evidence first. In fact, studies which do not meet NCLB Gold Standard often provide greater proof to practitioners since their format, data, and presentation are easier to understand and to apply to school settings. Other studies will follow in no particular order. Each selection is preceded by full APA citation. EdU rating of each study is indicated as such at the beginning of the discussion.   Gold Standard meets NCLB requirements. Strong Evidence indicates compelling data is included or cited but does not meet NCLB standards.  Strong Theoretical Framework indicates a scholarly article which relies on a review of existing literature and offers no real data of its own. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS  14 APPLIED indicates that the paper supports the learning principles behind CPS but does not constitute research directly involving CPS. Applied can precede any category.  Provides Evidence indicates that a paper has value within the overall portfolio but that it offers little evidence in and of itself. The credit applied for EdU certification is not discussed in this section. Rather, this section serves as further explanation for the ratings recorded in the Research Review Instruments included in the front of this report and as an appendix. GOLD STANDARD STUDY – Marissa Eagle [Eagle, M. (2006). Using multiple linear regressions to evaluate the use of a classroom performance system in an introductory statistics course. Unpublished master's thesis. Virginia Commonwealth University.] This study comprises the best evidence for the efficacy of CPS, according to NCLB standards. Although unpublished and not performed by a doctorate, this study was performed under a stringent review process in an accredited university and under the direct supervision of a doctorate (accepted practice for master’s theses). This study follows a quasi-experimental model. The data from this study were extensively analyzed and evaluated and determined that CPS was effective in improving student achievement: In an effort to determine whether or not the CPS has been a significant factor in a student’s final grade in STAT 210 at VCU, the analysis has shown that it is in fact significant. Based on the data from the eight semesters, by comparing the final grade in the course to whether or not the CPS was used in that semester. The fact that the CPS was used was shown to have a significantly positive effect. (Eagle, 2006) ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 15 This study not only supports its hypothesis that CPS enhanced instruction is effective; it offers strong evidence that CPS enhanced instruction may be more effective than the traditional instructional strategies employed in this course. Strong Evidence – Everett and Ranker [Everett, M. D., & Ranker, R. A. (2002). Classroom response system: evaluation at an easy-access regional university. Retrieved June 4, 2007, from http://www.einstruction.com/WhatisCPS/Research/index.cfm] This study provides very strong evidence supporting CPS efficacy, citing strong data and convincing analysis. Unfortunately, the format presented did not permit adequate review. Further investigation failed to produce a published copy from any scholarly journal or even ERIC, which often documents non-scholarly papers. Despite its strong promise, it falls short in too many areas as delineated by NCLB for SBR. Upon the next annual review, EdU has requested full documentation of this study. Such documentation could, in fact, raise this study to Gold Standard. EdU stresses that this study does offer strong evidence for efficacy. APPLIED Strong Theoretical Framework – Black and Wiliam Black, P., & Wiliam, D. (1998). Inside the black box: raising standards through classroom assessment. Phi Delta Kappan, 80(2), 139-148. This scholarly article provides a solid theoretical framework to support the need for future research regarding the underlying principles which CPS, but it does not constitute Gold Standard research. The researchers cite it as a ―prima facie case.‖ (p. 1) Black and Wiliam do not mention CPS in this study nor any other instructional device. Rather, they promote the efficacy of teaching strategies that eInstruction has applied to CPS. The need for constant and diligent formative assessment in the classroom – teacher ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS assessment and student self-assessment – seems to be the underlying concepts which 16 compliment CPS. Furthermore, this study relies entirely on data from studies published from 1986 to 1998, the vast-majority of them falling below the ten-year-old mark. Nonetheless, the assertions of this article do align logically with CPS’s promoted benefits. And since CPS has established sufficient evidence through at least one Gold Standard study (Eagle, 2006), this scholarly article presents a strong theoretical framework which can be applied to CPS. Provides Evidence Thalheimer, W. (2007). Questioning strategies for audience response systems: how to use questions to maximize learning, engagement, and satisfaction. Retrieved June12, 2007, from http://www.work-learning.com/catalog/ Thalheimer’s piece on CPS was commissioned directly from eInstruction. And while it does cite scholarly journals and studies, EdU considers this more of a promotional tool as opposed to scholarly research. EdU does not discount the integrity of Thalheimer nor the verity of his assertions. Rather, the classification of this paper discounts its value in meeting NCLB requirements. As a reference, however, it does earn a place in the CPS research portfolio. Strong Theoretical Framework -- Ward Ward, D. (2003). The classroom response system: the overwhelming research results supporting this teacher tool and methodology. Retrieved June 7, 2007, from http://www.einstruction.com/WhatisCPS/Research/index.cfm As the founder of the company, Darrell Ward definitely has a vested interest in proving the efficacy of CPS in the classroom setting (Ward, 2003). Within this piece, he cites several studies and builds a very convincing case for the effectiveness of CPS. Unfortunately, this paper must be considered promotional material. Several of the ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 17 references included are promising. EdU suggests that some of these studies be submitted in the following review process as independent components of the CPS Research Portfolio. Strong Evidence APPLIED -- Horowitz Horowitz, H. M. (1988). IBM study proves use of student response systems increases attentiveness, Sixth Conference of Interactive Instruction Delivery for the Society of Applied Learning Technology: Society of Applied Learning Technology. Horowitz’s study, commissioned by IBM, was nevertheless presented at a scholarly conference (a point which could not be verified by EdU). The study utilizes student response pads provided by Reactive Systems, Incorporated. Information on Reactive’s response pads could not be readily found, but the system was referenced in another study in the CPS portfolio (Everett & Ranker, 2002). In this case, evidence was good and the study does seem to employ quasi-experimental methods. Unfortunately, no independent verification of this article could be found by EdU. A similar article by Horowitz was found, but no article bearing this title. eInstruction did not include this article as part of its research portfolio. EdU discovered it on their website and decided to include it as it does offer strong supporting evidence for a student response mechanism such as CPS. Provides Evidence – Matassa Matassa, M. (2006). Just a click away. Edtechmag.com, March/April, 19-21. Two things primarily discounted the value of this article. First, its venue of publication – a publication which carries information on products sold through CDW-G. Second, the article itself states ―The results were not statistically significant because of the target ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS group’s small size.‖ (p. 21) These factors do not de-value its contribution entirely, however. Documentation of CPS’s implementation in a large school district definitely constitutes evidence in the opinion of EdU. From a practitioner’s standpoint, this may actually constitute even better evidence than Gold Standard research. The results are 18 certainly impressive, regardless of sample size or publication. Unfortunately, NCLB does not yet recognize this type of ―action research.‖ Provides Evidence -- Lewellyn Lewellyn, D. A. (2006). Report on the classroom performance system for use in the navy junior reserve officer training program United States Navy Junior Reserve Officer Training Program (http://www.einstruction.com/WhatisCPS/Research/index.cfm) Although this research is unpublished and not peer-reviewed, it again holds value in the overall research portfolio as providing ―real-world‖ evidence of CPS’s implementation. This article, more of an internal report, does outline strong reasons for the Junior ROTC to continue using CPS in educational settings. It does not meet NLCB standards, but it does deserve to be considered as supporting evidence within CPS’s overall research portfolio. If peer-reviewed with further documentation, this article may be considered as Gold Standard. Conclusion eInstruction Corporation has assembled strong evidence supporting the efficacy of the CPS in improving student achievement. The NCLB Gold Standard for research has been satisfied through a university sponsored, direct study involving CPS in classroom a classroom setting. Further research within the CPS research portfolio strongly supports the effectiveness of CPS in a variety of settings and under a variety of circumstances. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Additional theoretical evidence is supplied as well. Applied research and studies complete their portfolio in such a way that eInstruction earns solid marks from Educational Underwriters, Incorporated. CPS deserves the EdU Seal of Approval. 19 Electronic copies of this paper and the entire CPS research portfolio are available online at www.educationalunderwriters.org. Any questions regarding this white paper, the EdU review process, EdU, or the CPS research portfolio may be directed to Educational Underwriters, Incorporated 131 South Main, Geary, OK 73040. 1-800-753-5496 info@educationalunderwriters.org ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 20 References Black, P., & Wiliam, D. (1998). Inside the black box: raising standards through classroom assessment. Phi Delta Kappan, 80(2), 139-148. Eagle, M. (2006). Using multiple linear regression to evaluate the use of a classroom performance system in an introductory statistics course. Unpublished master's thesis. Virginia Commonwealth University, from http://www.einstruction.com/WhatisCPS/Research/index.cfm Everett, M. D., & Ranker, R. A. (2002). Classroom response system: evaluation at an easy-access regional university. Retrieved June 4, 2007, from http://www.einstruction.com/WhatisCPS/Research/index.cfm Horowitz, H. M. (1988). IBM study proves use of student response systems increases attentiveness, Sixth Conference of Interactive Instruction Delivery for the Society of Applied Learning Technology: Society of Applied Learning Technology. Lewellyn, D. A. (2006). Report on the classroom performance system for use in the navy junior reserve officer training program United States Navy Junior Reserve Officer Training Program (http://www.einstruction.com/WhatisCPS/Research/index.cfm). Matassa, M. (2006). Just a click away. Edtechmag.com, March/April, 19-21. Thalheimer, W. (2007). Questioning strategies for audience response systems: how to use questions to maximize learning, engagement, and satisfaction. Retrieved June12, 2007, from http://www.work-learning.com/catalog/ ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS United States Department of Education (2002). Retrieved March 5, 2006 from http://www.ed.gov/nclb/methods/whatworks/edpicks.jhtml?src=ln 21 Ward, D. (2003). The classroom response system: the overwhelming research results supporting this teacher tool and methodology. Retrieved June 7, 2007, from http://www.einstruction.com/WhatisCPS/Research/index.cfm ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS APPENDICES TABLE OF CONTENTS APPENDIX A – EXPANDED RESEARCH REVIEW INSTRUMENT WITH REVIEWER NOTES 22 APPENDIX B – eINSTRUCTION CORPORATION APPLICATION SUBMISSTION APPENDIX C – RESEARCH STUDIES ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 23 APPENDIX A EXPANDED RESEARCH REVIEW INSTRUMENT ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 24 EDUCATIONAL UNDERWRITERS PRODUCT RESEARCH REVIEW INSTRUMENT Component Points Awarded Application Submission ……………………………….…Possible Points: 30 22 ___3__(5 Possible) – Researcher(s)’ Credentials/ Professional Affiliations Research Reviewer Comments: The individuals cited below hold impressive credentials and have established, in most cases, successful careers. Only two of the studies, however, cited represent research which could meet the experimental or quasi-experimental definition. Both of those studies were performed by Masters’ degree holders and were not published. Marrissa Eagle’s study, while not being performed by a doctorate holder, was conducted at Virginia Commonwealth University under the direct supervision of a PhD – and therefore meets the gold standard. Had at another study met the Gold Standard with a PhD. Or Ed.D head researcher, the full points could have been awarded. Deduction of two points from the possible score was necessary to maintain the integrity of the research review since the majority of research presented by E-Instruction represents less than Gold Standard research. Paul Black, PhD    Dept of Physics at U of Birmingham (England) Director of the Centre for Science & Math @ Chelsea College in London Head of the King’s Centre for Educational Studies (KQC) Dylan Wiliam, PhD. University of London, 1993 Marrisa Eagle, MS, Virginia Commonwealth University ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Will Thalheimer, PhD. Columbia University 1996     American Educational Research Association American Psychological Association American Psychological Society Dissertation: Information-acquisition goals: How questions 25 produce learning through non-strategic processing. CDR David A. Lewellyn USN (ret), M. Ed Darrell Ward, PhD, Texas A&M University, 1973   Dissertation: Artificial Intelligence in Learning Programming Founder, E-Instruction Corporation ___2__(5 Possible) -- Research Publication Research Reviewer Comments: One study as cited by E-Instruction Corporation was published (Black and Wiliam) in Phi Delta Kappan; however, that study was not experimental or quasi-experimental research but was based on data which relates to Classroom Response Systems. The Navy Junior Reserve Officer Training Program study does follow a quasiexperimental model and presents very compelling evidence but was neither published nor peer-reviewed. Just a Click Away, despite showing compelling evidence of effectiveness as a pilot study of the Boulder Colorado School District, was published in a trade magazine. Thalheimer’s E-Instruction commissioned report was identified as such and contains references to research, but does not meet the standard for experimental/quasi-experimental research. Although research publication included Phi Delta Kappan, that paper was not a Gold Standard study. None of the remaining studies were published in peer-reviewed journals and represented reports. reviewed (as a thesis requirement) but not published. The one Therefore, exception was the Virginia Commonwealth study which was peer- ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS only two points out of a possible five could be awarded for research publication. 26 ___5___(5 Possible) -- Product/Company History E-Instruction Corporation, even as a relatively new company (established 1980), has a history of providing quality educational products which have been implemented in school settings very successfully worldwide. The Classroom Performance System, the company’s flagship product, is established as the leader of the student-response industry and boasts an impressive track record in schools, with teachers, and with students – over 3,000,000 users. 5-year rule fully satisfied. __4___(5 Possible) -- Company Mission Statement/Educational Philosophy Research Reviewer Comments: Documentation Pending, but points awarded based on www.einstruction.com website information, founder Darrell Ward’s statement, company history – and company assurances of pending documentation. __8___(10 Possible) -- Assurances ―Assurances‖ refers to the overall completeness of the application, submission of supporting documents, and appropriateness of the information provided. Reviewer Note: Phonic Ear, Inc., Front Row Division application submission complete: ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Membership Agreement IP Agreement Partner Registration Researcher Information Research Documents Review Application *Mission Statement (documentation pending, possible clerical error) Reviewer’s comments: Several areas of documentation were minimal and other areas were missing – fully expected in the first application round. Recommendations for improvement will be provided eInstruction for future reviews. 27 Application Submission Total_25_____ Research Requirements ……………………………………Possible Points: 70 __70____ (70 Possible) Direct Experimental or Quasi-Experimental The Eagle study does satisfy NCLB Gold Standard requirements. It is a master’s thesis but was executed under the direct supervision of a doctorate and conducted under the covering of an accredited university. The design, data, and analysis fit NCLB standards. As for publishing venue, it meets the peer-review requirement through the internal review processes inherent in VCU’s Master’s Degree Thesis requirements. ___70 __(25)___ (50 Possible) Applied Experimental or Quasi-Experimental eInstruction included several studies within its portfolio which could possibly meet NCLB standards with further documentation and/or the implementation of peer review processes. Black & Wiliam, Lewellyn, and Horowitz all provided strong evidence which did not necessarily ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS meet NCLB requirements but did provide strong evidence of efficacy. Inclusion of these points for the EdU process is insignificant since eInstruction has a study which meets the Gold Standard. ___(20)___ (30 Possible) Direct Qualitative or Theoretical Support Ward, Thalheimer, and Matassa all provided papers based on solid studies which were based on solid data – many of which may meet NCLB standards if submitted separately. ______ (20 Possible) Applied Qualitative or Theoretical Support Although some of the studies included could have fit into this area as well as others, eInstruction has met all required points and those studies were addressed within those areas. To qualify for full possible points, all research must satisfy AERA Standards for Reporting on Research Methods: 9. Problem Formulation 10. Design and Logic 11. Sources of Evidence 12. Measurement and Classification 13. Analysis and Interpretation 14. Extrapolation 15. Ethics in Reporting 16. Title, Abstract, and Headings Total Points EdU Rating: EdU review components comprise a 100 point score. Products which are listed receive a rating based on their EdU review: 90 -100 Superior 80-89 Excellent 70-79 Satisfactory 69 or Below – Provisional: Improvements, additions, or clarifications need to be made according to reviewer recommendations. (See accompanying EdU Report.) 28 __92____ Rating: SUPERIOR ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 29 APPENDIX B eINSTRUCTION APPLICATION FOR RESEARCH REVIEW ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 30 Educational Underwriters, Incorporated Research Review Application EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org EDU ID# DATE: Company Name: eInstruction Address: 308 North Carroll Blvd City: Denton Telephone: 940-565-0004 Fax: 940-565-0959 Website: www.einstruction.com State: TX ZIP Code: 76201 Authorized Company Representative: Michael Paul Direct telephone/extension: 940-565-0004 ext 307 E-mail: michael.paul@einstruction.com Product Information Name of Product: CPS (Classroom Performance System) Product Description (Brief Description only. Please Attach Additional Documentation as needed.): A software/hardware product that allows students to use handheld remotes to reply to instructor questions in class. The CPS software allows instructors to author questions and gather data from students using the handheld remotes. Number of years product has been on the market: 23 schools: 1987 Year product was introduced to Target Populations (Check all that Apply) Target Age Group: X Product not age specific All school ages 0-3 3-6 6-10 10-13 Target Grade(s): X Suitable for all grades Early Elementary Elementary Middle/Junior High High School College Target Demographics: X Special Education X Regular Education X Gifted and Talented X ESL ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 31 Educational Underwriters, Incorporated Research Review Application EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org 13-16 16-18 18+ Other: Graduate Grade Specific: Other: X Economically Disadvantaged X Male X Female X Other: Research Study Information Please list all product related research in APA Format. Each individual study requires a Research Documentation Form 1. 2004 The Classroom Performance System – A White Paper by Dr. Darrell Ward 2. Inside the Black Box by Black & Wiliam 3. Just a Click by Mical Matassa 4. NJROTC with CPS report by CDR David Lewellyn USN (ret) M. Ed 5. Questioning for Audience Response Systems by Will Thalheimer PhD 6. Using Multiple Linear Regression to Evaluate the Use of a Classroom Performance System in an Introductory Statistics Course If more than five studies are being submitted for review, please enumerate and continue on this sheet or add as attachment. Checklist X Contract with Educational Underwriters completed. X Research Documentation Form completed for each individual study. X Payment already remitted or PO established SIGNATURES/ASSURANCES Company Representative: Printed Name_______________________________________Title__________________ For EdU Use Only: Signature _______________________________________ Date_________________ ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 32 Educational Underwriters, Incorporated Research Review Application (Form is best completed electronically and then printed for signatures.) EDU ID# DATE: Company Name: Address: City: Denton eInstruction 308 North Carroll Blvd State: TX ZIP Code: 76201 Telephone: 940-565-0004 Fax: 940-565-0959 Website: www.einstruction.com Authorized Company Representative: Michael Paul ________________________________________________ Direct telephone/extension: 940-565-0004 E-mail: michael.paul@einstruction.com RESEARCHED PRODUCT(S) ext 307 (Additional Product Information Form Needed for Each Individual Product) Name of Product(s) with research to be reviewed: 1. CPS (Classroom Performance System) 2. 3. 4. 5. 6. 7. 8. 9. 10. (If more than 10 products’ research is to be reviewed, please enumerate and include in this section or add as an attachment.) Is company fewer than five years old? X No Year Company Gross Revenue latest Founded: 1988 fiscal Year in USD: 40,176,237 Is company a subsidiary or division of a larger company? X No Address of Company Tel: Name of Parent Company (if any): Headquarters if different than eInstruction Holdings, Inc above: Fax: Parent Company Contact and Title: Name:Tim Torno ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 33 Educational Underwriters, Incorporated Research Review Application (Form is best completed electronically and then printed for signatures.) Title: CFO Tel: 940-565-0004 ext 319 Checklist X Contract with Educational Underwriters completed – EDU ID# X Documentation/Statement of Gross Revenue attached. X Product Research Form for each individual product must be completed X Company Mission Statement (if available) X All products for which company needs research reviewed have been listed above or attached. X Payment arrangements/PO has been prepared. SIGNATURES/ASSURANCES Company Representative: ________________________________ Defects found on inspection: Status after Service: (Please circle) Complete/ Incomplete/ Pending for spares/ Under Observation/ Working solution provided Events: ( Date & Time) Start of Service: End of service: PLEASE RATE THIS CALL BY TICKING AN OPTION Extremely Satisfied Remarks: Satisfied Dissatisfied CUSTOMER FEEDBACK Annoyed Name : Email: Designation: Phone/Fax: ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 34 Educational Underwriters, Incorporated Research Review Application (Form is best completed electronically and then printed for signatures.) Signature: Date: Place: ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 35 Educational Underwriters, Incorporated Partner Registration EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org EDU ID# DATE: Company Name: eInstructiohn Address: 308 North Carroll Blvd. City: Denton Telephone: 940-565-0004 Fax: 940-565-0959 Website: www.einstruction.com State: TX ZIP Code: 76201 Authorized Company Representative: Michael Paul Direct telephone/extension: 940-565-0004 307 E-mail: michael.paul@einstruction.com RESEARCHED PRODUCT(S) ext (Additional Product Information Form Needed for Each Individual Product) Product(s) with research to be reviewed: Partner Only. No research to be reviewed. 1. CPS (Classroom Performance System) 2. 3. 4. 5. 6. 7. 8. 9. 10. (If more than 10 products’ research is to be reviewed, please enumerate and include in this section or add as an attachment.) Is company fewer than five years old? NO Year Founded: Company Gross Revenue 1988 latest fiscal Year in USD: 40,176,237 Is company a subsidiary or division of a larger company? NO ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 36 Educational Underwriters, Incorporated Partner Registration EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org Address of Parent Company Headquarters if different than above: Tel: Fax: Name of Parent Company (if any): eInstruction Holdings, Inc Parent Company Contact and Title: Name:Tim Torno Title: CFO Tel: 940-565-0004 ext 319 Checklist X Contract with Educational Underwriters completed – EDU ID# X Documentation/Statement of Gross Revenue attached. X Product Research Form for each individual product must be completed X Company Mission Statement (if available) X All products for which company needs research reviewed have been listed above or attached. X Payment arrangements/PO has been prepared. SIGNATURES/ASSURANCES Company Representative: Printed Name____Michael Paul___________Title_VP of Strategic Partnerships For EdU Use Only: Signature _______________________________________ Date__5-3-07 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 37 Educational Underwriters, Incorporated Research Review Application EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org EDU ID# DATE: Company Name: eInstruction Address: 308 North Carroll Blvd City: Denton Telephone: 940-565-0004 Fax: 940-565-0959 Website: www.einstruction.com State: TX ZIP Code: 76201 Authorized Company Representative: Michael Paul Direct telephone/extension: 940-565-0004 ext 307 E-mail: michael.paul@einstruction.com Product Information Name of Product: CPS (Classroom Performance System) Product Description (Brief Description only. Please Attach Additional Documentation as needed.): A software/hardware product that allows students to use handheld remotes to reply to instructor questions in class. The CPS software allows instructors to author questions and gather data from students using the handheld remotes. Number of years product has been on the market: 23 schools: 1987 Year product was introduced to Target Populations (Check all that Apply) Target Age Group: X Product not age specific All school ages 0-3 3-6 6-10 10-13 Target Grade(s): X Suitable for all grades Early Elementary Elementary Middle/Junior High High School College Target Demographics: X Special Education X Regular Education X Gifted and Talented X ESL ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 38 Educational Underwriters, Incorporated Research Review Application EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org 13-16 16-18 18+ Other: Graduate Grade Specific: Other: X Economically Disadvantaged X Male X Female X Other: Research Study Information Please list all product related research in APA Format. Each individual study requires a Research Documentation Form 1. 2004 The Classroom Performance System – A White Paper by Dr. Darrell Ward 2. Inside the Black Box by Black & Wiliam 3. Just a Click by Mical Matassa 4. NJROTC with CPS report by CDR David Lewellyn USN (ret) M. Ed 5. Questioning for Audience Response Systems by Will Thalheimer PhD 6. Using Multiple Linear Regression to Evaluate the Use of a Classroom Performance System in an Introductory Statistics Course If more than five studies are being submitted for review, please enumerate and continue on this sheet or add as attachment. Checklist X Contract with Educational Underwriters completed. X Research Documentation Form completed for each individual study. X Payment already remitted or PO established SIGNATURES/ASSURANCES Company Representative: Printed Name_______________________________________Title__________________ For EdU Use Only: Signature _______________________________________ Date_________________ ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 39 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org EDU ID# DATE: Company Name: eInstruction Product Name: CPS (Classroom Performance System) Research Information Name of Study: Report on the Classroom Performance System for use in the Navy Junior Reserve Officer Training Program Is study a dissertation? No Research Type: Experimental X Quasi-Experimental Applied Experimental X Qualitative Applied Qualitative Other: APA Format Publication Information (If any): Presentation Venue and Date (If any): Researcher Information Researcher Name: CDR David A. Lewellyn USN (ret), M. Ed. Researcher: Affiliated Professional Organization Memberships: 1. College or University: Degree Attained: Graduation Date: College or University: Graduation Date: Dissertation or Thesis Title: Degree Attained: Dissertation or Thesis Title: Lead College or University: Degree Attained ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 40 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org Graduation Date: Dissertation or Thesis Title: Researcher Information Researcher Name: Co-Researcher Affiliated Professional Organization Memberships: 2. College or University: Degree Attained: Graduation Date: Dissertation or Thesis Title: College or University: Degree Attained: Graduation Date: Dissertation or Thesis Title: College or University: Degree Attained: Graduation Date: Dissertation or Thesis Title: College or University: Degree Attained: Graduation Date: Dissertation or Thesis Title: Researcher Information Researcher Name: Lead/Co-Researcher Yes No Affiliated Professional Organization Memberships: 1. 2. College or University: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 41 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org Graduation Date: Dissertation or Thesis Title: Researcher Information Researcher Name: Researcher Yes No Affiliated Professional Organization Memberships: 1. 2. College or University: Lead/Co- Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: Dissertation or Thesis Title: ADDITIONAL RESEARCHER INFORMATION MAY BE ATTACHED IF NECESSARY SIGNATURES/ASSURANCES Company Representative: Printed Name_______________________________________Title__________________ For EdU Use Only: Signature _______________________________________ Date_________________ ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 42 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org EDU ID# DATE: Company Name: eInstruction Product Name: CPS (Classroom Performance System) Research Information Name of Study: Questioning Strategies for Audience Response Systems Is study a dissertation? No Research Type: Experimental Quasi-Experimental Applied Experimental X Qualitative Applied Qualitative Other: APA Format Publication Information (If any): Presentation Venue and Date (If any): Researcher Information Researcher Name: Will Thalheimer Affiliated Professional Organization Memberships: 2. 3. 4. American Educational Research Association American Psychological Association American Psychological Society Lead College or University: Columbia University Degree Attained: PhD Graduation Date: 1996 Dissertation or Thesis Title: Information-acquisition goals: How questions produce learning through non-strategic processing. College or University: Drexel University Graduation Date: 1986 Dissertation or Thesis Title: Degree Attained: MBA ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 43 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org College or University: Pennsylvania State Attained: BA Graduation Date: 1982 Dissertation or Thesis Title: Degree Researcher Information Researcher Name: Lead/Co-Researcher Affiliated Professional Organization Memberships: 1. 2. College or University: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: Dissertation or Thesis Title: Researcher Information Researcher Name: Lead/Co-Researcher Yes No Affiliated Professional Organization Memberships: 1. 2. College or University: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 44 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: Dissertation or Thesis Title: Researcher Information Lead/Co- Researcher Name: Researcher Yes No Affiliated Professional Organization Memberships: 1. 2. College or University: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: Dissertation or Thesis Title: ADDITIONAL RESEARCHER INFORMATION MAY BE ATTACHED IF NECESSARY SIGNATURES/ASSURANCES Company Representative: Printed Name_______________________________________Title__________________ For EdU Use Only: Signature _______________________________________ ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 45 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org Date_________________ ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 46 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org EDU ID# DATE: Company Name: eInstruction Product Name: CPS (Classroom Performance System) Research Information Name of Study: Using Multiple Linear Regression to Evaluate the Use of a Classroom Performance System in an Introductory Statistics Course Is study a dissertation? No Research Type: Experimental X Quasi-Experimental Applied Experimental X Qualitative Applied Qualitative Other: APA Format Publication Information (If any): Presentation Venue and Date (If any): Researcher Information Researcher Name: Marissa Eagle Researcher: Lead Affiliated Professional Organization Memberships: 5. College or University: Virginia Commonwealth University Graduation Date: 2004 Dissertation or Thesis Title: College or University: Virginia Commonwealth University Graduation Date: 2006 Dissertation or Thesis Title: College or University: Degree Attained Graduation Date: Dissertation or Thesis Title: Degree Attained: BS Degree Attained: MS ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 47 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org Researcher Information Researcher Name: Co-Researcher Affiliated Professional Organization Memberships: 2. College or University: Degree Attained: Graduation Date: Dissertation or Thesis Title: College or University: Degree Attained: Graduation Date: Dissertation or Thesis Title: College or University: Degree Attained: Graduation Date: Dissertation or Thesis Title: College or University: Degree Attained: Graduation Date: Dissertation or Thesis Title: Researcher Information Researcher Name: Lead/Co-Researcher Yes No Affiliated Professional Organization Memberships: 1. 2. College or University: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: Dissertation or Thesis Title: ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 48 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org Researcher Information Researcher Name: Researcher Yes No Affiliated Professional Organization Memberships: 1. 2. College or University: Lead/Co- Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: Dissertation or Thesis Title: ADDITIONAL RESEARCHER INFORMATION MAY BE ATTACHED IF NECESSARY SIGNATURES/ASSURANCES Company Representative: Printed Name_______________________________________Title__________________ For EdU Use Only: Signature _______________________________________ Date_________________ ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 49 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org EDU ID# DATE: Company Name: eInstruction Product Name: CPS (Classroom Performance System) Research Information Name of Study: The Classroom Performance System: The Overwhelming Research Results Supporting This Teacher Tool and Methodology Is study a dissertation? No Research Type: Experimental X Quasi-Experimental Applied Experimental X Qualitative Applied Qualitative Other: APA Format Publication Information (If any): Presentation Venue and Date (If any): Researcher Information Researcher Name: Darrell Ward Lead/Co-Researcher Affiliated Professional Organization Memberships: 6. College or University: Baylor University Graduation Date: 1966 Yes Degree Attained: BA Dissertation or Thesis Title: Degree Attained: MS College or University: University of Iowa Graduation Date: 1968 Dissertation or Thesis Title: College or University: Texas A&M Graduation Date: 1973 Degree Attained: PhD Dissertation or Thesis Title: Artificial Intelligence in Learning ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 50 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org Programming Researcher Information Researcher Name: Lead/Co-Researcher Affiliated Professional Organization Memberships: 1. 2. College or University: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: Dissertation or Thesis Title: Researcher Information Researcher Name: Lead/Co-Researcher Yes No Affiliated Professional Organization Memberships: 1. 2. College or University: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: Dissertation or Thesis Title: ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 51 Educational Underwriters, Incorporated Research Documentation EDUCATIONAL UNDERWRITERS INC. 131 W MAIN, GEARY, OK 73040 TEL: 405-884-2400 FAX: 405-884-2288 www.educationalunderwriters.org College or University: Degree Attained: Graduation Date: Dissertation or Thesis Title: Researcher Information Lead/Co- Researcher Name: Researcher Yes No Affiliated Professional Organization Memberships: 1. 2. College or University: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: College or University: Dissertation or Thesis Title: Degree Attained: Graduation Date: Dissertation or Thesis Title: ADDITIONAL RESEARCHER INFORMATION MAY BE ATTACHED IF NECESSARY SIGNATURES/ASSURANCES Company Representative: Printed Name_______________________________________Title__________________ For EdU Use Only: Signature _______________________________________ Date_________________ ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 52 APPENDIX C RESEARCH DOCUMENTATION ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 53 Title: Inside the Black Box., By: Black, Paul, Wiliam, Dylan, Phi Delta Kappan, 00317217, Oct98, Vol. 80, Issue 2 Database: Academic Search Elite Inside the Black Box: Raising Standards Through Classroom Assessment By Paul Black and Dylan Wiliam Firm evidence shows that formative assessment is an essential component of classroom work and that its development can raise standards of achievement, Mr. Black and Mr. Wiliam point out. Indeed, they know of no other way of raising standards for which such a strong prima facie case can be made. Raising the standards of learning that are achieved through schooling is an important national priority. In recent years, governments throughout the world have been more and more vigorous in making changes in pursuit of this aim. National, state, and district standards; target setting; enhanced programs for the external testing of students' performance; surveys such as NAEP (National Assessment of Educational Progress) and TIMSS (Third International Mathematics and Science Study); initiatives to improve school planning and management; and more frequent and thorough inspection are all means toward the same end. But the sum of all these reforms has not added up to an effective policy because something is missing. Learning is driven by what teachers and pupils do in classrooms. Teachers have to manage complicated and demanding situations, channeling the personal, emotional, and social pressures of a group of 30 or more youngsters in order to help them learn immediately and become better learners in the future. Standards can be raised only if teachers can tackle this task more effectively. What is missing from the efforts alluded to above is any direct help with this task. This fact was recognized in the TIMSS video study: "A focus on standards and accountability that ignores the processes of teaching and learning in classrooms will not provide the direction that teachers need in their quest to improve."1 In terms of systems engineering, present policies in the U.S. and in many other countries seem to treat the classroom as a black box. Certain inputs from the outside -- pupils, teachers, other resources, management rules and requirements, parental anxieties, standards, tests with high stakes, and so on -- are fed into the box. Some outputs are supposed to follow: pupils who are more knowledgeable and competent, better test ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 54 results, teachers who are reasonably satisfied, and so on. But what is happening inside the box? How can anyone be sure that a particular set of new inputs will produce better outputs if we don't at least study what happens inside? And why is it that most of the reform initiatives mentioned in the first paragraph are not aimed at giving direct help and support to the work of teachers in classrooms? The answer usually given is that it is up to teachers: they have to make the inside work better. This answer is not good enough, for two reasons. First, it is at least possible that some changes in the inputs may be counterproductive and make it harder for teachers to raise standards. Second, it seems strange, even unfair, to leave the most difficult piece of the standards-raising puzzle entirely to teachers. If there are ways in which policy makers and others can give direct help and support to the everyday classroom task of achieving better learning, then surely these ways ought to be pursued vigorously. This article is about the inside of the black box. We focus on one aspect of teaching: formative assessment. But we will show that this feature is at the heart of effective teaching. The Argument We start from the self-evident proposition that teaching and learning must be interactive. Teachers need to know about their pupils' progress and difficulties with learning so that they can adapt their own work to meet pupils' needs -- needs that are often unpredictable and that vary from one pupil to another. Teachers can find out what they need to know in a variety of ways, including observation and discussion in the classroom and the reading of pupils' written work. We use the general term assessment to refer to all those activities undertaken by teachers -- and by their students in assessing themselves -- that provide information to be used as feedback to modify teaching and learning activities. Such assessment becomes formative assessment when the evidence is actually used to adapt the teaching to meet student needs.2 There is nothing new about any of this. All teachers make assessments in every class they teach. But there are three important questions about this process that we seek to answer:    Is there evidence that improving formative assessment raises standards? Is there evidence that there is room for improvement? Is there evidence about how to improve formative assessment? In setting out to answer these questions, we have conducted an extensive survey of the research literature. We have checked through many books and through the past nine years' worth of issues of more than 160 journals, and we have studied earlier reviews of research. This process yielded about 580 articles or chapters to study. We prepared a lengthy review, using material from 250 of these sources, that has been published in a special issue of the journal Assessment in Education, together with comments on our ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 55 work by leading educational experts from Australia, Switzerland, Hong Kong, Lesotho, and the U.S.3 The conclusion we have reached from our research review is that the answer to each of the three questions above is clearly yes. In the three main sections below, we outline the nature and force of the evidence that justifies this conclusion. However, because we are presenting a summary here, our text will appear strong on assertions and weak on the details of their justification. We maintain that these assertions are backed by evidence and that this backing is set out in full detail in the lengthy review on which this article is founded. We believe that the three sections below establish a strong case that governments, their agencies, school authorities, and the teaching profession should study very carefully whether they are seriously interested in raising standards in education. However, we also acknowledge widespread evidence that fundamental change in education can be achieved only slowly -- through programs of professional development that build on existing good practice. Thus we do not conclude that formative assessment is yet another "magic bullet" for education. The issues involved are too complex and too closely linked to both the difficulties of classroom practice and the beliefs that drive public policy. In a final section, we confront this complexity and try to sketch out a strategy for acting on our evidence. Does Improving Formative Assessment Raise Standards? A research review published in 1986, concentrating primarily on classroom assessment work for children with mild handicaps, surveyed a large number of innovations, from which 23 were selected.4 Those chosen satisfied the condition that quantitative evidence of learning gains was obtained, both for those involved in the innovation and for a similar group not so involved. Since then, many more papers have been published describing similarly careful quantitative experiments. Our own review has selected at least 20 more studies. (The number depends on how rigorous a set of selection criteria are applied.) All these studies show that innovations that include strengthening the practice of formative assessment produce significant and often substantial learning gains. These studies range over age groups from 5-year-olds to university undergraduates, across several school subjects, and over several countries. For research purposes, learning gains of this type are measured by comparing the average improvements in the test scores of pupils involved in an innovation with the range of scores that are found for typical groups of pupils on these same tests. The ratio of the former divided by the latter is known as the effect size. Typical effect sizes of the formative assessment experiments were between 0.4 and 0.7. These effect sizes are larger than most of those found for educational interventions. The following examples illustrate some practical consequences of such large gains. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS  56  An effect size of 0.4 would mean that the average pupil involved in an innovation would record the same achievement as a pupil in the top 35% of those not so involved. An effect size gain of 0.7 in the recent international comparative studies in mathematics5 would have raised the score of a nation in the middle of the pack of 41 countries (e.g., the U.S.) to one of the top five. Many of these studies arrive at another important conclusion: that improved formative assessment helps low achievers more than other students and so reduces the range of achievement while raising achievement overall. A notable recent example is a study devoted entirely to low-achieving students and students with learning disabilities, which shows that frequent assessment feedback helps both groups enhance their learning.6 Any gains for such pupils could be particularly important. Furthermore, pupils who come to see themselves as unable to learn usually cease to take school seriously. Many become disruptive; others resort to truancy. Such young people are likely to be alienated from society and to become the sources and the victims of serious social problems. Thus it seems clear that very significant learning gains lie within our grasp. The fact that such gains have been achieved by a variety of methods that have, as a common feature, enhanced formative assessment suggests that this feature accounts, at least in part, for the successes. However, it does not follow that it would be an easy matter to achieve such gains on a wide scale in normal classrooms. Many of the reports we have studied raise a number of other issues.     All such work involves new ways to enhance feedback between those taught and the teacher, ways that will require significant changes in classroom practice. Underlying the various approaches are assumptions about what makes for effective learning -- in particular the assumption that students have to be actively involved. For assessment to function formatively, the results have to be used to adjust teaching and learning; thus a significant aspect of any program will be the ways in which teachers make these adjustments. The ways in which assessment can affect the motivation and self-esteem of pupils and the benefits of engaging pupils in self-assessment deserve careful attention. Is There Room for Improvement? A poverty of practice. There is a wealth of research evidence that the everyday practice of assessment in classrooms is beset with problems and shortcomings, as the following selected quotations indicate.  "Marking is usually conscientious but often fails to offer guidance on how work can be improved. In a significant minority of cases, marking reinforces underachievement and underexpectation by being too generous or unfocused. Information about pupil performance received by the teacher is insufficiently used ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 57    to inform subsequent work," according to a United Kingdom inspection report on secondary schools.7 "Why is the extent and nature of formative assessment in science so impoverished?" asked a research study on secondary science teachers in the United Kingdom.8 "Indeed they pay lip service to [formative assessment] but consider that its practice is unrealistic in the present educational context," reported a study of Canadian secondary teachers.9 "The assessment practices outlined above are not common, even though these kinds of approaches are now widely promoted in the professional literature," according to a review of assessment practices in U.S. schools.10 The most important difficulties with assessment revolve around three issues. The first issue is effective learning.    The tests used by teachers encourage rote and superficial learning even when teachers say they want to develop understanding; many teachers seem unaware of the inconsistency. The questions and other methods teachers use are not shared with other teachers in the same school, and they are not critically reviewed in relation to what they actually assess. For primary teachers particularly, there is a tendency to emphasize quantity and presentation of work and to neglect its quality in relation to learning. The second issue is negative impact.   The giving of marks and the grading function are overemphasized, while the giving of useful advice and the learning function are underemphasized. Approaches are used in which pupils are compared with one another, the prime purpose of which seems to them to be competition rather than personal improvement; in consequence, assessment feedback teaches low-achieving pupils that they lack "ability," causing them to come to believe that they are not able to learn. The third issue is the managerial role of assessments.    Teachers' feedback to pupils seems to serve social and managerial functions, often at the expense of the learning function. Teachers are often able to predict pupils' results on external tests because their own tests imitate them, but at the same time teachers know too little about their pupils' learning needs. The collection of marks to fill in records is given higher priority than the analysis of pupils' work to discern learning needs; furthermore, some teachers pay no attention to the assessment records of their pupils' previous teachers. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 58 Of course, not all these descriptions apply to all classrooms. Indeed, there are many schools and classrooms to which they do not apply at all. Nevertheless, these general conclusions have been drawn by researchers who have collected evidence -- through observation, interviews, and questionnaires -- from schools in several countries, including the U.S. An empty commitment. The development of national assessment policy in England and Wales over the last decade illustrates the obstacles that stand in the way of developing policy support for formative assessment. The recommendations of a government task force in 198811 and all subsequent statements of government policy have emphasized the importance of formative assessment by teachers. However, the body charged with carrying out government policy on assessment had no strategy either to study or to develop the formative assessment of teachers and did no more than devote a tiny fraction of its resources to such work.12 Most of the available resources and most of the public and political attention were focused on national external tests. While teachers' contributions to these "summative assessments" have been given some formal status, hardly any attention has been paid to their contributions through formative assessment. Moreover, the problems of the relationship between teachers' formative and summative roles have received no attention. It is possible that many of the commitments were stated in the belief that formative assessment was not problematic, that it already happened all the time and needed no more than formal acknowledgment of its existence. However, it is also clear that the political commitment to external testing in order to promote competition had a central priority, while the commitment to formative assessment was marginal. As researchers the world over have found, high-stakes external tests always dominate teaching and assessment. However, they give teachers poor models for formative assessment because of their limited function of providing overall summaries of achievement rather than helpful diagnosis. Given this fact, it is hardly surprising that numerous research studies of the implementation of the education reforms in the United Kingdom have found that formative assessment is "seriously in need of development."13 With hindsight, we can see that the failure to perceive the need for substantial support for formative assessment and to take responsibility for developing such support was a serious error. In the U.S. similar pressures have been felt from political movements characterized by a distrust of teachers and a belief that external testing will, on its own, improve learning. Such fractured relationships between policy makers and the teaching profession are not inevitable -- indeed, many countries with enviable educational achievements seem to manage well with policies that show greater respect and support for teachers. While the situation in the U.S. is far more diverse than that in England and Wales, the effects of high-stakes state-mandated testing are very similar to those of the external tests in the United Kingdom. Moreover, the traditional reliance on multiple-choice testing in the U.S. -- not shared in the United Kingdom -- has exacerbated the negative effects of such policies on the quality of classroom learning. How Can We Improve Formative Assessment? ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 59 The self-esteem of pupils. A report of schools in Switzerland states that "a number of pupils . . . are content to 'get by.' . . . Every teacher who wants to practice formative assessment must reconstruct the teaching contracts so as to counteract the habits acquired by his pupils."14 The ultimate user of assessment information that is elicited in order to improve learning is the pupil. There are negative and positive aspects of this fact. The negative aspect is illustrated by the preceding quotation. When the classroom culture focuses on rewards, "gold stars," grades, or class ranking, then pupils look for ways to obtain the best marks rather than to improve their learning. One reported consequence is that, when they have any choice, pupils avoid difficult tasks. They also spend time and energy looking for clues to the "right answer." Indeed, many become reluctant to ask questions out of a fear of failure. Pupils who encounter difficulties are led to believe that they lack ability, and this belief leads them to attribute their difficulties to a defect in themselves about which they cannot do a great deal. Thus they avoid investing effort in learning that can lead only to disappointment, and they try to build up their self-esteem in other ways. The positive aspect of students' being the primary users of the information gleaned from formative assessments is that negative outcomes -- such as an obsessive focus on competition and the attendant fear of failure on the part of low achievers -- are not inevitable. What is needed is a culture of success, backed by a belief that all pupils can achieve. In this regard, formative assessment can be a powerful weapon if it is communicated in the right way. While formative assessment can help all pupils, it yields particularly good results with low achievers by concentrating on specific problems with their work and giving them a clear understanding of what is wrong and how to put it right. Pupils can accept and work with such messages, provided that they are not clouded by overtones about ability, competition, and comparison with others. In summary, the message can be stated as follows: feedback to any pupil should be about the particular qualities of his or her work, with advice on what he or she can do to improve, and should avoid comparisons with other pupils. Self-assessment by pupils. Many successful innovations have developed self- and peerassessment by pupils as ways of enhancing formative assessment, and such work has achieved some success with pupils from age 5 upward. This link of formative assessment to self-assessment is not an accident; indeed, it is inevitable. To explain this last statement, we should first note that the main problem that those who are developing self-assessments encounter is not a problem of reliability and trustworthiness. Pupils are generally honest and reliable in assessing both themselves and one another; they can even be too hard on themselves. The main problem is that pupils can assess themselves only when they have a sufficiently clear picture of the targets that their learning is meant to attain. Surprisingly, and sadly, many pupils do not have such a picture, and they appear to have become accustomed to receiving classroom teaching as an arbitrary sequence of exercises with no overarching rationale. To overcome this pattern of passive reception requires hard and sustained work. When pupils do acquire such an overview, they then become more committed and more effective as learners. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 60 Moreover, their own assessments become an object of discussion with their teachers and with one another, and this discussion further promotes the reflection on one's own thinking that is essential to good learning. Thus self-assessment by pupils, far from being a luxury, is in fact an essential component of formative assessment. When anyone is trying to learn, feedback about the effort has three elements: recognition of the desired goal, evidence about present position, and some understanding of a way to close the gap between the two.15 All three must be understood to some degree by anyone before he or she can take action to improve learning. Such an argument is consistent with more general ideas established by research into the way people learn. New understandings are not simply swallowed and stored in isolation; they have to be assimilated in relation to preexisting ideas. The new and the old may be inconsistent or even in conflict, and the disparities must be resolved by thoughtful actions on the part of the learner. Realizing that there are new goals for the learning is an essential part of this process of assimilation. Thus we conclude: if formative assessment is to be productive, pupils should be trained in self-assessment so that they can understand the main purposes of their learning and thereby grasp what they need to do to achieve. The evolution of effective teaching. The research studies referred to above show very clearly that effective programs of formative assessment involve far more than the addition of a few observations and tests to an existing program. They require careful scrutiny of all the main components of a teaching plan. Indeed, it is clear that instruction and formative assessment are indivisible. To begin at the beginning, the choice of tasks for classroom work and homework is important. Tasks have to be justified in terms of the learning aims that they serve, and they can work well only if opportunities for pupils to communicate their evolving understanding are built into the planning. Discussion, observation of activities, and marking of written work can all be used to provide those opportunities, but it is then important to look at or listen carefully to the talk, the writing, and the actions through which pupils develop and display the state of their understanding. Thus we maintain that opportunities for pupils to express their understanding should be designed into any piece of teaching, for this will initiate the interaction through which formative assessment aids learning. Discussions in which pupils are led to talk about their understanding in their own ways are important aids to increasing knowledge and improving understanding. Dialogue with the teacher provides the opportunity for the teacher to respond to and reorient a pupil's thinking. However, there are clearly recorded examples of such discussions in which teachers have, quite unconsciously, responded in ways that would inhibit the future learning of a pupil. What the examples have in common is that the teacher is looking for a particular response and lacks the flexibility or the confidence to deal with the unexpected. So the teacher tries to direct the pupil toward giving the expected answer. In manipulating the dialogue in this way, the teacher seals off any unusual, often thoughtful ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 61 but unorthodox, attempts by pupils to work out their own answers. Over time the pupils get the message: they are not required to think out their own answers. The object of the exercise is to work out -- or guess -- what answer the teacher expects to see or hear. A particular feature of the talk between teacher and pupils is the asking of questions by the teacher. This natural and direct way of checking on learning is often unproductive. One common problem is that, following a question, teachers do not wait long enough to allow pupils to think out their answers. When a teacher answers his or her own question after only two or three seconds and when a minute of silence is not tolerable, there is no possibility that a pupil can think out what to say. There are then two consequences. One is that, because the only questions that can produce answers in such a short time are questions of fact, these predominate. The other is that pupils don't even try to think out a response. Because they know that the answer, followed by another question, will come along in a few seconds, there is no point in trying. It is also generally the case that only a few pupils in a class answer the teacher's questions. The rest then leave it to these few, knowing that they cannot respond as quickly and being unwilling to risk making mistakes in public. So the teacher, by lowering the level of questions and by accepting answers from a few, can keep the lesson going but is actually out of touch with the understanding of most of the class. The question/answer dialogue becomes a ritual, one in which thoughtful involvement suffers. There are several ways to break this particular cycle. They involve giving pupils time to respond; asking them to discuss their thinking in pairs or in small groups, so that a respondent is speaking on behalf of others; giving pupils a choice between different possible answers and asking them to vote on the options; asking all of them to write down an answer and then reading out a selected few; and so on. What is essential is that any dialogue should evoke thoughtful reflection in which all pupils can be encouraged to take part, for only then can the formative process start to work. In short, the dialogue between pupils and a teacher should be thoughtful, reflective, focused to evoke and explore understanding, and conducted so that all pupils have an opportunity to think and to express their ideas. Tests given in class and tests and other exercises assigned for homework are also important means of promoting feedback. A good test can be an occasion for learning. It is better to have frequent short tests than infrequent long ones. Any new learning should first be tested within about a week of a first encounter, but more frequent tests are counterproductive. The quality of the test items -- that is, their relevance to the main learning aims and their clear communication to the pupil -- requires scrutiny as well. Good questions are hard to generate, and teachers should collaborate and draw on outside sources to collect such questions. Given questions of good quality, it is essential to ensure the quality of the feedback. Research studies have shown that, if pupils are given only marks or grades, they do not benefit from the feedback. The worst scenario is one in which some pupils who get low marks this time also got low marks last time and come to expect to get low marks next ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 62 time. This cycle of repeated failure becomes part of a shared belief between such students and their teacher. Feedback has been shown to improve learning when it gives each pupil specific guidance on strengths and weaknesses, preferably without any overall marks. Thus the way in which test results are reported to pupils so that they can identify their own strengths and weaknesses is critical. Pupils must be given the means and opportunities to work with evidence of their difficulties. For formative purposes, a test at the end of a unit or teaching module is pointless; it is too late to work with the results. We conclude that the feedback on tests, seatwork, and homework should give each pupil guidance on how to improve, and each pupil must be given help and an opportunity to work on the improvement. All these points make clear that there is no one simple way to improve formative assessment. What is common to them is that a teacher's approach should start by being realistic and confronting the question "Do I really know enough about the understanding of my pupils to be able to help each of them?" Much of the work teachers must do to make good use of formative assessment can give rise to difficulties. Some pupils will resist attempts to change accustomed routines, for any such change is uncomfortable, and emphasis on the challenge to think for yourself (and not just to work harder) can be threatening to many. Pupils cannot be expected to believe in the value of changes for their learning before they have experienced the benefits of such changes. Moreover, many of the initiatives that are needed take more class time, particularly when a central purpose is to change the outlook on learning and the working methods of pupils. Thus teachers have to take risks in the belief that such investment of time will yield rewards in the future, while "delivery" and "coverage" with poor understanding are pointless and can even be harmful. Teachers must deal with two basic issues that are the source of many of the problems associated with changing to a system of formative assessment. The first is the nature of each teacher's beliefs about learning. If the teacher assumes that knowledge is to be transmitted and learned, that understanding will develop later, and that clarity of exposition accompanied by rewards for patient reception are the essentials of good teaching, then formative assessment is hardly necessary. However, most teachers accept the wealth of evidence that this transmission model does not work, even when judged by its own criteria, and so are willing to make a commitment to teaching through interaction. Formative assessment is an essential component of such instruction. We do not mean to imply that individualized, one-on-one teaching is the only solution; rather we mean that what is needed is a classroom culture of questioning and deep thinking, in which pupils learn from shared discussions with teachers and peers. What emerges very clearly here is the indivisibility of instruction and formative assessment practices. The other issue that can create problems for teachers who wish to adopt an interactive model of teaching and learning relates to the beliefs teachers hold about the potential of all their pupils for learning. To sharpen the contrast by overstating it, there is on the one hand the "fixed I.Q." view -- a belief that each pupil has a fixed, inherited intelligence that cannot be altered much by schooling. On the other hand, there is the "untapped ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 63 potential" view -- a belief that starts from the assumption that so-called ability is a complex of skills that can be learned. Here, we argue for the underlying belief that all pupils can learn more effectively if one can clear away, by sensitive handling, the obstacles to learning, be they cognitive failures never diagnosed or damage to personal confidence or a combination of the two. Clearly the truth lies between these two extremes, but the evidence is that ways of managing formative assessment that work with the assumptions of "untapped potential" do help all pupils to learn and can give particular help to those who have previously struggled. Policy and Practice Changing the policy perspective. The assumptions that drive national and state policies for assessment have to be called into question. The promotion of testing as an important component for establishing a competitive market in education can be very harmful. The more recent shifting of emphasis toward setting targets for all, with assessment providing a touchstone to help check pupils' attainments, is a more mature position. However, we would argue that there is a need now to move further, to focus on the inside of the "black box" and so to explore the potential of assessment to raise standards directly as an integral part of each pupil's learning work. It follows from this view that several changes are needed. First, policy ought to start with a recognition that the prime locus for raising standards is the classroom, so that the overarching priority has to be the promotion and support of change within the classroom. Attempts to raise standards by reforming the inputs to and measuring the outputs from the black box of the classroom can be helpful, but they are not adequate on their own. Indeed, their helpfulness can be judged only in light of their effects in classrooms. The evidence we have presented here establishes that a clearly productive way to start implementing a classroom-focused policy would be to improve formative assessment. This same evidence also establishes that in doing so we would not be concentrating on some minor aspect of the business of teaching and learning. Rather, we would be concentrating on several essential elements: the quality of teacher/pupil interactions, the stimulus and help for pupils to take active responsibility for their own learning, the particular help needed to move pupils out of the trap of "low achievement," and the development of the habits necessary for all students to become lifelong learners. Improvements in formative assessment, which are within the reach of all teachers, can contribute substantially to raising standards in all these ways. Four steps to implementation. If we accept the argument outlined above, what needs to be done? The proposals outlined below do not follow directly from our analysis of assessment research. They are consistent with its main findings, but they also call on more general sources for guidance.16 At one extreme, one might call for more research to find out how best to carry out such work; at the other, one might call for an immediate and large-scale program, with new guidelines that all teachers should put into practice. Neither of these alternatives is ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 64 sensible: while the first is unnecessary because enough is known from the results of research, the second would be unjustified because not enough is known about classroom practicalities in the context of any one country's schools. Thus the improvement of formative assessment cannot be a simple matter. There is no quick fix that can alter existing practice by promising rapid rewards. On the contrary, if the substantial rewards promised by the research evidence are to be secured, each teacher must find his or her own ways of incorporating the lessons and ideas set out above into his or her own patterns of classroom work and into the cultural norms and expectations of a particular school community.17 This process is a relatively slow one and takes place through sustained programs of professional development and support. This fact does not weaken the message here; indeed, it should be seen as a sign of its authenticity, for lasting and fundamental improvements in teaching and learning must take place in this way. A recent international study of innovation and change in education, encompassing 23 projects in 13 member countries of the Organisation for Economic Co-operation and Development, has arrived at exactly the same conclusion with regard to effective policies for change.18 Such arguments lead us to propose a four-point scheme for teacher development. 1. Learning from development. Teachers will not take up ideas that sound attractive, no matter how extensive the research base, if the ideas are presented as general principles that leave the task of translating them into everyday practice entirely up to the teachers. Their classroom lives are too busy and too fragile for all but an outstanding few to undertake such work. What teachers need is a variety of living examples of implementation, as practiced by teachers with whom they can identify and from whom they can derive the confidence that they can do better. They need to see examples of what doing better means in practice. So changing teachers' practice cannot begin with an extensive program of training for all; that could be justified only if it could be claimed that we have enough "trainers" who know what to do, which is certainly not the case. The essential first step is to set up a small number of local groups of schools -- some primary, some secondary, some innercity, some from outer suburbs, some rural -- with each school committed both to a school-based development of formative assessment and to collaboration with other schools in its local group. In such a process, the teachers in their classrooms will be working out the answers to many of the practical questions that the evidence presented here cannot answer. They will be reformulating the issues, perhaps in relation to fundamental insights and certainly in terms that make sense to their peers in other classrooms. It is also essential to carry out such development in a range of subject areas, for the research in mathematics education is significantly different from that in language, which is different again from that in the creative arts. The schools involved would need extra support in order to give their teachers time to plan the initiative in light of existing evidence, to reflect on their experience as it develops, and to offer advice about training others in the future. In addition, there would be a need for external evaluators to help the teachers with their development work and to collect ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS evidence of its effectiveness. Video studies of classroom work would be essential for disseminating findings to others. 65 2. Dissemination. This dimension of the implementation would be in low gear at the outset -- offering schools no more than general encouragement and explanation of some of the relevant evidence that they might consider in light of their existing practices. Dissemination efforts would become more active as results and resources became available from the development program. Then strategies for wider dissemination -- for example, earmarking funds for inservice training programs -- would have to be pursued. We must emphasize that this process will inevitably be a slow one. To repeat what we said above, if the substantial rewards promised by the evidence are to be secured, each teacher must find his or her own ways of incorporating the lessons and ideas that are set out above into his or her own patterns of classroom work. Even with optimum training and support, such a process will take time. 3. Reducing obstacles. All features in the education system that actually obstruct the development of effective formative assessment should be examined to see how their negative effects can be reduced. Consider the conclusions from a study of teachers of English in U.S. secondary schools. Most of the teachers in this study were caught in conflicts among belief systems and institutional structures, agendas, and values. The point of friction among these conflicts was assessment, which was associated with very powerful feelings of being overwhelmed, and of insecurity, guilt, frustration, and anger. . . . This study suggests that assessment, as it occurs in schools, is far from a merely technical problem. Rather, it is deeply social and personal.19 The chief negative influence here is that of short external tests. Such tests can dominate teachers' work, and, insofar as they encourage drilling to produce right answers to short, out-of-context questions, they can lead teachers to act against their own better judgment about the best ways to develop the learning of their pupils. This is not to argue that all such tests are unhelpful. Indeed, they have an important role to play in securing public confidence in the accountability of schools. For the immediate future, what is needed in any development program for formative assessment is to study the interactions between these external tests and formative assessments to see how the models of assessment that external tests can provide could be made more helpful. All teachers have to undertake some summative assessment. They must report to parents and produce end-of-year reports as classes are due to move on to new teachers. However, the task of assessing pupils summatively for external purposes is clearly different from the task of assessing ongoing work to monitor and improve progress. Some argue that these two roles are so different that they should be kept apart. We do not see how this can be done, given that teachers must have some share of responsibility for the former and must take the leading responsibility for the latter.20 However, teachers clearly face ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS difficult problems in reconciling their formative and summative roles, and confusion in teachers' minds between these roles can impede the improvement of practice. The arguments here could be taken much further to make the case that teachers should play a far greater role in contributing to summative assessments for accountability. One strong reason for giving teachers a greater role is that they have access to the performance of their pupils in a variety of contexts and over extended periods of time. 66 This is an important advantage because sampling pupils' achievement by means of short exercises taken under the conditions of formal testing is fraught with dangers. It is now clear that performance in any task varies with the context in which it is presented. Thus some pupils who seem incompetent in tackling a problem under test conditions can look quite different in the more realistic conditions of an everyday encounter with an equivalent problem. Indeed, the conditions under which formal tests are taken threaten validity because they are quite unlike those of everyday performance. An outstanding example here is that collaborative work is very important in everyday life but is forbidden by current norms of formal testing.21 These points open up wider arguments about assessment systems as a whole -- arguments that are beyond the scope of this article. 4. Research. It is not difficult to set out a list of questions that would justify further research in this area. Although there are many and varied reports of successful innovations, they generally fail to give clear accounts of one or another of the important details. For example, they are often silent about the actual classroom methods used, the motivation and experience of the teachers, the nature of the tests used as measures of success, or the outlooks and expectations of the pupils involved. However, while there is ample justification for proceeding with carefully formulated projects, we do not suggest that everyone else should wait for their conclusions. Enough is known to provide a basis for active development work, and some of the most important questions can be answered only through a program of practical implementation. Directions for future research could include a study of the ways in which teachers understand and deal with the relationship between their formative and summative roles or a comparative study of the predictive validity of teachers' summative assessments versus external test results. Many more questions could be formulated, and it is important for future development that some of these problems be tackled by basic research. At the same time, experienced researchers would also have a vital role to play in the evaluation of the development programs we have proposed. Are We Serious About Raising Standards? The findings summarized above and the program we have outlined have implications for a variety of responsible agencies. However, it is the responsibility of governments to take the lead. It would be premature and out of order for us to try to consider the relative roles in such an effort, although success would clearly depend on cooperation among government agencies, academic researchers, and school-based educators. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 67 The main plank of our argument is that standards can be raised only by changes that are put into direct effect by teachers and pupils in classrooms. There is a body of firm evidence that formative assessment is an essential component of classroom work and that its development can raise standards of achievement. We know of no other way of raising standards for which such a strong prima facie case can be made. Our plea is that national and state policy makers will grasp this opportunity and take the lead in this direction. 1. James W. Stigler and James Hiebert, "Understanding and Improving Classroom Mathematics Instruction: An Overview of the TIMSS Video Study," Phi Delta Kappan, September 1997, pp. 19-20. 2. There is no internationally agreed-upon term here. "Classroom evaluation," "classroom assessment," "internal assessment," "instructional assessment," and "student assessment" have been used by different authors, and some of these terms have different meanings in different texts. 3. Paul Black and Dylan Wiliam, "Assessment and Classroom Learning," Assessment in Education, March 1998, pp. 7-74. 4. Lynn S. Fuchs and Douglas Fuchs, "Effects of Systematic Formative Evaluation: A Meta-Analysis," Exceptional Children, vol. 53, 1986, pp. 199-208. 5. See Albert E. Beaton et al., Mathematics Achievement in the Middle School Years (Boston: Boston College, 1996). 6. Lynn S. Fuchs et al., "Effects of Task-Focused Goals on Low-Achieving Students with and Without Learning Disabilities," American Educational Research Journal, vol. 34, 1997, pp. 513-43. 7. OFSTED (Office for Standards in Education), Subjects and Standards: Issues for School Development Arising from OFSTED Inspection Findings 1994-5: Key Stages 3 and 4 and Post-16 (London: Her Majesty's Stationery Office, 1996), p. 40. 8. Nicholas Daws and Birendra Singh, "Formative Assessment: To What Extent Is Its Potential to Enhance Pupils' Science Being Realized?," School Science Review, vol. 77, 1996, p. 99. 9. Clement Dassa, Jesús Vazquez-Abad, and Djavid Ajar, "Formative Assessment in a Classroom Setting: From Practice to Computer Innovations," Alberta Journal of Educational Research, vol. 39, 1993, p. 116. 10. D. Monty Neill, "Transforming Student Assessment," Phi Delta Kappan, September 1997, pp. 35-36. 11. Task Group on Assessment and Testing: A Report (London: Department of Education and Science and the Welsh Office, 1988). 12. Richard Daugherty, National Curriculum Assessment: A Review of Policy, 1987-1994 (London: Falmer Press, 1995). 13. Terry A. Russell, Anne Qualter, and Linda McGuigan, "Reflections on the Implementation of National Curriculum Science Policy for the 5-14 Age Range: Findings and Interpretations from a National Evaluation Study in England," International Journal of Science Education, vol. 17, 1995, pp. 481-92. 14. Phillipe Perrenoud, "Towards a Pragmatic Approach to Formative Evaluation," in Penelope Weston, ed., Assessment of Pupils' Achievement: Motivation and School Success (Amsterdam: Swets and Zeitlinger, 1991), p. 92. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 68 15. D. Royce Sadler, "Formative Assessment and the Design of Instructional Systems," Instructional Science, vol. 18, 1989, pp. 119-44. 16. Paul J. Black and J. Myron Atkin, Changing the Subject: Innovations in Science, Mathematics, and Technology Education (London: Routledge for the Organisation for Economic Co-operation and Development, 1996); and Michael G. Fullan, with Suzanne Stiegelbauer, The New Meaning of Educational Change (London: Cassell, 1991). 17. See Stigler and Hiebert, pp. 19-20. 18. Black and Atkin, op. cit. 19. Peter Johnston et al., "Assessment of Teaching and Learning in Literature-Based Classrooms," Teaching and Teacher Education, vol. 11, 1995, p. 359. 20. Dylan Wiliam and Paul Black, "Meanings and Consequences: A Basis for Distinguishing Formative and Summative Functions of Assessment," British Educational Research Journal, vol. 22, 1996, pp. 537-48. 21. These points are developed in some detail in Sam Wineburg, "T. S. Eliot, Collaboration, and the Quandaries of Assessment in a Rapidly Changing World," Phi Delta Kappan, September 1997, pp. 59-65. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 69 Using Multiple Linear Regression to Evaluate the Use of a Classroom Performance System in an Introductory Statistics Course An applied project submitted in partial fulfillment of the requirements for the degree of Master of Science at Virginia Commonwealth University by Marissa Eagle Bachelor of Science, Virginia Commonwealth University, 2004 Director: Dr. D’Arcy P. Mays Associate Professor, Chairman Department of Statistical Sciences and Operations Research Virginia Commonwealth University Richmond, VA April, 2006 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 70 Acknowledgment First, I would like to thank my mom, dad, and my sister, Alex, for all the love and support that they have given me. They always have been and always will be there for me and I love them very much. I would also like to thank all of my wonderful friends for their continued support, including my boyfriend Greg for always loving me no matter what. Last, but not least, I would also like to thank Dr. Darcy Mays for all of his help with this project and throughout the years. He has helped guide me since my first years at VCU and he has been a great friend and teacher. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 71 Table of Contents List of Tables……………………………………………………………………………..iv List of Figures……………………………………………………………………………..v Abstract…………………………………………………………………………………...vi Executive Summary………….…………………………………………………………..vii Introduction………………………………………………………………………………..1 Literature Review………………………………………………………………………….4 Introduction to STAT 210…………………………………………………………………9 The Study………………………………………………………………………………...12 Results……………………………………………………………………………………13 Semesters before CPS……………………………………………………………13 Semesters after CPS……………………………………………………………...15 All Semesters Combined...……………………………………………………….16 All Individual Semesters…....................................................................................19 Conclusions………………………………………………………………………………21 Recommendations………………………………………………………………………..22 References………………………………………………………………………………..24 Appendix I: SAS Output for all semesters before CPS………………………………….25 Appendix II: SAS Output for all semesters after CPS…………………………………...45 Appendix III: SAS Output for all semesters combined………………………………….64 Appendix IV: SAS Output for all individual semesters………………………………….92 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 72 List of Tables Table 1: Results for physics courses at Eindhoven University of Technology……………...7 Table 2: ANOVA results for all individual semesters…………………………………...19 Table 3: Parameter estimates for all individual semesters……………………………….20 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 73 List of Figures Figure 1: Results for physics courses at Eindhoven University of Technology…………….8 Figure 2: Current CPS pad being used in STAT 210…………………………………….11 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 74 Abstract Using Multiple Linear Regression to Evaluate the Use of a Classroom Performance System in an Introductory Statistics Course By Marissa Eagle, B.S. Virginia Commonwealth University, 2006 Director: Dr. D’Arcy P. Mays Associate Professor, Chairman, Department of Statistical Sciences and Operations Research The Classroom Performance System (CPS) is an interactive teaching aid that is currently being used in Stat 210: Basic Practice of Statistics at Virginia Commonwealth University. By simply asking a question the instructor can receive almost instant feedback about the understanding and comprehension that students have about the material. The CPS consists of a receiver and software program where the teacher can receive the student’s answers by use of their CPS transmitters or ―clickers‖. Since incorporating the CPS into the course curriculum and making it a part of the grading process there has been a noticeable improvement in the performance of the students taking the course. By using data collected from eight semesters, four where the CPS was used and four when the CPS was not used, multiple regression techniques were used in order to determine whether there had been a significant impact on the students’ final grade in the course. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 75 Executive Summary Using Multiple Linear Regression to Evaluate the Use of a Classroom Performance System in an Introductory Statistics Course By Marissa Eagle, B.S. Virginia Commonwealth University, 2006 Director: Dr. D’Arcy P. Mays Associate Professor, Chairman, Department of Statistical Sciences and Operations Research When a professor is lecturing to his or her class, studies have shown that after about 15 to 20 minutes the students’ attention begins to lapse and they may no longer be paying attention (Johnstone and Percival, 1976). So how can students become more engaged in their classes and have a better learning experience? With the introduction of the Classroom Performance System (CPS) students attitudes towards learning as well as their grades in school have had a vast improvement. The CPS is an interactive teaching aid that is currently being used in classrooms throughout the world. From elementary school to college, and in businesses and the government, the CPS has become useful in many arenas. The CPS, produced by EInstruction, Inc. was produced in 2000 and has since become one of the leading providers of wireless interactive educational technology. The CPS consists of a receiver and software program where the teacher can receive the student’s answers by use of their CPS transmitters or ―clickers‖. By simply asking a question instructors can receive almost instant feedback about the understanding ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS and comprehension that their students have about the material. The question can be a 76 multiple choice, yes/no, or even a question where calculations are required and a student must enter a numerical value. This questioning approach has been proven to increase students’ understanding and learning by as much as 150%. From the interactive learning that the students’ have with their instructor by answering questions, it has been shown that the use of the CPS increases attentiveness, encourages participation, and increases the students’ comprehension. The CPS is also particularly useful in that it allows each and every student to answer the questions, confidentially. This in turn eliminates potential social dilemmas that may usually deter a student from answering a question. Of interest was to determine whether or not the CPS has been useful in helping students’ learn and hence perform better in their schoolwork. In particular, this study focuses on an introductory statistics course, Stat 210: Basic Practice of Statistics, which is offered at Virginia Commonwealth University (VCU). Since incorporating the CPS into the course curriculum and making it a part of the grading process there had been a noticeable improvement in the performance of the students taking the course. In order to evaluate the usefulness of the CPS, data was collected from eight semesters, four where the CPS was used and four when the CPS was not used, and multiple linear regression techniques were used in order to determine whether there had been a significant impact on the students’ final grade in the course. From the results, it was found that the CPS was in fact a statistically significant, positive factor in determining the final grade average of a student. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 77 It was found that the CPS was having a positive influence on each assignment that a student completed. Thus, since the students presumably understood the material better thanks to the CPS, it was not solely their final grade that was being influenced. The CPS positively affected the students’ performance on quizzes, homework, and tests, as well. Thus, overall the CPS has made an improvement in the students’ grades in STAT 210 at VCU. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 78 Introduction Standing in front of a lecture hall filled with a hundred students, the teacher asks a question…and no one raises their hand to provide an answer. For a moment, the teacher hesitates as if expecting a student to willingly answer the question. Instead, the teacher looks down at the computer screen in front and sees a chart of every student’s response. Now with a better understanding of the students’ comprehension of the question, the teacher continues on with the lecture… If you have ever been a teacher, then you might know how unwilling students can be to speak up. Whether the students are shy or just unsure of their answer, it can be hard to provoke a response. Yet with the application of interactive teaching tools, instead of that one ambitious student answering all the questions, you can get an answer from all of your students. Technology has improved our lives greatly in almost every aspect, why should it not improve the way we learn? In particular, computers and the internet have been a major influence, especially in the past few decades. When it comes to learning, the internet provides an endless supply of knowledge and it seems there are computer programs for everything from teaching new languages to helping tutor our children. The most recent and innovative application in interactive teaching is the use of a classroom performance system. The classroom performance system enables students in a classroom setting to interact in realtime with their teacher using a small computerized remote. This type of interactive teaching has been present since the early 1960’s (Judson and Sawada, 2002). Teaching with this type of technology can be utilized at any education level and is not just used in high schools and colleges. Originally, the response systems were used in ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 79 the corporate setting for uses such as employee training and conference meetings. They were also used by the government for voting and military training (Lowery, 2005). Now, similar systems are being used in education, from elementary and high school to higher education, and in the service sector, such as fire and police stations. There are currently several versions of these communication systems, to name a few; there are Classroom Response Systems (CRS), Classroom Communication Systems (CCS), and Classroom Performance Systems (CPS). Whatever their name may be they all work in a similar way. From now on we will refer to these as Classroom Performance Systems (CPS), from E-instruction, Inc., which is the specific type that our study focuses on. E-instruction created the CPS in 2000. The CPS consists of CPS pads or ―clickers‖ and a receiver. The receiver is placed in the classroom or lecture hall and each student must have their own CPS pad that transmits all answers to the receiver. CPS pads are relatively inexpensive and the student must also register the CPS pad with Einstruction for a minimal fee. Using the CPS, the teacher can ask a question in lecture and the students transmit their answers with the infrared transmitter. The questions can be multiple choice, true/false, or yes/no, as well as questions where a student can answer numerically. Once the students answer, the answers are reported instantly to the teacher. The CPS system includes a software program that reports the responses numerically and graphically. By having feedback instantly displayed the teacher knows how many were correct or incorrect. The CPS breakdown helps the teacher to gauge the students learning and understanding of topics. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 80 The use of the CPS has shown that the learning environment can be improved by as much as 150% with the use of a questioning approach (E-instruction). By using the questioning approach as a teaching method the students become more engaged, they focus on specific objectives, and practice answering questions. It also enables the teacher to receive feedback thus allowing students to understand why they missed specific questions (E-instruction). Polls from the 1960’s through the 1990’s confirmed this and also indicated that students were more likely to attend class and felt like the instructors knew more about them as students (Judson and Sawada, 2002). Students become more engaged because they are actively participating in lecture rather than just attending and listening. Studies show that most people’s attention lapses after a certain period of time and students can pay attention to a lecture no more than 15 to 20 minutes at a time (Johnstone and Percival, 1976). This active participation increases the learning and comprehension of students, and allows them to focus on the objective of the lecture. The students also tend to enjoy the class more because of the increased activity. ―Students have always favored the use of electronic response systems and attribute such factors as attentiveness and personal understanding to using electronic response systems‖ (Judson and Sawada, 1992). Studies show that the introduction of questioning into lectures can increase the level of dialogue between students and their instructor (Crouch and Mazur, 2001). Dialogue between a teacher and students often cannot take place in a lecture, especially not for each and every student, but that dialogue can be an important part of the students’ learning process. The CPS is a way to incorporate that dialogue into the lecture by asking ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 81 questions and allowing the students to answer back. This dialogue is also what allows the teacher to check for understanding that the students have about the subject. Literature Review Graphing calculators or mini-computers were a first, big step in helping teach courses such as in mathematics, physics, and engineering, and now these communication systems can be useful in any application. Yet the CPS produced by E-instruction has only been around since 2000, although some other versions have been around a bit longer. Now that they have been in use for some time, we can begin to start answering the question as to whether they really work. In 1973, the University of Iowa (UI) began using graphing calculators as a tool in teaching their introductory calculus course. At that time, the graphing calculators were called mini-computers and had all the functions of a calculator and could print out all calculations. In addition to this, the mobile ―mini-computer‖ could be programmed and was used both in the lecture and the mathematics laboratory. This was their answer to providing multiple students access to a computer because at that time computers were not as small and were expensive; therefore they were not as accessible to students as they are today (Hethcote and Schaeffer, 1972). The UI study looked at a one-semester course of introductory calculus and was divided into two sections. One section was taught conventionally through lecture without the use of the mini-computer. They were taught the basics of beginning calculus. The second section was taught the basics, with also a few more in-depth concepts, and used ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS the mini-computer in conjunction with the lecture. The students’ performance was measured by the tests given throughout the course (Hethcote and Schaeffer, 1972). Using analysis of variance it was determined that in some certain subjects there was no significant difference between the two sections, while in other areas there was a 82 significant increase in the students’ success in the course. Thus, they concluded that using the mini-computer was beneficial in that it increased the learning of certain subjects and mini-computers allowed the teachers to venture into new topics that were omitted from prior calculus courses. They also concluded that the mini-computers increased the students’ ability to solve certain problems and apply their learning in doing so (Hethcote and Schaeffer, 1972). While there is evidence that calculators have been useful in improving teaching of mathematics, one problem is that calculators most likely would not be useful in other academic areas, such as ethics, philosophy, or biology. Another problem is that by using calculators, the teacher is not able to ask a question and receive a response, but only able to show students how to calculate and solve quantitative problems. In order to encompass a broader application of these interactive teaching methods, a device must be able to have students respond to a variety of questions, such as multiple choice or yes/no questions and questions that are opinionated or ethical, and not just calculating an answer. One group of researchers in the Netherlands at Eindhoven University of Technology has been using a system called Audience Paced Feedback (APF) that would be able to handle this question and answer type of interaction. The APF is a more primitive version of what the CPS is today since it only has one button on the students’ remotes, yet it allows more of the aforementioned broader application than the calculator. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 83 To use the APF, teachers may only ask a ―yes or no‖ question, and students only press the single button if their answer is yes. Similar to the CPS, the APF transmits all answers to the teacher’s receiver unit and displays a report of all answers. It is also possible to ask multiple choice questions with the APF, but it requires more effort. The teacher must ask the question, and then present each answer where students answer when the answer that they think is right is presented (Gilbert et al. 1997). The APF had been used at Eindhoven University of Technology since 1966, so they felt that there had been considerable time for students to adjust to the technology. The study was conducted in a Physics lecture classes. The physics lectures consisted of 20 to 25 minutes each of using the APF and conventional lecturing. Based on the results of their study they found that there was some evidence that the students preferred and performed better with the use of the APF (Gilbert et al. 1997). They observed their results by looking at the pass rate of the students’ end-ofcourse exams. The data come from four semesters of courses in the period between 1979 and 1992. These periods were selected because there were no data for the years prior to 1979, and after 1992 the lecture material was no longer the same and thus was not relevant to the study. Overall there were 2841 students in the traditional lectures, and there were 2550 students in the lectures where APF was facilitated. By looking at Table 1 you can see that for the years where APF was used, years C and D, the pass rate significantly increased (Gilbert et al. 1997). ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 84 Table 1. The percentage pass rate and sample size by subject. Years C and D are those in which APF has been used, shown in bold; years A and B are the results of traditional lectures. Year A Subject Maxwell theory Vibration and waves Optics and Maxwell theory Mechanics and kinetic theory of gasses Kirchhoff, Mechanics and vibrations Energy management Statistical thermodynamic % ........S 38.....307 .... .... 42.....72 34......148 ..... ..... Year B % ........S 47......259 .... .... 60......84 39.......124 ..... ..... Year C % ........S 83.....269 87.....106 76.....165 93.....116 95.....107 82.....270 98.....249 Year D % ........S 87....230 87.....122 65....180 96.....112 84......98 74....265 88......261 Also, by looking at Figure 1, where the mean pass rate of the end-of-semester exam is displayed by traditional teaching versus teaching with the APF, you can see that the mean pass rate is significantly higher when using the APF during lecture. The standard deviation is also plotted within the figure, and it is evident that the standard deviation decreased when using the APF in lectures. This indicates that students are more consistent in performing on their exams. Therefore, the use of APF in this study was found to be significant in helping the students learn more and achieve higher scores and pass rates (Gilbert et al. 1997). ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Figure 1. The mean and standard deviation of the course pass rate of 2550 students attending lectures with APF and 2841 from traditional lectures. 100 90 Mean 80 70 60 Mean 50 40 30 20 10 0 85 Traditional APF With the APF proving to be useful in helping students perform well in their physics courses at Eindhoven University of Technology, we now look at the CPS. The only difference between the APF and the CPS is that the CPS becomes more userfriendly for both the teacher and the students. The CPS has more than one button, so the teacher can simply ask a question without having to display the multiple choice answers one at a time, and students can simply enter their choice. In an attempt to demonstrate that the CPS is useful in increasing student achievement, researchers administered the Test of Understanding College Economics to students in six separate classes of microeconomics at East Tennessee State University. The test was given at the beginning and the end of the course. This study compared test ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 86 results for Fall 2000 and Spring 2001, when the CPS system was not used, to when CPS was used in Summer 2001 and Fall 2001 semesters (Everett and Ranker, 2002). The results from the TUCE were compared to the national average of students who had taken the same test. It was shown that the averages for the test given at the beginning of the semester, for both the groups, were lower than the national average. On the contrary, it was shown that the averages for the test given at the end of the semester, for both the groups, were higher than the national average. It was also shown that the group using the CPS earned a slightly higher gain than the group that did not use the CPS. However, the researchers found no conclusive results and determined that further research should be made (Everett and Ranker, 2002). Introduction to STAT 210: Basic Practice of Statistics The objective of this analysis is to determine whether or not the Classroom Performance System (CPS) has been a useful tool in the introductory statistics course, STAT 210 Basic Practice of Statistics, at Virginia Commonwealth University. VCU is located in Richmond, Virginia and has over 29,000 students; it is the one of the five largest universities in Virginia. Large universities such as this typically have large lectures, particularly for their general education classes. At VCU, STAT 210 is considered a general education course and is required by many majors. Particularly, every student in the College of Humanities and Sciences who is seeking a Bachelor of Science degree must take a statistics course. This includes more than twenty majors, such as Biology, Psychology, Economics, and all Pre-Professional programs at VCU. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 87 The prerequisites for a student planning to enroll in STAT 210 are that the student must have previously passed an introductory mathematics course. The introductory mathematics course prerequisite may be replaced by a passing score on the VCU Mathematics Placement Test. These prerequisites were developed in order to assure that students are properly prepared for the mathematics needed in STAT 210. Completion of STAT 210 gives students three academic credits. The three credits include two lecture hours and one and a half laboratory hours. Thus, students attend lecture twice weekly and a lab/recitation once weekly. Topics covered throughout the course ―include examining distributions, examining relationships, producing data, sampling distributions and probability, introduction to inference.‖ (VCU) The average lecture has approximately 200 students each semester, while the laboratory sessions break the students into groups of 16 to 20 students. Due to the large number of students in lecture, the CPS was recently added to the course requirements in spring 2004, in addition to the homework, quizzes, and tests already being given. Students are expected to come to lecture with their CPS pads and answer the questions that the teacher asks. The CPS pad, shown in Figure 2, is a small handheld device that looks similar to a remote and students can key in the answer that they believe to be correct. The teacher then has instant feedback based on students’ answers and can see what type of understanding the class has of the question. The feedback is also visible to the students, so they can see how their answers compared to the answers of the class as a whole. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Figure 2. CPS infrared response pad. 88 The usefulness of the CPS is obvious, especially for large classes or lectures. In smaller classes the teacher can communicate more easily with students and can answer questions in a timely matter, something that is not always possible with larger classes. However, with the CPS the teacher can see instantly where students are having trouble and can focus more on the problems that most students are having, even in large classes. As mentioned, the STAT 210 courses at VCU use the CPS as a grading tool. The students also take a weekly quiz in their lab sections and have homework problems weekly, which are called focus exercises. They also have three tests throughout the semester and a final exam at the end of the semester. Each of these components makes up part of their final average at the end of the semester. While the quizzes, homework, and tests are not new to the course, the CPS is. The CPS has only been used in the STAT 210 course at VCU since the spring semester of 2004. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Quizzes are given weekly to students in their laboratory section and focus 89 exercises are assigned weekly; both are worth twenty-five points each. At the end of the semester the lowest quiz grade and the lowest focus exercise grade are dropped. The three tests and the final are worth one-hundred points each. The CPS questions are given during lecture and the following is the breakdown of the grading: Correctly answering 80% or more of the CPS questions Correctly answering between 60% and 80% of the questions Correctly answering between 45% and 60% of the questions Correctly answering between 30% and 45% of the questions Correctly answering between 15% and 30% of the questions 5 points 4 points 3 points 2 points 1 point The Study The objective of this analysis is to determine whether or not the Classroom Performance System (CPS) has been a useful tool in STAT 210. The data comes from eight semesters in the period between 2002 and 2005. In the fall and spring semesters of 2002 and 2003 the CPS was not being used as an instruction tool. In the following four semesters, the fall and spring of 2004 and 2005, the CPS was being used in the lecture. The same professor was the teacher for the lectures in all eight semesters within the study. The instructors for the laboratory sessions varied, and they were mostly adjunct faculty and graduate teaching assistants. To evaluate the use of the CPS linear regression techniques were used. Linear regression involves have one or more explanatory variables or regressors explain a response variable. From the linear regression the analysis of variance results were used to ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS explain the effectiveness of the overall model. The linear regression techniques used in 90 this analysis were model selection, multicollinearity diagnostics, and analysis of variance. Also, to determine whether or not the CPS was useful, the parameter coefficients were estimated and examined. Within the model are the following regressors: quizzes one through thirteen (Q1…Q13), focus exercises one through twelve (F1…F12), three tests (MT1, MT2, MT3), a final exam (FIN), the overall quiz average (QAVE), the overall focus exercise average (FEAVE), and where applicable, the CPS score (CPS). The response variable is the final average (FIN_AVE) that the student receives for the semester. The final average was calculated by making the quiz and focus exercise average worth 15% each, all tests worth 45%, and the final exam was worth 25% for the semesters before the CPS was used. For the semesters after the CPS began being used, the quiz and focus exercise averages became worth 14% each, all the test became worth only 42% each, and the final exam was still worth 25% of the final grade. The tests both before and after the use of the CPS were weighted such that the better tests were worth more. Results Semesters Before CPS The semesters before STAT 210 started using the CPS contained only the quizzes, focus exercises, tests, the final exam, and average grades. The data was assumed to be normal and have homogeneous variance. By performing a linear regression analysis on the full model, we can see that there is a good fit due to the highly significant and large F- ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS value as well as a large adjusted R-square value, but several of the variables are insignificant. 91 By looking at the multicollinearity diagnostics (see pages 26 - 31 in Appendix I) we can see that we have two very large variance inflation factors that are greater than ten, which indicates a serious multicollinearity problem. Then by looking at the eigenvalues and variance proportions we see that the quiz average and focus exercise average are highly correlated with the individual quizzes and focus exercises. This is illustrated by the small eigenvalues and large variance proportions. So there are two model options, one with individual grades and one with the averages. By performing regression on both options, we find that the model which contains the averages of the quizzes and focus exercises, as well as the test scores, yields a higher adjusted R-square value of 0.9988 along with a significant F test with a value of 105194. While the other model is also significant and has an adjusted R-square value of 0.9974, the model that includes the averages fits better and also has a lower MSE. This can be seen on pages 32 – 33 of Appendix I. The chosen model was then tested for multicollinearity, and none was found. Also, the model including the averages was the model chosen by several selection methods, including forward, backward, and stepwise selection. In this model all parameter estimates are significant and all coefficients are positive, indicating that they are good, positive indicators of the final average that a student earned in STAT 210 those semesters. The final model chosen is: Ŷ = 2.23626 + 0.16728MT1 + 0.13079MT2 + 0.13465MT3 + 0.24827FIN + 0.15418QAVE + 0.14306FEAVE ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS One interesting observation that can be seen in the fitted model was that the parameter estimates were very similar to what the variables actual worth was in the grading process. For example, the final was worth 25% of the students’ final grade, and 92 the parameter estimate for FIN was relatively close to 0.25. Similarly, the test were worth a total of 45% and the parameter estimates for MT1, MT2, and MT3 once added together came relatively close to 0.45. Semesters After CPS For the four semesters after the instructor started using CPS in the course, the variables being considered are the same, with the addition of CPS. Starting with the full model, we can see that the averages are highly correlated with the individual grades for the quizzes and focus exercises. Thus, we consider a model with the individual grades versus a model that includes the averages of those individual grades. By doing so we see that once again the model containing the averages is the model with the better fit (see Appendix II). The model containing all tests, the quiz average, the focus exercise average, and now the CPS grade, has an adjusted R-square value of 0.9993, with a significant F value of 164895, indicating a very good fit. Also, all parameter estimates are highly significant with positive coefficients. The important observation is that the CPS variable is significant and positive, indicating that CPS was important in determining the students’ final average, and that its impact was positive. In this model the multicollinearity diagnostics indicate that there is no multicollinearity, which is good. Selection methods also indicate that this is the best model in terms of fit. The final prediction model is: ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Ŷ = 1.96402 + 0.15416MT1 + 0.12828MT2 + 0.12898MT3 + 0.24745FIN + 0.13810QAVE + 0.13773FEAVE + 0.99190CPS 93 One again, the fitted model contained parameter estimate that were very similar to what the variables actual worth was in the grading process. This time the tests were worth a total of 42% and the parameter estimates for MT1, MT2, and MT3 once added together came relatively close to 0.42. However, the CPS estimate does not come close to 0.05, as it was worth 5%. This is due to the fact that all the other grading tools were out of 100 points, where the CPS grade was out of 5 points. Thus, the CPS parameter estimate is near 1 as it should be. All Semesters Combined To begin the analysis, the data for all eight semesters were combined. In addition to the regressors, an indicator variable was created in order to test for the significance of the semesters that used the CPS versus those that did not. The indicator variable became SEM, where SEM = 0 for the semesters in which the CPS was not used, and SEM = 1 for the semesters in which the CPS was used. Based on the full model which contained Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q9, Q10, Q11, Q12, Q13, F1, F2, F3, F4, F5, F6, F7, F8, F9, F10, F11, F12, MT1, MT2, MT3, FIN, QAVE, FEAVE, CPS, and SEM the fit cannot be determined due to the inclusion of the indicator variable. The inclusion of the CPS variable and the SEM variable within the same model causes bias. Thus we must consider the full model with either CPS or the full model with SEM separately. Since the variable SEM is a key ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS variable in determining whether or not CPS was a significant factor in the Stat 210 courses, the variable CPS was removed. 94 Using the ―scaled and centered‖ models to test for multicollinearity within the full model, two variables had variance inflation factors above ten; indicating a serious multicollinearity problem (see pages 65 - 70 of Appendix III). The two variables with variance inflation factors above ten were the focus exercise average and quiz average. From the eigenvalues and variance proportions we see that that the quiz average and focus exercise average are correlated with the individual quiz and focus exercise grades. Therefore, either the averages or the individual grades should be left in the model. Comparing the two models (see pages 71 and 72 in Appendix III), the adjusted Rsquare for the model containing the individual quiz and focus exercise grades was 0.9954 with a significant F-value of 7653.29. The adjusted R-square for the model containing the quiz and focus exercise averages was 0.9972 and a significant F value of 83046.8. The second model provides a slightly better fit, therefore the model with the averages was chosen over that with the individual grades. Now using the model which contains MT1, MT2, MT3, FIN, QAVE, FEAVE, and SEM, multicollinearity is checked once again (pages 73 – 74 of Appendix III). The model now does not have any variance inflation factors above four, thus there appear to be no serious dependencies. The model still has an adjusted R-square of 0.9972 with a significant F value of 83046.8. With a model free of multicollinearity, model selection was then used in order to determine which model was best (pages 75 – 84 in Appendix III). The forward, backward, and stepwise procedures were used. The same model including MT1, MT2, ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS MT3, FIN, QAVE, FEAVE, and SEM was chosen for all three procedures. Thus, the current model is a good model. In the final model chosen all parameter estimates are highly significant, and all 95 coefficients are positive with the exception of the coefficient for the SEM variable. This indicates that all tests, the quiz average, and the focus exercise average are positively affecting the final average that a student receives in the class. The final adjusted R-square is 0.9972 with a significant and large F value of 83046.8, and the MSE is small with a value of 0.64449. The following is the resulting prediction equation: Ŷ = 2. 18271 + 0.15783MT1 + 0.13018MT2 + 0.13334MT3 + 0.24873FIN + 0.15788QAVE + 0.15541FEAVE – 0.44508SEM As seen above, the coefficient for the SEM is negative, indicating that the use of CPS in the classroom is negatively affecting the final average. All other indications are that this inference is not correct and we investigate this by using a model containing only the SEM variable to predict the final average (page 85 in Appendix III). From this we find that the intercept coefficient is 76.83084 and the SEM coefficient is 3.72825. This means that if the CPS was not being used (SEM = 0), the predicted final average of a student would be 76.83084. However, if the CPS is being used (SEM = 1), the predicted final average of a student would be 80.55909. Hence, there is actually an increase in what a student’s final average is predicted to be if the CPS is used in the classroom. Similarly, each of the other variables within the model was tested as the response while the SEM variable was the regressor (see pages 86 – 91 in Appendix III). In all models, the predicted response variable was found to actually increase when the CPS was being used in the classroom versus when it was not used in the classroom. Since use of ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS the CPS actually increases every other grading tool, like quizzes and tests, this explains why we obtained a negative coefficient in our final model. Due to the CPS positively 96 affecting every grading tool, not just the final average, we have a negative coefficient for SEM, which compares the semesters that were not using CPS to the semesters that were using the CPS. All Individual Semesters An analysis similar to that of the semesters grouped together was performed on each individual semester as well. The full model once again demonstrates that the quiz and focus exercise averages are highly correlated with the individual grades for the quizzes and focus exercises, respectively. Therefore, as previously determined the model containing only the averages and not the individual quizzes and focus exercises provides the best fit. See output in Appendix IV. Table 2. All individual semesters’ adjusted R-square values, MSE, and F-values. Adjusted 2 R 0.9989 0.9990 0.9991 0.9988 0.9991 0.9993 0.9994 0.9995 Semester Spring 2002 Fall 2002 Spring 2003 Fall 2003 Spring 2004 Fall 2004 Spring 2005 Fall 2005 MSE 0.28977 0.21717 0.22550 0.26967 0.16922 0.13250 0.13987 0.11375 F 27157.2 29600.3 40061.1 26832.5 29819.8 37763.6 70136.6 52400.0 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS From Table 2, we can see that all semesters had very high adjusted R-square 97 values. Also, with the exception of the Fall 2003 semester, the adjusted R-square values increased slightly every semester. The MSE values are also small with relatively large Fvalues. All F-values were found to be significant, indicating that the models provided a good fit. Table 3 gives the parameter estimates for each semester’s model. All parameter estimates are positive and highly significant. The parameter estimates also do not change much over time. Table 3. All individual semesters’ parameter estimates. Semester Spring 2002 Fall 2002 Spring 2003 Fall 2003 Spring 2004 Fall 2004 Spring 2005 Fall 2005 Intercept 2.56281 3.78543 1.57147 2.84653 1.98602 2.02908 2.01277 1.39139 MT1 0.16393 0.16376 0.18102 0.15832 0.15457 0.14050 0.15224 0.16704 MT2 0.14126 0.12792 0.13183 0.12510 0.12843 0.11842 0.13525 0.12791 MT3 0.12916 0.12266 0.14214 0.14905 0.12801 0.14008 0.12735 0.12324 FIN 0.24495 0.25723 0.23973 0.24835 0.24664 0.25027 0.24632 0.24676 QAVE 0.15621 0.15613 0.14620 0.14688 0.13836 0.14932 0.13406 0.13217 FEAVE 0.14262 0.13764 0.14851 0.14840 0.13999 0.13458 0.13796 0.14149 CPS 0.94836 1.00999 1.01915 1.01220 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 98 Conclusions In an effort to determine whether or not the CPS has been a significant factor in a student’s final grade in STAT 210 at VCU, the analysis has shown that it is in fact significant. Based on the data from the eight semesters, by comparing the final grade in the course to whether or not the CPS was used in that semester, the fact that the CPS was used was shown to have a significantly positive effect. The effect of the CPS on students’ grades makes sense for several reasons. First, with the CPS being used as part of the students’ final grade, more students come to class in order to receive their points. With students attending class more regularly, it would be correct to assume that students are retaining more knowledge by attending lecture. Second, given students are being graded for giving correct answers and not just a participation grade, students are more likely to pay attention so that they may answer questions correctly. Finally, with the combination of students attending class on a regular basis and studying more in attempting to give correct answers, the overall comprehension and understanding of the course material is likely to increase. The overall increase in understanding, in turn, helps the students to perform better on other assignments such as quizzes, homework, and tests. This was also evident in the results from the data for all the semesters combined. As each grading tool was regressed as the response versus the variable SEM, there was found to be a positive and significant increase for all. For example, with the quiz average (QAVE) as the response and the SEM variable as the explanatory variable, the regression coefficient for SEM was found to be positive and significant. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 99 Besides the obvious benefits that may result from incorporating a tool such as the CPS, the instructor of the course also felt intuitively that the performance of his students was improving. Fortunately, the results from this analysis can confirm his observations. Therefore, VCU has made a rewarding investment in the use of E-Instruction’s CPS in its STAT 210 Course. Further Recommendations Due to time constraints, analysis of this data was preliminary and there could be numerous follow-up research options. The following are several recommendations and ideas for further research. There are several other courses at VCU who are incorporating the CPS into their syllabi. In particular, the physics department has students using CPS for several of their introductory courses. Therefore, it would be of interest to see a comparison of how student’s grades are doing in other courses and possibly make a comparison with all the courses combined. Another factor in this would be to see how these other instructors are using CPS. If other professors are using the CPS to ask questions but do not put any emphasis on the students having the correct question, it is possible that there may be different results. Due to the fact that students would just give any answer only wanting the credit, and not putting forth the effort to get the right answer, there would not necessarily be an increase in the understanding of the material. There are also several other factors that were also not accounted for in our analysis that may be refined in further research. For example, students who are repeating ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 100 the course were left within the data set, yet there final average may have increased due to more time and previous knowledge from taking the course beforehand and not from the use of the CPS. Also, the tests, quizzes, and homework assignments change every semester and are not the same. Using a standardized test and/or the same exact quizzes and homework assignments each semester could possibly give us a more accurate description of how the final grades are being affected by the CPS. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 101 References Crouch, C. and Mazur, E. ―Peer Instruction: Ten years of experience and results.‖ American Journal of Physics, Vol. 69, No. 9, pp. 970-977. (2001). E-Instruction. April 10, 2006. Everett, Michael D. and Ranker, Richard A. "Classroom Response System: An Evaluation at an Easy-Access Regional University." Presentation with paper, presented at University of Kentucky Economics Teaching Workshop, Lexington, KY, 2002. Gilbert, M., Massen, C., Poulis, J., & Robens, E. ―Physics lecturing with audience paced feedback.‖ Retrieved from the World Wide Web on March 13, 2006 from: http://www.bedu.com/Publications/PhysLectAPF.html. (1997). Hethcote, H. W. and Schaeffer, A. J. ―A Computer Laboratory Course for Calculus and Linear Algebra.‖ American Mathematical Monthly, Vol. 79, No. 3, pp. 290-293. (March 1972). Johnstone, A. H., and Percival, F. ―Attention breaks in lectures.‖ Education in Chemistry, 13, 49-50. (1976). Judson, Eugene and Sawada, Daiyo. ―Learning from Past and Present: Electronic Response Systems in College Lecture Halls.‖ Journal of Computers in Mathematics and Science Teaching, 21(2): 167-81. Retrieved March 16, 2006 from: http://www.aace.org/dl/files/JCMST/JCMST212167.pdf (2002). Lowery, Roger C., ―Teaching and learning with interactive student response systems: a comparison of commercial products,‖ presented at the Southwestern Social Science Association Annual Meeting, New Orleans, Louisiana. (2005). O’Loughlin, Thomas. ―Using Electronic Programmable Calculators (Mini-Computers) in Calculus Instruction.‖ The American Mathematical Monthly, Vol. 84, No. 4, pp. 281-283, (April 1976). Virginia Commonwealth University. April 10, 2006. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 102 Appendix I: Results for semesters without the use of CPS ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 103 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Before CPS.xls' OUT=BEFORE; MODEL FIN_AVE = Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE / VIF COLLINOINT; RUN; QUIT; /* Multicollinearity diagnostics for the semesters before CPS, including all tests, each of the quizzes and focus exercise grades, and the averages of the quizzes and focus exercises */ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Number of Observations with Missing Values Analysis of Variance Source Model Error Corrected Total DF 31 546 577 Sum of Squares 143735 145.29208 143880 0.51585 76.06931 0.67813 Mean Square 4636.60929 0.26610 F Value 17424.1 Pr > F <.0001 768 578 190 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9990 0.9989 Parameter Estimates Variable Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE Label Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE DF 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Parameter Estimate 2.50161 0.00443 -0.00315 -0.01332 0.00142 -0.00647 -0.00102 -0.00312 0.00592 -0.00217 -0.00595 -0.00432 0.00050884 -0.00877 -0.00549 0.00850 -0.00208 -0.01366 0.00225 -0.00362 -0.00224 -0.00866 -0.00067380 -0.00278 0.00415 -0.00369 0.17148 0.13151 0.13128 0.24836 0.16353 0.14831 Standard Error 0.19102 0.00433 0.00452 0.00452 0.00502 0.00417 0.00413 0.00411 0.00363 0.00405 0.00451 0.00422 0.00411 0.00413 0.00685 0.00526 0.00540 0.00590 0.00599 0.00490 0.00549 0.00564 0.00568 0.00598 0.00489 0.00434 0.00221 0.00154 0.00174 0.00181 0.00852 0.01101 t Value 13.10 1.02 -0.70 -2.95 0.28 -1.55 -0.25 -0.76 1.63 -0.54 -1.32 -1.02 0.12 -2.12 -0.80 1.62 -0.39 -2.32 0.38 -0.74 -0.41 -1.54 -0.12 -0.46 0.85 -0.85 77.54 85.35 75.58 136.92 19.20 13.47 Pr > |t| <.0001 0.3070 0.4864 0.0033 0.7771 0.1211 0.8047 0.4482 0.1041 0.5914 0.1879 0.3062 0.9016 0.0341 0.4225 0.1064 0.7003 0.0209 0.7076 0.4609 0.6827 0.1252 0.9057 0.6427 0.3963 0.3962 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Parameter Estimates Variable Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE Label Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE DF 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Variance Inflation 0 1.77097 1.80708 1.98400 2.08193 2.06555 2.14191 2.44833 2.28515 2.16559 2.33597 2.47472 2.39167 2.43548 2.13218 3.00485 2.67637 2.61952 3.42647 3.41388 3.39653 3.87946 3.05229 4.16335 3.81625 3.16764 1.82810 2.33289 3.39887 3.86199 40.38131 89.69663 104 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Eigenvalue 11.91788 1.70640 1.57422 1.11834 1.02410 0.96607 0.90076 0.84169 0.82752 0.79438 0.75279 0.71018 0.67357 0.64896 0.61515 0.57420 0.54890 0.53319 0.51482 0.49737 0.46995 0.43810 0.41901 0.39921 0.36900 0.34465 0.32176 0.28598 0.17917 0.02391 0.00877 Condition Index 1.00000 2.64277 2.75148 3.26446 3.41137 3.51232 3.63743 3.76290 3.79500 3.87333 3.97889 4.09653 4.20638 4.28540 4.40157 4.55585 4.65964 4.72779 4.81142 4.89510 5.03588 5.21569 5.33317 5.46385 5.68314 5.88044 6.08598 6.45552 8.15581 22.32523 36.85334 --------Proportion of Variation-------Q1 Q2 Q3 0.00056396 0.00556 0.00328 0.08213 0.01264 0.18359 0.00110 0.04904 0.04940 0.01814 0.02192 0.00895 0.07979 0.00901 0.02164 0.00006710 0.00038064 0.03135 0.01145 0.01580 0.00388 0.00677 0.00258 0.00001285 0.02568 0.00790 0.00380 0.03872 0.00504 0.21073 0.08908 0.00045573 0.03119 0.00145 0.02194 0.05918 0.01524 0.07577 0.05193 0.00786 0.11361 0.04154 0.13545 0.00047576 0.00001753 0.00942 0.00570 0.00122 0.01148 0.00190 0.03236 0.00002676 0.00001166 0.00230 0.00039846 0.00174 0.00486 0.03435 0.00598 4.391986E-7 0.26391 0.06824 0.00073386 0.01558 0.00263 0.10623 0.00481 0.00002797 0.00824 0.00190 0.02468 0.10536 0.00085983 0.00951 0.03644 0.06871 0.01539 0.00687 0.03204 0.00129 0.00387 0.01977 0.03957 0.15456 0.00529 0.01146 0.00155 0.00098995 0.00663 0.01683 0.00329 0.23060 0.06430 105 ----------------------Proportion of Variation---------------------Q4 Q5 Q6 Q7 Q8 0.00069325 0.02101 0.00140 0.00323 0.02489 0.02606 0.02273 0.07930 0.05272 0.01714 0.06132 0.06008 0.00013106 0.06562 0.02513 0.01744 0.02565 0.00579 0.00435 0.05684 0.01882 0.00087446 0.00175 0.00118 0.00432 0.00017838 0.00055314 0.03414 0.00550 0.29369 0.06749 0.00060970 0.00651 0.00138 0.04459 0.12998 0.01787 0.00014319 0.03941 0.00618 0.00914 0.04908 0.05435 0.00812 0.05225 0.01757 0.04693 0.02383 0.02369 0.00123 0.00290 0.00714 0.00003868 0.00350 0.00064014 0.00933 0.02578 0.00012515 0.04304 0.00169 0.25572 0.11725 0.00083318 0.01973 0.00186 0.03155 0.01252 0.00948 0.00850 0.01237 0.03646 0.03034 0.03507 0.03141 0.01935 0.01041 0.11239 0.08250 0.04391 0.00761 0.00002452 0.00079146 0.02467 0.11801 0.00578 0.00485 0.00031062 0.00012902 0.00760 0.00220 0.00035728 0.22066 0.10834 0.00097466 0.00881 0.00727 0.00158 0.02610 0.00078345 0.03507 0.00212 0.00432 0.01528 0.00595 0.08338 0.01494 0.02884 0.00367 0.00828 0.07974 0.05443 0.00646 0.00002851 0.00374 0.12579 0.00528 0.04991 0.02204 0.02167 0.02055 0.02550 0.00761 0.24029 0.08961 0.00077768 0.00397 0.01416 0.00798 0.03059 0.00059944 0.07985 0.04341 0.00129 0.02666 0.04386 0.00025016 0.08249 0.00429 0.00981 0.02584 0.00219 0.01223 0.00116 0.05889 0.06617 0.01369 0.00001379 0.02398 0.00146 0.05646 0.00623 0.00303 0.00573 0.25709 0.11586 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ----------------------Proportion of Variation---------------------Q9 Q10 Q11 Q12 Q13 0.00086524 0.00997 0.00509 0.03465 0.01760 0.00748 0.01204 0.00614 0.03710 0.04326 0.00337 0.01365 0.12591 0.03885 0.00341 0.13054 0.03141 0.00236 0.02406 0.00000282 0.04737 0.00636 0.00012646 0.01450 0.00603 0.00036268 0.02609 0.00930 0.00103 0.24414 0.09694 0.00091211 0.00481 0.01067 0.00424 0.00199 0.01868 0.00161 0.00995 0.03872 0.07046 0.00163 0.02614 0.00306 0.20834 0.01011 0.01681 0.00017701 0.00762 0.03621 0.02771 0.02575 0.06644 0.00013875 0.00938 0.01554 0.00076739 0.00031645 0.00192 0.00380 0.29009 0.08600 0.00114 0.00078541 0.00730 0.00435 0.01033 0.00583 0.00080588 0.02953 0.01906 0.00001046 0.05189 0.00645 0.00327 0.04291 0.00000380 0.02352 0.01105 0.01662 0.26168 0.01288 0.01037 0.00309 0.01176 0.04025 0.07696 0.00940 0.01101 0.01107 0.00274 0.21068 0.10324 0.00124 0.00172 0.00494 0.00608 0.00331 0.00624 0.00235 0.00166 0.05200 0.00007325 0.00117 0.00569 0.00542 0.05036 0.04270 0.01154 0.11781 0.08464 0.03164 0.00134 0.13021 0.05915 0.00460 0.05503 0.00051875 0.00139 0.03188 0.00267 0.00311 0.21072 0.06878 0.00119 0.00202 0.00092514 0.00310 0.00080850 0.03060 0.00394 0.01836 0.04018 0.00269 0.03724 0.01161 0.00508 0.00394 0.07588 0.03424 0.02519 0.00109 0.01011 0.16968 0.00172 0.01798 0.05083 0.03994 0.04346 0.00124 0.00597 0.07159 0.00032484 0.22090 0.06820 106 ----------------------Proportion of Variation---------------------F1 F2 F3 F4 F5 0.00093880 0.00248 0.02217 0.00025946 0.00102 0.02713 0.01285 0.07034 0.00019434 0.02742 0.05487 0.05139 0.03279 0.00029638 0.01002 0.00383 0.00068516 0.21207 0.00073099 0.04487 0.01911 0.03169 0.00182 0.01101 0.01836 0.03191 0.01252 0.00184 0.00265 0.02901 0.26372 0.00080482 0.00574 0.02161 0.00277 0.00049328 0.00332 0.02192 0.00054468 0.00071831 0.00028625 0.00317 0.02871 0.00045123 0.00533 0.00041163 0.06413 0.00810 0.03103 0.02290 0.01421 0.06612 0.15318 0.05763 0.00397 0.00819 0.01884 0.01484 0.02107 0.00045369 0.00712 0.41195 0.00058926 0.00319 0.03109 0.01551 0.02565 0.00251 0.02505 0.00039056 0.02151 0.00032129 0.00037802 0.03156 0.03913 0.01340 0.00004539 0.07834 0.00004258 0.00083176 0.02431 0.00770 0.15135 0.01379 0.02571 0.00991 0.00371 0.02012 0.00210 0.00137 0.00049285 0.02402 0.42587 0.00095638 0.00071775 0.01445 0.00232 0.00174 0.01781 0.00191 0.00165 0.00065769 0.02731 0.05621 0.04315 0.00022121 0.01784 0.11682 0.01512 0.00382 0.00209 0.00413 0.10615 0.01919 0.00623 0.00099280 0.05234 0.01388 0.05643 0.01592 0.02402 0.01693 0.01882 0.34018 0.00083781 0.00002624 0.01332 0.01227 0.00002887 0.00419 0.00535 0.00860 0.01417 0.00100 0.02613 0.00136 0.01232 0.00330 0.00156 0.02238 0.00003181 0.01698 0.05096 0.03128 0.00005676 0.00564 0.04906 0.02557 0.02642 0.21427 0.00019822 0.00001084 0.00894 0.01947 0.42426 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ----------------------Proportion of Variation---------------------F6 F7 F8 F9 F10 0.00095970 0.00008423 0.00124 0.00151 0.00215 0.02300 0.00784 0.01804 0.00538 0.00586 0.00054624 0.00001026 0.01948 0.00217 0.05546 0.00390 0.00570 0.04861 0.00038651 0.00056831 0.04629 0.00146 0.06722 0.03174 0.04478 0.15023 0.01089 0.00639 0.00136 0.00993 0.42680 0.00090255 0.00228 0.00191 0.00398 0.02649 0.00276 0.00599 0.00363 0.00774 0.01299 0.00508 0.00004696 0.00667 0.00001473 0.03455 0.00093069 0.09736 0.00037284 0.00015563 0.03723 0.01086 0.01649 0.00421 0.00304 0.19486 0.02434 0.00077478 0.05184 0.00020869 0.02455 0.41775 0.00084005 0.00731 0.00203 0.00433 0.00061944 0.00132 0.00018477 0.01125 0.00071539 0.00067018 0.03597 0.00084539 0.00571 0.00776 0.00018968 0.01315 0.00187 0.00094699 0.08345 0.00059435 0.01497 0.00788 2.180389E-8 0.00095300 0.11153 0.01007 0.13465 0.07467 0.00629 0.01582 0.44342 0.00096609 0.01093 0.00000539 0.00442 0.00000109 0.00016784 0.00013864 0.00627 0.03601 0.02085 0.00074537 0.01097 0.04177 0.00109 0.00011112 0.01825 0.06767 0.02863 0.03935 0.00118 0.02517 0.01639 0.00819 0.00046868 0.00418 0.05272 0.16288 0.06766 0.01093 0.02323 0.33866 0.00082764 0.00891 0.00047444 0.00004519 0.00137 0.00044099 0.00150 0.00787 0.00030064 0.00003916 0.00472 0.00266 0.00003014 0.00126 0.00006013 0.01018 0.04081 0.02849 0.01118 0.03699 0.00415 0.00872 0.08332 0.00184 0.10257 0.07160 0.01120 0.05700 0.00883 0.03851 0.45410 107 ----------------------Proportion of Variation---------------------F11 F12 MT1 MT2 MT3 0.00090178 0.00819 0.00219 0.00192 0.00037476 0.00177 0.00004593 0.00002868 0.00899 0.00177 0.00118 0.00195 0.00449 0.00124 0.00886 0.01152 0.01836 0.00047004 0.01647 0.00829 0.02491 0.02132 0.13044 0.14370 0.00569 0.05070 0.06728 0.00209 0.00312 0.01445 0.43730 0.00091548 0.01288 0.00002913 0.00072599 0.01259 0.00000119 0.00046172 0.02271 0.00613 0.00055500 0.00110 0.00152 0.03925 0.01783 0.02110 4.604117E-7 3.037774E-8 0.08527 0.00003244 0.04752 0.03363 0.00714 0.01685 0.02520 0.00249 0.01545 0.22029 0.01743 0.00172 0.00753 0.38165 0.00116 0.02974 0.01027 0.00067876 0.01758 0.00237 0.13097 0.00213 0.01747 0.00615 0.00075361 0.00094870 0.00017010 0.00016000 0.00150 0.01332 0.01311 0.00226 0.05123 0.00521 0.01246 0.04397 0.22071 0.14728 0.04821 0.04912 0.02428 0.12922 0.01564 8.92402E-7 0.00192 0.00135 0.00767 0.01745 0.00787 0.00035011 0.02521 0.00053841 0.00653 0.01736 0.00009224 0.00014043 0.00096124 0.00224 0.00334 0.03680 0.01297 0.01934 0.00028391 0.00301 0.01891 0.00072353 0.00894 0.02717 0.24482 0.01495 0.00373 0.12731 0.29426 0.09206 0.00043875 0.00318 0.00119 0.00271 0.00897 0.00345 0.00137 0.01069 0.01442 0.00030780 0.00052911 0.00535 0.00017121 0.00067232 0.00962 0.00057239 0.00621 0.00407 0.00231 0.00431 0.00395 0.00163 0.00027982 0.00502 0.02227 0.00571 0.02834 0.02973 0.06487 0.16478 0.58585 0.00559 0.00505 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ---------Proportion of Variation--------FIN QAVE FEAVE 0.00108 0.00015604 0.01313 0.00089077 0.00145 0.00660 0.00794 0.00328 0.00129 0.00584 0.00070669 0.00156 0.00827 0.00148 0.00105 0.00772 0.00000230 0.00232 0.00152 0.00872 0.00052502 0.00186 0.00981 0.00021483 0.02223 0.00265 0.00635 0.07872 0.79144 0.01027 0.00091662 0.00014804 0.00027655 0.00031635 2.957825E-7 0.00025314 0.00087805 0.00033710 0.00017245 0.00003328 0.00007274 0.00002300 0.00000312 0.00008601 0.00001229 0.00009181 0.00000234 0.00001285 0.00000257 0.00000658 0.00006118 0.00000693 0.00000615 0.00004572 0.00003872 0.00002205 0.00005299 0.00004943 0.00003855 0.00000610 0.75045 0.24650 0.00006839 0.00010997 0.00035894 3.759776E-8 0.00001504 0.00006537 3.005909E-7 0.00000207 0.00000172 0.00000155 0.00002285 4.646284E-7 0.00000518 0.00000242 0.00000946 0.00002626 0.00002385 0.00000495 3.866994E-7 0.00001246 3.492316E-7 0.00000136 0.00001007 3.163958E-7 0.00002091 0.00000211 0.00000219 0.00000366 0.00000397 0.04280 0.95642 108 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 109 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Before CPS.xls' OUT=BEFORE; PROC REG; MODEL FIN_AVE = Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12MT1 MT2 MT3 FIN; RUN; QUIT; /* Regression analysis for the semesters before CPS, including all tests and each of the quiz and focus exercise grades*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Number of Observations with Missing Values Analysis of Variance Source Model Error Corrected Total DF 29 548 577 Sum of Squares 143527 353.52707 143880 0.80320 76.06931 1.05587 Mean Square 4949.19493 0.64512 F Value 7671.71 Pr > F <.0001 768 578 190 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9975 0.9974 Parameter Estimates Variable Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN Label Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN DF 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Parameter Estimate 4.68904 0.04656 0.04850 0.03456 0.06274 0.03746 0.03924 0.04013 0.04464 0.04033 0.04751 0.03598 0.04194 0.03401 0.05462 0.04805 0.04918 0.03571 0.05670 0.03561 0.04916 0.04097 0.04967 0.06061 0.04653 0.02854 0.17286 0.13064 0.13601 0.25296 Standard Error 0.27122 0.00568 0.00580 0.00595 0.00630 0.00519 0.00530 0.00527 0.00451 0.00517 0.00560 0.00547 0.00546 0.00545 0.00897 0.00626 0.00626 0.00736 0.00698 0.00575 0.00640 0.00648 0.00709 0.00668 0.00566 0.00531 0.00344 0.00240 0.00269 0.00281 t Value 17.29 8.20 8.36 5.81 9.96 7.22 7.40 7.62 9.90 7.80 8.49 6.57 7.68 6.24 6.09 7.68 7.86 4.85 8.13 6.19 7.68 6.33 7.01 9.07 8.22 5.38 50.24 54.54 50.59 90.09 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 110 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Before CPS.xls' OUT=BEFORE; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE; RUN; QUIT; /* Regression analysis for the semesters before CPS, including all tests and the averages of the quiz and focus exercise grades*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 6 761 767 Sum of Squares 188473 227.24397 188700 0.54645 76.83084 0.71124 Mean Square 31412 0.29861 F Value 105194 Pr > F <.0001 768 768 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9988 0.9988 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE DF 1 1 1 1 1 1 1 Parameter Estimate 2.63626 0.16728 0.13079 0.13465 0.24827 0.15418 0.14306 Standard Error 0.14751 0.00190 0.00136 0.00148 0.00168 0.00214 0.00173 t Value 17.87 87.97 95.86 90.92 147.78 71.95 82.88 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 111 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Before CPS.xls' OUT=BEFORE; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE / VIF COLLINOINT; RUN; QUIT; /* Multicollinearity diagnostics for the semesters before CPS including all tests and the averages of the quizzes and focus exercises */ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 6 761 767 Sum of Squares 188473 227.24397 188700 0.54645 76.83084 0.71124 Mean Square 31412 0.29861 R-Square Adj R-Sq F Value 105194 Pr > F <.0001 768 768 Root MSE Dependent Mean Coeff Var 0.9988 0.9988 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE DF 1 1 1 1 1 1 1 Parameter Estimate 2.63626 0.16728 0.13079 0.13465 0.24827 0.15418 0.14306 Standard Error 0.14751 0.00190 0.00136 0.00148 0.00168 0.00214 0.00173 t Value 17.87 87.97 95.86 90.92 147.78 71.95 82.88 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE DF 1 1 1 1 1 1 1 Variance Inflation 0 1.53993 2.15572 2.92676 3.69725 3.07460 2.61255 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 Eigenvalue 4.03979 0.68008 0.45758 0.40192 0.23236 0.18826 Condition Index 1.00000 2.43725 2.97129 3.17036 4.16963 4.63232 --------Proportion of Variation-------MT1 MT2 MT3 0.01777 0.65270 0.25324 0.00696 0.04453 0.02480 0.01821 0.02425 0.53945 0.20256 0.11288 0.10264 0.01504 0.03606 0.00109 0.36165 0.00398 0.58218 112 Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 ---------Proportion of Variation--------FIN QAVE FEAVE 0.01329 0.00011944 0.03344 0.09500 0.03731 0.82084 0.01515 0.01271 0.03781 0.13404 0.76050 0.03977 0.01574 0.10256 0.14907 0.08377 0.57564 0.07321 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 113 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Before CPS.xls' OUT=BEFORE; PROC RSQUARE; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE / ADJRSQ; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE / SELECTION = FORWARD; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE / SELECTION = BACKWARD; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE / SELECTION = STEPWISE SLENTRY = .25 SLSTAY = .25; RUN; QUIT; /* Model selection the semesters before CPS, including all tests and the averages of the quizzes and focus exercises */ The RSQUARE Procedure Model: MODEL1 Dependent Variable: FIN_AVE R-Square Selection Method Number of Observations Read Number of Observations Used Number in Model Adjusted R-Square 768 768 R-Square Variables in Model 1 0.8612 0.8611 FIN 1 0.7282 0.7278 MT3 1 0.7181 0.7178 QAVE 1 0.6355 0.6351 MT2 1 0.6355 0.6350 FEAVE 1 0.4308 0.4301 MT1 ---------------------------------------------------------------2 0.9372 0.9370 FIN QAVE 2 0.9310 0.9308 FIN FEAVE 2 0.9059 0.9056 MT3 FIN 2 0.9055 0.9052 MT2 FIN 2 0.8901 0.8898 MT1 FIN 2 0.8703 0.8700 MT2 MT3 2 0.8685 0.8681 MT3 QAVE 2 0.8312 0.8308 MT2 QAVE 2 0.8279 0.8274 MT1 MT3 2 0.8278 0.8273 MT2 FEAVE 2 0.8185 0.8180 MT3 FEAVE 2 0.7878 0.7873 MT1 QAVE 2 0.7786 0.7780 QAVE FEAVE 2 0.7733 0.7728 MT1 FEAVE 2 0.7245 0.7238 MT1 MT2 ---------------------------------------------------------------3 0.9598 0.9596 MT2 FIN FEAVE 3 0.9574 0.9572 MT3 FIN QAVE 3 0.9561 0.9560 MT2 FIN QAVE 3 0.9556 0.9554 FIN QAVE FEAVE 3 0.9546 0.9544 MT1 FIN FEAVE 3 0.9511 0.9509 MT1 FIN QAVE 3 0.9462 0.9460 MT3 FIN FEAVE 3 0.9443 0.9441 MT2 MT3 FIN 3 0.9340 0.9337 MT1 MT3 FIN 3 0.9280 0.9277 MT2 MT3 QAVE 3 0.9231 0.9228 MT1 MT2 FIN 3 0.9160 0.9157 MT2 MT3 FEAVE 3 0.9111 0.9108 MT1 MT3 QAVE 3 0.9103 0.9100 MT1 MT2 MT3 3 0.8930 0.8926 MT1 MT3 FEAVE 3 0.8818 0.8814 MT3 QAVE FEAVE 3 0.8783 0.8778 MT1 MT2 FEAVE 3 0.8742 0.8737 MT2 QAVE FEAVE ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The RSQUARE Procedure Model: MODEL1 Dependent Variable: FIN_AVE R-Square Selection Method Number in Model R-Square Adjusted R-Square Variables in Model 114 3 0.8638 0.8633 MT1 MT2 QAVE 3 0.8445 0.8439 MT1 QAVE FEAVE ---------------------------------------------------------------4 0.9767 0.9766 MT2 MT3 FIN QAVE 4 0.9754 0.9753 MT1 MT2 FIN FEAVE 4 0.9745 0.9744 MT2 MT3 FIN FEAVE 4 0.9742 0.9741 MT2 FIN QAVE FEAVE 4 0.9729 0.9728 MT1 MT3 FIN QAVE 4 0.9713 0.9712 MT1 FIN QAVE FEAVE 4 0.9703 0.9701 MT1 MT3 FIN FEAVE 4 0.9676 0.9674 MT3 FIN QAVE FEAVE 4 0.9661 0.9659 MT1 MT2 FIN QAVE 4 0.9620 0.9618 MT1 MT2 MT3 FIN 4 0.9513 0.9510 MT1 MT2 MT3 QAVE 4 0.9500 0.9497 MT1 MT2 MT3 FEAVE 4 0.9397 0.9394 MT2 MT3 QAVE FEAVE 4 0.9258 0.9254 MT1 MT3 QAVE FEAVE 4 0.9069 0.9064 MT1 MT2 QAVE FEAVE ---------------------------------------------------------------5 0.9906 0.9905 MT1 MT2 MT3 FIN FEAVE 5 0.9879 0.9878 MT1 MT2 MT3 FIN QAVE 5 0.9865 0.9865 MT2 MT3 FIN QAVE FEAVE 5 0.9857 0.9856 MT1 MT2 FIN QAVE FEAVE 5 0.9843 0.9842 MT1 MT3 FIN QAVE FEAVE 5 0.9642 0.9640 MT1 MT2 MT3 QAVE FEAVE ---------------------------------------------------------------6 0.9988 0.9988 MT1 MT2 MT3 FIN QAVE FEAVE ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Forward Selection: Step 1 Variable FIN Entered: R-Square = 0.8612 and C(p) = 86918.44 Analysis of Variance Source Model Error Corrected Total Variable Intercept FIN DF 1 766 767 Parameter Estimate 34.22754 0.64455 Sum of Squares 162517 26183 188700 Standard Error 0.65288 0.00935 Mean Square 162517 34.18153 F Value 4754.53 Pr > F <.0001 768 768 115 Type II SS 93945 162517 F Value 2748.40 4754.53 Pr > F <.0001 <.0001 Bounds on condition number: 1, 1 -------------------------------------------------------------------------------Forward Selection: Step 2 Variable QAVE Entered: R-Square = 0.9372 and C(p) = 38953.34 Analysis of Variance Source Model Error Corrected Total Variable Intercept FIN QAVE DF 2 765 767 Parameter Estimate 16.54796 0.45615 0.37564 Sum of Squares 176841 11859 188700 Standard Error 0.72912 0.00883 0.01236 Mean Square 88420 15.50260 F Value 5703.59 Pr > F <.0001 Type II SS 7985.32096 41328 14324 F Value 515.10 2665.90 923.95 Pr > F <.0001 <.0001 <.0001 Bounds on condition number: 1.9694, 7.8778 -------------------------------------------------------------------------------Forward Selection: Step 3 Variable MT3 Entered: R-Square = 0.9574 and C(p) = 26156.95 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT3 FIN QAVE DF 3 764 767 Parameter Estimate Sum of Squares 180663 8037.73300 188700 Standard Error Mean Square 60221 10.52059 F Value 5724.09 Pr > F <.0001 Type II SS F Value Pr > F <.0001 <.0001 <.0001 <.0001 15.29439 0.60424 6740.48577 640.69 0.16120 0.00846 3821.75555 363.26 0.35712 0.00894 16780 1595.00 0.32139 0.01057 9724.50397 924.33 Bounds on condition number: 2.9731, 23.418 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Forward Selection: Step 4 Variable MT2 Entered: R-Square = 0.9767 and C(p) = 13973.56 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT2 MT3 FIN QAVE DF 4 763 767 Parameter Estimate 13.91537 0.14908 0.16241 0.28690 0.26491 Sum of Squares 184301 4399.02561 188700 Standard Error 0.45066 0.00593 0.00626 0.00719 0.00814 Mean Square 46075 5.76543 F Value 7991.65 Pr > F <.0001 116 Type II SS 5496.99366 3638.70739 3878.93774 9191.21966 6103.17614 F Value 953.44 631.12 672.79 1594.19 1058.58 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.5033, 42.493 -------------------------------------------------------------------------------Forward Selection: Step 5 Variable MT1 Entered: R-Square = 0.9879 and C(p) = 6873.704 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN QAVE DF 5 762 767 Parameter Estimate 5.45194 0.16009 0.13283 0.16769 0.25526 0.23997 Sum of Squares 186422 2278.32361 188700 Standard Error 0.45422 0.00601 0.00432 0.00451 0.00531 0.00594 Mean Square 37284 2.98993 F Value 12470.0 Pr > F <.0001 Type II SS 430.76111 2120.70201 2830.94019 4127.20746 6911.53604 4883.55924 F Value 144.07 709.28 946.83 1380.37 2311.61 1633.34 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.6879, 62.258 -------------------------------------------------------------------------------- ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Forward Selection: Step 6 Variable FEAVE Entered: R-Square = 0.9988 and C(p) = 7.0000 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE DF 6 761 767 Parameter Estimate 2.63626 0.16728 0.13079 0.13465 0.24827 0.15418 0.14306 Sum of Squares 188473 227.24397 188700 Standard Error 0.14751 0.00190 0.00136 0.00148 0.00168 0.00214 0.00173 Mean Square 31412 0.29861 F Value 105194 Pr > F <.0001 117 Type II SS 95.37596 2310.78206 2743.82552 2468.27829 6521.36316 1545.69688 2051.07964 F Value 319.40 7738.40 9188.59 8265.83 21838.9 5176.27 6868.70 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.6972, 96.041 -------------------------------------------------------------------------------All variables have been entered into the model. Summary of Forward Selection Variable Step Entered 1 2 3 4 5 6 FIN QAVE MT3 MT2 MT1 FEAVE Label FIN QAVE MT3 MT2 MT1 FEAVE Number Partial Model Vars In R-Square R-Square 1 2 3 4 5 6 0.8612 0.0759 0.0203 0.0193 0.0112 0.0109 0.8612 0.9372 0.9574 0.9767 0.9879 0.9988 C(p) F Value Pr > F 86918.4 4754.53 <.0001 38953.3 923.95 <.0001 26157.0 363.26 <.0001 13973.6 631.12 <.0001 6873.70 709.28 <.0001 7.0000 6868.70 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Backward Elimination: Step 0 All Variables Entered: R-Square = 0.9988 and C(p) = 7.0000 Analysis of Variance Source Model Error Corrected Total DF 6 761 767 Parameter Estimate 2.63626 0.16728 0.13079 0.13465 0.24827 0.15418 0.14306 Sum of Squares 188473 227.24397 188700 Standard Error 0.14751 0.00190 0.00136 0.00148 0.00168 0.00214 0.00173 Mean Square 31412 0.29861 F Value 105194 Pr > F <.0001 768 768 118 Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE Type II SS 95.37596 2310.78206 2743.82552 2468.27829 6521.36316 1545.69688 2051.07964 F Value 319.40 7738.40 9188.59 8265.83 21838.9 5176.27 6868.70 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.6972, 96.041 -------------------------------------------------------------------------------All variables left in the model are significant at the 0.1000 level. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Stepwise Selection: Step 1 Variable FIN Entered: R-Square = 0.8612 and C(p) = 86918.44 Analysis of Variance Source Model Error Corrected Total Variable Intercept FIN DF 1 766 767 Parameter Estimate 34.22754 0.64455 Sum of Squares 162517 26183 188700 Standard Error 0.65288 0.00935 Mean Square 162517 34.18153 F Value 4754.53 Pr > F <.0001 768 768 119 Type II SS 93945 162517 F Value 2748.40 4754.53 Pr > F <.0001 <.0001 Bounds on condition number: 1, 1 -------------------------------------------------------------------------------Stepwise Selection: Step 2 Variable QAVE Entered: R-Square = 0.9372 and C(p) = 38953.34 Analysis of Variance Source Model Error Corrected Total Variable Intercept FIN QAVE DF 2 765 767 Parameter Estimate 16.54796 0.45615 0.37564 Sum of Squares 176841 11859 188700 Standard Error 0.72912 0.00883 0.01236 Mean Square 88420 15.50260 F Value 5703.59 Pr > F <.0001 Type II SS 7985.32096 41328 14324 F Value 515.10 2665.90 923.95 Pr > F <.0001 <.0001 <.0001 Bounds on condition number: 1.9694, 7.8778 -------------------------------------------------------------------------------Stepwise Selection: Step 3 Variable MT3 Entered: R-Square = 0.9574 and C(p) = 26156.95 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT3 FIN QAVE DF 3 764 767 Parameter Estimate Sum of Squares 180663 8037.73300 188700 Standard Error Mean Square 60221 10.52059 F Value 5724.09 Pr > F <.0001 Type II SS F Value Pr > F <.0001 <.0001 <.0001 <.0001 15.29439 0.60424 6740.48577 640.69 0.16120 0.00846 3821.75555 363.26 0.35712 0.00894 16780 1595.00 0.32139 0.01057 9724.50397 924.33 Bounds on condition number: 2.9731, 23.418 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Stepwise Selection: Step 4 Variable MT2 Entered: R-Square = 0.9767 and C(p) = 13973.56 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT2 MT3 FIN QAVE DF 4 763 767 Parameter Estimate 13.91537 0.14908 0.16241 0.28690 0.26491 Sum of Squares 184301 4399.02561 188700 Standard Error 0.45066 0.00593 0.00626 0.00719 0.00814 Mean Square 46075 5.76543 F Value 7991.65 Pr > F <.0001 120 Type II SS 5496.99366 3638.70739 3878.93774 9191.21966 6103.17614 F Value 953.44 631.12 672.79 1594.19 1058.58 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.5033, 42.493 -------------------------------------------------------------------------------Stepwise Selection: Step 5 Variable MT1 Entered: R-Square = 0.9879 and C(p) = 6873.704 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN QAVE DF 5 762 767 Parameter Estimate 5.45194 0.16009 0.13283 0.16769 0.25526 0.23997 Sum of Squares 186422 2278.32361 188700 Standard Error 0.45422 0.00601 0.00432 0.00451 0.00531 0.00594 Mean Square 37284 2.98993 F Value 12470.0 Pr > F <.0001 Type II SS 430.76111 2120.70201 2830.94019 4127.20746 6911.53604 4883.55924 F Value 144.07 709.28 946.83 1380.37 2311.61 1633.34 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.6879, 62.258 -------------------------------------------------------------------------------- ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Stepwise Selection: Step 6 Variable FEAVE Entered: R-Square = 0.9988 and C(p) = 7.0000 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE DF 6 761 767 Parameter Estimate 2.63626 0.16728 0.13079 0.13465 0.24827 0.15418 0.14306 Sum of Squares 188473 227.24397 188700 Standard Error 0.14751 0.00190 0.00136 0.00148 0.00168 0.00214 0.00173 Mean Square 31412 0.29861 F Value 105194 Pr > F <.0001 121 Type II SS 95.37596 2310.78206 2743.82552 2468.27829 6521.36316 1545.69688 2051.07964 F Value 319.40 7738.40 9188.59 8265.83 21838.9 5176.27 6868.70 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.6972, 96.041 -------------------------------------------------------------------------------All variables left in the model are significant at the 0.2500 level. Summary of Stepwise Selection Variable Step Entered 1 2 3 4 5 6 FIN QAVE MT3 MT2 MT1 FEAVE Variable Removed Label FIN QAVE MT3 MT2 MT1 FEAVE Number Partial Model Vars In R-Square R-Square 1 2 3 4 5 6 0.8612 0.0759 0.0203 0.0193 0.0112 0.0109 0.8612 0.9372 0.9574 0.9767 0.9879 0.9988 C(p) F Value 86918.4 4754.53 38953.3 923.95 26157.0 363.26 13973.6 631.12 6873.70 709.28 7.0000 6868.70 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 122 Appendix II: Results for semesters with the use of CPS ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 123 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\After CPS.xls' OUT=AFTER; PROC REG; MODEL FIN_AVE = Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE CPS/ VIF COLLINOINT; RUN; QUIT; /* Multicollinearity diagnostics for the semesters after CPS, including all tests, each of the quizzes and focus exercise grades, and the CPS variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Number of Observations with Missing Values Analysis of Variance Source Model Error Corrected Total DF 32 446 478 Sum of Squares 103996 65.26305 104061 0.38253 80.26559 0.47658 Mean Square 3249.87237 0.14633 R-Square Adj R-Sq F Value 22209.2 Pr > F <.0001 852 479 373 Root MSE Dependent Mean Coeff Var 0.9994 0.9993 Parameter Estimates Variable Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE CPS Label Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE CPS DF 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Parameter Estimate 2.07432 0.00498 -0.00249 0.00102 0.00372 -0.00259 0.00031591 0.00489 -0.00499 -0.00273 -0.00381 -0.00232 0.00281 -0.00731 0.00664 0.00163 0.00881 0.01556 0.01335 0.00309 0.00735 0.00954 0.00808 0.00522 0.00500 0.00265 0.15064 0.13115 0.13011 0.24756 0.13863 0.11700 0.97659 Standard Error 0.18477 0.00205 0.00323 0.00350 0.00324 0.00353 0.00280 0.00278 0.00295 0.00366 0.00329 0.00298 0.00376 0.00308 0.00552 0.00346 0.00531 0.00599 0.00485 0.00411 0.00486 0.00427 0.00519 0.00505 0.00483 0.00411 0.00191 0.00157 0.00177 0.00163 0.00437 0.00857 0.02152 t Value 11.23 2.43 -0.77 0.29 1.15 -0.74 0.11 1.76 -1.69 -0.75 -1.16 -0.78 0.75 -2.37 1.20 0.47 1.66 2.60 2.75 0.75 1.51 2.23 1.56 1.03 1.04 0.64 79.04 83.34 73.38 151.42 31.71 13.65 45.37 Pr > |t| <.0001 0.0156 0.4414 0.7717 0.2518 0.4625 0.9102 0.0791 0.0908 0.4553 0.2475 0.4374 0.4551 0.0182 0.2294 0.6387 0.0979 0.0097 0.0062 0.4519 0.1313 0.0260 0.1200 0.3019 0.3011 0.5197 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 124 Parameter Estimates Variable Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE CPS Label Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE CPS DF 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Variance Inflation 0 1.31178 1.61891 1.69527 1.61381 1.57984 1.64203 1.84707 2.15699 2.00470 2.15796 1.93703 2.57645 2.08547 1.93419 2.43481 2.40182 3.39050 2.79078 3.11332 3.74905 3.32880 3.56527 4.09580 4.06333 3.21686 1.61440 3.17103 4.01303 4.18428 16.21302 70.06840 2.24824 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Eigenvalue 12.41631 1.97054 1.77392 1.45211 1.11646 1.01216 0.91797 0.89828 0.86777 0.70410 0.69146 0.68401 0.66276 0.61550 0.60152 0.55534 0.51965 0.49606 0.47595 0.44665 0.41926 0.38929 0.37189 0.33513 0.33188 0.31455 0.26498 0.25105 0.20628 0.16860 0.05674 0.01183 Condition Index 1.00000 2.51017 2.64563 2.92413 3.33484 3.50244 3.67775 3.71783 3.78263 4.19933 4.23751 4.26055 4.32831 4.49141 4.54330 4.72845 4.88808 5.00298 5.10759 5.27248 5.44198 5.64758 5.77813 6.08676 6.11652 6.28275 6.84521 7.03265 7.75829 8.58149 14.79233 32.40286 --------Proportion of Variation-------Q1 Q2 Q3 0.00019570 0.00043949 0.00076796 0.00975 0.03371 0.00000323 0.00379 0.00160 0.01506 0.14962 0.05006 0.01327 0.00040045 0.01217 0.07270 0.00194 0.00025047 0.00017585 0.15199 0.00176 0.05060 0.00265 0.01961 0.11251 0.11915 0.11877 0.02031 0.00383 0.06227 0.16889 0.02472 0.02416 0.00290 0.09532 0.00696 0.02564 0.00762 0.00016303 0.00197 0.00560 0.07931 0.01882 0.01478 0.18694 0.00224 0.06344 0.00107 0.00004788 0.10642 0.00462 0.01732 0.00767 0.04879 0.00008987 0.00594 0.00693 0.00949 0.01310 0.01986 0.08253 0.00787 0.07831 0.00000825 0.02265 0.02542 0.03379 0.00110 0.00855 0.09620 0.11826 0.01653 0.04764 0.00861 0.00057681 0.01923 0.01179 0.01228 0.00074667 0.00041087 0.00631 0.00075752 0.00016454 0.00909 0.00397 0.01320 0.01034 0.01195 0.00454 0.00035361 0.00138 0.02110 0.10352 0.16297 0.00237 0.04928 0.00603 125 ----------------------Proportion of Variation---------------------Q4 Q5 Q6 Q7 Q8 0.00087981 0.00069909 0.00092695 0.00103 0.00117 0.00842 0.00058290 0.02002 0.01314 0.00634 0.00037746 0.00071060 0.00074246 0.00604 0.00423 0.00359 0.00017793 0.01233 0.00018635 0.00006216 0.04964 0.04893 0.02532 0.00907 0.02660 0.03265 0.26750 0.00171 0.01329 0.02911 0.18570 0.02287 0.01832 0.02817 0.00126 0.00015492 0.00137 0.06305 0.04845 0.00054535 0.00262 0.01924 0.00941 0.01221 0.01709 0.01241 0.00163 0.02793 0.00021806 0.00776 0.01359 0.05885 0.24151 0.05596 0.01567 0.21729 0.07545 0.00850 0.06535 0.00402 0.00220 0.01117 0.08290 0.10435 0.00012653 0.00035668 0.11283 0.00148 0.01771 0.12070 0.07446 0.03534 0.03717 0.07246 0.00080183 0.03497 0.02629 0.07063 0.06388 0.00142 0.00759 0.00435 0.00000231 0.00017345 0.09876 0.02565 0.01219 0.00003425 0.06542 0.12195 0.01290 0.00733 0.11931 0.10937 0.07936 0.00126 0.02053 0.04592 0.00043533 0.02112 0.07245 0.00056801 0.01507 0.00107 0.00537 0.00262 0.01614 0.00158 0.00012241 0.05780 0.03917 0.00498 0.00150 0.02388 0.00006865 0.01326 0.00524 0.00124 0.00359 0.00598 0.00955 0.00119 0.03615 0.09636 0.00055915 0.02409 0.03112 0.02053 0.02500 0.12906 0.00035788 0.00013701 0.00271 0.00363 0.01698 0.00078546 0.00183 0.01996 0.00095928 0.06134 0.00080038 0.00084106 0.01466 0.00791 0.02104 0.00005138 0.00139 0.00120 0.00161 0.01100 0.12229 0.14536 0.07777 0.12006 0.09942 0.02787 0.06317 0.02042 0.02888 0.03330 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 126 Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ----------------------Proportion of Variation---------------------Q9 Q10 Q11 Q12 Q13 0.00121 0.00120 0.00118 0.00126 0.00111 0.00071481 0.00194 0.00516 0.00125 0.01063 0.00575 0.00648 0.01476 0.00322 0.00474 0.00082667 0.00227 0.02246 0.00035551 0.00280 0.00762 0.00042647 0.00123 0.00027225 0.00127 0.03138 0.01692 0.00145 0.00038428 0.00533 0.00362 0.02317 0.00288 0.00024083 0.00006099 0.03528 0.00324 0.00247 0.02449 0.00075273 0.00032526 0.01815 0.00114 0.03481 0.06673 0.14788 0.04250 0.02616 0.00014961 0.00721 0.02286 0.03434 0.00914 0.00821 0.01458 0.00174 0.07512 0.01033 0.00350 0.01693 0.00081440 0.00094912 0.11168 0.01257 0.08134 0.13946 0.00130 0.00468 0.00011549 0.00675 0.00124 0.06206 0.00334 0.04484 0.08533 0.04650 0.00573 0.03904 0.01148 0.06770 0.00296 0.12580 0.19794 0.00771 0.00580 0.07082 0.00335 0.00589 0.04047 0.24990 0.11283 0.10830 0.00101 0.05786 0.00027929 0.00947 0.01171 0.08058 0.07036 0.00039824 0.00156 0.00461 0.12283 0.00086353 0.00277 0.01193 0.11353 0.14817 0.28185 0.00068308 0.00467 0.02225 0.01604 0.03682 0.01327 0.00344 0.00005778 0.01187 0.00897 0.02343 0.00118 0.00428 0.00528 0.00844 0.02006 0.10640 0.08635 0.03071 0.04526 0.04280 0.03623 0.01028 0.00524 0.00055695 0.01913 0.00896 0.02308 0.00738 0.05914 0.02323 0.01889 0.00003881 0.06210 0.03625 0.04809 0.00360 0.00247 0.00700 0.00128 0.01197 0.14480 0.16498 0.02410 0.19566 0.15104 0.01501 0.02313 0.01678 0.00133 0.01392 ----------------------Proportion of Variation---------------------F1 F2 F3 F4 F5 0.00047936 0.00060257 0.00066513 0.00076810 0.00074782 0.00082081 0.00208 0.00078642 0.00001415 0.00031915 0.04029 0.02898 0.03664 0.01922 0.01190 0.02752 0.00433 0.00209 0.00676 0.02677 0.00315 0.00554 0.00007536 0.00004667 0.00509 0.00238 0.00596 0.00873 0.00203 0.00650 0.00145 0.03916 0.00513 0.00046375 0.02698 0.12407 0.00300 0.01679 0.00010738 0.00394 0.00290 0.01648 0.00073501 0.00010419 0.00048781 0.01790 0.00140 0.03591 0.01334 0.00074988 0.05035 0.00020739 0.02727 0.00018472 0.00161 0.01176 0.00965 0.00442 0.00203 0.00592 0.08166 0.13538 0.00058194 0.01991 0.01125 0.00647 0.00020626 0.09024 0.00575 0.00183 0.04303 0.08620 0.04015 0.00566 0.00575 0.00372 0.02187 0.02414 0.00012965 0.10868 0.03652 0.02735 0.04873 0.05495 0.01799 0.02846 0.00271 0.04351 0.00518 0.00605 0.00124 0.05293 0.12129 0.00020735 0.02730 0.00196 0.03447 0.04703 0.00293 0.08062 0.04835 0.00313 0.01994 0.11761 0.00014569 0.00127 0.01827 0.00391 0.06819 0.01891 0.04045 0.04961 0.00132 0.00634 0.08840 0.14841 0.05875 0.06534 0.23100 0.10375 0.00407 0.00540 0.00716 0.01081 0.00220 0.00198 0.00009519 0.00113 0.00073663 0.04286 0.00410 0.00382 0.00254 0.01781 0.00076736 0.03210 0.01655 0.01099 0.00598 0.04625 0.00445 0.00186 0.00238 0.00438 0.01328 0.00205 0.00187 0.00021823 0.02376 0.00156 0.00218 0.00043788 0.00022715 0.00224 0.00000812 0.22446 0.36170 0.32995 0.37135 0.33141 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS ----------------------Proportion of Variation---------------------F6 F7 F8 F9 F10 0.00091295 0.00134 0.00115 0.02010 0.00699 0.01140 0.00125 0.00005768 0.00018170 0.00549 0.00040574 0.00043983 0.00314 0.02385 0.00370 0.05556 0.02637 0.02151 0.00011319 0.00002225 0.07210 0.00292 0.31691 0.01517 0.00232 0.01847 0.00372 0.00127 0.03606 0.01780 0.00098362 0.32829 0.00093214 0.00004678 0.00000501 0.00455 0.00721 0.01478 0.00250 0.00913 0.00009476 0.00109 0.00269 0.00854 0.00192 0.01219 0.00147 0.00438 0.00014218 0.00883 0.00232 0.07804 0.00549 0.02182 0.01273 0.02352 0.29076 0.00000314 0.02996 0.07645 0.00289 0.00959 0.00671 0.35921 0.00093227 0.00427 0.00001782 0.00014183 0.00860 0.01651 0.00274 0.00360 0.00655 0.05519 0.00008068 0.00425 0.00022496 0.00056023 0.00062370 0.04027 0.00293 0.01423 0.01186 0.04378 0.00112 0.05536 0.04689 0.00539 0.13383 0.00256 0.02360 0.17523 0.00068243 0.01482 0.00898 0.31419 0.00089854 0.00586 0.00011550 0.00012988 0.00065863 0.00006548 0.00603 0.00493 0.03784 0.00259 0.03629 0.00740 0.00145 0.00736 0.00702 0.00055692 0.00103 0.02907 0.01025 0.00231 0.00016489 0.03127 0.00005187 0.00208 0.00000817 0.13325 0.13183 0.21163 0.01566 0.00670 0.00036889 0.30515 0.00079470 0.00857 0.00000218 0.00014580 0.00631 0.00189 0.00038069 0.00217 0.01496 0.00102 0.00392 0.00063096 0.00753 0.01268 0.00264 0.00555 0.00854 0.00099962 0.03888 0.01396 0.00343 0.00660 0.00229 0.01836 0.00010245 0.01090 0.40387 0.04402 0.02294 0.00201 0.00316 0.35076 127 Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ----------------------Proportion of Variation---------------------F11 F12 MT1 MT2 MT3 0.00087961 0.00650 0.00013676 0.00133 0.00154 0.00560 0.00092289 0.00024776 0.00125 0.00023823 0.01706 0.00305 0.00090144 0.01645 0.00065704 0.00003069 0.00228 0.00084338 0.01241 0.01433 7.058295E-7 0.02424 0.00301 0.00827 0.07914 0.14761 0.20951 0.08199 0.02623 0.00062430 2.40608E-10 0.33272 0.00093633 0.01210 0.00000440 0.00055571 0.01020 0.00216 0.00551 0.00011292 0.00737 0.01253 0.00527 0.02531 0.00134 0.00611 0.00004799 0.00001428 0.02476 0.00563 0.00000405 0.00001216 0.03779 0.00347 0.01610 0.11899 0.16944 0.06483 0.01042 0.16350 0.06158 1.470797E-7 0.00782 0.22608 0.00043494 0.04733 0.00208 0.00985 0.14572 0.00050999 0.00010615 0.00095735 0.00622 0.00003620 0.04244 0.00911 0.01260 0.04341 0.02634 0.00083700 0.03048 0.03542 0.00544 0.07646 0.10952 0.00950 0.16810 0.00368 0.03072 0.15318 0.00000195 0.01243 0.00143 0.00167 0.01089 0.00312 0.00076982 0.02089 0.00746 0.00384 0.00034608 0.00529 0.00662 0.01304 0.00233 0.00497 0.00050502 0.00975 0.00066760 0.00278 0.00031149 0.00938 0.00968 0.00883 0.00358 0.00009608 0.00091454 0.00037079 0.02715 0.00275 0.00504 0.03665 0.00811 0.05793 0.67460 0.05501 0.02020 0.00011438 0.00103 0.00015851 0.00290 0.00095031 0.00461 0.00130 0.00153 0.01635 0.00460 0.00375 0.00408 0.01755 0.00019687 0.00466 0.00009132 0.00001405 0.00268 0.00287 0.00008378 0.01004 0.00000456 0.02626 0.00119 0.00080583 0.00002364 0.01479 0.03973 0.00057641 0.28636 0.52421 0.02337 0.00321 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 128 Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 -----------------Proportion of Variation---------------FIN QAVE FEAVE CPS 0.00094888 0.00264 0.00536 0.00044315 0.00433 0.00008816 0.00294 0.00932 0.00007451 0.00223 0.00258 0.01360 0.00227 0.00588 0.00011072 0.00182 0.00374 0.01001 0.00030660 0.01187 0.00051401 0.00043611 0.00061286 0.00674 0.01370 0.01848 0.01010 0.00159 0.07503 0.78540 0.00671 0.00013153 0.00033362 0.00010556 0.00060043 0.00073565 0.00001051 0.00139 0.00003205 0.00095556 0.00010679 0.00055156 0.00009534 0.00027369 0.00059379 0.00022487 0.00001771 0.00018955 0.00004111 0.00021937 0.00024572 0.00003670 0.00063742 0.00001536 0.00004372 0.00011265 0.00016026 0.00031322 0.00039004 0.00003170 0.00253 0.00000939 0.85560 0.13340 0.00008070 0.00010897 0.00026680 0.00005635 4.17981E-7 0.00013578 0.00000280 0.00002331 0.00002860 0.00000210 0.00000213 0.00000582 0.00002262 0.00011815 0.00000565 0.00000392 0.00000238 0.00000368 0.00003891 0.00009312 0.00005916 0.00000822 0.00000374 3.776369E-7 0.00001578 0.00000108 0.00000662 0.00001992 5.520638E-7 0.00000201 0.00658 0.99230 0.00154 0.00030226 0.00273 0.00037202 0.00232 0.00044781 0.01001 0.00253 0.02321 0.00346 0.00939 0.00475 0.01303 0.00088125 0.00003677 0.09611 0.00212 0.03395 0.01346 0.13008 0.31287 0.00523 0.03629 0.09249 0.00422 0.00129 0.01574 0.15446 0.02605 0.00007478 0.00047278 0.00007739 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\After CPS.xls' OUT=AFTER; PROC REG; MODEL FIN_AVE = Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN CPS; RUN; ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS QUIT; /* Regression analysis for the semesters after CPS, including all tests, each of the quizzes and focus exercise grades, and the CPS variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Number of Observations with Missing Values Analysis of Variance Source Model Error Corrected Total DF 30 448 478 Sum of Squares 103765 295.77973 104061 0.81254 80.26559 1.01232 Mean Square 3458.84664 0.66022 F Value 5238.91 Pr > F <.0001 852 479 373 129 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9972 0.9970 Parameter Estimates Variable Label DF Parameter Estimate Standard Error t Value Pr > |t| <.0001 0.0004 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 0.0002 <.0001 <.0001 0.0254 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Intercept Intercept 1 4.83883 0.36115 13.40 Q1 Q1 1 0.01535 0.00430 3.57 Q2 Q2 1 0.02826 0.00635 4.45 Q3 Q3 1 0.04986 0.00684 7.29 Q4 Q4 1 0.03879 0.00640 6.06 Q5 Q5 1 0.03758 0.00672 5.59 Q6 Q6 1 0.02486 0.00567 4.39 Q7 Q7 1 0.03515 0.00549 6.40 Q8 Q8 1 0.02181 0.00590 3.70 Q9 Q9 1 0.04364 0.00718 6.08 Q10 Q10 1 0.03951 0.00636 6.21 Q11 Q11 1 0.01386 0.00618 2.24 Q12 Q12 1 0.06072 0.00726 8.36 Q13 Q13 1 0.03370 0.00602 5.60 F1 F1 1 0.04347 0.01032 4.21 F2 F2 1 0.02648 0.00589 4.49 F3 F3 1 0.04561 0.00926 4.92 F4 F4 1 0.06578 0.01010 6.51 F5 F5 1 0.04486 0.00845 5.31 F6 F6 1 0.03216 0.00716 4.49 F7 F7 1 0.05325 0.00827 6.44 F8 F8 1 0.04984 0.00749 6.65 F9 F9 1 0.04665 0.00921 5.06 F10 F10 1 0.04604 0.00865 5.32 F11 F11 1 0.03780 0.00844 4.48 F12 F12 1 0.03778 0.00765 4.94 MT1 MT1 1 0.15684 0.00402 39.00 MT2 MT2 1 0.12593 0.00333 37.82 MT3 MT3 1 0.14035 0.00373 37.68 FIN FIN 1 0.25266 0.00346 73.01 CPS CPS 1 0.98635 0.04572 21.57 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\After CPS.xls' OUT=AFTER; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE CPS; RUN; QUIT; ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 130 /* Regression analysis for the semesters after CPS, including all tests, the averages of the quizzes and focus exercise grades, and the CPS variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 7 844 851 Sum of Squares 181253 132.53201 181385 0.39627 80.55909 0.49190 Mean Square 25893 0.15703 F Value 164895 Pr > F <.0001 852 852 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9993 0.9993 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS DF 1 1 1 1 1 1 1 1 Parameter Estimate 1.96402 0.15416 0.12828 0.12898 0.24745 0.13810 0.13773 0.99190 Standard Error 0.11493 0.00127 0.00103 0.00124 0.00124 0.00158 0.00134 0.01607 t Value 17.09 120.92 123.96 103.91 200.18 87.64 102.77 61.72 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\After CPS.xls' OUT=AFTER; PROC RSQUARE; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE CPS / ADJRSQ; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE CPS / SELECTION = FORWARD; ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 131 PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE CPS / SELECTION = BACKWARD; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE CPS / SELECTION = STEPWISE SLENTRY = 0.25 SLSTAY = 0.25; RUN; QUIT; /* Model selection for the semesters after CPS, including all tests, the averages of the quizzes and focus exercise grades, and the CPS variable*/ The RSQUARE Procedure Model: MODEL1 Dependent Variable: FIN_AVE R-Square Selection Method Number of Observations Read Number of Observations Used Number in Model Adjusted R-Square 852 852 R-Square Variables in Model 1 0.8567 0.8566 FIN 1 0.7672 0.7670 MT3 1 0.6842 0.6839 QAVE 1 0.6543 0.6539 MT2 1 0.6145 0.6140 FEAVE 1 0.4538 0.4531 CPS 1 0.3011 0.3003 MT1 -------------------------------------------------------------------2 0.9277 0.9275 FIN FEAVE 2 0.9219 0.9218 FIN QAVE 2 0.9092 0.9090 MT3 FIN 2 0.9070 0.9068 FIN CPS 2 0.8967 0.8964 MT2 FIN 2 0.8839 0.8836 MT1 FIN 2 0.8715 0.8712 MT2 MT3 2 0.8631 0.8627 MT3 QAVE 2 0.8553 0.8550 MT2 QAVE 2 0.8445 0.8441 MT2 FEAVE 2 0.8345 0.8341 MT3 FEAVE 2 0.8240 0.8235 MT1 MT3 2 0.8179 0.8175 MT3 CPS 2 0.7817 0.7812 MT2 CPS 2 0.7563 0.7558 QAVE FEAVE 2 0.7559 0.7553 MT1 QAVE 2 0.7396 0.7390 MT1 FEAVE 2 0.7190 0.7183 QAVE CPS 2 0.6880 0.6873 MT1 MT2 2 0.6511 0.6503 FEAVE CPS 2 0.6168 0.6159 MT1 CPS -------------------------------------------------------------------3 0.9582 0.9581 MT2 FIN FEAVE 3 0.9553 0.9551 MT1 FIN FEAVE 3 0.9507 0.9505 MT2 FIN QAVE 3 0.9469 0.9467 MT3 FIN QAVE 3 0.9465 0.9463 FIN QAVE FEAVE 3 0.9465 0.9463 MT3 FIN FEAVE 3 0.9419 0.9417 MT1 FIN QAVE 3 0.9390 0.9387 MT2 FIN CPS 3 0.9375 0.9373 MT3 FIN CPS 3 0.9372 0.9370 MT2 MT3 FIN 3 0.9371 0.9369 FIN FEAVE CPS 3 0.9366 0.9364 FIN QAVE CPS ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The RSQUARE Procedure Model: MODEL1 Dependent Variable: FIN_AVE R-Square Selection Method Number in Model R-Square Adjusted R-Square Variables in Model 132 3 0.9336 0.9333 MT1 FIN CPS 3 0.9324 0.9322 MT1 MT3 FIN 3 0.9307 0.9305 MT2 MT3 QAVE 3 0.9205 0.9202 MT2 MT3 FEAVE 3 0.9101 0.9098 MT1 MT2 FIN 3 0.9062 0.9058 MT2 MT3 CPS 3 0.9002 0.8998 MT1 MT3 QAVE 3 0.8965 0.8962 MT2 QAVE FEAVE 3 0.8905 0.8901 MT1 MT2 MT3 3 0.8892 0.8888 MT1 MT3 FEAVE 3 0.8788 0.8784 MT3 QAVE FEAVE 3 0.8748 0.8744 MT2 QAVE CPS 3 0.8735 0.8730 MT1 MT2 FEAVE 3 0.8734 0.8730 MT3 QAVE CPS 3 0.8718 0.8714 MT1 MT2 QAVE 3 0.8714 0.8710 MT1 MT3 CPS 3 0.8594 0.8589 MT2 FEAVE CPS 3 0.8468 0.8462 MT3 FEAVE CPS 3 0.8273 0.8267 MT1 QAVE FEAVE 3 0.8118 0.8111 MT1 MT2 CPS 3 0.7907 0.7900 MT1 QAVE CPS 3 0.7672 0.7664 MT1 FEAVE CPS 3 0.7634 0.7626 QAVE FEAVE CPS -------------------------------------------------------------------4 0.9736 0.9735 MT1 MT2 FIN FEAVE 4 0.9733 0.9732 MT2 FIN QAVE FEAVE 4 0.9719 0.9717 MT2 MT3 FIN FEAVE 4 0.9715 0.9714 MT1 MT3 FIN FEAVE 4 0.9700 0.9699 MT2 MT3 FIN QAVE 4 0.9695 0.9694 MT1 FIN QAVE FEAVE 4 0.9659 0.9658 MT2 FIN FEAVE CPS 4 0.9657 0.9656 MT1 MT3 FIN QAVE 4 0.9643 0.9642 MT1 FIN FEAVE CPS 4 0.9636 0.9634 MT2 FIN QAVE CPS 4 0.9623 0.9622 MT2 MT3 FIN CPS 4 0.9611 0.9609 MT1 MT3 FIN CPS 4 0.9609 0.9607 MT1 MT2 FIN QAVE 4 0.9608 0.9606 MT3 FIN QAVE FEAVE 4 0.9580 0.9578 MT1 FIN QAVE CPS 4 0.9565 0.9563 MT3 FIN QAVE CPS 4 0.9539 0.9536 MT3 FIN FEAVE CPS 4 0.9533 0.9530 MT1 MT2 FIN CPS 4 0.9508 0.9505 FIN QAVE FEAVE CPS 4 0.9499 0.9497 MT1 MT2 MT3 FIN 4 0.9454 0.9451 MT2 MT3 QAVE FEAVE 4 0.9444 0.9442 MT1 MT2 MT3 QAVE 4 0.9411 0.9408 MT1 MT2 MT3 FEAVE 4 0.9395 0.9392 MT2 MT3 QAVE CPS 4 0.9284 0.9281 MT2 MT3 FEAVE CPS 4 0.9257 0.9254 MT1 MT2 MT3 CPS 4 0.9194 0.9190 MT1 MT3 QAVE FEAVE 4 0.9163 0.9159 MT1 MT2 QAVE FEAVE 4 0.9122 0.9118 MT1 MT3 QAVE CPS 4 0.9007 0.9002 MT2 QAVE FEAVE CPS 4 0.9002 0.8997 MT1 MT3 FEAVE CPS 4 0.8930 0.8925 MT1 MT2 QAVE CPS 4 0.8878 0.8873 MT1 MT2 FEAVE CPS 4 0.8824 0.8819 MT3 QAVE FEAVE CPS 4 0.8345 0.8337 MT1 QAVE FEAVE CPS ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The RSQUARE Procedure Model: MODEL1 Dependent Variable: FIN_AVE R-Square Selection Method Number in Model R-Square Adjusted R-Square Variables in Model 133 -------------------------------------------------------------------5 0.9864 0.9863 MT1 MT2 MT3 FIN FEAVE 5 0.9860 0.9859 MT1 MT2 FIN QAVE FEAVE 5 0.9836 0.9835 MT2 MT3 FIN QAVE FEAVE 5 0.9822 0.9821 MT1 MT3 FIN QAVE FEAVE 5 0.9813 0.9812 MT1 MT2 FIN FEAVE CPS 5 0.9803 0.9802 MT1 MT2 MT3 FIN QAVE 5 0.9787 0.9786 MT2 MT3 FIN QAVE CPS 5 0.9786 0.9785 MT1 MT3 FIN FEAVE CPS 5 0.9780 0.9779 MT2 MT3 FIN FEAVE CPS 5 0.9768 0.9767 MT2 FIN QAVE FEAVE CPS 5 0.9766 0.9764 MT1 MT3 FIN QAVE CPS 5 0.9759 0.9757 MT1 MT2 MT3 FIN CPS 5 0.9751 0.9749 MT1 MT2 FIN QAVE CPS 5 0.9741 0.9739 MT1 FIN QAVE FEAVE CPS 5 0.9643 0.9640 MT3 FIN QAVE FEAVE CPS 5 0.9614 0.9612 MT1 MT2 MT3 QAVE FEAVE 5 0.9544 0.9541 MT1 MT2 MT3 QAVE CPS 5 0.9490 0.9487 MT1 MT2 MT3 FEAVE CPS 5 0.9482 0.9478 MT2 MT3 QAVE FEAVE CPS 5 0.9234 0.9230 MT1 MT3 QAVE FEAVE CPS 5 0.9209 0.9204 MT1 MT2 QAVE FEAVE CPS -------------------------------------------------------------------6 0.9960 0.9959 MT1 MT2 MT3 FIN QAVE FEAVE 6 0.9926 0.9926 MT1 MT2 MT3 FIN FEAVE CPS 6 0.9901 0.9901 MT1 MT2 MT3 FIN QAVE CPS 6 0.9899 0.9899 MT1 MT2 FIN QAVE FEAVE CPS 6 0.9866 0.9865 MT2 MT3 FIN QAVE FEAVE CPS 6 0.9860 0.9859 MT1 MT3 FIN QAVE FEAVE CPS 6 0.9646 0.9643 MT1 MT2 MT3 QAVE FEAVE CPS -------------------------------------------------------------------7 0.9993 0.9993 MT1 MT2 MT3 FIN QAVE FEAVE CPS ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Forward Selection: Step 1 Variable FIN Entered: R-Square = 0.8567 and C(p) = 164629.8 Analysis of Variance Source Model Error Corrected Total Variable Intercept FIN DF 1 850 851 Parameter Estimate 34.68343 0.63196 Sum of Squares 155401 25985 181385 Standard Error 0.67074 0.00886 Mean Square 155401 30.57027 F Value 5083.39 Pr > F <.0001 852 852 134 Type II SS 81740 155401 F Value 2673.85 5083.39 Pr > F <.0001 <.0001 Bounds on condition number: 1, 1 -------------------------------------------------------------------------------Forward Selection: Step 2 Variable FEAVE Entered: R-Square = 0.9277 and C(p) = 82717.32 Analysis of Variance Source Model Error Corrected Total Variable Intercept FIN FEAVE DF 2 849 851 Parameter Estimate 19.42421 0.48772 0.29294 Sum of Squares 168264 13122 181385 Standard Error 0.71220 0.00804 0.01015 Mean Square 84132 15.45562 F Value 5443.44 Pr > F <.0001 Type II SS 11497 56807 12863 F Value 743.84 3675.51 832.25 Pr > F <.0001 <.0001 <.0001 Bounds on condition number: 1.6294, 6.5174 -------------------------------------------------------------------------------Forward Selection: Step 3 Variable MT2 Entered: R-Square = 0.9582 and C(p) = 47407.26 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT2 FIN FEAVE DF 3 848 851 Parameter Estimate Sum of Squares 173809 7576.82039 181385 Standard Error Mean Square 57936 8.93493 F Value 6484.23 Pr > F <.0001 Type II SS F Value Pr > F <.0001 <.0001 <.0001 <.0001 15.30901 0.56614 6533.27285 731.21 0.18589 0.00746 5544.99713 620.60 0.37063 0.00771 20626 2308.43 0.27418 0.00776 11162 1249.22 Bounds on condition number: 2.5915, 19.097 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Forward Selection: Step 4 Variable MT1 Entered: R-Square = 0.9736 and C(p) = 29678.25 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 FIN FEAVE DF 4 847 851 Parameter Estimate 4.78432 0.16893 0.14919 0.35127 0.27866 Sum of Squares 176593 4792.54759 181385 Standard Error 0.65428 0.00762 0.00616 0.00620 0.00618 Mean Square 44148 5.65826 F Value 7802.43 Pr > F <.0001 135 Type II SS 302.54827 2784.27280 3314.44255 18160 11518 F Value 53.47 492.07 585.77 3209.45 2035.55 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 2.6439, 31.642 -------------------------------------------------------------------------------Forward Selection: Step 5 Variable MT3 Entered: R-Square = 0.9864 and C(p) = 14912.02 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN FEAVE DF 5 846 851 Parameter Estimate 4.44458 0.16430 0.13533 0.14944 0.27569 0.22791 Sum of Squares 178912 2473.51490 181385 Standard Error 0.47048 0.00548 0.00446 0.00531 0.00520 0.00479 Mean Square 35782 2.92378 F Value 12238.4 Pr > F <.0001 Type II SS 260.93305 2631.50489 2693.92117 2319.03269 8209.32347 6614.50649 F Value 89.25 900.04 921.38 793.16 2807.78 2262.32 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.6024, 62.172 -------------------------------------------------------------------------------- ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Forward Selection: Step 6 Variable QAVE Entered: R-Square = 0.9960 and C(p) = 3815.968 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE DF 6 845 851 Parameter Estimate 0.99344 0.15208 0.13051 0.13287 0.24697 0.16123 0.16783 Sum of Squares 180655 730.80540 181385 Standard Error 0.26718 0.00299 0.00243 0.00291 0.00290 0.00359 0.00293 Mean Square 30109 0.86486 F Value 34813.9 Pr > F <.0001 136 Type II SS 11.95674 2235.80967 2500.74214 1803.68945 6268.13723 1742.70950 2838.20121 F Value 13.83 2585.17 2891.50 2085.53 7247.59 2015.02 3281.69 Pr > F 0.0002 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.7865, 95.557 -------------------------------------------------------------------------------Forward Selection: Step 7 Variable CPS Entered: R-Square = 0.9993 and C(p) = 8.0000 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS DF 7 844 851 Parameter Estimate 1.96402 0.15416 0.12828 0.12898 0.24745 0.13810 0.13773 0.99190 Sum of Squares 181253 132.53201 181385 Standard Error 0.11493 0.00127 0.00103 0.00124 0.00124 0.00158 0.00134 0.01607 Mean Square 25893 0.15703 F Value 164895 Pr > F <.0001 Type II SS 45.85809 2295.92649 2412.80003 1695.34430 6292.28246 1206.19651 1658.51306 598.27339 F Value 292.04 14621.1 15365.4 10796.4 40071.0 7681.39 10561.9 3809.97 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.7866, 129.67 -------------------------------------------------------------------------------All variables have been entered into the model. Summary of Forward Selection Variable Step Entered 1 2 3 4 5 6 7 FIN FEAVE MT2 MT1 MT3 QAVE CPS Label FIN FEAVE MT2 MT1 MT3 QAVE CPS Number Partial Model Vars In R-Square R-Square 1 2 3 4 5 6 7 0.8567 0.0709 0.0306 0.0154 0.0128 0.0096 0.0033 0.8567 0.9277 0.9582 0.9736 0.9864 0.9960 0.9993 C(p) F Value Pr > F 164630 5083.39 <.0001 82717.3 832.25 <.0001 47407.3 620.60 <.0001 29678.3 492.07 <.0001 14912.0 793.16 <.0001 3815.97 2015.02 <.0001 8.0000 3809.97 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Backward Elimination: Step 0 All Variables Entered: R-Square = 0.9993 and C(p) = 8.0000 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS DF 7 844 851 Parameter Estimate 1.96402 0.15416 0.12828 0.12898 0.24745 0.13810 0.13773 0.99190 Sum of Squares 181253 132.53201 181385 Standard Error 0.11493 0.00127 0.00103 0.00124 0.00124 0.00158 0.00134 0.01607 Mean Square 25893 0.15703 F Value 164895 Pr > F <.0001 852 852 137 Type II SS 45.85809 2295.92649 2412.80003 1695.34430 6292.28246 1206.19651 1658.51306 598.27339 F Value 292.04 14621.1 15365.4 10796.4 40071.0 7681.39 10561.9 3809.97 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.7866, 129.67 -------------------------------------------------------------------------------All variables left in the model are significant at the 0.1000 level. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Stepwise Selection: Step 1 Variable FIN Entered: R-Square = 0.8567 and C(p) = 164629.8 Analysis of Variance Source Model Error Corrected Total Variable Intercept FIN DF 1 850 851 Parameter Estimate 34.68343 0.63196 Sum of Squares 155401 25985 181385 Standard Error 0.67074 0.00886 Mean Square 155401 30.57027 F Value 5083.39 Pr > F <.0001 852 852 138 Type II SS 81740 155401 F Value 2673.85 5083.39 Pr > F <.0001 <.0001 Bounds on condition number: 1, 1 -------------------------------------------------------------------------------Stepwise Selection: Step 2 Variable FEAVE Entered: R-Square = 0.9277 and C(p) = 82717.32 Analysis of Variance Source Model Error Corrected Total Variable Intercept FIN FEAVE DF 2 849 851 Parameter Estimate 19.42421 0.48772 0.29294 Sum of Squares 168264 13122 181385 Standard Error 0.71220 0.00804 0.01015 Mean Square 84132 15.45562 F Value 5443.44 Pr > F <.0001 Type II SS 11497 56807 12863 F Value 743.84 3675.51 832.25 Pr > F <.0001 <.0001 <.0001 Bounds on condition number: 1.6294, 6.5174 -------------------------------------------------------------------------------Stepwise Selection: Step 3 Variable MT2 Entered: R-Square = 0.9582 and C(p) = 47407.26 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT2 FIN FEAVE DF 3 848 851 Parameter Estimate Sum of Squares 173809 7576.82039 181385 Standard Error Mean Square 57936 8.93493 F Value 6484.23 Pr > F <.0001 Type II SS F Value Pr > F <.0001 <.0001 <.0001 <.0001 15.30901 0.56614 6533.27285 731.21 0.18589 0.00746 5544.99713 620.60 0.37063 0.00771 20626 2308.43 0.27418 0.00776 11162 1249.22 Bounds on condition number: 2.5915, 19.097 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Stepwise Selection: Step 4 Variable MT1 Entered: R-Square = 0.9736 and C(p) = 29678.25 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 FIN FEAVE DF 4 847 851 Parameter Estimate 4.78432 0.16893 0.14919 0.35127 0.27866 Sum of Squares 176593 4792.54759 181385 Standard Error 0.65428 0.00762 0.00616 0.00620 0.00618 Mean Square 44148 5.65826 F Value 7802.43 Pr > F <.0001 139 Type II SS 302.54827 2784.27280 3314.44255 18160 11518 F Value 53.47 492.07 585.77 3209.45 2035.55 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 2.6439, 31.642 -------------------------------------------------------------------------------Stepwise Selection: Step 5 Variable MT3 Entered: R-Square = 0.9864 and C(p) = 14912.02 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN FEAVE DF 5 846 851 Parameter Estimate 4.44458 0.16430 0.13533 0.14944 0.27569 0.22791 Sum of Squares 178912 2473.51490 181385 Standard Error 0.47048 0.00548 0.00446 0.00531 0.00520 0.00479 Mean Square 35782 2.92378 F Value 12238.4 Pr > F <.0001 Type II SS 260.93305 2631.50489 2693.92117 2319.03269 8209.32347 6614.50649 F Value 89.25 900.04 921.38 793.16 2807.78 2262.32 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.6024, 62.172 -------------------------------------------------------------------------------Stepwise Selection: Step 6 Variable QAVE Entered: R-Square = 0.9960 and C(p) = 3815.968 Analysis of Variance Source Model Error Corrected Total DF 6 845 851 Sum of Squares 180655 730.80540 181385 Mean Square 30109 0.86486 F Value 34813.9 Pr > F <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Stepwise Selection: Step 6 Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE Parameter Estimate 0.99344 0.15208 0.13051 0.13287 0.24697 0.16123 0.16783 Standard Error 0.26718 0.00299 0.00243 0.00291 0.00290 0.00359 0.00293 Type II SS 11.95674 2235.80967 2500.74214 1803.68945 6268.13723 1742.70950 2838.20121 F Value 13.83 2585.17 2891.50 2085.53 7247.59 2015.02 3281.69 Pr > F 0.0002 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 140 Bounds on condition number: 3.7865, 95.557 -------------------------------------------------------------------------------Stepwise Selection: Step 7 Variable CPS Entered: R-Square = 0.9993 and C(p) = 8.0000 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS DF 7 844 851 Parameter Estimate 1.96402 0.15416 0.12828 0.12898 0.24745 0.13810 0.13773 0.99190 Sum of Squares 181253 132.53201 181385 Standard Error 0.11493 0.00127 0.00103 0.00124 0.00124 0.00158 0.00134 0.01607 Mean Square 25893 0.15703 F Value 164895 Pr > F <.0001 Type II SS 45.85809 2295.92649 2412.80003 1695.34430 6292.28246 1206.19651 1658.51306 598.27339 F Value 292.04 14621.1 15365.4 10796.4 40071.0 7681.39 10561.9 3809.97 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.7866, 129.67 -------------------------------------------------------------------------------All variables left in the model are significant at the 0.2500 level. All variables have been entered into the model. Summary of Stepwise Selection Variable Step Entered 1 2 3 4 5 6 7 FIN FEAVE MT2 MT1 MT3 QAVE CPS Variable Removed Label FIN FEAVE MT2 MT1 MT3 QAVE CPS Number Partial Model Vars In R-Square R-Square 1 2 3 4 5 6 7 0.8567 0.0709 0.0306 0.0154 0.0128 0.0096 0.0033 0.8567 0.9277 0.9582 0.9736 0.9864 0.9960 0.9993 C(p) F Value 164630 5083.39 82717.3 832.25 47407.3 620.60 29678.3 492.07 14912.0 793.16 3815.97 2015.02 8.0000 3809.97 Summary of Stepwise Selection Step Pr > F 1 2 3 4 5 6 7 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 141 Appendix III: Results for semesters with and without the use of CPS ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 142 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Combined.xls' OUT=COMBO; PROC PRINT DATA = COMBO PROC REG; MODEL FIN_AVE = Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE SEM; RUN; QUIT; /* SEM variable denotes which set of semesters that data comes from where 0 = semesters before CPS was used and 1 = semesters after CPS was used */ /* Multicollinearity diagnostics for the all semesters combined, including all tests, each of the quizzes and focus exercise grades, the quiz and focus exercise averages, and the SEM variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Number of Observations with Missing Values Analysis of Variance Source DF Sum of Squares 251933 620.89459 252554 0.77868 77.97093 0.99868 Mean Square 7872.89882 0.60634 R-Square Adj R-Sq F Value 12984.2 0.9975 0.9975 Pr > F <.0001 1620 1057 563 Model 32 Error 1024 Corrected Total 1056 Root MSE Dependent Mean Coeff Var Parameter Estimates Variable Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE SEM Label Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE SEM DF 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Parameter Estimate 2.60203 0.00363 0.00278 -0.00231 0.01300 -0.00017235 0.00474 0.00298 0.00516 0.00091394 -0.00031659 -0.00591 0.01261 -0.00477 0.00892 0.00427 0.00878 0.00678 0.01087 0.00334 0.00772 0.00830 -0.00372 0.00351 0.00905 -0.00157 0.15715 0.12817 0.13261 0.25076 0.14966 0.13617 -0.33782 Standard Error 0.22191 0.00330 0.00447 0.00470 0.00471 0.00444 0.00396 0.00399 0.00373 0.00448 0.00453 0.00414 0.00456 0.00430 0.00740 0.00517 0.00633 0.00699 0.00648 0.00540 0.00619 0.00593 0.00643 0.00660 0.00570 0.00500 0.00245 0.00183 0.00206 0.00209 0.00682 0.01170 0.05598 t Value 11.73 1.10 0.62 -0.49 2.76 -0.04 1.20 0.75 1.39 0.20 -0.07 -1.43 2.76 -1.11 1.20 0.83 1.39 0.97 1.68 0.62 1.25 1.40 -0.58 0.53 1.59 -0.31 64.26 69.94 64.46 120.02 21.95 11.64 -6.03 Pr > |t| <.0001 0.2717 0.5337 0.6223 0.0059 0.9690 0.2309 0.4543 0.1661 0.8385 0.9443 0.1537 0.0058 0.2671 0.2287 0.4089 0.1655 0.3320 0.0936 0.5364 0.2127 0.1619 0.5633 0.5949 0.1130 0.7529 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Parameter Estimates Variable Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE SEM Label Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN QAVE FEAVE SEM DF 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Variance Inflation 0 1.29103 1.54484 1.68411 1.62849 1.71395 1.65582 1.93178 1.89369 1.92413 2.02546 1.95360 2.26449 2.25215 1.93966 2.58563 2.46949 2.75580 2.97046 3.15085 3.37140 3.43529 3.04822 3.91339 3.67011 3.01542 1.62282 2.56929 3.42437 3.98110 21.64824 76.51822 1.35390 143 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Eigenvalue 11.89768 1.79078 1.65158 1.33551 1.08829 1.01070 0.92268 0.83892 0.82477 0.77209 0.74440 0.70329 0.67009 0.65491 0.61195 0.60177 0.56529 0.54486 0.52086 0.48747 0.48504 0.46760 0.45465 0.40387 0.38771 0.38098 0.34163 0.33189 0.27758 0.17782 0.04262 0.01074 Condition Index 1.00000 2.57757 2.68399 2.98475 3.30643 3.43099 3.59093 3.76593 3.79809 3.92553 3.99786 4.11304 4.21371 4.26225 4.40934 4.44649 4.58771 4.67294 4.77937 4.94034 4.95269 5.04421 5.11554 5.42766 5.53961 5.58828 5.90136 5.98736 6.54694 8.17974 16.70862 33.29094 --------Proportion of Variation-------Q1 Q2 Q3 0.00035122 0.00044609 0.00086979 0.03098 0.05363 0.00816 0.00464 0.00015160 0.00513 0.06183 0.02041 0.00226 0.04992 0.00766 0.07158 0.13077 0.00344 0.10492 0.03361 0.05355 0.01800 0.01939 0.01654 0.01423 0.00610 0.03656 0.04764 0.28463 0.24028 0.07273 0.00930 0.05004 0.02134 0.00043807 0.04012 0.02910 0.11680 0.09163 0.00004194 0.00238 0.00154 0.00212 0.00034198 0.00043170 0.06611 0.00601 0.01500 0.06244 0.02253 0.00417 0.01125 0.00645 0.09326 0.02332 0.00490 0.01815 0.00040253 0.00683 0.00765 0.00010027 0.00196 0.00386 0.01915 0.03497 0.00068592 0.00000363 0.02327 0.00313 0.19799 0.00162 7.968362E-8 0.00921 0.00102 0.00396 0.00079619 0.02995 0.01189 0.00037717 0.00004391 0.00587 0.00007005 0.00303 0.01419 0.00189 0.03001 0.00174 0.00735 0.00501 9.72292E-7 0.00000190 0.05532 0.16123 0.18597 0.01559 0.03878 0.01543 144 ----------------------Proportion of Variation---------------------Q4 Q5 Q6 Q7 Q8 0.00088911 0.00077793 0.00103 0.00116 0.00117 0.01389 0.00001855 0.02343 0.00835 0.00238 0.00068238 0.00914 0.00678 0.01229 0.01254 0.00118 0.05181 0.00073903 0.00132 0.00068084 0.00148 0.01542 0.00379 0.00450 0.00056111 0.00053532 0.06000 0.04109 0.00025062 0.00719 0.17102 0.01139 0.00707 0.06259 0.06434 0.00011980 0.07088 0.03846 0.00147 0.04510 0.14909 0.00187 0.03487 0.02345 0.00817 0.08122 0.00007780 0.00161 0.02196 0.00123 0.12046 0.15549 0.05846 0.02734 0.17569 0.03552 0.03146 0.02515 0.04165 0.01681 0.01163 0.01247 0.19944 0.00031576 0.00000569 0.03790 0.03495 0.00455 0.03086 0.03940 0.00164 0.06725 0.10459 0.04711 0.00092341 0.01509 0.13292 0.07046 0.05732 0.00154 0.00855 0.02825 0.01307 0.00046313 0.10983 0.02455 0.00742 0.00127 0.19967 0.00513 0.00024415 0.00032963 0.13406 0.04036 0.00182 0.00076847 0.00085848 0.01319 0.03407 0.02235 0.04741 0.00327 0.02420 0.05028 0.02170 0.00003377 0.00564 0.00028319 0.00967 0.10764 0.00361 0.00175 0.02654 0.08408 0.02096 0.01690 0.00112 0.00124 0.00238 0.02149 0.00460 0.01381 0.00333 0.00378 0.02450 0.00500 0.00412 0.00569 0.00145 0.06668 0.00068580 0.00320 0.00000432 0.00177 0.00023020 0.01911 0.01288 0.00143 0.00599 0.02035 0.01403 0.01176 0.00236 0.01856 0.00026466 0.00224 0.00155 0.00008316 0.00640 0.00736 0.18352 0.18952 0.11897 0.16189 0.14495 0.02642 0.05859 0.03274 0.03724 0.04700 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ----------------------Proportion of Variation---------------------Q9 Q10 Q11 Q12 Q13 0.00117 0.00120 0.00138 0.00144 0.00124 0.00829 0.00457 0.00369 0.00362 0.00925 0.00415 0.00356 0.00453 0.00298 0.00140 0.00003977 0.01405 0.01164 0.00017909 0.00023852 0.00007641 0.00958 0.02607 0.00813 0.02351 0.04439 0.00003646 0.00070845 0.00214 0.00049044 0.01609 0.00175 0.00017182 0.00232 0.00004572 0.02875 0.05860 0.00001806 0.01467 0.07264 0.01397 0.01593 0.00088580 0.00848 0.00481 0.01657 0.00113 0.00685 0.01001 0.00870 0.00329 0.01834 0.00933 0.00000862 0.00000508 0.11780 0.00001627 0.16560 0.00218 0.00288 0.00015824 0.00886 6.744034E-7 0.00259 0.00115 0.00491 0.08805 0.02005 0.00747 0.00284 0.01857 0.20174 0.01336 0.01412 0.00062955 0.26807 0.05456 0.02054 0.03669 0.01651 0.03244 0.00982 0.03624 0.07153 0.00257 0.07061 0.00352 0.00855 0.04548 0.09537 0.00877 0.09253 0.00777 0.11237 0.00082004 0.01431 0.00006427 0.05444 0.04972 0.20667 0.00001730 0.03694 0.06363 0.04372 0.14738 0.00204 0.07757 0.31220 0.00011239 0.01947 0.00094285 0.00002468 0.00080709 0.24769 0.03542 0.00070751 0.00769 0.05294 0.02066 8.962543E-7 0.04960 0.01060 0.00542 0.02426 0.03641 0.02734 0.00129 0.02504 0.05942 0.00173 0.01253 0.00022119 0.00001975 0.00165 0.00796 0.00340 0.02782 0.00030239 0.00854 0.00667 0.00340 0.00302 0.00874 0.00000142 0.06211 0.00009312 0.00378 0.00131 0.00031147 0.00007593 0.19153 0.21337 0.10131 0.18532 0.20756 0.03601 0.02978 0.03646 0.01220 0.02344 ----------------------Proportion of Variation---------------------F1 F2 F3 F4 F5 0.00079340 0.00074406 0.00067797 0.00097133 0.00088067 0.00844 0.01220 0.00387 0.00114 0.00056127 0.02503 0.02139 0.01975 0.01161 0.00885 0.00038522 0.00113 0.02808 0.01552 0.01361 0.02308 0.00253 0.00522 0.00556 0.01839 0.00483 0.01498 0.00033680 0.00006593 0.01224 0.00786 0.00053213 0.01750 0.00355 0.00273 0.00044180 0.00043338 0.02574 0.00129 0.00102 0.13974 0.00862 0.00575 0.00000245 0.01039 0.00015615 0.00431 0.00117 0.00054705 0.00000666 0.02174 0.01312 0.01771 0.00012919 5.25807E-11 0.00345 0.01910 0.00014714 0.00065643 0.02057 0.14082 0.05399 0.04485 0.03840 0.00915 0.04162 0.03989 0.00996 0.03100 0.06783 0.00066474 0.00472 0.11201 0.02063 0.01633 0.03949 0.00424 0.00467 0.01364 0.00224 0.04993 0.05485 0.03488 0.11136 0.00005243 0.01290 0.02972 0.01935 0.00409 0.00125 0.03639 0.00557 0.00103 0.01951 0.00116 0.01774 0.03059 0.03127 0.08616 0.02302 0.00045895 0.02144 0.09586 0.05726 0.00135 0.06300 0.08724 0.08008 0.00096360 0.03264 0.06045 0.12568 0.00319 0.03145 0.03242 0.00077207 0.00420 0.01431 0.05452 0.03463 0.00015347 0.00634 0.00176 0.07505 0.16080 0.01814 0.02535 0.01680 0.00189 0.05063 0.00000274 0.00003972 0.00034239 0.00766 0.07797 0.00207 0.01875 0.00042230 0.02474 0.00003455 0.00014910 0.00493 0.00056123 0.00741 0.00309 0.00301 0.00118 0.00012244 0.01346 0.00566 0.00706 0.00031610 0.00530 0.00664 0.00478 0.26925 0.38187 0.39730 0.35313 0.38570 145 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ----------------------Proportion of Variation---------------------F6 F7 F8 F9 F10 0.00103 0.00098776 0.00096001 0.00106 0.00088436 0.00008845 0.00022057 0.00425 0.00725 0.00598 0.00051279 0.00130 0.00309 0.00126 0.00332 0.00433 0.00095081 0.00220 0.00359 0.00433 0.03509 0.02596 0.00745 0.00000594 0.00001788 0.00042307 0.00200 0.00573 0.00051745 0.00106 0.00643 0.00664 0.00020960 0.00007729 0.00344 0.00090833 0.00790 0.00026707 0.02966 0.00651 0.00011172 0.00422 0.00002673 0.00423 0.00066125 0.00029426 0.00750 0.00265 0.00046574 0.00286 0.00051916 0.00733 0.00105 0.00223 0.00180 0.00109 0.01181 0.02304 0.03506 0.00523 0.00286 0.00007784 0.00509 0.00174 0.00132 0.01175 0.00496 0.07205 0.01375 0.00015950 0.02982 0.00291 0.00067936 0.00183 0.00153 0.00149 0.00503 0.00148 0.00030085 0.00407 0.06702 0.00007246 0.01116 0.00025386 0.00047857 0.04522 0.00190 0.01266 0.01788 0.00908 0.00479 0.07538 0.00111 0.13735 0.04875 0.00894 0.01514 0.01626 0.03096 0.03133 0.00465 0.00062100 0.00597 0.01480 0.01011 0.00840 0.06928 0.01592 0.00035116 0.00298 0.01078 0.00291 0.00642 0.01015 0.00120 0.01295 0.05637 0.08232 0.00086031 0.03788 0.07826 0.00019584 0.01927 0.01967 0.02720 0.19568 0.15599 0.00891 0.01351 0.03134 0.06827 0.00850 0.02992 0.07261 0.26358 0.00144 0.07117 0.24025 0.22114 0.03366 0.01005 0.04163 0.01377 0.00957 0.01438 0.00836 0.00062479 0.00774 0.01357 0.00186 0.00153 0.00839 0.00766 0.00151 0.01133 0.37690 0.40203 0.39043 0.33278 0.43169 ----------------------Proportion of Variation---------------------F11 F12 MT1 MT2 MT3 0.00099233 0.00103 0.00089657 0.00118 0.00125 0.00587 0.01035 0.03584 0.00568 0.00115 0.00311 0.00161 0.01618 0.02488 0.00400 0.00202 0.00405 0.00002294 0.00313 0.00060550 0.00126 0.00001148 0.00848 0.01356 0.00189 0.00039384 0.00283 0.08149 0.00106 0.01394 0.00008428 0.00156 0.09982 0.00013083 0.00395 0.00584 0.03550 0.00113 0.00159 0.00002440 0.00016499 0.00007484 0.00487 0.02465 0.01458 0.00096477 0.00001952 0.01573 0.00853 0.00136 0.00264 0.00246 0.00502 0.00001819 0.00165 0.00518 0.00006246 0.00373 0.00065661 0.00695 0.00005903 0.01578 0.00046647 0.00030618 0.00852 0.00362 0.02686 0.01706 0.00724 0.00291 0.00606 0.00647 0.00004289 0.00318 0.00532 0.00951 0.00120 0.00556 0.00012735 0.00304 0.01528 0.01594 0.00363 0.02024 0.00187 0.00008600 0.01190 0.00815 0.00079829 0.00828 0.00730 0.00031318 0.00015929 0.00430 0.00130 0.03042 0.07078 0.00054540 0.00002438 0.00033320 0.00865 0.04166 0.17231 0.01215 0.00528 0.00559 0.00568 0.00900 0.00226 0.00001792 0.00072041 0.00769 0.03198 0.00009964 0.01969 0.07356 0.11944 0.20588 0.09392 0.00105 0.15067 0.02804 0.10847 0.01984 0.01731 0.00746 0.00514 0.13136 0.02001 0.02992 0.22587 0.02018 0.00068720 0.01965 0.00189 0.02905 0.17563 0.01707 0.00926 0.02586 0.00073605 0.06276 0.00339 0.59816 0.26470 0.00313 0.00079151 0.00636 0.09985 0.54376 0.00092112 0.00327 0.00466 0.00216 0.00224 0.39277 0.32094 0.00000388 0.00136 0.00538 146 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 147 Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 -----------------Proportion of Variation---------------FIN QAVE FEAVE SEM 0.00109 0.00027495 0.00008029 0.00009944 0.00006610 0.00007352 0.00003315 0.01558 0.01009 0.00076099 0.00045985 0.01836 0.00005484 0.00000361 0.00000847 0.18201 0.00153 0.00162 0.00011889 0.04969 0.00909 0.00035137 9.165072E-7 0.00326 0.00171 0.00008513 2.400628E-7 0.00766 0.00000244 0.00000892 0.00000215 0.08842 0.01317 0.00076224 0.00000647 0.03878 0.00019354 0.00018854 0.00000852 0.02912 0.00015189 0.00000491 9.179691E-8 0.01165 0.00340 0.00025779 2.341146E-7 0.03249 0.00535 0.00000539 9.279708E-7 0.00617 0.00276 0.00011823 0.00000864 0.00023553 0.00245 0.00006373 0.00000133 0.03060 0.00157 0.00003243 0.00003479 0.00108 0.00173 0.00013248 0.00000612 0.02804 0.00340 0.00000969 0.00002429 0.12814 0.00011825 0.00010733 0.00002073 0.02188 0.00103 0.00002915 0.00001030 0.00340 0.00170 5.489377E-9 0.00000534 0.04393 0.00001130 0.00009035 7.792695E-7 0.00872 0.00779 0.00000250 0.00001002 0.01499 0.00614 0.00002543 2.764297E-7 0.01113 0.02661 5.829588E-8 0.00000619 0.00004556 0.00326 0.00007857 0.00001554 0.08145 0.01195 0.00040645 4.751982E-7 0.00284 0.00007970 0.00010881 0.00000243 0.00292 0.05843 0.00004534 0.00000106 0.05367 0.81602 0.00006942 0.00000169 0.00982 0.00881 0.85044 0.01051 0.06321 0.00024184 0.14384 0.98862 0.01061 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 148 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Combined.xls' OUT=COMBO; PROC PRINT DATA = COMBO PROC REG; MODEL FIN_AVE = Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 MT3 FIN SEM; RUN; QUIT; /* SEM variable denotes which set of semesters that data comes from where 0 = semesters before CPS was used and 1 = semesters after CPS was used */ /* Regression analysis for the all semesters combined, including all tests, each of the quizzes and focus exercise grades, and the SEM variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Number of Observations with Missing Values Analysis of Variance Source Model Error Corrected Total DF 30 1026 1056 Sum of Squares 251430 1123.55685 252554 1.04646 77.97093 1.34212 Mean Square 8381.00333 1.09508 R-Square Adj R-Sq F Value 7653.29 Pr > F <.0001 1620 1057 563 Root MSE Dependent Mean Coeff Var 0.9956 0.9954 Parameter Estimates Variable Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 Label Intercept Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 MT1 MT2 DF 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Parameter Estimate 4.94740 0.02110 0.04236 0.04630 0.06006 0.04187 0.03530 0.03904 0.03580 0.04519 0.04821 0.02309 0.06078 0.04143 0.06065 0.03558 0.05756 0.05897 0.05935 0.03881 0.05793 0.05543 0.03823 0.05998 0.04721 0.03206 0.16189 0.12647 Standard Error 0.27666 0.00428 0.00541 0.00569 0.00567 0.00521 0.00491 0.00482 0.00453 0.00535 0.00535 0.00517 0.00553 0.00512 0.00847 0.00547 0.00660 0.00753 0.00681 0.00574 0.00642 0.00620 0.00708 0.00666 0.00600 0.00554 0.00328 0.00246 t Value 17.88 4.93 7.83 8.14 10.60 8.04 7.19 8.09 7.91 8.44 9.00 4.47 11.00 8.10 7.16 6.50 8.73 7.83 8.72 6.76 9.02 8.94 5.40 9.00 7.87 5.79 49.38 51.40 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS MT3 FIN SEM MT3 FIN SEM 1 1 1 0.13725 0.25589 -0.01725 0.00275 0.00280 0.07256 49.87 91.55 -0.24 <.0001 <.0001 0.8121 149 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Combined.xls' OUT=COMBO; PROC PRINT DATA = COMBO PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE SEM; RUN; QUIT; /* SEM variable denotes which set of semesters that data comes from where 0 = semesters before CPS was used and 1 = semesters after CPS was used */ /* Regression analysis for the all semesters combined, including all tests, the averages of the quizzes and focus exercise grades, and the SEM variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 7 1612 1619 Sum of Squares 374661 1038.92071 375700 0.80280 78.79163 1.01889 Mean Square 53523 0.64449 F Value 83046.8 Pr > F <.0001 1620 1620 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9972 0.9972 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE SEM Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE SEM DF 1 1 1 1 1 1 1 1 Parameter Estimate 2.18271 0.15783 0.13018 0.13334 0.24873 0.15788 0.15541 -0.44508 Standard Error 0.15761 0.00188 0.00144 0.00164 0.00175 0.00220 0.00179 0.04064 t Value 13.85 83.74 90.47 81.29 142.13 71.83 86.83 -10.95 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 150 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Combined.xls' OUT=COMBO; PROC PRINT DATA = COMBO PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE SEM; RUN; QUIT; /* SEM variable denotes which set of semesters that data comes from where 0 = semesters before CPS was used and 1 = semesters after CPS was used */ /* Multicollinearity diagnostics for the all semesters combined, including all tests, the averages of the quizzes and focus exercise grades, and the SEM variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 7 1612 1619 Sum of Squares 374661 1038.92071 375700 0.80280 78.79163 1.01889 Mean Square 53523 0.64449 F Value 83046.8 Pr > F <.0001 1620 1620 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9972 0.9972 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE SEM Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE SEM DF 1 1 1 1 1 1 1 1 Parameter Estimate 2.18271 0.15783 0.13018 0.13334 0.24873 0.15788 0.15541 -0.44508 Standard Error 0.15761 0.00188 0.00144 0.00164 0.00175 0.00220 0.00179 0.04064 t Value 13.85 83.74 90.47 81.29 142.13 71.83 86.83 -10.95 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE SEM Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE SEM DF 1 1 1 1 1 1 1 1 Variance Inflation 0 1.42176 2.26440 3.09568 3.78907 2.91225 2.52861 1.03520 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 7 Eigenvalue 4.01353 0.98862 0.74910 0.46158 0.35358 0.25070 0.18289 Condition Index 1.00000 2.01488 2.31470 2.94877 3.36914 4.00119 4.68456 --------Proportion of Variation-------MT1 MT2 MT3 0.01667 0.00012030 0.66658 0.26090 0.02764 0.02465 0.00344 0.01805 0.00220 0.02314 0.35302 0.41467 0.08204 0.10688 0.01487 0.00372 0.01563 0.02107 0.37951 0.01018 0.55502 151 Collinearity Diagnostics (intercept adjusted) Number 1 2 3 4 5 6 7 -----------------Proportion of Variation---------------FIN QAVE FEAVE SEM 0.01324 3.098853E-7 3.216007E-7 0.05755 0.06658 0.04309 0.81955 0.01582 0.00100 0.01816 0.08094 0.10253 0.71866 0.06289 0.01599 0.00619 0.08621 0.18086 0.04421 0.58397 0.08258 0.00185 0.94307 0.01505 0.01912 0.01202 0.00362 0.00527 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 152 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Combined.xls' OUT=COMBO; PROC PRINT DATA = COMBO TITLE 'STAT 210 CLASS GRADES'; PROC RSQUARE; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE SEM / ADJRSQ; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE SEM / SELECTION = BACKWARD; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE SEM / SELECTION = FORWARD; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE SEM / SELECTION = STEPWISE SLENTRY = .25 SLSTAY = .25; RUN; QUIT; /* SEM variable denotes which set of semesters that data comes from where 0 = semesters before CPS was used and 1 = semesters after CPS was used */ /* Model selection for the all semesters combined, including all tests, the averages of the quizzes and focus exercise grades, and the SEM variable*/ The RSQUARE Procedure Model: MODEL1 Dependent Variable: FIN_AVE R-Square Selection Method Number of Observations Read Number of Observations Used Number in Model R-Square Adjusted R-Square 1620 1620 Variables in Model 1 0.8609 0.8608 FIN 1 0.7463 0.7462 MT3 1 0.7054 0.7052 QAVE 1 0.6500 0.6498 MT2 1 0.6266 0.6264 FEAVE 1 0.3669 0.3665 MT1 1 0.0149 0.0143 SEM -------------------------------------------------------------------2 0.9304 0.9303 FIN FEAVE 2 0.9303 0.9302 FIN QAVE 2 0.9085 0.9084 MT3 FIN 2 0.9016 0.9015 MT2 FIN 2 0.8884 0.8883 MT1 FIN 2 0.8723 0.8721 MT2 MT3 2 0.8659 0.8657 MT3 QAVE 2 0.8611 0.8609 FIN SEM 2 0.8444 0.8442 MT2 QAVE 2 0.8383 0.8381 MT2 FEAVE 2 0.8252 0.8250 MT3 FEAVE 2 0.8227 0.8225 MT1 MT3 2 0.7748 0.7746 MT1 QAVE 2 0.7703 0.7700 QAVE FEAVE 2 0.7574 0.7571 MT1 FEAVE 2 0.7497 0.7494 MT3 SEM 2 0.7072 0.7068 MT1 MT2 2 0.7060 0.7056 QAVE SEM 2 0.6500 0.6496 MT2 SEM 2 0.6308 0.6303 FEAVE SEM 2 0.3729 0.3721 MT1 SEM -------------------------------------------------------------------3 0.9591 0.9591 MT2 FIN FEAVE 3 0.9555 0.9554 MT1 FIN FEAVE 3 0.9530 0.9529 MT2 FIN QAVE 3 0.9526 0.9525 MT3 FIN QAVE 3 0.9516 0.9515 FIN QAVE FEAVE 3 0.9470 0.9469 MT1 FIN QAVE 3 0.9469 0.9468 MT3 FIN FEAVE 3 0.9412 0.9411 MT2 MT3 FIN ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 3 0.9336 0.9335 MT1 MT3 FIN 3 0.9307 0.9305 FIN QAVE SEM 3 0.9304 0.9303 FIN FEAVE SEM 3 0.9302 0.9301 MT2 MT3 QAVE 3 0.9193 0.9191 MT2 MT3 FEAVE 3 0.9169 0.9168 MT1 MT2 FIN 3 0.9085 0.9083 MT3 FIN SEM 3 0.9054 0.9052 MT1 MT3 QAVE 3 0.9025 0.9023 MT2 FIN SEM 3 0.8999 0.8997 MT1 MT2 MT3 3 0.8893 0.8891 MT1 MT3 FEAVE 3 0.8886 0.8884 MT1 FIN SEM 3 0.8862 0.8860 MT2 QAVE FEAVE 3 0.8802 0.8800 MT3 QAVE FEAVE 3 0.8764 0.8762 MT1 MT2 FEAVE 3 0.8724 0.8721 MT2 MT3 SEM 3 0.8678 0.8676 MT1 MT2 QAVE 3 0.8666 0.8663 MT3 QAVE SEM 3 0.8447 0.8444 MT2 QAVE SEM 3 0.8383 0.8380 MT2 FEAVE SEM 3 0.8374 0.8371 MT1 QAVE FEAVE 3 0.8277 0.8274 MT3 FEAVE SEM 3 0.8247 0.8244 MT1 MT3 SEM 3 0.7751 0.7747 MT1 QAVE SEM 3 0.7711 0.7707 QAVE FEAVE SEM 3 0.7595 0.7591 MT1 FEAVE SEM 3 0.7072 0.7066 MT1 MT2 SEM -------------------------------------------------------------------4 0.9744 0.9743 MT1 MT2 FIN FEAVE 4 0.9734 0.9733 MT2 MT3 FIN FEAVE 4 0.9733 0.9733 MT2 MT3 FIN QAVE 4 0.9733 0.9733 MT2 FIN QAVE FEAVE 4 0.9710 0.9710 MT1 MT3 FIN FEAVE 4 0.9706 0.9705 MT1 FIN QAVE FEAVE 4 0.9695 0.9695 MT1 MT3 FIN QAVE 4 0.9645 0.9644 MT3 FIN QAVE FEAVE 4 0.9631 0.9630 MT1 MT2 FIN QAVE 4 0.9595 0.9594 MT2 FIN FEAVE SEM 4 0.9559 0.9558 MT1 MT2 MT3 FIN 4 0.9555 0.9554 MT1 FIN FEAVE SEM 4 0.9539 0.9538 MT2 FIN QAVE SEM 4 0.9526 0.9525 MT3 FIN QAVE SEM 4 0.9517 0.9516 FIN QAVE FEAVE SEM 4 0.9479 0.9477 MT1 MT2 MT3 QAVE 4 0.9474 0.9472 MT1 FIN QAVE SEM 4 0.9469 0.9468 MT3 FIN FEAVE SEM 4 0.9453 0.9452 MT1 MT2 MT3 FEAVE 4 0.9432 0.9430 MT2 MT3 QAVE FEAVE 4 0.9414 0.9413 MT2 MT3 FIN SEM 4 0.9336 0.9335 MT1 MT3 FIN SEM 4 0.9302 0.9301 MT2 MT3 QAVE SEM 4 0.9221 0.9219 MT1 MT3 QAVE FEAVE 4 0.9193 0.9191 MT2 MT3 FEAVE SEM 4 0.9177 0.9174 MT1 MT2 FIN SEM 4 0.9115 0.9113 MT1 MT2 QAVE FEAVE 4 0.9058 0.9056 MT1 MT3 QAVE SEM 4 0.9000 0.8997 MT1 MT2 MT3 SEM 4 0.8908 0.8905 MT1 MT3 FEAVE SEM 4 0.8863 0.8860 MT2 QAVE FEAVE SEM 4 0.8811 0.8808 MT3 QAVE FEAVE SEM 4 0.8764 0.8761 MT1 MT2 FEAVE SEM 4 0.8681 0.8677 MT1 MT2 QAVE SEM 4 0.8379 0.8375 MT1 QAVE FEAVE SEM 153 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The RSQUARE Procedure Model: MODEL1 Dependent Variable: FIN_AVE R-Square Selection Method Number in Model R-Square Adjusted R-Square Variables in Model 154 -------------------------------------------------------------------5 0.9883 0.9882 MT1 MT2 MT3 FIN FEAVE 5 0.9854 0.9854 MT1 MT2 FIN QAVE FEAVE 5 0.9849 0.9849 MT2 MT3 FIN QAVE FEAVE 5 0.9840 0.9840 MT1 MT2 MT3 FIN QAVE 5 0.9832 0.9831 MT1 MT3 FIN QAVE FEAVE 5 0.9747 0.9747 MT1 MT2 FIN FEAVE SEM 5 0.9739 0.9738 MT2 FIN QAVE FEAVE SEM 5 0.9737 0.9736 MT2 MT3 FIN QAVE SEM 5 0.9735 0.9734 MT2 MT3 FIN FEAVE SEM 5 0.9710 0.9709 MT1 MT3 FIN FEAVE SEM 5 0.9707 0.9706 MT1 FIN QAVE FEAVE SEM 5 0.9696 0.9695 MT1 MT3 FIN QAVE SEM 5 0.9645 0.9644 MT3 FIN QAVE FEAVE SEM 5 0.9639 0.9638 MT1 MT2 FIN QAVE SEM 5 0.9626 0.9625 MT1 MT2 MT3 QAVE FEAVE 5 0.9561 0.9559 MT1 MT2 MT3 FIN SEM 5 0.9479 0.9477 MT1 MT2 MT3 QAVE SEM 5 0.9453 0.9452 MT1 MT2 MT3 FEAVE SEM 5 0.9432 0.9430 MT2 MT3 QAVE FEAVE SEM 5 0.9226 0.9224 MT1 MT3 QAVE FEAVE SEM 5 0.9116 0.9113 MT1 MT2 QAVE FEAVE SEM -------------------------------------------------------------------6 0.9970 0.9970 MT1 MT2 MT3 FIN QAVE FEAVE 6 0.9884 0.9883 MT1 MT2 MT3 FIN FEAVE SEM 6 0.9859 0.9858 MT1 MT2 FIN QAVE FEAVE SEM 6 0.9852 0.9852 MT2 MT3 FIN QAVE FEAVE SEM 6 0.9843 0.9842 MT1 MT2 MT3 FIN QAVE SEM 6 0.9832 0.9831 MT1 MT3 FIN QAVE FEAVE SEM 6 0.9626 0.9624 MT1 MT2 MT3 QAVE FEAVE SEM -------------------------------------------------------------------7 0.9972 0.9972 MT1 MT2 MT3 FIN QAVE FEAVE SEM ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Backward Elimination: Step 0 All Variables Entered: R-Square = 0.9972 and C(p) = 8.0000 Analysis of Variance Source Model Error Corrected Total DF 7 1612 1619 Parameter Estimate 2.18271 0.15783 0.13018 0.13334 0.24873 0.15788 0.15541 -0.44508 Sum of Squares 374661 1038.92071 375700 Standard Error 0.15761 0.00188 0.00144 0.00164 0.00175 0.00220 0.00179 0.04064 Mean Square 53523 0.64449 F Value 83046.8 Pr > F <.0001 1620 1620 155 Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE SEM Type II SS 123.60281 4518.97627 5275.46066 4259.25262 13019 3325.30665 4859.14886 77.29399 F Value 191.78 7011.69 8185.46 6608.70 20200.1 5159.58 7539.51 119.93 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.7891, 119.33 -------------------------------------------------------------------------------All variables left in the model are significant at the 0.1000 level. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Forward Selection: Step 1 Variable FIN Entered: R-Square = 0.8609 and C(p) = 79482.89 Analysis of Variance Source Model Error Corrected Total Variable Intercept FIN DF 1 1618 1619 Parameter Estimate 34.51852 0.63690 Sum of Squares 323432 52268 375700 Standard Error 0.46445 0.00637 Mean Square 323432 32.30381 F Value 10012.2 Pr > F <.0001 1620 1620 156 Type II SS 178436 323432 F Value 5523.69 10012.2 Pr > F <.0001 <.0001 Bounds on condition number: 1, 1 -------------------------------------------------------------------------------Forward Selection: Step 2 Variable FEAVE Entered: R-Square = 0.9304 and C(p) = 38976.78 Analysis of Variance Source Model Error Corrected Total Variable Intercept FIN FEAVE DF 2 1617 1619 Parameter Estimate 19.46103 0.48884 0.29270 Sum of Squares 349540 26160 375700 Standard Error 0.49853 0.00582 0.00729 Mean Square 174770 16.17837 F Value 10802.7 Pr > F <.0001 Type II SS 24654 114126 26107 F Value 1523.86 7054.21 1613.71 Pr > F <.0001 <.0001 <.0001 Bounds on condition number: 1.6695, 6.6781 -------------------------------------------------------------------------------Forward Selection: Step 3 Variable MT2 Entered: R-Square = 0.9591 and C(p) = 22213.58 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT2 FIN FEAVE DF 3 1616 1619 Parameter Estimate Sum of Squares 360345 15355 375700 Standard Error Mean Square 120115 9.50210 F Value 12640.9 Pr > F <.0001 Type II SS F Value Pr > F <.0001 <.0001 <.0001 <.0001 15.92009 0.39623 15340 1614.34 0.17917 0.00531 10805 1137.12 0.38027 0.00550 45406 4778.53 0.26843 0.00563 21598 2272.98 Bounds on condition number: 2.5393, 18.992 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Forward Selection: Step 4 Variable MT1 Entered: R-Square = 0.9744 and C(p) = 13314.62 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 FIN FEAVE DF 4 1615 1619 Parameter Estimate 5.35882 0.17650 0.14917 0.35055 0.26841 Sum of Squares 366081 9618.79220 375700 Standard Error 0.46283 0.00569 0.00432 0.00446 0.00446 Mean Square 91520 5.95591 F Value 15366.3 Pr > F <.0001 157 Type II SS 798.44158 5736.59997 7114.41517 36805 21596 F Value 134.06 963.18 1194.51 6179.66 3625.93 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 2.6621, 31.859 -------------------------------------------------------------------------------Forward Selection: Step 5 Variable MT3 Entered: R-Square = 0.9883 and C(p) = 5232.034 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN FEAVE DF 5 1614 1619 Parameter Estimate 4.78079 0.17436 0.14249 0.14638 0.27106 0.21641 Sum of Squares 371292 4408.34565 375700 Standard Error 0.31370 0.00385 0.00293 0.00335 0.00353 0.00324 Mean Square 74258 2.73132 F Value 27187.7 Pr > F <.0001 Type II SS 634.35396 5597.15943 6473.14919 5210.44655 16144 12149 F Value 232.25 2049.25 2369.97 1907.67 5910.67 4448.04 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.6289, 61.257 -------------------------------------------------------------------------------- ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Forward Selection: Step 6 Variable QAVE Entered: R-Square = 0.9970 and C(p) = 125.9302 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE DF 6 1613 1619 Parameter Estimate 2.04586 0.15822 0.12893 0.13458 0.24725 0.15698 0.15591 Sum of Squares 374584 1116.21471 375700 Standard Error 0.16281 0.00195 0.00149 0.00170 0.00181 0.00228 0.00185 Mean Square 62431 0.69201 F Value 90216.2 Pr > F <.0001 158 Type II SS 109.27623 4542.58142 5207.47824 4359.47405 12942 3292.13095 4894.26332 F Value 157.91 6564.31 7525.13 6299.71 18701.3 4757.33 7072.52 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.7663, 95.723 -------------------------------------------------------------------------------Forward Selection: Step 7 Variable SEM Entered: R-Square = 0.9972 and C(p) = 8.0000 Analysis of Variance Source Model Error Corrected Total DF 7 1612 1619 Parameter Estimate 2.18271 0.15783 0.13018 0.13334 0.24873 0.15788 0.15541 -0.44508 Sum of Squares 374661 1038.92071 375700 Standard Error 0.15761 0.00188 0.00144 0.00164 0.00175 0.00220 0.00179 0.04064 Mean Square 53523 0.64449 F Value 83046.8 Pr > F <.0001 Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE SEM Type II SS 123.60281 4518.97627 5275.46066 4259.25262 13019 3325.30665 4859.14886 77.29399 F Value 191.78 7011.69 8185.46 6608.70 20200.1 5159.58 7539.51 119.93 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.7891, 119.33 -------------------------------------------------------------------------------All variables have been entered into the model. Summary of Forward Selection Variable Step Entered 1 2 3 4 5 6 7 FIN FEAVE MT2 MT1 MT3 QAVE SEM Label FIN FEAVE MT2 MT1 MT3 QAVE SEM Number Partial Model Vars In R-Square R-Square 1 2 3 4 5 6 7 0.8609 0.0695 0.0288 0.0153 0.0139 0.0088 0.0002 0.8609 0.9304 0.9591 0.9744 0.9883 0.9970 0.9972 C(p) 79482.9 38976.8 22213.6 13314.6 5232.03 125.930 8.0000 F Value Pr > F 10012.2 1613.71 1137.12 963.18 1907.67 4757.33 119.93 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Stepwise Selection: Step 1 Variable FIN Entered: R-Square = 0.8609 and C(p) = 79482.89 Analysis of Variance Source Model Error Corrected Total Variable Intercept FIN DF 1 1618 1619 Parameter Estimate 34.51852 0.63690 Sum of Squares 323432 52268 375700 Standard Error 0.46445 0.00637 Mean Square 323432 32.30381 F Value 10012.2 Pr > F <.0001 1620 1620 159 Type II SS 178436 323432 F Value 5523.69 10012.2 Pr > F <.0001 <.0001 Bounds on condition number: 1, 1 -------------------------------------------------------------------------------Stepwise Selection: Step 2 Variable FEAVE Entered: R-Square = 0.9304 and C(p) = 38976.78 Analysis of Variance Source Model Error Corrected Total Variable Intercept FIN FEAVE DF 2 1617 1619 Parameter Estimate 19.46103 0.48884 0.29270 Sum of Squares 349540 26160 375700 Standard Error 0.49853 0.00582 0.00729 Mean Square 174770 16.17837 F Value 10802.7 Pr > F <.0001 Type II SS 24654 114126 26107 F Value 1523.86 7054.21 1613.71 Pr > F <.0001 <.0001 <.0001 Bounds on condition number: 1.6695, 6.6781 -------------------------------------------------------------------------------Stepwise Selection: Step 3 Variable MT2 Entered: R-Square = 0.9591 and C(p) = 22213.58 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT2 FIN FEAVE DF 3 1616 1619 Parameter Estimate Sum of Squares 360345 15355 375700 Standard Error Mean Square 120115 9.50210 F Value 12640.9 Pr > F <.0001 Type II SS F Value Pr > F <.0001 <.0001 <.0001 <.0001 15.92009 0.39623 15340 1614.34 0.17917 0.00531 10805 1137.12 0.38027 0.00550 45406 4778.53 0.26843 0.00563 21598 2272.98 Bounds on condition number: 2.5393, 18.992 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Stepwise Selection: Step 4 Variable MT1 Entered: R-Square = 0.9744 and C(p) = 13314.62 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 FIN FEAVE DF 4 1615 1619 Parameter Estimate 5.35882 0.17650 0.14917 0.35055 0.26841 Sum of Squares 366081 9618.79220 375700 Standard Error 0.46283 0.00569 0.00432 0.00446 0.00446 Mean Square 91520 5.95591 F Value 15366.3 Pr > F <.0001 160 Type II SS 798.44158 5736.59997 7114.41517 36805 21596 F Value 134.06 963.18 1194.51 6179.66 3625.93 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 2.6621, 31.859 -------------------------------------------------------------------------------Stepwise Selection: Step 5 Variable MT3 Entered: R-Square = 0.9883 and C(p) = 5232.034 Analysis of Variance Source Model Error Corrected Total Intercept MT1 MT2 MT3 FIN FEAVE DF 5 1614 1619 4.78079 0.17436 0.14249 0.14638 0.27106 0.21641 Sum of Squares 371292 4408.34565 375700 0.31370 0.00385 0.00293 0.00335 0.00353 0.00324 Mean Square 74258 2.73132 634.35396 5597.15943 6473.14919 5210.44655 16144 12149 F Value 27187.7 Pr > F <.0001 232.25 2049.25 2369.97 1907.67 5910.67 4448.04 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.6289, 61.257 -------------------------------------------------------------------------------Stepwise Selection: Step 6 Variable QAVE Entered: R-Square = 0.9970 and C(p) = 125.9302 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE DF 6 1613 1619 Parameter Estimate 2.04586 0.15822 0.12893 0.13458 0.24725 0.15698 0.15591 Sum of Squares 374584 1116.21471 375700 Standard Error 0.16281 0.00195 0.00149 0.00170 0.00181 0.00228 0.00185 Mean Square 62431 0.69201 F Value 90216.2 Pr > F <.0001 Type II SS 109.27623 4542.58142 5207.47824 4359.47405 12942 3292.13095 4894.26332 F Value 157.91 6564.31 7525.13 6299.71 18701.3 4757.33 7072.52 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.7663, 95.723 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Stepwise Selection: Step 7 Variable SEM Entered: R-Square = 0.9972 and C(p) = 8.0000 Analysis of Variance Source Model Error Corrected Total Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE SEM DF 7 1612 1619 Parameter Estimate 2.18271 0.15783 0.13018 0.13334 0.24873 0.15788 0.15541 -0.44508 Sum of Squares 374661 1038.92071 375700 Standard Error 0.15761 0.00188 0.00144 0.00164 0.00175 0.00220 0.00179 0.04064 Mean Square 53523 0.64449 F Value 83046.8 Pr > F <.0001 161 Type II SS 123.60281 4518.97627 5275.46066 4259.25262 13019 3325.30665 4859.14886 77.29399 F Value 191.78 7011.69 8185.46 6608.70 20200.1 5159.58 7539.51 119.93 Pr > F <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 Bounds on condition number: 3.7891, 119.33 -------------------------------------------------------------------------------All variables left in the model are significant at the 0.2500 level. All variables have been entered into the model. Summary of Stepwise Selection Variable Step Entered 1 2 3 4 5 6 7 FIN FEAVE MT2 MT1 MT3 QAVE SEM Variable Removed Label FIN FEAVE MT2 MT1 MT3 QAVE SEM Number Partial Model Vars In R-Square R-Square 1 2 3 4 5 6 7 0.8609 0.0695 0.0288 0.0153 0.0139 0.0088 0.0002 0.8609 0.9304 0.9591 0.9744 0.9883 0.9970 0.9972 C(p) 79482.9 38976.8 22213.6 13314.6 5232.03 125.930 8.0000 F Value 10012.2 1613.71 1137.12 963.18 1907.67 4757.33 119.93 Summary of Stepwise Selection Step Pr > F 1 2 3 4 5 6 7 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 162 OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Combined.xls' OUT=COMBO; PROC PRINT DATA = COMBO PROC REG; MODEL FIN_AVE = SEM; RUN; QUIT; /* SEM variable denotes which set of semesters that data comes from where 0 = semesters before CPS was used and 1 = semesters after CPS was used */ /* Regression analysis for the final grade average versus the SEM variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 1 1618 1619 Sum of Squares 5614.29573 370086 375700 15.12383 78.79163 19.19472 Mean Square 5614.29573 228.73034 F Value 24.55 Pr > F <.0001 1620 1620 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.0149 0.0143 Parameter Estimates Variable Intercept SEM Label Intercept SEM DF 1 1 Parameter Estimate 76.83084 3.72825 Standard Error 0.54573 0.75252 t Value 140.78 4.95 Pr > |t| <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Combined.xls' OUT=COMBO; PROC PRINT DATA = COMBO PROC REG; MODEL QAVE = SEM; RUN; QUIT; 163 /* SEM variable denotes which set of semesters that data comes from where 0 = semesters before CPS was used and 1 = semesters after CPS was used */ /* Regression analysis for the quiz average versus the SEM variable*/ The REG Procedure Model: MODEL1 Dependent Variable: QAVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 1 1618 1619 Sum of Squares 5387.12246 383116 388503 15.38778 82.13789 18.73408 Mean Square 5387.12246 236.78368 F Value 22.75 Pr > F <.0001 1620 1620 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.0139 0.0133 Parameter Estimates Variable Intercept SEM Label Intercept SEM DF 1 1 Parameter Estimate 80.21719 3.65204 Standard Error 0.55526 0.76565 t Value 144.47 4.77 Pr > |t| <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Combined.xls' OUT=COMBO; PROC PRINT DATA = COMBO PROC REG; MODEL FEAVE = SEM; RUN; QUIT; 164 /* SEM variable denotes which set of semesters that data comes from where 0 = semesters before CPS was used and 1 = semesters after CPS was used */ /* Regression analysis for the focus exercise average versus the SEM variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FEAVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 1 1618 1619 Sum of Squares 2702.02094 506055 508757 17.68518 86.60696 20.42005 Mean Square 2702.02094 312.76565 F Value 8.64 Pr > F 0.0033 1620 1620 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.0053 0.0047 Parameter Estimates Variable Intercept SEM Label Intercept SEM DF 1 1 Parameter Estimate 85.24669 2.58643 Standard Error 0.63816 0.87997 t Value 133.58 2.94 Pr > |t| <.0001 0.0033 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Combined.xls' OUT=COMBO; PROC PRINT DATA = COMBO PROC REG; MODEL FIN = SEM; RUN; QUIT; 165 /* SEM variable denotes which set of semesters that data comes from where 0 = semesters before CPS was used and 1 = semesters after CPS was used */ /* Regression analysis for the final versus the SEM variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 1 1618 1619 Sum of Squares 17039 780299 797339 21.96046 69.51358 31.59161 Mean Square 17039 482.26165 F Value 35.33 Pr > F <.0001 1620 1620 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.0214 0.0208 Parameter Estimates Variable Intercept SEM Label Intercept SEM DF 1 1 Parameter Estimate 66.09766 6.49507 Standard Error 0.79243 1.09269 t Value 83.41 5.94 Pr > |t| <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Combined.xls' OUT=COMBO; PROC PRINT DATA = COMBO PROC REG; MODEL MT1 = SEM; RUN; QUIT; 166 /* SEM variable denotes which set of semesters that data comes from where 0 = semesters before CPS was used and 1 = semesters after CPS was used */ /* Regression analysis for the MT1 versus the SEM variable*/ The REG Procedure Model: MODEL1 Dependent Variable: MT1 Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 1 1618 1619 Sum of Squares 1448.64300 256468 257917 12.59005 84.06111 14.97726 Mean Square 1448.64300 158.50946 F Value 9.14 Pr > F 0.0025 1620 1620 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.0056 0.0050 Parameter Estimates Variable Intercept SEM Label Intercept SEM DF 1 1 Parameter Estimate 83.06510 1.89382 Standard Error 0.45430 0.62645 t Value 182.84 3.02 Pr > |t| <.0001 0.0025 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Combined.xls' OUT=COMBO; PROC PRINT DATA = COMBO PROC REG; MODEL MT2 = SEM; RUN; QUIT; 167 /* SEM variable denotes which set of semesters that data comes from where 0 = semesters before CPS was used and 1 = semesters after CPS was used */ /* Regression analysis for the MT2 versus the SEM variable*/ The REG Procedure Model: MODEL1 Dependent Variable: MT2 Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 1 1618 1619 Sum of Squares 17427 687478 704905 20.61295 73.61728 28.00015 Mean Square 17427 424.89376 F Value 41.01 Pr > F <.0001 1620 1620 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.0247 0.0241 Parameter Estimates Variable Intercept SEM Label Intercept SEM DF 1 1 Parameter Estimate 70.16276 6.56846 Standard Error 0.74381 1.02565 t Value 94.33 6.40 Pr > |t| <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS NODATE LS=80; PROC IMPORT DATAFILE = 'f:\Applied Project\Combined.xls' OUT=COMBO; PROC PRINT DATA = COMBO PROC REG; MODEL MT3 = SEM; RUN; QUIT; 168 /* SEM variable denotes which set of semesters that data comes from where 0 = semesters before CPS was used and 1 = semesters after CPS was used */ /* Regression analysis for the MT3 versus the SEM variable*/ The REG Procedure Model: MODEL1 Dependent Variable: MT3 Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 1 1618 1619 Sum of Squares 4108.75983 737448 741556 21.34894 77.05370 27.70657 Mean Square 4108.75983 455.77724 F Value 9.01 Pr > F 0.0027 1620 1620 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.0055 0.0049 Parameter Estimates Variable Intercept SEM Label Intercept SEM DF 1 1 Parameter Estimate 75.37630 3.18943 Standard Error 0.77036 1.06227 t Value 97.85 3.00 Pr > |t| <.0001 0.0027 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 169 Appendix IV: Results for the individual semesters ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS LS=80 NODATE; PROC IMPORT DATAFILE = 'f:\Applied Project\Spring 2002.xls' OUT=SPRING02; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE; RUN; QUIT; /* Regression analysis for the spring of 2002, including all tests and averages of the quizzes and focus exercises*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 6 166 172 Sum of Squares 47216 48.10158 47264 0.53830 72.45788 0.74292 Mean Square 7869.30439 0.28977 F Value 27157.2 Pr > F <.0001 173 173 170 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9990 0.9989 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE DF 1 1 1 1 1 1 1 Parameter Estimate 2.56281 0.16393 0.14126 0.12916 0.24495 0.15621 0.14262 Standard Error 0.31233 0.00411 0.00297 0.00313 0.00340 0.00463 0.00349 t Value 8.21 39.89 47.56 41.32 71.98 33.73 40.81 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS LS=80 NODATE; PROC IMPORT DATAFILE = 'f:\Applied Project\Fall 2002.xls' OUT=FALL02; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE; RUN; QUIT; /* Regression analysis for the fall of 2002, including all tests and averages of the quizzes and focus exercises*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 6 179 185 Sum of Squares 38570 38.87416 38609 0.46602 78.85992 0.59095 Mean Square 6428.41339 0.21717 F Value 29600.3 Pr > F <.0001 186 186 171 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9990 0.9990 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE DF 1 1 1 1 1 1 1 Parameter Estimate 3.78543 0.16376 0.12792 0.12266 0.25723 0.15613 0.13764 Standard Error 0.28949 0.00388 0.00201 0.00226 0.00268 0.00392 0.00321 t Value 13.08 42.25 63.61 54.25 96.13 39.87 42.82 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS LS=80 NODATE; PROC IMPORT DATAFILE = 'f:\Applied Project\Spring 2003.xls' OUT=SPRING03; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE; RUN; QUIT; /* Regression analysis for the spring of 2003, including all tests and averages of the quizzes and focus exercises*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 6 212 218 Sum of Squares 54203 47.80651 54251 0.47487 76.55205 0.62032 Mean Square 9033.87338 0.22550 F Value 40061.1 Pr > F <.0001 219 219 172 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9991 0.9991 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE DF 1 1 1 1 1 1 1 Parameter Estimate 1.57147 0.18102 0.13183 0.14214 0.23973 0.14620 0.14851 Standard Error 0.24090 0.00319 0.00254 0.00285 0.00295 0.00391 0.00304 t Value 6.52 56.79 51.82 49.93 81.40 37.41 48.90 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS LS=80 NODATE; PROC IMPORT DATAFILE = 'f:\Applied Project\Fall 2003.xls' OUT=FALL03; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE; RUN; QUIT; /* Regression analysis for the fall of 2003, including all tests and averages of the quizzes and focus exercises*/ 173 The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 6 183 189 Sum of Squares 43416 49.35007 43465 0.51930 79.14752 0.65612 Mean Square 7235.97966 0.26967 F Value 26832.5 Pr > F <.0001 190 190 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9989 0.9988 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE DF 1 1 1 1 1 1 1 Parameter Estimate 2.84653 0.15832 0.12510 0.14905 0.24835 0.14688 0.14840 Standard Error 0.30464 0.00394 0.00289 0.00324 0.00399 0.00402 0.00343 t Value 9.34 40.20 43.22 45.94 62.31 36.51 43.31 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS LS=80 NODATE; PROC IMPORT DATAFILE = 'f:\Applied Project\Spring 2004.xls' OUT=SPRING04; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE CPS; RUN; QUIT; /* Regression analysis for the spring of 2004, including all tests and averages of the quizzes and focus exercises, as well as the CPS variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 7 181 188 Sum of Squares 35323 30.62864 35353 0.41136 80.21912 0.51280 Mean Square 5046.07615 0.16922 F Value 29819.8 Pr > F <.0001 189 189 174 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9991 0.9991 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS DF 1 1 1 1 1 1 1 1 Parameter Estimate 1.98602 0.15457 0.12843 0.12801 0.24664 0.13836 0.13999 0.94836 Standard Error 0.29751 0.00319 0.00230 0.00278 0.00270 0.00363 0.00324 0.03992 t Value 6.68 48.43 55.96 46.03 91.24 38.15 43.27 23.76 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS LS=80 NODATE; PROC IMPORT DATAFILE = 'f:\Applied Project\Fall 2004.xls' OUT=FALL04; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE CPS; RUN; QUIT; /* Regression analysis for the fall of 2004, including all tests and averages of the quizzes and focus exercises, as well as the CPS variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 7 176 183 Sum of Squares 35025 23.31940 35048 0.36400 82.03591 0.44371 Mean Square 5003.55344 0.13250 F Value 37763.6 Pr > F <.0001 184 184 175 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9993 0.9993 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS DF 1 1 1 1 1 1 1 1 Parameter Estimate 2.02908 0.14050 0.11842 0.14008 0.25027 0.14932 0.13458 1.00999 Standard Error 0.23693 0.00283 0.00197 0.00274 0.00272 0.00326 0.00242 0.03205 t Value 8.56 49.59 60.06 51.06 92.06 45.76 55.52 31.51 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS LS=80 NODATE; PROC IMPORT DATAFILE = 'f:\Applied Project\Spring 2005.xls' OUT=SPRING05; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE CPS; RUN; QUIT; /* Regression analysis for the spring of 2005, including all tests and averages of the quizzes and focus exercises, as well as the CPS variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 7 282 289 Sum of Squares 68668 39.44209 68707 0.37399 80.29587 0.46576 Mean Square 9809.70016 0.13987 F Value 70136.6 Pr > F <.0001 290 290 176 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9994 0.9994 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS DF 1 1 1 1 1 1 1 1 Parameter Estimate 2.01277 0.15224 0.13525 0.12735 0.24632 0.13406 0.13796 1.01915 Standard Error 0.17928 0.00218 0.00176 0.00210 0.00203 0.00253 0.00227 0.02428 t Value 11.23 69.77 76.83 60.69 121.04 53.00 60.81 41.97 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS OPTIONS LS=80 NODATE; PROC IMPORT DATAFILE = 'f:\Applied Project\Fall 2005.xls' OUT=FALL05; PROC REG; MODEL FIN_AVE = MT1 MT2 MT3 FIN QAVE FEAVE CPS; RUN; QUIT; /* Regression analysis for the fall of 2005, including all tests and averages of the quizzes and focus exercises, as well as the CPS variable*/ The REG Procedure Model: MODEL1 Dependent Variable: FIN_AVE Number of Observations Read Number of Observations Used Analysis of Variance Source Model Error Corrected Total DF 7 181 188 Sum of Squares 41722 20.58793 41742 0.33726 79.86519 0.42229 Mean Square 5960.25868 0.11375 F Value 52400.0 Pr > F <.0001 189 189 177 Root MSE Dependent Mean Coeff Var R-Square Adj R-Sq 0.9995 0.9995 Parameter Estimates Variable Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS Label Intercept MT1 MT2 MT3 FIN QAVE FEAVE CPS DF 1 1 1 1 1 1 1 1 Parameter Estimate 1.39139 0.16704 0.12791 0.12324 0.24676 0.13217 0.14149 1.01220 Standard Error 0.22463 0.00248 0.00203 0.00215 0.00217 0.00375 0.00246 0.03207 t Value 6.19 67.46 62.93 57.20 113.96 35.25 57.61 31.56 Pr > |t| <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 <.0001 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 178 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 179 Monday April 01, 2002: CLASSROOM RESPONSE SYSTEM: AN EVALUATION AT AN EASY-ACCESS REGIONAL UNIVERSITY by Michael D. Everett, PhD, Department of Economics, East Tennessee State University, and Richard A. Ranker, EdD, Office of Information Technology, East Tennessee State University INTRODUCTION Classroom Response Systems (CRSs) are a class of instructional tools that elicit student interaction by collecting and displaying student responses to questions. This paper describes one CRS, its equipment requirements, typical use and costs. The interaction provided by this CRS is studied, using students in introductory economics classes at an easy-access regional university. The time required to learn and use the system is reported. Non-CRS- and CRSusing student pre- and post-test performance on a standardized test are compared. CRS student satisfaction data from an eight-question survey are presented. Finally, reflections on the impact the CRS has had on the researcher's teaching are presented. DESCRIPTION Classroom Response Systems are one way of encouraging student interactions. The CRS computer software allows the instructor to write questions and then project them on a large screen to the class. Students signal their answers via a small handheld infrared transmitter. After the instructor calls time and ends the input period, the computer instantly tabulates the answers and gives a read out on the screen of the number of students selecting each choice. The instructor can give immediate correction if enough students missed the right answer. In an alternate application of the CRS, the instructor can print out a series of questions on paper and give them to the students to take at their own pace by clicking an answer for each question. Then the instructor can call up a report which summarizes the class performance in a number of ways. For example, the instructor can project an item analysis, which shows the percent of students who choose each possible answer (a, b, c, d, e) on a question, and give in-class feedback on the questions which large numbers of students missed. CRSs have been in use since at least 1976. The first were permanently-mounted (hard-wired) systems; these were followed by portable systems. For example, the Q-System 1 by Reactive Systems, Inc. (RSi), was introduced in the mid-1980s. These systems of up to 128 response pads could be connected using standard telephone wire and RJ-11 connectors through an interface box to the computer via the RS-232 serial port of an IBM PC or PC Junior. Similarly, the Classtalk by Better Education, Inc., was introduced in 1985 2 Many have seen a CRS in use on the television show, "Who Wants to Be a Millionaire?"; it is the TV-remote sized tool used to pool the audience. There are several manufacturers of CRSs. They include Classroom Performance System (CPS) from eInstruction, Inc., the Personal Response System (PRS) from Better Education, Inc., the Audience Participation System from Reactive Systems, Inc, and the TI-Navigator from Texas Instruments, Inc. The CRS described and used in this research is the CPS. Components: The CPS (Figure 1, below) consists of an infrared transmitter (or pad), a receiver, and software that collects the individual student responses to instructor-posed questions and displays the results. One battery-powered infrared transmitter or pad is normally given to each student. Typically sold in sets of sixteen to thirty-two, these transmitters have eight buttons on them and must generally be pointed in the direction of the ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 180 receiver as the students' responses are sent. Each transmission contains both the student's response and the unique number of the transmitter or pad. The optimal range of a pad is 35 to 40 feet, but can function upwards of 80 feet. Currently the CPS handles up to a maximum of 256 pads. Figure 1: The components of the Classroom Performance System (CPS) showing five transmitters, the receiver (top left) and the carrying bag with software. The receiver converts the infrared signal from each transmitter to a digital signal which it sends, via a normal RS-232 cable, to the computer software. The software collects the digital signals and stores the student response in the location designated for that transmitter number. It collects all responses and displays which transmitter numbers have responded to that question (Figure 2). When the instructor has all responses, he or she can advance to the next question. Results can be displayed numerically, or as a histograph (a horizontal bar graph, vertical bar graph or pie chart.) Student performance data can be exported in a variety of formats including .doc, .rtf, .xls, .pdf or the proprietary grade book file (.gbf). ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 181 Figure 2: Example of a screen generated by CPS. At the bottom of the screen, the grid of numbers indicates student pads that have responded in dark, pads that have not responded in white, and pads not in use in gray (25 - 32). Equipment requirements: The software was made to run on a PC in a MS Windows environment. A Macintosh version was just released in January 2002. The PC system requirements are very low - Windows 95 or better. However, the CPS requires a computer to drive the software and a projector is recommended. Typical use: According to the CPS website at http://www.eInstruction.com/highered.htm, the CPS can be used to "take attendance; give and grade objective pop quizzes; stimulate class discussion with subjective and objective questions using CPS's ad hoc or formal question authoring capabilities built within CPS; give a formal paper-based class test with multiple versions of the test using CPS to grade the exam; use the CPS Gradebook to manage all aspects of your students' grades while leveraging CPS to collect and record them instantly." At http://www.eInstruction.com/learnaboutcps.htm, the uses of the CPS are listed and include: "Streamline your grading. Your CPS results can be easily exported to Excel, Word, PDF, or the CPS Gradebook.  Pass out a paper test and allow students to answer at their own pace. CPS' automated assessment feature lets students answer test questions at their own pace while keeping track of answers and grades behind the scenes.  Use CPS to increase benchmark test scores. CPS Online gives you immediate results school- and district-wide - making CPS an ideal tool for benchmark testing.  Provide a non-threatening environment allowing all students to participate - even the shy ones."  Cost: According to the CPS site at http://www.eInstruction.com/cpspricing.htm, the cost of a complete CPS is "$1,195-$3,995 depending on the size of classroom from 16 to 32 students. Pricing for larger classes, up to 256 members, is also available." However, this does not ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 182 include the cost of the computer required to drive the software or of the (optional but recommended) projector to display the results. Additional equipment purchased at ETSU includes a carrying bag for $30 and an extra receiver for $250. INTERACTION The CPS allows students to respond to multiple quizzes, test and surveys in class. This can increase students' active participation in their learning, particularly in institutions with passive learning cultures where there is typically little student time spent outside of class on homework and course preparation. If the active learning model -- demonstration followed by extensive student practice with corrective feedback (Chickering and Gamson 3 , and Everett and Zinser 4 ) - is correct, then CRS should increase student learning. Through the use of quizzes on homework topics at the beginning of class, frequent feedback surveys at the end of each lecture segment, and discussions based on student response patterns, students should be more actively involved in their learning. This becomes important in easy-access schools with low standards and students who will not spend much time studying on their own outside of class. CPS use should result in measurable learning gains above traditional passive lecture classes. Active learning is most often the result of "interactive teaching" according to Abrahamson 5 . He expands on the options that CRSs provide, and suggest using a variety of approaches. These provide the ability to quickly check on both the student understanding of the content of a presentation and the accuracy of student notes right after a lecture. The above benefits should increase student satisfaction with the course. Structure, clarity and enthusiasm have been the major determinants of student evaluation scores. If the instructional use of the CPS increases clarity, student evaluations should go up. On the other hand, passive lectures and recognition of pre-digested bits of information on objective exams (paraphrased from Richardson, et. al., cited in Everett and Zinser 4) constitutes the educational culture in many easy-access regional schools. To move from this culture to active learning risks creating confusion, frustrations and lower student evaluations in such institutions. An example of the frustration that can result from an interactive teaching approach is provided by Dufresne, et al 6 . A CRS was being used at the Harvard Business School by Professor Elon Kohlberg teaching managerial economics. Kohlberg pointed out that the CRS-charted histogram was evenly split three ways. In the ensuing discussion, he sought different students to defend their particular choices. Then he invited the class to change their answers as he conducted another survey. The resulting histogram was split two ways now. An impatient student asked, "You can see we don't know where we are going with this. Why don't you just tell us what you want 7?" Although Kohlberg was able to quell this student's obvious frustration and engage him in the presentation of an argument that later produced unanimity, it is important to note that not all students find it easy to be interactive learners. A CRS can involve the entire class in discussions of issues, instead of being limited to just a few, if any, willing discussants. For example, the instructor or a student can present an issue, then class members can discuss it in a full class or small group setting, and then everyone can respond. Because of the anonymity in CRSs, even the shy students can interact without cause for concern. This too should increase student satisfaction as metered in end of semester student evaluations of faculty. The type and focus of questions are critical to effective use of CRSs. Abrahamson7 stresses the importance of questioning style in his piece on interactive teaching. His conclusion ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 183 includes the statement that "Good questions asked in the right context have a remarkable property to transfer a classroom." A study of IBM management trainees at the IBM Corporate Education Center 8 using the Q-System CRS from RSi also showed that questioning style was the key. In this study, researchers recognized five types of questions (true/false, multiple choice, mean numeric entry, correct numeric entry and rating scale 1 to N) as well as several interesting variations of those five. After "relevance and quality of the student response questions were enhanced and soliciting student responses more timely and frequent, i.e., at least once every 15 to 20 minutes, " attentiveness and retention were improved. "Thought provoking questions stimulated the students' desire to seek self-discovery and provided them with the opportunity to compare their answers to those of the rest of the class and their peer groups. The subsequent discussions explored issues at greater depths and encouraged participation from a much larger percentage of the students. Classroom time was used more productively and [the materials could] be covered in the same allocated time. The technology assisted the facilitator to complete all the learning points within the allotted time and in some cases the total number of learning points were increased by 20 to 30 per cent." The researchers of this paper sought to determine if the use of the CPS would have a similar effect on student learning, as measured in a standardized test of comprehension, or student satisfaction, as measured in end of course student ratings of faculty and a short survey of satisfaction with the CPS system. PERFORMANCE The researcher administered questions 1 through 30 of the Test of Understanding College Economics, which had a national norming group of about 2,700 students from Ivy League to Junior College. Students in six different classes of microeconomics were administered the same test at the beginning and at the end of the course. Performance data are presented in Table 1. Table 1: Scores of Everett's microeconomics student on the Test of Understanding College Economics. Data were grouped by treatment; the Fall 2000 and Spring 2001 students did not use the CPS system and thus their test scores were grouped together. Similarly, the Summer 2001 and Fall 2001 students did use the CPS and thus their test scores were grouped together. Although the total number of students involved in these treatments were small enough (60 in the non-CPS group and 56 in the CPS group) to make statistical comparisons questionable, some comments are appropriate. First, the averages for both the non-CPS and CPS groups were slightly lower (32.2 and 31.3) than the national average (35.7) on the pretest. Second, the averages for both the non-CPS and CPS groups were slightly higher (52.9 and 53) than the national average (51.2) on the post-test. However, the researcher had used an aggressive active learning approach in the non-CRS courses, using much in-class and homework drill. Third, since the CPS group started slightly lower and ended up slightly higher than the non-CPS group, it earned a slightly higher gain (21.7 percent or 69.2 percent gain) than the non-CPS group (20.7 percent or 69.2 percent gain). Fourth, the gains for each class in the CPS group are very consistent - within 0.4 percent of each other - while the gains ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 184 for the three classes in the non-CPS group vary by 10.6 percent. In summary, the pre-post comparison of the non-CPS and CPS students on the TUCE, while not demonstrative, are encouraging and warrant continued study. This is especially true since the professor prided himself on the fact that prior to use of the CPS, he considered himself to be very interactive in his treatment of the students. STUDENT SATISFACTION SURVEYS There were two separate tests of student satisfaction. The first was the plan to compare end-ofsemester student ratings of faculty; the second was a plan to survey students on their satisfaction with the CPS. Details of both follow. Student Ratings of Faculty: Students have only formally evaluated two of the three CPS classes. Those evaluations were in the same range of average evaluations normally obtained in microeconomics classes by this professor. Because of the low numbers of evaluations completed, no certain generalizations could be presented. However, it appears that the one class found the presentations very clear, a major determinant of relatively high evaluations, and the other class found the presentations unclear. Thus, the ability to use the CPS to quickly survey the students' understanding of basic points and clarify points which students missed after a short lecture, apparently may not have increased the overall clarity. The low number of participants in these evaluations raises serious reliability issue. Perhaps further evaluations (more years) will show a more positive trend. However, the professor's rigorous active learning approach in a passive lecture environment may continue to negatively affect evaluations. Survey of Student Satisfaction with CPS: On the affective (feeling) side, students seemed to like the CRS. On written surveys administered to four separate classes, 60 students indicated a high level of satisfaction with the CPS [Table 2]. The surveys consisted of eight statements listed below. Students were asked to respond to each statement using: a. Strongly Agree, b. Agree, c. Disagree, d. Strongly Disagree and e. No Opinion. Responses were converted to ordinal data by equating a Strongly Agree response to 1, Agree to 2, Disagree to 3, Strongly Disagree to 4 and not including No Opinion responses. Average responses for each of the eight statements are presented in Table 2. For purposes of interpretation, any statement with an average response of: 1.0 to 1.5 was considered to be in the "Strongly Agree" range;  1.51 to 2.5 was considered to be the "Agree" range;  2.51 to 3.5 was considered to be in the "Disagree" range; and  3.51 to 4.0 was considered to be in the "Strongly Disagree" range. The eight survey statements on the CPS (called the Student Class Response System or SCRS in the survey), the average numerical response each was given by the students, and their interpretation were: 1. The SCRS helped me understand the lectures better by catching and correcting my mistakes. - 1.53 - Agree 2. Using the SCRS after a short lecture took time that could have been more productively devoted to lecture. - 3.20 - Disagree 3. We really needed to write the basic concepts out first before trying to answer objective questions on them. - 1.90 - Agree 4. The SCRS helped increase learning outputs by involving students in more active learning. 1.53 - Agree 5. Learning takes place best when instructors clearly provide information to students rather than requiring students to write, think about, and answer questions on information given them. - 2.99 - Disagree 6. I enjoyed the increased class involvement the SCRS provided me. - 1.68 - Agree 7. The SCRS system increased my level of anxiety and tension since I had to participate  ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 185 whether I wanted to or not. - 3.20 - Disagree 8. I do not like active learning and thus do not like the SCRS. - 3.51 - Strongly Disagree Table 2: Results of survey of 60 students enrolled in microeconomics classes in which the CPS was used. In summary, the students either disagreed or strongly disagreed with all statements that denigrated active learning and/or the use of the CPS. The students agreed with all statements that spoke well of the CPS (SCRS) use. The CPS was a big hit with the students. TIME DEMANDS In addition to the modest monetary costs, the researcher invested substantial but manageable time costs. Most of these costs involved time which could have been devoted to other teaching, research, or service activities. For example, about 50 hours were spent in first-time set-up costs learning the program (mainly on his own), writing the questions before each class, and arranging for assignment of classes to multimedia classrooms (see http://ats.etsu.edu/mc/) to assure availability of the equipment needed to use the CPS. Now that the initial problems associated with initial set-up have been resolved, the researcher or the Office of Information Technology staff could probably teach the program to other faculty in a two- to three-hour workshop. Most of the other set-up time would also be significantly diminished. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 186 Table 3: Researcher's estimate of time spent on various tasks associated with CPS use. Note that First Time Overall Set-Up Costs and Overall Initial Class Set-Up Time are reported in hours; Per class Additional Time Costs are reported in minutes. Some time costs associated with CPS use will not be diminished. The researcher estimates between 5 and 25 minutes additional time per class in setting up and using the system for that class; the total could reach up to 40 minutes, if questions need modified. Table 3 describes the tasks associated with CPS use, provides rough estimates of the minimum and maximum time the researcher spends completing those tasks, and provides a format for more systematically collecting time and motion type cost data. Time demands, while formidable in the first-time set up period, are much more manageable for the next faculty users. DISCUSSION OF IMPACT ON TEACHING This study has attempted to identify and measure a wide range of benefits and costs associated with using the CPS in introductory economics classes at an easy access regional university typical of many regional universities, colleges, and junior colleges. Time costs, such as learning the program, preparing questions, accessing technical equipment, and daily loading and unloading class data can be substantial. However, the next faculty to use this system will face a much easier path since the initial training package has been developed and the technology has been tested and integrated into the technology classrooms on the campus. The remaining time costs may be counterbalanced by other benefits, such as student interest and attention and a feeling of connecting with a normally passive class. The following anecdotal evidence comes from the researcher's experience teaching three, and half way through two more, introductory economics classes with the Classroom Performance System. Table 1 shows the class scores on national standardized exams remained about the same as other recent classes the researcher has taught: relatively low pre test scores and relative high post test scores and gains, compared to national averages. However, it does not reflect the observation that these classes seemed to cover the basics more quickly and thus had more time for broader discussions. In previous (non-CPS) classes, the researcher would give short lectures and then assign in-class exercises and practice quizzes. Time would be ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 187 spent walking around the classroom checking students progress and/or have other students provide feedback on each other's work. This took more time than the CPS and provided much less reliable information on problem areas experienced by the students. On the other hand, the CPS, with its objective question formats, emphasizes recognition of the "right" answer rather than working through the graphical analysis. This points to the need to improve the question drafting process to use more graphically based questions. The CPS seemed to leave more time for broad analyses of articles and cases in the textbook. It improved these discussions by letting all students vote on policy issues which had been debated in class. This gave the researcher a feeling of connection with classes of students who traditionally are very passive, uninvolved, and often bored with virtually any kind of academic material These studies have not yet yielded data on the CPS's ability to motivate real homework study as opposed to just going through the motions. In the initial design, the CPS was to be used to quiz students on the homework concepts, thereby motivating students to complete all assignments prior to coming to class and to actually learn the principles through practice. However, transferring the data from the CPS Gradebook program to the class' official grade spreadsheet remains time consuming (at least 5 minutes per assignment) and did not get done. An update of the current version of the CPS software may result in a more efficient grade book. Also, commercial Internet testing services (e.g., http://www.Gradesummit.com) may perform this function more efficiently by allowing instructors to assign an exam over topics in a textbook to test students' knowledge of that topic. Again, the adoption of this technology will depend on the ease-of-use of its record keeping formats. These researchers believe the CPS and other CRSs can efficiently improve active learning and learning outputs in easy access schools where most students spend very little time studying outside of class. Surveys indicate students like the CPS. Pre-post standardized testing hints at the possibility of increased learning due to the CPS use, but more study is required. Perhaps the lack of significant comparative evidence of gains on the TUCE may stem mainly from the use of a very rigorous active learning approach before the CPS was introduced. Class sizes of 20 to 30 students allowed this. The apparent lack of impact of the CPS on student evaluations may be caused by the frustration students feel when their overall passive lecture system - which most students have experienced and seem to generally prefer - is disturbed by the very active, mastery learning approach. However, two sets of student evaluations are too few to draw a valid conclusion. There is a shortage of hard evidence that CPS use increases learning outputs or student evaluations in our introductory economics classes, even though the evidence is positive and hopeful. The major intuitive benefit for the researchers is a feeling of connecting with classes of traditionally very passive students. This comes both from feedback on basic lecture material and class discussions on broader issues which the CPS system facilitates. CONCLUSION In conclusion, the theoretical and intuitive benefits seem to outweigh the preparation and set up costs. CPS use appears to increase active learning by devoting some classroom time to student involvement in active feedback such as quizzes on lectures. The theoretical and empirical literature on active learning, cited above, seems compelling. These observations hold particularly true in easy-access regional schools where students do very little work outside of class. On the other hand, such schools tend to emphasize short-term student satisfaction rather than actual learning. This constitutes a major barrier to the use of active ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS learning technologies like the CPS. 188 Nonetheless, the pre-post TUCE results did appear to favor the CPS group, the end of course faculty evaluation data were mixed and too few to allow generalization, the student endorsement of CPS use was strong, and the time studies indicate that the time commitment for other faculty to use the CPS will have been significantly diminished. The major intuitive benefit for the researchers is a feeling of connecting with classes of traditionally very passive students and the time it makes available for other higher-order thinking. Further study seems to be in order. The possibility of further study will be enhanced by the recent decision to place the CPS in the remaining multimedia classrooms at ETSU and make it part of the standard package for new classroom installations. REFERENCES 1 Q System Class Product advertising sheet, Reactive Systems, Inc., Englewood, NJ 2 Better Education brief history, http://www.bedu.com/ 3 Arther Chickering and A. F. Gamson, "Principles of Good Practice in Undergraduate Education," The Wingspread Journal, special section, June 1987 and reprinted. 4 Michael D. Everett and Otto Zinser, "Interdisciplinary Social Science Courses: Using a Critical Thinking Approach," Journal of General Education, Vol. 4, No. 3. 1998 5 A. Louis Abrahamson, "What IS Interactive Teaching?", http://www.bedu.com/interactive.html 6 Dufresne, R.J., Gerace, W.J., Leonard, W.J., Mastre, J.P., and Wenk, L., "Classtalk: A Classroom communication System for Active Learning," Journal of Computing in Higher Education, 7, 3-47, 1996 A. Louis Abrahamson, "An overview of Teaching and Learning Research with Classroom Communicatins Systems (CCSs)," paper presented at the International Conference of the Teaching of Mathematics, Village of Pythagorion, Samos, Greece, Conference Proceedings by John Wiley & Sons, Inc., June 3-6, 1998; also available at http://www.bedu.com/Publications/Samos.html 8 7 Harold M. Horowitz, "Student Response Systems: Interactivity in a Classroom Environment" Xerox Palo Alto Research Center, 1984; also available at http://www.rsicommunications.com/keypads/study.htm; later updated at http://www.meetingnet.com/users/classroom_interactivity.pdf About eInstruction Headquartered in Denton, Texas, eInstruction Corporation is a leader in the educational technology industry. With the introduction of the Classroom Performance System (CPS) in 2000, eInstruction is now the unquestioned educational leader in real-time, interactive wireless response pad technology with over 2 million response pads now being used in all 50 states in thousands of k-12 schools as well as over 600 universities and 10 foreign countries. eInstruction was founded in 1981, by Dr. Darrell L. Ward, a long-time researcher and teacher. He recognized a significant need in educational institutions for innovative ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS technology-based products. Many years of consistent cuttingedge research and development have earned eInstruction a reputation of excellence in the education market. eInstruction Corporation is committed to providing innovative products that enhance the learning process in corporations and educational institutions through the use of computer-based technology, software, and the Internet. 189 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 190 IBM Study Proves Use of Student Response Systems Increases Attentiveness Wednesday August 01, 2001: The attached was presented February 24, 1988 by Harold M. Horowitz, Ph.D., Program Director of Educational Technology, IBM Corporation at the Sixth Conference of Interactive Instruction Delivery for the Society of Applied Learning Technology (SALT). This Paper describes the Advanced Technology Classroom at the IBM Corporation Management Development Center, and the application of interactive student response units along with the educational developments derived. Student Response Systems: Interactivity in Harold M. Program Director IBM Corporate Thornwood, New York 10594 ABSTRACT This Paper describes the research and development activities in educational technology which preceded the installation of the new Advanced Technology Classrooms at the IBM Corporate Management Development Center. Specifically, the application of interactive student response units is presented along with the educational benefits derived. The impact of this interactive capability on the design and development of courses is described in terms of goals, question categories and potential for improving instructional design methods. Finally, traditional and interactive classroom environments are compared based on students’ reactions and retention scores. Background During the past three years, the Corporate Management Development Center (MDC) Located at IBM Corporate Headquarters in Armonk, New York has served as a test-bed in experimenting with new concepts for brining technology into the industrial classroom. The culmination of these efforts resulted in the development of the Advance Technology Classrooms which are being used by IBM today for management training at corporate headquarters. This paper focuses on a major component of this new classroom - the student response system. This system enables each student to participate by responding to questions during the learning process. This interactive process was designed to increase the students’ attentiveness, aid in individual knowledge discovery and increase retention of key learning points. Training Environments IBM’s training policy and practice requires for all new managers to attend a one week course at corporate headquarters within three months after appointment. The topics taught include IBM’s history and values, personnel practices, leadership, communication, performance planning, reviews, counseling, appraisals, compensation, delegation and employee relations. The students come to the class with varied experiences and represent virtually every function, occupation and IBM location in the United States. The primary objective of this training is to provide these new managers with insight on how to manage employees within the spirit of IBM’s beliefs and value system. This training objective is accomplished by using lectures, case studies and discussions which are taught by instructors who have been identified as future "high-potential" managers. These appointments to the school are made by each executive areas and consist of a two year chair assignment. IBM has always placed special emphasis on Management Development and the requirement for continual management training on an annual basis has contributed to the conclusion found in numerous surveys which report that IBM is among the best managed companies in the world. This high standard for education and quality of instruction at the Management Development Center created a difficult environment to initiate research and experimentation searching for innovative ways to improve the process. Indeed, as this effort began, doubts were raised about the importance of studying the existing MDC classroom environment which, in the minds of many both inside and outside the company, already a classroom Horowitz, Educational Education environment Ph.D. Technology Corporation Center ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS represented the leading edge in industrial education. Traditional Classroom Observations During a six month time period from January through June 1984, detailed observations were made of management training for newly appointed managers. A typical class consisted of one-hundred students divided into five color groups of twenty students each. Each color group was assigned a separate classroom for interactive training and the entire class met in a large "main tent" on occasion for special presentations which warranted little or no interaction. The findings reported in this paper were based on data collected in "live classroom" courses by observation without any intervention in the learning process. The following four questions guided the data collection process: How was class time spent by instructor and students? What was students’ attentiveness and apparent interest? What was the nature of students’ questions? How effective was the use of audio/visual aids? In order to decrease unexplainable variability of the findings, one specific course was selected which was taught every week by multiple instructors. This was a six hour course hundred key learning points which were covered in approximately 300 minutes of productive class time excluding breaks. This course was well designed and considered among the very best taught at the school. It consisted of about 100 key learning points. A key learning point was defined as a specific interpretation of policy and/or practice which the student was expected to know and remember when back on the job. The course material was presented using visual aids including approximately seventy transparencies (foils), numerous flip charts and handouts. A laboratory exercise was also included to enable the students to apply and practice the learned material and a video tape was used for presenting a case study to demonstrate the process using role models. The bottom line of the observations of this our course was – the instruction was effective on the whole and did achieve the objectives as stated in the syllabus. But basic questions about possible improvements in productivity and effectiveness still needed answers including whether technology could enhance the process. The following five generic observations were cited as having potential for improvements to this traditional classroom environment: Observation 1: Visual Aids For the most part, visuals were busy, difficult to read and predominately consisted of bullets of associated word phases. These "word charts" received only periodic glances by students and did not appear to capture and hold their attention. However, when clear, conceptual visuals, diagrams and model representations were used, the students appeared to be more interested in the visuals and more attentive to the accompanying explanations by the instructor. Observation 2: Logistics At some point in each class observed, the instructor experienced difficulties with the logistics of using multimedia, adjusting lighting levels and locating transparencies properly on the projector. These logistical difficulties resulted in nonproductive time in the classroom and while these situations were normally handled in a humorous vein, they were frequently disruptive to the continuity of the learning process. Observation 3: Time Management Allocated course time did not always permit all the key learning points to be covered if the instructor did encourage students’ questions and comments as part of the learning process. Ratio of instructors’ presentation/lecture time to students’ question and answer time varied considerably among instructors. Observation 4: Student Interaction Participation was not evenly distributed among students. In a typical class, between 10 and 20 percent of the students dominated the discussion, i.e., these vocal students asked the most questions, offered most of the unsolicited comments and were more likely to volunteer to answer the questions posed by the instructor. The remaining 80 to 90 percent of the students contributed only occasionally to the discussion unless specifically asked to do so by the instructor. Observation 5: Attentiveness Students’ apparent interest and attentiveness while course material was presented tended to decrease during pure lectures which did not encourage student participation and increased as the instructor served more as a facilitator/enabler who encouraged students towards interaction and participation. Time Allocation 191 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS How time was spent in the classroom was very much a function of the teaching instructors were observed teaching the same course – two who favored facilitation mode. While every instructor used both styles at times, there appeared to be a dominate style which was used more frequently. Three major categories of classroom activities which were derived from these styles were identified: Lecture (Tell) The prepared verbal and visual presentation communicated by the instructor which included the theory behind the key learning points and associated explanations. Dialogue (Share) The unplanned or unstructured discussion by the instructor and the students which covered related material and was designed to amplify the understanding of the key learning points and interpret their application on the job. Question and Answer (Query) The specific questions and answers generated by both the instructor and students on key learning points and other related and non-related points. This Q & A normally expanded the application of the lecture and dialogue to specific situations, resolved confusion and provided some indicators to the instructor about the level of understanding by the students. Time Management vs. Attentiveness Lecture, Dialogue and Q & A activities were observed and a comparison was made between different instructor styles. A dichotomy was uncovered between efficient time allocation and student attentiveness. In summary, lecture style used time more effectively in terms of covering required material and learning points in the allotted time. Facilitation style tended to require more time but appeared to sustain the students’ interest and attentiveness. Time Management When an instructor primary used a lecture style without encouraging questions and comments by the students, virtually all of the learning points were covered in the allotted time. However, the facilitator had more difficulty in completing all required material in the allotted time. In a typical class, the facilitator required 10 to 15 percent more time to cover the same number of learning points. Figure 1 represents a composite example of time allocation for the instructors who primarily used a lecture style. Approximately half of the available class time was used lecturing and the remainder was used for dialogue and Q & A. Approximately ¼ of the time was spent with Q & A but for every three questions, the instructor originated two and the students just one. 192 On the other hand, the facilitator spent less time lecturing and used about 43 percent of the time for Q & A. The distribution of origination of questions between the instructor and students was about even as indicated in composite example shown in Figure 2. Attentiveness While determining whether or not a student is exhibiting attentive behavior is subjective and judgmental, an attempt was made to compare the classroom environments based on instructor’s teaching styles. Body language signals were categorized using head, eye, hand, leg and sitting positions to depict possible attentive or non-attentive behavior. (1) & (2) Observations were made in a class of twenty students in 5 minute intervals and an instant determination was made for each student. While this determination has substantial potential for erroneous conclusions for any one student, consistent criteria was used for each student and the composite findings showed an interesting contrast between teaching styles. The observations summarized in Figure 3 compared lecture and facilitation styles in terms of resulting student attentiveness. At the beginning of each class, most students exhibited attentive behavior which diminished rapidly within 20 minutes – after which the average number of students who exhibited apparent attentiveness stabilized. However, each student exhibited both attentive and non-attentive behavior at different observations and the recording of individual student’s fluctuating attentiveness was not performed as part of this study. ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS In order to compare teaching styles, an index was established to represent a composite of the area under the attentiveness curve. An index of 100 indicates attentiveness of every student at every observation point. In the lecture style, this index was 47 or just under half of the class. This index of attentive behavior increased to 68 for the class taught with facilitation style. Short – Term Retention The students were tested anonymously to determine their understanding of the key learning points which were taught using both teaching styles. The results of testing showed that while facilitation required additional class time for the same number of learning points, the average short-term retention (3 days) was about 19 percent higher using facilitation when only considering those learning points which time permitted to be covered adequately. This finding corresponds to the 22 percent increase of average scores reported by Paul Macali in 1981 as the result of using Socratic questioning methods. Technology in the Classroom The initial observations were analyzed further in different training environments. The conclusion reached was that technology could play a vital role in focusing attention, supporting interactivity, improving logistics and fostering the facilitation style of an instructor. An experimental classroom was completed during the summer of 1985 which incorporated promising technologies to address the problem areas found during the investigation. Observation 1 (Visual Aids) was addressed by replacing foil transparencies and slides with computer-generated graphics which focused on conceptual visualization rather than "word charts" . Observation 2 (Logistics) was resolved by incorporating an IBM PC/AT coupled with a plasma panel display podium into the classroom. The keyboard was eliminated and replaced by a podium. The IBM PC manages all the classroom logistics for the instructor including graphics, audio/video, lighting levels and automatic switching to connect any required component to the video projection system. The resulting "Advanced Technology Classroom" has been successfully used for management training and is currently under evaluation in other diverse educational functions within IBM. Further descriptions of this new classroom concept can be found in papers by Levine, Garwin and Shappert and Vadas. The primary purpose of this paper is to report on one major aspect of technology brought into the classroom, namely the student response system which in conjunction with conceptual visuals and quality instructional design of courses addressed the opportunities described in Observations 3,4 and 5. Interactive Classroom Concept The design concept of the Advanced Technology Classroom was guided by the premise that the learning process could be improved if the visuals and the instructional design of courses would expand students’ interest and self-discovery through a high level of interactivity. Strong support for this basic concept for effective teaching is found in the Paideia Proposal which calls for radical reform of basic schooling in the United States. The interactivity aspect of this design concept has its roots in Socratic teaching principles which encourages questioning and active participation as the keys to more productive and enlarged learning. The research and implementation by the Amherest H. Wilder Foundation in conjunction with the St. Paul School District in using a key board for each student to support group instruction in a classroom setting provides further evidence for merits of this interactive concept. Student response systems Could higher levels of student learning and retention be achieved by providing each student with a device to facilitate the Q & A process and would increase the students’ involvement and interaction in the classroom environment? To answer this question a student response system was incorporated in the classroom to enable each student to respond to questions during the learning process and to become a more active participant in the process. Goals of Student Response The concept of incorporating student response units (or keypads) for use by each student in the classroom was conceived and guided by five goals: 1) Student Activity Stimulate the active processing of data information, ideas, viewpoints and beliefs at the same time as the learning is taking place. The opportunity for participation and contribution should be available equally to all students. 193 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 2) Communication Create an environment in the classroom where differences in answers and opinions as a group can be observed and discussed immediately upon tabulation while keeping each student’s specific response anonymous. 3) Learning Desire and Commitment Provide students with frequent indicators of both individual and class learning progress which include comparisons with peer groups, previous classes and demographic subgroups – to encourage positive effects of self-assessment and competition among students. 4) Customized Instruction Provide the means for both preplanned questioning and ad-hoc questioning including the opportunity for students themselves to initiate the solicitation of class responses. 5) Data Collection Capture data on student responses divided into demographic categories to facilitate course revisions, to provide input to students on demographic positions; and to provide information for personnel research into critical topic areas. Hardware The experimentation with alternative student response systems was conducted using products manufactured and distributed by Reactive Systems, Inc. The Student Response Unit (keypad) which was selected for implementation in the classroom was a device comprised of ten keys for data entry and five function keys as illustrated in Figure 4. An LCD readout screen for eight numeric digits is included to enable the student to receive confirmation of the data input. The function keys provide capability to: 1) clear the screen, 2) answer more than one question at a time, 3) recall last answer, 4) ask for help and 5) send the student input to the system. up to 128 response units are cabled to an interface box using telephone-type wire. The interface box is connected to an RS 232 asynchronous serial communications port on a PC. Software Software which is resident in the PC polls, the response units, tabulates results and presents graphics simultaneously to the students and instructor. The software provided by Reactive Systems which polls, tabulates and creates graphical results is called Instant Feedback. The instructor activates keypads and tabulates results by pressing appropriate function keys on a PC. There are four primary modes in this program: Instructor, Quiz, Reports and Roster. The Instructor mode enables the presenter to activate the keypads, poll the audience and display results as bar graphs. The Quiz mode is used for answering multiple (batched) questions. Reports are printed which summarize all responses by question for both individuals and the entire class. These reports include student names (if input in the Roster menu mode). While this program supplied by Reactive Systems was effective in our early experiments, the complex requirements for student response in the Advanced Technology Classroom necessitated the development of a special purpose program which could run under the control of a Command Processor in the PC. This program activates and polls keypads from a remote control device, accounts for the different types of questions asked, includes paraphrasing on the resulting graphs, stores and retrieves demographic data for comparisons, and creates specially designed output displays on the presentation screen in the classroom. Categories of Questions Experience with using student response capabilities in the classroom has yielded an array of categories and types of questions which can be asked of the students for keypad input during the learning process. Listed below are examples of the categories and types of questions used to elicit input from the students during a course: Yes/No or True/False This is the simplest of all question types which asks for a positive or negative response to a statement, situation or condition. Typically, this type of question elicits opinions, viewpoints and experiences with respect to key issues. Also an answer to a stated question can be presented for the student to either agree or disagree with the accuracy of the answer. Yes/No type responses can either be asked one at a time or in pairs where two questions are related. In the latter case, both graphical results are shown on the same screen for comparison. Multiple Choice This question consists of a stem followed by alternative response choices. Typically , the stem consists of a question or an incomplete sentence and the alternative answers range from three to ten. The student selects the best answer or the best ending from the alternatives or "None of the 194 ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS above" is used if all other alternatives are incorrect. Mean Numeric Entry This type of question requires the student to answer with a specific numeric value. Usually, there is no correct answer in this category but rather a range of answers which are of interest to the class along with a computed mean. The output display presents the mean, upper and lower limit as well as the historical mean from previous classes. An example question would be "How long have you been with the company?" Correct Numeric Entry This type of numeric response is targeted towards eliciting a specific correct answer to a stated problem. Each student’s answer is compared against the predetermined correct answer and the percent of the students who obtained the correct answer is shown on the output display along with historical data of previous correct answers in other classes. This type of question requires more time for response than any other because some thought is required before entering answer. An example of a Correct Numeric Entry type question would be "what is the mean of the following set of numbers: (10, 22, 38, 5, 25). If 15 out of 20 students specified the correct answer, the output display would provide the correct answer (which is 20) and would indicate that 75% of the class provided the correct answer. Rating Scale 1 to N This category allows the student to express his or her feelings or opinions about a particular situation or topic. A statement asks for a rating from 1 to n (where n usually ranges from 5 to 10). Each alternative numeric response is assigned a condition such as very poor, poor, average, good, very good in a graduated scale. The student selects the rating which best represents his or her reaction and the result is shown as a vertical bar chart containing the collective responses. A variation of this category is the "Consensus/No Consensus" which takes the distribution of responses and computes the mean and standard deviation. Consensus is assumed if the standard deviation is less than a predetermined value (such as 1.5 or 2). An example question would be "On a scale of 1 to 5 (where 1 is poor, 3 is average and 5 is outstanding) how would you rate the manager’s handling of this personnel situation?" If mean was 4 and the standard deviation was within the tolerance previously established, the output would indicate a consensus among the class respondents. Group Response When group activities are determined to be an effective method to stimulate learning of a particular topic, the Group Response variation is very effective. In this approach, students are assigned to one of two, three or four groups and the tabulation of results compares the collective summary totals from each group on the output display. Group Response Sequences further stimulate interest by promoting healthy competition among groups. The "Game" environment creates a peer pressure to participate and the desire to win encourages higher levels of attentiveness in order to provide correct answers and contribute to the success of the Group. Group Response is used in one of two approaches- On-line Class & Offline Breakout. On-line Class variation can be used in conjunction with any of the previously described keypad sequences. At the beginning of a class, the instructor establishes groups either arbitrarily or for a specific purpose. Examples of group segmentation are gender, birth location, occupation, division, time in the business, etc. Once this determination is made, each student enters his or her affiliation with a specific group via the keypad. Whenever responses are tabulated and displayed, group results are shown in addition to the total class results. The rankings of the groups in terms of score achievement are determined by the system at the end of the course or at key milestones. Off-line Breakout variation is used when groups meet separately to discuss issues and to derive a consensus which is reported back to the class. The group responses are tabulated and reported as each spokesperson enters the group’s consensus. However, all students are encouraged to remain active participants by providing their individual answers which may be different from the consensus opinion. The extent of minority opinions can be observed in the results and the resulting discussion includes all members of the entire class rather than predominately the spokesperson for each group. The rankings of the groups plus the individual scores are tabulated and both group winners and individual winners are determined by the system. 195 RESULTS Pilot Classroom The installation of an initial student response system in a pilot classroom did stimulate student interest beyond just a "Hawthorne Effect" but it did not make the dramatic differences in the classroom environment as was predicted. An analysis revealed that this was due to the fact that the instructors were unfamiliar with the system, the system was used too infrequently and the questions were almost all "multiple choice" and the students would find the correct answer about 90 percent of the time. Another finding was that when there were more than four alternative choices, too much time was spent by the ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS students in the selection process. Prototype Classroom Based on the experienced gained in the pilot classroom, a prototype Advanced Technology Classroom was built. The graphical presentation and interactivity processes were improved and the results were very positive. The relevance and quality of the student response questions were enhanced and soliciting student responses more timely and frequent, i.e., at least once every 15 to 20 minutes. In addition, keypad questions were inter-weaved into the overall instructional design of the course and a variety of student response categories were used beyond just multiple choice. Thought provoking questions stimulated the students’ desire to seek self-discovery and provided them with the opportunity to compare their answers to those of the rest of the class and their peer groups. The subsequent discussions explored issues at greater depths and encouraged participation from a much larger percentage of the students. Classroom time was used more productively and substantially more learning points could be covered in the same allocated time. The technology assisted the facilitator to complete all the learning points within the allotted time and in some cases the total number of learning points were increased by 20 to 30 percent. Student Attentiveness and Retention When the same criteria to measure students’ body language was applied to classroom with student response units, the index of apparent attentiveness was found to be 83 as shown in Figure 5. In this classroom environment, the facilitator used the keypads to solicit student responses six times during the 90 minutes of instruction and the interactive process tended to peak students’ interest and attentiveness which would then decrease somewhat until the next opportunity for the students to respond. The test scores were higher in this environment – from the 19 percent improvement reported for the facilitation style to 27 percent when this style was coupled with the student response system. 196 Student Reactions The students were asked to express their reaction to the interaction and feedback provided by the student response system. A scale of 1 to 7 was established to compare the conventional Q & A approach to the student response unit approach. A value of "4" was designated to represent an equal attitude and feeling between the two classroom approaches. A rating of 1 would be a strong vote for conventional Q & A while a rating of 7 would be a strong vote for the new student response system. Ratings in between these values would be graduated towards those indications but to a lesser degree. The results were a 6.6 out of 7 in favor of student response systems. Conclusions Based on the experimentation and findings described in this paper, interactive classrooms which use student response capabilities have been shown to improve the learning process and this concept should be explored further as we look for technology’s role in the "classroom of the future" for both industrial and public education. However, much additional research is required beyond the limited studies presented here. Within IBM, the Advanced Technology Classroom concept is being currently expanded into other learning environments beyond management development to determine its suitability and needs for enhancements in hardware and courseware to address new educational requirements. Most seem to agree that education is the key to the future of our society. Unfortunately, the classroom suffers from technical neglect and a lack of creativity which would enhance the instructor teaching capabilities. While technology has provided our society with vast improvements in quality of life and productivity during this century, the classroom has not yet been a prime benefactor of technical innovation and ingenuity. most of today’s educational research is focused on interactive video and related self-learning concepts but the classroom requires some special focus and attention since it will likely remain our primary educational delivery system for many future generations. This paper suggests that computer supported interactive video and related self-learning concepts but the classroom requires some special focus and attention since it will likely remain our primary educational delivery system for many future generations. This paper suggests that computer supported interactive classrooms could enhance learning by supplying the teacher with relatively inexpensive technology. About The Author Dr. Harold Horowitz is the Program Director of Educational Technology with International Business ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS Machines at Corporate Headquarters. He is currently responsible for research and development in instructional applications for the Advanced Technology Classroom, satellite remote training and individualized learning. Prior to this assignment, he was the principal architect in the design and development of the Advanced Technology Classroom. Dr. Horowitz has spent over 31 years with IBM and served as an engineer, systems analyst, operations research analyst, project manager, program director and educator. He holds a Bachelors of Electrical Engineering degree, a Masters of Business Administration in Operations Research and a Ph.d. in Educational Research. He has served as a lecturer at the University of Maryland and as an Adjunct Professor in the graduate program at the University of Connecticut. References McConnell, Charles R., Learn To Read Nonverbal Trainee Messages. Training. May 1978. Pease, Alan, Signals: How To Use Body Language. New York: Bantam Books. 1984. Micali, Paul J. The Power of the Questioning Approach, Training. March 1981. Levine, James L., Garwin Richard L. and Schappert, Michael S., An Electonic Podium fo rthe Classroom. 1987 International Symposium Digest of Technical Papers. Vidas, Judith E., Interacive Videodisc for Management Training in a Classroom Environment. Eighth Annual Conference on Interactive Videodisc in Education and Training, August 20-22, 1986. Adler, Mortimer J., The Paideia Program: An Educational Syllabus. New York: MacMillan Publishing Company, 1984. Robinson, Steven L., Technology and Group Instruction: A Communication and Management Tool for Teachers. 29th International Conference of the Association for the Development of Computer-Based Instructional Systems. November 1987. 197 About eInstruction Headquartered in Denton, Texas, eInstruction Corporation is a leader in the educational technology industry. With the introduction of the Classroom Performance System (CPS) in 2000, eInstruction is now the unquestioned educational leader in real-time, interactive wireless response pad technology with over 2 million response pads now being used in all 50 states in thousands of k-12 schools as well as over 600 universities and 10 foreign countries. eInstruction was founded in 1981, by Dr. Darrell L. Ward, a long-time researcher and teacher. He recognized a significant need in educational institutions for innovative technology-based products. Many years of consistent cuttingedge research and development have earned eInstruction a reputation of excellence in the education market. eInstruction Corporation is committed to providing innovative products that enhance the learning process in corporations and educational institutions through the use of computer-based technology, software, and the Internet. Home | What is CPS? | About Us | Products | Solutions | User Area ©2007 Educational Underwriters, Incorporated. All rights Reserved EDU, CPS 198 Report on the Classroom Performance System for use in the Navy Junior Reserve Officer Training Program By CDR David A. Lewellyn USN (ret), M. Ed. 21 July 2006 ©2007 Educational Underwriters, Incorporated. All rights Reserved 1. Introduction: Most educators will agree that due in part to a change in cultural norms, high school students learn differently today than in the past. It is naïve of us to think that we will be able to change the way that these high school students learn, so we must explore different ways to effectively teach them. A. Why does the NJROTC need a polling system? There are three principal issues that NJROTC must address for our curriculum to remain relevant for future cadets. First, we must move away from our lecture-and-listen teaching style. Today’s educational research is full of discussions on the advantages of ―student-centered learning‖ models. We have known for some time that lecture is not the best way to learn. The learning pyramid (enclosure (1)), which has been part of teacher education for decades, rates lecture as achieving only a 5% retention, while use of audio-visual, demonstrations and discussion groups achieve retention of 20%, 30% and 50% respectively. Second, we need classrooms that reflect today’s technology and engage students in classroom discussions. We need a class model that invites students to participate and is the envy of the non-cadets in the school. This is one way to help our program recruiting efforts. Lastly, NJROTC instructors are provided only basic training in how to teach at the high school level and tend to revert to the method that they experienced in the military or in their own educational background, which is most likely lecture-and-listen. Our instructors need a model that will provide a template for successful high school teaching in a student-centered environment. B. Why look at CPS? A large part of the overall answer may lie with the Classroom Performance System (CPS). Conclusions reached in a 2003 study of CPS at the Trident Training Facility (TTF) suggest that CPS would address the needs of our program listed above. Additionally, CPS has been successfully used at the United States Naval Academy, United States Military Academy, public schools throughout the country and, most importantly, since they are also teaching at the high school level, with Army JROTC. Therefore it made sense to look into how CPS may complement the NJROTC curriculum. A 2004 Department of the Navy Application Analysis conducted by the ©2007 Educational Underwriters, Incorporated. All rights Reserved Functional Area Manager (FAM) for Training and Education on portable/remote Polling Applications reviewed five of the most popular systems on the market and concluded: ―Based on the percentage of functional requirements and lack of substantial price differences, the FAM has determined that CLASSROOM PERFORMANCE SYSTEM (CPS) is the preferred application for the T&E Polling Application group….CPS is the preferred Navy application and must be used for all new acquisitions.‖ Based on the current uses of CPS and the results of the FAM study CPS was a natural choice for helping NJROTC improve its curriculum model and meet its requirements. 2. CPS Description: The Classroom Performance System (CPS) by eInstruction is an educational tool designed to enhance interaction between students and teachers, and encourage students to participate in their own learning. It consists of an infrared remote control device called a response pad, CPS software, and an infrared receiver. The teacher uses his/her own questions, either in writing or via audio-visual inputs, projected on a large screen display and cadets respond anonymously via their assigned response pad. The computer instantaneously tabulates the percentage of correct/incorrect responses and frequency distribution. Instructors can use this information to monitor class understanding of the material. It promotes questioning and checks cadets’ understanding of the material. ―Checking for understanding‖ is one of the most fundamental of teaching processes but one of the most often overlooked, and is a key to achieving factors that promote learning. The response information is stored so the teacher can review individual responses at a later time to see who may be having difficulty with the material and adjust teaching techniques accordingly. Cadets can also take their tests using CPS. CPS will grade the tests and input the grades directly into an electronic grade book provided with the software. This has the potential to save instructors hours of time normally spent grading and recording grades for cadet quizzes and tests. The CPS software can turn the results of the test into 20+ different ©2007 Educational Underwriters, Incorporated. All rights Reserved reports appropriate for parents, evaluation of test item validity, and assessment of class performance as a whole. There is significant potential for time savings once instructors become adept at using the system. To determine the applicability of CPS to the NJROTC classroom the NJROTC program office conducted a small scale study using NJROTC instructors and cadets. Leveraging on the detailed study already completed by the TTF, we examined whether gains in test scores, student participation and teaching methodology would easily translate to the NJROTC environment. 3. NJROTC Evaluation Plan Summary: To assess the applicability of the CPS system in the NJROTC classroom, a study was conducted with 13 NJROTC units. At no cost to the Navy eInstruction, supplied a 32 response pad CPS system to each participating unit. For the purposes of the evaluation, each participating unit was to teach a meteorology lesson from Naval Science 2 (Junior Year) using a prepackaged CPS style course of instruction that included CPS in-class questions, 4 quizzes and 1 end-of-unit exam. The same instructor was to teach all classes, CPS and non-CPS. The experimental class was taught using CPS delivery and test taking. Control classes were taught as they would normally be without CPS. All quizzes and the end-of-unit exam were the same in the experimental group as in the control group(s). In units large enough to have several classes the first class of the day was taught without CPS, the next class with CPS and if there were any additional classes they were taught without CPS. Therefore, a comparison could be made between CPS and include the additional classroom dynamics caused by daily routine or increased teacher familiarity with the material. NJROTC participating units had to complete the meteorology unit by the end of the spring semester of the 2005-2006 school year. 4. Trident Training Facility (TTF) 3-year Study: Capt Harkin and LCDR Brady completed an in-depth study in 2003 on the use of CPS at TTF. The results demonstrated significant increase in the same functional areas NJROTC identified as needing improvement. Therefore this study was a cornerstone for the NJROTC investigation and ©2007 Educational Underwriters, Incorporated. All rights Reserved NJROTC needed only to confirm that the findings in the TTF study would translate to an NJROTC classroom. A. TTF CPS/non-CPS grade comparison: The TTF study compared final grades prior to CPS being used in the classroom to final grades after CPS was introduced to the classroom. The ―without CPS‖ comparison used historical exam grades. Their findings, illustrated in figure 1 below, show a statistically significant increase in the final exam grades for each of the courses of instruction using CPS. The courses used in their evaluation were in Missile Technician Courses (MTA), Junior Officer Qualification Courses (JO) and Advanced Sonar. Figure 1. TTF Final Exam Grade comparison of historical and CPS classes (Harkins and Brady 2003). B. TTF survey results: The exam data was supplemented by survey data from instructors and students. The TTF student surveys demonstrated an ―overall favorable impression‖ of CPS. TTF Instructors also had an overall favorable impression of CPS, but there was a concern that material could not be taught in the allotted time. The authors of the study felt this was because instructors were forced to spend more time checking for ©2007 Educational Underwriters, Incorporated. All rights Reserved student understanding and re-teaching those areas in which CPS indicated a lack of comprehension (Harkins and Brady 2003). 5. NJROTC Evaluation Quiz/Test Results: The NJROTC evaluation was slightly different than the TTF evaluation in that it compared data between non-CPS and CPS classrooms taken at the same time vice using historical data. This was done because of the lack of historical data and the inherent inconsistency in testing between units spread over a broad geographic area. Another difference between the TTF study and the NJROTC study was that the TTF study compared only final exams while the NJROTC study compared 4 quiz score averages and one end-of-unit exam average between non-CPS classrooms and CPS classrooms. Figure 2 below illustrates the comparison of the quiz and test scores for the NJROTC study. As in the TTF study, CPS classes demonstrate an obvious consistent advantage over non-CPS classrooms. CPS vs Non CPS Classrooms 90.00 80.00 70.00 60.00 50.00 40.00 30.00 20.00 10.00 0.00 Quiz 1 Quiz 2 Quiz 3 Quiz 4 Test 1 1 Non-CPS 2 With CPS 3 Non-CPS 4 Non-CPS Figure 2: NJROTC exam comparison between non-CPS and CPS classrooms. 6. NJROTC Survey Results: The surveys used in NJROTC were derived from the surveys used in the TTF study and adapted for NJROTC cadets. The cadet survey was given only to the cadets in the CPS ©2007 Educational Underwriters, Incorporated. All rights Reserved classroom as non-CPS classrooms could not comment on the use of CPS. The survey was completed by 251 cadets and 12 instructors. A. NJROTC cadet surveys (Figure 3): Questions 1, 2, 3, 4, 6 and 9 indicate a strong positive response by the cadets to the CPS system. Question 5 response indicates that cadets felt that the use of CPS was worth extra classroom time. Question 7 response indicates that cadets were not intimidated by the use of the system in the classroom. This could be in part because of their familiarity with technology. Question 8 response indicates that students like to participate in class using CPS. Overall, this survey illustrates a very strong positive response by cadets to the use of CPS in the classroom. This is supported by the final open ended question of the survey which asked if given the choice between a CPS classroom and a non-CPS classroom, which they would choose? Of the 227 total cadet responses to this question 91% or 207 said they would choose the CPS classroom. 4.0 Questions Strongly Agree 1. I have enjoyed using CPS in the classroom. 2. CPS helped me learn the material taught in class. 3. CPS helped me pay closer attention to the ideas and concepts presented in class. 4. CPS helped me understand the material better because it showed me the mistakes I made. 5. Using CPS after a short lecture wasted time that could have been used for class. 6. I enjoyed getting more involved with the class using CPS. 3.0 Agree 2.0 Disagree 1..0 Strongly Disagree *3.44 *3.21 *3.15 *3.19 *1.76 *3.20 ©2007 Educational Underwriters, Incorporated. All rights Reserved 7. CPS made me uneasy because I had to participate whether I wanted to or not. *1.65 8. I do NOT like the wireless system, because I do NOT like to participate in class. *1.37 *3.28 9. CPS helped me check my understanding of the material. Figure 3: Average scores for NJROTC cadet survey results. The general comparison between the TTF student survey and the NJROTC cadet survey suggests the cadets have a more favorable impression of CPS than the students in the TTF study. This may be due to the short duration and uniqueness of CPS to the cadets while the TTF students worked with CPS in more classes and over a longer period of time. Therefore, we could infer that a student survey taken after a longer period of use may dip somewhat but would remain positive. B. NJROTC instructor surveys (Figure 4): The strong positive responses to questions 1, 2, 3, 4, 9, 10, 11, and 12 indicate a favorable disposition toward CPS and support the idea that CPS helped the instructors teach, and cadets learn the material. The less strong responses in questions 5, 8, and 10 indicate some reservations about using the CPS system. It appears from the comments discussed in the following paragraphs that their reservations center on the concern of additional time requirements that may be incurred when using CPS. There is an interesting dichotomy between cadet question 5, where the cadets disagree with the statement that class time was wasted using CPS and instructor question 5, where the instructors agreed with a similar statement. A similar situation was encountered in the Harkins and Brady study. In that study, it was hypothesized that without CPS, instructors would ―plow‖ through material without taking time for questioning and discussion. With CPS, more time may be required to prepare questions and participate in discussions, and more class time may be required to ensure that the material was understood, therefore less material could be covered in a given class time. ©2007 Educational Underwriters, Incorporated. All rights Reserved However, the material that is covered would be better understood by the students, resulting in more productive classroom time as indicated by the higher test scores. 4.0 Questions Strongly Agree 1. I have enjoyed teaching using CPS in the classroom. 2. CPS helped me teach the material presented in the lecture. 3. CPS helped cadets pay closer attention to the ideas and concepts presented in class. 4. CPS helped increase understanding by involving lecture participants in more active learning. 5. Using CPS after a short lecture wasted time that I could have been productively devoted to lecture. 6. I enjoyed the increased lecture involvement CPS provided cadets. 7. While using CPS classroom lecture time was used more productively and the material could be covered in the same allocated time. . 8. CPS assisted me to complete all learning points within the allocated time and in some cases the number of learning points was increased. 9. CPS helped me to quickly check for cadet understanding of basic points and clarify points which student missed after a short lecture. 10. Thought provoking questions stimulated the cadet’s desire to seek self discovery and provided them 3.0 Agree 2.0 Disagree 1..0 Strongly Disagree *3.58 *3.33 *3.25 *2.93 *2.44 *3.40 *2.92 *3.00 *3.67 *3.00 ©2007 Educational Underwriters, Incorporated. All rights Reserved with the opportunity to compare their answers to those of the rest of the class. The subsequent discussions explored issues at greater depth and encouraged participation from a much larger percentage of the class. 11. The use of interactive engagement strategies such as CPS can increase course effectiveness well beyond that obtained by traditional methods. 12. I would recommend that CPS be used in all NJROTC classrooms. *3.50 *3.67 Figure 4: Average scores for NJROTC instructor survey. C. NJROTC instructor survey comments: In addition to the scaled survey the instructors were asked to answer several questions concerning their experience. Their responses are summarized below under the questions. (1) Does the wireless system have the potential to save the classroom teacher time and effort? This was one of the most significant concerns of the instructors. Eleven of the twelve instructors agreed that CPS will eventually be a time saver – but they had several reservations. The first concern is the amount of time spent on familiarization of the program and how to operate it within the classroom. One instructor recognized that there was a ―steep learning curve on preparing for class.‖ Another instructor agreed by saying that ―if lessons and evaluation tools are created prior to delivery, the time savings will be significant.‖ All of the instructors were impressed with how quickly CPS graded exams and how this immediate feedback of student performance was a catalyst in overall classroom success. One teacher observed: ©2007 Educational Underwriters, Incorporated. All rights Reserved ―…automatic grading of the quizzes and tests combined with instantaneous feedback to students provides instructors with more time to answer questions FOLLOWING an exam, while the system does the grading grunt work and provides detailed study guides to the student.‖ Another instructor commented positively, saying that ―overall it made me a better teacher as I got almost immediate feedback.‖ The few teachers that complained about time being an issue pointed out that they ―see a problem with having to spend additional time out of the classroom with each revision of the curriculum. ― (2) Do you think you could use CPS without any additional curriculum items? In other words, if you had NOT been given the CPS meteorology disc, could you have used CPS and still been effective? The issue surrounding whether or not CPS is capable of being added to the curriculum without additional curriculum materials goes back to the amount of work necessary to create the lessons. One teacher is concerned that in order for ―CPS to be used to its maximum potential, instructors would require much greater training than we received.‖ This instructor also points out that ―instructors who are less proficient with computers would probably give up in frustration and not use the system.‖ There are those who were successful at lesson creation and felt ―…it would be cumbersome to alternate between instructional mediums, but having CPS would still allow for concept polling and testing thereby providing advantages.‖ (3) If you had a choice between updating a year of curriculum or purchasing CPS, what would you do? All of the instructors agreed that CPS is a program worthy of purchasing and implementing into the classroom. Most of the responses concerning this question were similar to ―buy CPS!‖ or ―purchase CPS, definitely!‖ Others would purchase the system, but ―only with (curriculum) discs (provided by headquarters)!‖ 7. Discussion: ©2007 Educational Underwriters, Incorporated. All rights Reserved Army JROTC is spearheading the further development of CPS for the JROTC environment, including the creation of sophisticated games and standard CPS formats for questions that we can use in our curriculum to help instructors use CPS. NJROTC will be able to use, without development costs, those same games, formats and questions. In order to assess the general feeling of more of the instructors throughout the country, CPS demonstrations were conducted during in-service training sessions for all eleven areas. The response was overwhelmingly positive with many instructors wanting immediate access to CPS. Others took CPS purchase information back to their schools to lobby the school to purchase the CPS system. However, similar concerns were expressed about the amount of time instructors would have to spend developing CPS-style lessons. Some instructors were adamantly against fielding the system without the curriculum being built around it. Current funding levels will not support both routine annual curriculum reviews and the development of a new curriculum model using CPS. This is why the last question on the instructor survey concerning the priority use of resources between maintaining the current model and changing our curriculum model was asked. The answer was an enthusiastic ―yes‖ that our resources should focus on CPS and moving toward a student centered learning model. The same response was received during in-service briefs. In order to provide the best possible product for the instructors and cadets, additional specific funding is required for CPS so curriculum reviews could continue during which lessons could be updated to include CPS related material in a student centered learning model. CPS appears to be the best way to improve the learning of the cadets as well as have a positive effect on the teaching methods used by our instructors. It gets NJROTC moving in the right direction immediately. Regardless of whether NJROTC acquires CPS, a change must be made in the NJROTC’s curriculum model. Lecture-and-listen is not effective in a high school environment. We need to shift to a student-centered learning/active learning model. We need to make this change for two reasons. First, if we are going to teach the curriculum, then we want the ©2007 Educational Underwriters, Incorporated. All rights Reserved students to retain the material. Second, we want to attract more students via dynamic, contemporary, discussion-lead classes that use the latest technology. Lecturing teenagers is not an effective way to teach. It won’t attract students to the program. The NJROTC curriculum office is reviewing how we may be able to adapt some of the other Services’ student-centered learning curricula to our curriculum to save developmental costs. 8. Conclusions: A. The data in our study and in all other studies available to us demonstrate that CPS is effective in improving student performance. B. The surveys support the adaptability of CPS to the NJROTC curriculum and show overwhelming positive support by cadets in the NJROTC program C. Instructors support using the CPS system but are concerned that they will not get the curriculum support if and when they get the system. D. Quality of instruction is improved when using CPS by forcing instructors to think more about questioning and discussion in the classroom. Additionally, it requires the instructors to periodically check for understanding, not just from one student, but from the class as a whole, and adjust the teaching methodologies accordingly E. CPS is the first step in moving our curriculum away from lecture-and-listen toward a student-centered/active learner approach. It is the most important of many changes that need to be implemented since it changes the way in which we teach. 9. Recommendations: A. As soon as practical, using whatever resources are available, acquire CPS and CPS related equipment. This position is supported by the instructors who participated in the evaluation as well as the curriculum committee during their last meeting in February 06. The sooner we get the system to the instructors the sooner we will be able to learn CPS best practices and apply it to our curriculum. ©2007 Educational Underwriters, Incorporated. All rights Reserved B. Request funding from Naval Service Training Command to support our change to a student-centered learning/active approach. (1) Provide funding for CPS thus freeing up funding to conduct the necessary curriculum revisions to include student-centered learning methodology into our curriculum. This would permit all NJROTC instructors, regardless of their proficiency with CPS, to use the system to a greater potential, as well as help save time for the instructors. (2) As discussed above, CPS is the first step in overcoming the larger problem of curriculum model change. In the near future, NJROTC needs to leverage off Army JROTC educational research to make our curriculum more effective and more desirable to the general high school student body. C. Since it is unlikely that we will be able to purchase CPS for everyone at one time, distribute CPS only to those instructors motivated to use this new technology. In this manner we will have motivated instructors establishing the lesson plans and models that can be used for all instructors at a later date. Our web portal has already been set up with a CPS lessons learned forum where instructors can share ideas and best practices. ©2007 Educational Underwriters, Incorporated. All rights Reserved Learning Pyramid The Learning Pyramid charts the average retention rate for various methods of teaching. These retention percentages represent the results of research conducted by National Training Laboratories in Bethel, Maine. According to the chart, lecture, the top of the pyramid, achieves an average retention rate of 5%. On the opposite end of the scale, the "teach others/immediate use" method achieves an average retention rate of 90%. The Abilene Christian University Center for Teaching Excellence Adam’s Center http://www.acu.edu/cte/activelearning/whyuseal2.htm ©2007 Educational Underwriters, Incorporated. All rights Reserved Enclosure (1) ©2007 Educational Underwriters, Incorporated. All rights Reserved Print March/April 2006 (Page 19) pwf.texterity.com/edtech/20060304/templates/pageviewer_... ://edtech- ©2007 Educational Underwriters, Incorporated. All rights Reserved 1 of 1 ©2007 Educational Underwriters, Incorporated. All rights Reserved Print March/April 2006 (Page 20) ://edtech-pwf.texterity.com/edtech/20060304/templates/pageviewer_... ©2007 Educational Underwriters, Incorporated. All rights Reserved 1 of 1 5/23/2006 4:33 PM ©2007 Educational Underwriters, Incorporated. All rights Reserved Print March/April 2006 (Page 21) pwf.texterity.com/edtech/20060304/templates/pageviewer_... ://edtech- ©2007 Educational Underwriters, Incorporated. All rights Reserved