Development of Two-Tier Diagnostic Test Wang, Jing-Ru Development of Two-Tier Diagnostic Test for Investigating Students’ Understanding of Plant Transport and Human Circulation Jing-Ru Wang Dept. of Science Education, National Pingtung Teachers college Taiwan 1. Introduction In biology, two-tier diagnostic test had been only used to investigate students’ understanding of photosynthesis and respiration (Haslam & Treagust, 1987). In their report, Haslam and Treagust only stated the process of developing the instrument but did not provide the evidentceof validation, such as the interview data and how the interview data became the choices of the item. This study was to duplicate the method of two-tier item development about students’ understanding and alternative conceptions on the materials transported in human body and plant. In addition, this report will provide the evidence of validity and discuss statistical reliability of the instrument. 2. Research methodology The two-tier multiple-choice diagnostic instrument described by Treagust (1988) was use in this research. Of which, the first tier of each item consists of a content question with two or three choices; the second part of each item contains a set of three to five justifications for the answer to the first part. Included in these justifications are the correct answer and two to four distracters. Distracters are derived from actual students’ alternative explanations gathered from the literature, interviews and free responses. The design of the diagnostic instrument was based on a process described by Treagust (1988) and Haslam and Treagust (1987) that involved three phases and 10 steps. In which three phases were: (a) defining the content boundary in terms of prepositional knowledge statements and concept map for the specific concept; (b) obtaining information about students’ alternative conceptions, and (c) developing the instrument. In order to make the research steps more logically linked and duplicable, the current research refined the process from original 10 steps to 15 steps. For example, in phase1, the content boundary was based on the step 1, reviewing science textbooks and teaching guides. In phase 2, before step 8, conducting interviews, teachers were trained with interview skills (step 7). In phase 3, the refinement of the instrument was based on step 12,expert validation and step 13, pilot tested. Figure 1 is the flowchart of the developmental process. The convenience sampling was used in this research from the student population in southern part of Taiwan across different educational levels: grade 4 (n=132), grade 6 (n=147), grade 7 (n=151), grade 8 (n=153), grade 10 (n=153), grade 11 with non-science track (n=162), and grade 11 with science track (n=165). The total number of students was 1063. Of which there were 501males and 562 females. Development of Two-Tier Diagnostic Test Wang, Jing-Ru Phase 1: Defining the content boundaries Identifying propositional No knowledge statements. （Step 2） Relating Reviewing science textbooks propositional and teaching guides. Validating the knowledge to the content. （Step 1） content map. Yes （Step 5） ( Step 4） Developing a concept map （Step 3） Phase 2: Identifying students’ misconceptions Developing multiple Examining related Training teachers with Conducting interviews. choice content items literature. interview skills. （Step 8） with free response. （Step 6） （Step 7） （Step 9） Phase 3: Instrument development Developing the two tier diagnostic tests. Validating the instrument. （Step 10） （Step 12） Refinement. Conducting statistical （Step 14） analysis. Designing a specific grid. Conducting pilot test. （Step 15） （Step 11） （Step 13） Figure 1. Flow chart of content and item validity. Development of Two-Tier Diagnostic Test Wang, Jing-Ru 3. Result and discussion 1. Validity （1）Content validity The final versions of the propositional knowledge statements and concept maps were constructed for the development of test items. Of that there were five key areas involved as following: a) Functions of transport system b) Routes of plant transport c) Mechanism of plant transport d) Route of blood flow in human body e) Mechanisms of blood flow in human body Each key area contained two or three propositional knowledge statements. （2）Item validity The three versions of two-tier multiple-choice items were constructed based the interview data. One version was for students of elementary school level (grade 4 to grade 6), one for middle school level (grade 7 and grade 8) and one for high school level (grade 10 and up). The distracters of each item were from students interview data and validated by nine experts. (3) Reliability The version of middle school level showed highest reliability than other two versions. First, The one-month interval test-retest reliability was estimated to be .569 for elementary school level, .804 for middle school level and .753 for high school level. Although the version of elementary school level showed lowest reliability, the whole reliability of the three versions reached .01 significantly correlated. This evidence implied that the development of test items through the process of student interviews and data analysis might lead to the stability of the test result. (4) Difficulty indices and discrimination indices In terms of difficulty indices, the version of elementary school level was most difficult with mean value .31; the version of high school level was easiest with the mean value of .40. The difficulty indices of most items ranged from .20 to .79, providing a wise range of difficulty items. Regarding the range of discrimination, the discrimination indices of most items were in the range from .2 to .8, only a few items had low discrimination indices with value below .2. A summary of the test characters of the instruments is presented in Table 1. Development of Two-Tier Diagnostic Test Wang, Jing-Ru Table 1. Characters of the instruments for three educational levels: elementary school, middle school and high school. Elementary Middle High sch. sch. sch. No. of items 23 28 25 Cronbach Alpha 0.618 0.855 0.785 Pearson correlation 0.574** 0.804** 0.753** No. of items with r=0.05 5 2 4 No. of items with r=0.01 7 19 13 Discrimination Indices Mean 0.47 0.45 0.37 No. of items (Range .6≦d< .8) 6 8 2 No. of items (Range.4≦d< .6) 11 8 4 No. of items (Range.2≦d< .4) 4 10 13 No. of items (Range .d<.2) 2 2 4 Difficulty Indices Means 0.31 0.33 0.40 No. of items (Range .6≦p< .8) 1 2 4 No. of items (Range.4≦p< .6) 4 6 8 No. of items (Range.2≦p< .4) 14 15 10 No. of items (Range .p<.2) 4 5 3 4. Conclusions In this research, although the test-retest reliability, discrimination indices of all three test versions were found to be proper, the elementary school version had lower degree of reliability and difficulty indices than other two versions. These results indicated that the boundary content of plant transport and human circulation defined in this research might be too difficult for the elementary students. The instrument of elementary version should be used with caution. Researchers should combine other research methods in their studies to provide closer picture of educational phenomena. Based on the statistical data, researcher could choose proper items for measuring their interested educational levels and concept areas. Development of Two-Tier Diagnostic Test Wang, Jing-Ru 5. References Haslam, F. & Treagust, D. F. (1987). Diagnosing secondary students’ misconceptions of photosynthesis and respiration in plants using a two-tier multiple choice instrument. Journal of Biological Education, 21(3), 203-211. Treagust, D. F. (1988). Development and use of diagnostic tests to evaluate students’ misconceptions in science. Journal of Biological Education, 10(2), 159-169.
Pages to are hidden for
"Development of Two-Tier Diagnostic Test for Investigating Students"Please download to view full document