Your Federal Quarterly Tax Payments are due April 15th Get Help Now >>

Development of Two-Tier Diagnostic Test for Investigating Students by vev19514


									Development of Two-Tier Diagnostic Test
Wang, Jing-Ru

    Development of Two-Tier Diagnostic Test for Investigating
Students’ Understanding of Plant Transport and Human Circulation
                                       Jing-Ru Wang
               Dept. of Science Education, National Pingtung Teachers college

1. Introduction
In biology, two-tier diagnostic test had been only used to investigate students’ understanding
of photosynthesis and respiration (Haslam & Treagust, 1987). In their report, Haslam and
Treagust only stated the process of developing the instrument but did not provide the
evidentceof validation, such as the interview data and how the interview data became the
choices of the item. This study was to duplicate the method of two-tier item development
about students’ understanding and alternative conceptions on the materials transported in
human body and plant. In addition, this report will provide the evidence of validity and
discuss statistical reliability of the instrument.

2. Research methodology
The two-tier multiple-choice diagnostic instrument described by Treagust (1988) was use in
this research. Of which, the first tier of each item consists of a content question with two
or three choices; the second part of each item contains a set of three to five justifications for
the answer to the first part. Included in these justifications are the correct answer and two
to four distracters. Distracters are derived from actual students’ alternative explanations
gathered from the literature, interviews and free responses. The design of the diagnostic
instrument was based on a process described by Treagust (1988) and Haslam and Treagust
(1987) that involved three phases and 10 steps. In which three phases were: (a) defining the
content boundary in terms of prepositional knowledge statements and concept map for the
specific concept; (b) obtaining information about students’ alternative conceptions, and (c)
developing the instrument. In order to make the research steps more logically linked and
duplicable, the current research refined the process from original 10 steps to 15 steps. For
example, in phase1, the content boundary was based on the step 1, reviewing science
textbooks and teaching guides. In phase 2, before step 8, conducting interviews, teachers
were trained with interview skills (step 7). In phase 3, the refinement of the instrument
was based on step 12,expert validation and step 13, pilot tested. Figure 1 is the flowchart
of the developmental process.
The convenience sampling was used in this research from the student population in southern
part of Taiwan across different educational levels: grade 4 (n=132), grade 6 (n=147), grade
7 (n=151), grade 8 (n=153), grade 10 (n=153), grade 11 with non-science track (n=162),
and grade 11 with science track (n=165). The total number of students was 1063. Of
which there were 501males and 562 females.
Development of Two-Tier Diagnostic Test                                                                                                         Wang, Jing-Ru

Phase 1: Defining the content boundaries

                                                  Identifying propositional                                                       No
                                                   knowledge statements.
                                                         (Step 2)
   Reviewing science textbooks                                                                  propositional
      and teaching guides.                                                                                               Validating the
                                                                                              knowledge to the             content.
           (Step 1)                                                                             content map.                                    Yes
                                                                                                                          (Step 5)
                                                                                                  ( Step 4)

                                                 Developing a concept map
                                                       (Step 3)
Phase 2: Identifying students’ misconceptions

                                                                                                                 Developing multiple
         Examining related                  Training teachers with              Conducting interviews.           choice content items
            literature.                        interview skills.                    (Step 8)                      with free response.
            (Step 6)                              (Step 7)                                                            (Step 9)

Phase 3: Instrument development

                Developing the two tier
                   diagnostic tests.                      Validating the instrument.
                     (Step 10)                                  (Step 12)
                                                                                                 Refinement.           Conducting statistical
                                                                                                 (Step 14)                  analysis.
               Designing a specific grid.                   Conducting pilot test.                                        (Step 15)
                     (Step 11)                                 (Step 13)

                                                       Figure 1. Flow chart of content and item validity.
Development of Two-Tier Diagnostic Test                                       Wang, Jing-Ru

3. Result and discussion
1. Validity
(1)Content validity
The final versions of the propositional knowledge statements and concept maps were
constructed for the development of test items. Of that there were five key areas involved as
   a) Functions of transport system
   b) Routes of plant transport
   c) Mechanism of plant transport
   d) Route of blood flow in human body
   e) Mechanisms of blood flow in human body
Each key area contained two or three propositional knowledge statements.
(2)Item validity
The three versions of two-tier multiple-choice items were constructed based the interview
data. One version was for students of elementary school level (grade 4 to grade 6), one for
middle school level (grade 7 and grade 8) and one for high school level (grade 10 and up).
The distracters of each item were from students interview data and validated by nine experts.

(3) Reliability
The version of middle school level showed highest reliability than other two versions. First,
The one-month interval test-retest reliability was estimated to be .569 for elementary school
level, .804 for middle school level and .753 for high school level. Although the version of
elementary school level showed lowest reliability, the whole reliability of the three versions
reached .01 significantly correlated. This evidence implied that the development of test
items through the process of student interviews and data analysis might lead to the stability of
the test result.

(4) Difficulty indices and discrimination indices
In terms of difficulty indices, the version of elementary school level was most difficult with
mean value .31; the version of high school level was easiest with the mean value of .40. The
difficulty indices of most items ranged from .20 to .79, providing a wise range of difficulty
Regarding the range of discrimination, the discrimination indices of most items were in the
range from .2 to .8, only a few items had low discrimination indices with value below .2.
A summary of the test characters of the instruments is presented in Table 1.
Development of Two-Tier Diagnostic Test                                         Wang, Jing-Ru

Table 1. Characters of the instruments for three educational levels: elementary school, middle
school and high school.
                                  Elementary         Middle              High
                                      sch.             sch.               sch.
No. of items                           23               28                 25
Cronbach Alpha                       0.618            0.855              0.785
Pearson correlation                 0.574**          0.804**            0.753**
   No. of items with r=0.05             5                2                  4
   No. of items with r=0.01             7               19                 13
Discrimination Indices
             Mean                         0.47        0.45               0.37
 No. of items (Range .6≦d< .8)              6           8                  2
  No. of items (Range.4≦d< .6)            11            8                 4
  No. of items (Range.2≦d< .4)             4           10                13
   No. of items (Range .d<.2)              2            2                 4

Difficulty Indices
              Means                       0.31        0.33               0.40
 No. of items (Range .6≦p< .8)              1           2                  4
  No. of items (Range.4≦p< .6)             4            6                 8
  No. of items (Range.2≦p< .4)            14           15                10
   No. of items (Range .p<.2)              4            5                 3

4. Conclusions
In this research, although the test-retest reliability, discrimination indices of all three test
versions were found to be proper, the elementary school version had lower degree of
reliability and difficulty indices than other two versions. These results indicated that the
boundary content of plant transport and human circulation defined in this research might be
too difficult for the elementary students. The instrument of elementary version should be
used with caution. Researchers should combine other research methods in their studies to
provide closer picture of educational phenomena. Based on the statistical data, researcher
could choose proper items for measuring their interested educational levels and concept areas.
Development of Two-Tier Diagnostic Test                                     Wang, Jing-Ru

5. References
     Haslam, F. & Treagust, D. F. (1987). Diagnosing secondary students’ misconceptions of
photosynthesis and respiration in plants using a two-tier multiple choice instrument. Journal
of Biological Education, 21(3), 203-211.

    Treagust, D. F. (1988). Development and use of diagnostic tests to evaluate students’
misconceptions in science. Journal of Biological Education, 10(2), 159-169.

To top