Improving and Assessing Instructional Effectiveness

Document Sample
Improving and Assessing Instructional Effectiveness Powered By Docstoc
					Improving and Assessing
Instructional Effectiveness
                         Theodore Frick
              Department of Instructional Systems Technology
                           School of Education
                     Indiana University Bloomington


     Invited Lecture to IUB Medical Sciences Faculty
                     October 14, 2002
Overview

   Confounding of instructional
    effectiveness by other factors
   Types of knowledge about education
   Type 2, 3 and 4 examples
   Recommendations




                   Improving and Assessing Instructional
October 14, 2002        Effectiveness -- Ted Frick         2
The Logic of Effectiveness
   Before instruction      After instruction                   Conclusion

   1. Student nonmastery   Student mastery                     Instruction appears to be
                                                               effective enough, but …


   2. Student nonmastery   Student nonmastery                  Instruction apparently was not
                                                               effective enough

   3. Student mastery      Student mastery                     Can't tell if instruction is
                                                               effective, since student had
                                                               achieved the goal before
                                                               instruction began

   4. Student mastery      Student nonmastery                  ??? Something wrong with the
                                                               content - factual errors,
                                                               incorrect procedures, bad
                                                               models ??? Bad luck ???



                           Improving and Assessing Instructional
 October 14, 2002               Effectiveness -- Ted Frick                                      3
Confounding Factors

   1.      Learning occurs without intentional
           instruction.
   2.      Learning occurs despite poor
           instruction.
   3.      We‟re not the only teachers a student
           has.
   4.      Results of instruction may occur LONG
           afterwards.
                     Improving and Assessing Instructional
 October 14, 2002         Effectiveness -- Ted Frick         4
Conflation of Research Methods and
Outcomes
    In addition to the confounding problem in
     determining effectiveness of instruction,
     there has been a lot of debate in
     education about inquiry methods – e.g.,
     quantitative vs. qualitative.
    Continuing problem of bridging research
     and practice in education


                    Improving and Assessing Instructional
 October 14, 2002        Effectiveness -- Ted Frick         5
Consider this Debate
   Imagine for a moment three carpenters arguing about
     which tool is best.

      Quanta: "Hammers and nails are clearly superior."
      Qualia: "I disagree. Screwdrivers and screws are
       much more effective."
      Performa: "You're both wrong. Saws are best for
       cutting wood."
      Quanta: "Who said anything about cutting wood? I
       thought we were talking about fastening wood
       together."
      Qualia: "Right. Who needs saws?"
      Performa: "I do. I need to cut this board in half."

                       Improving and Assessing Instructional
 October 14, 2002           Effectiveness -- Ted Frick         6
Premise


      The kind of knowledge about
      education that we create through
      disciplined inquiry determines what
      research methods are appropriate and
      useful.



                   Improving and Assessing Instructional
October 14, 2002        Effectiveness -- Ted Frick         7
Types of Educology: Knowledge
about Education
      Methodology of Theory Building (Elizabeth
       Steiner, 1988) (link)
                Non-axiological knowledge – what is
                Axiological knowledge – pertaining to values:
                       Instrumental value: good for – what is effective
                       Intrinsic value: good in itself – what is worthwhile
      The Dependability of Behavioral
       Measurements: Theory of Generalizability for
       Scores and Profiles (Lee Cronbach, et al.,
       1972)
                                 Improving and Assessing Instructional
 October 14, 2002                     Effectiveness -- Ted Frick               8
Six Types of Knowledge (Frick)

                                                  Scope of Knowledge
        Purpose of Inquiry                              Unique         Generalizable

                    What is?                                1               2
           What is effective?                               3               4
        What is worthwhile?                                 5               6


                      See Frick: R690 Syllabus for further details



                               Improving and Assessing Instructional
 October 14, 2002                   Effectiveness -- Ted Frick                         9
Six Types of Knowledge (Frick)

                                                    Scope of Knowledge
          Purpose of Inquiry                             Unique         Generalizable

                    What is?                                 1               2
           What is effective?                                3               4
        What is worthwhile?                                  5               6

             Example: Type 1 Knowledge Claim

              „Elizabeth Steiner is an educational philosopher.‟

                                Improving and Assessing Instructional
 October 14, 2002                    Effectiveness -- Ted Frick                         10
Six Types of Knowledge (Frick)

                                                  Scope of Knowledge
          Purpose of Inquiry                           Unique         Generalizable

                    What is?                               1               2
           What is effective?                              3               4
        What is worthwhile?                                5               6

             Example: Type 2 Knowledge Claim
             „Students who spend more time engaged successfully in
             arithmetic and reading tasks score more highly on achievement
             tests in those content areas.‟ (Fisher, et al., 1976)
                              Improving and Assessing Instructional
 October 14, 2002                  Effectiveness -- Ted Frick                         11
Six Types of Knowledge (Frick)

                                                  Scope of Knowledge
          Purpose of Inquiry                           Unique         Generalizable

                    What is?                               1               2
           What is effective?                              3               4
        What is worthwhile?                                5               6

             Example: Type 3 Knowledge Claim
             „The DISTAR program for teaching arithmetic and reading to
             elementary students works.‟

                              Improving and Assessing Instructional
 October 14, 2002                  Effectiveness -- Ted Frick                         12
Six Types of Knowledge (Frick)

                                                      Scope of Knowledge
          Purpose of Inquiry                               Unique         Generalizable

                    What is?                                   1               2
           What is effective?                                  3               4
        What is worthwhile?                                    5               6
             Example: Type 4 Knowledge Claim
             „Instruction is most effective when it is problem-based, activates prior
             learner knowledge, demonstrates what is to be learned, provides
             opportunities for guided practice, and encourages integration with
             everyday life.‟ (Merrill, 2001)
                                  Improving and Assessing Instructional
 October 14, 2002                      Effectiveness -- Ted Frick                         13
Six Types of Knowledge (Frick)

                                                  Scope of Knowledge
          Purpose of Inquiry                           Unique         Generalizable

                    What is?                               1               2
           What is effective?                              3               4
        What is worthwhile?                                5               6

             Example: Type 5 Knowledge Claim

             „The corporal punishment policy in Houston schools is a bad
             policy.‟
                              Improving and Assessing Instructional
 October 14, 2002                  Effectiveness -- Ted Frick                         14
Six Types of Knowledge (Frick)

                                                  Scope of Knowledge
          Purpose of Inquiry                           Unique         Generalizable

                    What is?                               1               2
           What is effective?                              3               4
        What is worthwhile?                                5               6

             Example: Type 6 Knowledge Claim

             „Teachers and students should respect each other.‟

                              Improving and Assessing Instructional
 October 14, 2002                  Effectiveness -- Ted Frick                         15
Do not confuse ends and means
  The kind of knowledge about education that we
    create through disciplined inquiry determines
    what research methods are appropriate and
    useful.
           Outcome of research is knowledge.
           Inquiry methods are a means to that end.
           Criteria for evaluating adequacy of research
            methodology are NOT the same for each
            knowledge type – e.g.,
                  criteria for statistical inference to a broad population do not
                   apply to Types 1, 3, and 5;
                  effectiveness not of concern in Types 1, 2, 5, and 6, but
                   main concern in 3 and 4.

                                Improving and Assessing Instructional
October 14, 2002                     Effectiveness -- Ted Frick                  16
For today…

  Focus on a few examples of Types 2, 3
   and 4 knowledge of education
         Type 2: Academic learning time research
         Type 3 outcomes: Program/product
          evaluation – Kirkpatrick‟s 4 levels (1959)
         Type 4 process: Effective Web Instruction –
          Frick & Boling (2002)
         Type 4: Principles of 5-Star Instruction –
          David Merrill (2001)
                     Improving and Assessing Instructional
October 14, 2002          Effectiveness -- Ted Frick         17
Six Types of Knowledge (Frick)

                                               Scope of Knowledge
          Purpose of Inquiry                        Unique         Generalizable

                    What is?                            1               2
           What is effective?                           3               4
        What is worthwhile?                             5               6

          Type 2 Example: Relationship of academic
          learning time and student achievement

                           Improving and Assessing Instructional
 October 14, 2002               Effectiveness -- Ted Frick                         18
Type 2: Academic Learning Time

                                                                       Task Success

                                                                       Engaged Time

                                                                       Allocated Time




                ALT is positively correlated with academic
                achievement in the same content areas (Fisher,
                et al., 1976; Rieth & Frick, 1983; Berliner; 1985).

                               Improving and Assessing Instructional
 October 14, 2002                   Effectiveness -- Ted Frick                          19
Six Types of Knowledge (Frick)

                                               Scope of Knowledge
          Purpose of Inquiry                        Unique         Generalizable

                    What is?                            1               2
           What is effective?                           3               4
        What is worthwhile?                             5               6

           Type 3: Evaluation of a particular instructional
           program or product

                           Improving and Assessing Instructional
 October 14, 2002               Effectiveness -- Ted Frick                         20
Type 3: Program/Product
Evaluation – Kirkpatrick‟s Levels
   Techniques for Evaluating Training
      Programs -- Donald Kirkpatrick (1959)

         1.     Reaction (satisfaction)
         2.     Learning (achievement)
         3.     Behavior (transfer to real context)
         4.     Results (impact on organization/context)

   More on Kirkpatrick‟s levels from Encyclopedia of Educational Technology

                            Improving and Assessing Instructional
 October 14, 2002                Effectiveness -- Ted Frick                   21
Six Types of Knowledge (Frick)

                                               Scope of Knowledge
          Purpose of Inquiry                        Unique         Generalizable

                    What is?                            1               2
           What is effective?                           3               4
        What is worthwhile?                             5               6

           Type 4 Example: how to create effective
           instructional products – design theory

                           Improving and Assessing Instructional
 October 14, 2002               Effectiveness -- Ted Frick                         22
Type 4: Effective Web Instruction: An
Inquiry-Based Process – Frick & Boling




                                                            More

                    Improving and Assessing Instructional
 October 14, 2002        Effectiveness -- Ted Frick                23
Type 4: First Principles of
Instruction – David Merrill


              Integration                                 Activation



                                  Problem


              Application                                 Demonstration



                            Improving and Assessing Instructional
 October 14, 2002                Effectiveness -- Ted Frick               24
Type 4: First Principles of
Instruction (cont‟d)
   “Learning is facilitated when:
         1. Learners are engaged in solving real-world
            problems.
         2. Existing knowledge is activated as a foundation
            for new knowledge.
         3. New knowledge is demonstrated to the learner.
         4. New knowledge is applied by the learner.
         5. New knowledge is integrated into the learner‟s
            world.” (Merrill, 2001, p. 2)
         Does your instruction rate 5 stars? A rating scale

                        Improving and Assessing Instructional
 October 14, 2002            Effectiveness -- Ted Frick         25
Six Types of Knowledge (Frick)

                                               Scope of Knowledge
          Purpose of Inquiry                        Unique         Generalizable

                    What is?                            1               2
           What is effective?                           3               4
        What is worthwhile?                             5               6

           Type 3: Indiana University physician education
           program

                           Improving and Assessing Instructional
 October 14, 2002               Effectiveness -- Ted Frick                         26
Recommendations

  In physician education, focus on Type 3
    knowledge:
         Apply Merrill‟s 5 principles as criteria for
          design. See video (requires RealPlayer).
         Use Kirkpatrick‟s 4 levels of evaluation for
          your programs at IU.




                      Improving and Assessing Instructional
October 14, 2002           Effectiveness -- Ted Frick         27
Questions?

     This presentation is available at:

        http://www.indiana.edu/~tedfrick/improveinstruction.ppt


     Contact: frick@indiana.edu




                           Improving and Assessing Instructional
October 14, 2002                Effectiveness -- Ted Frick         28

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:7
posted:4/17/2011
language:English
pages:28