dependent) variable on dependent variables is central to

Document Sample
dependent) variable on dependent variables is central to Powered By Docstoc
					JOURNAL OF APPLIED BEHAVIOR ANALYSIS                      1978, 113,357-362            NUMBER     3 (FALL 1978)

                                   TERRY F. PECHACEK1
                                         UNIVERSITY OF MINNESOTA

           Without internal validity, experimental data are uninterpretable. With intensive designs,
           most methods presented to quantify a design's internal validity have been subject to
           criticism. A probabilistic model of intensive designs is presented that demonstrates the
           high degree of internal validity of these designs without relying on adaptations from
           traditional inferential statistics. Where the experimenter is able to conform to the re-
           strictions of the model, the equations provide an estimation of internal validity for
           either reversal or multiple-baseline designs. More importantly, the model provides
           mathematical bases for some of the common recommendations and design considerations
           in intensive research (such as the desirability of within-subject replications and of four or
           more multiple baselines).
              DESCRIPTORS: experimental design, intensive design, probabilistic model of signifi-
           cance, methodology, multiple baselines, guidelines, single organism research

   Demonstration of internal validity of infer-            hypotheses not based on independent variables,
ences about the effects of an experimental (in-            especially those based on extraneous variables
dependent) variable on dependent variables is              such as history (psychosocial events concurrent
central to experimental investigation (Campbell            with but unrelated to experimental influences)
and Stanley 1966). While traditional inferential           (Campbell and Stanley, 1966, Kazdin and Ko-
statistics provide numeric estimations of internal         pel, 1975).
validity, it can be meaningfully argued that                  Kazdin (1976) reviewed both the appropri-
they can obscure the true relationship between             ateness and controversy regarding efforts to
independent and dependent variables by aver-               adapt group-comparison statistics for single-
aging within-subject variability of response to            subject designs. The following is an attempt to
treatment (Hersen and Barlow, 1976; Sidman,                demonstrate by means of a probabilistic model
1960). Hersen and Barlow (1976) outlined                   that intensive designs do have a high degree of
the advantages of single-subject designs in this           internal validity without relying on adaptations
area, but the internal validity of these designs           from traditional group statistics. In formula-
bears further elucidation.                                 tions of a more general nature than those pre-
   Internal validity concerns the ability of the           sented by Revusky (1967), the internal validity
design to discount or eliminate plausible rival            of both the reversal and multiple-baseline de-
   1 Portions of this paper were presented at the an-
                                                           signs can be demonstrated. Also, these formula-
nual meeting of the Association for Advancement of
                                                           tions offer mathematical rationales for some
Behavior Therapy, Atlanta, December 1977. The ba-          of the common recommendations for using
sic work on this manuscript was completed while the        these designs.
author was a clinical psychology intern at the Palo           In this model, the issue of what magnitude of
Alto Veterans Administration Hospital, Palo Alto,
California. The author would like to express his grati-    observed effects or change is significant will not
tude to Alan E. Kazdin for his editorial assistance on     be addressed;2 rather, consistent with the basic
this manuscript. Reprints may be obtained from Terry
F. Pechacek, Laboratory of Physiological Hygiene,             2The reader is referred to the discussion of clinical
Stadium Gate 27, University of Minnesota, Minne-           versus statistical significance in Hersen and Barlow
apolis, Minnesota 55455.                                   (1976) and to the chapter by Kazdin (1976) on the
358                                         TERRY F. PECHACEK

philosophy of single-subject research, "change"             phase. A statistic can be formulated that esti-
will be defined as a departure from baseline of             mates the likelihood that treatment and reversal
a magnitude that would be judged of clinical                effects could be due to extraneous, random
significance by the research community at                   factors. This statistic is based on the number of
large; "no change" is defined as all other occur-           possible patterns of results that could occur if
rences. Therefore, these formulations are not               the outcomes were due only to the random fac-
dependent on traditional statistical analyses, nor          tors. In the most general cases, random factors
do they preclude their use.                                 could produce three outcomes at each phase.
   In the development of this model, a basic as-            (no change, an increase, or a decrease). Since the
sumption must be made. Consistent with infer-               experimenter is a priori defining a single, ex-
ential statistics, plausible rival hypotheses for           pected outcome for each phase, there is a proba-
observed treatment effects are considered as a              bility of 0.33 that this definition is correct if the
single null hypothesis and expressed as the po-             outcome is due to random factors. In a design
tential for random change (that is, not system-             with N experimental, postbaseline4 phases, the
atically related to the experimental variables).            probability that the outcome in all phases would
This basic assumption states that the observed              be correctly defined is equal to 0.33 * N.5
behavior has an equal probability of not chang-                In some experimental analyses, the observed
ing or changing (increasing or decreasing) at               behavior may not have the freedom either to
any time when measured in the natural state.                increase or decrease. Such a case would occur
   This assumption does not imply that proxi-               if the frequency was at or reduced almost to a
mate observation or measurement are not corre-              zero rate or near the physical or biological
lated. Rather, this assumption states that the              maximum rate of occurrence. In such cases, the
effects of the extraneous factors can become                probability of correct definition would be in-
manifest at any time and, a priori, there is no             creased to 0.50. Table 1 presents the probabili-
more than a random chance to suspect that                   ties for increasing Ns in both the unrestricted
they will coincide with the experimental vari-              and common restricted types of cases.
ables. If this assumption is accepted, the model               The statistic can be simply and directly com-
computes an estimate of the likelihood that                 puted for any design. Moreover, if each phase is
rival hypotheses based on extraneous, random                continued for a sufficient period of observations,
factors (such as history) could account for the             the intervention or reversal phases can be subdi-
observed results in reversal or multiple-baseline           vided. For example, the standard A-B-A design
intensive designs.3                                         of two experimental phases can be expanded
                                                              4The initial baseline cannot be considered as an
             REVERSAL DESIGNS                               experimental phase in this model since the require-
                                                            ments for obtaining a stable baseline preclude accept-
   In reversal designs, the experimenter defines            ance of the basic assumption. If such extraneous
the direction of change that is expected in each            factors were to produce significant variability, the
                                                            logic of the design requires the baseline to be con-
                                                            tinued until it restabilized.
appropriations of various statistical analysis in single-      5In reality, the probability (Pa) of the common
case designs.                                               patterns of data may not be 0.33 * N but would ac-
   3While this assumption is constructed to be as           tually be a function of the true probability (Pal) of
consistent as possible with the phenomenon of in-           the behavior changing in the desired direction in
tensive research, the application of the hypothesis         each of the N phases. Hence, the true probability
testing approach, which is standard to most psycho-         (Pa) =  (P.z)(Pa2)  . . . (PN), where 0 < Pai       1.0.
                                                                                          in the first experimental
logical research, does not always suit the endeavors        In many situations, such as
of the applied researcher whose goal often is the           phase after the initial baseline, Pa, may be quite
discovery, rather than testing of potent independent        small, making Pai = 0.33 a generally conservative
variables.                                                  estimate.
                                       A PROBABILISTIC MODEL                                                  359
                         Table 1                         the experiment during the intervention phase
Probabilities for reversal designs with N experimental   (A-B-A-B design) (Barlow and Hersen, 1973;
phases.                                                  Hersen and Barlow, 1976), the experimenter
                         Number of                       may want to include more than the two experi-
N         Type of Design Patternsa Probabilityb          mental phases. This is especially true where the
       DESIGNS WITH UNRESTRICTED RANGE                   dependent measure is at or near either a zero
 2         A-B-A                   9                     rate or the physical or biological maximum rate
 3         A-B-A-B                27          0.037      during baseline or after intervention. Likewise,
 4         A-B-A-B-A              81          0.012      the model suggests that more complex designs
 5         A-B-A-B-A-B           243          0.004
 6         A-B-A-B-A-B-A         729          0.001      (see Barlow and Hersen, 1973, or Hersen and
    DESIGNS WITH INITIAL BASELINE HAVING                 Barlow, 1976) should have good internal va-
                 RESTRICTED RANGE                        lidity.
 2         A-B-A                   6
 3         A-B-A-B                12
 4         A-B-A-B-A              36                            MULTIPLE-BASELINE DESIGNS
 5         A-B-A-B-A-B            72
 6         A-B-A-B-A-B-A         210                        Within multiple-baseline experiments, sev-
    DESIGNS WITH RESTRICTED RANGE AFTER                  eral baselines are formed and then the indepen-
                    INTERVENTION                         dent variables (intervention) are sequentially
 2        A-B-A                    6          0.167
 3        A-B-A-B                 18          0.056      introduced into the environments of each base-
 4        A-B-A-B-A               36          0.028      line. Treatment effects are demonstrated when
 5        A-B-A-B-A-B            108          0.009      each baseline changes in the desired direction
 6        A-B-A-B-A-B-A          216          0.005
                                                         only after the intervention. Kazdin and Kopel
   aNumber of Patterns = number of permutations of       (1975) stated that the internal validity of the
change that could occur due to random factors.
   bProbability that the expected pattern occurred due   multiple-baseline experiment is a function of a
to random factors (p = (1 . number of patterns).         number of factors, including the rapidity with
                                                         which the behaviors change in the desired di-
to a   A-B1-B2-A design of three experimental            rection, the variability in the data, and the num-
phases. However, such subdivision accentuates            ber of baselines used. Again, a probabilistic
within-subject variability, which may obscure            model can quantify some of these factors, dem-
ability to define whether the predicted outcome          onstrating the high validity of a well-con-
occurred or not. Within the model, it is primary         structed, multiple-baseline experiment and pro-
that results be distinguishable as either signifi-       viding specific recommendations regarding the
cant change from baseline level or reversal back         important issues of independence of baselines
toward baseline level.                                   and the number of baselines required. The
   On the basis of this model, the internal va-          model is derived from the basic process of the
lidity of reversal designs can be demonstrated.          multiple-baseline design. When an experi-
While the model concludes that the traditional           menter selects the order of introducing the
A-B-A design produces a probability estimate             independent variables into baselines, an a priori,
greater  than the generally accepted criterion           expected order of independent, sequential
of p < 0.05, this estimate may be conservative           changes in the baselines has been defined.6 If
(see Footnote 4) and does not take into consid-
eration that the data from many A-B-A experi-               6This requirement for a priori definition can run
                                                         counter to the logic of some intensive experiments,
ments could have been considered as A-B1-BW-A            since it restricts the flexibility of the experimenter to
or A-B-A1-A2 experiments. Nevertheless, in line          a predefined order of treatment. Hence, the model
with recommendations for direct replications             applies only to those designs wherein the order of
                                                         treatment can be flexible and not affect the intent
(A-B-A-B-A designs) and for termination of               of the experiment.
360                                      TERRY F. PECHACEK
                                                    Table 2
                                  Probabilities for Multiple-Baseline Designs
                                                         Number of Baselines
                        3                 4                 5                   6                   7
                                                SIMPLE CASEa
Number of Patternsb      6                 24             120                      720                5040
Probability         0.1670             0.0417          0.0083                   0.0014              0.0002
Number of Patterns         15                64            325                  1956  1.37 X 104
Probability           0.0667             0.0156         0.0031                0.0005 7.3 X 10-5
Number of Patterns         35               208           1233           2.10 X 104   1.00 X 105
Probability           0.0286             0.0048         0.0008           4.8 X 10-5  1.0 X 10-S
   aDerived with formula fi(n) = n!.
  bNumber of permutations of change that could occur due to random factors.
    Probability of predicted pattern occurring due to random factors.
   dDerived with Equation 1.
  eDerived with Equation 2.

the observed effects are inferred to be due to        Unfortunately, the multiple-baseline experi-
extraneous, random factors, then the observed      ment is somewhat more complex. First, the re-
order of change could have been any one of         quirement of independence of change across
several. If the basic assumption of this model,    baselines is difficult to achieve (Kazdin and
namely: that each observed behavior has an         Kopel, 1975). Often, the baselines are not truly
equal probability of not changing, changing in     independent, and change in one increases the
the undesired direction, or changing in the de-    probability of change in others. Moreover, the
sired direction at any time when measured in       simple formula expressed above only considers
the natural state,' is accepted, then the number   sequential, nonsimultaneous changes in base-
of possible orders of changes can be computed      lines. The reality of interdependence creates
by the formula, fi(n) - n!, where n is the num-    the potential for simultaneous changes (defined
ber of baselines in the design. When the experi-   as the case where two or more baselines display
menter can a priori define a single pattern of     significant changes in the desired direction the
expected results (see Footnote 6), the likelihood  intervention is introduced in only one of the
that this would occur due to random factors        baselines). When this aspect of the design is
would be 1- n!. Table 2 provides those proba-      incorporated into the model, the formula be-
bilities for increasing numbers (ns) of baselines. comes more complex. Equation 1 will compute
                                                   the number of possible orders of change under
  7While each behavior may not have an equal prob- these conditions.
ability of changing or not changing, the likelihood                             nf(n
that baselines will change in the desired direction is                                                  (1)
                                                                         f2()                   ,

of critical importance. Therefore, the experimenter                              -1    (n
                                                                                        (- )!
may need to select randomly the order of introducing
the intervention into baselines or, where that is not    Table 2 provides probability estimates for these
possible for therapeutic or experimental reasons, be-    more complex cases.
gin with the baseline with the least probability of
change and conclude with the baseline of highest            In the simple formula, f, = n!, and in Equa-
probability for desired change. The model is specifi-    tion 1, it is assumed that all baselines will be
cally inappropriate where the order of treatment is      observed to change during the experimental
selected to maximize the likelihood of producing the     analysis. In reality, one or more of these base-
desired results at each stage, since this approach
maximizes the interdependence of the baselines.          lines may in fact show no changes judged as
                                                A PROBABILISTIC MODEL                                              361

significant (see Footnote 2). Equation 2 com-                     provides a mathematical rationale supporting
putes the number of possible patterns of results                  the most common recommendations regarding
that could be observed under conditions that                      the need to use four or more baselines (Jeffrey,
allow for one or more baselines to change simul-                  1975; Kazdin and Kopel, 1975). In fact, the
taneously and/or one or more baselines not to                     model demonstrates the rapidly increasing
change significantly.                                             power of the design when more than four base-

                                                                  lines are used (see Table 2).
f3(n) = n +                                                          The model also enables the experimenter
               =                                                  to evaluate the less desirable pattern of results
       t                      t!

       t   2                                                      when interdependence of baselines produces
  {    >2E[
       r            )     ((t - )!)         +
                                                t!t               nonsequential or simultaneous change in two
                                                            (2)   or more baselines. Kazdin and Kopel (1975)
Table 2 provides the resulting probabilities                      discussed the problems in this area and the diffi-
for increasing numbers of baselines under these                   culty in predicting the sought interdependence
conditions.                                                       in baselines. However, the probabilistic model
   All probabilities in Table 2 are based on                      can quantify the situation of two baselines
the optimum outcome in a well-designed, mul-                      changing simultaneously and displaying inter-
tiple-baseline experiment, namely, an observed                    dependence and offers some estimates about
pattern of independent, sequential, and signifi-                  the relevance of the results. Equation 3 com-
cant changes across baselines in the exact order                  putes the number of ways that this type of pat-
in which the independent variables were intro-                    tern could occur with n baselines plus the single,
duced. When this does occur, even under condi-                    desired pattern of independent change across
tions where the baselines are somewhat inter-                     all baselines.
dependent, if the order of introducing the                    f4(n) = + ((n            1      (3)   2)!)

intervention into the baselines was not biased
toward the desired results (see Footnote 7),        The results of Equation 3 define the subsam-
the model would predict that possibly with three ple of patterns that, when compared to the re-
and more definitely with four baselines, plausi- sults from Equations 1 and 2, provide ratio
ble rival hypotheses for the results could be estimates of the number of baselines that will be
rejected with a p < 0.05. Therefore, the model needed in order to reject the influences of ex-
                                                   Table 3
                        Ratios with Designs Having Potential for Interdependent Baselines
                                                                    Number of Baselines
                                   3                    4               5                  6                   7
                                            RATIOS BASED ON EQUATION 1
Possible experimental
  patternsa                             6                   12            20                   30                  42
Total patternsb                        15                   64           325                 1956          1.37 X 104
Ratio                          0.4000             0.1875              0.0615               0.0153              0.0031
                                            RATIOS BASED ON EQUATION 2
Possible experimental
  patterns                       6                        12              20                  30                   42
Total patternsc                 35                       208            1233          2.10 X 104           1.00 X 105
Ratio                       0.1714                    0.0577          0.0162              0.0014               0.0004
  aDerived with Equation 3.
  bDerived with Equation 1.
  cDerived with Equation 2.
362                                    TERRY F. PECHACEK

traneous factors with an acceptable degree of         of the strength of the obtained results. How-
certainty should some pair of baselines display       ever, the model does not offer a solution to the
a degree of interdependence. Table 3 shows            controversy regarding the best method in which
these ratios for increasing numbers of baselines.     equivocal results should be analyzed. Rather,
In the interpretation of these results, a decision    the model is based on the assumption that
must be made regarding the appropriateness            interventions will produce clinically significant
of the ratios based on either Equation 1 or Equa-     results, with the significance defined by the
tion 2. The ratios based on Equation 2 require        experimenter or experiment as appropriate.
that all baselines display unequivocal change         Hopefully, the application of the model where
in the desired direction. Ratios based on Equa-       appropriate will help to support the legitimacy
tion 1 are valid as long as all baselines display     of single-subject designs and encourage their
a pattern of change distinctly related to the in-     application to even more diverse experimental
tervention, except for the dependent member of        questions.
the pair. Therefore, the model provides a mathe-
matical rationale for using five, six, or more                          REFERENCES
baselines depending on the experimenter's con-        Barlow, D. H. and Hersen, M. Single-case experi-
fidence in the expected results.                          mental designs: Uses in applied clinical research.
                                                          Archives of General Psychiatry, 1973, 29, 319-
    Overall, this probabilistic model demon-               325.
strates the high degree of internal validity that     Campbell, D. T. and Stanley, J. C. Experimental and
well-constructed, multiple-baseline experiments           quasi-experimental designs for research. Chicago:
can display. In fact the model predicts that as            Rand McNally, 1966.
                                                      Hersen, M. and Barlow, D. H. Single case experi-
few as three baselines are sufficient for an ex-          mental designs: strategies for studying behavior
periment with independent baselines displaying             change. New York: Pergamon Press, 1976.
sequential changes distinctly related to the in-      Jeffrey, D. B. Treatment evaluation issues in research
                                                           on addictive behaviors. Addictive Behaviors, 1975,
tervention. Moreover, the model permits the                1, 23-26.
evaluation of less obvious results, such as those     Kazdin, A. E. Methodological and assessment con-
that occur when a pair of baselines display                siderations in evaluating reinforcement programs
                                                           in applied settings. Journal of Applied Behavior
interdependent change; however, ensuring                   Analysis, 1973, 6, 517-531.
against this occurrence may require the planned       Kazdin, A. E. Statistical analyses for single-case
use of at least six baselines.                             experimental designs. In M. Hersen and D. H.
    In summary, where the experimenter is able             Barlow, (Eds), Single case experimental designs:
                                                           strategies for studying behavior change. New
to conform to the a priori definition process              York: Pergamon Press, 1976.
outlined in the model, and when the obtained          Kazdin, A. E. and Kopel, S. A. On resolving am-
results are judged sufficiently definitive and             biguities of the multiple-baseline design: Prob-
                                                           lems and recommendations. Behavior Therapy,
unequivocal, the probabilistic model provides               1975, 6, 601-608.
an estimation of internal validity for either a       Revusky, S. H. Some statistical treatments compati-
reversal or multiple-baseline type intensive ex-           ble with individual organism methodology. Jour-
                                                           nal of Experimental Analysis of Behavior, 1967,
periment. In a process similar to inferential              10, 319-330.
statistics, plausible, rival hypotheses are com-      Sidman, M. Tactics of scientific research. New York:
bined into a single null hypothesis that can be            Basic, 1960.
 rejected with a specific limit of confidence. In     Received 18 November 1975.
 this process, the model provides specific evidence   (Final Acceptance 3 February 1978.)

Description: dependent-and-independent-variable pdf