# Experimental design

Document Sample

```					Statistics 203: Introduction to Regression
and Analysis of Variance
Experimental design

Jonathan Taylor

- p. 1/20
Today

q Today
q Why is design important?
q What makes a good
experiment?
s   Power.
q Power in a multivariate setting
q Power example: one-way            s   Classical designs.
ANOVA
q Determining sample size:
power
s   Criteria of optimality.
q Scheffé’s procedure
q Determining sample size: CIs
q Classical designs
q Randomized design
q Randomized block design
q Nested designs
q Nested design: ANOVA table
q Latin square
q Latin square ANOVA table

q2
k   factorial designs
q Fractional design: example
q Fractional design: example
q Design criteria

- p. 2/20
Why is design important?

q Today
q Why is design important?
s   Entire courses based just on design: only a brief overview
q What makes a good
experiment?
today.
q Power in a multivariate setting
q Power example: one-way
s   Industrial experiments. Often each trial can be very
ANOVA
q Determining sample size:              expensive: imagine modelling crash test data for Jaguar....
power
q Scheffé’s procedure
q Determining sample size: CIs
s   Clinical trials. To rule out unexpected selection bias, it is
q Classical designs                     important to sufﬁciently randomize the study, perhaps
q Randomized design
q Randomized block design               keeping some control over heterogeneity.
q Nested designs
q Nested design: ANOVA table        s   A topic of active research: ﬁeld has progressed well beyond
q Latin square
q Latin square ANOVA table              its origins in agricultural ﬁeld trials. Modern response surface
q2
k   factorial designs
q Fractional design: example
design problems are used frequently in industry.
q Fractional design: example
q Design criteria
s   Current examples: CFD (Computational Fluid Dynamics)
models take ages to run on a computer and one might want
to understand how the model depends on the parameters
input into it: try a few values of the parameters and
“interpolate” (i.e. run a regression).

- p. 3/20
What makes a good experiment?

q Today
q Why is design important?
s   Is the design subject to unforeseen (selection) bias?
q What makes a good
experiment?
s   Is it powerful enough to detect a given effect size?
q Power in a multivariate setting
q Power example: one-way            s   Is the precision of a certain number of estimators sufﬁcient?
ANOVA
q Determining sample size:
power
i.e. if we assume we know σ 2 , can we get a tight enough
q Scheffé’s procedure
q Determining sample size: CIs
conﬁdence interval for β1 , say?
q Classical designs
q Randomized design
s   Multivariate generalizations: D-optimality, V -optimality.
q Randomized block design
q Nested designs
q Nested design: ANOVA table
q Latin square
q Latin square ANOVA table

q2
k   factorial designs
q Fractional design: example
q Fractional design: example
q Design criteria

- p. 4/20
Power in a multivariate setting

q Today
q Why is design important?
s   To talk about power in a multivariate setting, one needs to
q What makes a good
experiment?
know about non-central χ2 , F, t.
q Power in a multivariate setting
q Power example: one-way
s   Non-central χ2 : suppose Z ∼ N (µ, I) ∈ Rk . Then
ANOVA
q Determining sample size:
2
power
q Scheffé’s procedure
Z       ∼ χ2 ( µ 2 ).
k
q Determining sample size: CIs
q Classical designs
q Randomized design
and µ 2 is called the non-centrality parameter: 0
q Randomized block design
q Nested designs
corresponds to usual χ2 .
k
q Nested design: ANOVA table
q Latin square
s   Non-central F : G1 ∼ χ21 (h), G2 ∼ χ22 (0) then
ν             ν
q Latin square ANOVA table
k
q2       factorial designs
G1 /ν1
q Fractional design: example
q Fractional design: example
F =          ∼ Fν1 ,ν2 (h).
q Design criteria
G2 /ν2
s   Non-central tk (h): “square root” of F1,k (h2 ).

- p. 5/20
Power example: one-way ANOVA

q Today                                  Source                     SS                      df              E(M S)
q Why is design important?                                                                                  Pr        2
q What makes a good                 s    Treatments   SST R =
Pr       “
ni Y i· − Y ··
”2
r−1        σ 2 +   i=1 ni αi
i=1                                             r−1
experiment?                                                     Pn i
− Y i· )2                         σ2
Pr                          Pr
Error        SSE =             (Y               i=1 ni − r
q Power in a multivariate setting                             i=1   j=1 ij
q Power example: one-way
ANOVA
q Determining sample size:
s   Power for testing H0 : α1 = · · · = αr = 0. Under
power
q Scheffé’s procedure
H a : r ni α i = h
i=1
2
q Determining sample size: CIs
q Classical designs
M ST R
q Randomized design
∼ FdfT R ,dfE (φ)
q Randomized block design
q Nested designs
M SE
q Nested design: ANOVA table
q Latin square                          where
q Latin square ANOVA table                                                 2
k                                                                  ni α i
i         h
q2     factorial designs
φ=           2
=          2
.
q Fractional design: example
q Fractional design: example
(r − 1)σ      (r − 1)σ
q Design criteria
Therefore, if we reject at level α
Power = 1 − PdfT R ,dfE ,φ (FdfT R ,dfE ,1−α ).
where Pν1 ,ν2 ,φ is the distribution function of Fν1 ,ν2 (φ)

- p. 6/20
Determining sample size: power

q Today
q Why is design important?
s   Suppose that ni = n, then the non-centrality parameter
q What makes a good
n       ˜
φ > (max αi − min αi )2
experiment?
q Power in a multivariate setting
2
=φ
q Power example: one-way                                  i         i        (r − 1)σ
ANOVA
q Determining sample size:
power
q Scheffé’s procedure
s   As power is monotone increasing in φ, it is possible to just
q Determining sample size: CIs                                                                   ˜
specify this “range” of α’s relative to σ in the form of φ.
q Classical designs
q Randomized design
q Randomized block design
s   To determine sample size for a given power 1 − β and Type I
q Nested designs
q Nested design: ANOVA table
error rate α we would plot Power as a function of sample
q Latin square
q Latin square ANOVA table
size and take the smallest sample size that yields sufﬁcient
q2
k   factorial designs              power.
q Fractional design: example
q Fractional design: example
q Design criteria
s   Here is the example.

- p. 7/20
Scheffé’s procedure

q Today
q Why is design important?
s   We saw Bonferroni correction for simultaneous inference for
q What makes a good
experiment?
residuals.
q Power in a multivariate setting
q Power example: one-way
s   Suppose we wanted conﬁdence intervals for many contrasts
ANOVA
q Determining sample size:
p
power
q Scheffé’s procedure
q Determining sample size: CIs
a j βj .
q Classical designs
j=0
q Randomized design
q Randomized block design
q Nested designs
q Nested design: ANOVA table
For instance, all pairwise differences of main effects in a
q Latin square                          two-way ANOVA model.
q Latin square ANOVA table

q2
k   factorial designs          s   We can use Bonferroni, but as the number of contrasts grows
q Fractional design: example
q Fractional design: example            intervals get wider even though they are based on only p
q Design criteria
random variables: cannot use this to get conﬁdence bands.
s   Scheffé’s procedure: conﬁdence interval for each contrast is
         
p                    p
aj βj ± SE          a j βj    pFp,n−p,1−α .
j=0                  j=0

- p. 8/20
Determining sample size: CIs

q Today
q Why is design important?
s   Suppose that there is a set of k contrasts that we wish to
q What makes a good
experiment?
estimate and each one has a pre-speciﬁed target width wi .
q Power in a multivariate setting
q Power example: one-way
s   Scheffé’s procedure at level α tells us that the CI for the j-th
ANOVA
q Determining sample size:              contrast of interest
power                                                             r
q Scheffé’s procedure
q Determining sample size: CIs                                           ai,j µj
q Classical designs
q Randomized design                                                j=1
q Randomized block design
q Nested designs                        has width
q Nested design: ANOVA table
q Latin square
r
q Latin square ANOVA table
k
a2
i,j
q2     factorial designs
Wi = 2    σ 2 · (rFr,(n−1)r,1−α )
q Fractional design: example
q Fractional design: example
j=1
n
q Design criteria

s   Can replace rFr,(n−1)r,1−α by Bonferroni correction
t2
(n−1)r,1−α/k .
s   Choose n large enough so that Wi ≤ wi , 1 ≤ i ≤ k.

- p. 9/20
Classical designs

q Today
q Why is design important?
s   Randomized.
q What makes a good
experiment?
s   Randomized complete block.
q Power in a multivariate setting
q Power example: one-way            s   Nested.
ANOVA
q Determining sample size:
power
s   Repeated measures – will come later in random effects.
q Scheffé’s procedure
q Determining sample size: CIs      s   Latin square.
q Classical designs
q Randomized design
q Randomized block design
s   Factorial / fractional factorial.
q Nested designs
q Nested design: ANOVA table
q Latin square
q Latin square ANOVA table

q2
k   factorial designs
q Fractional design: example
q Fractional design: example
q Design criteria

- p. 10/20
Randomized design

q Today
q Why is design important?
s   This design controls selection bias in the experiment: i.e. by
q What makes a good
experiment?
assigning “ﬁtter” people to treatment vs. control, looks like
q Power in a multivariate setting
q Power example: one-way
treatment is more effective than it is.
ANOVA
q Determining sample size:
s   Given r treatments, and nr subjects, assign subjects to
power
q Scheffé’s procedure                   treatment are random.
q Determining sample size: CIs
q Classical designs                 s   Assumes implicitly that all subjects are identical – no
q Randomized design
q Randomized block design               “controlling” for variables such as gender, age, etc.
q Nested designs
q Nested design: ANOVA table        s   Reduces to a one-way ANOVA model for the treatment
q Latin square
q Latin square ANOVA table              effects.
q2
k   factorial designs
q Fractional design: example
q Fractional design: example
q Design criteria
Yij = µ· + αi + εij , 1 ≤ i ≤ r, 1 ≤ j ≤ n
(with usual constraints)

- p. 11/20
Randomized block design

q Today
q Why is design important?
s   If subjects are heterogeneous then some of the variance σ 2
q What makes a good
experiment?
can be attributed to this heterogeneity.
q Power in a multivariate setting
q Power example: one-way
s   One may “block” subjects into n homogeneous groups and
ANOVA
q Determining sample size:              randomize the r treatments within each block.
power
q Scheffé’s procedure
q Determining sample size: CIs
s   Reduces to a two-way ANOVA model for the block and
q Classical designs                     treatment effects with no interactions.
q Randomized design
q Randomized block design
q Nested designs
q Nested design: ANOVA table                                    Yij = µ·· + ρi + τj + εij
q Latin square
q Latin square ANOVA table

q2
k   factorial designs
(with usual constraints)
q Fractional design: example                               SS                                df              E(M S)
q Fractional design: example                                                                                     P    2
2 + r i ρi
P “           ”2
q Design criteria                               SSBL = r i Y i· − Y ··                      n−1          σ
s                                                                              r−1
τ2
P
”2
P “
SST R = n j Y ·j − Y ··                     r−1          σ 2 +n j j
”2                              r−1
2
P   “
SSBL.T R = r i,j Yij − Y i· − Y ·j + Y ··      (n − 1)(r − 1)         σ

- p. 12/20
Nested designs

q Today
q Why is design important?
s   Example: suppose we are studying the performance of
q What makes a good
experiment?
different schools on standardized tests based on the
q Power in a multivariate setting
q Power example: one-way
performance of classes within the schools.
ANOVA
q Determining sample size:
s   Each school 1 ≤ i ≤ a has b classes taking the tests, of
power
q Scheffé’s procedure                   which each class had a different teacher.
q Determining sample size: CIs
q Classical designs                 s   It is natural to think of “school” effect and “teacher” effect, but
q Randomized design
q Randomized block design               the teachers taught only within one school: they are nested
q Nested designs
q Nested design: ANOVA table            within schools. (Perhaps better to treat this as random ...)
q Latin square
q Latin square ANOVA table

q2
k   factorial designs
q Fractional design: example
q Fractional design: example
q Design criteria

- p. 13/20
Nested design: ANOVA table

q Today
q Why is design important?
s   Model: like a two-way ANOVA model
q What makes a good
experiment?
q Power in a multivariate setting
Yijk = µ·· + αi + βj(i) + εijk
q Power example: one-way
ANOVA
q Determining sample size:              with 1 ≤ i ≤ a, 1 ≤ j ≤ b, 1 ≤ k ≤ n.
power
q Scheffé’s procedure                   (with usual constraints)
q Determining sample size: CIs
q Classical designs
q Randomized design
s   Note the βj(i) ’s are not “shared” across schools and can only
q Randomized block design               be estimated within a given school.
q Nested designs
q Nested design: ANOVA table                           SS                       df             E(M S)
q Latin square                                                                                      P   2
2 + bn i αi
P “              ”2
q Latin square ANOVA table                 SSA = bn i Y i·· − Y ···            a−1         σ
k                              s                                                            P a−1
q2       factorial designs                                                                            β2
2 + n i,j j(i)
P      “            ”2
q Fractional design: example             SSB(A) = n i,j Y ij· − Y i··        a(b − 1)    σ
”2                          a(b−1)
q Fractional design: example
σ2
P       “
SSE =   i,j,k   Yijk − Y ij·       ab(n − 1)
q Design criteria

- p. 14/20
Latin square

q Today
q Why is design important?
s   r treatments: two blocking variables:             each block gets all r
q What makes a good
experiment?
treatments
q Power in a multivariate setting
q Power example: one-way
s   Example:
ANOVA
q Determining sample size:                                          Time
power
q Scheffé’s procedure                                                A B                    C
q Determining sample size: CIs
q Classical designs                                         Subject B C                     A
q Randomized design
q Randomized block design
q Nested designs
C A                    B
q Nested design: ANOVA table
q Latin square                      s   Model
q Latin square ANOVA table
k
q2       factorial designs
q Fractional design: example
Yijk = µ... + ρi + κj + τk + εijk                  1 ≤ i, j, k ≤ r
q Fractional design: example
q Design criteria
but only r 2 observations.
s   Similar to two-way ANOVA model with no interactions, one
replication per cell.
s

1                        1                        1
Y i··   =         Yijk , Y ·j·   =         Yijk , Y ··k   =           Yijk .
r   j
r   i
r   i,j            - p. 15/20
Latin square ANOVA table

q Today
q Why is design important?
s   Predicted values
q What makes a good
experiment?
q Power in a multivariate setting
Yijk = Y i·· + Y ·j· + Y ··k − 2Y ···
q Power example: one-way
ANOVA
q Determining sample size:          s
power
q Scheffé’s procedure                                 SS                                    df            E(M S)
q Determining sample size: CIs
2                               ρ2
P
2
q Classical designs
q Randomized design
SSROW = r         i   Y i·· − Y ···               r−1          σ + r r−1i
i
P 2
q Randomized block design                                                      2                          2      j κj
q Nested designs                         SSCOL = r      j      Y ·j· − Y ···               r−1          σ + r r−1
q Nested design: ANOVA table                                                                                   P 2
2                                 kτ
q Latin square
q Latin square ANOVA table
SST R = r     k Y ··k − Y ···                    r−1          σ 2 + r r−1k
q2
k   factorial designs
q Fractional design: example
SSRem =      i,j,k (Yijk − Yijk )2            (r − 1)(r − 2)        σ 2
q Fractional design: example
q Design criteria

- p. 16/20
2k factorial designs

q Today
q Why is design important?
s   Given k factors with 2 levels each there are 2k possible
q What makes a good
experiment?
combinations. Designs including all levels are called 2k
q Power in a multivariate setting
q Power example: one-way
factorial designs.
ANOVA
q Determining sample size:          s   To estimate all of them (with replications) becomes quite
power
q Scheffé’s procedure                   “expensive”.
q Determining sample size: CIs
q Classical designs
q Randomized design
s   Most of the degrees of freedom goes to estimating higher
q Randomized block design               order interactions (which may be less of interest).
q Nested designs
q Nested design: ANOVA table
q Latin square
s   A 2k−f design is a design that drops some combinations in
q Latin square ANOVA table
k
the interest of “cost”, but introduces some confounding to the
q2       factorial designs
q Fractional design: example            model.
q Fractional design: example
q Design criteria                   s   For instance if f = 1 then the experimenter only uses half of
the combinations. This means that it may be impossible to
“separate” some low order interactions effects with higher
order interactions (i.e. they will be confounded)
s   Which effects are confounded depends on the deﬁning
relation of the fractional study. Good deﬁning relations leave
as many low order interactions (including main effects)
- p. 17/20
estimable.
Fractional design: example

q Today
q Why is design important?
s   Three factors:   full design
q What makes a good
experiment?                                                                               
q Power in a multivariate setting
1     1     1       1    1   1     1    1
q Power example: one-way
                                           
ANOVA
q Determining sample size:                        1      1     1      −1    1   −1   −1   −1
power                                                                                      
q Scheffé’s procedure                             1      1    −1       1   −1   1    −1   −1
q Determining sample size: CIs                                                               
q Classical designs                               1      1    −1      −1   −1   −1   1    1
                                           
q Randomized design
                                           
q Randomized block design
1     −1     1       1   −1   −1    1   −1
q Nested designs
                                           
q Nested design: ANOVA table
1     −1     1      −1   −1   1    −1   1
q Latin square
                                           
q Latin square ANOVA table
                                           
q2
k   factorial designs                        1     −1    −1       1    1   −1   −1   1
q Fractional design: example
q Fractional design: example                        1    −1    −1      −1   1    1     1   −1
q Design criteria

- p. 18/20
Fractional design: example

q Today
q Why is design important?
s   Three factors: half design
q What makes a good
experiment?                                                               
q Power in a multivariate setting
q Power example: one-way
1    1 1   1  1    1  1  1
ANOVA                                           1    1 1 −1 1     −1 −1 −1
q Determining sample size:                                                   
power                                                                      
q Scheffé’s procedure                             1    1 −1 1 −1     1 −1 −1
q Determining sample size: CIs
q Classical designs
q Randomized design
1    1 −1 −1 −1   −1 1  1
q Randomized block design
q Nested designs
q Nested design: ANOVA table
q Latin square
q Latin square ANOVA table

q2
k   factorial designs
q Fractional design: example
q Fractional design: example
q Design criteria

- p. 19/20
Design criteria

q Today
q Why is design important?
s   In incomplete fractional designs, it gets very tricky to sort out
q What makes a good
experiment?
what is confounded with what.
q Power in a multivariate setting
q Power example: one-way
s   Need for general criteria to compare designs.
ANOVA
q Determining sample size:
power
s   D-criterion
q Scheffé’s procedure
q Determining sample size: CIs
D = det(X t X)
q Classical designs
q Randomized design                 s   V -criterion: given a collection {X1 , . . . , Xk } of “points of
q Randomized block design
q Nested designs                        interest” in the predictor space
q Nested design: ANOVA table
q Latin square
k
q Latin square ANOVA table                                      1
q2
k   factorial designs                                  V =                         t
Xi (X t X)−1 Xi .
q Fractional design: example                                    k   i=1
q Fractional design: example
q Design criteria

- p. 20/20

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 2 posted: 12/12/2011 language: English pages: 20
How are you planning on using Docstoc?