Experimental Designs
AP Statistics Presentation 3.8
Simple Experiments
• With the simple experiment, bias often rears its
ugly head.
• Consider the scenario:
– You are testing a new medication that claims to cure
migraines. In order to run your experiment on
humans, you must disclose that you are testing a new
medication. Let’s assume the medication does not
work at all. What are the chances that some of your
subjects will believe (by simply convincing
themselves) that the drug works and report that their
suffering is lessened?
• This is known as the Placebo Effect.
Simple vs. Comparative
Experiments
• Due to the Placebo Effect, we need a comparison.
• In order to avoid the bias, we do the following:
– Give half of the patients a sugar pill (takes, looks, feels, smells
like the real pill but contains no medication) and the other half
the real medication. The patients do not know which pill they are
receiving (this is referred to as blinding as they cannot ‘see’
which treatment they are receiving). Then compare the results of
the sugar pill (this fake treatment is called a placebo as it
accounts for the placebo effect) and the actual medication.
• We are therefore comparing a fake pill to the actual
medication.
• The group receiving the placebo is called the control
group.
• The group receiving the actual medication is called the
experimental group.
More Bias??
• Now that we are conducting comparative experiments,
could bias still be present?
• Consider the scenario:
– In an experiment to test the effectiveness of a new diet pill, the
experimenters separate subjects into an experimental group and
a control group (which will receive a placebo). They deviously or
perhaps unintentionally place the most overweight subjects into
the experimental group. The results of the experiment, no
surprise, show that the subjects who used the diet pill lost much
more weight than those who used the placebo.
• Of course, they had much more weight that they could
lose in the first place.
• To avoid such bias, an experiment must be completely
randomized.
More Bias??
• Now that we are completely randomized, could bias still
be present?
• Let’s continue with the diet pill scenario:
– In an experiment to test the effectiveness of a new diet pill, the
experimenters separate subjects into an experimental group and
a control group (which will receive a placebo). They
unintentionally end up with nearly all the female subjects in the
experimental group and nearly all males in the control group. Are
the results of the experiment necessarily reliable? Could there
be differences between the genders with regards to dieting?
• To avoid such bias, an experiment may need to be
blocked.
Blocking in Experiments
• Blocking occurs in order to account for different
variables.
• Still using the diet pill scenario:
– In an experiment to test the effectiveness of a new diet pill, the
experimenters separate subjects into an experimental group and
a control group (which will receive a placebo). In order to
account for differences between males and females, they first
separate their patients into two groups (male and female). They
then take the male group and randomly split them. One of these
groups of males takes the treatment the other the placebo. They
then take the female group and randomly split them. One of the
groups of females takes the treatment the other the placebo.
Comparisons in data are then made.
• This is an example of an experiment that has been
blocked on gender.
Blocking in Experiments
• Blocking is frequently done to account for suspected differences in
the experimental units or subjects.
• If you conduct an experiment on the effectiveness of a television
advertising campaign, you might consider blocking by:
– Age group (do the ads appeal equally or are they more effective only to
certain ages?)
– Gender (are the ads more effective with men or women?)
– Socioeconomic Status (are the ads more effective with wealthy vs.
middle class?)
– Race (are the ads more effective with different cultures or ethnic
background?)
– Geographic region (are the ads more effective on the west coast vs. the
east coast vs. the south etc?)
– Political preferences, educational background, occupation, or any other
variable which may have an impact on results.
Matching in Experiments
• Matching (or conducting an experiment using matched pairs) is
another common strategy in experimental design.
• Matching Example #1
– Before and After cases.
• Many people believe that taking an SAT prep course can significantly boost
scores. This is a classic opportunity to run a matched pairs experiment. In a
matched pairs experiment, each subject may (or may not) act as their own
control. Each subject would first take the SAT, then take the course,
followed by again taking the SAT. The difference in their scores would
constitute the data used in the analysis. You could (should) also set up a
separate control group that would simply take the SAT a second time without
having taken the course.
– In this example, each person is their own control (as you have data both
with and without the treatment).
– In addition, you have a control group to account for the fact that scores
may increase on the second try just because the subjects are seeing
the test for a second time.
Matching in Experiments
• Matching Example #2
– Matched Pairs Experiment
• Consider an experiment in which you are testing
out a new sinus medication. You typically have a
very limited number of subjects to work with. Let’s
presume you have 20 subjects. You examine
relevant variables (age, gender, etc.) and match up
similar patients in pairs. Then, for each pair,
randomly assign one subject to the treatment and
the other to the control (placebo).
Randomized Design
• Regardless of whether you employ
blocking, matching, placebo, or run the
simplest of experiments, you must
ALWAYS USE RANDOMIZATION in your
design.
• Blocking should be used to control the
variables you know about, randomization
controls for the variables you don’t know
about.
Experimental Design Layout
Treatment
Group #1
(Block A)
Measure Variable (s)
Treatment
Randomly Allocate
Group #2
(Block B) Compare
Experimental Units Results
(Subjects)
Treatment
Group #3
(Block C)
Control
Group
Principles of Experimental Design
• CONTROL
– Control the effects of lurking variables on the
response (through comparison, blocking, placebos,
etc.)
• RANDOMIZATION
– The use of random chance to assign subjects to
treatments (controls unknown variables)
• REPLICATION
– Use a substantial number of subjects in order to
reduce chance variation
Experiments
• This concludes this presentation.