The Basics of Study Design
Barry Braun, PhD, FACSM
Associate Professor H
CH2OH
H
Director, Energy Metabolism Laboratory OH
OH
H
OH
H OH
Department of Kinesiology
University of Massachusetts
Amherst, MA
Barry Braun, Ph.D. Basics of Study Design
A fairy tale
While boardsailing in Belize, physician/
scientist Dr. Dulcinea Toboso gets hit on the head by
her mast and knocked unconscious. She wakes up in
a hut where she is cared for by a tribe of people who
share a remarkable characteristic; every person is
lean and toned, even though they eat massive meals
and do absolutely no exercise. They tell her the
secret is the bark of a rare tree that only grows in the
misty cloud forests that hide the interior of the island.
The bark smells like elephant feces and somehow,
tastes even worse.
Barry Braun, Ph.D. Basics of Study Design
Though it is strictly forbidden, Dr.
Toboso leaves with several
kilograms of bark hidden in her
bathing suit. She
flies to San Francisco and heads to her laboratory
to isolate the active ingredient, which she plans to
market as "Bark-a-lounge", a dietary supplement
designed to cause fat loss and muscle growth
without any need for exercise. As a conscientious
scientist, she decides to do a research study to
show how well it works. She writes the study
design on her prescription pad and orders her
long-suffering assistant to do the following study:
Barry Braun, Ph.D. Basics of Study Design
A group of 12 men she knows from her gym will
participate in the study. They will weigh themselves
at home and then come to the laboratory so their
body fat can be measured using skin fold calipers.
Then they will do as many pushup and situps as
they possibly can. They will be given 30 doses of
"Bark-a-Lounge" in pill form and told to
take 2 per day for about 15 days. Then, they
will re-weigh themselves, come back to the lab to
have body fat re-measured and do as many
pushups and situps as possible. Dr. Toboso is sure
that the men will lose fat but gain strength after
taking "Bark-a-Lounge" for 15 days.
Barry Braun, Ph.D. Basics of Study Design
Objective
Although we have to give Dr. Toboso credit for
even considering actually subjecting her product to
scientific testing, many of you recognize that her
study design is not optimal. The overall goal of this
lecture is to allow you to recognize the strengths
and the flaws in published studies and media
reports. If you plan to conduct your own studies, this
lecture will aid you in designing them in a way that
maximizes their contribution to the body of scientific
knowledge that is used to enhance the performance
of athletes and the health of the general public.
Barry Braun, Ph.D. Basics of Study Design
Plan of attack
Part 1: “True Lies”
What kind of study? Epidemiology vs. experiment;
cross sectional vs. longitudinal, association and
causality, validity and reliability
Part 2: “Of Mice and (Wo)Men”:
Humans, animals or cells? Controlling
confounding variables vs. real world application.
Barry Braun, Ph.D. Basics of Study Design
More plan of attack
Part 3: “Sub-divide and conquer”
How do you attack big important questions?
One big study or many small ones?
Part 4: “The Color of Money”
Can the funding source affect the study
design? The results?
Part 5: “You can’t always get what you want”
All studies have flaws. Why continue to do them?
Barry Braun, Ph.D. Basics of Study Design
Some useful terms
Subjects: participants in a study (usually only
used when participants are human)
Variable: Something that can be measured.
Independent variables are controlled by the
investigator (research scientist). Dependent
variables are not.
Treatment: What subjects are “exposed” to. Also
called exposure or condition.
Barry Braun, Ph.D. Basics of Study Design
Outcomes: The dependent variables. The
answers to the question you are interested in.
Control group or condition: What the treatment or
exposure is compared with. Can be the initial state
(baseline) or can be a group that is either given no
treatment or a non-functional placebo.
Relative to starting weight (baseline), what is
effect on body weight (outcome) when I give 100
people (subjects) three pints of ice cream per day
for 6 months (treatment) as compared with 100
people who get no ice cream (control group)?
Barry Braun, Ph.D. Basics of Study Design
Epidemiological Studies
One or more characteristics of a
population (e.g. weight or blood lipids or
dietary habits) are assessed (usually by using
questionnaires but other techniques used as well).
Subjects are not asked to change behavior or
subjected to treatments like exercise or diet change.
Researchers do not control the experimental
conditions; they are trying to understand behavior or
physiology or metabolism in a “natural” setting.
Barry Braun, Ph.D. Basics of Study Design
Cross Sectional
The variables of interest are measured once. E.g.,
survey 600 subjects (300 W and 300 M) and
measure height. Exposure is gender and the
outcome is height.
Mean (average) height for men = 175 cm
Mean height for women = 165 cm
Based on your data, you might conclude that men
are taller than women.
Barry Braun, Ph.D. Basics of Study Design
Note that EVERY man was not taller than
EVERY woman. There is a lot of variation in
human height (let’s say men in your sample
ranged from 155-195 cm and women from
148-185 cm).
But the average or mean height for men (175 cm) is
greater than the mean height for women (165 cm).
148 165 175 195
Barry Braun, Ph.D. Basics of Study Design
Because there is so much variation in height within
each gender (about 30 cm in your sample)
compared to the mean DIFFERENCE in height
(only 10 cm), you need to study a lot of subjects to
see a difference between men and women that
accurately represents the population.
Barry Braun, Ph.D. Basics of Study Design
Although very useful to illustrate a relationship
between exposures and outcomes, a problem with
observational studies is that you often can’t
determine if the exposure caused the outcome.
Let’s say you are interested in whether doing a lot of
aerobic exercise lowers the risk for getting cancer;
in particular, skin cancer. You send out surveys to
hundreds of people asking about their exercise
habits and whether they had skin cancer. This is a
case-control study; it compares people who got a
disease (“cases”) with those who didn’t (“controls”).
Barry Braun, Ph.D. Basics of Study Design
Retrospective studies
You could do this study “retrospectively”, that is,
you could look through medical records, find cases
of skin cancer, and mail surveys to the people you
identified asking them about their exercise habits.
The downside to this approach is that you depend
on people’s memory of their past habits. You might
minimize this problem by having people mail you
their training diaries but many will be non-existent or
incomplete and you have no way to determine
whether or not they are accurate.
Barry Braun, Ph.D. Basics of Study Design
Prospective studies
You can also do this study prospectively. You start
with a group of individuals who DON’T have the
disease and track them for some period of time.
Then, you look for differences between people
who got the disease vs. those who didn’t.
You might randomly contact 5000 people from the
phone book and assess their exercise habits every
year. At the end of 5 years you would see who got
skin cancer and if there was a relationship between
time spent exercising and a diagnosis of skin cancer.
Barry Braun, Ph.D. Basics of Study Design
The advantage of a prospective design is that the
subjects are followed “longitudinally”, that is; over
time; rather than cross-sectionally; which only
gives a single “snapshot” at one time point.
But to get meaningful comparisons you need to
have a fairly large number of people who get the
disease so that you can separate them into groups
that differ by exercise habits. And some of the
subjects will move away or lose interest over time.
So to get accurate results often requires recruiting
and tracking thousands of people for multiple years.
Barry Braun, Ph.D. Basics of Study Design
Questions and answers
Lets say that your results show that people who run
and cycle and swim > 20 hours/week have higher
rates of skin cancer than people who don’t exercise
at all. Can you conclude that triathlon training
causes skin cancer? Alert the media!
Most triathletes spend an enormous amount
of time outdoors with a lot of skin exposure
to the sun. So is it exercise that causes more skin
cancer or is it more exposure to UV radiation from
the sun. Unless you collected data on sun exposure
in your survey, you would have no way to know
Barry Braun, Ph.D. Basics of Study Design
Isolating the outcome of interest
With enough subjects and enough information
there are statistical methods to “separate” the
key variables. E.g., if you had good data on both
exercise habits and sun exposure you would see
that if you “remove” or factor out the sun exposure
variable, there is no longer any association
between exercise habits and skin cancer. So it is
sun exposure, not exercise, that increases the risk
for skin cancer.
Barry Braun, Ph.D. Basics of Study Design
Take another example. Let’s say you want to test the
hypothesis that a high intake of fat increases the risk
for heart disease. You would need to:
1. accurately identify the men and women in
the population who get heart disease
2. accurately assess how much fat is in the
diet of each person
3. compare dietary fat in people who get heart
disease with dietary fat in people who don’t
Barry Braun, Ph.D. Basics of Study Design
get heart disease
% of people who
0 20 40 60 80
10 30 50 70
dietary fat as a % of total kilocalories
This graph (I made it up) says that the number
of people who get heart disease increases as
the amount of fat in the diet increases.
What are potential problems with this story? Well, did
we measure what we thought we were measuring?
Barry Braun, Ph.D. Basics of Study Design
Validity
Validity refers to the accuracy or truthfulness of a
measurement. In other words, are you actually
measuring what you think you are measuring?
This can be obvious (using a body weight scale to
measure body fat), less obvious (are lower blood
lipids after starting exercise training due to training
or accompanying weight loss?) or very subtle (do
athletes perform better when given carbohydrate
during exercise because the sugar does something
directly or because they think they should do better
when given carbohydrate?)
Barry Braun, Ph.D. Basics of Study Design
Measuring physical activity
Activity monitors are a good example of how
difficult it can be to develop tools that yield valid
measurements of physical activity. There are
many types of activity monitors available;
pedometers, accelerometers, etc.
If you are a scientist interested in accurately
measuring daily physical activity how valid are
these tools?
Barry Braun, Ph.D. Basics of Study Design
For example, you decide that collecting physical
activity information using questionnaires is too
subjective and prone to bias so you decide to
measure it objectively using an activity monitor
that is worn on the hip and is sensitive to motion.
You give the accelerometers to 20 people and
measure their activity for 7 days to assess their
physical activity. 10 of your subjects
are world class cyclists and 10 are typical college
students. After 7 days your measurements indicate
the college students are more active than the elite
cyclists! How can this be?
Barry Braun, Ph.D. Basics of Study Design
Since the activity monitor only measures
movement in the vertical plane, the 600 miles each
of your cyclists covered during the week on their
bicycles was not detected as movement by the
monitor.
This is an extreme case but researchers
are constantly forced to consider “am I
really measuring what I need to measure?”.
Barry Braun, Ph.D. Basics of Study Design
What do your subjects eat?
One of the most common measurements
attempted in Sport Nutrition is diet analysis. It
seems straightforward; you collect information
from subjects about what they eat over the course
of a few days and enter the foods into a database
which spits out grams of carbohydrate and protein
and thiamine and iron and vitamin C, etc.
In reality, the measurement is fraught with
potential inaccuracy.
Barry Braun, Ph.D. Basics of Study Design
Sources of potential error
How do you account for portion size? Estimate
based on showing the subjects plastic food models
before you start the study? Have them weigh their
food? Better but they have to carry their scales
everywhere with them. What about combination
foods? How do they tell you ingredients and
portion sizes of the seafood paella they had at
their best friends wedding? And how do you know
they are remembering to report
everything they ate?
Barry Braun, Ph.D. Basics of Study Design
And the process of having to weigh their food and
write everything down changes their typical behavior.
People avoid foods that are difficult to record
accurately and start choosing easy things like
prepackaged foods that are conveniently labeled.
Diet records are often inaccurate even in
the hands of experienced users. Many subjects
under-report their actual food intake by hundreds of
kilojoules/day. In contrast , women with eating
disorders may OVER-report actual food intake.
Barry Braun, Ph.D. Basics of Study Design
Internal Validity
Chance: what is the chance that the outcome you
observe could occur even with NO association
between the exposure and outcome you measure?
Measured statistically and reported as a “p-value”
showing probability of obtaining the result by chance.
Commonly define p-value <.05 (5%) as “statistically
significant”. This means there is a 95% chance that
the observed effect is NOT due to chance alone.
Is this good enough? Is it too restrictive?
Barry Braun, Ph.D. Basics of Study Design
What are the consequences of getting it wrong?
Willing to accept an error rate higher than 5% if the
consequence is getting the wrong sandwich.
Not willing to accept error rate greater than 0.1% if
consequence is landing on jagged rocks.
Every reader will have to use their own judgment
regarding their comfort level with a given
probability that the results are due to chance. Most
journal editors have a comfort level right at 5%.
Barry Braun, Ph.D. Basics of Study Design
Bias – a systematic error that misrepresents the
association between the treatment and outcome.
Investigators may design the study in a way that
makes it more likely to get a particular outcome.
Or, in conducting the study, they may treat the
subjects in one group differently than in the other
group (e.g. more encouragement during a maximal
exercise test with the treatment than the placebo)
Subjects can bias a study as well. Food intake is
often not accurately reported; e.g. faulty memory or
wanting to supply the “right” answer.
Barry Braun, Ph.D. Basics of Study Design
Reliability
Reliability refers to the reproducibility of a
measurement. Measurement tools (surveys, activity
monitors, etc) are often tested extensively before
being used in studies to determine if the values they
report are reproducible. Reliability is the main reason
researchers often need to make multiple
measurements over several days .
Barry Braun, Ph.D. Basics of Study Design
Reliability
It is important to be clear on the distinction between
validity and reliability. A measurement can be
reliable but not valid; i.e., it measures incorrectly
every time. Investigators require results to be both
reliable and valid.
Reliable but
Neither not valid Reliable AND Valid
x x xxx
xx
x xx
x xx
x
x x
x
Barry Braun, Ph.D. Basics of Study Design
Reliability influences # of measurements
Some measurements, e.g. maximal oxygen
consumption (VO2max) are very reliable. You can
measure VO2max on different days, different times
of day, before or after a snack, and the results will
almost always be within a few % of each other.
On the other hand, resting metabolic rate varies
day to day and is very sensitive to time of day,
food intake, exercise, room temperature, etc.
Need very controlled conditions and have to repeat
measurements at least 3 times
Barry Braun, Ph.D. Basics of Study Design
get heart disease
% of people who
0 20 40 60 80
10 30 50 70
dietary fat as a % of total kilocalories
Back to the made-up graph which indicates that the
number of people who get heart disease increases as
the amount of fat in the diet increases.
What are other potential problems with this story?
Did account for all the other confounding variables?
Barry Braun, Ph.D. Basics of Study Design
A confounding variable is associated with both the
exposure and the outcome and that affects the
association between the exposure and outcome.
more exercise more skin cancer
hours per week
more sun exposure
The relationship between exercise and skin cancer is
confounded by strong relationships between exercise and
sun exposure and between sun exposure and skin cancer.
Trying to minimize confounding variables is the most
difficult and time-consuming part of study design
Barry Braun, Ph.D. Basics of Study Design
Can we accurately measure the rate of heart
disease (probably) and the amount of fat in the diet
(much more problematic)?
Do other factors need to be considered?
* gender (true for men AND women?),
* age (maybe elderly people eat more fat)
* ethnicity (directly or indirectly)
* other “risky behavior” (smoking, lack of exercise,
less frequent physicals, etc.) in people who eat
more fat in diet?
Barry Braun, Ph.D. Basics of Study Design
Can you consider all the other factors?
Clearly not b/c we don’t even know what they all are
(e.g. there is a lot of recent evidence that the
conditions a fetus encounters in utero can have an
impact on adult-onset disease).
Even if you could, does a positive relationship
between 2 things (as 1 goes up, the other also
goes up) prove that one causes the other?
Barry Braun, Ph.D. Basics of Study Design
price of gasoline
distance from the Earth to Saturn
During this time period (2005), there was strong
association between the distance from Earth to
Saturn and the price of gasoline. Did gasoline prices
rise because Earth was getting farther from Saturn?
The relationship is a coincidence:
Association does not mean causality
Barry Braun, Ph.D. Basics of Study Design
So, epidemiological studies are difficult to design
in a way that gives you clear, definitive answers.
To get a sharper picture of the causal relationships
between diet and health or performance you can do
an experimental study.
Take a group of healthy people, feed them different
amounts of fat, and see who gets heart disease?
Barry Braun, Ph.D. Basics of Study Design
Experimental Studies
The key difference from an observational study is
that the investigator actively manipulates the
treatment instead of letting things happen by
chance. Because the experimental conditions are
controlled, there is a much greater chance that
the outcomes are directly related to the treatment.
A disadvantage is that by manipulating the
conditions, the results may have less direct
relevance to what happens in the “real-world”
Barry Braun, Ph.D. Basics of Study Design
Experimental Studies
In experimental research, study subjects (whether
human or animal) are selected according to relevant
characteristics and then assigned to either an
experimental group or a control group. The subjects
in the experimental group receive treatment and the
control group receives no treatment or a placebo. If
you do this correctly, you can assume that
differences between the groups at the end of the
study were caused by the treatment.
Barry Braun, Ph.D. Basics of Study Design
Experimental: Cross Sectional
Experimental studies can be cross-sectional (multiple
groups getting a single treatment) or cross-over (one
group getting multiple treatments including control). In
a cross-sectional design, subjects are randomly
assigned to either a treatment or a control group.
They are exposed to the treatment or control for a
period of time and then the outcome is compared
between the two groups. Let’s say you wanted to test
whether consuming only simple sugars for 28 days
would cause more synthesis of muscle glycogen
compared with a “normal” diet.
Barry Braun, Ph.D. Basics of Study Design
Your cross-sectional design might look something
like this:
Group 1
Group 2
Baseline Groups 28
Re-test of
test of randomly days
muscle
muscle assigned glycogen
glycogen synthesis
synthesis
Barry Braun, Ph.D. Basics of Study Design
Assigning subjects to groups
One of the keys to doing this right is to ensure that
the 2 groups of subjects are as similar as possible.
To do this, subjects are usually randomly assigned
to the placebo or control group.
An alternative is to match subjects in each group
on some key characteristics (e.g. age, weight,
training status, aerobic capacity). This helps to
distribute any characteristics that might influence
the results across the groups.
Barry Braun, Ph.D. Basics of Study Design
An example of why randomization is important can
be seen in the following example:
Researchers want to determine if a high fat diet
during marathon training can improve performance.
They do a baseline (before any treatment) test of
aerobic fitness to all of the potential subjects. Then
they assign them to different groups; 20 to the high-
fat diet group and 20 to the high-carbohydrate diet
group. Then they train them using the different diets
for 12 weeks.
Barry Braun, Ph.D. Basics of Study Design
At the end of that time, they redo the test of
aerobic fitness and find that the high-fat group
has improved considerably more (increased
VO2max from 45 to 52 ml/kg/min) than the high-
carbohydrate group (only increased from 68 to 70
ml/kg/min). They report in all of the media outlets
that runners can gain twice the training effect by
using a high-fat diet. Is this reasonable?
Barry Braun, Ph.D. Basics of Study Design
Notice that the baseline VO2max was considerably
higher in the high-fat group. Runners were clearly
not randomly assigned; the high-carbohydrate
group seems to have contained really fit elite
runners (whose VO2max is already about as high
as it can be) and the high-fat group look like mainly
novice runners (who can improve a lot with training).
If the groups had been randomly assigned, the
baseline VO2max would have been similar in the 2
groups. In that case, a larger improvement in the
high-fat group could be interpreted as due to the
diet (assuming everything else had been done right!)
Barry Braun, Ph.D. Basics of Study Design
Blinding
Randomization is often blinded to limit
experimental bias (an interest in having a particular
result). Blinding is used to prevent bias from
influencing the behavior of both the investigators
and the subjects. There are two types of blinding,
single blind and double blind. In a single blinded
study the investigators know which treatment the
subjects are getting but the participants do not. In a
double blinded study, a neutral third party assigns
the groups and neither the investigators nor the
participants are aware of the group assignments.
Barry Braun, Ph.D. Basics of Study Design
A drawback of cross-sectional study design is that
no matter how well you “match” the 2 groups on
important characteristics like age, height, weight,
fitness, etc., there is no way to do this perfectly.
Two groups may be similar but they can’t be
identical, meaning “inter-individual variability”
(genetic and other differences between people) will
be a limitation to showing clear differences
between the treatment and the control groups.
Wouldn’t it be great if you could clone each
subject and use their clone in the other group?
Barry Braun, Ph.D. Basics of Study Design
Experimental: Cross Over
In a cross over design, subjects serve as their own
controls. Half of the subjects get the treatment and
the other half get placebo. Then the same subjects
undergo the opposite protocol.
½ of group
½ of group
Baseline order of 28 28
Re-test of Final test
test of treatment days muscle
1 month days of muscle
muscle randomly washout
glycogen glycogen
glycogen assigned synthesis synthesis
synthesis
Barry Braun, Ph.D. Basics of Study Design
Washout period
A potential problem with the cross-over design is
that effects of the first condition (e.g. treatment)
may have an impact on the response to the second
treatment (e.g. control). The solution is to put a
“washout” period between the 2 conditions to allow
the effects of the first condition to disappear.
This washout period may be long (months for
some interventions like training or lipid-soluble
anabolic agents). This makes the study very lengthy
and it can be difficult to keep subjects in the study.
Barry Braun, Ph.D. Basics of Study Design
External Validity
Also referred to as generalizability; meaning how
applicable are the results to the general population.
To increase the external validity, investigators can
study subjects varying in gender, race, ethnicity,
age, weight, etc. By doing this, it is more likely that
results can be applied to the general population.
Barry Braun, Ph.D. Basics of Study Design
Overgeneralizing
Many classic studies in nutrition (for example; the
response to semi-starvation and re-feeding; human
protein requirements) were performed almost
solely using Caucasian, male, healthy subjects in
their 20’s and 30’s.
Nutritional requirements were generalized from
those studies to the entire population, despite few
data on women, children, ethnic/racial minorities or
people with underlying health problems
Barry Braun, Ph.D. Basics of Study Design
Trade-offs
All major funding agencies now mandate inclusion
of women and minorities or require a strong
justification for not doing that.
Why not include as many types of subjects as
possible in order to maximize the external validity?
Increasing external validity also means increasing
the number of potential confounding variables. In
some studies, it is more prudent to use a specific
population to minimize confounding variables
Barry Braun, Ph.D. Basics of Study Design
“Basic” research studies
Experiments under highly controlled conditions are
often necessary to confirm observations or uncover
how a process works (the mechanism of action).
They may be conducted in vitro (e.g. with cell
populations on culture plates) or with animals.
These studies allow the investigator to isolate one
variable of interest without confounding variables
such as environmental factors, genetic variation, and
differences in dietary or physical activity patterns.
Barry Braun, Ph.D. Basics of Study Design
One of the advantages of doing studies using cells
or animals is that tissues not available in humans
can be isolated (e.g. whole muscle, liver, heart,
etc.) and life spans are much shorter. For example,
if we were to do our study of high fat diets and
heart disease in mice instead of humans, the study
would take a couple of years instead of decades.
And researchers could sacrifice the mice at the end
of the study and look directly at the effects of the
diets on their arteries, muscle, liver, etc.
Barry Braun, Ph.D. Basics of Study Design
Due to differences in physiology and the fact that
animals are routinely exposed to levels of
compounds far higher than those humans typically
encounter, results from studies with animals are
not directly generalizable to humans.
In addition, there are moral issues regarding animal
experimentation that can’t be ignored. Some people
feel strongly that no experimentation on animals is
ever justified. Some people have no problem at all
with scientific experimentation on animals.
Barry Braun, Ph.D. Basics of Study Design
The great majority of individuals, both within and
outside the scientific community see this as a
complicated issue. There are benefits to animal
research (potentially lifesaving cures for human
disease; many dogs were sacrificed
in the hands of Banting and Best
before they were able to isolate and
purify the insulin that has saved
the lives of millions of people with diabetes).
And certainly costs (nobody enjoys the idea of
submitting creatures to experimental procedures
that often end with their death).
Barry Braun, Ph.D. Basics of Study Design
And the type of animal is certainly a factor in
people’s discomfort with animal research: few
people object to research on flies, a few more to
fish or frogs, many become uncomfortable
with experiments on mice, rats, and rabbits, and
even more people feel strongly about research on
cats, dogs and primates.
To balance these competing forces, universities
and other research organizations follow strict
guidelines to help ensure that research on animals
is conducted in the most humane possible way
Barry Braun, Ph.D. Basics of Study Design
Researchers are required to justify why the
research is essential (disease yes, performance
no), to use statistical analysis to minimize the
number of animals they intend to study ,and to
maximize the comfort and well-being of the
animals in their care.
As new experimental and mathematical modeling
techniques are developed, the justification for
doing research on animals is expected to diminish
in the near future.
Barry Braun, Ph.D. Basics of Study Design
Human Experimentation
The moral issues of experimentation extends to
humans as well. Before organizations began to
regulate the conduct of experiments on humans,
experiments were sometimes done without
subjects consent and with little regard for their
health or well-being.
Human research in most countries
is regulated to ensure that subjects
can truly give informed consent to the procedures
and that potential benefits outweigh risks
Barry Braun, Ph.D. Basics of Study Design
Potential subjects have to be recruited in ways that
are not coercive and they must be in a position to
refuse to participate or to leave the study partway
though without adverse consequence (so no
prisoners, people who are institutionalized,
children unless with parental consent).
What about students in a class being taught by the
researcher? Grad students in the lab? The
researcher has to convince an institutional review
board that participation or non-participation will
have absolutely no consequences with respect to
their grade in the class or graduation, etc.
Barry Braun, Ph.D. Basics of Study Design
Review boards weigh the potential benefits from
the research with the stress, physical and mental
discomfort, time commitment, etc. that the subjects
are required to undergo.
During the study itself, procedures must be in
place to ensure that health and well-being of the
subjects are a higher priority than the data.
Subjects are often compensated financially for
their participation; it is important that compensation
be sufficient but not excessive (i.e. coercive)
Barry Braun, Ph.D. Basics of Study Design
Because the priority to maximize health and well-
being of the subjects and to ensure they are not
coerced into continuing participation in a study can
conflict with the need to collect vital research data,
doing human studies in a way that both prioritizes
subject well-being AND maintains maximal scientific
rigor is very difficult.
Barry Braun, Ph.D. Basics of Study Design
get heart disease
% of people who
0 20 40 60 80
10 30 50 70
dietary fat as a % of total kilocalories
So let’s return to this made-up association. Could you
do an experimental study in which you recruit
subjects without heart disease, feed them several
different amounts of dietary fat and look at the
relationship between dietary fat and the rate of heart
disease over time?
Barry Braun, Ph.D. Basics of Study Design
Yes. But would require studying hundreds of
people for decades, providing all of their meals and
controlling dozens of other things that affect risk for
heart disease (like smoking and exercise and
aspirin use and on and on).
This would cost tens of
millions of dollars, take
several decades and
would still be almost
impossible to
do b/c most volunteers
would leave the study
Barry Braun, Ph.D. Basics of Study Design
So, how do you design a study that can answer an
important question and that is doable in a
reasonable time frame and for a reasonable
amount of $$?
You have to take a big important question and
reduce it to a much smaller, more focused
question. You have to do a series of small studies,
each one building on the one before, until you
accumulate enough evidence to support or
disprove your idea.
Barry Braun, Ph.D. Basics of Study Design
Barry Braun, Ph.D. Basics of Study Design
Matching subjects on key characteristics
To compare whether men responded to cookie
cream similarly to women, you plan to do a
2nd group composed of men. What kind of men?
Well, you can recruit men matched to the
characteristics of the women. How about
VO2max? OK, but a woman with VO2max of 60
ml/kg/min is often a lean athlete in hard training
whereas elite male athletes have a higher
VO2max. So men and women matched on
VO2max will usually differ on body fat and training
Barry Braun, Ph.D. Basics of Study Design
How about body fat? OK, but an average body fat
for a man, let’s say 15% would be very low for a
woman, and again you are likely to end up with
trained female athletes with very high VO2max and
moderately trained, recreationally active men.
There is actually almost no way to match men and
women for both aerobic capacity AND body fat.
Need to choose the one that is MOST critical. Or
another characteristic that CAN be matched (e.g.
training status)
Barry Braun, Ph.D. Basics of Study Design
Recruiting subjects for studies requires balancing
many competing factors. A more diverse subject
pool gives you more generalizability but also more
confounding variables, increasing the number of
subjects required.
A more homogeneous group (e.g. highly trained,
college-age women in the luteal phase) reduces
the confounding variables and allows you to do the
study with fewer subjects but also makes the
results less generalizable
Barry Braun, Ph.D. Basics of Study Design
The trade-offs illustrate why studies at all levels of
generalizability are required to answer important
questions. Epidemiological studies using large,
heterogeneous sample sizes can point to
interesting associations that are worth pursuing
(e.g. more physical activity is associated with lower
rates of diabetes).
Basic scientists can look at potential mechanism
(e.g. isolated rat muscle electrically stimulated to
contract takes up more sugar than resting muscle)
Barry Braun, Ph.D. Basics of Study Design
In between are human experimental trials ranging
from simulating the rat study (putting in arterial and
venous catheters in the leg of a volunteer to see if
exercised human muscle also takes up more sugar
for the blood) to testing different intensities and
durations of physical activity on groups of free-
living people to determine which combination has
the biggest impact to reduce the risk for diabetes.
The best study designs build on the
results that have come before and add
another key piece to the jigsaw puzzle.
Barry Braun, Ph.D. Basics of Study Design
New York Times, 5-27-01
“Coke formed a partnership with
Procter & Gamble earlier this spring.
The companies are now preparing to introduce a
drink called Elations. Each bottle of Elations
contains 1500 milligrams of glucosamine, a dietary
supplement that has been popular among people
with arthritis for years.”
Procter officials insist that sound science is what
distinguishes Elations from the many herbal
concoctions currently transforming the market.”
Barry Braun, Ph.D. Basics of Study Design
New York Times, 5-27-01 cont’d
“The National Institutes of Health is conducting a
comprehensive 4-year study on glucosamine. But
neither Coke nor Procter felt they could afford to
wait for the results. ““The game will be over if
anybody isn't in it by then," said the assistant
director of Procter's Nutrition Science Institute””.
Can research be done in a way that balances needs
of the scientific community (a line of research
studies that tell the whole story) with needs of
industry (no research or a single study showing the
product works)?
Barry Braun, Ph.D. Basics of Study Design
For industry, enough research is ....
sufficient to convince enough consumers
to buy the product that $$ from sales exceeds the
costs of manufacture, distribution and advertising.
To do more violates the interests of employees,
shareholders and, in terms of price, consumers.
Doing more than the minimum research needed
to maximize sales is not only unnecessary but
even incompatible with interests of the company.
Barry Braun, Ph.D. Basics of Study Design
For academic scientists, enough
research is …..
First: efficacy (does it work?)
safety (does it harm?) but also:
Research scientists are charged with understanding
context: mechanism of action, effects on other
metabolic pathways etc.
Doing less than the minimum research required
to understand the physiological context is
incompatible with responsibilities as scientists.
Barry Braun, Ph.D. Basics of Study Design
Is there a way to meet halfway?
YES, in the sense that both groups share the same
basic goals of optimizing safety, health, and
performance
NO, in the sense that there are fundamental
disagreements about who (target population), what
(top priorities), why (knowledge/sales), when (how
soon), and how (single study vs. line of research)
research is done
Barry Braun, Ph.D. Basics of Study Design
A fairy tale revisited
A group of 12 men she knows from her gym will
participate in the study. They will weigh themselves
at home and then come to the laboratory so their
body fat can be measured using skin fold calipers.
Then they will do as many pushup and situps as
they possibly can. They will be given 30 doses of
"Bark-a-Lounge" in pill form and told to take 2 every
day for about 15 days. After 15 days they will re-
weigh themselves and come back to the laboratory
to have body fat re-measured and do as many
pushups and situps as possible. Dr. Toboso
is sure that the men will lose fat but gain strength
after taking "Bark-a-Lounge" for 15 days.
Barry Braun, Ph.D. Basics of Study Design
Doing studies correctly is hard. Why
keep doing them?
“It’s supposed to be hard. That’s what
makes it great. If it was easy, anybody
could do it.”
Barry Braun, Ph.D. Basics of Study Design