The Basics of Designing and Running an Economics Experiment
Economics 328
Spring 2005
The Purposes of Running an Experiment
Test a Theory
Gather Empirical Regularities to Inform a Theory
Test Institutions
What Makes a Good Experiment?
Should an experiment replicate reality?
No! We have field studies for that.
Should an experiment replicate a formal model?
No! We have theory for that.
A good experiment tries to capture the most relevant
features of reality in a simple, carefully controlled
environment. Good experiments are usually
designed to test specific hypotheses, sometimes
derived from the implications of some economic
theory and other times based on previous
observations in either experiments or field data.
What is a Controlled Experiment?
As much as possible, the experiment controls all
elements of the environment in which the experiment
takes place.
The experimenter determines the rules – what choices are
available to subjects, when decisions are made, and what
the consequences of these decisions will be.
The experimenter controls subjects’ payoffs as a function of
the actions they take.
The experimenter controls the information that is available
to subjects.
What is a Controlled Experiment?
Between treatments, the experimenter only changes variables which
are directly relevant to the hypothesis being tested, otherwise holding
the environment fixed.
Control vs. Treatment Sessions
Avoid Confounds (e.g. Don’t change more than one thing at a time.)
Variables that cannot be directly controlled are typically controlled via
randomization.
Example: Many experiments are designed to test subjects’ attitudes
towards fairness are affected by some treatment variable. To the extent
that subjects enter the lab with differing attitudes about fairness, a true
controlled experiment can’t be run. However, by randomly assigning
subjects to treatments, we can eliminate subjects differing attitudes as a
cause of differences between treatments. This relies on the law of large
numbers, implying that a large sample may be necessary.
Designing an Experiment
Identify an interesting question or questions. These should be issues that are
better addressed through a controlled experiment than through gathering field
data.
Determine hypotheses you would like to test.
Example: Increased payoffs lead to fewer mistakes in solving logic problems.
Example: Face-to-face interaction will lead to greater concern with fairness.
Design a simple environment that allows you to test the hypotheses you are
interested in. Remember, the more complicated the environment is, the more
likely you are to lose control. An experiment is not meant to replicate reality.
The reason we use controlled experiments is because reality is too complicated.
Brandts and Cooper:
The Weak Link Game
The employee payoff Worker i's Payoff Table, B = 6
function is given by the Minimum Effort by Other Workers
following equation: 0 10 20 30 40
E
0 200 200 200 200 200
Effort 10 150 210 210 210 210
π 200 5E i B min
i
e j
by 20 100 160 220 220 220
j1,2,3,4 Worker i 30 50 110 170 230 230
40 0 60 120 180 240
Having all five players
choose the same effort level
is a Nash equilibrium.
The critical variable here is Worker i's Payoff Table, B = 14
B, the bonus rate. Higher Minimum Effort by Other Workers
values of B give greater 0 10 20 30 40
benefits to successful Effort
0
10
200
150
200
290
200
290
200
290
200
290
coordination. by 20 100 240 380 380 380
Trying to move to a higher
Worker i 30 50 190 330 470 470
40 0 140 280 420 560
effort level is a risky
strategy.
Brandts and Cooper:
Research Questions
Can an increase in the bonus rate enable a firm to overcome
coordination failure?
Does the magnitude of the bonus rate increase matter or is the
simple fact of an increase effective as such?
If an increase in the bonus rate brings about improved coordination,
can the bonus rate increase be revoked without affecting the
improved outcome?
Does the length of time a firm has been underperforming affect the
impact of the increase?
Designing an Experiment
Test your hypotheses by varying a small number of variables
within the experiment. If at all possible, you should not vary
more than one variable at a time.
Within vs. Between Subject Designs
Some important factors to remember . . .
Subjects get bored.
Subjects get confused.
Subjects learn as they gain experience within the experiment.
You should anticipate changes in their behavior (order effects).
Design for Brandts and Cooper
List of Treatments
Cell 1 Cell 2 Cell 3 Cell 4 Cell 5 Cell 6
Bonus Rate
6 6 6 6 6 6
Rounds 1 – 10
Bonus Rate
14 10 8 14 14 6
Rounds 11 – 20
Bonus Rate
14 10 8 10 6 14
Rounds 21-30
Results for Brandts and Cooper
Comparison of Treatments, Second Block
40
30
Average Minimum Effort
20
10
0
10 11 12 13 14 15 16 17 18 19 20
Period
Treatment 6/6/14 Treatment 6/8/8 Treatment 6/10/10 Treatment 6/14/X
Results of Classroom Experiment
Effect of Bonus Rate Increase
40
30
Average Minimum Effort
20
10
0
5 6 7 8 9 10
Round
B = 8 or B = 10 B = 14
Results of Classroom Experiment
Effect of Bonus Rate Increase
40
30
Average Effort
20
10
0
5 6 7 8 9 10
Round
B = 8 or B = 10 B = 14
Designing an Experiment
Avoid the use of deceit. This is a very tempting trap, but one that is
highly frowned upon by the profession. In particular, what is going to
happen the next time you try to run an experiment?
Anonymity: Most experiments guarantee subjects anonymity. In other
words, subjects are guaranteed that no other subjects (or indeed,
nobody other than the researchers) will be able to ever identify their
actions or payoffs. For some experiments, abandoning anonymity is an
important part of the design. If so, subjects should know what
information about them is be revealed publicly. Subjects should have
the option of withdrawing if they do not want information about them
revealed publicly.
Subjects
What population should you use as subjects?
Undergraduates are the easiest to get, but subjects with relevant experience are often
more interesting to study.
How do you get subjects?
Advertisements can be placed on posters, electronic bulletin boards, or in newspapers.
Spamming, while effective, is generally not a good way to make friends.
What should be in the advertisement?
It should include a very brief description, a summary of payoffs (average payoff is
sufficient), the time necessary to complete the experiment, and contact information.
(E-mail addresses work well. A website where people can sign up for the experiment
works even better.)
The advertisement should stress the monetary payoffs. Remember, you want
controlled experiments. This means that you want subjects who care about the
monetary payoffs being offered.
Play Games!!!
Earn Cash!!!
Participate in an Experiment in
Economic Decision Making!!!
Average Earnings = $20 - 25!!!
Takes 1½ Hours or Less!!!
Wednesday, September 4th, 4:30
Thursday, September 5th, 4:30
Monday, September 9th, 4:30
Tuesday, September 10th, 9:30
More Times Available
Sign up at :
www.weatherhead.cwru.edu/exper
Play Games!!!
Earn $$$
Subjects
Avoid unintentional selection of the subject population. For example, if you post
all of you ads in the weight room in Veale, your subject population will be
skewed towards men. You should also avoid unintentional clustering (e.g. don’t
sign up ten people from the same fraternity for a single session).
Have times and locations for sessions selected before posting any ads. Be
certain to schedule potential subjects as soon as they respond to your ad. Be
certain to get contact information from your subjects. Confirm that all subjects
will attend the day before the session. Even with confirmation, you should
anticipate that only about 80% of subjects will show up (CWRU students tend to
be pretty good about showing up).
Keep careful track of all contacts with potential subjects. You should never use
a subject twice for the same set of experiments. Most experimental groups
keep a black list – subjects who don’t show up or disrupt sessions are
undesirable.
Instructions
Most experiments include some form of consent form. For those of you who
plan on publishing your research, this brings up the issue of the dread IRB . . .
The overarching goal of instructions is to make certain that the subjects
understand the rules of the experiment. If subjects do not understand the
rules, you have immediately lost control.
Instructions should be as clear and complete as possible. Any critical points
should be repeated at least once. If in doubt, make the instructions too
detailed. Nobody likes sitting through lengthy instructions, but remember your
goal – the subjects need to understand the rules of the experiment.
Context: Many experiments like to frame their instructions in a generic context
that does not have any real world connotations. In my opinion, this often
generates confusion for experimental subjects, but that is a controversial point.
All experimenters agree that you should avoid any loaded terms.
Example: In a Prisoner’s Dilemma Game, you wouldn’t want to label the two
strategies as “Cooperate” and “Cheat” even though this is how game theorists often
refer to them.