Field Research Methods
in Psychology
Dr. John Hetherington
Fall 1997
• Research Methods and Statistics
are tools.
• Concern is:
• How to use them?
• When to use them?
What is Science?
• Science is not a state;
• But a process by which one
establishes knowledge or
obtains information.
• At very least seek a reduction of
uncertanity
• No such thing as “TRUTH”
Ways of Knowing (Kerlinger, 1986)
1. Method of Tenacity (Always Believed)
2. Method of Intuition (Feels Good)
3. Method of Authority (Respected Source)
4. Method of Science (Empirical/Objective)
What are the Goals
of Science?
Goals of Science:
1. Description
2. Explanation
3. Prediction
4. Control
Scientific Methodology =
a systematic analysis of the rational
and experimental principles which
guide an inquiry
Scientific Method involves:
1. Theory = an integrated set of
principles that explain and predict
facts
2. Hypothesis = a prediction of what
is the case (fact) based on theory
3. Observation = a comparison of
hypothesis to what is the case
Physical Sensory Mental
Reality 1 Experience 2 Constructs 3
induction
Theory
Observation deduction
Immanuel testing
Kant Hypothesis
1=2 2=3
Radical Empiricism Logical Positivism
Two distinct contexts of scientific
method (Reichenbach, 1951):
1. Context of Discovery =
creative, arational, no justification
or responsibility for statements
2. Context of Justification=
rigorous, austere, stringent req.
for considering information factual
Function of Theories:
Evaluation Criteria
1. Predict relationships between
variables
Education SES
Self-
Esteem
Salary
2. Summarize large amounts of data
(parsimony)
e= mc2
3. Application to many different
phenomena (generalizability)
4. Generate additional research by
suggesting novel relationships
(new hypotheses)
5. Apply research to practical
problems (utilitiy)
Research Methods
Researchable Research
Content
Questions Design
Area
Measurement Sampling Data
Methods Collection
Statistical
?
Report
Analysis Writing
Assessment of Observation
(Measurement)
AAaahh!!
A shark!
Observed Score = True Score + Error
Error component may be either:
• Random Error = Varaiation due to
unknown or uncontrolled factors
• Systematic Error = variation due to
systematic but irrelevant elements
of the design
• Concern of scientific research is
management of the error
component
• Number of criteria by which to
evaluate success
1. Reliability
• Does the measure consistently reflect
changes in what it purports to
measure?
• Consistency or stability of data across
• Time
• Circumstances
• Balance between consistency and
sensitivity of measure
2. Validity
• Does the measure actually
represent what it purports to
measure?
• Accuracy of the data (for what?)
• Number of different types:
A. Internal Validity
• Effects of an experiment are due solely
to the experimental conditions
• Extent to which causal conclusions can
be drawn
Semmelweis
Pasteur
Lister
• Dependent upon experimental control
• Trade-off between high internal
validity and generalizability of results
B. External Validity
• Can the results of an experiment be
applied to other individuals or
situations?
• Extent to which results can be
generalized to broader populations or
settings
• Dependent upon sampling subjects and
occasions
• Trade-off between high
generalizability and internal validity
C. Construct Validity
• Whether or not an abstract,
hypothetical concept exists as
postulated
• Examples of Constructs:
• Intelligence
• Personality
• Conscience
Based on:
• Convergence = different measures that
purport to measure the same construct
should be highly correlated (similar)
with one another
• Divergence = tests measuring one
construct should not be highly
correlated (similar) to tests purporting
to measure other constructs
D. Statistical Conclusion Validity
• The extent to which a study has used
appropriate design and statistical
methods to enable it to detect the
effects that are present
• The accuracy of conclusions about
covariation made on the basis of
statistical evidence
•Based on appropriate:
•Statistical Power
•Methodological Design
•Statistical Analyses
Can have a reliable, but invalid
measure.
If measure is valid, then
necessarily reliable.
3. Utility
• Usefulness of methods gauged in terms
of:
A. Efficiency
B. Generality
A. Efficient Methods provide:
• Precise, reliable data with relatively
low costs in:
• time
• materials
• equipment
• personnel
B. Generality
• Refers to the extent to which a method
can be applied successfully to a wide
range of phenomena
• a.k.a. Generalizability
Threats to Validity
• Numerous ways vailidity can be
threatend
• Related to Design
• Related to Experimenter
Related to Design
1. Threats to Internal Validity
(Cook & Campbell, 1979)
A. History = specific events occurring to
individual subject
B. Testing = repeated exposure to testing
instrument
C. Instrumentation=changes in the
scoring procedure over time
D. Regression = reversion of scores
toward the mean or toward less
extreme scores
E. Mortatility = differential attrition
across groups
F. Maturation = developmental
processes
G. Selection = differential composition
of subjects among samples
H. Selection by Maturation interaction
I. Ambiguity about casual direction
J. Diffusion of Treatments =
information spread between groups
K. Compensatory Equalization of
Treatments = lack of treatment
integrity
L. Compensatory Rivalry = “John
Henry” effect on nonparticpants
2. Threats to External Validity
(LeCompte & Goetz, 1982)
A. Selection = results sample-specific
B. Setting = results context-specific
C. History = unique experiences of
sample limit generalizability
D. Construct efffects = constricts are
sample specific
Related to Experimenter
1. Noninteractional Artifacts
A. Observer Bias = over/under
estimation of phenomenon (schema)
B. Interpreter Bias = error in
interpretation of data
C. Intentional Bias = fabrication or
fraudulent interpretation of data
2. Interactional Artifacts
A. Biosocial Effects = errors attributable
to biosocial attributes of researcher
B. Psychosocial Effects = errors
attributable to psychosocial attributes
of researcher
C. Situational Effects = errors
attributable to research setting and
participants
D. Modeling Effects = errors
attributable to example set by
researcher
E. Experimenter Expectancy Bias =
researchers treatment of participants
elicits confirmatory evidence of
hypothesis
Basic Applied
Purpose •Expand Knowledge •Understand Specific Problem
Context •Academic Setting •Real-World Setting
•Single Researcher •Multiple Researchers
•Less Time/Cost Pressure •More Time/Cost Pressure
Methods •Internal Validity •External Validity
•Cause •Effect
•Single Level of Analysis •Multiple Levels of Analysis
•Single Method •Multiple Methods
•Experimental •Quasi-Experimental
•Direct Observations •Indirect Observations
Only substantial difference
between applied and basic
research:
• Basic = Experimental Control
• Applied = Statistical Control
Research Planning Model
Planning research is iterative
process
• Research plan constantly being
updated throughout development and
implementation
• Emphasize both "rational and
experimental principles" or
methodology
Two major phases of Research
Planning:
1. Planning Phase = defining focus of
research and developing research plan
2. Execution Phase = implementation
and monitoring success of research
plan
Model of Planning Phase
(Hedrick, Bickman, & Rog, 1993)
A. Research Definition Stage (I)
• Understand the Problem
• Identify Questions (Hypothesize)
B. Research Design\Plan Stage (II)
• Choose Design
• Data Collection Approaches
Understand the Problem
STAGE 1:
RESEARCH
Identify Questions
DEFINITION
Refine/Revise Questions
STAGE 2:
RESEARCH Choose Design
PLAN/DESIGN
Determine Trade-offs Inventory Resources
Assess Feasibility
Planning Phase
A. Defining scope of research (Stage I)
• Identifying and specifying subject
area for investigation
• Involves:
Understand the Problem
Identify Questions
Refine/Revise Questions
1. Developing an understanding
of issue or problem
• Gathering information about subject
(literature review, informal
discussions--focus groups, on-site
evaluations)
• Multiple sources of information
(convergence and divergence)
2. Identifying specific/researchable
questions
• Specification of variables of interest
and the relationships amongst them
• Two Types of questions:
• Descriptive Questions
• Normative Questions
a. Descriptive Questions
• Provide factual information about the
characteristics of some entity
• Factual information about the nature of
a problem, the objectives of program,
or needs of a population
• Can also refer to describing
relationship between variables of
interest (a.k.a., correlative questions)
b. Normative Questions
• Compares current state of affairs to
evaluative, a priori criteria
• Standards could be set by legislative
objectives, professional standards,
program goals, or any other normative
criteria
• Can also refer to impact or cost-
effectiveness questions
3. Refining and revising the
questions
• Assemble questions or group of
questions into research hypotheses
• Are these questions explicit, logical,
and testable?
• Concerns of reliability and validity
B. Developing a research plan
(Stage II)
Choose Design
Determine Trade-offs Inventory Resources
Assess Feasibility
•Balance and weigh importance of:
• Credibility = validity of design to
support conclusions
• Utility = design will address specific
questions of interest
• Feasibility = research design and plan
are reasonable given time and resource
constraints
Several Key Elements of
Research Design:
1. Operational definitions of variables of
interest
• Detailed description of important
variables
• A definition in terms of exact
operations that will be used to
manipulate or measure each variable
2. Identification of what
comparisons will be made
• Specification of relations between
variables
• Order of relations (causal or
associative)
3. Specification of level of analysis
• Individual or aggregate data
collection\analysis
• Implications for data collection,
analyses, prediction, generalizability,
and explanation
4. Representativeness of sampling
• Populational, geographic, and temporal
generalizability
• Identification and selection of
sampling strategy
5. Determine level of precision of
results
• Rigor of design and measures affects
how precise answers to research
questions will be
• Size of sample also affects statistical
power
Selecting a Research
Design
Research Design
• Refers to the outline, plan, or strategy
specifying the procedure to be used in
answering research questions
• Determines the when (a procedural
sequence) but not the how of:
1. Sampling Techniques and
representativeness of data sources
2. Data Collection
• Time frame of measurement
• Methods of measurement
3. Data Analysis
Three Major Approaches
to Research Designs
1. Descriptive Approach
2. Experimental Approach
3. Quasi-Experimental Approach
1. Descriptive Approach
• Represent or provide accurate
characterization of phenomenon under
investigation
• Provide a “picture” of a phenomenon
as it naturally occurs
Key Features:
• Not designed to provide information
on cause-effect relationships, therefore
internal validity is not a concern
• Because focus is on providing
information regarding some population
or phenomena, external validity is
major concern
Variations:
• Exploratory
• Goal: to generate ideas in field of
inquiry that is relatively unknown
• Least structured to gather more
descriptive information
• Frequently used as first in series of
studies on a specific topic
•Process evaluation
• Goal: to identify the extent to which a
program (or policy) has been implemented,
how that has occurred, and what barriers
have emerged
• Program as implemented vs. program as
intended
• Designed to guide development of new
program (normative), summarize the
structure of program prior to studying its
effects, or assess feasance of pre-existing
program
•Strengths (Descriptive):
• Generally lower costs (depend upon sample
size, number of data sources, and
complexity of data collection methods)
• Relative ease of implementation
• Ability to yield results in relatively short
amount of time
• Data analysis straight-forward
• Results easy to communicate to non-
technical population
•Limitations (Descriptive):
• Cannot answer questions of causal
nature
2. Experimental Approach
• Primary purpose is to empirically test
the existence of causal relationship
among two or more variables
• Systematic variation of independent
variable (IV) and measure its effects on
dependent variable (DV)
Kerlinger (1973) MAX-MIN-CON
Approach
• MAXimize systematic variance (exp
cond as different as possible)
• MINimize error variance (accuracy of
assessment)
• CONtrol extraneous systematic
variance (homogeneity of conditions)
Key Features:
• Random assignment of individuals or
entities to the levels or conditions of
the study
• Control biases at time of assignment
• Ensure only independent variable(s)
differs between conditions
Key Features (cont):
• Emphasis placed on maximizing
internal validity by controlling possibly
confounding variables
• Creation of highly controlled
conditions may reduce external validity
Variations:
1. Between Group Designs
A. Post-only design
• Subjects randomly assigned to
experimental\control groups
• Introduction of IV in experimental
condition
• Measurement of DV (single or
multiple instances)
Post-only design
Randomized IV DV
Group 1 R X O
Group 2 R O
B. Pre and post design:
• Subjects randomly assigned to
experimental\control groups
• Preliminary measurement of DV
before before treatment (check of
random assgn)
• Introduction of IV in experimental
condition
• Measurement of DV (single or
multiple instances)
Pre- and Post- design
Randomized DV IV DV
Group 1 R O1 X O2
Group 2 R O1 O2
C. Multiple Levels of single IV:
• Subjects randomly assigned to
experimental\control groups
• Introduction of multiple levels of IV
in experimental condition
• Measurement of DV across different
conditions
Multiple Levels design
Randomized IV DV
Group 1 R X1 O
Group 2 R X2 O
Group 3 R X3 O
Group 4 R O
D. Multiple Experimental and Control
Groups (Solomon Four-Group design):
• Subjects randomly assigned to
experimental\control groups
• Preliminary measurement of DV in one
exp\control pair
• Introduction of IV in both
experimental conditions
• Measurement of DV (assess effects of
pretest)
Multiple Levels design
Randomized DV IV DV
Group 1 R O1 X O2
Group 2 R O1 O2
Group 3 R X O2
Group 4 R O2
E. Multiple IVs (Factorial Design):
• Subjects randomly assigned to
experimental\control groups
• Introduction of multiple levels of IVs
in experimental condition
• Measurement of DV across different
conditions (cells)
Multiple IVs design
Randomized IV IV DV
Group 1 R X1 Y1 O
Group 2 R X2 Y2 O
Group 3 R X1 Y2 O
Group 4 R X2 Y1 O
2. Within-Group Designs
• Repeated Measures
• Each Subject is presented with two or
more experimental conditions
• Comparisons are made between
conditions within the same group of
subjects
Within SS design
Randomized IV DV IV DV
Subject 1 R X1 O1 X2 O2
Subject 2 R X1 O1 X2 O2
Subject 3 R X1 O1 X2 O2
Subject 4 R X1 O1 X2 O2
•Strengths (Expl):
• Experimental control over threats to
internal validity
• Ability to rule out possible alternative
explanations of effects
•Limitations (Expl):
• More resemble controlled study, less
resembles usual real-world
intervention (decrease external
validity)
• Experimental Realism = engagement
of subject in experimental situation
• Mundane Realism = correspondence of
experimental situation to „real-world‟
or common experience
•Limitations (Expl):
• Randomized experiments difficult to
implement with integrity(practical or
ethical reasons)
• Attrition over time = non-equivalent
designs
3. Quasi-Experimental Approach
• Primary purpose is to empirically test
the existence of causal relationship
among two or more variables
• Employed when random assignment
and experimental control over IV is
impossible or impractical
•Key Features:
• Other design features are substituted
for randomization process
• Quasi-experimental comparison base:
• Addition of non-equivalent
comparison groups
• Addition of pre- and post-treatment
observations
•Variations:
A. Non-equivalent Comparison Group
• Post Only, Pre-Post, Multiple Treatments
• No random assignment into exp\con
• “Create” comparison groups
• Selection criteria or eligibility protocol
• Partial out confounding variance
• (statistical control)
B. Interrupted Time Series
• Multiple observations before and after
treatment or intervention is introduced
• Examine changes in data trends
(slope and intercept)
• Investigate effects of both onset and
offset of interventions
Interrupted Time Series
16
14
12
10
8
6
4
2
0
Intervention
C. Regression Discontinuity
• Separate sample based on some
criterion (pre-test)
• One group administered treatment,
other is control group
• Examine trends in both groups;
hypotheisze equivalent
Regression Discontinuity
No Treatment Treatment
Post
Test
Scores
Pre-test Cut-off Point
•Strengths (Quasi):
• Approximation of experimental design,
thereby allowing causal inference
• Garner internal validity through
statisticalcontrol, not experimental
• Use where experimental designs are
impractical or unethical
•Limitations (Quasi):
• Uncertainty about comparison base:
Is it biased?
• Statistical control based on known
factors. If unknown or unmeasureable,
threat to validity
• Data collection schedule and measures
very important
2 2
2
Statistics Review
Measurement
• Assignment of numbers or values to
levels of a variable according to a set
of rules
• Basis of scientific observation
Variable Types
• Dichotomous = variable that has only
two categories (either\or)
• Discrete = variables that increase or
decrease by whole units
• Continuous = variables that can
theoretically assume infinite number of
values
Scales of Measurement
(Stevens, 1951)
1. Nominal or Categorical
2. Ordinal
3. Interval
4. Ratio
1. Nominal or Categorical
• Classification according to presence or
absence of qualities
• No information provided on order or
magnitude of differences
• Because nominal scales have no
quantitative properties, data consist of
frequencies only
• E.g., sex, race, religion, political party
2. Ordinal
• Classification according to degree of
quality present
• Distinguish between ordered
relationships between classes or
characteristics, but no information
about the magnitude of difference
• E.g., tall > normal > short,
first > second > third
3. Interval
• Addition of a meaningful unit of
measure: equal size interval
• Consistent and useful unit of measure
allows the use of basic arithmetic
functions (addition, subtraction,
multiplication, division)
• E.g., Fahrenheit scale, shoe size
4. Ratio
• Addition of an absolute zero point to
interval scale
• Zero implies total absence of the
characteristic
• Ability to utilize ratio statements (2:1,
1:5)
• E.g., Height and weight
Higher order scales can readily be
translated into lower order scales,
but not vice versa.
Because higher order scales
contain more information, detail is
lost going to a lower-order scale of
measurement.
Three main ways to describe
distributions of data:
1. Shape of distribution
2. Measures of central tendency
(typical score)
3. Measures of variability
(differences between scores)
1. Shape of Distribution
• Concerned with two distinctive
features of distributions
A. Symmetry = similarity of right\left
halves
B. Peak (mode) = most frequently
occurring score
A. Symmetry
• Bell-shaped curves = describe in terms
of its kurtosis (curvature)
1. Leptokurtic = thin distribution
(concentrated at midpoint)
2. Mesokurtic = normal distribution
3. Platykurtic = flat distribution
•Non-symmetrical
1. Positive Skew = high number of low
scores
2. Negative Skew = high number of high
scores
•Non-bell shaped curves
1. U-shaped curves
2. Bimodal or Multi-modal distributions
B. Mode
1. Unimodal = single peak
2. Bimodal = double peak
2. Measures of Central Tendency
• Asks the question: "What is the typical
score in a distribution?"
• Three different methods:
1. Mode
2. Median
3. Mean
A. Mode
• most frequently occurring
value (peak)
• Advantages
• Requires no difficult arithmetic
computations
• Used at nominal level of
measurement
• Readily identifiable on polygon
distribution
•Limitations
• May not be very descriptive of
distribution
• May not be unique
• Overly affected by sampling variation
B. Median
• 50th percentile or that point at which
exactly 50% of the score values are
above or below
• Requires at least ordinal level, better if
interval level
•Advantages:
• Precise midpoint of distribution
• Compare two different scores on same
distribution
• Because not as sensitive to extreme
scores, good measure for skewed
distribution
•Limitations:
• Arithmetic operations (algebraic)
• Difficult to generalize to multiple
variable statistics
C. Mean
• arithmetic sum of scores in distribution
divided by total number of scores
X X
X n
N
• Requires at least interval level of
measurement
•Advantages:
• Sensitive to extreme scores
• Best predictor of any score in
distribution
• Total sum deviations around mean
always = 0
Relationship between measures of
Central Tendency
• Normal Curve = Mean, mode, and
median are same score
Mean. Mode & Median
Median Mode Mode Median
Mean Mea
n
• Skewed Curve =
• Mode = peak of distribution
• Median = in middle
• Mean = closest to tail of distribution
16 16
16
3. Measures of Variability
• Asks question: "What are the
differences between scores?"
• Three Different Methods:
1. Range
2. Variance
3. Standard Deviation
A. Range
• Highest score - lowest score
• General indicator of dispersion, higher
range = higher deviations
• Advantages:
• Easy to compute
•Limitations:
• Takes into account only two most
extreme scores
• Extremely sensitive to single outlier
• Sample range poor estimate of
populational range
• Miss patterns in data
B. Variance
• Mean of the squared deviations from
the mean
2
=populational s 2
=sample
• Requires at least interval level measure
•Definitional Formulas
X
2
2
N
X X
2
s
2
n 1
•Calculational Formulas
X 2
X 2
2
N
N
X 2
X n 2
s
2
n 1
• Advantages:
• Good measure of dispersion or
variability
• Allows for partitioning of
distribution
• Limitations:
• Measure is in squared units
C. Standard Deviation
• Square root of variance (or mean of the
squared deviation from the mean)
=populational s =sample
• Returns unit of measure back to
original unit of measure
•Definitional Formulas
X
2
N
X X
2
s n 1
•Calculational Formulas
X 2
X 2
N
N
X 2
X n 2
s n 1
•Advantages:
• Average distance of mean and scores
in original units of measure
• Accurate measure of dispursion
Graphing Data
• Market Analogy (Henry, 1995)
• Does the audience have access?
• Is the audience motivated?
• Meaningful data = sufficient to initiate and
maintain interest without overloading
• Does the targeted audience have ability to
understand?
• Graphical competence: literacy, numeracy,
graphicacy
Graphical Competence (Henry, 1995)
1. Knowledge of the Subject
Supply:
2. Choose Meaningful Data
Researcher
3. Tailor Display Designs
4. Display of Data
5. Access to Data Display
Demand:
6. Motivation to Know
Audience
7. Interpret Display
8. Knowledge of the Subject
Bar Graphs
• Qualitative Data (Nominal\Ordinal)
• Width of the bars is constant
• Bars separated by constant distance
• Normally height of bar corresponds to
frequency of category
• Concerns:
• Orientation (horiz vs. vertical)
• Grid lines
• Axes & Tickmarks
• Fill
• Order
Figure 1.
Prevalence of Eye Color
6
5
4 Blue
Frequency 3
Brown
Green
2
Black
1
0
Eye Color
Elements needed:
•Identification (Figure #)
•Title
•Labels\Headings
Remember: Figure should read like a
self-contained paragraph.
Quantitative Data (Interval\Ratio)
• Histogram (similar to Bar Graph)
• Okay to put breaks in axis where set of
values omitted
• Bar widths represent real limits
• Therefore, touch
• Keep bar widths constant
• Frequency Polygon
• Values represented as points above
interval
Figure 2.
Scores on First Exam
15
10
Frequency
5
0
95-99 94-90 89-85 84-80 79-75 74-70 69-65 64-60 59-55 54-50
Test Scores
Elements needed:
•Identification (Figure #)
•Title
•Labels\Headings
Remember: Figure should read like a
self-contained paragraph.
Figure 3.
Scores on Psyc 489 First Exam
15
10
Frequency
5
0
95-99 94-90 89-85 84-80 79-75 74-70 69-65 64-60 59-55 54-50
Test Scores
Elements needed:
•Identification (Figure #)
•Title
•Labels\Headings
Remember: Figure should read like a
self-contained paragraph.
Correlation
•Measure of Association
• Describes the degree of relationship
that exists between two variables
• Bivariate relationships
• Descriptive statistic
•Represent relationship graphically
1. Direction of Relationship
•Positive
Y
X
•Negative Y
X
2. Form of Relationship
•Linear Y
X
Y
•Curvilinear
X
3. Degree of Relationship
•Strong
Y
X
•Weak
Y
X
•Correlation Coefficient
• A quantitative description of the
magnitude and direction of the linear
relationship between two variables
Number of different algorithms used
to calculate coefficient
• Example:
• phi
• point biserial
• tetrachoric
• Spearman's rank order
• Kendall's tau
•Depends upon:
1. Scale of measurement
(Nominal, Ordinal, Interval, Ratio)
2. Continuous or discrete
3. Characteristics of distribution
(normal or skewed)
Pearson's Product Moment
Correlation Coefficient (1896)
rxy = correlation between x and y
• Calculation based on deviation from
the mean for each variable
•Uses of coefficient:
1. Prediction - if related systematically
use one variable to predict the other
2. Validity - measures of the same
construct should have high degree
of relationship
3. Theory verification - test specific
predictions
4. Reliability - relationship across time
or separate parts of test
•Covariance
• Average of the multiplied deviation
scores
• Average of the cross products of
deviation scores
• Measure of the extent to which two
variables co-vary from their respective
means across an entire group
• If cross products consistently positive,
their average will be positive.
• If cross products consistently negative,
their average will be negative
Definitional Formula
X X Y Y
Covxy n 1
Computational Formula
X Y
XY
Covxy n 1
n
• Covariance = same as variance with
two different variables
• Variance = sum of squares
• Covariance = sum of products
• Larger the absolute number = stronger
relationship
•Limitations
• Larger ranges = larger deviations
• As with variance, not in original units
• Possibility of different scales of
measure
Correlation
• Standardized scale: Therefore, range =
+1.0 to -1.0
degree to which X and Y vary together
r = degree to which X and Y vary separately
covariability of X and Y
r = variability of X and Y separately
Definitional Formula
Cov xy
r sx s y
Zx Z y
r n 1
Computational Formula
X X Y Y
r
X X Y Y
2 2
Computational Formula
X Y
XY
r X 2
n
Y 2
X 2
nx
Y 2
ny
X Y (X-X) (Y-Y) (X-X)(Y-Y)
6 3 1 -3 -3
9 10 4 4 16
4 7 -1 1 -1
2 4 -3 -2 6
4 6 -1 0 0
=25 30 =0 0 =18
X=5 Y=6
Covxy = 18 = 3.6
5
X Y (X-X)2 (Y-Y)2
6 3 1 9
9 10 16 16
4 7 1 1
2 4 9 4
4 6 1 0
=25 30 =28 30 (28x30=840)
X=5 Y=6
Corxy = 18 = 18 = .62
840 29
X Y X2 Y2 XY
6 3 36 9 18
9 10 81 100 90
4 7 16 49 28
2 4 4 16 8
4 6 16 36 24
=25 30 =153 210 =168
X Y
XY
r X 2
n
Y 2
X2 nx
Y2 ny
25 30
168
r 5
153 25 2 210 30 2
5
5
168 150
r
153 125 210 180
r 18
840
18
29 .62
•Interpretation of r
1. rxy = ryx
• Correlation of one variable on
another is same reversed.
2. Size of r
• Scale is ordinal
• Arbitrary what you decide to be
low, moderate or high
3. Linear relationship between two
variables
4. Correlation is measure of association,
not causality
•Correlation necessary, but not
sufficient to demonstrate causal
relationship exists
•Factors that affect r
1. Sample characteristics
• cut outliers = lower r (decrease sd)
• only use extreme groups = higher r
2. Combining populations
• population correlations may differ
• population means may differ on one or
both of the variables
Pos/Neg = Two no corr groups=
no correlation slightly positive corr
Linear Regression
Linear Regression
• Mathematical description of the
characteristics of a line that best
captures the relationship between two
variables
• Predictions of values are based on the
assumption that the relationship is
linear
• Statistical procedure for predicting one
variable from another variable by using
a linear prediction rule
•The Straight Line
Y = a + bX
where:
• a = y-intercept
• b = slope
• a = y intercept
Y = a + bX
= value of Y when X is 0
• b = slope of the line
= measure of the change in Y for
every change in X (rise/run)
= Y2 - Y1
X2 - X1
•Consider a and b to be fixed constants
How to predict scores
Regression coefficents
sy
b r xy sx
Cov xy
b sx 2
Regression coefficents
Y = a + bX
a = Y - bX
sy
a Y r xy sx X
X Y X2 Y2 XY
6 3 36 9 18
9 10 81 100 90
4 7 16 49 28
2 4 4 16 8
4 6 16 36 24
=25 30 =153 210 =168
X=5 Y=6
X Y
XY
r X 2
n
Y 2
X 2
nx
Y 2
ny
25 30
168
r 5
153 25 2 210 30 2
5
5
168 150
r
153 125 210 180
r 18
840
18
29 .62
X 2
X 2
sx n 1
n
25 2
153
sx 4
5
sx 28
4 2.64
Y 2
Y n
2
sy n 1
30 2
210
sy 4
5
sy 30
4 2.74
sy
b r
b .62 2 .74
2 .64 xy sx
sy
a Y r X
.64
xy sx
a 6 .62 2.74
2.64 5
2.8
Y = 2.8 + .64X
Regression Line
.64 = rise = b
1.0 run
a=2.8
Y = 2.8 + .64X
Solving for Y'
Y' = a + bX
Y' = [Y - rxysyX] + [ rxysyX ]
sx sx
Y' = Y - rxysy(X - X)
sx
Y' = Y + b (X-X)
sy
•Solve for: rxy = 0 b rxy sx
sy
b = 0 (sy/sx) = 0 a Y r xy sx X
a = Y - 0(X) = Y
: rxy = 1
b = 1 (sy/sx) = sy/sx
a = Y - (sy/sx)X = Y - sy/sxX
•As the absolute value of r increases,
so too does the absolute value of b
Prediction Error
• Unless r = 1.0 or -1.0,
Y' rarely = Y
• Looking for an average difference
between predicted scores and
obtained scores
• Evaluate how well the regression line
“fits” the data
Total Dev. = Explained Dev. + Unexplained Dev.
(Y - Y)2 = (Y' - Y)2 + (Y - Y')2
Total Variation Explained Variation
(Y - Y)2 (Y' - Y)2
Unexplained Variation (Y - Y')2
Coefficient of Determination
•Proportion of total variance in one variable
that is predictable from another variable
r2 = Explained variation = (Y' - Y)2
Total variation (Y - Y)2
r=.62 r2=.38
Coefficient of Nondetermination
•Proportion of total variance in one
variable that is not predictable from
another variable
1 - r2 = Unexplained variation = (Y - Y')2
Total variation (Y - Y)2
r=.62 1 - r2=.62
Variance error of estimate (s2Y-Y‟)
(or Residual variance)
•Average prediction error
(conceptually similar to variance)
= assume
errors
normally
distributed
Definitional
Y Y ' 2
2 SSY Y '
s Y Y ' df n 2
Computational
n 1
s 2
Y Y ' s Y (1 r )
2 2
n2
s 2
Y Y ' s Y (1 r )
2 2 n 1
n2
s2Y-Y‟ = 7.5(.62) 4
3
s2Y-Y‟ = 7.5(.62)(1.33)
s2Y-Y‟ = 6.18
Standard error of estimate (sY-Y‟)
• Square root of residual variance
• Returns back to original scale of
measurement
Computational
sY Y ' s Y (1 r )
2 2 n 1
n2
sY-Y‟ = 6.18 = 2.49
• When r = 0, Y' = Y,
thus the prediction errors Y-Y' = Y-
Y
• When r = 1, Y' = Y,
thus the prediction error Y-Y' = 0
2
Chi-Square Statistic
Descriptive vs. Inferential statistics
• Descriptive statistics = summarization
of data such that important features
become salient (only concern is with
sample)
• Inferential statistics = infer
characteristics of larger group from
data collected from sample
Hypothesis Testing Review:
Hypothesis:
• prediction of fact based on theory
• define an expectation about outcome
Null Hypothesis (H0):
• data represent some chance outcome
of random events
Alternative Hypothesis (H1):
• data represent nonrandom,
systematic effect
• Strategy is to assume that H0 is correct,
until one has evidence that it is not
• Reject H0 based on pre-established
criteria (alpha-level)
• If difference within criterion: reject
H0
• Accept H1, but do not prove its
truth
• If difference not within criterion, fail
to reject H0
• Never accept or affirm H0
.025 .136 .341 .341 .136 .025
-2 -1 0 1 2
• Region of Rejection = set of least
likely, most extreme possible outcomes
which are consistent with H1
• When the event occurs within this
region, assert that results are not due to
random factors
Decision Making and Error:
• Type I Error:
• Reject H0 when it is true (falsely
conclude systematic relationship exists)
• Experimenter sets level
• Type II Error:
• Failure to reject H0 when it is false
(falsely accept H1)
• Function of alpha level and sample size
()
(as alpha level lowered, increases beta
error)
ACTUAL STATE OF NATURE
H0 True H0 False
I
N
H0 TYPE II
F CORRECT
True ERROR
E
R
E
N TYPE I
H0 CORRECT
C False ERROR
E
•Chi-Square
• Nonparametric test
• Measurement of data is nominal or
ordinal
• Data = frequencies
• Categories must be mutually
exclusive
•Hypotheses concern distribution
of frequencies
1. Goodness of fit
• Sample distribution corresponds to
some actual or theoretical distribution
A. No preference between two choices
B. No difference between two groups
C. No change in distribution over time
D. No difference from theoretical
distribution
2. Contingency test
• Independence of two categorical
variables
• Asks question whether or not variables
are systematically related
•Therefore, interested in difference
between observed (O) and expected (E)
frequencies
O E
2
2
E
2 O2
E N
O E
2
2 2(1) = 4
E
Heads Tails
E 50 50 100
O 60 40 100
O-E 10 10
(60-50)2 + (40-50)2 = 100 + 100 = 4
50 50 50 50
df = k-1 = 2-1 = 1
N
2
2 O 2(1) = 4
E
Heads Tails
E 50 50 100
O 60 40 100
O-E 10 10
(60)2 + (40)2 - 100 = 72 + 32 - 100 = 4
50 50
df = k-1 = 2-1 = 1
Interpretation of chi-square
• Distribution similar to t and F
• Therefore, need to keep track of df
(df = k-1), where k= # groups
• 30+ df to approach z distribution
(z=22-2df-1)
Test of independence
• Relationship between two variables,
H0=no relationship
• Expected frequencies should be equal
across categories or determined a
priori
• Sum column and row totals, determine
freq., calculate chi2
• df = (r-1)(c-1)
O Norm R-G B-Y
M 320 70 10 400
F 580 10 10 600
900 80 20 1000
E Norm R-G B-Y
M 360 32 8 400
F 540 48 12 600
900 80 20 1000
Expected: 400(900) = 360, 400(80) = 32
1000 1000
2 = (320-360)2 + (580-540)2 + (70-32)2 + (10-48)2
360 540 32 48
+ (10-8)2 + (10-12)2
8 12
= 4.44 + 2.76 + 45.12 + 30.08 + .5 + .5
= 83.44
2(3) = 83.44
df = (3-1)(2-1) = 2
•Residual
• Quantitative difference between
obtained and expected frequencies
O E
group residual E
• Larger values = greater discrepancy
between obtained and expected
frequencies
• Square and sum residuals = Chi-square
value
Heads Tails
E 50 50 100
O 60 40 100
O-E 10 10
(60-50) = 10 = 1.41
50 7.07
Data Collection Techniques
Major Issues of Data Collection
1. Sources of Data
2. Form of Data
3. Amount of Data
4. Accuracy\Reliability of Data
5. Design Fit
1. Sources of Data
Two broad categories:
1. Primary Data = information obtained
exclusively for current research
2. Secondary Data = archival or
previously recorded information
Advantages of Primary data
collection include:
• Collection based on researcher's need
• Control over measurement selection
and execution
•Advantages of Secondary data
collection include:
• Little cost or time required to access
data
• Not confined to immediate level or unit
of analysis
2. Form of Data
A. Self-Report Data
• Subject provides account of attitude,
opinion, memory, personal
characteristics, or circumstances
• Examples:
• Surveys\Questionnaires
• Interviews
B. Observational Data
• Recording of events, actions, or
circumstances of behavior
• Examples:
• Naturalistic Inquiry
• Participatory Observation
C. Archival Data
• Previously collected data
• Examples:
• Prior research
• Procedural documents
3. Amount of Data
• Determine amount of data needed to
conduct study
• Data sources, time periods, and
number of units sampled
• Involves sampling techniques
•Sampling
• Aim of sampling is to equate unknown
characteristics that may influence
variation and to preserve the
representativeness of the sample
•Rummel's Data Cube:
sample populations, occasions, and
characteristics
populations
occasions
characteristics
Two Classes of Sampling
Techniques:
1. Nonprobability Sampling
2. Probability Sampling
1. Nonprobability Sampling
• Common feature is that subjective
judgments are used to determine the
population that are contained in the
sample
A. Convenience sampling = select
cases based on their availability
for the study
B. Judgmental sampling - select cases
based on some purpose
(Most similar\dissimilar,
Typical or Critical cases)
C. Systematic Sampling - select cases based
on some predefined criteria
(Interval sampling or Snowball)
Every 4th
Advantages of Nonprobability
sampling
• Expedient, low effort\cost methods
• Useful in exploratory research
2. Probability Sampling
• Common feature is that each unit in
the population has a known, nonzero
probability of being included in the
sample
A. Simple Random Sample - Each member
of the study population has an equal
probability of being selected
B. Stratified Random Sample - Each
member of a population is assigned
to a group or stratum, then random
sample is drawn from each stratum
(ensures levels represented)
C. Proportional Random Sample - Each
member of a population is assigned to
a sub-group, then representative
sample is drawn from each group
proportional to population
Advantages of Probability
Sampling
• Objective standards remove possibility
of unknown confounds
• Intent to remove bias in selection
process
4. Accuracy and Reliability of Data
• Issues of data quality: validity,
reliability and utility of measurement
• Reduction of error in measurement
5. Design Fit
• Statistical Conclusion Validity
• Utility (Efficiency/Generality)
Survey Research Methods
Research Planning Model
• Research Definition Stage (I)
• Understand the Problem
• Identify Questions (Hypothesize)
• Research Design\Plan Stage (II)
• Choose Design
• Data Collection Approaches
Understand the Problem
STAGE 1:
RESEARCH
Identify Questions
DEFINITION
Refine/Revise Questions
STAGE 2:
RESEARCH Choose Design
PLAN/DESIGN
Determine Trade-offs Inventory Resources
Assess Feasibility
Design of Survey\Questionnaire
Instrument
• As with research plan in general,
survey design is an iterative process
• Ask self series of questions:
1. What do you need to know?
• Based on hypotheses identified in
Stage I of Research Planning Model
• Most important question researcher
can ask before begin writing
• Survey-type instruments can yield
three types of information
A. Reports of Fact - self-disclosure of
some objective information (e.g., age,
gender, education, behavior)
B. Ratings of Opinion or Preference -
evaluative response to statement
(e.g., satisfaction, agreement,
like\dislike)
C. Reports of Intended Behavior -
self-disclosure of motivation or
intention (e.g., likeliness, willingness)
2. How will administration be
accomplished?
A. Self-administered surveys - subject
responds to printed questions (e.g.,
group or mail surveys)
•Advantages
• Ask questions with long, complex or
visual response categories
• Ask batteries of similar questions
• Respondent does not share answers
with immediate person
•Disadvantages
• Careful questionnaire design is
required
• Open response questions not useful
• Good reading and writing skills by
respondents are needed
• Very little quality control over
administration
B. Other-administered surveys
• Subject responds to questions directly
posed by researcher (e.g., interview,
phone survey)
•Advantages
• Most effective in enlisting cooperation
(initial and length)
• Opportunity to answer respondent
questions and ensure quality of data
(e.g., probe adequate answers, answer
all questions)
• Rapport and confidence building
possible
•Disadvantages
• Cost and time requirements
• Adequate training of staff
• Accessibility of sample
3. What type of population are
you sampling?
• Very important consideration before
one begins to write questions
• Consider number of qualities
respondents possess:
A. Level of Education
(specifically reading level)
B. Limits of Attention
(avoid fatiguing respondents)
C. Motivation
(why is respondent going to
participate)
4. What type of response format
is appropriate (for each question)?
A. Open-ended Questions - permits
subject freedom to answer question in
own words (without pre-specified
alternatives
•Advantages
• Obtain unanticipated answers
• May better reflect respondents
thoughts\beliefs
• Appropriate when list of possible
answers is excessive
•Disadvantages
• Flexibility in responses difficult to
code and analyze
• Provide incomplete or unintelligible
answers
B. Close-ended Questions - subject
selects from list of pre-determined,
acceptable responses
Types of closed-ended questions
1. Checklists - respondent selects certain
number of pre-specified categories
(nominal data)
Types of Exercises:
Aerobics
Basketball
Swimming
Weightlifting
2. Two-way (Forced Choice) -
respondent must select between two
alternatives (crude ordinal\nominal)
Do you always wake
up before 8:00am?
Yes No
3. Ranked - respondent must place items
in order of importance or value
(ordinal)
Rank in order of importance:
Coursework
Part-time employment
Party hardy
Close relationship
4. Multiple-Choice (Likert) - respondent
selects between range of alternatives
along pre-specified continuum
(ordinal\interval?)
Strongly Strongly
Agree Agree NeutralDisagree Disagree
1 2 3 4 5
Advantages of Closed-ended
Questions
• Obtain more reliable answers
• Meaning of responses more
meaningful to researcher
• Straight-forward analysis
Disadvantages of Closed-ended
Questions
• Answers relative to response scale
provided
• Respondent's choice not among listed
alternatives
• Choices listed communicate kind of
response wanted
Writing good survey questions
• Differences in answers should stem
from differences among respondents
rather than differences in the stimuli
• Question's wording is obviously a
central part of the stimulus
Survey Ploys:
1. Did you kill your wife: YES NO
2. What percent of men kill their wives?
10% 30% 60% 90%
3. Lots of men are killing their wives these days. Did you by
any chance kill yours? YES NO
4. Complete the following sentence:
•“Killing my wife .”
5. Tell me a story about this picture:
1. Simple sentences
• No double negatives
• Eliminate vagueness (poorly defined
terms)
• Objectionable\Irrelevant questions
•No double negatives
It is not the case that I have never cheated on my tax
returns.
Never should one not help others.
*The likelihood of depresion recurring after the
discontinuation of psychotropic drug treatment is
greater than if drug treatment is never used as part
of therapy.
•Eliminate vagueness
•poorly defined terms
•ensure consistent meaning for all respondents
How many times in the past year have you talked with
a doctor about your health?
Is health care easily accessible for your family?
*Studying accounts fo a majority of activities I do at
college.
*Tests are stressful.
*I relax by using drugs.
•Objectionable\Irrelevant questions
•ruin rapport with respondent
•yield missing data
How old are you?
Have you ever tested positive for HIV virus?
Have you answered each question truthfully?
How many years of education were you able to
complete?
*I believe crack is one of the four food groups.
2. Discrete questions\responses
• No double-barrel questions
• Balanced questions\responses
• Exhaustive\mutually exclusive
categories
•No double-barrel questions
Is your doctor friendly and reasonably priced?
Were your caregivers courteous and friendly?
*I am often figeting and on edge.
*I find that I am more attentive and remember more if
I have eaten before a study session.
•Balanced questions\responses
•removes potential response bias
How was the service at this hospital?
Excellent Very Good Great
Are you depressed frequently?
Sad is the best descriptor of me right now.
My depressed mood keeps me from doing fun things.
•Exhaustive\mutually exclusive categories
•ensure accurate data collection
•ensure no duplicate responses
What is your age?
under 10 10-20 20-30 30-40 40-50
How did you last travel to the supermarket?
car, bus, foot, walking, public transportation
What is your maritial status?
single, married, divorced
3. Limit response format (7±2)
• Even vs. Odd categories
• Allow expression of variability
Strongly Agree Disagree Strongly
Agree Disagree
Strongly Agree Neutral Disagree Strongly
Agree Disagree
4. Match response to item
• Frequency (Never-All the time)
• Likert Scaling (Disagree-Agree)
• Quality (Poor-Excellent)
• Service (Not Well-Extremely Well)
5. Overall Format
• General to specific order of questions
• Employ "filtering" questions (If “Yes”)
• Mix question\response types to remove
response bias
• Minimize judgment and emphasize
accuracy (social desirability)
Survey Data Analysis
Data Entry
• Process of taking completed
questionnaires\surveys and putting
them into a form that can readily be
analyzed
• Series of options need to consider:
1. Decide on a file format.
A.The way the data will be
organized
• Order of information collected
• How subject is referenced
B. Constructing individual records
• History of 80-column format
• Application to statistics programs
2. Devise code for analysis
• Set of rules that translates answers
into discrete values
• Alphabetical or Numerical depending
on measurement scale
• Preserve level of measurement for
each item
General Considerations
(closed questions):
A. Make coding translation simple (or
non existent!)
• Minimize effort and risk of coding
errors
• Facilitates data interpretation
Item-level:
• Leave #s as #s (#s can be nominal).
• Reverse coding/Unfolding complex
response formats.
•Test-level:
• Code questions in order of appearance.
• Be consistent in assigning values with
similar responses
• Identify question groups within test.
B. How missing data are treated
1. Non ascertained Information:
information not obtained because of
interviewer or respondent performance.
• Failure to ask question
• Failure to obtain appropriate
response
• Refusal to answer question
(separate)
2. Inapplicable Information:
information does not apply to a
particular respondent
3. Unknown information:
information as to respondent's claim
of awareness (How to treat "Don't
know" option)
3. Entry of Data
• Number of translation steps between
subject's response and readable data
file
•Computer assisted techniques: 1
•Digital answer format (Scantron): 3
•Entry by hand: 4
• Impacts ability to check quality of data
entry (accuracy, reliability)
4. Clean Data File
• Examine each data file to ensure each
record is complete and in order
• Remove non-legal codes
• Replace with information from original
response format
• Importance of verification
Data Analysis
• Organization of data to better
understand the distribution
• Ordered listing or array of scores
Frequency Distribution
• Record the frequency (i.e., # of
respondents) within each category
• Numerical Table (representation)
• Use original measurement scale of
each item
Frequency Table
• Qualitative Data: relative frequencies
1. Proportion - frequency within
category divided by total number
2. Percentage - proportion (100)
n prop %
Clinical 16 .40 40
Counseling 8 .20 20
Experimental 10 .25 25
Social 6 .15 15
Total 40
Frequency Table
• Quantitative Data: cumulative freq
1. Proportion\Percentage
2. Cumulative Frequency
3. Cumulative Percentages
n cumf prop cum%
4 Strng Agree 8 40 .20 100
3 Agree 14 32 .35 80
2 Disagree 12 18 .30 45
1 Strng Disagree 6 6 .15 15
Total 40
Graphs
• Qualitative Data: Bar Graph
1. Width of bars constant
2. Bars separated by constant dist
3. Height of bar corresponds to freq
of category
Figure 1. Student Eye Color
Freq
30
25
20
15
10
5
0
Blue Brown Green Black
Graphs
Quantitative Data:
1. Histogram
• Similar to Bar Chart
• Bar width represents real limits
Figure 2. Parking expense by years of education
E
x $50
p $40
e $30
n $20
s $10
e $0
1 2 3 4 5 6 7
Graphs
Quantitative Data:
2. Frequency Polygon
• Values represented as points
• Express time\duration
Figure 3. Parking expense by years of education
E
x $50
p $40
e $30
n $20
s $10
e $0
1 2 3 4 5 6 7
Interview Research Methods
Research Planning Model
• Research Definition Stage (I)
• Understand the Problem
• Identify Questions (Hypothesize)
• Research Design\Plan Stage (II)
• Choose Design
• Data Collection Approaches
Understand the Problem
STAGE 1:
RESEARCH
Identify Questions
DEFINITION
Refine/Revise Questions
STAGE 2:
RESEARCH Choose Design
PLAN/DESIGN
Determine Trade-offs Inventory Resources
Assess Feasibility
Design of Survey\Questionnaire
Instrument
• As with research plan in general,
survey design is an iterative process
• Ask self series of questions:
1. What do you need to know?
A. Reports of Fact
B. Ratings of Opinion or Preference
C. Reports of Intended Behavior
2. How will administration be
accomplished?
• Self-administered surveys
• Other-administered (Interviews)
Administration of Interviews
and Phone Surveys
• Main Advantages:
Enlisting cooperation, Ability to
monitor quality of response, Rapport
• Main Disadvantages:
Cost\Time requirements, Interviewer
training, Sample accessibility
Development of Questions:
• Again full consideration given to:
3. What type of population are
you sampling?
A. Level of Education
B. Limits of Attention
C. Motivation
4. What type of response format is
appropriate for Interviews?
• Open-ended questions:
generally used in other-administered
surveys (possibility to clarify
ambiguities)
• Close-ended questions:
limited utility to response formats that
have few number of alternate
responses
Ask for each question
Role of the Interviewer
• Interviewer plays central role in
gathering of data
• Constantly monitoring of potential
influence or bias
Interviewer Objectives:
1. Locate and enlist cooperation of
selected respondents
• Available when respondent is
• Confident assertive style in
presenting study
• Responsive to personal needs and
concerns of each individual
2. Motivate respondents to do well
• Important role in setting respondent
performance
• Speed of asking question influences
speed of response
• Encouragement influences how
subjects perceive task
• Explicit and Implicit actions may bias
subject‟s resp
3. Ensure data meets question
objectives: Standardization
A. How study presented:
• Common understanding of purpose
of study, confidentiality,voluntary
nature of project, and context of
interview
B. How questions are asked:
• Wording of question
• Placing of vocal emphasis
• Filtering:
• Funnel (General to specific
questions)
• Inverted Funnel (Specific to general)
C. Probing:
• Standard follow-up questions that are
non-directive
• Addition Probe: elicits additional
information ("I see,""Yes," "Uh-huh,"
"I understand")
• Reflecting Probe: asks respondent to
consider answer provided (mirroring)
("What do you mean by ")
• Transitional Probe: extends range of
response to include other topics
("Another...")
• Situational Probe: queries about
specifics of situation in which
respondent reacts
(What was it like..?")
• Emotion Probe: Reflects depth or
degree of feeling associated
(How do you feel about..?)
• Climate Probe: queries about how
respondent feels about the interview
context
(“You seem to be uncomfortable
answering these questions. Is there
some problem talking about this
matter?")
D. How answers recorded:
• Verbatim answers (taped?) for open-
ended questions,
• Appropriate response for close-ended
questions
E. How rapport managed:
• Emphasis placed on professional rather
than personal interaction
• Interviewers should not:
• tell stories about self
• indicate judgment of respondents
answer, or opinions about subject
matter
Interview Data Analysis
Data Entry
• Process of taking completed
questionnaires\surveys and putting
them into a form that can readily be
analyzed
• Series of options need to consider:
1. Decide on a file format.
• Order of information collected
• How subject is referenced
2. Devise code for analysis
• Set of rules that translates answers
into discrete values
• Preserve level of measurement for
each item
General Considerations
(closed-ended questions):
A. Make coding translation simple (or
non existent!)
B. How missing data are treated
• Nonascertained Infromation
• Inapplicable Information
• Unknown information
General Considerations
(open-ended questions):
A. Record interview data verbatim
(code responses later)
B. Multiple or Single raters
C. Importance of objective coding
schemes
D. Determine code a priori or
a posteriori
Context of Discovery
Context of Justification
3. Entry of Data
4. Clean Data File
Interview Data Analysis
•Organization of data to better
understand the distribution
• Frequency Distribution
• Frequency Table
• Qualitative Data: relative frequencies
• Quantitative Data: cumulative freq
•Graphs
• Qualitative Data: Bar Graph
• Quantitative Data: Histogram &
Frequency Polygon
•Other Analyses:
• Chi-square statistic
• Pearson Product-Moment Correlation
(closed)
• Content Analysis
Content Analysis
• Method of studying and analyzing
communications in a systematic,
objective, and quantitative manner
• As much a method of observation as
method of analysis
•Steps of Content Analysis
1. Define and Categorize Universe of
Content
• Identification of variables of interest
• Implications for coding scheme:
a priori or a posteriori
2. Units of Analysis (Berelson, 1954)
• Words (types or frequencies)
• Themes (persevering concept)
• Character (projective)
• Items (publication)
• Space-and-Time Measures (physical
measure)
3. Analysis
A. Quantification
• Nominal frequency
• Judges rank or rate other responses
B. Enthnographic Reporting
- -
Observational
Research Methods
Research Planning Model
• Research Definition Stage (I)
• Understand the Problem
• Identify Questions (Hypothesize)
• Research Design\Plan Stage (II)
• Choose Design
• Data Collection Approaches
Understand the Problem
STAGE 1:
RESEARCH
Identify Questions
DEFINITION
Refine/Revise Questions
STAGE 2:
RESEARCH Choose Design
PLAN/DESIGN
Determine Trade-offs Inventory Resources
Assess Feasibility
Design of Observational
Instrument
• As with research plan in general,
survey design is an iterative process
• Ask self series of questions:
1. What do you need to know?
• Based on hypotheses identified in
Stage I of Research Planning Model
• Most important question researcher
can ask
• Have to be selective: Can‟t observe
everything
2. How will observation be
accomplished?
• Variably intrusive: How far intrude?
• From what vantage point do make
observations?
A. Secret Outsider
• Distant observer unknown to
participants in natural setting
• Non-intrusive
• Major Disadvantage: Removed from
the immediacy of the action
B. Recognized Outsider
• Firsthand observer made known to
participants
• Intrusive: Hawthorne effect
• Western Electric Company
(Roethlisberger & Dixon, 1939)
• Effect of known observation: increase
performance
• Minimize Hawthorne effect by ss
adapting to presence
• Major Disadvantage: Recognized
authority affiliation
C. Marginal Participant
• Adopt position of commonly accepted
and unimportant participant
• Non-intrusive as long as subjects
unaware of being observed
• Choice of clothes\objects carried
• Physical posturing
•Major Disadvantage: Familiarity
with situation influences what data is
recorded
D. Full Participant
• Adopt position of central importance
in situation
• Intrusive if not perceived as resident
or fail to meet membership criteria
• Major Disadvantage: Ability to
unintentionally change others behavior
3.What type of population are
you sampling?
• Very important consideration before
one begins to write questions
• Consider number of qualities
respondents possess:
A. Access to subjects' behavior
B. Amount of activity
C. Awareness of being observed
4. How will observation be
recorded?
• Recording devices depend upon:
• Detail required of information
gathered
• Amount previously known
(exploratory vs. confirmatory)
A. Notation
• Recording behavior in verbal or
diagrammatic notes
• Decide what to record and what to
overlook on the spot
• Write as much as possible as event
occurs (steno mask)
B. Pre-coded Checklists
• Select from series of pre-coded
checklists
• Decide important information
beforehand (participant characteristics,
place, time, situational characteristics)
• Requires previous diagnostic
evaluation
C. Maps
• Recording of activity
on floor plans, diagrams or maps
• Convenient to record behavior of
several people in one general area at
the same time
• Useful to record sequences of behavior
D. Photographs\Videotape
• Accurate recording of behavior
• Record only what expect (direction of
lens, focus, other noises)
• Requires additional observation of
images captured
•What to observe?
• Actor (Who is?)
• Act (doing What?)
• Significant others (with Whom?)
• Relationships (in what Relationship?)
• Sociocultural context (in what
Context?)
• Physical setting (Where?)
• Time (Across what times?)
•Actor
• Identify populations of interest
• Identify characteristics of interest
• Level of analysis (individual or group)
•Act
• Decide on level of abstraction
(molar vs molecular)
• How distinguish between individual
acts from connected sequence of acts
(episode)
•Significant Others
• Acts partly defined by how other
people are or are not included as
participants
• Influence of presence and absence
•Relationships
• Connections and separations between
people
• Meaningfulness of relationships among
participants
•Context
• Situational and cultural contexts
• Reactions under variety of conditions
•Time
• Time scale of observation vs time scale
of behavior
• Important element to be sampled
•Physical Situations
• Environment as influence of behavior
• Rules that reflect in behavior based on
surrounding environment
•Observing Physical Traces
• Systematically looking at physical
surroundings to find reflections of
previous activity
• Purposive and unintentional behaviors
•Inferences:
• How environment acquired its current
state
• How people actually use space
• How people feel toward surroundings
• How environment meets needs of users
•Advantages
• Unobtrusive
• Durable (traces do not quickly
disappear)
Types of traces
A. By-products of use
(Reflect what people do in settings)
1. Erosions = use wears away parts of
the environment
2. Leftovers = physical objects as the
result of some activities get left behind
3. Missing traces = indication of what
people are not doing
B. Adaptions for use
(Reflect how people change surrounding
environment)
1. Props = addition or removal of objects
from setting
2. Separations = division of spaces
formerly together
3. Connections = physical adaptions to
allow interaction or movement
C. Displays of self
(Communication)
1. Personalization = expression of
uniqueness and individuality
2. Identification = allow others to
recognize territory
3. Group membership = displays of
affiliation
D. Public messages
1. Official = communicate publicly in
settings designed for it
2. Unofficial = communicate publicly in
settings not designed for it
3. Illegitimate = not planned or approved
Methods of Recording
A. Annotated Diagrams
• Note where objects located during
interview\survey
• Standard symbols to refer to objects
B. Photographs
• Useful in generation of hypotheses
and determining categories
• Decide what to include beforehand
C. Counting
• Quantification of traces
• Number or metric
• Mechanical or Observer
Observational
Research Methods
Research Planning Model
• Research Definition Stage (I)
• Understand the Problem
• Identify Questions (Hypothesize)
• Research Design\Plan Stage (II)
• Choose Design
• Data Collection Approaches
Understand the Problem
STAGE 1:
RESEARCH
Identify Questions
DEFINITION
Refine/Revise Questions
STAGE 2:
RESEARCH Choose Design
PLAN/DESIGN
Determine Trade-offs Inventory Resources
Assess Feasibility
Design of Observational
Instrument
• As with research plan in general,
survey design is an iterative process
• Ask self series of questions:
1. What do you need to know?
• Based on hypotheses identified in
Stage I of Research Planning Model
• Most important question researcher
can ask
• Have to be selective: Can‟t observe
everything
2. How will observation be
accomplished?
• Variably intrusive: How far intrude?
• From what vantage point do make
observations?
A. Secret Outsider
• Distant observer unknown to
participants in natural setting
• Non-intrusive
• Major Disadvantage: Removed from
the immediacy of the action
B. Recognized Outsider
• Firsthand observer made known to
participants
• Intrusive: Hawthorne effect
• Western Electric Company
(Roethlisberger & Dixon, 1939)
• Effect of known observation: increase
performance
• Minimize Hawthorne effect by ss
adapting to presence
• Major Disadvantage: Recognized
authority affiliation
C. Marginal Participant
• Adopt position of commonly accepted
and unimportant participant
• Non-intrusive as long as subjects
unaware of being observed
• Choice of clothes\objects carried
• Physical posturing
•Major Disadvantage: Familiarity
with situation influences what data is
recorded
D. Full Participant
• Adopt position of central importance
in situation
• Intrusive if not perceived as resident
or fail to meet membership criteria
• Major Disadvantage: Ability to
unintentionally change others behavior
3.What type of population are
you sampling?
• Very important consideration before
one begins to write questions
• Consider number of qualities
respondents possess:
A. Access to subjects' behavior
B. Amount of activity
C. Awareness of being observed
4. How will observation be
recorded?
• Recording devices depend upon:
• Detail required of information
gathered
• Amount previously known
(exploratory vs. confirmatory)
A. Notation
• Recording behavior in verbal or
diagrammatic notes
• Decide what to record and what to
overlook on the spot
• Write as much as possible as event
occurs (steno mask)
B. Pre-coded Checklists
• Select from series of pre-coded
checklists
• Decide important information
beforehand (participant characteristics,
place, time, situational characteristics)
• Requires previous diagnostic
evaluation
C. Maps
• Recording of activity
on floor plans, diagrams or maps
• Convenient to record behavior of
several people in one general area at
the same time
• Useful to record sequences of behavior
D. Photographs\Videotape
• Accurate recording of behavior
• Record only what expect (direction of
lens, focus, other noises)
• Requires additional observation of
images captured
•What to observe?
• Actor (Who is?)
• Act (doing What?)
• Significant others (with Whom?)
• Relationships (in what Relationship?)
• Sociocultural context (in what
Context?)
• Physical setting (Where?)
• Time (Across what times?)
•Actor
• Identify populations of interest
• Identify characteristics of interest
• Level of analysis (individual or group)
•Act
• Decide on level of abstraction
(molar vs molecular)
• How distinguish between individual
acts from connected sequence of acts
(episode)
•Significant Others
• Acts partly defined by how other
people are or are not included as
participants
• Influence of presence and absence
•Relationships
• Connections and separations between
people
• Meaningfulness of relationships among
participants
•Context
• Situational and cultural contexts
• Reactions under variety of conditions
•Time
• Time scale of observation vs time scale
of behavior
• Important element to be sampled
•Physical Situations
• Environment as influence of behavior
• Rules that reflect in behavior based on
surrounding environment
•Observing Physical Traces
• Systematically looking at physical
surroundings to find reflections of
previous activity
• Purposive and unintentional behaviors
•Inferences:
• How environment acquired its current
state
• How people actually use space
• How people feel toward surroundings
• How environment meets needs of users
•Advantages
• Unobtrusive
• Durable (traces do not quickly
disappear)
Types of traces
A. By-products of use
(Reflect what people do in settings)
1. Erosions = use wears away parts of
the environment
2. Leftovers = physical objects as the
result of some activities get left behind
3. Missing traces = indication of what
people are not doing
B. Adaptions for use
(Reflect how people change surrounding
environment)
1. Props = addition or removal of objects
from setting
2. Separations = division of spaces
formerly together
3. Connections = physical adaptions to
allow interaction or movement
C. Displays of self
(Communication)
1. Personalization = expression of
uniqueness and individuality
2. Identification = allow others to
recognize territory
3. Group membership = displays of
affiliation
D. Public messages
1. Official = communicate publicly in
settings designed for it
2. Unofficial = communicate publicly in
settings not designed for it
3. Illegitimate = not planned or approved
Methods of Recording
A. Annotated Diagrams
• Note where objects located during
interview\survey
• Standard symbols to refer to objects
B. Photographs
• Useful in generation of hypotheses
and determining categories
• Decide what to include beforehand
C. Counting
• Quantification of traces
• Number or metric
• Mechanical or Observer
Observation Data Analysis
Data Entry
1. Decide on a file format.
2. Devise code for analysis
3. Entry of Data
4. Clean Data File
Data Analysis
• Frequency Distribution
• Frequency Table
• Qualitative Data: relative frequencies
• Quantitative Data: cumulative freq
•Graphs
• Qualitative Data: Bar Graph
• Quantitative Data: Histogram &
Frequency Polygon
•Other Analyses:
• Chi-square statistic
• Pearson Product-Moment Correlation
• Behavioral Maps (spatial analysis)
Sits: 2min
Stands>
SS talk re:
Unable to leave weather
same time
Case Study
Characteristics of Case Study
1. Intensive investigation of an
individual unit of analysis
• Person or Group
• Situation
• Occasion
2. Reliance on anecdotal evidence
• Life “histories”
3. Absence of experimental controls
• Viewed as lower end of continuum
of experimental control
• Vary to extent to which valid
conclusions can be drawn
Type of Data
1. Anecdotal information
• Narrative accounts (retrospective)
• Reports of Others
• Informal observations of researcher
2. Objective information
• Systematic and Quantifiable data
• Bolster causal inferences
Because of numerous threats to
validity and causal inference:
Case studies must use multiple
measures to control for possible
threats to valid conclusions.
Assessment Occasions
• Number of separate assessments
• Timing of sampling occasions
• Different Types of Schedules:
1. Pre- and Post-assessment
• Necessity of baseline assessment to
determine effects, if any, of treatment
• Even if objective measure, haven‟t
ruled out history, maturation, testing,
or instrumentation
Treatement
Pre-assessment Post-assessment
2. Repeated assessment
• Necessity of baseline assessment
• Includes removal and reintroduction
of treatment
• Ethical and practical implications
• Necessity of multiple cases to rule out
history and maturation threats
From Quattrochi-Tubin & Jason (1980)
Baseline Refreshments Baseline Refreshments
Number
of
Elderly
Participants
= Television watching
= Interactions
= Attendance
Subject Characteristics
1. Number of subjects
• Greater number, less chance
extraneous factor responsible for
change
• Removal of “threats”
2. Heterogeneity of subjects
• Variety of demographic, histories,
and maturational differences
Advantages of Case Study
Method
1. Much cheaper than large-sample
research
2. Better opportunity to investigate
corollary hypotheses
3. Detailed information
• breadth and depth
• “thick” descriptions
Advanced Research
Design
The game of science is, in principle, without end.
He who decides one day that scientific statements
do not call for any further test, and that they can be
regarded as finally verified, retires from the game.
Sir Karl Popper
The Logic of Scientific Discovery
Science is simply common sense at its best: that is,
rigidly accurate in observation, and merciless to
fallacy in logic
Thomas H. Huxley
On the Origin of Species
Two Criticisms of Scientific
Approach:
1. Thomas Kuhn (1962\1970)
Structure of Scientific Revolutions
2. Paul Feyerabend (1975, 1981)
Against Method
Problems of Empiricism
1. Thomas Kuhn
Structure of Scientific Revolutions
• Basic Premise is that Science is not a
steadily accumulating body of
knowledge
• Historical process of competition
amongst segments of scientific
community
Paradigm
• Framework for characterizing
phenomena that a particular discipline
takes as its subject matter
• General (meta) theory that instructs
how scientific theories or models are to
be developed and applied in further
research
• Includes:
• General theoretical assumptions
• Methodological techniques
• Development of Paradigms =
maturational process
• Series of Five Stages
• Characterized by type of activities
Stage 1:
Pre-science or Immature Science
• Scientific activities not guided by
generally accepted paradigm
• Number of competing schools of
thought
• Disagree about theory and what
constitutes observational phenomena
Stage 2:
Normal Science
• One school of thought adopted
• Unites scientific community into one
research program
• Extension of theory and method to
other problems or content areas
Stage 3:
Crisis!!
• Unsolved problems or anomalies build
up over course of application
• Increasing number of anomalies and
reduced rate of progress
• Begin to doubt future potential of
paradigm = relaxation of rules
Stage 4:
Revolutionary Science
• Active struggle between defenders of
old paradigm and proponents of new
paradigm
• Each one tries to solve the greatest
number of anomalies
• Incommensurability = inability to
directly compare because different
theory\methods
Stage 5:
Resolution
• One paradigm becomes dominant
• Generates new period of Normal
Science (Stage 2)
• Because of incommensurability, choice
between paradigms is fundamentally
not rational, but matter of taste or
preference
Stage 1
Pre-Science Unification
Stage 5 Stage 2
Generates new
Resolution Normal Science
Anomalies build
Domination
Stage 4 Stage 3
Revolution Crisis!!
Competition
2. Paul Feyerabend
Against Method
• Basic Premise is scientific methods
are inherently theory-laden or
dependent
• Scientific methods, therefore, are not
commensurable nor objective in
evidence they provide
Solution:
Methodological Anarchism
• Because not one method provides
objective information, use all sorts of
methods
• Denies existence of any sound, rational
principles of science (Reliability,
Validity, Utility)
• “Anything Goes”
Science
•Not a state, but a process
•Critical examination of methods used
to obtain knowledge and assess the
validity of the same
All sciences, in spite of the manifold differences, have
in common that they are devoted to endeavor to understand
the world
Ernest Mayr
The Growth of Biological Thought
Critical Multiplism
• Direct response to criticisms of Kuhn
and Feyerabend
• Strategy = approach entire research
situation as opposed to the specific
tactics (methods) used
• Emphasis placed on multiple methods
• But, not just mindless multiplism of
“Anything Goes”
• Want to be “Critical” in methods one
selects
• Identify strengths, biases, and
assumptions of different methods or
theories one selects
• Develop “Heterogeneity of Bias”
Thoughtless Thoughtful
Single Mindless Rigid
Method Monism Monism
poor science Newtonian Science
Rear-End Validity MAX-MIN-CON
Multiple Mindless Critical
Methods
Multiplism Multiplism
poor science
Anything Goes
The point of planned multiplism is not that more is
better--more operations, more methods, more measures
and more theories. The point is to provide tools to
help scientists explore the boundaries of their
knowledge.
William Shadish
Critical Multiplism: A Reserach Strategy and Its Attendant Tactics
The men of experiment are like the ant, they only collect and
use; the reasoners resemble spiders, who make cobwebs out
of their own substance. But the bee takes the middle course:
it gathers its material from the flowers of the garden and of
the field, but transforms it and digests it by a power of its own.
Francis Bacon
The Novum Organum
Critical Multiplism as Design
Strategy
1. Multiple Working Hypotheses
• Consideration given to multiple
predictors by seeking alternative
hypotheses and rival causal theories
• Allows for strong inference about
validity of conclusion
2. Multiple Outcomes
• Specification of complex causal
network by identification of multiple:
• Criterion variables
• Stakeholders
• Utilities
• Ecological or Systems approach
Simple Reality Complex Reality
S B S1 B1
S2 B2
S3 B3
Sophisticated Reality
S1 B1
S2 B2
S3 B3
S4 B4
3. Multiple Indicators
• Examine converging and diverging
operations of multiple measures
• Permits use of fallible indicators,
because can assess error and bias
• a.k.a. Multiple Operationalism
4. Multiple Replications
• Integration of different research by
different investigators having
different theoretical approaches
• Enhances Generalizability
•Rummel's Data Cube:
sample populations, occasions, and
characteristics
populations
occasions
characteristics
Two Fundamental World Views
of Science
1. Newtonian World View
2. Darwinian World View
1. Newtonian World View
A. Elementalistic = phenomena broken
down into parts
B. Essentialistic = ideal “types” or
categories (nomothetic)
C. Deterministic = simple causality
vary one variable and hold all others
constant
•Experimentally translates into:
• Systematic Designs (MaxMinCon)
• Rigid Experimental Control
• Null Hypothesis Testing
• Single Experiments (Critical)
The world is a machine
2. Darwinian World View
A. Systemic = phenomena inherently
associated with other phenomena
B. Individualistic = populational
thought, individual differences or
idiographic (idionomothetic)
C. Probabilistic = complex causality
embrace indeterminacy
•Experimentally translates into:
• Representative Designs (Ecological)
• Statistical Control
• Parametric Testing
• Multiple Experiments
(Critical Multiplism)
The world is a jungle
Applied Research Ethics
What are Ethics?
What are Values?
1. Normative Ethics
• Fundamental concern = development
and justification of systems of moral
rules which guide conduct
• Statements of ought
2. Descriptive Ethics
• Fundamental concern = accurate
identification of moral rules which
actually do guide conduct
• Statements of fact
What are Research Ethics?
• Set of principles to assist community
of experimenters in deciding which
goals are most important in reconciling
conflicting values
• Characteristically Normative
Ethical concerns divided into three
areas: (Diener & Crandall, 1978)
1. Relationship between society and
science
2. Professional issues
3. Treatment of subjects
1. Relationship between society
and science
• Extent to which societal concerns and
cultural values should and do direct the
course of scientific investigation
• How much autonomy?
• How much responsibility to societal
needs?
A. Direct social influence = research
incentives, grants, requests for
proposals (RFPs)
B. Indirect social influence = culturally
based interests
2. Professional issues
• Scientific misconduct as specified and
regulated by professional organizations
and groups
• Peer Review process
A. Fraudulent activity
• Presentation or publication of forged,
falsified, or manipulated data
• Between 1950 and 1979: 14 cases
• Between 1980 and 1987: 26 cases
• Cyril Burt
B. Research Publication Issues
• Plagiarism = present substantial
portions or elements of another's work
or data as their own
• Publication Credit = Authorship and
publication credit for only work
actually performed
• Partial Publication = publication of
several articles based on one large set
of data
• Dual Publication = publishing the same
data and results in more than one
journal or publication
C. Financial Conflict of Interest
• Fiducial interest of investigator
confounded with production of
research
Development of APA Ethical
Principles in the Conduct of Research
with Human Participants
•History
• Committee on Ethical Standards in
Psychological Research (1953)
• Questionnaire: 9,000 + psychologists
• Interview: 35 researchers written on
topic of research ethics
• Dissemination of draft
• City, state, regional and national
meetings
• Published in Monitor
• Revised draft gained acceptance in
1973 by general membership of APA
• Proposed revision every five years
• Latest revision: December, 1992
• Present 10 Principles published in
1982, ease of understanding
Ethical Principles
•Principle A:
In planning a study, the
investigator has responsibility to
make a careful evaluation of its
ethical acceptability.
•Principle B:
Determine degree of risk to subject
(deception, stressful conditions, or
take medication).
•Principle C:
Investigator always retains
responsibility for ethical practice.
•Principle D:
Prior to conducting research, the
investigator must disclose obligations
and responsibilities of both subject
and investigator (influence willingness
to participate).
Not for: anonymous surveys or naturalistic
observation.
•Principle E:
If deception is going to be used:
(1) determine if justified;
(2) identify alternatives to deception,
if any;
(3) ensure participants provided with
sufficient explanation as
immediate as possible.
•Principle F:
Investigator respects individual's
freedom to decline at any time during
course of experiment.
•Principle G:
Investigator protects participant from
physical and mental discomfort, harm,
and danger that may arise from
research procedures.
•Principle H:
After collection of data, researcher
provides participant with information
about the nature of the study
(debriefing).
•Principle I:
If research procedures result in
undesirable consequences for
individual participant, investigator
has responsibility to detect and
remove or correct these consequences
(long-term).
•Principle J:
Information obtained about research
participant during the course of an
investigation is confidential unless
otherwise agreed upon in advance.