Reliability
Reliability
Reliability means repeatability or
consistency
A measure is considered reliable if it
would give us the same result over
and over again (assuming that what
we are measuring isn’t changing!)
IOP 301-T Mr. Rajesh Gunesh
Definition of Reliability
Reliability usually “refers to the consistency
of scores obtained by the same persons
when they are reexamined with the same
test on different occasions, or with
different sets of equivalent items, or under
other variable examining conditions
(Anastasi & Urbina, 1997).
Dependable, consistent, stable, constant
Gives the same result over and over again
IOP 301-T Mr. Rajesh Gunesh
Validity vs Reliability
IOP 301-T Mr. Rajesh Gunesh
Variability and reliability
What is the acceptable range of error in
measurement
– Bathroom scale ±1 kg
– Body thermometer ±0.2 C
– Baby weight scale ±20 g
– Clock with hands ±5 min
– Outside thermometer ±1 C
IOP 301-T Mr. Rajesh Gunesh
Variability and reliability
We are completely comfortable with a
bathroom scale accurate to ±1 kg, since
we know that individual weights vary
over far greater ranges than this, and
typical changes from day to day are
about the same order of magnitude.
IOP 301-T Mr. Rajesh Gunesh
Reliability
True Score Theory
Measurement Error
Theory of reliability
Types of reliability
Standard error of measurement
IOP 301-T Mr. Rajesh Gunesh
True Score Theory
IOP 301-T Mr. Rajesh Gunesh
True Score Theory
Every measurement is an additive
composite of two components:
1. True ability (or the true level) of the
respondent on that measure
2. Measurement error
IOP 301-T Mr. Rajesh Gunesh
True Score Theory
Individual differences in test scores
– “True” differences in characteristic
being assessed
– “Chance” or random errors.
IOP 301-T Mr. Rajesh Gunesh
True Score Theory
What might be considered error
variance in one situation may be true
variance in another (e.g Anxiety)
IOP 301-T Mr. Rajesh Gunesh
Can we observe the true score?
X = T + ex
We only observe the measurement,
we don’t observe what’s on the right
side of equation (only God knows
what those values are)
IOP 301-T Mr. Rajesh Gunesh
True Score Theory
var(X) = var(T) + var(ex)
The variability of the measure is the
sum of the variability due to true score
and the variability due to random error
IOP 301-T Mr. Rajesh Gunesh
What is error variance?
Conditions irrelevant to purpose of the
test
– Environment (e.g., quiet v. noisy)
– Instructions (e.g., written v. verbal)
– Time limits (e.g., limited v. unlimited)
– Rapport with test taker
All test scores have error variance.
IOP 301-T Mr. Rajesh Gunesh
Measurement Error
Measurement error:
–Random
–Systematic
IOP 301-T Mr. Rajesh Gunesh
Measurement Error
IOP 301-T Mr. Rajesh Gunesh
Measurement Error
Random error: effects are NOT
consistent across the whole sample,
they elevate some scores and depress
others
– Only adds noise; does not affect
mean score
IOP 301-T Mr. Rajesh Gunesh
Measurement Error
Systematic error: effects are generally
consistent across a whole sample
– Example: environmental conditions for
group testing (e.g., temperature of the
room)
– Generally either consistently positive
(elevate scores) or negative (depress
scores)
IOP 301-T Mr. Rajesh Gunesh
Measurement Error
IOP 301-T Mr. Rajesh Gunesh
Measurement Error
IOP 301-T Mr. Rajesh Gunesh
Theory of Reliability
IOP 301-T Mr. Rajesh Gunesh
Reliability
The variance of the true score
Reliability =
The variance of the measure
Var(T)
Reliability =
Var(X)
IOP 301-T Mr. Rajesh Gunesh
How big is an estimate of Reliability?
Subject variability
Reliability =
Subject variability + measurement error
Var(T) Var(T)
Reliability = =
Var(X) Var(T) + Var(e)
IOP 301-T Mr. Rajesh Gunesh
We can’t compute reliability because
we can’t calculate the variance of the
true score; but we can get an estimate
of the variability.
IOP 301-T Mr. Rajesh Gunesh
Estimate of Reliability
Observations would be related to
each other to the degree that they
share true scores. For example
consider the correlation between X1
and X2:
covariance ( X 1 , X 2 )
var ( X 1 ) var ( X 2 )
IOP 301-T Mr. Rajesh Gunesh
RELIABILITY
Stability Equivalence Internal consistency
Test-Retest Alternate-form Split-half
Inter-scorer Kuder-Richardson
Cronbach Alpha
IOP 301-T Mr. Rajesh Gunesh
Types of Reliability
1. Test-Retest Reliability
Used to assess the consistency of a
measure from one time to another
2. Alternate-form Reliability
Used to assess the consistency of
the results of two tests constructed
the same way from the same content
domain
IOP 301-T Mr. Rajesh Gunesh
Types of Reliability
3. Split-half Reliability
Used to assess the consistency of results
across items within a test by splitting them
into two equivalent halves
Kuder-Richardson Reliability
Used to assess the extent to which items are
homogenous when items have a dichotomous
response, e.g. “yes/no” items.
IOP 301-T Mr. Rajesh Gunesh
Types of Reliability
Cronbach’s alpha (α) Reliability
Compares the consistency of response of
all items on the scale (Likert scale or
linear graphic response format)
4. Inter-Rater or Inter-Scorer Reliability
Used to assess the concordance between
two or more observers scores of the same
event or phenomenon for observational
data
IOP 301-T Mr. Rajesh Gunesh
Test-Retest Reliability
Definition: When the same test is
administered to the same individual (or
sample) on two different occasions
IOP 301-T Mr. Rajesh Gunesh
Test-Retest Reliability:
Used to assess the consistency of a measure from one
time to another
IOP 301-T Mr. Rajesh Gunesh
Test-Retest Reliability
Statistics used
– Pearson r or Spearman rho
Warning
– Correlation decreases over time because error
variance INCREASES (and may change in
nature)
– Closer in time the two scores were obtained,
the more the factors which contribute to
error variance are the same
IOP 301-T Mr. Rajesh Gunesh
Test-Retest Reliability
Warning
– Circumstances may be different for
both test-taker and physical
environment.
– Transfer effects like practice and
memory might play a role on the
second testing occasion
IOP 301-T Mr. Rajesh Gunesh
Alternate-form Reliability
Definition:
Two equivalent forms of the same
measure are administered to the same
group on two different occasions
IOP 301-T Mr. Rajesh Gunesh
Alternate-form Reliability:
Used to assess the consistency of the results of two tests
constructed same way from the same content domain
IOP 301-T Mr. Rajesh Gunesh
Alternate-form Reliability
Statistic used
– Pearson r or Spearman rho
Warning
– Even when randomly chosen, the two
forms may not be truly parallel
– It is difficult to construct equivalent
tests
IOP 301-T Mr. Rajesh Gunesh
Alternate-form Reliability
Warning
– Even when randomly chosen, the two
forms may not be truly parallel
– It is difficult to construct equivalent
tests
– The tests should have the same number
of items, same scoring procedure,
uniform content and item difficulty
level
IOP 301-T Mr. Rajesh Gunesh
Split-half Reliability
Definition: Randomly divide the test
into two forms; calculate scores for
Form A, B; calculate Pearson r as
index of reliability
IOP 301-T Mr. Rajesh Gunesh
Split-half Reliability
IOP 301-T Mr. Rajesh Gunesh
Split-half Reliability
2rhh
rtt
1 rhh
(Spearman-Brown formula)
IOP 301-T Mr. Rajesh Gunesh
Split-half Reliability
Warning
The correlation between the odd and even
scores are generally an underestimation of
the reliability coefficient because it is
based only on half the test.
IOP 301-T Mr. Rajesh Gunesh
Cronbach’s alpha &
Kuder-Richardson-20
Measures the extent to which items
on a test are homogeneous; mean of
all possible split-half combinations
– Kuder-Richardson-20 (KR-20): for
dichotomous data
– Cronbach’s alpha: for non-dichotomous
data
IOP 301-T Mr. Rajesh Gunesh
Cronbach’s alpha (α)
IOP 301-T Mr. Rajesh Gunesh
Cronbach’s alpha (α)
n
st
2
2
si
n 1
2
st
(Coefficient alpha)
IOP 301-T Mr. Rajesh Gunesh
Kuder-Richardson (KR-20)
n
rtt
st
2
pq
n 1
2
st
IOP 301-T Mr. Rajesh Gunesh
Inter-Rater or Inter-Observer Reliability:
Used to assess the degree to which different raters or
observers give consistent estimates of the same phenomenon
IOP 301-T Mr. Rajesh Gunesh
Inter-rater Reliability
Definition
Measures the extent to which
multiple raters or judges agree when
providing a rating of behavior
IOP 301-T Mr. Rajesh Gunesh
Inter-rater Reliability
Statistics used
– Nominal/categorical data
•Kappa statistic
– Ordinal data
•Kendall’s tau to see if pairs of ranks for
each of several individuals are related
–Two judges rate 20 elementary school
children on an index of hyperactivity
and rank order them
IOP 301-T Mr. Rajesh Gunesh
Inter-rater Reliability
Statistics used
– Interval or ratio data
•Pearson r using data obtained from
the hyperactivity index
IOP 301-T Mr. Rajesh Gunesh
Factors affecting Reliability
Whether a measure is speeded
Variability in individual scores
Ability level
IOP 301-T Mr. Rajesh Gunesh
Whether a measure is speeded
For speeded measures, test-retest and
equivalent-form reliability are more
appropriate. Split-half techniques may
be considered if the split occurs
according to time rather than number
of items.
IOP 301-T Mr. Rajesh Gunesh
Variability in individual scores
Correlation is normally affected by the
range of individual differences in a
group. Sometimes, smaller subgroups
display correlation coefficients which
are completely different from that of
the whole group. This phenomenon is
known as range restriction.
IOP 301-T Mr. Rajesh Gunesh
Ability level
One must also consider the variability
and ability levels of samples. It is
advisable to compute separate
reliability coefficients for homogeneous
and heterogeneous subgroups.
IOP 301-T Mr. Rajesh Gunesh
Interpretation of Reliability
One must ask oneself the following
questions:
How high must the coefficient of
reliability be?
How is it interpreted?
What is the standard error of
measurement?
IOP 301-T Mr. Rajesh Gunesh
Magnitude of reliability coefficient
Anastasi & Urbina (1997) 0.8 – 0.9
Huysamen (1996)
at least 0.85 for individuals
at least 0.65 for groups
Smit (1996)
0.8 – 0.85 for personality & interest
at least 0.9 for aptitude
IOP 301-T Mr. Rajesh Gunesh
Standard Error of the Measurement
Definition: Estimate of the amount
of error usually attached to an
individual’s obtained test score
– As SEM ↑, test reliability ↓
– As SEM ↓, test reliability ↑
IOP 301-T Mr. Rajesh Gunesh
Standard Error of the Measurement
SEM s t 1 rtt
IOP 301-T Mr. Rajesh Gunesh
Standard Error of the Measurement
Confidence Interval: Uses SEM to
calculate a band or range of scores that
has a high probability of including the
person’s true score.
Example: 95% confidence interval means
only 5 times in 100 will the person’s
TRUE score lie outside this range of
scores.
IOP 301-T Mr. Rajesh Gunesh
Reliability
Formula: CI = Obtained score + z(SEM)
z = 1.0 for 68% level
z = 1.44 for 85% level
z = 1.65 for 90% level
z = 1.96 for 95% level
z = 2.58 for 99% level
IOP 301-T Mr. Rajesh Gunesh
Reliability of standardized tests
An acceptable standardized test should
have reliability coefficients of at least:
0.95 for internal consistency
0.90 for test-retest (stability)
0.85 for alternate-forms (equivalency)
IOP 301-T Mr. Rajesh Gunesh
Reliability: Implications
Evaluating a test
– What types of reliability have been
calculated and with what samples?
– What are the strengths of the
reliability coefficients?
– What is the SEM for a test score
– How does this information influence
decision to use and interpret test
scores?
IOP 301-T Mr. Rajesh Gunesh