Construct Validity and Reliability Overview
Research Methods (PSY 353) Dr. Hughes, fall 2011
Today we will think about issues of reliability and validity. As discussed at the beginning of class, the issues of variables
and their measurement is important not only in a practical sense (operational definitions, construct validity), but also from
a philosophical standpoint. Abstractly, what are we measuring? What is the meaning of the construct that we have in
mind? What assumptions are we making about the construct and its measurement? All of these considerations might
influence the way we design our measures and how we our design our experiments.
One important distinction that should become clear after today’s lab is the difference between internal validity threats and
construct validity. On the one hand we are talking about potential problems that might arise when you are doing
experiments and on the other, we are talking about measurement scales.
Identify different ways to demonstrate a measures validity
Use SPSS to calculate the correlation between different dimensions/constructs in a career inventory and
Determine, using evidence and logic, whether a career inventory is valid
Determine what information is needed to make a validity judgment regarding a personality scale
Calculate the reliability of a career inventory
Recommend additional ways to provide evidence of reliability
Part 1. Types of Validity
Since today’s lab is focused on construct validity and reliability we will be analyzing measurement scales. Construct
validity is an assessment of whether one measured what one set out to measure. Importantly, like all things in science,
construct validity is not something you prove, but something you argue with evidence. This “evidence” often comes in
form of quantitative measures of overlap (e.g., correlation or covariations). For example, say you are interested in creating
a new scale of depression. To provide evidence of the scale’s validity you could show
+/-/0 covariation Type of construct validity
_________________ A validated depression scale _____________________
_________________ A scale of sadness _____________________
_________________ Happiness scales _____________________
_________________ Dancing joy scale _____________________
_________________ % change in depression later in life_____________________
In the first part of today’s lab, you will run some tests using SPSS and make some decisions regarding the construct
validity of a measure. Then, we will examine scale reliability. For both examples we will use the data from your ORVIS
Part 2. Define Pearson’s product-moment correlation r. Using the textbook and help from your teammates, describe
Pearson’s r conceptually. What is this statistic measuring? What kind of inference can we make based on this calculation?
Open SPSS file with the classes ORVIS and TIPI data. What GUI menus must you use to calculate a correlation?
Describe them with arrows below. I’ll start the GUI sequence for you:
Part 3. Construct Validity: Please use the ORVIS & TIPI handout as you work through the problems below.
A) Before we perform any computations, what are your expectations based on the descriptions of the ORVIS and Big-5
1. Do you think any of the ORVIS constructs will be positively correlated with one another?
2. Do you think any of the ORVIS constructs will be negatively correlated with one another?
3. Do you think any of the Big-5 constructs will be positively correlated with one another?
4. Do you think any of the Big-5 constructs will be negatively correlated with one another?
5. Do you think any of the Big-5 constructs will be positively correlated with any of the ORVIS constructs?
6. Do you think any of the Big-5 constructs will negatively correlated with any of the ORVIS constructs?
B) One way to show that an inventory is valid is to show that its constructs are independent. That is, if you have scale
that purports to measure different, unique constructs then those constructs should not be correlated with one another.
Let’s test this idea and see if there is evidence for the validity of the ORVIS. For each question, report the
correlation as follows: r(N) = ___, p = ___.
Calculate Pearson’s product moment correlation r between the leadership and creativity constructs.
Calculate Pearson’s product moment correlation r between the altruism and creativity constructs.
C) Another way to demonstrate validity is to show that things that should be related to one another are in fact related.
Calculate Pearson’s product moment correlation r between the production and adventure constructs.
D) Still another way to demonstrate validity of the ORVIS is to show that people who are already in an occupation (and
presumably one they enjoy) score high on the constructs for which they should score highly (e.g., artist should score
high on creativity).
Which constructs would you predict that psychology majors would score high on?
Test your expectation above by examining the mean scores for our class by category.
E) Finally, test whether your expectations regarding the correlations between the Big-5 and the ORVIS constructs (from
part A) were correct. List the two constructs and the correlations that you examine below.
F) Go through questions B through E and try to guess which type of validity was tested. Place your answer under the
question. Feel free to use the internet, your textbook, or any other method to guess.
G) Is there evidence for the construct validity of the ORVIS? Explain.
H) What additional information about the class would help you decide whether or not the Big-5 personality measure is a
Part 4. Reliability. Examine whether the ORVIS demonstrates high test-retest reliability. Using the GUI menu in SPSS,
go to Analyze Scale Reliability Analysis. Now, enter leadership time 1 and leadership time 2 into the box. Give the
scale a label “leadership reliability” or something similar, and then click OK.
Report the test-retest reliability using the alpha symbol α = ___(value)___
Examine the reliability of each of the scales and report them below.
Overall, is the ORVIS a reliable measure? Why or why not?