VIEWS: 25 PAGES: 3 POSTED ON: 11/12/2011
Construct Validity and Reliability Overview Research Methods (PSY 353) Dr. Hughes, fall 2011 Today we will think about issues of reliability and validity. As discussed at the beginning of class, the issues of variables and their measurement is important not only in a practical sense (operational definitions, construct validity), but also from a philosophical standpoint. Abstractly, what are we measuring? What is the meaning of the construct that we have in mind? What assumptions are we making about the construct and its measurement? All of these considerations might influence the way we design our measures and how we our design our experiments. One important distinction that should become clear after today’s lab is the difference between internal validity threats and construct validity. On the one hand we are talking about potential problems that might arise when you are doing experiments and on the other, we are talking about measurement scales. Identify different ways to demonstrate a measures validity Use SPSS to calculate the correlation between different dimensions/constructs in a career inventory and personality measure Determine, using evidence and logic, whether a career inventory is valid Determine what information is needed to make a validity judgment regarding a personality scale Calculate the reliability of a career inventory Recommend additional ways to provide evidence of reliability Part 1. Types of Validity Since today’s lab is focused on construct validity and reliability we will be analyzing measurement scales. Construct validity is an assessment of whether one measured what one set out to measure. Importantly, like all things in science, construct validity is not something you prove, but something you argue with evidence. This “evidence” often comes in form of quantitative measures of overlap (e.g., correlation or covariations). For example, say you are interested in creating a new scale of depression. To provide evidence of the scale’s validity you could show +/-/0 covariation Type of construct validity _________________ A validated depression scale _____________________ _________________ A scale of sadness _____________________ _________________ Happiness scales _____________________ _________________ Dancing joy scale _____________________ _________________ % change in depression later in life_____________________ In the first part of today’s lab, you will run some tests using SPSS and make some decisions regarding the construct validity of a measure. Then, we will examine scale reliability. For both examples we will use the data from your ORVIS survey. Part 2. Define Pearson’s product-moment correlation r. Using the textbook and help from your teammates, describe Pearson’s r conceptually. What is this statistic measuring? What kind of inference can we make based on this calculation? Open SPSS file with the classes ORVIS and TIPI data. What GUI menus must you use to calculate a correlation? Describe them with arrows below. I’ll start the GUI sequence for you: Analyze Part 3. Construct Validity: Please use the ORVIS & TIPI handout as you work through the problems below. A) Before we perform any computations, what are your expectations based on the descriptions of the ORVIS and Big-5 constructs? 1. Do you think any of the ORVIS constructs will be positively correlated with one another? 2. Do you think any of the ORVIS constructs will be negatively correlated with one another? 3. Do you think any of the Big-5 constructs will be positively correlated with one another? 4. Do you think any of the Big-5 constructs will be negatively correlated with one another? 5. Do you think any of the Big-5 constructs will be positively correlated with any of the ORVIS constructs? 6. Do you think any of the Big-5 constructs will negatively correlated with any of the ORVIS constructs? B) One way to show that an inventory is valid is to show that its constructs are independent. That is, if you have scale that purports to measure different, unique constructs then those constructs should not be correlated with one another. Let’s test this idea and see if there is evidence for the validity of the ORVIS. For each question, report the correlation as follows: r(N) = ___, p = ___. Calculate Pearson’s product moment correlation r between the leadership and creativity constructs. Calculate Pearson’s product moment correlation r between the altruism and creativity constructs. C) Another way to demonstrate validity is to show that things that should be related to one another are in fact related. Calculate Pearson’s product moment correlation r between the production and adventure constructs. D) Still another way to demonstrate validity of the ORVIS is to show that people who are already in an occupation (and presumably one they enjoy) score high on the constructs for which they should score highly (e.g., artist should score high on creativity). Which constructs would you predict that psychology majors would score high on? Test your expectation above by examining the mean scores for our class by category. E) Finally, test whether your expectations regarding the correlations between the Big-5 and the ORVIS constructs (from part A) were correct. List the two constructs and the correlations that you examine below. F) Go through questions B through E and try to guess which type of validity was tested. Place your answer under the question. Feel free to use the internet, your textbook, or any other method to guess. G) Is there evidence for the construct validity of the ORVIS? Explain. H) What additional information about the class would help you decide whether or not the Big-5 personality measure is a valid instrument? Part 4. Reliability. Examine whether the ORVIS demonstrates high test-retest reliability. Using the GUI menu in SPSS, go to Analyze Scale Reliability Analysis. Now, enter leadership time 1 and leadership time 2 into the box. Give the scale a label “leadership reliability” or something similar, and then click OK. Report the test-retest reliability using the alpha symbol α = ___(value)___ Examine the reliability of each of the scales and report them below. Overall, is the ORVIS a reliable measure? Why or why not?
Pages to are hidden for
"Validity and reliability lab"Please download to view full document