"Bill of Materials for Construction Samples"
Sampling and Questionnaires Bill Reimer October 31, 2005 firstname.lastname@example.org http://reimer.concordia.ca/teaching 11/22/2010 Bill Reimer 1 Outline Sampling Questionnaire construction 11/22/2010 Bill Reimer 2 Sampling Objective Get a lot of information with little cost Sample Population 11/22/2010 Bill Reimer 3 Sample sizes for a population of 10,000 4500 4147 4000 3500 Sample Size 3000 2401 2500 95% Confidence 2000 99% Confidence 1500 1037 1000 600 461 500 96 166 150259 267 0 10% 8% 6% 4% 2% Range of acceptable error (Accuracy) 11/22/2010 Bill Reimer 4 3 Rules of Scientific Sampling each unit in the population has an equal chance of being chosen we must know the chance of each member being chosen each selection must be independent from the others 11/22/2010 Bill Reimer 5 To what do you want to generalize? People Households Trees Leaves These are the Units of Analysis 11/22/2010 Bill Reimer 6 To which units do you want to generalize? People in Canada? Women in Québec? Citizens of Montréal? Trees in boreal forests? This is the Population 11/22/2010 Bill Reimer 7 The Population Is the collection of units to which you want to generalize It may be abstract It may be concrete But it must be clear 11/22/2010 Bill Reimer 8 From what will you choose your sample? A list of units? A region or place? A drawer or computer file? A particular time? This is your sample frame 11/22/2010 Bill Reimer 9 The Sample Frame may not match the Population Population Sample Frame …but try to get them close 11/22/2010 Bill Reimer 10 Sampling Requirements: Identify the population Is the sample frame reasonable? Does it meet the 3 conditions of scientific sampling? We can seldom meet all 3 conditions! 11/22/2010 Bill Reimer 11 Types of Samples Probability Generalizability Efficient Non-probability Exploratory Strategic 11/22/2010 Bill Reimer 12 Simple Random Sampling Equal chance of being chosen We know the chance Each selection independent From: O’Leary, Z (2004) The Essential Guide to Doing Research, London: Sage, Ch 8. 11/22/2010 Bill Reimer 13 Systematic Sampling Random selection Determine sample rate (8/40) Random selection of 1st case Every nth case (n=5) Watch for regularities in the sample frame From: O’Leary, Z (2004) The Essential Guide to Doing Research, London: Sage, Ch 8. 11/22/2010 Bill Reimer 14 Stratified Random Sampling Separate sample frame into strata Take random (or systematic) sample from each strata May be proportionate or disproportionate From: O’Leary, Z (2004) The Essential Guide to Doing Research, London: Sage, Ch 8. 11/22/2010 Bill Reimer 15 Cluster Sampling Divide population into clusters Randomly select clusters Collect data on all cases in cluster Watch cluster composition From: O’Leary, Z (2004) The Essential Guide to Doing Research, London: Sage, Ch 8. 11/22/2010 Bill Reimer 16 Cluster Sampling - Montréal 11/22/2010 Bill Reimer 17 Non-Probability Samples Accidental Purposive 11/22/2010 Bill Reimer 18 Convenience Sample Choose most convenient people Subject to multiple biases From: O’Leary, Z (2004) The Essential Guide to Doing Research, London: Sage, Ch 8. 11/22/2010 Bill Reimer 19 Volunteer Sampling Ask for volunteers Directly Ads Selected groups Unclear biases From: O’Leary, Z (2004) The Essential Guide to Doing Research, London: Sage, Ch 8. 11/22/2010 Bill Reimer 20 Strategic Sampling Handpick respondents for strategic purposes Useful for marginalized or small groups Control biases by specifying criteria From: O’Leary, Z (2004) The Essential Guide to Doing Research, London: Sage, Ch 8. 11/22/2010 Bill Reimer 21 Snowball Sampling Identify strategic respondent Ask for referrals Continue the process Use multiple starts to avoid network biases From: O’Leary, Z (2004) The Essential Guide to Doing Research, London: Sage, Ch 8. 11/22/2010 Bill Reimer 22 Probability Samples – Multiple Mixture of various techniques 11/22/2010 Bill Reimer 23 Sample Size Depends On Size of the population Accuracy desired Confidence desired Variation in the phenomenon investigated 11/22/2010 Bill Reimer 24 Assignment Specify a research question you would like to answer. Describe a sampling procedure along with the data you would collect to answer the question. Identify the following: the population the sample unit the sample frame the type of sample 11/22/2010 Bill Reimer 25 Samples Non-probability Accidental Purposive Probability Simple random Systematic Stratified: • Proportionate • Disproportionate Cluster Multiple 11/22/2010 Bill Reimer 26 Constructing Questionnaires 11/22/2010 Bill Reimer 27 Types of Research Interviews Questionnaires Closed and some open-ended questions Semi-structured interviews Open and some closed-ended questions Unstructured interviews Open-ended questions Research Interviews are NOT what you see on TV 11/22/2010 Bill Reimer 28 Advantages and Disadvantages Questionnaires Semi-structured Economic High response rates Speedy Can ask complicated Minimal interviewer questions bias Can follow-up Qs Anonymity Can observe Low response rates Expensive Limited exploration Time consuming Comparison easy Comparison difficult 11/22/2010 Bill Reimer 29 Preparing Questions Resist the temptation to start with specific questions Clarify the RESEARCH question Identify the types of information required to answer the research question Within each of the types of information brainstorm for specific questions to ask Rearrange questions 11/22/2010 Bill Reimer 30 Question Order Leave sensitive issues for later - once rapport has been established Follow from general to specific Use tables and grids where appropriate Use cards for sensitive issues if appropriate Prepare your probes if interview Follow a natural flow 11/22/2010 Bill Reimer 31 Anticipate the Analysis Produce dummy tables Work backwards from your analysis 11/22/2010 Bill Reimer 32 Format examples Contingency question Full filter question Quasi-filter question Closed question Partially open question Open question Probes 11/22/2010 Bill Reimer 33 Simple, Mutually Exclusive What is your gender? (Circle the appropriate number) Female . . . . . . . . . . . . . . . 1 5 Male . . . . . . . . . . . . . . . . . 2 11/22/2010 Bill Reimer 34 Multiple Choice, not Mutally Exclusive What modes of transportation did you used to get to school last week? (Circle all that apply) • Metro . . . . . . . . . . . . . . . . . 1 6 • Bus . . . . . . . . . . . . . . . . . . . 1 7 • Car . . . . . . . . . . . . . . . . . . . 1 8 • Bicycle . . . . . . . . . . . . . . . . . 1 9 • Walk . . . . . . . . . . . . . . . . . . 1 10 • Other (please specify) _____________________ 1 |__|__| 11-12 11/22/2010 Bill Reimer 35 Rank-order Rank order the 3 most important aspects of your job (Place 1 beside the most important, 2 beside the next most important, and 3 beside the next most important) • Money . . . . . . . . . . . . . . . . . . |___| 13 • Security . . . . . . . . . . . . . . . . . |___| 14 • New experiences . . . . . . . . . . |___| 15 • Power . . . . . . . . . . . . . . . . . . |___| 16 • Excitement . . . . . . . . . . . . . . |___| 17 • Other (please specify) ________________________ |___| |__|__| 18-19 11/22/2010 Bill Reimer 36 Pretest, pretest, pretest Use your friends and family Discuss Where it was clear or not clear What they thought about when responding to each question What they thought you were getting at Move to colleagues Check it out on people who are similar to those you will eventually survey 11/22/2010 Bill Reimer 37 Problem Questions Jargon, slang, abbreviations What is your Internet browser? Ambiguity, confusion, vagueness What is your income? Emotional language and prestige bias Should we put murderous terrorists in jail? Do you support Prime Minister Chretien’s policy on world trade? Double-barrelled questions Should marijuana be legalized for medical or other purposes? Problem Questions – con’t Leading questions Do you feel that governments should have less power to interfere in business? Beyond respondent’s competence Does your mother support extramarital sex? False premises How can we halt the rising crime rate? 11/22/2010 Bill Reimer 39 Problem Questions – con’t Future intentions How will you vote in the next federal election? Overlapping or unbalanced response categories What is your opinion regarding abortion? • Very favourable • Favourable • Neutral 11/22/2010 • Opposed Bill Reimer 40 Questionnaire Construction The amount of time spent in the preparation of the questionnaire is directly related to the value of the results. Surveys usually cost a lot and you can't redo them. Know what you are wanting to discover 11/22/2010 Bill Reimer 42 Research Strategies Questionnaires for generalizability Exploration: well selected, in-depth interviews Use interviews to design survey Use comparisons 11/22/2010 Bill Reimer 43 Sampling and Questionnaires Bill Reimer October 31, 2005 email@example.com http://reimer.concordia.ca/teaching 11/22/2010 Bill Reimer 44