Lecture 4 - Survey design • Sampling • Sample size/precision • Data collection issues • Sources of bias Why do surveys? • Information on particular population – prevalence of a disease – behaviour, knowledge, attitude • Planning of services • Collect information on data not routinely available: – e.g., mental health status, health behaviours • Repeat surveys to monitor trends (serial cross- sectional studies) Bias and precision of the survey estimates • Bias: – selection bias relates to sample selection – information bias relates to information collected • Precision – relates to sample size Reasons to sample • Reduce cost • Increase accuracy and quality of data collected Definitions • Sampling unit – person or group (e.g., household) • Sampling frame – list of sampling units in the population • censuses • electoral lists • telephone lists • are institutional populations excluded (e.g., prisons, nursing homes) Target and study population • Target population: – population for generalization of results • Study population: – population for collection of data – may be total target population or a sample Types of sample • Non-representative – convenience – volunteers • Representative – simple random – systematic – cluster – multistage Simple random sample • Each sampling unit in the population has equal probability of being included • Sampling with replacement: – each unit placed back in pool • Sampling without replacement (usual method): – each unit selected is kept out of pool Simple random sample (cont’d) • Methods: – manual – tables of random numbers – computer-generated random numbers Systematic sample • Select every nth individual from a list – can use existing numbers – e.g., patient appointments, medical records • Advantages: – Does not require complete sampling frame – Simple to carry out • Disadvantages: – May be unsuitable for cyclic or ordered data (e.g., every 5th patient when only 5/day) Stratified sampling • Separate sample selected from different strata of population • Requires separate sampling frame for each stratum • Useful if there are small but important subgroups of the population (e.g., very old, very young, institutionalized, sick) Cluster sampling • Sampling frame comprises groups (households, villages, schools) • Step 1: Simple random sample of groups • Step2: All individuals in each group included in survey • Advantages: – enumeration of population not needed – more efficient use of resources Multistage sampling • Larger units sampled in first stage, smaller units later • e.g.: – stage 1 - sample of towns – stage 2 - sample of city blocks or census tracts – stage 3 - sample of households Sampling for “hidden populations” • Homosexual men: – gay bars, newspapers • Injection drug users: – convenience sample (e.g., treatment facilities) – snowball sampling (through networks) • Capture-recapture methods – identify biases of sampling method Planning a survey • Define target population • Select method of sampling – sampling unit, sampling frame, etc • Calculate sample size • Define survey data collection methods • Non-respondents – number of attempts to reach – different days, times Sample size estimations • Requirements: – level of precision (width of confidence interval) – expected variability (estimated from previous studies, pilot study, or literature) Design of questionnaires • List study variables • Collect existing questions and instruments • Adapt and/or develop new questions • Format questionaire • Pre-testing (timing, responses, clarity, etc.) • Revise, determine priorities, shorten Question wording: clarity • Use concrete rather than abstract terms, e.g., – During a typical week, how many hours do you spend doing vigorous exercise? – Not: How much exercise do you get? • Avoid jargon, technical terms, slang • Avoid double-negatives (Do you disagree that doctors should not make house calls?) • Use active vs passive voice (Has a doctor ever told you vs Have you ever been told by a doctor?) Question wording: clarity – Break long sentences into short ones (20 word or fewer) – Use good grammar but use informal style – Avoid hypothetical questions – Evaluate reading level (normally not more than 8th grade) Question wording: neutrality • Do not suggest desirable response, e.g.: – Not: do you ever drink alcohol? – Better: how often do you drink alcohol? • Give permission to give undesirable response e.g.: – Sometimes people forget to take medications their doctor prescribes. Do you ever forget (or how often do you forget) to take your medications? Question wording • Introduce attitude questions, e.g.: – People have different opinions about their medical care. We are interested in your opinion. • Avoid double-barreled questions – How much coffee or tea do you drink each day? • Avoid assumptions – How much help do you get from your family? Response wording • Make them short • Use as few options as possible • Consider different types of non-response: – refuse – don’t know – no opinion – not applicable – omission by subject or interviewer Response wording • Make sure responses are mutually exclusive (or give instructions to “check all that apply”) • Consider use of response card for multiple questions with same set of responses Organization of questionnaire • Group questions by subject matter • Introduce each group with short descriptive statement (e.g., now I am going to ask you some questions about your use of health services) • Begin with more emotionally neutral questions • More sensitive questions (e.g., income, sexual function) near end of questionnaire Organization of questionnaire • interviewer-administered: repeat time frame fairly frequently • self-administered: repeat time frame at top of each page or each set of questions, e.g.: During the past year, how many times have you: – Visited a doctor? – Been a patient in an emergency department? – Been admitted to hospital? Organization of questionnaires • Group questions with similar response scale • Format skip patterns – screener questions – branching questions • Time frame – group questions that ask about same time frame – “usual” behavior vs specified time period – assist respondent with milestones to help define reference time frame Questionnaire mode • Face-to-face • Telephone • Mail • Other: – diaries • Mixed mode Face-to-face interviews: advantages • reduce items with no response • easier for older, less educated, lack of fluency in language • some formats easier to administer: – skip patterns to avoid irrelevant questions – open-ended questions - can probe for more complete response Face-to-face interviews: disadvantages • cost • time • effort (interviewer training, evaluation of inter-rater reliability) • interviewer biases • differences in sociodemographic characteristics of interviewer and subject Telephone interviews: advantages • less expensive than face-to-face • reduce items with non-response • some formats easier to administer: – skip patterns to avoid irrelevant questions – open-ended questions - can probe for more complete response • large, representative samples can be organized from one office • avoids bias associated with appearance of interviewer Telephone interviews: disadvantages • misses households without telephone • misses those with unlisted ‘phone numbers • bias when calls made during day • multiple calls may be needed • perceived as intrusive by some • difficult to administer items with multiple response options Mailed questionnaires: advantages • least expensive • can be coordinated from one office • social desirability minimized • inconsistent results on completeness of reporting (e.g., for # MD visits) Mailed questionnaires: disadvantages • relatively low response rates – multiple mailings, cover letter, letterhead, advance warning, token of appreciation, SSAE • difficult to get information on non-respondents – differences between early and late responders • items may be omitted: 5-10% may be unusable • cannot control order of questions • postal strikes Analysis of surveys • Missing data – exclude – imputation: e.g., based on characteristics of respondents – sensitivity of estimate to method of imputation • Weighting of estimates – for stratified samples Analysis of surveys (cont’d) • Crude estimates, confidence intervals – Continuous data: Mean, median, quartile – Categorical data: proportion – Confidence intervals to describe precision Bias and precision of the survey estimates • Bias: – selection bias relates to sample selection – information bias relates to information collected • Precision – relates to sample size Selection bias in surveys • Does the final analysis sample represent the original target population? • Sources of bias: – sampling method – non-response – missing data Information bias in surveys • Bias in measurement of outcomes • Sources of information bias: – non-validated measurement instrument – unblinded or poorly trained data collectors – response set – etc.
Pages to are hidden for
"Surveys"Please download to view full document