Achieving Quality Data Seminar Series Through Survey Testing Elise Kennedy firstname.lastname@example.org Thanks for attending introduce self Overview What can testing do for me? Seminar Series Why should I test? Types of testing Analysing the results Developing a testing strategy Consider the benefits of testing types of testing including skirmishes through to dress rehearsals brief examples of analysing the results overview of how to design a testing strategy What can testing do for me? Seminar Series In this section I will go through some of the things testing can do for you - benefits What can testing do for me? Testing can help you assess the: Seminar Series processing system processing system Find out the automatic optical character reading system misreads a question crucial to output. If hadn't tested then all 10,000 forms need to be altered by manual intervention. What can testing do for me? Testing can help you assess the: Seminar Series processing system cost cost - get a good estimate here particularly if interviewer based, how much will you have to pay them? Can you afford it? If employing a company to conduct for you then the duration of interview is important - we review a lot of CATI surveys where interview length is fixed cost say 15 minutes and if goes over costs more What can testing do for me? Testing can help you assess the: Seminar Series processing system cost organisation of fieldwork organisation of fieldwork - Are the interviewer instructions working? What can testing do for me? Testing can help you assess the: Seminar Series processing system cost organisation of fieldwork questionnaire Questionnaire for example are the data available? can data be found in records or not? What can testing do for me? Testing can help you assess the: Seminar Series variability of responses Range: using ranges to collect turnover, all respondents are ticking the same range. Could change the ranges variability - sample could be significantly reduced. Eg $ value of rent paid for warehouse space in certain geographical area. testing: not much variability may be able to reduce sample size What can testing do for me? Testing can help you assess the: Seminar Series variability of responses non-response rate non-response rate - questionnaire sensitive high rate of refusals need to rethink. Eg questions relating to how waste is dealt with from an environment department What can testing do for me? Testing can help you assess the: Seminar Series variability of responses non-response rate method method: Want to collect financial information over the phone, CATI tests show people can't find data during the interview, call backs necessary and high item non-response. test again on mail based find higher response better quality data What can testing do for me? Testing can help you assess the: Seminar Series variability of responses non-response rate method sample design Find that focus groups provide enough info. Eg client feedback would they like the training to be presented in an online format focus groups resounding no. Change to a two phase approach - interested in ordering on line - testing shows that most respondents do not complete most of the form 90% two forms one short one long short determines interest in online and long asks more specific questions about online ordering habits What can testing do for me? Testing can help you assess the: Seminar Series variability of responses non-response rate method sample design list of businesses List of businesses - testing very useful if unsure of how many units are out of scope or dead use yellow pages but you want only those who have a website. testing gives an idea of proportion who have a website and whether you need to use a new frame Why should I test? Testing Seminar Series will improve quality of data can be done cheaply is as important as the sample design We have people say all the time; we don't have the $ or the time to test Have person saying I have 2 weeks, contracted for 800 interviews 50 more for a test will cost us extra We weigh up the alternatives benefits stated before: Apply a question and find 50% of people misinterpret it - data are rubbish Seminar Series Types of Testing Set scene before you test expect that you have undertaken a development process literature search consulted experts or others with similar experience Going to talk about 6 types of testing, this does not cover all options for testing out there but main ones not all are relevant to all types of research depends on aims (talk about later) testing should be flexible and aimed to be cost effective these are tools to cover now and how to use get the most out of will be covered later Types of Testing Skirmish Seminar Series Focus Groups Cognitive Interviewing Observational Studies qualitative Types of Testing Pilot tests Seminar Series Dress Rehearsal quantitative but with some exceptions Seminar Series Skirmish What is a Skirmish? An informal test where you show the form to a few people you know Note many different definitions of a skirmish here I have chosen the very basic one. You probably do this anyway EXAMPLE. So say my area is starting a new collection, we need to collect data on salary sacrifice. I have no idea about the topic and no form to copy, no one else has done any research about it what do I do? Makes logical sense to write some questions and take the form round the office or to people with experience or knowledge in area or with surveys Can be used in development of new questions too for some topics might be appropriate to speak to industry boards and associations or lobby groups particularly for common terms Use with mates in the industry Skirmish Can identify understanding of definitions and Seminar Series terms how answers are formulated possible range of answers level of knowledge required flaws in design awkward wording So with my Salary sacrifice example - what can I get out of a skirmish? Do they know what salary sacrifice means? Do they use the same definition as me? I might find that there are other words like salary packaging, non-cash remuneration When I ask them how much did they sacrifice Do they think of the last financial year or the last week? range of types of salary sacrifice I thought of cars and super but there are others like share options, childcare and school fees, self education - can add them as categories on my final collection Do they need to know the tax law to answer? it is complex or easy? questionnaire flaws, routing instructions, superfluous questions wording does it sound good or bad? Skirmish Benefits Seminar Series compare alternative designs Limitations Can not be used as only form of testing respondents not as diverse as true population With my sacrificing survey I have two designs with different questions can use skirmish to see which one is better easier to read Once I've skirmished not stop there do other forms of testing, mainly because I've only asked some of my mates in big business and the public service what about people from non-profit organisations who have a much broader range of allowances under the law? Focus Groups What is a focus group? Seminar Series A facilitated discussion about the form with people who represent the target population Groups of approximately 5-8 In my example before salary sacrifice find out that non-profit organisations do have so much more rich diverse experience than I was aware of. v important that I collect info correctly from them. They have a great impact on my outputs for my survey I decide to conduct a focus group I line up some representatives from NPIs not too many so everyone can talk but big enough to obtain a range of ideas Employ a professional facilitator who is aware of leading a group discussion, not leading respondent specific outcomes well articulated prior develop a discussion guide Allow 2hrs on the day Focus Groups Can Identify Seminar Series culture of respondent understanding of concepts abilities of group reaction to form best person in business to complete the questionnaire Focus group helps determine things like Culture: people in these organisations who look after the finances are generally part timers, there is high turnover and records are kept but in very different ways, Some organisations no $ to buy computer systems. So I can keep this in mind. Ask for last pay period records, simplify questions into broad categories, understanding - words, definitions are there acronyms or terms commonly used by the group, eg terminology used. Find out common terminology Discover they are diverse in their educational background. simple words to ensure all understand reaction to form - layout, best person in business to complete - Find out as expect that financial related questions should be directed to the financial officer but should be addressed to the HR officer Focus Groups Limitations needs a skilled facilitator Seminar Series tendency for problems to be exaggerated targets only sub groups of the population resource intensive should be used with other forms of testing In general there is some advice: Groups can get out of hand so best to have an experienced facilitator Sometimes the groups do not provide answers at all. Need to have a protocol written prior with clear aims/answers want covered Even when you conduct lots of groups the groups are relatively small so we can't represent all types of respondents in the target population Not representative it should be used as the sole method of testing As with any group there can be a very vocal person, their problems may be exaggerated, alternatively problems can be exaggerated just due to tendencies in groups to agree with one another. Finally not good if want to extract specific information about the business - sensitive Cognitive Interviewing What is a cognitive interview? Seminar Series An in depth one-on-one interview with a respondent where the interviewer asks questions to determine how the respondent constructs answers Cog interview this time I will take my example from personal experience when I worked on the National Health survey Repeat survey, last time had a nutritionist survey some individuals but this time proposed to collect through interviewer based We were unclear on respondent understanding of question, their ability to consistently answer Cog test - good op to try to understand what is going on in mind of respondent Cognitive Interviewing Can identify Seminar Series impression created by questionnaire respondent thought processes understanding of concepts ease of answering questions..... One area of interest was eg how many serves of fruit/vegetables did you eat yesterday Thought processes include What they were thinking when answered - what fruit?, how did they formulate the answer did they think for breakfast I ate? or did they just know? time periods thought of - yesterday all day only main meals includes and excludes from a question what fruit? was there a consistent understanding. Was it easy to remember fruit they ate Was there a consistent understanding of fruit? Do all perceive the same eg tomatoes Cognitive Interviewing Can identify (continued) availability of data Seminar Series level of knowledge needed to answer awkward wording whether all questions interpreted similarly by all respondents Could they remember fruit? was it easier to remember yesterday than last week Was the wording easy or difficult? think about last week, that is the week beginning Monday 14 February and ending Sunday 20 February How many half cup fulls of cut fruit did you consume last week. The fruit could be chopped, fresh, whole and cooked. Business survey example cog testing showed that the target population did not understand or interpreted differently the term 'online' (lots in area non-English speaking background) when changed to over the Internet Cognitive Interviewing Limitations Seminar Series Respondents may not represent entire target population Respondent does not focus solely on questionnaire should be used with other forms of testing For our test we were aware that since interviewing during the day had mainly retired or at home parents. Did not have representation from workers, all education backgrounds We knew that only focus on one part of questionnaire, no idea about overall impressions of people of form. So schedule a pilot interviewers may bias the respondent Assumes respondent can verbalise how they construct their thoughts Observational Studies What is an observational study? Seminar Series Respondent completes the questionnaire in the presence of an observer Respondent then explains their understanding of the collection Back to salary sacrifice: Create the part of questionnaire I conducted some observational studies. Observer takes notes on the respondent body language or reactions as they complete the questionnaire Eg pauses, wrong directions to wrong question, where had to look up info Observer asks in retrospect - what they were thinking as they answered, where they got the info. Prompt the respondent Another example which is not quite observational testing - send form and ask them to do it then call after and ask for their response (Instructions given before and after) Observational Studies Can identify problem questions Seminar Series data availability time taken for particular questions form navigation issues appropriate person to supply data respondent understanding Sample representative of all businesses (not just NPIs) Found problem questions: people consistently frowned and went to records for one data avail Found that data couldn't be separated into categories we required - just wasn't in their records like that Found had to be flexible on time periods because some paid in regular pay but others salary sac only reconciled every 6 months Observational Studies Limitations Seminar Series respondents may not display any outward evidence of confusion Need skilled observer, some set questions travelling to organisations can be time consuming and expensive Pilot Testing What is pilot testing? Seminar Series A test on a small representative sample and conducted in an environment as close to the real situation as possible Most people we review conduct pilot test. One of best ways to determine if collection going to work. Pilot just take small representative sample and do survey on them - can be as little as 10 units for CATI survey. Pilot Testing Can identify whether data is what you want Seminar Series non-response adequacy of: collection method instructions processing system questionnaire mock tabulations, see if data can be input to the tables? scales: can they be created/used given info received? is there a high correlation between questions, then can remove one item non response: is it too hard, sensitive, routing, etc. full non-response, sensitivity, time given to complete, frame issues, method for cost and respondent burden. If CATI important to keep it down to 20 minutes alternative versions of form compared instructions: on the form eg routing, or to get it back to the office to interviewers, processing system - editing does it work? questionnaire: format, problems with the form, format, wording open ended questions to determine categories Pilot Testing Limitations Seminar Series not suitable for testing respondent understanding Dress Rehearsal Seminar Series What is a dress rehearsal? A formal final trial run Census Large important surveys - ASK Michael Dress Rehearsal Can identify Seminar Series preliminary estimates of population variances adequacy of processing system administration processes administration costs does it all work? instructions are they clear? can data be processed? administration costs - how much will final really cost, duration of interview how much have to pay interviewers pop variances: help to predict sampling errors eg for big stuff like census Dress Rehearsal Limitations Seminar Series not suitable for testing questionnaire can be expensive sample may overlap with the survey sample Expense not worth it for repeating surveys with very little change Very large surveys the sample may overlap. Can avoid by selecting the sample first and then testing from left over Final test What should I do as a final Seminar Series test? Get someone who has not worked on the project to read it through can find editorial mistakes eg question asking strongly opposed, somewhat opposed, somewhat favour, strongly oppose Seminar Series Analysing Results Some examples of how to analyse test results Problem high refusal rate, or high partial non-response Consider a few common problems Questionnaire design Analysing Results Possible reasons instructions unclear Seminar Series introduction unclear long questionnaire sensitive sequencing multiple options when only one appropriate To determine possible reasons contact people who have not responded and ask them why. instructions to respondent paper form they don't know what to do? intro unclear CATI - interviewer not pushing the right buttons, not doing a sufficient sell job might be some misunderstanding of questions Analysing Results Problem Seminar Series lots of out of scope units and dead businesses Reason inappropriate frame flawed process for determining the right person in the business eg you want to collect information about advertising but you are talking to the firm accountant Testing Strategy What should I consider when Seminar Series developing a testing strategy? How big is the survey? Large - over 1000 responding businesses Medium 200- 1000 Small less than 200 Testing Strategy Large scale (over 1000)? Seminar Series robust strategy allocate 5-10% of budget plan for all types of testing really risk management there are too many things which could go wrong if allocate 5-10% of budget you will have sufficient funds to undertake all necessary testing. from SCH reviews, larger 9-14% mail of budget, 3% for CATI smaller Large survey: government survey on how a new tax law will effect businesses. Need good estimates from each industry Testing Strategy Medium (200-1000) Seminar Series three to four types of testing skirmish one of focus groups, cognitive or observational studies small pilot skirmish cheap and easy look at relative merits of cog, focus and observational studies look at if suits you pilot tests so many things really good form of testing Most surveys are like this Testing Strategy Small (up to 200) Seminar Series skirmish pilot skirmish pilot of 10 respondents - who represent the population Testing Strategy What if I do not have time? Seminar Series skirmish (1-2hrs) contact a few businesses request advice from SCH consult experts in survey design warning this slide should be read with care need to plan and conduct adequate testing - you need to make time It is a real risk not testing if you are pushed for time: skirmish - short, easy - probably already done You will have contacts round, contact a few business owners and ask them to complete, do via phone, e-mail 10 businesses achievable in a day - 1 day is not too much to loose when the risk is that you could lose three months of development time Ask do they understand the concepts, does it make sense SCH can give advice on areas which may need to be tested. contact experts: there are many companies round, some may provide an advice service. Do a literature search consult others who have approached your target population or conducted research on a similar topic see SCH website for list of approved surveys Important to remember that creating a good rapport with clients through a well conducted survey is important. Esp if you are going to repeat or conduct other surveys with same group Sources ABS/SCH/NSS websites Seminar Series www.sch.abs.gov.au (see Reference Material) Moser CA and Kalton G; (1986) Survey Methods in Social Investigation; Gower; Vermont.