Online Surveys Guidelines and Summary of Research

ARTHRITIS SOURCE Online Surveys: Guidelines and Summary of Research Julianne Fondiller March 16, 2000 Purpose and Goals of Online Survey The purpose of the survey is to collect data on and from actual users of the Arthritis Source. This data will be used to accomplish the following goals:    Inform the re-design Help to understand and prioritize actual users’ issues with the Source and, thus, direct the focus of the field and lab studies Gather potential subjects/participants for the upcoming field and lab studies Goal 1: Inform the re-design. Collecting this data will allow us to know our users and, therefore, make educated decisions about the most appropriate content, language, style, tone, structure, and use of stories for the Source. Goal 2: Prioritize and direct the focus of the field and lab studies. Understanding our users will allow us to make informed decisions about what to focus further research on. It will help us understand what issues the users are currently facing, their assumptions, problems, mistakes, etc. The survey information will help to guide the choice of tasks we ask users about or ask them to perform because it will ensure that they are realistic tasks. For instance, if we find out in the survey that almost no primary care physicians use the Source, then we wouldn’t want to study a sample of them using the Source in the field, since they wouldn’t be representative of the actual users. Perhaps we’d want to study why they weren’t using the Source. In another example, we may determine from the survey that arthritis patients are the biggest user group; their most common goal is to see how to perform exercises, but most have trouble accomplishing this task. So, we may want to focus the lab studies on why they’re having trouble. Goal 3: Gather potential subjects/participants for the upcoming field and lab studies In the survey, we will ask the respondents if they’d be willing to be contacted for more questions, to solicit their feedback, and to possibly participate in another study. Finding actual users is important for studies, so this will be an invaluable way to make connections with them. Online Survey Design After reviewing the literature on survey methodology and online surveys, we determined that there were a number of critical areas in online survey design. For our purpose, these core areas of concern included:  Survey validity- compensating for inherent sample bias from web audience and trying to ensure valid responses;  Convincing people to take the survey - attention spans are so short online that catching people’s attention and convincing them to take time for the survey would be difficult.  Survey screen design and survey placement- where could we place the survey on our website so that people would most likely respond and not immediately close it?  Survey length- how long would people be willing to spend answering a survey online  Asking questions;  Preserving anonymity GUIDELINES Below we’ve assembled a list of guidelines for conducting an online survey. It’s not comprehensive, but it will get you started. Also, we’ve included examples or “applications” of how we applied many of these principles in our online survey for the Arthritis Source. Striving for Validity  There is an inherent sample bias with online surveys that you’ll need to reflect in your interpretations and conclusions (1, 2, 7). Realize that you are only getting a limited viewpoint- you are only getting responses from people already visiting your site. For instance, you won’t know why others are not visiting your site. Application: Since we know that we are only sampling people already using the Arthritis Source, we are also planning on surveying arthritis patients to be distributed through the mail and in waiting rooms of doctors’ offices.  Try to oversample to compensate for non-random sample and self-selection (1, 7). In other words, try to get as many responses as possible to help off-set the fact that people are choosing to take your survey, thus biasing your survey. Application: Since we don’t know what our response rate will be yet, we are tentatively planning on leaving our survey up for two weeks.  Do internal consistency checks in order to keep track of the quality of responses. If you have the length and time, you may want to check consistency of responses, which can be done by using different question formats referring to the same item (10).  Consider asking people to provide e-mail addresses (will have to do this if providing a prize) because this may help ensure more thought out responses. Knowing that there’s identification with their responses, people will usually spend a little more time and thought on their responses; however, you will have to weigh this with the value preserving in anonymity. Application: Since our survey was a part of a research project, we had to be very careful about collecting identifying data. We only asked for respondents for an e-mail address if they wanted to be in the drawing for the reward, and we ensured that responses and personal identifiers were kept separate. Marketing the Survey  Use an eye-catching subject heading and description of the research (1). Application: We used the heading “Help Us Improve the Arthritis Source” as the hotlink to the survey.  Give potential respondents a time estimate of how long it will take to answer the survey; it is difficult to gauge this online (2). You may also want to provide page numbers as another frame of reference. Application: We told users that the survey would take less than 10 minutes. We also provided the page number out of total pages on each page of the survey (for example, Page 2 of 5). Page 2 of 17  Publish a disclaimer before and after the survey to tell respondents that information they enter may be used or published (5). Application: At the UW, research like our survey requires approval by the Human Subjects Division. They required a consent form for the respondents, which embodied a disclaimer.  Reward users for their input. It increases response rates and/or rates of completion (3, 5). Rewards could be monetary or even real-time results of the survey. However, you will need to decide the value of the reward. Offering too little may not be incentive for people to take or complete the survey, but if you offer too much, people may come to your site only because of the reward, and you will get respondents who are unrepresentative of your users. Application: We told users they would be entered into a drawing to win a $50 gift certificate to Amazon.com Survey layout and design    Avoid drop-down boxes and scroll bars; it can cause respondents to accept unintentional defaults or miss questions (2). Offer a text box that allows the respondent to fill in whatever they want at the end of the survey (8, 9). If your survey is very long, present questions in survey categories; one long survey may discourage potential respondents, and increases survey complexity (9). Application: We used survey categories as headings, including: Visiting the Arthritis Source, Who are you?, How do you learn about arthritis?, Some general information about you, What do you think of the site?, and more.  Utilize adaptive questioning to reduce number and complexity of the questions, also sets question context (5, 7, 9). Adaptive questioning provides a more natural conversation-style progression of questions. Application: After we asked respondents what category best described them, we adapted the next questions to fit their answer. For example, if a person answered that they were a “person with arthritis,” next, they would be asked “Have you been diagnosed with arthritis?”   Include adequate white space (5); follow standard UI and screen layout guidelines. Use standard web terms and buttons such as reset/clear, submit, save (5). Survey Placement  Consider, if possible, arranging the site so the respondent fills out the survey immediately after using the site (8). They will be better to accurately recall their experience. However, you also want to ensure that as many people as possible see the survey. So you will have to decide where to place it so that people have a recent experience to respond about while also allowing for many people to see the survey. Application: We chose to put the link to the survey in the top right corner of every page on the Arthritis Source. We wanted to allow for people coming into the site on any page, while also allowing for people to get their tasks/browsing done (why they came to the site in the first place!) and then be able to take the survey at their convenience. Page 3 of 17  Consider providing a way for each respondent to rank each section of a site while in that section; that way the information is fresh in their mind (8). Application: We chose not to do this since our site was too big for this to be feasible. Asking Questions For examples of how we applied the “asking questions” principles, please see the Appendix for our Online Survey.  Think through every question you want to ask. Make sure you know the purpose of each question and answer (11). You only have so much time and space to ask questions, so you’ll want to carefully think about every question. Ask yourself what answers are nice to know and what are need to know?  Don’t ask users about the value of some proposed change; it’s an unreliable judge of their opinions. Instead ask about specific existing features, or give them an example of the proposed change (such as a beta site) (10, 8). People have a hard time predicting what they will do. They will certainly tell you what they think they will do, but they are not very accurate. Application: We focused our questions on what users did when they came to the Arthritis Source. We decided against asking them if they would print out a .pdf file on arthritis conditions.    Use an even number of choices in rating scales, if you want to force respondents away from a middle choice (5). Don’t ask people to rank more than 5-7 items (11, 5). Use open-ended questions as little as possible (11, 5) because they are difficult to analyze. However, they may be necessary if your list of options are too long or you don’t want to make assumptions about the users’ responses. Application: We asked only 3 open-ended questions, including “What have you visited the site today?” because we didn’t want to try to assume their answer or bias their answer by providing why we thought they were visiting. We also used an open-ended question to get users feedback on the site; we wanted to provide an opportunity for them to say anything.    Utilize other survey questions (11). Look for other surveys and use the relevant questions; it’ll save you time and work. Be careful with asking low-frequency, low-involvement events- may get biased responses (11). Low involvement events have less processing so may create biases in their responses. Give a time period if you want to know about usual behavior. It is more reliable to be specific and give a time period (11). For example: DON’T ask: How often do you access the internet? DO ask: How many times did you access the internet last week? Add memory cues to a question, even if it means increasing the question length. Adding memory cues may increase the quality of reporting (11). Memory cues are intended to help the respondent give a more accurate answer. They may give the respondent a context, a time period, etc. for the information you want.  Page 4 of 17  Knowledge questions should be asked before attitude questions. This order is essential if knowledge level is used to screen out respondents who don’t have sufficient knowledge. Further, respondents will be less likely to overclaim knowledge if knowledge questions come before attitude. (11). Use multiple questions for determining knowledge; more reliable measures of an individual’s knowledge are obtained with multiple questions. (11). If general and specific attitude questions are related, ask the general questions first (11). Rating scales: start with the end of the scale that is least socially desirable. Otherwise, respondents may choose a socially desirable answer without hearing the rest of the choices (11). Start with easy, salient, non-threatening, but necessary questions first. Put difficult potentially threatening questions near end (11). Application: Our first questions asked, “How many times have you visited the site?” and “How did you find us?” since these set the context, were important for us to know, non-threatening, and (hopefully) easy to answer. Our last questions were demographics, such as “Do you speak English fluently?” “What is your highest level of education?” and “What is your age?” These questions could be seen as threatening if people don’t like revealing personal information.       Avoid asking demographic questions first since some are seen as threatening, unless you need them to screen people out (11). If dealing with more than one topic, ask all questions on one topic before moving on to the next (11). Survey Length  Keep the survey as short as possible- no longer than 15 minutes (2). For highly-salient topics, 12-16 screens of non-scrollable text is reasonable; however, for low-saliency topics aimed at a general population, 2-4 screens of non-scrollable text is good (5, 11). Highly-salient topics are ones that respondents are personally interested in and may have a personal stake in the outcome. Application: We kept the survey length to 10 minutes, and we told respondents it would only take 10 minutes at the beginning of the survey. Submitting Questionnaires  Don’t accept questionnaires until they’re completed; thus, users won’t accidently skip questions (7). However, provide a non-response option such as “Not applicable” or “Rather not say” (3). Respondents may not have the knowledge to answer every question and may be offended with being forced to (2). Make sure people can review and change their answers easily. Provide either a Back button or a Review step (2). Highlight the unanswered questions (7). The Review step could be a summary screen that displays a summary of the responses and asks the respondents if they are correct. Application: We provided both a Back button and a Review screen, which summarized the respondents’ answers.  Thank the user for input (5).  Page 5 of 17 Testing the Survey  Usability test the questions to make sure people understand and respond to the questions in the way you thought they will (2). Usability test with people who are in or representative of your target audience. Test the technical functionality (2). You want to be sure that the respondents won’t get a technical error- that’s a sure way to lose respondents! Test how long your survey will take. Be sure and test with people who are representative of your target audience (and who haven’t worked on the survey design because they’ll know it too well!).   Protecting the Respondents’ Identities If you need to collect identifying information from respondents, you will need to preserve confidentiality. If you don’t need any identifying information, then you should ideally preserve anonymity. For more information on confidentiality and anonymity with UW studies, please refer to the Human Subjects Division website at http://depts.washington.edu/hsd. To preserve anonymity, consider:     Use a user-selected ID and password for formal surveys (5). Do not ask questions that enable respondents to be identified (5). Explain to respondents that there is no way to trace them (5). Give respondents the option to print and mail in the survey if they so choose (5). Application: In our case, we wanted to capture respondents’ identities, so we could potentially contact them for follow-up interviews. To address possible confidentiality issues, we chose to shore the names/contact information separate from the subjects’ responses. Page 6 of 17 RESOURCES [1] Coomber, R. (1997) “Using the Internet for Survey Research.” Sociological Research Online, vol.2, no.2. http://www.socresonline.org.uk/socresonline/2/2/2.html, accessed January 26, 2000. [2] Gould, E., Gurevich, M., & Pagerey, P. (1998). “Creating Surveys for the World Wide Web.” Intercom, September/October 1998. [3] GVU’s 10th WWW User Survey. http://www.cc.gatech.edu/gvu/user_surveys/survey-1998-10/, accessed January 26, 2000. [4] Health on the Net Foundation. Surveys: Evolution of Internet use for health purposes, http://www.hon.ch, accessed January 2000. [5] Huson, L. (1999) Are Guidelines for Online Survey Methodology Empirically Supported. Unpublished paper, University of Washington. [6] Huson, L. (1999) Cultural Influences on Online Surveys: A Signal for Change in Survey Methodology. Unpublished paper, University of Washington. [7] Kehoe, C.M & Pitkow, J.E. (1998) “Surveying the Territory: GVU’s Five User Surveys.” The World Wide Web Journal, Vol. 1, no. 3. http://www.cc.gatech.edu/gvu/user_surveys/papers/w3j.html, accessed January 26, 2000. [8] Nielsen, J. (1999). “Collecting Feedback from Users of an Archive.” www.useit.com, Alertbox. http://www.useit.com/alertbox/990110.html, accessed January 12, 2000. [9] Pitkow, James E. & Recker, Margaret M. (1995) “Using the Web as a Survey Tool: Results from the Second WWW User Survey.” Journal of Computer Networks and ISDN systems, Vol. 27, no. 6. http://www.cc.gatech.edu/gvu/user_surveys/papers/survey_2_paper.html, accessed January 26, 2000. [10] Root, R. & Draper, S. (1983) “Questionnaires as a Software Evaluation Tool.” Human Factors in Computing Systems, CHI ’83 Conference Proceedings, December. [11] Sudman, S., Bradburn, N.M., & Schwarts, N. (1982). Asking Questions. San Francisco: Jossey Bass Publishers. Page 7 of 17 Appendix: Arthritis Source Online Survey Help us Improve our Website! Get a chance to win a $50 gift certificate. We are a team of researchers at the University of Washington in Seattle, Washington. We are working to improve the Arthritis Source website for you, but we need your input. Please take 10 minutes to answer some important questions. We greatly appreciate it! Visiting the Arthritis Source 1.     How many times have you visited the site? This is my first time 2-5 times 6-10 times More than 10 times How did you find us?  By a search engine called _________________  Through a website called _________________  By a bookmark/favorite on my computer  I was referred to the site by _________________ (e.g. a doctor, a friend, my sister, etc.)  Other _________________ Why have you visited the site today? 2. 3. 4. If you had to evaluate your success at The Arthritis Source today, which of these categories fits best? You have been:  Very successful  Somewhat successful  Fairly successful  unsuccessful  Don’t know yet Please check all (if any) of the following activities that you have completed at The Arthritis Source:  Looked at information about medications  Looked at sensitive information that may be embarrassing to ask about, such as "sex and arthritis"  Watched a video on how to perform an exercise  Searched The Arthritis Source to follow up on information from a doctor's visit  Searched The Arthritis Source for a second opinion about a diagnosis  Referred other patients to the site 5. Who are you? 6. Which category best describes you?  Person with arthritis  Relation of person with arthritis (e.g., parent of, child of, friend of, caregiver of, etc.)  Medical professional (e.g., doctor, nurse, caregiver, etc.)  Medical student  Student (K-12, college, etc.)  Researcher (e.g., librarian, paralegal, etc.)  Other Page 8 of 17 Patient Branch More about you 1. Have you been diagnosed with arthritis?  Yes  No If yes, when were you diagnosed? (month/year) ______________ 2. What type of arthritis do you have?  _________________ This button comes set as the default  Don’t know  Rather not say Page 9 of 17 family/caregiver branch More about you 1. What is your relation to the person with arthritis? You are his/her _________________ (e.g., parent, child, friend, caregiver) Has your relation been diagnosed with arthritis?  Yes  No If yes, when were you diagnosed? (month/year) ______________ 3. What type of arthritis do they have?  _________________ This button comes set as the default  Don’t know  Rather not say 2. Page 10 of 17 Medical professional branch More about you 1. 2. What type of medical professional are you? ____________________________ What percent of your work involves arthritis issues?  0-5%  6-25%  26-50%  51-75%  76-100% Are you using The Arthritis Source on behalf of a specific patient?  Yes  No How will you use what you find on The Arthritis Source? 3. 3. Page 11 of 17 Medical student branch More about you 1. 2. What type of medical student are you? How would you describe your studies? _________________ What percent of your work involves arthritis issues?  0-5%  6-25%  26-50%  51-75%  76-100% . Are you using The Arthritis Source on behalf of a specific patient?  Yes  No 3. 4. How will you use what you find on The Arthritis Source? Page 12 of 17 Student Branch More about you 1. 2. What type of student are you? _________________ (e.g., 6th grade, freshmen in college) How will you use what you find on The Arthritis Source? Page 13 of 17 Researcher Branch More about you 1 How will you use what you find on The Arthritis Source? Page 14 of 17 Resume the rest of the survey How do you learn about arthritis? 1. Where do you get information on arthritis? (Check all that apply)  Websites  Medical books and journals for medical professionals  Medical books for the general public  Brochures (like those found in doctors’ offices)  Internet groups (news groups, chat rooms, bulletin boards)  Newspapers and magazines  Other Which of these things have you done on a computer? (Check all that apply)  Watched a video on the web  Listened to audio on the web  Received email  Sent email  Participated in an online chat or discussion (not including email)  "Bookmarked" a webpage or saved it as a "Favorite"  Customized a webpage for yourself (e.g., My Yahoo, CNN Custom News)  Created a webpage  Changed your "cookie" preferences (continued) How confident are you that you could accomplish the things that you didn’t check, if you needed to?  Very confident  Fairly confident  Somewhat confident  Not confident 3. Do you feel that you can visit your general practitioner, if you need to?  Yes  No  Don't know Do you feel that you can visit an arthritis specialist, if you need to?  Yes  No  Don't know 2. 4. Some general information about you 1. Where are you located?  Africa  Antarctica  Asia  Oceania (Australia, New Zealand, etc.)  Europe  USA  Canada  Mexico  Central America  South America  Middle East Page 15 of 17  West Indies 2. Do you speak English Fluently  Yes  No Describe the area where you are located:  urban  suburban  rural  other _________________  don’t know What is your highest level of education?  Grammar school  High school or equivalent  Vocational/Technical school (2 year)  Some college  College graduate (4 year)  Masters degree (M.A. or M.S.)  Doctoral degree (Ph.D.)  Other _________________ What is your age?  Under 20  21-30  31-40  41-50  51-60  71-80  81 and over  Rather not say What is your sex?  Female  Male  Rather not say 3. 4. 5. 6. What do you think of the site? 1. Would you recommend the site to someone else looking for information on arthritis?  Yes  No  Don’t know Please give us any comments you have about the site! All suggestions are welcome and we want to know what you think! 2. Page 16 of 17 Can we contact you? 1. If you would like to be entered into the drawing for a $50 dollar gift certificate, please enter your email and/or phone number: Email _________________ Phone Number (with area code) _________________ Would you be willing to participate in a follow up interview?  Yes  No If yes, please fill out one or more of the following boxes: Email _________________ Phone Number (with area code) _________________ Address _________________ Thank you very much for taking the time to complete our survey! 2. Julianne Fondiller Page 17 of 17 06/08/09