Guide to MN WorkForce Center System Job Seeker Satisfaction Survey

Guide to MN WorkForce Center System Job Seeker Satisfaction Survey Making Sense of the Quarterly Charts and Reports September 1, 2006 1 About the Survey Telephone interviews are conducted with:  Exiters from Workforce Investment Act (WIA) Adult, WIA and State Dislocated Worker, WIA and State Youth, and Minnesota Family Investment Program (MFIP) Exiter and in-progress Rehabilitation Services (RS) and State Services for the Blind (SSB) customers Core Services customers (“universal customer,” formerly Job Service).   Visit the link below for specific selection criteria information: www.deed.state.mn.us/customersurvey/Assets/jobseeker/doc/SurveyCriteria0207.doc September 1, 2006 2 About the Survey Survey Vendor: Strategic Research Group (SRG)  Chosen based on best response to our Request for Proposals   Based in Columbus, Ohio Has conducted WIA customer satisfaction surveys for State of Ohio  Out-of-state vendor helps limit survey bias September 1, 2006 3 About the Survey Survey questions provide a “score” as well as descriptive comments:  Minnesota Customer Satisfaction Index (MnCSI) is a numeric score based on 3 standard questions about general satisfaction (more on this in the „Questions‟ slide). Descriptive comments are from open-ended questions asking about what services/activities were of most help, and what about services could be improved.  September 1, 2006 4 About the Survey The survey presents a broad view of customer satisfaction:  Local area results are based on balanced input from local area customers. Results represent a range of customers: most satisfied, least satisfied, and everyone in-between. Results are produced quarterly from on-going survey.   September 1, 2006 5 Why is this survey important?  Job Seeker Satisfaction Survey (JSSS) is based on large, random samples; these produce a credible picture of overall satisfaction for the whole system / program / local area. JSSS focuses on the customer‟s entire services experience, particularly in relation to finding a job. JSSS complements results of Mystery Shoppers that focus on quality of WorkForce Center facilities and shortterm interaction with staff. 6   September 1, 2006 Why is this survey important? (cont.)  JSSS is on-going, with new results each quarter; this allows tracking over time of customer response to changes in service strategy. JSSS provides detailed understanding of the customer‟s experience by capturing verbatim responses to “which services were most help [in getting a job]” and “what about the services could be improved”.  September 1, 2006 7 Reports Produced     WorkForce Center System (minus MFIP) Each major program at the statewide level Each local area, all programs combined. Each local area, MFIP-only (brief report every 6 months) September 1, 2006 8 Samples  For a WorkForce Service Area (WSA), the proportion of interviews from a program roughly equals that program's portion of the total customer population. One exception: Core Services customers receive lessintense services than customers in other programs, so the Core Services population is halved.   Example: Program Adult Dislocated Worker Youth RS Core Services Population in WSA 80 600 240 900 2800 (use 1400) Sample 8 60 24 90 140 9 September 1, 2006 Samples  SSB and MFIP are not included in local area samples or results. SSB program results are based on a statewide sample (~260 interviews over 12 months). MFIP program results are based on a statewide sample (~375 interviews over 12 months). In addition, a small sample (~40-70) is drawn for each local area. 10   September 1, 2006 Questions Three questions ask about general satisfaction:    Overall satisfaction with the services? To what extent have the services met expectations? How well do the services compare with ideal?  Questions are asked on a 1-to-10 scale where “1” is least satisfied and “10” is most satisfied. Responses to the three questions are averaged and converted to 0-to-100 scale to create MnCSI (Minnesota Customer Satisfaction Index). MnCSI can be used to compare results over time. 11   September 1, 2006 Questions Two questions asked about staff:   How satisfied that staff understand your needs? How satisfied that staff responded to request for help in reasonable amount of time?  Responses are “very satisfied,” “satisfied,” “dissatisfied,” and “very dissatisfied” Two additional questions asked for RS also use “very satisfied” to “very dissatisfied”    How satisfied that were given enough information to make good choices for employment plan? How satisfied that had an active role in decisions about services? 12 September 1, 2006 Questions Two open-ended questions ask:   Which activities/services were most help to you [in getting a job]? What about the services could be improved?  Verbatim responses sorted into categories (e.g., service quality, staff, level of service)   Each category reported as a percent of total responses Customers often mention more than one topic in their response (e.g., “the computer class, learning what they are looking for in a resume, and the interview techniques . . .”) September 1, 2006 13 Three Things to Keep in Mind When Viewing Quarterly MnCSI Results  Results fluctuate from quarter to quarter; watch them over a longer period. After five or six quarters, ask:    Is there any consistent pattern to our WSA [program] score? Is the score generally rising or falling? If we changed our services or processes, did the score change?  Smaller samples give less exact results. Always check the width of the confidence intervals. It‟s best not to compare scores among WSAs. Differences may be due to variations in sample make-up or:       local economic conditions customer-to-staff ratios financial resources program strategies customer characteristics, etc. 14 September 1, 2006 Figure 1a. Overall Satisfaction with Services: Local Area, Region, State Customer Satisfaction: Lake Woebegone WSA Average MnCSI over past 5 quarters (12-month rolling) (N=Current N to date) 80.0 Lake Woebegone (N=455) Minnesota (N=4781) Greater MN (N=2834) Minnesota Customer Satisfaction Index (MnCSI) 75.0 70.0 65.0 60.0 Apr 05-Jun 05 Apr 05-Sept 05 Apr 05-Dec 05 Year-to-date by Quarter Apr 05-Mar 06 April 05-June 06    Track the changes over time. Remember, the larger the sample (the bigger the “N”) the more precise the result. Local area MnCSI scores are put in context by comparing to State and corresponding region (Metro or Greater MN) MnCSI scores. The chart uses an enlarged scale range (60.0 – 80.0) so you can better see trends in MnCSI scores. 15 September 1, 2006 Figure 1b. Overall Satisfaction with Services: Local Area Scores Customer Satisfaction: Lake Woebegone WSA Average MnCSI over past 5 quarters (12-month rolling) (Current to date N=455) 100.0 90.0 Minnesota Customer Satisfaction Index (MnCSI) Lake Woebegone 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0 Apr 05-Jun 05 Apr 05-Sept05 Apr 05-Dec05 Year-to-date by Quarter 71.6 73.5 74.3 74.4 Apr 05-Mar 06 Apr 05-June06    This chart shows average MnCSI score over time. Examine the confidence intervals, the black bars extending above and below the average scores. The bar indicates we are 95 percent confident the true score lies within the length of the bar. The narrower the bar, the more precise the MnCSI score. Think of confidence intervals as “margin of error,” much like political polls that refer to a “margin of error of plus or minus three points.” 16 September 1, 2006 Summaries of answers to open-ended questions  Tables provide a quick summary of topics mentioned in answers to:  Which activities/services were most help to you [in getting a job]?  What about the services could be improved? The local area summary is put in context by comparing to State and region (Greater MN or Metro) summaries .  September 1, 2006 17 Which activities/services were most help to you [in getting a job]?  Responses categorized into 10 major groups (Learning job search skills, Staff helpfulness, etc.) and 38 more detailed subgroups. Compare to State and Region (Metro or Greater MN) by looking across the rows.  September 1, 2006 18 What about these services that you received could be improved?  Responses categorized into 7 major groups (Provide more services, Improve quality of services, etc.) and 30 more detailed sub-groups. Compare to State and Region (Metro or Greater MN) by looking across the rows.  September 1, 2006 19 Comments on Good Services, June 2006 Results  Youth • “My counselor went way beyond just finding me a job, he hooked me up with finding scholarships, that is what I am most thankful for.” (WSA 3) “A program called Brighter Futures. All of these programs boil down to the individual that has hands on contact. The person that I had contact with was wonderful.” ( WSA 5) “My job coach. They helped me adjust problems to my employer properly, and also helped me find affordable transportation.” (WSA 17) “The weekly group session, job club, we got a lot of different ideas on how to react in interviews.” (WSA 8)  Rehab Services • •  Adult / Dislocated Worker • September 1, 2006 20 Feedback on Good Services (cont.) • • “Resume writing, interviewing skills, creative job searching. They seemed to be very current in terms of what employers wanted.” (WSA 12) “I went to a three day boot camp; it was beneficial to see how things changed over the years. I also received two computer classes and I really enjoyed those.” (WSA 16)  Core Services • • • “I had a one on one meeting with a job placement worker and he spent a lot of time going over possible classes, training and useful internet job searching tools.” (WSA 4) “They are actually helping me look for jobs by providing referrals via email and phone calls.” (WSA 12) “The combination of everything is important, talking one-on-one with a counselor is important so that they can get a feel for who you are and your needs. The classes and knowing the website is important. There is a wide range of needs so one-on-one talking is important.” (WSA 17) 21 September 1, 2006 Flexibility of the Survey  Local areas that would like more detailed results may purchase a larger sample so their results can be broken-out by program. (One local area has enlarged its sample to break-out results in 3 groups: Rehab Services, Core Services, and Adults + Dislocated Workers + Youth.)  Local areas or programs can explore additional topics by purchasing additional questions asked just of their customers. (State Services for the Blind and LMI Research have purchased additional questions.) September 1, 2006 22 Sample Web Page Look for the link to your local area or program September 1, 2006 23 Need Further Information?  Visit www.deed.state.mn.us/customersurvey/index.htm to view the entire Customer Satisfaction Survey results. Contact Devin Colvin of the Policy, Planning and Measures Unit via email, devin.colvin@state.mn.us or by phone, 651-284-3516, with questions, comments or requests about the presentation or results.  September 1, 2006 24