Shared by: tek31120
FOOD SECURITY ASSESSMENT ACCURACY CHECKLIST Definition of FSA accuracy: Likelihood that an assessment reflects the reality of conditions on the ground, especially in regards to the scale, severity of food insecurity and type of humanitarian need. Structure of checklist: The checklist covers 21 issues that are expected to influence EFSA accuracy. They are divided into four parts: 1. Process Management, Capacities & Partnerships, 2. Assessment Preparation, 3. Field work, 4. Data entry and analysis. Purpose of checklist: 1. Raise awareness on factors potentially limiting accuracy in preparation of and throughout the assessment process; 2. Pointing out limitations to accuracy incurred by a specific assessment. Who should fill out the checklist: Staff involved in the specific assessment in close consultation with mission members to reach agreement on answers. The checklist is only applicable to assessments that include primary data collection! Instructions: Please click appropriate answers. Scores will be calculated automatically. *: answer the questions only if applicable ► FSA title ►Team leader ► Country --- Select Country --- ► Reviewer ► Crisis type --- Select Crisis Type --- QUESTION TO ASK ANSWERS AND CORRESPONDING ACCURACY SCORE RATIONALE [Part I] Process Management, Capacities and Partnerships I.1 What was the time available for the Initial and Rapid EFSAs need to compromise between assessment (incl. field work and completed within less than 7 days completed within 1-3 weeks completed within 4-8 weeks completed in more than 8 weeks accuracy and timeliness. report writing)? I.2 What was the level of partner Many partners involved during Partnerships in assessments help to validate the approach Few partners involved during Many partners involved during all involvement during the assessment Not involved certain stages or few partners and the analysis and increase the amount of information certain stages stages process (preparation to report during all stages and expertise available to the assessment team. writing)? I.3 What was the level of experience of Vast assessment and leadership It is likely that the accuracy of findings increases with the involved involved the mission leader ? 1st assessment experience experience (e.g. advanced assessment experience of the person who has the in 1-3 assessments before in 4-6 assessments before assessor) ultimate responsibility for data analysis and interpretation. I.4 Did the mission leader oversee the Oversaw most parts of the Being involved in all steps of the assessment process No, provided inputs only selectively Oversaw few parts of the process, Oversaw the entire process, incl. whole assessment process, from process, including in the country including field work facilitates interpretation of results by and mainly remotely including in the country itself all in-country assessment stages mission planning to report writing? itself mission leader when writing the report. Continuous supervision also helps to keep data collection and analysis on track. I.5 Were interviewers knowledgeable If interviewers are insufficiently knowledgeable about the about the area studied? Hardly/very few Some Most All context in which they are working they might miss essential information or interpret responses in an inappropriate way. I.5 Were interviewers knowledgeable If interviewers are insufficiently knowledgeable about the about the area studied? context in which they are working they might miss essential information or interpret responses in an inappropriate way. Average Score Part I #DIV/0! [Part II] ASSESSMENT PREPARATION II.1 Was an analysis plan outlining A data analysis plan - if used properly - helps to guarantee information requirements prepared Yes, used during many stages by Yes, used throughout the entire that all necessary data are collected and analysed to No Yes, but only selectively used prior to the data collection and was it some team members process by the team respond to the salient assessment questions used throughout the assessment? II.2 Were reliable population baseline As primary data are collected from a sample that Used, but reliability low (e.g. old; Reliable data used (e.g. recent, data of the study zone available and No, either not available or not used Used, but reliability medium represents the larger population in the affected area, it is politically influenced) verified) used by the assessment? key to have accurate and up-to-date population data to draw a proper sample and to get accurate estimates of people in need. II.3 Were secondary data (including Modestly used, but reliability of The more data from different sources are used and cross- baseline data) used throughout the Used, with fair reliability of most Reliable secondary data were used checked the higher the likelihood that conclusions are No, either not available or not used most data low (e.g. old; politically assessment process? data extensively accurate. influenced) II.4 Which sampling method was used? Because probability sampling employs random selection Mainly random with 1st stage techniques it is more objective than non-probability Mixed random-purposive purposive sampling (e.g. purposive Probability Simple or Stratified Non- probability sampling sampling. Probability sampling also allows for the degree sampling selection of study area within crisis- Random Samples of error around food security estimates to be quantified. affected zone) Results from non-probablity sampling can only be generalized with caution. II.5 Rule of thumb suggests a sample size of 150 to 250 * In case of household survey, how <300 households 300 to 500 households 501 to 800 households >800 households households for each strata (population group or many households were interviewed? geographic area). Towards the upper end of this scale reliability of results and validity of extrapolation increases. II.6 Were interviewers sufficiently Surveyors have to be trained on the objectives of the Less than 1 day of training and no 1-2 days of training and no field More than 2 days of training and trained on the assessment tools used 1-2 days of training and field test study and on the specific survey instruments. They have to field test test field test (i.e. participatory appraisal methods, understand all terms used in the data collection survey questionnaire etc.) and the instruments specifics of the context? Average Score Part II 0.00 [Part III] FIELD WORK III.1 Have the selected communities been Informants frequently see the assessment as an repeatedly studied, and/or have they repeatedly studied and long term repeatedly studied or long term studied a couple of times and/or opportunity for personal (or community) gain and give the been recipients of (food) aid never studied or assisted impression that the situation is worse than it actually is in recipients recipients irregular recipients order to receive assistance. They may also give misleading information about the impact of the crisis on different groups. III.2 Have data collection tools been The questionnaire should be translated into the language translated (if required), field tested translated (if required), field- thoroughly translated (if required), in which it will be administered. Once translated, it should not translated, quickly field-tested and subsequently revised? No tested and revised (within 2-3 extensively field-tested and be “back-translated” into the original language to check and revised (within one day) days) revised (more than 3 days) on the accuracy of the translation. Field testing of tools helps to reduce further ambiguousness and increases specificity of questions. III.3 Were language/ translation problems Misunderstandings and misinterpretation during encountered during data collection? Frequently, by most interviewers in Frequently, by many interviewers Occasionally, by some interviewers No interviews lowers the accuracy of primary data. most locations in many locations in some locations III.4 Were interviewers provided During a long days of interviews, interviewers often get Adequately (field work regularly sufficient rest during field work? tired and feel as if they are hearing the same responses Hardly, assessment schedule was To some extent, field work ended stopped (late) afternoon, for a Reasonably, field work ended repeatedly. They are less inclined to query the information very tight, teams worked until late irregularly without sufficient max. of 5 days worked, one day regularly late afternoon and are less alert to inconsistencies and new information. evening breaks during the day rest was provided, lunch breaks were scheduled) III.5 Informants may be happy to discuss their lives but, as the * How much time did a typical More than one hour 46-60 minutes 30-45 minutes Less than 30 minutes interview progresses, they may lose interest and the structured household interview quality of their response declines. take? III.6 Participatory techniques and Proportional Piling: There is * If proportional piling, focus group Rather low Medium, discussion was dominated by a few Medium, most groups participated (e.g. women) High, all groups actively participated usually a lot of debate about the relative size of the piles; discussions or other participatory this encourages participation and enhances accuracy. methods were used, was both debate and participation high? Average Score Part III 0.00 [Part IV] DATA ENTRY AND ANALYSIS IV.1 Was data entry and cleaning Improper data entry and cleaning can be major causes of Yes, thoroughly supervised and conducted by trained and No Partially Mostly assessment inaccuracy. spot checked experienced staff; was it rigorously supervised and spot checked? IV.2 Were primary data collected from Different data sources allow for triangulation and More than 2 information sources different sources and triangulated? 2 information sources used and More than 2 information sources crosschecking of findings. Rule of thumb: If at least three Only one source used used and triangulated using trianguled used and trianguled diverse and reliable sources provide consistent statistical methods information, and an insignificant number of sources provide inconsistent information, the information can be used with confidence. IV.3 extensive statistical analysis, Statistical analysis allows to correlate data and to identify * Were statistical methods No Frequencies and means generated Frequencies, means and cross- including PCA and statistical tests main determinants of food insecurity in a given situation. employed for correlation and causal tabulations generated (CI) conducted analysis? IV.4 Were results from data analysis Consultatively interpreted on few Consultatively interpreted on Consultatively interpreted Discussing rassessment findings with different key Interpreted by only 1-2 people interpreted in a consultative way? occasions many occasions throughout analysis process informants increases the likelihood of an accurate interpretation/understanding of the situation Average Score Part IV 0.00 PROCESS MANAGEMENT, CAPACITIES AND ASSESSMENT PREPARATION FIELD WORK DATA ENTRY AND ANALYSIS PARTNERSHIPS #DIV/0! 0.00 0.00 0.00 TOTAL Average Score #DIV/0! You have 21 mandatory 1.1 2.1 4.1 3.1 question(s) not ansewered. 1.2 2.2 3.2 4.2 Please note that the highlighted cells in red correspond to mandatory question(s) that 1.3 2.3 3.3 4.3 have not been answered. In order to have a 1.4 2.4 3.4 4.4 proper calculation, please make sure that you answer (click) all mandatory questions. 1.5 2.5 3.5 2.6 3.6 Average Scores and Levels of Accuracy Score Range Accuracy Classification >5 High #DIV/0! #DIV/0! 3.1 - 5 Moderate to High #DIV/0! #DIV/0! 1 to 3 Moderate #DIV/0! #DIV/0! <1 Low #DIV/0! #DIV/0!