Get documents about "FSA_AccuracyChecklist
Document Sample
FOOD SECURITY ASSESSMENT ACCURACY CHECKLIST
Definition of FSA accuracy: Likelihood that an assessment reflects the reality of conditions on the ground, especially in regards to the scale, severity of food insecurity and type of humanitarian need.
Structure of checklist: The checklist covers 21 issues that are expected to influence EFSA accuracy. They are divided into four parts: 1. Process Management, Capacities & Partnerships, 2. Assessment Preparation, 3. Field work, 4. Data entry and
analysis.
Purpose of checklist: 1. Raise awareness on factors potentially limiting accuracy in preparation of and throughout the assessment process; 2. Pointing out limitations to accuracy incurred by a specific assessment.
Who should fill out the checklist: Staff involved in the specific assessment in close consultation with mission members to reach agreement on answers.
The checklist is only applicable to assessments that include primary data collection!
Instructions: Please click appropriate answers. Scores will be calculated automatically. *: answer the questions only if applicable
► FSA title ►Team leader
► Country --- Select Country --- ► Reviewer
► Crisis type --- Select Crisis Type ---
QUESTION TO ASK ANSWERS AND CORRESPONDING ACCURACY SCORE RATIONALE
[Part I] Process Management, Capacities and Partnerships
I.1 What was the time available for the Initial and Rapid EFSAs need to compromise between
assessment (incl. field work and completed within less than 7 days completed within 1-3 weeks completed within 4-8 weeks completed in more than 8 weeks accuracy and timeliness.
report writing)?
I.2 What was the level of partner Many partners involved during Partnerships in assessments help to validate the approach
Few partners involved during Many partners involved during all
involvement during the assessment Not involved certain stages or few partners and the analysis and increase the amount of information
certain stages stages
process (preparation to report during all stages and expertise available to the assessment team.
writing)?
I.3 What was the level of experience of Vast assessment and leadership It is likely that the accuracy of findings increases with the
involved involved
the mission leader ? 1st assessment experience experience (e.g. advanced assessment experience of the person who has the
in 1-3 assessments before in 4-6 assessments before
assessor) ultimate responsibility for data analysis and
interpretation.
I.4 Did the mission leader oversee the Oversaw most parts of the Being involved in all steps of the assessment process
No, provided inputs only selectively Oversaw few parts of the process, Oversaw the entire process, incl.
whole assessment process, from process, including in the country including field work facilitates interpretation of results by
and mainly remotely including in the country itself all in-country assessment stages
mission planning to report writing? itself mission leader when writing the report. Continuous
supervision also helps to keep data collection and analysis
on track.
I.5 Were interviewers knowledgeable If interviewers are insufficiently knowledgeable about the
about the area studied? Hardly/very few Some Most All context in which they are working they might miss
essential information or interpret responses in an
inappropriate way.
I.5 Were interviewers knowledgeable If interviewers are insufficiently knowledgeable about the
about the area studied? context in which they are working they might miss
essential information or interpret responses in an
inappropriate way.
Average Score Part I #DIV/0!
[Part II] ASSESSMENT PREPARATION
II.1 Was an analysis plan outlining A data analysis plan - if used properly - helps to guarantee
information requirements prepared Yes, used during many stages by Yes, used throughout the entire that all necessary data are collected and analysed to
No Yes, but only selectively used
prior to the data collection and was it some team members process by the team respond to the salient assessment questions
used throughout the assessment?
II.2 Were reliable population baseline As primary data are collected from a sample that
Used, but reliability low (e.g. old; Reliable data used (e.g. recent,
data of the study zone available and No, either not available or not used Used, but reliability medium represents the larger population in the affected area, it is
politically influenced) verified)
used by the assessment? key to have accurate and up-to-date population data to
draw a proper sample and to get accurate estimates of
people in need.
II.3 Were secondary data (including Modestly used, but reliability of The more data from different sources are used and cross-
baseline data) used throughout the Used, with fair reliability of most Reliable secondary data were used checked the higher the likelihood that conclusions are
No, either not available or not used most data low (e.g. old; politically
assessment process? data extensively accurate.
influenced)
II.4 Which sampling method was used? Because probability sampling employs random selection
Mainly random with 1st stage
techniques it is more objective than non-probability
Mixed random-purposive purposive sampling (e.g. purposive Probability Simple or Stratified
Non- probability sampling sampling. Probability sampling also allows for the degree
sampling selection of study area within crisis- Random Samples
of error around food security estimates to be quantified.
affected zone)
Results from non-probablity sampling can only be
generalized with caution.
II.5 Rule of thumb suggests a sample size of 150 to 250
* In case of household survey, how <300 households 300 to 500 households 501 to 800 households >800 households
households for each strata (population group or
many households were interviewed? geographic area). Towards the upper end of this scale
reliability of results and validity of extrapolation increases.
II.6 Were interviewers sufficiently Surveyors have to be trained on the objectives of the
Less than 1 day of training and no 1-2 days of training and no field More than 2 days of training and
trained on the assessment tools used 1-2 days of training and field test study and on the specific survey instruments. They have to
field test test field test
(i.e. participatory appraisal methods, understand all terms used in the data collection
survey questionnaire etc.) and the instruments
specifics of the context?
Average Score Part II 0.00
[Part III] FIELD WORK
III.1 Have the selected communities been Informants frequently see the assessment as an
repeatedly studied, and/or have they repeatedly studied and long term repeatedly studied or long term studied a couple of times and/or opportunity for personal (or community) gain and give the
been recipients of (food) aid never studied or assisted impression that the situation is worse than it actually is in
recipients recipients irregular recipients
order to receive assistance. They may also give misleading
information about the impact of the crisis on different
groups.
III.2 Have data collection tools been The questionnaire should be translated into the language
translated (if required), field tested translated (if required), field- thoroughly translated (if required), in which it will be administered. Once translated, it should
not translated, quickly field-tested
and subsequently revised? No tested and revised (within 2-3 extensively field-tested and be “back-translated” into the original language to check
and revised (within one day)
days) revised (more than 3 days) on the accuracy of the translation. Field testing of tools
helps to reduce further ambiguousness and increases
specificity of questions.
III.3 Were language/ translation problems Misunderstandings and misinterpretation during
encountered during data collection? Frequently, by most interviewers in Frequently, by many interviewers Occasionally, by some interviewers No interviews lowers the accuracy of primary data.
most locations in many locations in some locations
III.4 Were interviewers provided During a long days of interviews, interviewers often get
Adequately (field work regularly
sufficient rest during field work? tired and feel as if they are hearing the same responses
Hardly, assessment schedule was To some extent, field work ended stopped (late) afternoon, for a
Reasonably, field work ended repeatedly. They are less inclined to query the information
very tight, teams worked until late irregularly without sufficient max. of 5 days worked, one day
regularly late afternoon and are less alert to inconsistencies and new information.
evening breaks during the day rest was provided, lunch breaks
were scheduled)
III.5 Informants may be happy to discuss their lives but, as the
* How much time did a typical More than one hour 46-60 minutes 30-45 minutes Less than 30 minutes
interview progresses, they may lose interest and the
structured household interview quality of their response declines.
take?
III.6 Participatory techniques and Proportional Piling: There is
* If proportional piling, focus group Rather low
Medium, discussion was
dominated by a few
Medium, most groups participated
(e.g. women)
High, all groups actively
participated usually a lot of debate about the relative size of the piles;
discussions or other participatory this encourages participation and enhances accuracy.
methods were used, was both
debate and participation high?
Average Score Part III 0.00
[Part IV] DATA ENTRY AND ANALYSIS
IV.1 Was data entry and cleaning Improper data entry and cleaning can be major causes of
Yes, thoroughly supervised and
conducted by trained and No Partially Mostly assessment inaccuracy.
spot checked
experienced staff; was it rigorously
supervised and spot checked?
IV.2 Were primary data collected from Different data sources allow for triangulation and
More than 2 information sources
different sources and triangulated? 2 information sources used and More than 2 information sources crosschecking of findings. Rule of thumb: If at least three
Only one source used used and triangulated using
trianguled used and trianguled diverse and reliable sources provide consistent
statistical methods
information, and an insignificant number of sources
provide inconsistent information, the information can be
used with confidence.
IV.3 extensive statistical analysis, Statistical analysis allows to correlate data and to identify
* Were statistical methods No Frequencies and means generated
Frequencies, means and cross-
including PCA and statistical tests main determinants of food insecurity in a given situation.
employed for correlation and causal tabulations generated
(CI) conducted
analysis?
IV.4 Were results from data analysis Consultatively interpreted on few Consultatively interpreted on Consultatively interpreted Discussing rassessment findings with different key
Interpreted by only 1-2 people
interpreted in a consultative way? occasions many occasions throughout analysis process informants increases the likelihood of an accurate
interpretation/understanding of the situation
Average Score Part IV 0.00
PROCESS MANAGEMENT, CAPACITIES AND ASSESSMENT PREPARATION FIELD WORK DATA ENTRY AND ANALYSIS
PARTNERSHIPS
#DIV/0! 0.00 0.00 0.00
TOTAL Average Score #DIV/0!
You have 21 mandatory 1.1 2.1 4.1
3.1
question(s) not ansewered.
1.2 2.2 3.2 4.2
Please note that the highlighted cells in red
correspond to mandatory question(s) that 1.3 2.3 3.3 4.3
have not been answered. In order to have a
1.4 2.4 3.4 4.4
proper calculation, please make sure that
you answer (click) all mandatory questions. 1.5 2.5 3.5
2.6 3.6
Average Scores and Levels of Accuracy
Score Range Accuracy Classification
>5 High #DIV/0! #DIV/0!
3.1 - 5 Moderate to High #DIV/0! #DIV/0!
1 to 3 Moderate #DIV/0! #DIV/0!
<1 Low #DIV/0! #DIV/0!
Related docs
