; Models of SNAP-Ed and Evaluation
Learning Center
Plans & pricing Sign in
Sign Out
Your Federal Quarterly Tax Payments are due April 15th Get Help Now >>

Models of SNAP-Ed and Evaluation

VIEWS: 210 PAGES: 10

  • pg 1
									                                         Models of SNAP-Ed and Evaluation

                                  Questions from Potential Project Partners

FNS is pleased to have the opportunity to respond to the questions received from numerous nutrition
educators considering applying to become demonstration projects in the second wave of the Models of
SNAP-Ed and Evaluation study. Without exception every question was thoughtful and provides proof
that riding the second wave will be just as exhilarating as the first!

While each one has a unique feature to it, we have attempted to group the questions and FNS responses
according to four broad themes. Almost half of the inquiries asked FNS to describe the nature of the
potential relationships between the demonstration projects and the FNS evaluation contractor,
especially with respect to the independence of data collections and methodologies. Another theme of
inquiry sought clarification on appropriate target audiences. Still another asks about the appropriate
use of incentive payments, “customary and usual” SNAP-Ed funding, and participant/subject incentives.
The final “theme” is more of an “other” category. The questions within this theme are no less important
than the others; just a bit more difficult to categorize.

While reviewing the questions and answers we ask that you keep the following points in mind.

      Information required on the application form. When completing the Application, please
       describe your proposed FY 2012 demonstration intervention and evaluation as precisely as you
       expect to describe it in your State’s SNAP-Ed plan. All anticipated expenses should be the result
       of your SNAP-Ed intervention and evaluation, and should not be related to your activities as a
       part of the FNS evaluation.

      Incentive payments as a state cost share. Incentive payments are derived from Federal funds.
       As such, they cannot be used as a basis to obtain additional Federal reimbursement. If selected
       as a demonstration project, you will receive reimbursements for 50% of approved, allowable
       administrative costs as outlined in the SNAP-Ed Plan Guidance.

      Incentive payment uses. FNS expects that selected demonstration projects will incur costs as a
       direct result of their participation in this study. Incentive payments should be used, for
       example, to cover the cost of travel to FNS headquarters for two meetings and the support of a
       liaison to FNS and its evaluation contractor. At the implementing agency’s discretion, incentive
       funds may be used to author and disseminate academic papers, travel to and from conferences
       and other appropriate purposes. An implementing agency may choose to devote any or all of its

      incentive payments for its intervention and/or evaluation expenses; however, such expenses
      will not receive Federal reimbursement.

Demonstration Projects & the FNS Evaluation Contractor

  1. How is the independent contractor selected? Does the model program have any input into the
     evaluation design?

      Answer: The independent contractor is selected by FNS through standard, competitive, Federal
      procurement procedures.

      Cooperation between demonstration projects and the FNS evaluation contractor is very
      important, and it is FNS’ wish that demonstration projects remain as unfettered as possible in
      the development and administration of their interventions and evaluation. However, there are
      limits. It must be the case that demonstration projects follow SNAP-Ed Guidelines, and remain
      replicable and evaluable.

  2. Is the FNS independent evaluator going to collect data different and apart from what we will
     collect for our self-initiated evaluation?

      Answer: Generally, yes. One of the goals of the Models of SNAP-Ed and Evaluation study is to
      identify logistically practical yet methodologically robust evaluation techniques. Therefore, to
      the greatest extent possible, the demonstration project and FNS will strive to maintain
      independence between their respective impact/outcome evaluations. Perhaps paradoxically,
      maintaining independence can require a fair amount of coordination.

  3. If [the] FNS independent evaluator is going to collect different data, [are they] going to use our
     data as well in their analysis?

      Answer: The FNS impact evaluation will rely on data collected by the FNS evaluation contractor.
      As part of the assessment of the demonstration project’s self-initiated evaluation, the FNS
      contractor will examine the results of the demonstration project’s data collection ergo its data.
      That data will not be used, though, to generate FNS’ quantitative results.

  4. Will the FNS independent evaluator be using data collection tools and methodologies different
     from us?

      Answer: FNS will employ the tools and techniques that will maximize our ability to detect and
      describe behavioral change. The demonstration project may or may not choose similar tools and

  5. Will there be duplication and overlaps in data collection between us and the FNS independent
    Answer: There will certainly be some overlap. For example, both FNS and the demonstration
    project will necessarily collect impact/outcome data from at least a subset of the target
    population – those that will take part in the demonstration project’s intervention, i.e. those in
    the experimental condition.

6. Will the FNS independent evaluator be focused on collecting data only from the control groups/
   subjects or from both the treatment and control groups/subjects?

    Answer: FNS will collect data from both treatment and control groups, pre- and post-

7. Will the FNS independent evaluator do their analysis independently from ours, or will they work
   with us to analyze our data and their data?

    Answer: The FNS analyses will be independent from those of the demonstration projects, though
    the independent evaluator will be gathering information that describes the processes associated
    with the demonstration projects’ analyses and they will be comparing their results with yours.

8. Will the FNS independent evaluator be a resource available to help us develop our data
   collection tools and methodologies or must we be totally independent from them?

    Answer: No. The FNS evaluation contractor should not be considered a resource for the
    development of any part of your intervention or evaluation.

9. How is the FNS independent evaluator going to evaluate our evaluation? What does that entail?

Answer: As succinctly as possible, the demonstration projects’ evaluations will be assessed on the
basis of eight criteria.

    -   Research Objectives and Hypotheses
                   i. Clarity of research questions/hypotheses
                  ii. Alignment of evaluation goals and objectives with intervention activities
    -   Viability of the Comparison Strategy
                   i. Appropriateness of the control and/or comparison groups
                  ii. Threats to the validity of the design
    -   Sampling Size and Strategy
                   i. Sample size estimations
                  ii. Appropriateness of sample frame
                 iii. Methods of selecting sample participants from the population
                 iv. Recruitment
    -   Outcome Measures
                   i. Quality of the data collection instruments
                  ii. Alignment of evaluation measures with intervention activities
    -   Data Collection
                   i. Timing and practicality of the data collection schedule
                  ii. Rigor of the data collection process
                iii. Quality of the data collection process (data collector training and procedures
                     that reduce bias and do not promote non-response)
    -   Data Analysis
                  i. Sample characteristics and baseline comparability
                 ii. Statistical methods used to assess the program impacts
                iii. Methods for handling attrition bias; procedures for accounting for missing data;
                     and subgroup analyses
    -   Attrition
    -   Missing data and the identification of, and correction for, associated biases

10. A requirement for participating states: Selected demonstration projects will be required to
    cooperate with the independent, impact evaluation conducted by FNS through its contractor.
    The FNS evaluation will assess each project's intervention and evaluation. Each selected
    demonstration project will receive $100,000 to offset costs associated with participating in the
    independent FNS impact evaluation.

    In order to determine if the intervention and control/comparison groups could meet this
    requirement, we need to know more about the potential burden of an independent impact
    evaluation. Could you describe potential activities of the independent evaluator? How would
    the external evaluator interact with the internal research and evaluation effort?

    Answer: The independent evaluator’s activities have been have been sketched within the above

    As a prerequisite to selection, demonstration projects must agree to attend two meetings at FNS
    headquarters in Alexandria, Virginia. The expenses associated with these trips are not costs
    associated with the administration and evaluation of a SNAP-Ed intervention and are therefore
    not subject to 50% Federal reimbursement. This is an example of costs that will be covered out
    of the incentive. Other examples of burdens that would have to be borne with the use of the
    incentive might be the time and effort implementing agency staff might spend facilitating access
    to clients; coordinating evaluations; responding to/participating in contractor process evaluation
    questionnaires/interviews; etc.

11. How does FNS envision the two evaluation efforts (FNS’ independent evaluation and the
    demonstration project’s self-initiated evaluation) working effectively, without tripping over one
    another or alienating participants? Is it valid for the self-initiated evaluation to be conducted at
    a different time with a similar, but not identical, group of participants?

    Answer: This is one of the great challenges of this study that must be met on a case-by-case
    basis. In Wave I, FNS is attempting to ensure the independence of the evaluations by
    maintaining temporal separation; by randomly staggering the administration of evaluation
    instruments; by measuring correlated, but not identical outcomes; by obtaining responses from
    different affected audiences (e.g. children’s caregivers as opposed to the children themselves);
    and through various combinations of all of these methods.

      Again, what is or is not valid must be considered on a case-by-case basis.

Target Audiences

  12. In the 2009 ASNNA Conference PowerPoint, 2 groups were noted as hard to reach: The Elderly
      and Newly Eligible. However, the new 16 core messages all focus on Moms and/or Children. In
      the first wave, 3 programs targeting small children and one program targeting women were
      winners. Should we target based on Core Messages or Based on Hard to Reach?

      Answer: With respect to the prospective demonstration project’s target audience, seven (7)
      points will be award if the applicant can demonstrate that their intervention will effectively
      target households participating in or eligible for SNAP. The determination of the target audience
      – mothers, children, the elderly, etc. – is incumbent upon the applicant. Only the nutrition
      educator can know how the needs of his/her community can best be met. However, conducting
      a needs assessment can best help you describe your target audience.

      The last paragraph under SNAP-Ed Guiding Principle 3 (SNAP-Ed Plan Guidance, p. 93) states:

              “FNS’ national focus on women and their children does not preclude States from also
              offering SNAP-Ed to other SNAP audience segments such as the elderly, men, or adults
              without children. A needs assessment of the SNAP eligible population will help States
              target SNAP-Ed effectively and efficiently to yield the greatest change in dietary behavior
              among the largest number of SNAP eligibles.”

  13. Based upon the completion of the First Wave of selected projects, are there other target
      audiences, nutrition education methodologies, evaluation strategies or other demographics that
      FNS would like to see addressed in “Wave 2”?

      Answer: The selection criteria for “Wave 2” are not substantially different from that used to
      choose demonstration projects for the first wave. However, the application period for Wave 2 is
      significantly longer, as is the period that a demonstration project will have to stage and evaluate
      its intervention (October 2011 through June 2012). These elements alone are likely to lead to a
      wider variety of applicants than could not participate in the First Wave, but it is paramount that
      selected demonstration projects stage interventions that are consistent with SNAP-Ed Guidance,
      and that they be replicable, and evaluable.

  14. A selection criterion states: The intervention effectively targets households participating in or
      eligible for SNAP. Does this include all three categories of eligible audiences? Will each category
      receive the same number of points?

      Answer: All three categories of eligible audiences will receive the same number of points.

  15. A selection criterion states: The proposed demonstration project has indicated that if other
      nutrition education or promotions are delivered to the target audience they are delivered to
      both the treatment and control groups during the course of the project. Could you define and
      provide examples of promotions in context of a school-based intervention?

      Answer: The purpose of this criterion is to provide reasonable assurance that subject assignment
      to experimental conditions can be made in a fashion that allows FNS to isolate the effect of the
      demonstration project’s intervention. The word “promotion” in the statement of the criterion is
      used to invoke the broadest possible definition of nutrition education or message. What is
      important here is that it must be possible to create treatment and control groups in which if one
      group is exposed to additional messages that can be reasonably interpreted as either reinforcing
      or undermining the themes of the nutrition education intervention, so must the other.

  16. We have rolling recruitment for this program, and have approximately 25 families enrolled in
      different counties at any one time. The timeline specifies that all pre-intervention data
      collection will have occurred by Oct 2011. Can the intervention of the first group start while the
      pre-intervention for the next group is not yet completed?

      Answer: Please keep in mind that all interventions must be finished by mid-June, 2012, so that
      post-intervention data can be collected in a timely manner. Selected demonstration projects will
      have the option to begin their interventions as early as October 1, 2011, but no sooner.

      Also, due to the challenges that can be associated with seasonally adjusting data, FNS will be
      reluctant to attempt the evaluation of an intervention spread over multiple time periods. For
      example, an intervention that is staged in the spring, and then again in the fall, might produce
      data that is not comparable due to such factors as weather, and the price and availability of
      fruits and vegetables.

  17. Can we share the curriculum in advance of the FNS demonstration evaluation?

      Answer: Assuming that doing so does not threaten the integrity of the evaluation by making the
      selection of valid control groups difficult, for example, then, yes. FNS wants to avoid exposing
      individuals not formally a part of the intervention to the treatment. Presenting your curriculum
      at a professional conference would be acceptable, but we would hope that you share our interest
      in insuring that this would not lead to its administration to the FNS control group.

Funding, the Use of Incentive Payments, and Incentivizing

  18. What specific aspects of development can [incentive payments] fund? [For example], creation
      of data collection tools, testing of the tools, software, hardware, training, administration,

      support staffing, testing, etc.? Are there any restrictions to the spending of the $100K incentive

      Answer: The $100,000 incentive, administered in three installments under subcontract to the FNS
      evaluation contractor, can be put to any good purpose. The only restriction on its use is that it
      may not be used as the basis for matching to any other source of Federal Funds.

  19. For pre/post and analysis evaluation activities during the implementation of the project in
      FY2012, can they also be funded by the 50% match against local, county or state match?

      Answer: If your FY 2012 SNAP-Ed plan has been approved for these activities, then yes.

  20. What types of incentives is FNS willing to provide to organizations where the independent
      evaluation will be conducted? What about participants? Will any of these incentives be paid by
      FNS directly, or will the Implementing Agencies need to draw on the $100,000 incentive
      awarded by FNS?

      Answer: Many of the demonstration projects in Wave I of the study are providing incentives to
      study participants. These incentives will be paid from the $100,000 nutrition educators receive
      as an incentive for project participation.

      The FNS evaluation contractor plans to incentivize the participation of organizations and
      individual respondents to its surveys, but it does not cover, compensate, or otherwise
      remunerate the incentives offered by the nutrition educators for the completion of their
      independent instruments.

  21. Would the control groups identified be compensated for time and effort related to participation
      as a control group in this evaluation study?

      Answer: FNS is incentivizing participation in the control group for its contractor-led evaluation in
      Wave I, and it is anticipated that we will do so again in Wave II. However, whether control group
      participation is incentivized in your self-evaluation, is up to you and would have to be
      compensated by you.

Other Topics

  22. Is it possible for a SNAP-Ed organization to make more than one application in “Wave 2” for 2 or
      more different projects?

      Answer: FNS wishes to identify as many effective, innovative and evaluable SNAP-Ed
      interventions, each with its own impact/outcome evaluation, as possible. If a single SNAP-Ed
      organization is staging more than one unique, nutrition education intervention that might serve
      as a model to others, and if each intervention has its own impact/outcome evaluation
    component, then a single SNAP-Ed organization may apply for each unique intervention being
    conducted under its aegis.

23. The selection criteria clearly favors evaluation studies that support the random assignment of
    multiple units (person, classes, stores, etc.) to treatment and control conditions than the quasi-
    experimental, non-random assignment of matched units to both treatment and comparison
    groups by giving the scientific method 10 more possible points. What category of evaluation
    study is being employed by the 4 winners of the First Wave?

    Answer: Please review Part III, on page 3, of the Demonstration Project Selection Criteria which
    addresses the suitability of the proposed demonstration project for an FNS evaluation. A
    maximum of twenty (20) points will be awarded if the project can support the random
    assignment of multiple units to treatment and control conditions **OR** the quasi-
    experimental, non-random assignment of matched units to both treatment and comparison

    Although an experimental design is the gold standard for evaluation, it is not always possible to
    conduct a study that supports random assignment. In such cases a strong quasi-experimental
    design should be used. In the first wave of the Models of SNAP-Ed and Evaluation study, FNS’
    evaluation contractor is employing quasi-experimental designs in three instances, and a fully
    randomized, experimental design in the fourth.

    Of the outcome/impact, self-initiated evaluations to be conducted by the four demonstration
    projects in Wave I, one will be fully randomized, experimental; one will be quasi-experimental;
    and two will administer pre- and post-tests without a control group.

24. A selection criterion states: The proposed intensity and/or duration of the intervention provides
    individuals with multiple exposures to the nutrition education message and is sufficient to
    produce a significant, measurable effect.

    Does FNS have parameters for the number of exposures that would be considered sufficient?
    Does "significant” imply statistical significance (e.g., p-value less than .05), and does
    "measurable" imply an effect size calculation (e.g., Cohen's d)?

    Answer: Through the literature review, results of formative research, and/or pilot test findings
    that you provide in your application to justify your approach and expected effect(s), the selection
    committee will decide whether your proposed intervention shows the potential for measurable

    FNS is designing its own independent evaluations of the selected projects such that each
    evaluation can detect a change of 0.3 standard deviations (Cohen’s d) at a significance level of
    0.05 (Type I error) and a power (1 – Type II error) of 0.8. The selection committee will be

    interested in proposed interventions that can generate at least a 0.3 effect in an environment
    that will facilitate sample design of the prescribed power and precision.

25. What number of participants and what number of controls are expected?

    Answer: It depends upon the project. A single set of numbers that can be applied in all
    circumstances cannot be provided. For its purposes, FNS will require samples large enough to
    detect a 30% shift in the consumption distribution with a power of 0.8 and 95% confidence. How
    large these samples must be will depend on the size of the target population, whether it can be
    meaningfully stratified, the extent to which the population is distributed into clusters (schools, or
    classes, for example), the level to which individuals within these clusters resemble one another
    with respect to their tastes, habits and influences, our ability to measure and include relevant
    covariates, the expected attrition rate and many other factors.

26. A selection criterion states: The intervention budget is provided as per SNAP-Ed annual

    Rural communities are obviously important in a State like [ours]. They also can provide excellent
    settings for research. However, it is difficult to use rural communities in a SNAP-Ed model that
    examines whether a multi-channel intervention would be more effective than an intervention
    limited to school-based education. This is due to the fact that allowability of costs is based on
    census tracts or high-volume grocery stores that are not available in most rural settings. Would
    it be possible to request a waiver that would allow the intervention costs of the research project
    for one year, in order to determine the effectiveness of a multi-channel approach?

    Answer: A principal goal of this study is to provide examples of successful nutrition education
    interventions that are with reasonably minor modifications “doable” by others. It is difficult to
    conceive of how an intervention that requires a waiver of regulations could be easily replicated
    by other nutrition educators. It must be possible to use the standard procedures currently
    published within the SNAP-Ed Guidance to reproduce your study.

27. The timeline mentions that Demonstration Project members will attend two 2-day meetings at
    FNS in Alexandria, VA. What additional time commitment is expected for implementing the FNS
    evaluation outside of direct program time? Or put another way, what percentage FTE is
    expected to be devoted to the FNS evaluation not counting the program time itself?

    Answer: First a point of clarification: Demonstration project representatives will attend two full
    (one) day meetings at FNS in Alexandria, VA.

    Even with more, project-specific information, it would not be possible to hazard a guess at the
    “percentage FTE *that might be+ expected to be devoted to the FNS evaluation …” Yet FNS is
    confident that the total incentive payment of $100,000 that demonstration projects will receive

    for their participation should adequately cover all expenses incurred, directly or indirectly, as a
    consequence of participation in the study.

28. We have conference presentations already scheduled for this project. Does this constitute

    Answer: Yes, it does.

29. Can we publish parts of this project unrelated to the FNS evaluation?

    Answer: You can publish findings from your evaluation but not in advance of the release of the
    FNS report. If it is part of the project; it is related to the FNS evaluation.


To top