immunization by xiuliliaofz


									                                                               OMB Program Assessment Rating Tool (PART)

                                                                             Competitve Grant Programs

Name of Program: 317 Immunization Program
Section I: Program Purpose & Design (Yes,No, N/A)

            Questions                        Ans.                     Explanation                                               Evidence/Data                              Weighting Weighted Score
  1    Is the program purpose clear?         Yes    The goal of the National Immunization Program          Cited in the NIP Strategic Plan mission and GPRA plan.            20%           0.2
                                                    (NIP) is to prevent disease, disability and death in   The 317 program is authorized through the Public Health
                                                    children (and increasingly) adults through             Service Act Section 317j, to provide vaccines for
                                                    vaccination. NIP is comprised of two primary grant     individuals (later specified as children, adolescents and
                                                    programs to states - 1) the discretionary 317          adults) free of charge and to provide preventive health
                                                    program; and 2) the mandatory Vaccines for             services related to the delivery of immunizations. With
                                                    Children (VFC) program. The 317 grant program          the establishment of VFC in 1994, the 317 program
                                                    provides some vaccines for those who are not           shifted more of its efforts towards vaccine assurance
                                                    eligible to receive vaccines under any other           rather than direct provision of vaccines. For global
                                                    insurance program, but primarily focuses on            activities, Congress authorizes NIP's global activities
                                                    assuring vaccines for the entire population through:   through appropriations language and NIP's strategic plan
                                                    1) public information and outreach; 2) quality         includes a goal to eliminate and eradicate diseases
                                                    assurance within the medical community; 3)             globally as well as domestically. However, there is no
                                                    assessment of immunizations within the population;     clear guiding principle for how CDC prioritizes its global
                                                    4) surveillance of disease and vaccine safety; 5)      activities other than that CDC works closely with WHO
                                                    immunization registries; 6) vaccine management.        and its priorities to determine what international activities
                                                    CDC also supports global efforts such as               to undertake.
                                                    eradicating polio and eliminating measles because
                                                    to eliminate/eradicate diseases in the U.S.
                                                    completely it is necessary to eliminate/eradicate
                                                    them internationally.

  2    Does the program address a            Yes    CDC focuses on activities (including service            In the U.S., 11,000 babies are born each day that must           20%           0.2
       specific interest, problem or need?          delivery and supportive services) to ensure that       be vaccinated (approximately 4 million per year), and
                                                    children domestically (and increasingly adults) and    need to receive 12-16 doses of vaccine by 18 months,
                                                    internationally receive the appropriate and            and 16-20 doses through childhood. The immunization
                                                    recommended vaccines. CDC is also using the 317        rates for newer vaccines such as varicella and Hep. B
                                                    program to try and reach "pockets-of-need," or         have not yet reached 90 percent coverage. 317 also
                                                    specific populations where immunization rates are      serves as a gap-filler for those children who are not
                                                    much lower than the national average.                  receiving vaccines from any other provider.

                                                                                                                                                                                       FY 2004 Budget
                                                                                                 1                                                                                         Fall Review
   3   Is the program designed to have      Yes    Although there are no good estimates for how much         For vaccine purchase, in FY 2001, CDC estimates that            20%           0.2
       a significant impact in addressing          states contribute to vaccine purchase/infrastructure      states provided $116 million in purchases through the
       the interest, problem or need?              activities, NIP estimates that it provides the majority   Federal contract (excluding how much states spent
                                                   of the public funding for vaccine purchase and            independantly purchasing vaccines), while CDC spent
                                                   assurance activities. For vaccine purchase, the           $201 million in 317 funds. NIP has helped increase
                                                   Federal contribution (both 317 and VFC) represents        overall childhood immunization rates from 55 percent in
                                                   a majority of the funds (a 2000 IOM report                1992 to an all-time high of approximately 80% in 2000.
                                                   estimates the state contribution to vaccines on the
                                                   Federal contract ranges from less than 10 to 30
                                                   percent) so that increases and decreases in Federal
                                                   vaccine purchase funds will have an impact on
                                                   coverage levels.

   4   Is the program designed to make      Yes    The 317 program provides vaccines for those that                                                                          20%           0.2
       a unique contribution in                    do not receive vaccines through other private or
       addressing the interest, problem            public insurance programs (largely the underinsured
       or need (i.e., not needlessly               with large copayments), and also supports
       redundant of any other Federal,             outreach, education, and quality assurance
       state, local or private efforts)?           activities.

   5   Is the program optimally designed    Yes    CDC provides direct financial assistance to                                                                               20%           0.2
       to address the interest, problem            grantees for infrastructure activities and a line of
       or need?                                    credit for vaccine purchase since it is from a single

Total Section Score                                                                                                                                                          100%         100%

Section II: Strategic Planning (Yes,No, N/A)

            Questions                       Ans.                      Explanation                                                Evidence/Data                             Weighting Weighted Score
   1   Does the program have a limited      Yes    CDC's overall outcome goal is to reduce the               Strategic Plan examples: 1) Eradicate/eliminate/control all     14%           0.1
       number of specific, ambitious               number of indigenous cases of vaccine preventable         vaccine-preventable disease disability and death in the
       long-term performance goals that            diseases in the U.S. to 0 by 2010. NIP uses               U.S. and globally ; 2) Raise and sustain vaccine coverage
       focus on outcomes and                       Healthy People 2010, its strategic plan and GPRA          levels in all populations for all recommended vaccines.
       meaningfully reflect the purpose            to guide and measure its activities. The five-year
       of the program?                             strategic plan (2000-2005) is more qualitative and
                                                   process-oriented, and is more of a vision document
                                                   to help guide CDC's overall activities, while GPRA
                                                   is used to measure progress on achieving specified
                                                   Healthy People 2010 goals.

                                                                                                                                                                                       FY 2004 Budget
                                                                                                 2                                                                                         Fall Review
2   Does the program have a limited    Yes   The GPRA plan includes several goals to help            Examples: 1) The number of indigenous cases of: a)             14%       0.1
    number of annual performance             measure progress on this long-term goal annually        measles will go from 63 in FY 2000 to 60 in FY 2002 to
    goals that demonstrate progress          including vaccine coverage levels, annual targets       50 in FY 2004; b) rubella will go from 176 in FY 2000 to
    toward achieving the long-term           for specific diseases, and global polio eradication     20 in FY 2002 to 15 in FY 2004; c) Hib from 183 in FY
    goals?                                   efforts.                                                2001 to 175 in FY 2002 to 150 in FY 2004; c) polio will
                                                                                                     remain at 0; 2) achieve or sustain immunization coverage
                                                                                                     of at least 90% in children 19-35 months of age for
                                                                                                     recommended vaccines each year; 3) achieve and
                                                                                                     sustain zero cases of polio by 2005.

3   Do all partners (grantees, sub-    Yes   In the FY 2003 grant announcement, NIP will             In FY 2003, grantees will be required to develop               14%       0.1
    grantees, contractors, etc.)             require grantees to develop measurable outcomes         measurable objectives in relation to the following GPRA
    support program planning efforts         in relation to five of its GPRA goals. Previously,      goals: 1) Reduce the number of indigenous cases of
    by committing to the annual              NIP included 15 HP 2010 goals as the objectives         vaccine-preventable disease; 2) ensure that 2 year-olds
    and/or long-term goals of the            that grantees should be working towards and             are appropriately vaccinated; 3) improve vaccine safety
    program?                                 reporting progress on in their applications.            surveillance; 4) increase routine vaccination coverage
                                                                                                     levels for adolescents; 5) increase the proportion of adults
                                                                                                     who are vaccinated annually against influenza and ever
                                                                                                     vaccinated against pneumoccocal diseases. Previously,
                                                                                                     grantees were required to develop and measure progress
                                                                                                     on their own objectives that were in support of CDC's
                                                                                                     overarching goals.

4   Does the program collaborate       Yes   CDC leverages the National Vaccine Program                                                                             14%       0.1
    and coordinate effectively with          Office to coordinate activities among different HHS
    related programs that share              agencies. CDC collaborates closely with NIH on
    similar goals and objectives?            IOM vaccine trials and CMS on the development of
                                             GPRA goals, reimbursement rates, and
                                             administration fees.

5   Are independent and quality        No    While NIP has undertaken several management             Two divisions of the program have had an independent           14%       0.0
    evaluations of sufficient scope          evaluations over the past few years to see if certain   review of their management structure and operations
    conducted on a regular basis or          aspects of the program can be improved, there           within the last few years; NIP recently undertook an
    as needed to fill gaps in                have been no comprehensive evaluations looking at       evaluation of its NIP-wide IT systems, which will have
    performance information to               how well the program is structured/managed to           recommendations in the Fall; an independent contractor
    support program improvements             achieve its overall goals. A 2000 IOM report, while     was brought in to review and help develop the NIP
    and evaluate effectiveness?              comprehensive in scope, focused more on how the         strategic plan; NIP brought in an independent contractor
                                             Federal government could improve its ability to         to review its indirect cost rates.
                                             address childhood immunizations rather than
                                             evaluating how well the 317 and VFC programs, as
                                             currently structured and operated, were improving
                                             immunization rates among children.

                                                                                                                                                                          FY 2004 Budget
                                                                                          3                                                                                   Fall Review
   6   Is the program budget aligned         No    For the vaccine purchase activities, yes, for state      There is no specific mechanism or measurement that               14%           0.0
       with the program goals in such a            infrastructure, no. For the infrastructure activities,   links NIP's infrastructure budget and activities to its
       way that the impact of funding,             there are a lot of different activities that comprise    performance goals.
       policy, and legislative changes on          infrastructure (education, outreach, administration
       performance is readily known?               of vaccines), so it's unclear exactly how
                                                   funding/policy/legislative changes will affect

   7   Has the program taken                 No    NIP has not addresssed any of the above cited                                                                             14%           0.0
       meaningful steps to address its             deficiencies (comprehensive evaluations to
       strategic planning deficiencies?            determine program effectiveness or aligning the
                                                   immunization infrastructure budget with
                                                   performance goals).

Total Section Score                                                                                                                                                          100%         57%

Section III: Program Management (Yes,No, N/A)

            Questions                       Ans.                       Explanation                                              Evidence/Data                              Weighting Weighted Score
   1   Does the agency regularly collect    Yes    CDC collects grantee information from a variety of       Disease rates from surveillance and the National                 10%           0.1
       timely and credible performance             sources including annual progress reports from           Immunization Survey have helped CDC determine internal
       information, including information          states, a financial status report, and at least one      priorities (e.g., what diseases/populations scientists
       from key program partners, and              site visit per year. CDC also receives information       should be looking at), and their activities in collaboration
       use it to manage the program and            quarterly from the National Immunization Survey          with states, as well as how well their grantees are
       improve performance?                        (NIS) on immunization coverage across all 50             achieving immunization coverage levels. For grantees, if
                                                   states, and disease surveillance information. CDC        CDC sees that there are low immunization levels within a
                                                   is moving towards a more formula-based grant in          jurisdiction, CDC may provide technical assistance or
                                                   FY 2003 that will take into account more objective       direct additional funds to this area.
                                                   criteria, including performance. NIP's project
                                                   officers have constant contact with grantees to
                                                   determine if a change in program direction is
                                                   warranted. NIP also conducts quality assurance
                                                   reviews of private providers to make sure that they
                                                   are administering the vaccines properly, and
                                                   storing/rotating them.

                                                                                                                                                                                       FY 2004 Budget
                                                                                                   4                                                                                       Fall Review
2   Are Federal managers and           No    NIP's Federal program managers, while responsible                                                                     10%       0.0
    program partners (grantees,              for cost and schedule, do not have performance-
    subgrantees, contractors, etc.)          based contracts that integrate program performance
    held accountable for cost,               into their personnel evaluations. Within CDC, only
    schedule and performance                 SES have performance-based contracts and NIP
    results?                                 has no SES. For grantees, while NIP reviews
                                             grantees' vaccine coverage levels and progress
                                             reports to determine if they are meeting their stated
                                             objectives, NIP doesn't reallocate funds as a result
                                             of grantees not meeting their objectives, and tends
                                             to provide technical assistance instead.

3   Are all funds (Federal and         Yes   NIP generally obligates almost all of its funds by the   Grantees tend to have less than 10% of their obligations     10%       0.1
    partners’) obligated in a timely         end of the year, and has many mechanisms to              carried over to the following year (approx. $1,000-
    manner and spent for the                 make sure that granteees spend their funding for         $100,000) and have to use their carry-over in lieu of new
    intended purpose?                        the intended purpose.                                    funds. NIP also conducts site visits to assess grantee
                                                                                                      obligation patterns and how funds are spent, and interacts
                                                                                                      frequently with grantees through conference calls to
                                                                                                      monitor activities and progress. Grantees are required to
                                                                                                      provide a detailed budget by object class, so if they want
                                                                                                      to move funds around they have to notify CDC. CDC's
                                                                                                      central program and grants office has also started site
                                                                                                      visits to focus on management/funding issues.

4   Does the program have              No    While NIP is starting to undertake efforts such as a     Efficiencies: NIP is converting to some electronic           10%       0.0
    incentives and procedures (e.g.,         review of its IT structure to improve the                processing, including its disease reporting system,
    competitive sourcing/cost                effectiveness and efficiency of the the program,         vaccine ordering system, and collecting records from
    comparisons, IT improvements)            generally there are no existing procedures in place      providers to improve efficiency, and is undertaking a
    to measure and achieve                   to achieve efficiencies and cost-effectiveness in        comprehensive review of its IT positions/activities. While
    efficiencies and cost                    administering the program.                               CDC centrally cost-competes for certain procurement and
    effectiveness in program                                                                          other administrative activities, the program doesn't cost-
    execution?                                                                                        compete for services. Cost-Effectiveness: There are no
                                                                                                      dollars per unit service. CDC has achieved some cost
                                                                                                      savings in vaccine purchase through having a single
                                                                                                      Federal contract, contracts with multiple manufacturers
                                                                                                      and re-competing vaccine bids every four years. NIP also
                                                                                                      contracts with GSA to help states establish vaccine

                                                                                                                                                                         FY 2004 Budget
                                                                                           5                                                                                 Fall Review
   5     Does the agency estimate and          No    While CDC includes the full cost of its activities                                                                    10%       0.0
         budget for the full annual costs of         including overhead, program performance cannot
         operating the program (including            be readily identified with changes in funding levels.
         all administrative costs and
         allocated overhead) so that
         program performance changes
         are identified with changes in
         funding levels?

   6     Does the program use strong           No    The HHS Financial Statement Audit cited two               The HHS Financial Statement Audit cites no material         10%       0.0
         financial management practices?             reportable conditions regarding the manual nature         weaknesses, but two reportable conditions: (a) Financial
                                                     of CDC's accounting processes, although it did not        statements had to be prepared manually to ensure
                                                     find any internal material weaknesses. Until the          accuracy; (b) CDC had to undertake a cumbersome
                                                     HHS-wide Unified Financial Management System is           process to reconcile its reimbursable agreements at the
                                                     in place, CDC will not be able to fully automate its      end of the year.
                                                     financial accounting practices. However, CDC has
                                                     generally made improvements to its financial
                                                     management processes over the past few years,
                                                     including restructing its budget and financial
                                                     accounting system to more accurately track CDC's
                                                     expenditures and hiring a consulting firm to develop
                                                     a more consistent and accurate system for charging

   7     Has the program taken                 Yes   In its FY 2003 application, NIP is trying to formalize    Grantee applications will be ranked based on: 1) plan; 2)   10%       0.1
         meaningful steps to address its             its application criteria, requiring grantees to provide   objecitves; 3) methods; 4) evaluation; previously,
         management deficiencies?                    more quantifiable objective information in its            grantees were primarily funded based on population and
                                                     application and annual progress reports, and              need.
                                                     developing more clear evaluation criteria. NIP has
                                                     also contracted with a firm to review its IT
                                                     organizational structure and develop a 5-year plan
                                                     to help improve the efficiency of NIP.

8 (Co 1.) Are grant applications               Yes   NIP assigns project officers to review the         In FY 2003, NIP will use the following criteria to rank            10%       0.1
          independently reviewed based on            applications and determine how much funding each applications: 1) plan; 2) objectives; 3) methods; 4)
          clear criteria (rather than                state should receive. Before FY 2003, the funding evaluation.
          earmarked) and are awards made             decisions were based upon the information included
          based on results of the peer               in individual grantee applications, taking into
          review process?                            account historical funding levels and factors like
                                                     state need/population/poverty levels. In FY 2003,
                                                     CDC is formalizing this process to include clear
                                                     criteria for allocating resources.

                                                                                                                                                                                 FY 2004 Budget
                                                                                                    6                                                                                Fall Review
9 (Co 2.) Does the grant competition                   NA        NIP provides funding to all 50 states.                                                                                   0%
          encourage the participation of
          new/first-time grantees through a
          fair and open application process?

10 (Co 3.) Does the program have oversight             Yes       CDC collects information from a variety of sources,                                                                     10%                  0.1
           practices that provide sufficient                     including disease surveillance reports, annual
           knowledge of grantee activities?                      progress reports, and site visits. States also
                                                                 conduct annual program reviews of local health
                                                                 departments and intensive reviews of immunization
11 (Co 4.) Does the program collect                    Yes       NIP makes both aggregate and state performance                                                                          10%                  0.1
           performance data on an annual                         information on coverage levels and disease burden
           basis and make it available to the                    available through its website and Morbitidy and
           public in a transparent and                           Mortality Weekly reports.
           meaningful manner?

Total Section Score                                                                                                                                                                     100%                 60%

Section IV: Program Results (Yes, Large Extent, Small Extent, No)

                Questions                             Ans.                          Explanation                                            Evidence/Data                             Weighting Weighted Score
    1     Has the program demonstrated            Large Extent CDC has made significant progress in achieving its                                                                        25%                  0.2
          adequate progress in achieving its                   long-term goals.
          long-term outcome goal(s)?

                            Long-Term Goal I:                                           Reduce the number of cases of indigenous cases of vaccine-preventable diseases by 2010.
                                          Target: Goal is 0: Polio (from 0), Rubella (from 181 in 97), Measles (From 81 in 97), Diptheria (from 3 in 97), Congenital Rubella (from 5 in 1997), Mumps (from 683 in 97),
                                                  Tetanus (From 50 in 97)
             Actual Progress achieved toward                            FY 2001 Data: Polio: 0; Rubella: 19; Measles: 61; Hib: 183; Diptheria: 2; Congenital Rubella: 2; Tetanus: 27, Mumps: 231.
    2     Does the program (including             Large Extent CDC has largely achieved its annual goals.                                                                                25%                  0.2
          program partners) achieve its
          annual performance goals?

                                   Key Goal I:                      Achieve or Sustain Immunization Coverage of at Least 90% in children 19-35 months of age for recommended vaccines every year.
                          Performance Target:                                                                                    90%
                          Actual Performance:                                                           All at or past 90% except Varicella at 68%
                                  Key Goal II:                           Assist Domestic and International Partners to help achieve WHO's goal of global polio eradication by 2005.
                          Performance Target:                                                              FY 02: 500 cases; FY 03: 200 cases.
                          Actual Performance:                                                                       FY 2001: 483 cases
                                  Key Goal III:                                                      Reduce vaccine-preventable diseases in the U.S.

                                                                                                                                                                                                        FY 2004 Budget
                                                                                                              7                                                                                             Fall Review
                      Performance Target:                       FY 04: polio: 0; measles 50; rubella: 15; Hib: 150; Diptheria: 5; Congenital Rubella: 5; Tetanus: 25; Mumps: 200.
                      Actual Performance:                           FY 2001: polio: 0; measles: 61; Hib 183; Diptheria: 2; Congenital Rubella: 2; Tetanus: 27; Mumps: 231
                                            Footnote: Performance targets should reference the performance baseline and years, e.g. achieve a 5% increase over base of X in 2000.
   3   Does the program demonstrate              No        While NIP has achieved some cost savings                                                                                 25%        0.0
       improved efficiencies and cost                      through negotiating a single Federal contract,
       effectiveness in achieving                          the program does not have a stated efficiency
       program goals each year?                            or cost-effectiveness goal to measure
                                                           progress in this area.
   4   Does the performance of this              N/A       While VFC is similar to the 317 program, VFC                                                                             0%
       program compare favorably to                        serves a distinct population and focuses primarily
       other programs with similar                         on vaccine purchase. The 317 program does some
       purpose and goals?                                  vaccine purchase but also provides a lot of support
                                                           for activities that cover the entire population
                                                           including education, outreach, and surveillance.

   5   Do independent and quality            Small Extent While the more comprehensive IOM report                                                                                   25%        0.1
       evaluations of this program                        indicated that childhood immunization levels are at
       indicate that the program is                       an all-time high and the program has helped
       effective and achieving results?                   contribute to this outcome, this report focused more
                                                          on the appropriate role of the Federal government
                                                          rather than evaluating whether the 317 program, as
                                                          currently structured/managed was effective at
                                                          improving immunization rates among children.

Total Section Score                                                                                                                                                                 100%      42%

                                                                                                                                                                                           FY 2004 Budget
                                                                                                        8                                                                                      Fall Review

To top