OMB Program Assessment Rating Tool (PART)
Competitve Grant Programs
Name of Program: 317 Immunization Program
Section I: Program Purpose & Design (Yes,No, N/A)
Questions Ans. Explanation Evidence/Data Weighting Weighted Score
1 Is the program purpose clear? Yes The goal of the National Immunization Program Cited in the NIP Strategic Plan mission and GPRA plan. 20% 0.2
(NIP) is to prevent disease, disability and death in The 317 program is authorized through the Public Health
children (and increasingly) adults through Service Act Section 317j, to provide vaccines for
vaccination. NIP is comprised of two primary grant individuals (later specified as children, adolescents and
programs to states - 1) the discretionary 317 adults) free of charge and to provide preventive health
program; and 2) the mandatory Vaccines for services related to the delivery of immunizations. With
Children (VFC) program. The 317 grant program the establishment of VFC in 1994, the 317 program
provides some vaccines for those who are not shifted more of its efforts towards vaccine assurance
eligible to receive vaccines under any other rather than direct provision of vaccines. For global
insurance program, but primarily focuses on activities, Congress authorizes NIP's global activities
assuring vaccines for the entire population through: through appropriations language and NIP's strategic plan
1) public information and outreach; 2) quality includes a goal to eliminate and eradicate diseases
assurance within the medical community; 3) globally as well as domestically. However, there is no
assessment of immunizations within the population; clear guiding principle for how CDC prioritizes its global
4) surveillance of disease and vaccine safety; 5) activities other than that CDC works closely with WHO
immunization registries; 6) vaccine management. and its priorities to determine what international activities
CDC also supports global efforts such as to undertake.
eradicating polio and eliminating measles because
to eliminate/eradicate diseases in the U.S.
completely it is necessary to eliminate/eradicate
them internationally.
2 Does the program address a Yes CDC focuses on activities (including service In the U.S., 11,000 babies are born each day that must 20% 0.2
specific interest, problem or need? delivery and supportive services) to ensure that be vaccinated (approximately 4 million per year), and
children domestically (and increasingly adults) and need to receive 12-16 doses of vaccine by 18 months,
internationally receive the appropriate and and 16-20 doses through childhood. The immunization
recommended vaccines. CDC is also using the 317 rates for newer vaccines such as varicella and Hep. B
program to try and reach "pockets-of-need," or have not yet reached 90 percent coverage. 317 also
specific populations where immunization rates are serves as a gap-filler for those children who are not
much lower than the national average. receiving vaccines from any other provider.
FY 2004 Budget
1 Fall Review
3 Is the program designed to have Yes Although there are no good estimates for how much For vaccine purchase, in FY 2001, CDC estimates that 20% 0.2
a significant impact in addressing states contribute to vaccine purchase/infrastructure states provided $116 million in purchases through the
the interest, problem or need? activities, NIP estimates that it provides the majority Federal contract (excluding how much states spent
of the public funding for vaccine purchase and independantly purchasing vaccines), while CDC spent
assurance activities. For vaccine purchase, the $201 million in 317 funds. NIP has helped increase
Federal contribution (both 317 and VFC) represents overall childhood immunization rates from 55 percent in
a majority of the funds (a 2000 IOM report 1992 to an all-time high of approximately 80% in 2000.
estimates the state contribution to vaccines on the
Federal contract ranges from less than 10 to 30
percent) so that increases and decreases in Federal
vaccine purchase funds will have an impact on
coverage levels.
4 Is the program designed to make Yes The 317 program provides vaccines for those that 20% 0.2
a unique contribution in do not receive vaccines through other private or
addressing the interest, problem public insurance programs (largely the underinsured
or need (i.e., not needlessly with large copayments), and also supports
redundant of any other Federal, outreach, education, and quality assurance
state, local or private efforts)? activities.
5 Is the program optimally designed Yes CDC provides direct financial assistance to 20% 0.2
to address the interest, problem grantees for infrastructure activities and a line of
or need? credit for vaccine purchase since it is from a single
contract.
Total Section Score 100% 100%
Section II: Strategic Planning (Yes,No, N/A)
Questions Ans. Explanation Evidence/Data Weighting Weighted Score
1 Does the program have a limited Yes CDC's overall outcome goal is to reduce the Strategic Plan examples: 1) Eradicate/eliminate/control all 14% 0.1
number of specific, ambitious number of indigenous cases of vaccine preventable vaccine-preventable disease disability and death in the
long-term performance goals that diseases in the U.S. to 0 by 2010. NIP uses U.S. and globally ; 2) Raise and sustain vaccine coverage
focus on outcomes and Healthy People 2010, its strategic plan and GPRA levels in all populations for all recommended vaccines.
meaningfully reflect the purpose to guide and measure its activities. The five-year
of the program? strategic plan (2000-2005) is more qualitative and
process-oriented, and is more of a vision document
to help guide CDC's overall activities, while GPRA
is used to measure progress on achieving specified
Healthy People 2010 goals.
FY 2004 Budget
2 Fall Review
2 Does the program have a limited Yes The GPRA plan includes several goals to help Examples: 1) The number of indigenous cases of: a) 14% 0.1
number of annual performance measure progress on this long-term goal annually measles will go from 63 in FY 2000 to 60 in FY 2002 to
goals that demonstrate progress including vaccine coverage levels, annual targets 50 in FY 2004; b) rubella will go from 176 in FY 2000 to
toward achieving the long-term for specific diseases, and global polio eradication 20 in FY 2002 to 15 in FY 2004; c) Hib from 183 in FY
goals? efforts. 2001 to 175 in FY 2002 to 150 in FY 2004; c) polio will
remain at 0; 2) achieve or sustain immunization coverage
of at least 90% in children 19-35 months of age for
recommended vaccines each year; 3) achieve and
sustain zero cases of polio by 2005.
3 Do all partners (grantees, sub- Yes In the FY 2003 grant announcement, NIP will In FY 2003, grantees will be required to develop 14% 0.1
grantees, contractors, etc.) require grantees to develop measurable outcomes measurable objectives in relation to the following GPRA
support program planning efforts in relation to five of its GPRA goals. Previously, goals: 1) Reduce the number of indigenous cases of
by committing to the annual NIP included 15 HP 2010 goals as the objectives vaccine-preventable disease; 2) ensure that 2 year-olds
and/or long-term goals of the that grantees should be working towards and are appropriately vaccinated; 3) improve vaccine safety
program? reporting progress on in their applications. surveillance; 4) increase routine vaccination coverage
levels for adolescents; 5) increase the proportion of adults
who are vaccinated annually against influenza and ever
vaccinated against pneumoccocal diseases. Previously,
grantees were required to develop and measure progress
on their own objectives that were in support of CDC's
overarching goals.
4 Does the program collaborate Yes CDC leverages the National Vaccine Program 14% 0.1
and coordinate effectively with Office to coordinate activities among different HHS
related programs that share agencies. CDC collaborates closely with NIH on
similar goals and objectives? IOM vaccine trials and CMS on the development of
GPRA goals, reimbursement rates, and
administration fees.
5 Are independent and quality No While NIP has undertaken several management Two divisions of the program have had an independent 14% 0.0
evaluations of sufficient scope evaluations over the past few years to see if certain review of their management structure and operations
conducted on a regular basis or aspects of the program can be improved, there within the last few years; NIP recently undertook an
as needed to fill gaps in have been no comprehensive evaluations looking at evaluation of its NIP-wide IT systems, which will have
performance information to how well the program is structured/managed to recommendations in the Fall; an independent contractor
support program improvements achieve its overall goals. A 2000 IOM report, while was brought in to review and help develop the NIP
and evaluate effectiveness? comprehensive in scope, focused more on how the strategic plan; NIP brought in an independent contractor
Federal government could improve its ability to to review its indirect cost rates.
address childhood immunizations rather than
evaluating how well the 317 and VFC programs, as
currently structured and operated, were improving
immunization rates among children.
FY 2004 Budget
3 Fall Review
6 Is the program budget aligned No For the vaccine purchase activities, yes, for state There is no specific mechanism or measurement that 14% 0.0
with the program goals in such a infrastructure, no. For the infrastructure activities, links NIP's infrastructure budget and activities to its
way that the impact of funding, there are a lot of different activities that comprise performance goals.
policy, and legislative changes on infrastructure (education, outreach, administration
performance is readily known? of vaccines), so it's unclear exactly how
funding/policy/legislative changes will affect
performance.
7 Has the program taken No NIP has not addresssed any of the above cited 14% 0.0
meaningful steps to address its deficiencies (comprehensive evaluations to
strategic planning deficiencies? determine program effectiveness or aligning the
immunization infrastructure budget with
performance goals).
Total Section Score 100% 57%
Section III: Program Management (Yes,No, N/A)
Questions Ans. Explanation Evidence/Data Weighting Weighted Score
1 Does the agency regularly collect Yes CDC collects grantee information from a variety of Disease rates from surveillance and the National 10% 0.1
timely and credible performance sources including annual progress reports from Immunization Survey have helped CDC determine internal
information, including information states, a financial status report, and at least one priorities (e.g., what diseases/populations scientists
from key program partners, and site visit per year. CDC also receives information should be looking at), and their activities in collaboration
use it to manage the program and quarterly from the National Immunization Survey with states, as well as how well their grantees are
improve performance? (NIS) on immunization coverage across all 50 achieving immunization coverage levels. For grantees, if
states, and disease surveillance information. CDC CDC sees that there are low immunization levels within a
is moving towards a more formula-based grant in jurisdiction, CDC may provide technical assistance or
FY 2003 that will take into account more objective direct additional funds to this area.
criteria, including performance. NIP's project
officers have constant contact with grantees to
determine if a change in program direction is
warranted. NIP also conducts quality assurance
reviews of private providers to make sure that they
are administering the vaccines properly, and
storing/rotating them.
FY 2004 Budget
4 Fall Review
2 Are Federal managers and No NIP's Federal program managers, while responsible 10% 0.0
program partners (grantees, for cost and schedule, do not have performance-
subgrantees, contractors, etc.) based contracts that integrate program performance
held accountable for cost, into their personnel evaluations. Within CDC, only
schedule and performance SES have performance-based contracts and NIP
results? has no SES. For grantees, while NIP reviews
grantees' vaccine coverage levels and progress
reports to determine if they are meeting their stated
objectives, NIP doesn't reallocate funds as a result
of grantees not meeting their objectives, and tends
to provide technical assistance instead.
3 Are all funds (Federal and Yes NIP generally obligates almost all of its funds by the Grantees tend to have less than 10% of their obligations 10% 0.1
partners’) obligated in a timely end of the year, and has many mechanisms to carried over to the following year (approx. $1,000-
manner and spent for the make sure that granteees spend their funding for $100,000) and have to use their carry-over in lieu of new
intended purpose? the intended purpose. funds. NIP also conducts site visits to assess grantee
obligation patterns and how funds are spent, and interacts
frequently with grantees through conference calls to
monitor activities and progress. Grantees are required to
provide a detailed budget by object class, so if they want
to move funds around they have to notify CDC. CDC's
central program and grants office has also started site
visits to focus on management/funding issues.
4 Does the program have No While NIP is starting to undertake efforts such as a Efficiencies: NIP is converting to some electronic 10% 0.0
incentives and procedures (e.g., review of its IT structure to improve the processing, including its disease reporting system,
competitive sourcing/cost effectiveness and efficiency of the the program, vaccine ordering system, and collecting records from
comparisons, IT improvements) generally there are no existing procedures in place providers to improve efficiency, and is undertaking a
to measure and achieve to achieve efficiencies and cost-effectiveness in comprehensive review of its IT positions/activities. While
efficiencies and cost administering the program. CDC centrally cost-competes for certain procurement and
effectiveness in program other administrative activities, the program doesn't cost-
execution? compete for services. Cost-Effectiveness: There are no
dollars per unit service. CDC has achieved some cost
savings in vaccine purchase through having a single
Federal contract, contracts with multiple manufacturers
and re-competing vaccine bids every four years. NIP also
contracts with GSA to help states establish vaccine
registries.
FY 2004 Budget
5 Fall Review
5 Does the agency estimate and No While CDC includes the full cost of its activities 10% 0.0
budget for the full annual costs of including overhead, program performance cannot
operating the program (including be readily identified with changes in funding levels.
all administrative costs and
allocated overhead) so that
program performance changes
are identified with changes in
funding levels?
6 Does the program use strong No The HHS Financial Statement Audit cited two The HHS Financial Statement Audit cites no material 10% 0.0
financial management practices? reportable conditions regarding the manual nature weaknesses, but two reportable conditions: (a) Financial
of CDC's accounting processes, although it did not statements had to be prepared manually to ensure
find any internal material weaknesses. Until the accuracy; (b) CDC had to undertake a cumbersome
HHS-wide Unified Financial Management System is process to reconcile its reimbursable agreements at the
in place, CDC will not be able to fully automate its end of the year.
financial accounting practices. However, CDC has
generally made improvements to its financial
management processes over the past few years,
including restructing its budget and financial
accounting system to more accurately track CDC's
expenditures and hiring a consulting firm to develop
a more consistent and accurate system for charging
overhead.
7 Has the program taken Yes In its FY 2003 application, NIP is trying to formalize Grantee applications will be ranked based on: 1) plan; 2) 10% 0.1
meaningful steps to address its its application criteria, requiring grantees to provide objecitves; 3) methods; 4) evaluation; previously,
management deficiencies? more quantifiable objective information in its grantees were primarily funded based on population and
application and annual progress reports, and need.
developing more clear evaluation criteria. NIP has
also contracted with a firm to review its IT
organizational structure and develop a 5-year plan
to help improve the efficiency of NIP.
8 (Co 1.) Are grant applications Yes NIP assigns project officers to review the In FY 2003, NIP will use the following criteria to rank 10% 0.1
independently reviewed based on applications and determine how much funding each applications: 1) plan; 2) objectives; 3) methods; 4)
clear criteria (rather than state should receive. Before FY 2003, the funding evaluation.
earmarked) and are awards made decisions were based upon the information included
based on results of the peer in individual grantee applications, taking into
review process? account historical funding levels and factors like
state need/population/poverty levels. In FY 2003,
CDC is formalizing this process to include clear
criteria for allocating resources.
FY 2004 Budget
6 Fall Review
9 (Co 2.) Does the grant competition NA NIP provides funding to all 50 states. 0%
encourage the participation of
new/first-time grantees through a
fair and open application process?
10 (Co 3.) Does the program have oversight Yes CDC collects information from a variety of sources, 10% 0.1
practices that provide sufficient including disease surveillance reports, annual
knowledge of grantee activities? progress reports, and site visits. States also
conduct annual program reviews of local health
departments and intensive reviews of immunization
clinics.
11 (Co 4.) Does the program collect Yes NIP makes both aggregate and state performance 10% 0.1
performance data on an annual information on coverage levels and disease burden
basis and make it available to the available through its website and Morbitidy and
public in a transparent and Mortality Weekly reports.
meaningful manner?
Total Section Score 100% 60%
Section IV: Program Results (Yes, Large Extent, Small Extent, No)
Questions Ans. Explanation Evidence/Data Weighting Weighted Score
1 Has the program demonstrated Large Extent CDC has made significant progress in achieving its 25% 0.2
adequate progress in achieving its long-term goals.
long-term outcome goal(s)?
Long-Term Goal I: Reduce the number of cases of indigenous cases of vaccine-preventable diseases by 2010.
Target: Goal is 0: Polio (from 0), Rubella (from 181 in 97), Measles (From 81 in 97), Diptheria (from 3 in 97), Congenital Rubella (from 5 in 1997), Mumps (from 683 in 97),
Tetanus (From 50 in 97)
Actual Progress achieved toward FY 2001 Data: Polio: 0; Rubella: 19; Measles: 61; Hib: 183; Diptheria: 2; Congenital Rubella: 2; Tetanus: 27, Mumps: 231.
goal:
2 Does the program (including Large Extent CDC has largely achieved its annual goals. 25% 0.2
program partners) achieve its
annual performance goals?
Key Goal I: Achieve or Sustain Immunization Coverage of at Least 90% in children 19-35 months of age for recommended vaccines every year.
Performance Target: 90%
Actual Performance: All at or past 90% except Varicella at 68%
Key Goal II: Assist Domestic and International Partners to help achieve WHO's goal of global polio eradication by 2005.
Performance Target: FY 02: 500 cases; FY 03: 200 cases.
Actual Performance: FY 2001: 483 cases
Key Goal III: Reduce vaccine-preventable diseases in the U.S.
FY 2004 Budget
7 Fall Review
Performance Target: FY 04: polio: 0; measles 50; rubella: 15; Hib: 150; Diptheria: 5; Congenital Rubella: 5; Tetanus: 25; Mumps: 200.
Actual Performance: FY 2001: polio: 0; measles: 61; Hib 183; Diptheria: 2; Congenital Rubella: 2; Tetanus: 27; Mumps: 231
Footnote: Performance targets should reference the performance baseline and years, e.g. achieve a 5% increase over base of X in 2000.
3 Does the program demonstrate No While NIP has achieved some cost savings 25% 0.0
improved efficiencies and cost through negotiating a single Federal contract,
effectiveness in achieving the program does not have a stated efficiency
program goals each year? or cost-effectiveness goal to measure
progress in this area.
4 Does the performance of this N/A While VFC is similar to the 317 program, VFC 0%
program compare favorably to serves a distinct population and focuses primarily
other programs with similar on vaccine purchase. The 317 program does some
purpose and goals? vaccine purchase but also provides a lot of support
for activities that cover the entire population
including education, outreach, and surveillance.
5 Do independent and quality Small Extent While the more comprehensive IOM report 25% 0.1
evaluations of this program indicated that childhood immunization levels are at
indicate that the program is an all-time high and the program has helped
effective and achieving results? contribute to this outcome, this report focused more
on the appropriate role of the Federal government
rather than evaluating whether the 317 program, as
currently structured/managed was effective at
improving immunization rates among children.
Total Section Score 100% 42%
FY 2004 Budget
8 Fall Review