Comparative Effectiveness Research International Experiences and

Document Sample
Comparative Effectiveness Research International Experiences and Powered By Docstoc
					Comparative Effectiveness
Research: International
Experiences and Implications
for the United States
by Kalipso Chalkidou, M.D., Ph.D., and Gerard Anderson, Ph.D.

                                                     July 2009
    Introduction                                  innovative CE programs. We discuss
                                                  programs operating in:
                                                                                                       – How is it governed?

                                                                                                       – How is it funded?
    The idea of developing a comparative          •   Australia                                        – What is its size and cost?
    effectiveness (CE) program in the United
    States is not new. More than 15 years
                                                  •   Brazil                                           – How are its decisions appealed?
    ago, the Office of Technology Assessment      •   Canada
                                                                                                   •   Methods to inform decision making
    (OTA) published a report discussing the       •   Germany
    rationale and obstacles associated with                                                            – Who decides what topics to address?
    the creation of a comparative effectiveness
                                                  •   England and Wales

    entity. “The justification for most medical   •   France                                           – Who conducts analyses?

    practices used in the United States today     •   Russian Federation                               – What are the primary data sources/
    rests on the experience and expertise of      •   South Korea                                        techniques?
    clinicians and patients rather than on
    objective evidence that these practices
                                                  •   Sweden                                           – How is the input of various
    can measurably improve people’s               •   Turkey                                             constituencies included in the
                                                                                                         analysis or its interpretation?
    health. Compiling objective evidence is
    considered by some…highly controversial,      Focus of Report                                      – Are costs an important
    because the evidence might be applied in      Consistent with the CBO definition of                  consideration?
    ways that would limit individuals’ choices    CE (Appendix 1) and with international
    of medical treatments.”1                      experience, this report focuses on medical       •   Dissemination and impact
                                                  technologies, including drugs and devices,           – How is information disseminated?
    To date, comparative effectiveness research   surgical interventions, and diagnostic tests.
                                                  Some CE entities focus only on drugs                 – Who enforces the recommendations,
    activity in the United States has been
                                                  (Australia) while others focus solely on               if at all?
    limited in its funding and scope, although
    that appears to be changing given its         medical devices and procedures (Ontario).            – Has the information led to changed
    prominence in the American Recovery and       The UK also conducts CE for public health              medical practices or altered costs?
    Reinvestment Act. Other countries, who        interventions and programs. The present
    have been using comparative effectiveness     analysis, however, does not consider CE’s
    to make decisions for some time, may offer
    instructive examples for policymakers in
                                                  role in public health. Although some have
                                                  proposed that CE entities should also address    Core Attributes
    the United States.                            the CE of alternative service delivery models,   of Comparative
                                                  these models are beyond the scope of this
                                                  analysis and are generally not part of the
                                                                                                   Effectiveness Entities:
    In this report, we examine how CE
    programs operate in other countries.          domain of CE programs in other countries.        Lessons for the
    We focus on four policy issues and how                                                         United States
    various countries have addressed them.        This report focuses on four main topics:         Before describing various core attributes
    We reviewed CE programs in several                                                             of the CE entities studied here, we draw
    industrialized countries (primarily the       •   Establishment and evolution of               on international comparisons to propose
    United Kingdom and Germany) but                   comparative effectiveness agencies           some possible lessons for the United States.
    also included the experiences of several
                                                      – What was the original reason
    middle-income countries. While CE                                                              Organizational Forms
                                                        for the establishment of the CE
    research must recognize country-by-                                                            CE entities were either established as
    country differences in medical practices                                                       part of a comprehensive policy strategy or
    and ability to pay, much of the clinical          – How is it organized and related to         rapidly integrated into the overall health
    information is based on international               other entities?                            system. What differentiates CE from
    peer-reviewed literature, thus making CE                                                       conventional medical research is that it is
    information applicable internationally. As    •   Governance and accountability
                                                                                                   demand-driven rather than investigator-
    a result, several middle-income countries                                                      driven. Demand is defined by the needs
    have been able to take advantage of the           – What is the CE organization’s legal        of decision makers, including public
    available information to introduce                  status?                                    and private payers, clinicians, patients,
                                                                                                   and professionals. Most CE entities were
                                                                                                   established as part of their respective

nations’ comprehensive restructuring of           Methodological rigor is critical to the         Applicability to different populations is
their health care system. International           acceptance of the CE entity. Methodological     essential. International experience shows
experience demonstrates that the more             rigor forces the debate to focus on issues      that the application of methodological
“hard-wired” a CE agency into its nation’s        rather than on politics and helps protect       tools such as individual patient data
health care system, the greater is its impact.    the CE entity from the influence of vested      pooling from trials or decision analyses
                                                  interests. Buy-in from academic groups          based on observational and RCT data
Independence from government is                   often promotes rigor.                           ensure that studies are relevant to given
important. While most of the CE entities                                                          subpopulations. In the UK, the need
examined for this analysis were the               Transparency is crucial. At the United          for subpopulation-specific information
creation of their respective government           Kingdom’s National Institute for Clinical       has triggered public investment in
and receive different levels of government        Excellence (NICE), all information that         methodological research and human
funding, they all exhibit some level of           drives decisions, including CE evidence         capacity building through formal training
independence from government.                     and committee deliberations, is available       programs in UK universities.
                                                  to the public. NICE scrutinizes all
Independent oversight boards are common.          confidential academic and commercial            Costs are an important consideration. Some
One or more independent boards of                 submissions and encourages sources to           CE programs did not initially consider
directors ensure good governance and              release the information to the public as        costs, although all do so today. Countries
accountability.                                   soon as possible. Other countries have          have found different ways to incorporate
                                                  adopted similar transparency principles.        costs. England’s approach is the most
Some CE entities set standards while others                                                       explicit, specifying a cost-effectiveness
only identify and recommend standards.            Timeliness of decisions is important.           cut-off range (threshold range). Australia
Some of the CE programs only identify             Resource allocation decisions need to be        is less clear about having an explicit
standards, leaving final decisions to a           made quickly. CE entities use data from         threshold, but open about costs being a
separate entity. More commonly, however,          a variety of sources and often must make        key consideration in decision-making.
the CE entity sets standards under the            tentative decisions until more information      In fact, Australia was the first country to
supervision of an independent board               becomes available.                              include such cost considerationss. Ontario
of directors, with oversight/monitoring                                                           is undergoing a gradual move toward
provided by another component of the              CE entities use different types of data         consistent consideration of costs. Germany
health system.                                    to reach recommendations. Analytic              is in the process of devising a completely
                                                  approaches to making recommendations            new method for economic evaluation that
Most CE entities are relatively small and         vary from primarily relying on                  differs from standard cost-effectiveness
operate with a modest budget. Most CE             randomized clinical trails (RCTs) to            analysis and allows each sickness fund
entities operate with a few hundred people        relying on claims data, meta-analyses, and      (insurer) to set its own threshold.
and a budget under $60 million. They              other types of data. In addition, CE entities
outsource most of their research and rely         sometimes make use of manufacturers’            Economic evaluations are often included
on other entities to conduct analyses and         data and unpublished data and sometimes         in comparative clinical trials even in
disseminate results.                              re-examine existing evidence as new             settings where cost-effectiveness is not a
                                                  information becomes available.                  requirement. While some stakeholders may
Methods                                                                                           resist the incorporation of cost-effectiveness
The prioritization process for selecting          There is increasing use of conditional          considerations into the decision-making
topics for review is critical. A prioritization   coverage as a means of linking prospective      process, countries have yet to tackle
process closely linked to the needs of            evidence generation to decision making. CE      resistance to cost considerations through
decision makers guides the selection of           entities engage in evidence synthesis as        the introduction of some type of “firewall”
topics for analysis. Increasingly, decision       their main type of research largely because     that isolates clinical from cost-effectiveness
makers are interested in evaluating               of resource constraints and the timeliness      issues. Instead, CE entities have gradually
established practices with a view to              of current evidence. Increasingly, however,     assumed responsibility for economic
eliminating wasteful activities.                  CE is being linked to prospective trials.       analysis as part of a longer-term maturation
                                                                                                  process. Despite the many perspectives for
                                                                                                  measuring costs (societal, patient, insurer,
                                                                                                  and government), most CE entities focus
                                                                                                  on the payer perspective.

    Dissemination and Impact                        pressures of dissatisfied stakeholders,       During this period, the government
    Inclusiveness of all stakeholders, including    including manufacturers, patients, or         introduced a series of programs that
    manufacturers, is critical. Gaining             professionals. In isolated instances, the     modified operation of the NHS. As
    stakeholder buy-in is essential in ensuring     government has tried to interfere or          described in a white paper announcing
    that CE-based decisions have the                influence a CE entity’s decisions, but such   the restructuring of the NHS, the National
    desired impact. Inclusiveness depends           cases have generally occurred only once.2     Institute for Clinical Excellence (NICE)
    on engaging all stakeholders not only in                                                      was part of a comprehensive strategy
    the decision-making process but also in         Most CE entities rely on other entities to    aimed at better informing practice and
    methodology design.                             enforce and disseminate their findings.       policy decisions.3,4 According to the white
                                                    Most CE entities cannot force stakeholders    paper, “The Government is determined
    Most CE entities are not charged with           to implement their recommendations            that the services and treatment that
    saving costs. Most CE entities were not         and instead rely on other entities to do      patients receive across the NHS should be
    initially charged with cost containment         so. Dissemination and implementation          based on the best evidence of what does
    as their primary goal. In fact, most CE         support are increasingly important            and does not work and what provides
    programs have been responsible for              components of CE, and CE entities are         best value for money. . . . At present
    health care cost increases in that they         gradually assuming responsibility for such    there are unjustifiable variations in the
    approve new technologies that are more          components.                                   application of evidence on clinical and
    expensive than the technologies being                                                         cost-effectiveness…The Government will
    replace. Because CE programs are part                                                         spread best practice….by establishing
    of a larger heatlh system, it is difficult to                                                 a new National Institute of Clinical
    assess what cost savings have been realized     Why Have Countries                            Excellence which will promote clinical and
    or to estimate what overall spending            Created Comparative                           cost-effectiveness by producing clinical
                                                                                                  guidelines and audits, for dissemination
    might be if CE entities were not in place.
    Furthermore, most agencies have been            Effectiveness Entities?                       throughout the NHS.”5
    operating for only a short period, making       In most cases, the creation of a CE entity
     it diffult to assess their impact on longer-   represented a response to concerns about      In the same white paper establishing
     term spending and health outcomes.             obtaining value, limiting unwarranted         NICE, the government reinforced
     Nonetheless, general speculation holds         variation in medical practice, and the cost   its support of the NHS research and
     that CE entities are responsible for           and diffusion of new technologies. Cost       development program to undertake
     allowing payers, consumers, and industry       containment was not part of the original      research assessing the clinical and cost-
     to develop a better understanding              impetus for CE entities. In most countries,   effectiveness of health technologies. The
     of health outcomes, increase the               the CE program was generally part of a        white paper also announced the launch of
     transparency of health care investment,        larger reform effort.                         a new “horizon scanning” function “for
     and reduce regional variation.                                                               emerging clinical innovations. . .to help set
                                                    England (UK)                                  research priorities, to provide information
    An appeal process is essential. Many            The British National Health Service           for planning services, and to identify the
    countries have established an appeals           (NHS) was established in 1948 as a            need for clinical and service guidelines
    mechanism for decisions made by the CE          publicly financed, single-payer system        which the new National Institute may be
    entity; the format varies from country to       providing free universal access to health     commissioned to develop.” These new
    country.                                        care. Following a period of perceived         government initiatives were expected to
                                                    chronic under funding of health               work collaboratively to rationalize health
                                                    care relative to other industrialized         allocation decisions.
    Strong political endorsement can be
                                                    countries, the NHS benefited from
    critical. The CE entity often makes
                                                    the Labour government’s (under Tony           Germany
    decisions that are unpopular with some
                                                    Blair) commitment to increased public         The German health care system dates
    constituencies. In all the international
                                                    funding as of 2000. Between 2002 and          to the end of 19th century amid the
    examples studied here, the country’s
                                                    2008, the NIS experienced a planned 50        Bismarckian reforms that created
    central or provincial government grants
                                                    percent increase in real spending that was    social health insurance. The system is
    full support to the CE entity in nearly all
                                                    designed to bring health spending levels      decentralized, with more than 90 percent
    cases. The political and legislative support
                                                    closer to the European Union average.         of the population covered through
    received by CE entities means that they
    can withstand the political and lobbying                                                      mandatory social health insurance offered

by several sickness funds. High-income          Australia                                         Given that Australia operates a positive listing
individuals may opt out of the system           Australia’s Pharmaceutical Benefits Scheme        system, PBAC must consider all new drugs,
and purchase private health insurance.          (PBS) has its antecedents in a limited            and its approval is a prerequisite for listing a
The presence of several payers may have         program established after World War I to          drug on the national formulary. It does not
influenced the creation, organization, and      provide pharmaceuticals free to returning         make pricing recommendations but instead
scope of Germany’s CE entity.                   veterans. In the following decades, the           considers the comparative clinical and cost-
                                                idea of a system to provide medications           effectiveness of technologies based on the
In the early 2000s, Germany set forth several   to all Australians gained increasing              unit price proposed by the manufacturer and
objectives in reforming its health care         acceptance and was adopted during                 then makes recommendations to the pricing
system: consolidating insurance funds and       World War II. In 1953, the government             authority. The authority in turn advises the
broadening their financial base; reducing       granted the Pharmaceutical Benefits               government as to the most appropriate price.
fragmentation in financing streams and          Advisory Commission (PBAC) authority
increasing the proportion of tax funding;       to recommend drugs for addition to the            France
imposing a single premium (as a proportion      formulary.7 Initially, PBAC based its             The French High Health Authority (HAS)
of income); and offering citizens the right     coverage recommendations for drugs solely         was established in 2005 as an independent
to choose their insurance fund and, through     on perceptions of clinical need. “It is not the   public body charged with a wide range
choice, introducing regulated competition       intention to limit the inclusion of any drug      of responsibilities, including provider
between funds.6 The 2004 creation of            or compound on the score of cost. The only        accreditation, guideline development,
the German Institute for Quality and            question asked will be: Does it contribute        definition of the basic package for
Efficiency in Healthcare (IQWiG) was part       anything towards the medical efficiency of        chronic disease patients, promotion of
of the overall reform strategy. Initiating      the formulary?”8                                  information technology tools in health
operations in 2005, IQWiG introduced                                                              care, and informing the French insurance
cost-effectiveness and clinical effectiveness   Operating with only limited transparency,         system’s decisions about listing and
as conditions for evaluating coverage and       PBAC initially consisted of senior public         reimbursement for medical technologies
reimbursement of health technologies.           servants directly appointed by the Minister       and services. Although not explicitly
                                                for Health. Gradually, however, as cost           stated in its responsibilities, one of HAS’s
At the outset, IQWiG’s main role was to         pressures and patient expectations grew,          political objectives was to help rationalize
identify national quality standards by          PBAC changed its administrative style. First,     spending on health care through quality
collating and presenting to decision-making     operation of the committee became more            improvement10 and, since January 2008,
bodies an assessment of the evidence of         transparent, and membership expanded              through adoption of cost-effective
the comparative clinical effectiveness of       to draw from the clinical and academic            technologies.
various medical interventions. German           communities rather than only from the
policymakers saw a pressing need for            government civil service. In the 1970s, the       Middle-Income Country Examples
an independent “standard-identifying”           committee started making the reasons for its      Several middle-income countries are
(rather than “standard-setting”) body to        decisions known to the medical community          creating CE entities. As a starting point, they
inform negotiations between insurers and        and then to the broader public. Second,           draw from the guidelines created by NICE,
professionals. IQWiG therefore transmits        as cost became a concern in the 1960s and         IQWiG, and other CE programs and then
its information to another decision-making      1970s, PBAC introduced a means-tested             adapt them to their own setting, subject
body that comprises sickness funds and          patient co-payment. In the late 1980s, PBAC       to their ability to finance interventions
providers. This approach stands in contrast     became the world’s first reimbursement            and management practices. They illustrate
to the UK’s approach, whereby some types        committee to consider drug costs directly         additional reasons for establishing CE
of NICE guidance are mandatory (discussed       in its decision making. As of 1993, PBAC          entities.
later in more detail). In the 2007 reforms,     considers comparative value in its decision
IQWiG’s responsibilities were expanded to       making in accordance with the principle           Turkey is undertaking a series of significant
incorporate cost-effectiveness data into its    of “purchasing outcomes”; that is, a higher       reforms of its health care system. In
assessment portfolio, although the means of     price is acceptable only when a drug offers       particular, it is establishing a national entity
such incorporation is still under discussion.   additional clinical benefit (either greater       responsible for reviewing the evidence of
IQWiG is not expected to issue cost-            efficacy or reduced toxicity or both)             what works in health care in order to set
effectiveness advice before 2009.               compared to current alternatives.9                quality standards, provide professional
                                                                                                  guidance, and inform coverage decisions as
                                                                                                  part of its guaranteed package of services

    (including drugs). Turkey’s political
    commitment to improving quality while
                                                    Governance and                                    Size and budget. When NICE was first
                                                                                                      established in 1999, it employed six people,
    guaranteeing a package of services seems        Accountability                                    including the directors of its programs,
    to be driving the government’s interest         Arrangements                                      and operated with a budget of $15 million.
    in using CE evidence to guide day-to-day        CE entities’ governance and accountability        Over the years, the organization has grown
    coverage decisions.                             arrangements vary with respect to                 and, in 2007, employed a staff of 270 with
                                                    legal status and statutory functions;             an annual budget of $70 million. The
    In many countries, an important impetus         funding mechanisms; the size and cost             Department of Health channels additional
    for CE is guidance in the design of benefit     of operations; collaboration with other           funding (approximately $8 million)
    packages. Increasingly, countries rely on CE    entities in gathering and disseminating           directly to the academic networks that
    to ensure that the services promised in the     information; and appeal processes.                undertake CE research to inform NICE
    benefit package are accessible. Many Latin                                                        guidance. NICE has always been a “virtual”
    American countries, for example, face the       England (UK)                                      organization and, despite its growth, still
    challenge of a particularly generous state-     Legal status, funding, and statutory              relies largely on a network of external
    promised benefit package in a climate of        functions. The UK’s NICE is a Special             experts, including members of the public,
    limited financial resources such that few       Health Authority that operates under              health care professionals, academics, and
    drugs are available when a patient arrives at   a 12-member Board of Directors that               commercial entities, to identify, develop,
    an outpatient clinic or hospital. Countries     consists of eight non-executive directors         and disseminate its recommendations.
    adopt CE to ensure that listed drugs are in     and a chair, all appointed by the                 More than 2,000 people work with NICE
    fact available.                                 independent Appointments Commission,              over any one year to help develop guidance.
                                                    and four executive directors, all appointed       Most volunteer their time and expertise.
    Brazil, for example, uses the NICE              by the non-executive directors. The
    guidelines as the first step in a process       board determines policy and strategy              Status of NICE guidance. The status of
    to ensure that services in the benefit          and holds the executives to account for           NICE guidance varies across the member
    package will be available. Brazil’s Ministry    the achievement of objectives specified           nations of the UK, although the present
    of Science reviews the NICE guidelines          in the strategic and business plans.              analysis focuses exclusively on England,
    in the context of the country’s medical         Among several subcommittees are audit,            where NICE recommendations for the
    care system while Brazilian health              risk management, and citizens’ council            use of technologies became mandatory
    economists evaluate the cost of the NICE        subcommittees. NICE also has a Partners’          in July 2003.12 Local public purchasers of
    recommendations in the Brazilian context.       Council of major stakeholders appointed           care (Primary Care and Hospital Trusts)
    This information, in combination with the       by the secretary of state. Stakeholders           have a three-month period to identify the
    requests of physicians, hospital managers,      include representatives of patient groups,        additional funding needed to implement
    and patient advocates, is presented to the      the health professions, NHS management,           NICE-recommended technologies.
    Minister of Health for a final decision on      industry, and trade unions. The Partner’s         Given that NICE has no budget to fund
    what should be included in the final benefit    Council meets annually to review NICE’s           its recommendations, Primary Care
    package. Once a drug or device is included      annual report and provide strategic advice        Trusts must identify additional funding
    in the benefit package, the expectation is      to the institute.                                 in order to comply with the institute’s
    that it will be available to all patients.11                                                      recommendations. Increasingly, where
                                                    NICE is funded by the Department of               NICE guidance applies, additional funding
    In the case of the Russian Federation,          Health and issues its guidance directly           is made available for specific Health
    different parts of Russia (oblasts) have        to the NHS Health Ministers responsible           Resource Groups (the NHS equivalent
    used CE information to make resource            for its work program (topic selection),           of DRGs) in the form of “tariff uplifts.”
    allocation decisions and improve technical      although the institute is administratively        However, some argue that mandated
    efficiency. The current system of resource      independent of the Department of                  NICE recommendations distort local
    allocation is highly centralized and heavily    Health. Government departments and                priorities and crowd out non-drug, service
    hospital-oriented. CE offers a means to         other stakeholders may participate in the         delivery–type interventions for which such
    improve allocative efficiency by reducing       guidance development process through              mandates usually do not apply. In response,
    hospitalization, expanding the outpatient       designated consultation periods. However,         NICE provides Primary Care Trusts
    drug benefit, and setting professional          reliance on government funding has led            with planning tools to prepare them for
    standards for best practice.                    some to question NICE’s independence              additional costs, and it increasingly focuses
                                                    from political pressures within NHS,              on identifying inefficient practice. In June
                                                    especially in the early years of its operation.   2008, an “NHS Constitution” that describes
                                                                                                      the basic standards of care to be expected

from the NHS was proposed. Access             government funding of comparative               network that encompasses England’s
to NICE-approved treatments from its          effectiveness research through public           Royal Colleges (equivalent to professional
technology appraisal program will become      agencies, namely, the National Institute for    societies such as the American College
a right if this legislation is enacted.13     Health Research (NIHR).                         of Radiologists). NICE maintains the
                                                                                              network by contracting with the colleges
To promote high-quality care by ensuring      The data collection program was                 and their affiliated teams of systematic
NHS compliance with national standards,       established14 with the dual objectives to       reviewers and, more recently, with health
including NICE guidance, the government       “commission research focused on the             economists to produce NICE guidelines,
established a monitoring body, known as       needs of patients and the public” and           the full version of which is owned by
the Healthcare Commission (HCC), at           develop “evidence to inform and underpin        the respective college. Each college
the same time that it established NICE.       health and social care policy.” NIHR            maintains the right to distribute the full
Currently, HCC reviews the performance        includes a health technology assessment         guideline that carries that college’s logo.
of NHS providers against several core         program and a horizon-scanning service,         NICE may also use the material in the
and developmental standards. The              both commissioned by the Department             full guideline for its own purposes such
former are mandatory and include NICE         of Health, to assist NICE with topic            as the development of implementation
guidance on the safety and efficacy of        prioritization and evidence syntheses.          support or audit tools. The intellectual
new interventional procedures and on                                                          property arrangement between NICE
NICE recommendations for specific             Further, NIHR has provided financial            and the Royal Colleges has expedited
health technologies. For the latter, NHS      support to the Cochrane Collaboration,          the establishment of the NICE clinical
providers must be able to demonstrate         supports research into methods of critical      guidelines program—predominantly
implementation plans that include NICE        appraisal and economic evaluation, and          through the analytic work of the colleges,
clinical guidelines. In 2010, the HCC         builds professional capacity through            which agreed to buy into the NICE
will be replaced by the Care Quality          fellowship training programs across             standardization and quality assurance
Commission, which will introduce a            major UK universities. A recent reform          processes—and has contributed to
new system of registration standards          of the public funding arrangements for          increased acceptance and implementation
for providers, including compliance           supporting health-related methodological        of NICE recommendations among clinical
with NICE technology appraisals and           research has transferred responsibility         professionals who see the final output
interventional procedure reviews.             for funding research on methodological          as a product of their own professional
                                              issues to the Medical Research Council          association rather than an externally
Overall, instead of relying on rigid          (MRC). The newly formed (2007)                  imposed direction with little clinical input.
rules and regulations that force              Methodology Research Program, with
clinicians and public payers to adapt         a total budget of $120 million, aims            Appeal process. Stakeholders may appeal
its recommendations, NICE relies on           to “support the development of new              NICE recommendations for the use of
frontline NHS staff and patients for          and improved systems and theories for           technologies on the following grounds:
implementation of its guidance. It strives    health research” and is charged with            (1) perversity in that no reasonable
to gain stakeholder buy-in through            providing support to decision makers            group of people would have formulated
the inclusive nature of its guidance          such as NICE by addressing important            recommendations as presented by NICE,
development processes (see below for          methodological challenges.15 NICE now           (2) violation of NICE procedural rules,
the Royal Colleges’ participation in the      works closely with MRC to develop the           and (3) violation of NICE scope of
process).                                     research briefs that call for research into     responsibilities. Appeals are heard by a
                                              specific areas of interest to the institute,    panel consisting of at least two NICE
Data collection. NICE is a user of CE         such as calibration of its cost-effectiveness   non-executive directors and three other
research to inform coverage decisions and     threshold by using empirical evidence and       members with patient advocacy, industry,
to set quality standards across the NHS; it   the development of appropriate criteria for     and NHS experience. Approximately one
does not directly engage in comparative       making conditional coverage decisions for       in three appraisals is appealed, with almost
effectiveness research. It is heavily         new emerging technologies with a weak           half of appeals upheld. If unsuccessful at
dependent on the UK’s pre-existing            evidence base.                                  the appeal stage, stakeholders may seek
infrastructure to produce needed evidence,                                                    a judicial review of the guidance as has
mostly in the form of systematic reviews      In addition to NHS research and                 occurred only once thus far (Alzheimer’s
and evidence syntheses, in order to make      development, the NICE clinical guidelines       drugs); a decision is pending.
coverage and policy decisions. NICE’s         program is built on a pre-existing
data needs have resulted in increased         professional guidelines’ development

    Germany                                                                                        the relevant evidence by using IQWiG-
    Legal status, funding, and statutory
                                                   •   Assessment of the benefit and cost of
                                                                                                   designed “evidence table templates.”
                                                       drugs; and
    functions. To ensure stakeholder buy-in,                                                       IQWiG undertakes (or commissions)
    IQWiG has assumed a strictly advisory                                                          little primary research in the form of
    role with respect to the Joint Federal         •   Provision to citizens of easily
                                                                                                   prospective clinical trials and registries,
                                                       understandable general information
    Committee (JFC), which has final                                                               although JFC may commission prospective
                                                       on the quality and efficiency of health
    decision-making authority. JFC consists of                                                     studies in instances of important evidence
                                                       care services and on the diagnosis
    providers (including hospitals and health                                                      gaps. Similarly, IQWiG does not perform
                                                       and therapy of diseases of substantial
    care professionals) and insurance funds,                                                       claims data analyses, as is the case in other
                                                       epidemiological relevance.
    with both groups equally represented.                                                          countries. Furthermore, given its primary
    Patient representatives with no right to                                                       reliance on meta-analyses of RCTs, IQWiG
                                                   In 2007, the Ministry of Health added to
    vote also sit on the JFC as does a “neutral                                                    operates with a relatively underdeveloped
                                                   the institute’s charge responsibility for
    vote,” which determines the final decision                                                     network of external CE support structures
                                                   making recommendations to insurance
    in the case of a disagreement between                                                          compared to other similar entities abroad.
                                                   funds for the ceiling (maximum) price of
    professionals/providers and insurers.16
                                                   new (or existing) listed technologies, based
                                                                                                   Appeals process. Relevant stakeholders
                                                   on evidence of comparative value. IQWIG
    IQWiG is an independent nonprofit non-                                                         may appeal IQWIG decisions.
                                                   has yet to issue any such recommendations
    governmental entity with a 30-member
                                                   and instead is developing its methodology
    Board of Trustees, a Scientific Advisory                                                       Some Unique Aspects of CE
                                                   for undertaking the required cost-
    Board, and a Steering Committee that                                                           Govenence in Other Countries
                                                   effectiveness analyses.
    includes the institute’s management and                                                        Canadian Common Drugs Review (CDR)
    department heads and a Methods Group                                                           and Ontario Health Technology Advisory
                                                   Size and budget. IQWiG operates with
    that is responsible for the methodology                                                        Committee (OHTAC)
                                                   a staff of approximately 90 and a budget
    applied by IQWiG in its assessments and                                                        In Canada, the provinces are responsible
                                                   of about $30 million (2008). It is funded
    for institute processes.                                                                       for managing, organizing, and delivering
                                                   to “50% by a levy on every hospital case
                                                                                                   health care and for funding more than
                                                   to be invoiced and to 50% by a tax on
    The Federal Ministry of Health approved                                                        80 percent of health care costs (public
                                                   ambulatory and outpatient services
    IQWiG’s establishment in July 2004,17 with                                                     and private); the federal government
                                                   reimbursed by statutory health insurance.”
    the following responsibilities:                                                                funds the balance. Through the Common
                                                                                                   Drugs Review (CDR), which forms part
                                                   Status of IQWIG guidance. JFC is not
    •   Research, presentation, and evaluation
                                                   required to accept IQWiG’s advice but
                                                                                                   of the Canadian Agency for Drugs and
        of the current medical evidence base for                                                   Technologies (CADTH), the federal
                                                   must explain its rationale if it decides
        diagnostic and therapeutic procedures                                                      government assesses drugs (with the
                                                   not to do so. In a sense, JFC performs the
        for selected diseases;                                                                     exception of Quebec). In light of a CDR
                                                   “appraisal” part of the decision-making
                                                                                                   recommendation, each drug plan in
                                                   process, partly based on evidence collated
    •   Production of scientific reports, expert   and interpreted by IQWiG. IQWiG fits
                                                                                                   Canada makes its own decision. Overall
        opinions, and statements on the quality                                                    compliance with CDR guidance exceeds
                                                   into a highly decentralized environment,
        and efficiency of services delivered                                                       90 percent; however, CDR decisions
                                                   with strong professional, union, and
        within the framework of the statutory                                                      apply only to the Canadian provinces’
                                                   industry interest groups operating within a
        health insurance system, taking into                                                       public programs, thus accounting for
                                                   relatively opaque negotiation framework.
        consideration factors concerning age,                                                      approximately half of overall spending on
        gender, and life circumstances;                                                            pharmaceuticals. To make its decisions,
                                                   Data collection. IQWiG carries out CE           CDR relies on evidence submitted by the
                                                   in the form of evidence syntheses rather        drug and device companies.
    •   Assessment of evidence-based               than through prospective studies, both
        guidelines for the epidemiologically       in-house and through external groups
        most important illnesses;                                                                  Responsibility for generating CE to
                                                   (“collectives”) in Germany and other
                                                                                                   inform coverage decisions for non-
                                                   EU countries. External commissioning
                                                                                                   drug technologies tends to rest with
    •   Submission of recommendations on           takes place mostly in disease areas in
                                                                                                   the provinces (e.g., Institute for
        disease management programs;               which IQWiG lacks relevant expertise.
                                                                                                   Health Economics in Alberta; Agence
                                                   In such cases, IQWiG commissions
                                                                                                   d’évaluation des technologies et des modes
                                                   external experts, mostly clinical specialists
                                                                                                   d’intervention en santé [AETMIS] in
                                                   with critical appraisal skills, to collate
                                                                                                   Quebec; and Medical Advisory Secretariat/

Ontario Health Technology Advisory               to predetermined criteria19—high-priority         The UK’s Department of Health funds
Committee [MAS/OHTAC]). Ontario                  topics and the most appropriate process           the National Institute for Health Research
has developed a unique scheme whereby            (clinical guidelines, public health guidance,     (NIHR), which undertakes three core
CE evidence, ranging from independent            or technology appraisal) for developing           functions that feed into NICE processes:
evidence syntheses to prospective trials         guidance on these topics. The Minister of         (1) horizon scanning to inform the topic
and registries, is directly linked to coverage   Health retains final responsibility for topics    selection process; (2) evidence synthesis,
decisions. OHTAC, established in 2003            referred to NICE.20                               including a systematic review of the
following an initiative led by Ontario                                                             evidence and decision analysis modeling
hospital CEOs,18 uses dedicated networks         NICE relies on several sources to prioritize      or a critique of manufacturers’ models, to
of researchers, mostly funded by the             the selection of technologies and medical         inform the development of guidance on
provincial government, to evaluate health        conditions, including a dedicated horizon-        the use of specific technologies; and (3)
technologies upon request from potential         scanning service funded by the Department         starting in 2007, prospective real-world
technology purchasers, including the             of Health and suggestions by the public           trials to address specific uncertainties
Ministry of Ontario or hospitals located in      through an open Web-based process.                identified during the guidance
the province.                                    NICE places the recommended topics                development process to inform future
                                                 before panels led by the National Clinical        updates of the guidance “direct access.”21
In the case of insufficient clinical and cost    Directors (government advisors) in the
data to make fully informed decisions,           respective clinical areas (e.g., cardiovascular   The Medical Research Council receives
Ontario has implemented a highly                 disease, public health, mental health,            public funding to support research into
innovative and relatively unique model           cancer). The panels draw their membership         methodologies for developing NICE
for generating and assessing evidence.           from professional associations, industry,         guidance, including modeling tools for
Through what it calls conditionally              academia, and the general public.                 making conditional coverage decisions
funded field evaluations (CoFFE), OHTAC                                                            and ways for incorporating equity
funds prospective trials that compare the        Calls by various stakeholders (including          considerations into the decision-making
clinical and cost-effectiveness of non-drug      patient groups and industry) to allow             algorithm. At the same time, NICE
technologies.                                    more time to consider evidence, engage            contracts with the Royal Colleges to
                                                 in the process, and appeal unfavorable            synthesize and appraise the evidence.
Australian Pharmaceutical Benefits               decisions have led to a protracted selection
Advisory Committee (PBAC)                        process. Responding to concerns over              The various sources of evidence reviewed
CE entities in Australia are part of federal     the lengthening process, NICE has                 during the assessment phase are
or provincial governments rather than            abbreviated some consultation rounds and          synthesized and used to develop a decision
independent public agencies. Given               accelerated the evidence analysis stage.          analysis model. NICE has developed
that it operates a positive listing system,      While transparency and inclusiveness are          explicit guidance as to the type, format,
PBAC must consider all drugs licensed in         important characteristics of the NICE topic       and sources of evidence that decision-
Australia. Drugs rejected by PBAC are not        selection process, the institute must balance     making committees consider during the
listed on the formulary.                         its consultative responsibilities against the     appraisal process.22 The evidence includes
                                                 pressure to issue timely guidance.                meta-analyses and systematic reviews of
                                                                                                   RCTs, head-to-head RCT comparisons
                                                 Assessment and appraisal. NICE’s                  of the technologies under consideration,
Methods to Inform                                charge is to help the NHS make evidence-          and different types of quasi-experimental
Decision Making                                  informed decisions regarding the most
                                                 clinically effective and cost-effective
                                                                                                   studies, such as prospective cohort,
                                                                                                   registries, and epidemiological analyses.
CE entities must select technologies for
review, appraise the technologies, and           application of specific technologies and
nearly always consider costs before making       disease management strategies. In this            Unpublished academic or proprietary
recommendations.                                 context, the synthesis and critical appraisal     commercial information is also eligible for
                                                 of available evidence (assessment phase)          consideration, although NICE encourages
                                                 is one input in a broader decision-               stakeholders to keep such submissions to
England (UK)
                                                 making process that generates final               a minimum, requires full justification of
Topic selection. NICE plays an increasingly
                                                 recommendations (appraisal phase).                the confidential nature of such evidence,
central role in selecting the technologies
                                                 NICE carries out its synthesis and critical       and expects the evidence to be made public
and medical conditions for which it issues
                                                 appraisal functions in conjunction with           following the licensing of a technology or
guidance. As of 2006, NICE is responsible
                                                 other public entities.                            publication of an academic analysis. Finally,
for identifying and selecting—according
                                                                                                   NICE considers patient surveys and patient
                                                                                                   and professional expert opinion.
     The appraisal phase--rather than the             CE research, the council report concluded        generation.25 Risk-sharing schemes or
     assessment phase--typically captures             that “patients would be reassured to know        conditional reimbursement decisions
     professional opinion while the NICE              that clinicians and the healthcare system in     (similar to CMS’s Coverage with Evidence
     Citizens’ Council—a citizens’ jury               general could face up to uncertainty, and        Development) are particularly relevant in
     consisting of 30 laypersons—elicits societal     were confident enough to deal with it in a       circumstances of increased uncertainty
     views on issues such as the importance of        mature, scientific way, and avoid wasting        as in the case of new drugs at the time
     equity or age in making resource allocation      money on unproven technologies.”                 of receiving marketing authorization or
     decisions. The views of this citizens’ jury                                                       diagnostic tests and surgical procedures
     are captured and published in the NICE           Drawing on the report, NICE asked its            usually accompanied by limited evidence of
     guideline on Social Value Judgments.23           advisory bodies (Social Value Judgments)24       impact on health outcomes.26
                                                      to consider recommending the use of an
     Often, CE entities are asked to make             emerging technology only in the context          Consideration of costs. “Even if NHS
     recommendations and issue guidance based         of a well-designed research project when         funding is significantly increased that single
     on minimal publicly available information.       limited evidence is available to support the     truth will remain…resources do not stretch
     In January 2007, the Citizens’ Council           technology’s broad adoption across the           to satisfying the demands placed on them by
     was asked to consider the role of decision       NHS.                                             everyone. No healthcare system in the world
     makers such as NICE in encouraging the                                                            begins to meet, and match, the aspirations of
     generation of CE evidence and reducing           NICE’s overall approach to evidence              all those who work in it or use it.” 27
     uncertainty around the effects of treatment.     evaluation does not always follow
     After three days of deliberations that           conventional evidential hierarchies.             NICE considers costs from the NHS
     involved researchers from UK universities        Instead, it is driven largely by the type of     perspective (and, in exceptional
     and the Cochrane Collaboration, ethics           policy and clinical practice questions that      circumstances, in terms of broader indirect
     committee members and ethicists, patients        need to be answered and focuses on the           societal costs) by requesting the calculation
     who had participated in clinical trials,         quality (rather than the type) of the studies    of an incremental cost-effectiveness ratio
     researchers, industry representatives, and       that address such questions. Further, given      (ICER) that is measured as the cost per
     public payers, the council unanimously           increasing pressures for making timely           quality adjusted life year (QALY). Before
     endorsed the idea of conditional coverage        decisions, evidential synthesis rather           reaching a decision, the NICE decision-
     subject to further evidence. In its report,      than primary research often provides the         making committees consider the ratio
     the council identified several criteria that     impetus for the NICE approach, thereby           in addition to other non-quantifiable
     should guide decision makers in making           allowing for:                                    factors, such as equity implications or the
     recommendations for additional research:                                                          availability of alternative treatments. The
                                                      •   Consideration of several sources of          threshold range applied by NICE to assess
     •   The extent to which the research is likely       evidence;                                    the appropriateness of a technology’s
         to reduce current uncertainty;                                                                ICER is between $30,000 and $45,000
                                                      •   Extrapolation beyond the usually short       per QALY, although advisory bodies may
     •   The value-for-money of the research;             time horizons of most RCTs;                  consider and have accepted ICERs out of
                                                                                                       range. In response to the negative reaction
     •   The budgetary implications of making a       •   Incorporation of epidemiological data        triggered by a provisional NICE decision
         positive recommendation based on the             specific to the UK population such as        to restrict access to four renal cancer drugs
         evidence;                                        baseline risk or usual treatment patterns;   that extend life by an average few months
                                                                                                       at more than $40,000 per patient per year,
                                                                                                       NICE has proposed to revise its threshold
     •   The existence of an ongoing study or the     •   Consideration of alternative technologies
                                                                                                       for end-of-life treatments. If approved,
         feasibility of initiating a study within a       and their associated costs; and
         realistic time frame; and                                                                     the proposal will make it easier for
                                                                                                       treatments targeting terminally ill patients
                                                      •   Quantification of uncertainty and the
                                                                                                       to make it past NICE’s cost-effectiveness
     •   Issues of patient access to the study            implications of making the wrong
                                                                                                       threshold.28 The current threshold was
         across geographic areas.                         decision.
                                                                                                       initially based on anecdotal evidence from
                                                                                                       technology adoption decisions made by
     In discussing the need for explicit criteria     Increasingly, NICE is experimenting with
                                                                                                       local purchasers. More recently, program
     when making conditional coverage                 decision options that link policy and
                                                                                                       budgeting and marginal analysis data
     decisions for more CE and considering            practice recommendations to evidence
                                                                                                       released by the Department of Health made
     the difficulties of implementing needed

it possible to estimate the substitution       studies and, where available, patient-level     Consideration of costs. As of 2007,
rate (or NHS productivity) at the local        data. They rarely adapt evidence syntheses      IQWiG’s scope of responsibility extends
level. According to the analyses, the NICE     undertaken in other countries to the            to costs. For many years, Germany used a
threshold range is broadly compatible          German context and do not consider any          national system for reference pricing for
with NHS’s return on investment in terms       confidential commercial information; a          pharmaceuticals with generic equivalents.
of reduction in mortality and (less easily     requirement mandates that all unpublished       However, increasing pressures from
measured) morbidity in the major disease       data considered by IQWiG must be made           expensive new drugs and the burgeoning
areas such as cancer and cardiovascular        public. The limits on acceptable evidence       costs of long-term care led to the
disease.29                                     often trigger complaints by industry and        introduction of a more central role for
                                               physician-proponents that the evidence          IQWiG in assessing cost-effectiveness.
NICE does not directly consider                base is inadequate.                             Starting in 2009, IQWiG will be responsible
affordability but does provide budgetary                                                       for advising insurance funds of a ceiling
impact analyses for all recommendations        Industry has also complained about its lack     price for new and existing pharmaceuticals
in order to support implementation. It         of involvement in evidence assessment. In       based on evidence of comparative cost-
also directs its committees to exercise        response, IQWiG now permits industry            effectiveness. When the ceiling price is
greater caution and expect higher levels       comment at the various stages of the            below the manufacturer’s list price, the
of certainty around the expected ICER          assessment, including the early priority-       manufacturer will have to lower its price; if
when the budgetary implications of a           setting and preliminary report stages.          not, insurance funds will shift the extra cost
recommendation are significant.                                                                to patients through co-payments.
                                               With regard to prospective trials, the law
Germany                                        allows IQWiG to sponsor prospective trials      IQWiG has adopted a unique approach
Topic selection. JFC, which is responsible     to assess the comparative effectiveness         to considering costs. As detailed in the law
for prioritizing topics referred to it         of medical devices (rather than drugs),         expanding IQWiG’s responsibilities, IQWiG is
by various professional associations,          reflecting Germany’s relatively more relaxed    not charged with considering the opportunity
providers, and insurance funds, refers the     regulatory requirements for devices than        cost of coverage decisions across different
vast majority of topics to IQWiG. About        drugs. In addition, Germany has developed       technologies or disease areas but only within
10 percent of topics come directly from        a strong medical devices industry, although     those. So, for example, it does not need to
the German Ministry of Health and 2            most of the device companies operating in       compare the effectiveness of cancer drugs to
to 3 percent from IQWiG. The pattern           Germany have limited financial means to         the effectiveness of drugs for hypertension
of referrals is consistent with IQWiG’s        sponsor RCTs. Similarly, IQWiG has not          but instead the cost effectiveness of an
broader model of operation, which serves       yet sponsored clinical trials of such devices   antihypertensive drug against other drugs
as a “decision support tool” for JFC, with     largely because of the lack of dedicated        to treat the same condition. Furthermore,
JFC stakeholders providing the large share     funding.                                        only technologies with proven superiority
of its financial support and representing                                                      to comparable technologies are subject
the primary users of its advice. IQWiG         JFC is the major customer for IQWiG’s           to comparative evaluation of their costs.
is primarily responsible for selecting         advice. Traditionally, JFC’s decision-making    Individual insurance funds then use the
the topics for ”information for patients”      process has been consensus-driven with          ceiling price proposed by IQWIG to define
publications.                                  relatively opaque decision rules that make      the “appropriate” level of reimbursement,
                                               it difficult to assess the impact of IQWiG’s    subject to JFC decisions.30 In fact, the intent
Assessment and appraisal. IQWiG is             advice on the final decision. As mentioned,     is to encourage individual insurance funds to
responsible for the assessment part of the     JFC is not obligated to accept IQWIG’s          negotiate prices down on a case-by-case basis,
decision-making process. As opposed to         advice but must discuss its rationale for       well below IQWiG’s ceiling price, through
NICE, which hosts the decision-making          not accepting such advice. The general          direct negotiation with manufacturers in
committees, JFC and local providers make       lack of transparency at the appraisal and       an environment of increased competition
final listing and reimbursement decisions.     negotiation stage is now being addressed        between funds.
                                               by opening JFC meetings to the public and
IQWiG operates with clear rules concerning     encouraging the publication of IQWiG’s          Excluding consideration of opportunity
the nature of evidence deemed appropriate      evidence syntheses and the rationale for        costs and avoiding comparisons across
to inform its advice to JFC. RCTs sit at the   IQWiG’s advice at the assessment stage.         diseases conveniently eliminates the need
top of the evidential hierarchy. IQWiG                                                         for an explicit threshold for quantifying
specialists seek access to individual trial                                                    benefits with application of a common
protocols and information on unpublished                                                       utility measure (such as the QALY in the
trials, especially for industry-sponsored                                                      case of England and Australia).
     Another concern is consideration of societal     commissions its networks to undertake           Canadian Institutes of Health Research.
     (indirect) costs such as productivity.           conditionally funded field evaluations          ICES’s main focus is on undertaking
     Currently, IQWiG adopts a payer                  (CoFFE).31 The networks include the             population-level studies by using claims
     perspective (similar to the NICE model).         Program for Assessment of Technology            and administrative data through linked
                                                      in Health (PATH) based at McMaster              databases of individual patient information
     Perhaps the most innovative aspect of            University and St. Joseph’s Health Centre,      (up to 12 million Ontarians) instead of
     Germany’s costing model is the adoption          the Toronto Health Economics and                relying mostly on meta-analyses of RCTs
     of an “efficiency frontier” approach for         Technology Assessment (THETA) based at          (IQWiG) or decision analytic modeling
     determining costs. This approach compares        the University of Toronto, and the Institute    (NICE). With a staff of over 100 scientists,
     efficiency of individual treatments for a        for Clinical and Evaluative Sciences            ICES’s objectives are to:
     single indication to alternatives only after     (ICES). With a budget of approximately
     establishing their superior comparative          $8-10 million per annum (which in most          •   Carry out population-based health
     clinical effectiveness. The upper price          cases does not include the cost of the new          services evaluation relevant to clinical
     limit that the health care system should         technologies32), OHTAC has thus far (2008)          practice and health policy development;
     be prepared to pay for the treatments is         completed more than 10 field evaluations
     then estimated in accordance with the            with another 25 studies underway,               •   Document province-wide patterns and
     unit cost set by the manufacturer and the        including a combination of registries and           trends in health care delivery;
     efficiency (benefit/cost ratio) of existing      RCTs for evaluating drug-eluting stents,
                                                      a cardiac PET and an implantable cardiac
     treatments for the same indication. In
                                                      defibrillator (ICD) registry, and two PET
                                                                                                      •   Develop and share evidence to inform
     a sense, the efficiency ratio of new (or                                                             decision making by policymakers,
     existing) technologies is benchmarked            RCTs for staging early lung and breast              managers, clinicians, planners, and
     against ratios for other technologies with       cancer.33                                           consumers;
     the same indication rather than against a
     set threshold reflecting societal willingness    The OHTAC model is one of the few
     to pay or affordability within a set budget.     international examples of CE research in
                                                                                                      •   Promote linkages among health services
                                                                                                          researchers and decision makers; and
     This approach avoids the controversial           the form of prospective trials, as opposed
     acknowledgment of finite resources and           to evidence synthesis and claims’ data
     the recognition of opportunity costs.            analysis, that is directly linked to decision
                                                                                                      •   Train researchers and promote a wider
                                                                                                          understanding of clinical epidemiology
     At the same time, it involves significant        making. OHTAC and the Medical Advisory
                                                                                                          and health services evaluation.
     informational requirements…More                  Secretariat (MAS) call for research through
     important, it carries the risk of perpetuating   the “field evaluation” model (similarly to
                                                                                                      In addition to conducting epidemiological
     inefficiencies for indications where current     CMS’s coverage with evidence development
                                                                                                      studies to assess health trends and identify
     practice is wasteful. However, it may help       or NICE’s “only in research” scheme).
                                                                                                      high-priority diseases in Ontario, ICES is
     contain costs by setting an upper price          What makes the Ontario model unique
                                                                                                      commissioned by Ontario’s Ministry of
     ceiling and may represent an important first     is that decision makers (not the National
                                                                                                      Health to evaluate policy initiatives, such
     step toward allowing decision makers to          Institutes of Health or NIHR in the case
                                                                                                      as the impact of interventions to reduce
     consider costs explicitly and consistently.      of the United States or UK, respectively)
                                                                                                      waiting times for elective surgery for
                                                      control the research budget, thus making
                                                                                                      high-priority procedures (e.g., cataract or
     Some Unique Aspects of Methods to                implementation of calls for research much
                                                                                                      knee replacement surgery) on procedures
     Inform Decisions in Other Countries              more feasible.
                                                                                                      not included on the priority list. ICES
     While most countries use some variation of
                                                                                                      also provides information to OHTAC. At
     topic selection, assessment and appraisal,       Ontario’s Institute for Clinical Evaluative
                                                                                                      the request of OHTAC, ICES has set up
     and consideration of costs as implemented        Sciences (ICES)
                                                                                                      an Ontario-wide Web-based registry to
     by NICE and IQWiG, some countries                ICES ( was established in
                                                                                                      monitor the use and effectiveness of ICDs.
     demonstrate unique approaches.                   1992 to “provide unique scientific insights
                                                                                                      ICD use data are collected in real time
                                                      to help policymakers, managers, planners,
                                                                                                      and may be linked to individual patient
     Ontario Health Technology Advisory               practitioners and other researchers shape
                                                                                                      characteristics and outcomes at the patient
     Committee (OHTAC)                                the future direction of Ontario’s health
                                                                                                      level through ICES’s large administrative
     When conducting a first assessment               care system.” Funded by Ontario’s Ministry
                                                                                                      databases. The results of analyses, published
     of available evidence in the case of             of Health (at approximately $5 million
                                                                                                      in peer-reviewed journals, inform province-
     an insufficient basis for making an              per annum), ICES staff may receive grants
                                                                                                      wide coverage decisions.
     adoption (or rejection) decision, OHTAC          from federal funding agencies, such as the

Health Insurance Review Agency                   a “maybe,” a “no” is always a “no.” In only        alternative to resubmission when the PBAC
(HIRA)—Republic of Korea                         one instance have political leaders ignored        declines to recommend the listing of a new
In 2007, Korea abandoned reference pricing       PBAC advice. In 2001, PBAC rejected                drug (or new indication for an already
and switched to a positive drug listing          Herceptin for breast cancer as not cost-           listed drug). The independent review is not
system that calls for price negotiation based    effective. The Minister disagreed and, to          an appeal mechanism but rather, considers
on evidence of comparative clinical and          bypass PBAC’s recommendation and list              issues in dispute and cannot repeal a PBAC
cost-effectiveness (value-based pricing). The    Herceptin, he created a new program,               recommendation. Moreover, the outcome
shift highlighted the need to link decision      the ongoing Herceptin program, which is            of a review does not mandate PBAC’s
making to evidence generation. In response,      outside PBAC’s jurisdiction. This direct           reconsideration of an application.
Korean policymakers introduced a system of       political intervention challenged PBAC’s
conditional coverage for orphan drugs and        independence and led to a strong response.         High Health Authority—France
inpatient cancer drugs based on what they                                                           HAS plays a central role in pricing new
termed weak evidence of CE. Furthermore,         PBAC’s deliberations are informed by a             medical technologies. Its specialized
to support HIRA’s decision-making role,          detailed appraisal of the applicant’s submission   committees review the absolute and
the Korean parliament enacted a law in           and the evidence contained within it, as           comparative clinical effectiveness of new
February 2008 to establish a national            prepared by four evaluation groups based           technologies and then rank them on a
medical research institute (NECA) charged        at Australian universities, and by additional      5-point scale, from no improvement to
with undertaking research to address HIRA        advice provided by specialist committees that      significant improvement. Then, a separate
needs. Preliminary discussions indicate          take into account economic considerations          entity determines the price premium
decision-maker interest in both head-to-         and utilization patterns (Economics and Drug       justified by the clinical improvement.
head trials and claims analyses as well as in    Utilization Subcommittees).34 Occasionally,
methods research, including social value         the Department prepares the needed analyses.       Since 2008, HAS is required by law to
elicitation from the Korean population and       The evaluation groups generate evidence            consider costs when issuing its advice.
adaptation of QALY or another outcome            consisting mainly of evidence synthesis,           Economic considerations are included in
measure to the Korean setting.                   “model-busting” of manufacturers’ models,          those cases where HAS considers classes
                                                 and budgetary impact analyses. PBAC cannot         of drugs or other technologies rather
Pharmaceutical Benefits Advisory                 recommend either conditional coverage of           than individual products; accordingly, the
Committee (PBAC)—Australia                       drugs or further research; it can, however,        introduction of economic evaluation is not
In Australia, drug companies nominate            propose post-listing follow-up, especially when    expected to affect pricing decisions.
the price they are seeking. Based on             the industry submission is based on surrogate
comparative clinical and cost-effectiveness      endpoints. It can also recommend, through          Pharmaceutical Benefits Board
evidence, PBAC then decides whether              the pricing authority, the establishment of a      (LFN)—Sweden
the price is acceptable. In the absence of       risk-sharing scheme, usually in the form of a      Sweden’s Pharmaceutical Benefits Board
evidence of increased clinical benefit, price    price-volume agreement between government          (LFN) was established in 2002 and is
negotiation does not occur such that the         and the manufacturer.                              responsible for developing and maintaining
analysis is a cost-minimization analysis with                                                       a national positive list for drugs whose costs
the “equieffective” dose as the primary issue.   Recent years have seen several                     are reimbursed by the health care system. In
If the drug is cost-effective at the proposed    developments in how the Australian system          making its listing decisions, LFN considers
price, then no price negotiation is needed;      operates, including a commitment to                comparative clinical and cost-effectiveness,
rather, negotiation focuses on risk-sharing      increasing public involvement in decision          clinical need (severity of disease), and
schemes or price-volume agreements,              making; an open call for submissions by            equality. In its economic evaluation, LFN
especially in the case of considerable           interested parties for all drug evaluations        adopts a societal (rather than payer)
uncertainty around the clinical effect and       to be considered by PBAC alongside the             perspective, including, where appropriate,
any risk of “indication creep.”                  evaluation commentary; the introduction            factors such as productivity costs.
                                                 of cost-recovery arrangements to support
A positive recommendation by PBAC                PBAC processes by levying service fees on          LFN decisions, in conjunction with
usually means that a drug is listed, although    submitting companies; public disclosure of         mandatory generic substitutions for
listing always occurs at the prerogative         PBAC deliberations (in the form of Public          branded drugs (otherwise the patient
of the Minister for Health and Ageing.           Summary Documents) in response to the              pays the difference out of pocket), have
However, the Minister may not reverse            recent Free Trade Agreement between                prompted many drug companies to
a negative PBAC recommendation. In               Australia and the United States; and an            lower their prices on branded drugs.
other words, whereas a “yes” by PBAC is          option for an independent review as an             Manufacturers want to maintain their drugs

     in the positive list. Generic substitution     maintaining a database of uptake studies        Perhaps NICE’s greatest contribution
     and LFN CE reviews of classes of drugs,        from across the UK (            has been an increased awareness of the
     such as proton pump inhibitors, migraine       ernie).                                         importance of evidence-informed health
     drugs, and antihypertensives, have resulted                                                    care resource allocation decisions in a
     in approximately $700 billion in savings in    The evaluation of the impact of NICE is         transparent, inclusive, and methodologically
     recent years.                                  partially based on ideology and partially       robust way. In its 2008 parliamentary
                                                    based on indirect measures of impact. For       enquiry into NICE, the multipartisan
                                                    example, there is concern that NICE slows       Health Select Committee concluded, “NICE
                                                    the diffusion of new technologies and           does a vital job in difficult circumstances.
     Dissemination and                              recognition of the occasional difficulty in     The development of more and more health
     Impact                                         linking clinical and economic evaluations
                                                    given different evaluation methods.
                                                                                                    technologies and procedures, alongside
                                                                                                    rising patient expectations and the ageing
     CE entities need to be able to communicate
                                                    Not withstanding these concerns, most           population, is going to make it even
     their findings to a wide audience. Countries
                                                    evaluations are positive.                       more difficult in the future. Healthcare
     rely on a variety of mechanisms to
                                                                                                    budgets in England, as in other countries,
     disseminate their finding and evaluate the
                                                    According to the latest assessment of           are limited. Patients cannot expect to
     impact of their recommendations.
                                                    NIHR’s health technology assessment             receive every possible treatment. Demand
                                                    program, 96 percent of academic groups          outstrips resources and priorities have to
     England (UK)
                                                    providing evidence for consideration by         be determined. In other words rationing is
     The initial establishment of NICE
                                                    NICE committees reported an impact              essential, and NICE has a key role to play.
     explicitly excluded implementation from
                                                    on policy versus 60 percent of those            Given the difficult environment, NICE
     the institute’s responsibilities. However,
                                                    undertaking other NIHR-funded research.36       requires the backing of the Government.
     unwarranted geographic variation persisted
                                                    Researchers who want their research to have     NICE must not be left to fight a lone battle
     despite national guidance. As a result, the
                                                    an impact are apparently satisfied.             to support cost- and clinical effectiveness in
     implementation of NICE recommendations
                                                                                                    the NHS.”38
     became a government priority. Among the
     measures introduced to reduce regional         It is methodologically challenging to assess
                                                    the impact of NICE guidance on practice         Germany
     variation was the requirement that
                                                    patterns and even more so on health             Implementation of IQWiG’s
     primary care trusts make funds available
                                                    outcomes. The lack of a control group and       recommendations has been difficult to
     to implement NICE decisions. The recent
                                                    the wide range of government policies           evaluate given (1) the strictly advisory
     announcement that NHS patients will
                                                    and non-health–related activities make it       nature of IQWiG’s guidance as defined
     (subject to enactment of the necessary
                                                    impossible to attribute causality to NICE.      by law and (2) the lack of explicitness
     law) be entitled to access new technologies
                                                    However, numerous case studies show the         in justifying JFC coverage decisions in
     approved by NICE in the context of the
                                                    impact of the use of CE to inform coverage      light of IQWiG’s advice. In the absence
     NHS Constitution35 is expected to increase
                                                    decisions on unwarranted variation in           of a mandate for JFC to adopt IQWiG’s
     uptake of NICE recommendations and
                                                    practice and spur the diffusion of new          recommendations, the requirement for JFC
     reduce geographic variation.
                                                    treatment modalities across the NHS.            to explain its grounds for rejection, coupled
                                                    According to a report by the National           with the recently announced opening of its
     NICE’s implementation directorate has
                                                    Director for Cancer, adoption of cancer         decision-making meetings to the public,
     developed several tools and interventions
                                                    drugs appraised by NICE increased by            will increase the pressure on JFC to rely on
     for supporting the adoption of NICE
                                                    almost 50 percent across the country            IQWiG’s evidence or produce convincing
     guidance at the local level, including audit
                                                    between 2003 and 2005, with geographic          reasons for reaching an alternative
     criteria, educational tools, a network of
                                                    variation declining from three- to eight-fold   conclusion.
     “implementation consultants” operating
     at the local level, guides to changing         to two- to three-fold over the same period.37
                                                                                                    The introduction of an IQWiG-set price
     provider behavior, budget impact tools
                                                    NICE was not intended to generate cost          ceiling for medical technologies is also
     adaptable to the local setting, and a
                                                    savings when it was established. Adoption       expected to increase IQWiG’s impact on
     “forward planner” to help commissioners
                                                    decisions have been estimated to cost $1.6      industry pricing and physician prescribing
     plan for NICE guidance in the pipeline.
                                                    to $3.2 billion per year. In 2006–2007, NICE    practices. However, the German model
     The implementation directorate is also
                                                    guidance absorbed more than a tenth of the      of delegating decision-making power to
     responsible for collating and sharing case
                                                    growth in health care spending across the       insurance funds means that IQWiG has
     studies of best practice in implementing
                                                    NHS.                                            an indirect and difficult-to-assess impact
     NICE guidance and for developing and

on decisions at the national level. When       to public payers only (employer-sponsored
JFC accepts IQWIG’s advice, the advice         plans make their own listing decisions in
                                                                                               •   NICE: Sir Michael Rawlins, Andrew
tends to be binding for insurance funds.       Canada).
However, the link between IQWiG’s advice
                                                                                               As well as to the many individuals who
and JFC decisions remains unclear.
                                               Acknowledgments                                 provided insight and feedback on the
                                               The authors are are grateful to the             manuscript, including:
Australia and Canada                           following experts who consulted on the
PBS in Australia and CDR in Canada make
listing decisions for drugs for the whole of
                                               role and functions of:                          •   Robert Crane, Kaiser Permanente

their respective populations (in the case of
CDR, for public plans only). Consideration     •   OHTAC and CDR: Irfan Dhalla, Les            •   Amanda Folsom, AcademyHealth
                                                   Levine, Danielle Whicher
and approval by PBAC is a prerequisite
for a drug’s inclusion in the Australian                                                       •   Patricia Pittman, AcademyHealth
formulary while, even though state-run         •   PBS: Ruth Lopert
public plans in Canada operate with a                                                          The views expressed herein are those of
degree of autonomy, the vast majority of       •   HAS: Lise Rochaix                           the authors not of their employers, the
plans follow CDR’s recommendations. In                                                         experts consulted or of the sponsors of
this sense, both entities have considerable    •   IQWiG: Charalbos-Markos Dintsios,           this analysis.
impact. However, their scope is limited to         Jonas Schreyoegg
pharmaceuticals and, in the case of CDR,

 This report is part of a three year initiative to expand senior health policy leaders’ knowledge of
 health systems and delivery innovations in other industrialized countries, and to promote the use
 of this knowledge to advance system improvements in the United States. AcademyHealth gratefully
 acknowledges the Kaiser Permanente Institute for
 Health Policy and the National Institute for Health Care
 Management Foundation for their generous support of this
 work. See for more information.                            FOUNDATION
     Appendix 1: Defining Comparative Effectiveness
     Table 1 depicts some alternative definitions of CE as proposed by various organizations in the United States prior to the
     release of the Federal Coordinating Council for Comparative Effectiveness Research. Despite considerable overlap in the
     definitions, the table indicates significant differences, which were indicative of some of the debate over the role of CE.

     Table 1: Definitions of Comparative Effectiveness by Various U.S. Organizations
      Organization                                   Definition

                                                     “Comparative effectiveness is the evaluation of the relative (clinical) effectiveness,
      American College of Physicians                 safety, and cost of 2 or more medical services, drugs, devices, therapies, or
                                                     procedures used to treat the same condition.”39

                                                     “. . .comparative effectiveness [is] the comparison of one diagnostic or treatment
                                                     option to one or more others. In this respect, primary comparative effectiveness
                                                     research involves the direct generation of clinical information on the relative
      Institute of Medicine
                                                     merits or outcomes of one intervention in comparison to one or more others, and
                                                     secondary comparative effectiveness research involves the synthesis of primary
                                                     studies to allow conclusions to be drawn.”40

                                                     “Comparative-effectiveness analysis compares the relative value of drugs, devices,
                                                     diagnostic and surgical procedures, diagnostic tests, and medical services. By
      Medicare Payment Advisory Committee            value, we mean the clinical effectiveness of a service compared with its alternatives.
                                                     Comparative-effectiveness information has the potential to promote care of higher
                                                     value and quality in the public and private sectors.”41

                                                     “As applied in the health care sector, an analysis of comparative effectiveness is
      Congressional Budget Office                    simply a rigorous evaluation of the impact of different options that are available for
                                                     treating a given medical condition for a particular set of patients.”42

                                                     “Comparative effectiveness reviews expand the scope of a typical systematic
                                                     review, which focuses on the effectiveness of a single intervention, by comparing
                                                     the relative benefits and harms among a range of available treatments or
      Agency for Healthcare Research and Quality     interventions for a given condition. In doing so, [they] more closely parallel the
                                                     decisions facing clinicians, patients and policymakers, who must choose among
                                                     a variety of alternatives in making diagnostic, treatment, and health care delivery

      Pharmaceutical Research and Manufacturers of   “. . .the development of high quality [sic] information about patients’ medical
      America                                        treatment options.”44

Appendix 2: Comparative Effectiveness Research Infrastructure
Even with some agreement regarding the definition of comparative effectiveness, differences in the preferred organization and
domains of the CE entity were prevalent prior the to the passage of the ARRA legislation. For example, various organization in the
United States had proposed a wide range of alternative organizational forms with a variety of public and private roles (Table 2).

Table 2: Organizational Forms for Comparative Effectiveness
 Organization                     Proposal

                                  Establish a national (public/private) entity to evaluate new and existing health care services and
                                  technologies. The new entity (which could be known as a Comparative Effectiveness Board) would be
 America’s Health Insurance       responsible for (1) comparing the clinical and cost-effectiveness of new and existing drugs, devices,
 Plans                            procedures, therapies, and other health care services; (2) assessing alternative uses of treatments
                                  currently in practice; and (3) disseminating the resultant information in a useful format, allowing patients
                                  and clinicians to make informed health care decisions.45

                                  Establish a Center for Medical Effectiveness and Health Care Decision-Making to improve decision
                                  making and incorporate information about the relative clinical benefits and cost-effectiveness of
 The Commonwealth Fund            alternative treatment options into insurance benefit design. By generating the information and creating
                                  payment and cost-sharing incentives for providers and consumers, this policy option could result in
                                  estimated health system savings of $368 billion shared over 10 years by all payers.46

                                  Congress should direct the U.S. Department of Health and Human Services to establish a program
                                  with the necessary authority, expertise, and resources to set priorities for evaluating clinical services
 Institute of Medicine            and to conduct systematic reviews of the evidence. The program would develop and promote rigorous
                                  standards for creating clinical practice guidelines, which could help minimize use of questionable
                                  services and target services to patients most likely to benefit.47

                                  Congress should charge an independent entity to sponsor credible research on comparative
                                  effectiveness of health care services and disseminate this information to patients, providers, and
 Medicare Payment Advisory
                                  public and private payers. Such an entity would be independent, transparent, and inclusive, with
                                  secure funding, and would be responsible for regular review, dissemination of its findings to relevant
                                  stakeholders and would be overseen by an independent Board.48

                                  The Centers for Medicare and Medicaid Services launched the Evidentiary Priorities Initiative in 2007 by
 Medicare Coverage Advisory
                                  convening a panel of the Medicare Evidence Development & Coverage Advisory Committee (MedCAC)
                                  to identify high-priority topics for CE.49

                                  Given that any private sector entity (such as a health plan) has only a limited incentive to produce or pay
                                  for information that could benefit many entities—including its competitors—it is reasonable to argue for
                                  a larger federal role in coordinating and funding research on comparative effectiveness. In addition, with
 Congressional Budget Office
                                  federal health insurance programs playing a major role in financing medical care and accounting for a
                                  large share of the federal budget, the federal government has an interest in generating evaluations of the
                                  effectiveness of different approaches to health care.50

                                  As just one example, the Conrad/Baucus Bill (August 2008), which is entitled the Comparative
 Congressional Bills              Effectiveness Research Act of 2008, recommends the establishment of a comparative effectiveness
                                  research fund to prioritize and undertake CE research.

                                  A 2008 position statement51 called for an “adequately funded, trusted national entity to prioritize,
 American College of Physicians
                                  sponsor, and produce comparative-effectiveness and cost-effectiveness information.”

In addition, CE has been the topic of a series of academic publications52 and policy reports53 developed by U.S. health
research and policy organizations, including AcademyHealth.54

     Endnotes                                                  18 In Canada, general hospitals are nonprofit entities
                                                                  operating on budgets negotiated with provincial
                                                                                                                        31 Levin, L., et al., “Establishing a Comprehensive
                                                                                                                           Continuum from an Evidentiary Base to Policy
     1 Office of Technology Assessment, Congress of the           Ministries of Health with limited capacity (or           Development for Health Technologies: The
        United States, “Identifying Health Technologies           incentive) to invest in new technologies due             Ontario Experience,” International Journal of
        That Work: Searching for the Evidence,”                   to financial pressures. These attributes of the          Technology Assessment in Health Care, Vol. 23, No.
        September 1994.                                           demand side may explain the establishment and            3 (2007):299-309.
     2 For example, the decision of the Australian                buy-in to the OHTAC model in Ontario.                 32 The cost of the new technologies is also usually
        Minister to ignore the negative recommendation         19 Topic selection criteria include broad policy            covered by the Ministry through a separate fund,
        by PBAC on Herceptin resulted in a series of              priorities, potential budgetary impact, potential        thus making implementation of field evaluations
        resignations of committee members while a                 to improve health outcomes, current variation            a relatively appealing option for manufacturers,
        similar decision by the UK government to ignore           in practice, availability of relevant evidence, and      especially if the alternative is no adoption by local
        NICE’s recommendation on drugs for multiple               potential of NICE guidance to add value.                 providers.
        sclerosis has been one of the most controversial       20 National Institute for Health and Clinical            33 Whicher, D., et al. Comparative Effectiveness
        decisions in the organization’s history.                  Excellence, “NICE Topic Selection Process,” http://      Research in Ontario, Canada: Producing Relevant
     3 Department of Health, “The New NHS: Modern,                     and Timely Information for Health Care Decision
        Dependable,” December 1997, http://tinyurl.               topicselectionconsiderationpanels/topic_                 Makers Milbank Quarterly, fall 2009 issue (in
        com/4asgpy.                                               selection_consideration_panels.jsp, accessed             press)
     4 For a detailed description of NICE and its functions,      October 2008.                                         34 Affordability is a key consideration for PBAC, in
        see Sorenson, C. et al., “NICE: How Does It Work       21 Chalkidou, K. et al., “Evidence-Informed                 addition to comparative cost-effectiveness; for
        and What Are the Implications for the US?” Funded         Evidence-Making,” Journal of Health Services             approved technologies with an estimated budget
        by the National Pharmaceutical Council, 2008.             Research & Policy, Vol. 13, No. 3 (2008):167-73.         impact of more than AUS$10 million, the Cabinet
     5 Ibid., DH, The New NHS, paragraph 7.6, 1997.            22 Guide to the Methods of NICE Technology                  (rather than the Minister) grants final approval.
     6 Cheng, M. and U. Reinhardt, “Shepherding Major             Appraisals, NICE, 2008; also see Claxton, K. et       35 Ibid., NHS Constitution.
        Health System Reforms: A Conversation with                al., “Probabilistic Sensitivity Analysis for NICE     36 Hanney S., et al. “An Assessment of the Impact
        German Health Minister Ulla Schmidt,” Health              Technology Assessment: Not an Optional Extra,”           of the NHS Health Technology Assessment
        Affairs, Vol. 27, No. 3 (2008):w204-w213. There           Health Economics, Vol. 14, No. 4 (2005):339-47.          Programme,” Health Technology Assessment, Vol.
        are no pay-for-performance schemes in Germany.         23 National Institute for Health and Clinical               11, No. 53 (2007).
        Sickness funds are paid according to a risk-              Excellence, “Social Value Judgments:                  37 Department of Health, “Usage of Cancer Drugs
        adjusted per capita formula, and sickness funds           Principles for the Development of NICE                   Approved by NICE,” September 20, 2006, http://
        that produce better outcomes are not rewarded.            Guidance,” 2005–2008,     
     7 Sloan, C. “A History of the Pharmaceutical                 uk/getinvolved/patientandpublicinvolvement/              Publications/PublicationsPolicyAndGuidance/
        Benefits Scheme 1947–1992,” Commonwealth                  opportunitiestogetinvolved/citizenscouncil/              DH_4139061.
        Department of Human Services and Health.                  socialvaluejudgementsguidelines/social_value_         38 The House of Commons Health Select
        Canberra, Australia, 1995.                                judgements_guidelines.jsp.                               Committee, “First Report, Session 2007–08:
     8 Speech by the Minister of Labour and National           24 See above                                                National Institute for Health and Clinical
        Service of Australia, 1944.                            25 Chalkidou, K., et al., “Making a Decision to Wait        Excellence,” January 10, 2008, http://www.
     9 Lopert, R. “Pharmacoeconomics and Drug                     for More Evidence: When NICE Recommends        
        Subsidy in Australia–An Account of the Australian         a Technology Only in the Context of Research,”        39 American College of Physicians, “Information
        Approach. In Patients Physicians and Pharma               Journal of the Royal Society of Medicine 100             on Cost-Effectiveness: An Essential Product of a
        Workshop Report, Israel National Institute for            (2007):453-60; K. Chalkidou et al., “Evidence-           National Comparative Effectiveness Program,“
        Health Policy Research, Haifa, April 2006, http://        Based Decision Making: When Should We Wait               Annals of Internal Medicine 148 (2008).           for More Information?” Health Affairs, Vol. 27,       40 Institute of Medicine Roundtable on Evidence
        88bbd73a2f1bf941f.pdf; also see National Health           No. 6, in press.                                         Based Medicine, “Learning What Works Best:
        Act 1953, S101.                                        26 Tunis, S. and S. Pearson, “Coverage Options For          The Nation’s Need for Evidence of Comparative
     10 Rochaix, L. and D. Wilsford. “State Autonomy,             Promising Technologies: Medicare’s ‘Coverage             Effectiveness in Healthcare,” September 2007.
        Policy Paralysis: Paradoxes of Institutions and           With Evidence Development,” Health Affairs,           41 Medicare Payment Advisory Commission, Report
        Culture in the French Health Care System,”                Vol. 25, No. 5 (2006): 1218–1230; Chalkidou,             to Congress, “Promoting Greater Efficiency in
        Journal of Health Politics, Policy and Law, Vol. 30,      K. and J. Lord, et al., “Evidence based decision         Medicare,” June 2007.
        No. 1-2 (Feb.–April 2005).                                making in healthcare: when should we wait for         42 Ibid., CBO, 2007.
     11 Conversations at the Ministry of Health Brasilia,         more information?” Health Affairs, Vol. 27, No. 6     43 Agency for Healthcare Research and Quality,
        Brazil, with Dr. Anderson.                                (2008): 1642-1653                                        “Methods Reference Guide for Effectiveness and
     12 Department of Health, “Directions to Primary           27 National Institute for Health and Clinical               Comparative Effectiveness Reviews,” October
        Care Trusts and NHS Trusts in England                     Excellence, “Speech to St. Paul Healthcare by            2007.
        Concerning Arrangements for the Funding of                Professor Sir Michael Rawlins,” September             44 PhRMA, “Principles for Government-Supported
        Technology Appraisal Guidance from NICE,” July            7, 1999,               Health Outcomes Research on Medical
        1, 2003,                       whatwedo/niceandthenhs/speech_to_st_paul_                Technologies and Services,” http://www.phrma.
     13 Department of Health, “The National Health                healthcare_by_professor_sir_michael_rawlins.jsp.         org/health_outcomes_research/.
        Service Constitution,” June 30, 2008, http://          28 National Institute for Health and Clinical            45 Ibid., AHIP, 2007; also see America’s Health               Excellence, “Appraising End-of-Life Medicines:           Insurance Plans, “A Shared Responsibility:
        Publications/PublicationsPolicyAndGuidance/               A NICE Consultation,” November 2008,                     Advancing Toward a More Accessible, Safe, and
        DH_085814.                                                     Affordable Health Care System for America,” May
     14 Department of Health, “Research for Health: A             EndOfLifeMedicinesConsultation.jsp.                      2008.
        Research and Development Strategy for the NHS,”        29 Martin, S., et al., “The Link between Healthcare      46 Commonwealth Fund, “Bending the Curve:
        London, 1991.                                             Spending and Health Outcomes: Evidence from              Options for Achieving Savings and Improving
     15 See              English Programme Budgeting Data,” June 2007,            Value in the U.S. Health Spending,” December,
        News/MRC004215.                                           2007.
     16 Gemeinsamer Bundesausschuss, “The German                  reports/the_link_between.html and follow-on           47 Institute of Medicine, “Knowing What Works
        Health Care System and the Federal Joint                  May 2008,         in Healthcare: A Roadmap for the Nation,”
        Committee,”           research_reports/spending_outcomes.html.                 Washington, DC: National Academies Press, 2008,
        english/.                                              30 However, in reality, individual sickness funds have
     17 Healthcare Modernization Act of 2004,paragraph            little power to amend JFC decisions, which tend
        139a.                                                     to be binding.

48 Medicare Payment Advisory Commission,                 Effectiveness of Medical Treatments,” December        Outcomes Assessment Initiative,” Journal of the
   “Producing Comparative-Effectiveness                  2007.                                                 American Medical Association 298 (2007):1323-5.
   Information.” In Report to the Congress,           51 Kirschner N. “Improving Availability of            53 Congressional Research Service, Report to
   Promoting Greater Efficiency in Medicare,             Comparative Effectiveness and Cost-Effectiveness      the U.S. Congress, “Comparative Clinical
   Washington, DC:, Medicare Payment Advisory            Information: An Essential Feature for a High-         Effectiveness and Cost-Effectiveness Research:
   Commission, 2007,            Quality and Efficient United States Health Care       Background, History, and Overview,” October
   Jun07_Ch02.pdf.                                       System,” A Position Paper of the American             2007.
49 Centers for Medicare and Medicaid Services,           College of Physicians, Philadelphia: American      54 Holve E. and Pittman, P. “A First Look at
   “Medicare Evidentiary Priorities,” October 2007,      College of Physicians, 2008.                          Comparative Effectiveness Research in the United         52 Wilensky G.R. “Developing a Center for                States,” 2009, AcademyHealth.
   EvidentiaryPriorities.asp, accessed April 2008.       Comparative Effectiveness Information.” Health
50 Congress of the United States, Congressional          Affairs 25 (2006) w572-85; E.J. Emanuel et
   Budget Office, ”Research in the Comparative           al., “Essential Elements of a Technology and