National Institute for Health and Clinical Excellence
Centre for Health Technology Evaluation
Diagnostics Assessment Programme
Report on pilot project on the assessment of non-
invasive diagnostic assessment tools for the
detection of liver fibrosis in patients with suspected
alcohol related liver disease
March 2011
1 of 17
Contents
Background and summary ............................................................................... 3
Objectives of the pilot....................................................................................... 4
Pilot process .................................................................................................... 5
Key groups involved in the pilot .................................................................... 5
The pilot schedule ........................................................................................ 6
Pilot methods ................................................................................................... 8
Engagement of key stakeholder groups........................................................... 9
Evaluation and lessons learned ..................................................................... 10
Appendix: Members of groups involved in pilot project .................................. 14
Diagnostics Methods Working Group ......................................................... 14
Medtech Stakeholder Reference Group ..................................................... 15
Evaluation Pathway Project Board ............................................................. 16
2 of 17
Background and summary
1. In 2008 NICE and the British In Vitro Diagnostics Association (BIVDA)
attended a workshop to discuss potential approaches to the evaluation
of diagnostic technologies by NICE. This built on NICE’s previous
experience in evaluating a range of health technologies, and
developing methods and processes for their evaluation.
2. The Diagnostics Assessment Programme (DAP) was set up by NICE in
2009/10 to evaluate innovative medical diagnostic technologies with
the aim of supporting the NHS in adopting clinically and cost effective
technologies more rapidly and consistently.
3. As part of this, NICE set up a pilot project. It was designed so that the
programme could learn from the pilot on a rolling basis, and so that the
new programme could start without waiting for the pilot to finish. The
outputs of the pilot were used to develop the processes and methods
for the substantive programme, and the draft programme manual is due
to be presented to the NICE Board in Spring 2011.
4. The pilot has now been successfully completed, and the purpose of this
report is to describe the pilot and highlight how NICE responded to the
comments of stakeholders about the methods and processes of
diagnostic assessment.
5. All the pilot objectives listed in paragraph 7 have been met. NICE
identified important lessons from the pilot and all of these have directly
influenced the work of the substantive programme:
The importance of developing a detailed scope that includes
information on how the test is used in the care pathway
The importance of the involvement of specialist committee
members, who are decision-makers with expert knowledge, to
ensure that decisions are clinically robust
The importance of a clearly defined role for specialist committee
members.
The need for flexibility in evaluation methods as the nature of
diagnostic tests and their uses are varied
The ability to evaluate multiple products if appropriate.
3 of 17
6. When NICE began working with industry representatives on the
evaluation of diagnostic tests, NICE was asked to develop methods
and processes that addressed the unique features of diagnostics and
their usual evidence base, while at the same time applying the highest
possible methodological rigour. The development of realistic methods
was a major focus of the pilot, and a large number of academic,
regulatory and other stakeholders have advised NICE. Although NICE’s
methods for Technology Appraisals were the starting point, in the
course of the pilot we considered what was different about evaluating
diagnostics and aimed to capture this in our methods for the new
programme. An interim methods statement was published in November
2009 to describe the methods that would be followed in the pilot. A
manual outlining the programme’s methods and processes is being
developed, building on the experience of the pilot.
Objectives of the pilot
7. The objectives of the pilot were outlined in the project initiation
document (PID) of the project to set up the DAP:
‘To identify issues in relation to applying the draft methods in
practice.
To highlight any omissions from the [interim] methods
statement.
To inform the development of the methods guide for the
programme.
To allow stakeholder input into the development of the methods
design.
To produce an assessment report for consideration by the
diagnostics advisory committee. Although the assessment report
from the pilot will not make formal recommendations on the
product being assessed, it will be available for review by the
diagnostics advisory committee who will formulate
recommendations.’ (DAP PID June 2009)
8. In addition, the programme team found that the pilot project was
invaluable for testing the proposed processes and timetable of the new
programme.
4 of 17
Pilot process
Key groups involved in the pilot
9. The following groups were involved in the pilot:
10. Diagnostics Advisory Committee (DAC). This new type of
Committee was established between late 2009 and early 2010. It has
both standing members and specialist members. Standing members
have experience across a range of disciplines, while specialist
members have specific expertise in the topic being evaluated. Standing
members were recruited according to NICE’s standard procedures for
recruiting advisory committee members. For the pilot, specialist
committee members (SCMs) were recruited with expertise in liver
fibrosis and/or alcohol-related liver disease. NICE’s standard
committee recruitment procedures were adopted, but specific process
steps were developed for recruitment of these members due to the
need to recruit them on a frequent basis in the ongoing programme.
11. External assessment group (EAG). The EAG for the pilot was
ScHARR (School of Health and Related Research, University of
Sheffield). ScHARR is an experienced assessment group for NICE’s
technology appraisals programme and had some prior experience of
assessing diagnostics. They carried out the evidence assessment and
produced the evidence assessment and analysis report (EAAR),
guided by the interim methods statement.
12. Pilot-registered stakeholders. Organisations and individuals with an
interest and/or expertise in the pilot topic were identified from
December 2009 onwards and invited to register as stakeholders. One
of the requirements for being a registered stakeholder was to sign a
confidentiality agreement. Registered stakeholders received the EAAR
and draft pilot recommendations for comment, and the pilot discussion
document for information. The registered stakeholders could attend the
DAC meetings in September and November 2010. During the course of
the pilot, the DAP team simplified the registration process and now
automatically registers the manufacturers of technologies being
assessed, and SCM applicants, as stakeholders.
13. Methods working group (‘methods WG’). As provided for in the PID,
this group was set up in July 2009 to advise on the development of the
methods for the assessment of diagnostics. The group includes highly
respected academics in the field of health technology assessment,
representatives from the diagnostics industry, the Department of Health
and NICE staff. Membership of the group is listed in the appendix. See
5 of 17
‘Pilot methods’ for more information about the development of the
programme’s methods and the role of the methods WG.
14. Medtech stakeholder reference group (‘medtech SRG’). This group,
originally set up by the Department of Health in 2008, supports the
development of NICE’s medical technologies evaluation activities,
including the DAP. The group has representatives from the medtech
industry, the NHS, patient and carer organisations, the Department of
Health and others. The group’s membership is listed in the appendix;
there is some crossover in membership between the methods WG,
DAC standing members and pilot registered stakeholders.
15. Evaluation Pathway project board. This project board existed for 1
year (May 2009 to April 2010) and oversaw the establishment of the
new medical technologies evaluation programme (formerly called the
evaluation pathway programme for medical technologies) and the early
establishment of the DAP. Membership of the project board is listed in
the appendix.
16. NICE DAP team. The early stages of the pilot were managed by a part-
time home-based technical adviser and a temporary assistant project
manager, with support from the programme director. Programme staff
were recruited during the course of the pilot and contributed to its
development.
The pilot schedule
17. The pilot followed these steps:
18. Choice of topic: When the pilot topic was chosen the programme
team was not in post and the normal topic selection mechanism for the
programme was not yet in place. The process followed is therefore not
representative of current topic selection processes for the programme
and will not be repeated in the future. The topic was selected by the
methods WG based on nominations previously identified to NICE
through various mechanisms, notably horizon scanning:
Two topics were shortlisted in July 2009 – one was ‘new tests
for liver fibrosis’.
Draft scopes were drawn up for these two topics during July and
August 2009.
The final topic was chosen in August 2009 – ‘non-invasive
diagnostic tests for the evaluation of liver fibrosis’.
6 of 17
19. Scoping:
A scoping workshop in October 2009 was attended by invited
representatives from industry, academia and lay organisations.
The format was similar to a technology appraisal scoping
workshop. As there are multiple causes of liver fibrosis, the
workshop considered five possible causes and narrowed the
scope down to two – hepatitis B and alcohol-related liver
disease.
For pragmatic reasons, the NICE DAP team chose a single
aetiology for the pilot assessment, alcohol-related liver disease,
in October 2009.
The final scope was agreed and signed off by the centre director
in December 2009 as ‘non-invasive diagnostic assessment tools
for the detection of liver fibrosis in patients with suspected
alcohol-related liver disease’.
20. Assessment:
The assessment protocol was agreed between NICE and
ScHARR in early January 2010. The assessment protocol
outlined how the EAG would assess the evidence and develop
the EAAR.
A topic background meeting was held between the newly
recruited SCMs, the DAC chair, the EAG and the DAP team in
March 2010.
A technical meeting of DAC took place in May 2010 to review
the draft EAAR.
The final EAAR was ready in July 2010.
Registered stakeholders were given two weeks to comment on
the EAAR between July and August 2010.
21. The Diagnostics Advisory Committee:
DAC met on 24 September 2010 to review the EAAR and
registered stakeholders’ comments, and agree draft pilot
recommendations.
NICE drafted pilot guidance based on DAC’s draft pilot
recommendations.
7 of 17
Pilot registered stakeholders had three weeks to comment on
the draft pilot guidance in October and November 2010.
DAC met on 23 November 2010 to review the comments and
agree final pilot recommendations.
22. The final pilot discussion document was developed by NICE based
on DAC’s final pilot recommendations and was circulated to registered
stakeholders for information in January 2011. The pilot did not aim to
develop ‘live’ guidance to the NHS, and none was produced.
Pilot methods
23. The assessment of diagnostic technologies is complex, as patient
benefits need to be modelled through the post diagnosis care
pathway(s). The benefits to the patient are determined by what
happens after the diagnostic technology is used. These downstream
benefits need to be modelled in order to assess the value of the
technology. Where there is a nationally agreed clinical guideline,
decisions about what to include in the assessment of the post
diagnosis care pathway are more straightforward. When no clinical
guideline exists, additional expert input is needed, and a number of
options for post diagnosis care pathways may need to be modelled.
24. The evidence base for diagnostic technologies is typically much thinner
than that for pharmaceuticals due to the less onerous regulatory
framework. In this context, trialling assessment methods and refining
them in the light of experience was a major dimension of the pilot. This
allowed tailoring of bespoke methods and processes, designed
specifically for the evaluation of diagnostic technologies.
25. Due to the complexity of diagnostics assessment and the lack of
generally accepted methodologies, it was identified at a very early
stage that receiving expert input and building expert consensus was
key to the success of the programme. The methods WG was set up to
provide this expertise.
26. The methods WG met seven times over an 18-month period to January
2011. In addition, the members commented on the two key methods
documents developed by NICE based on the group’s input: the interim
methods statement (see Diagnostics assessment) and the draft
methods guide. (The draft methods guide will be incorporated into a
single programme manual.)
8 of 17
27. The methods WG also contributed expertise to the design and delivery
of a methodology workshop for external assessment groups in autumn
2010. The complexity of diagnostics assessment is such that the NHS’
National Institute for Health Research (NIHR) recognised that
additional training was required for assessment group members. The
session was delivered by leading academics, industrialists and NICE
staff.
Engagement of key stakeholder groups
28. NICE was keen to ensure that the methods and processes of the new
programme were developed with significant input from key stakeholder
groups, to ensure that the programme elements were fit for purpose
and also to build a consensus of support for the programme. Key
stakeholder groups were identified and their involvement sought as
outlined below.
29. Manufacturers of diagnostic technologies: The programme is driven
by manufacturer notifications, and industry is an important stakeholder.
Manufacturers were represented on the methods WG, the medtech
SRG and the EP project board, by the trade industry associations
Association of British Healthcare Industries (ABHI), British In Vitro
Diagnostics Association (BIVDA) and the Association for Healthcare
Technology Providers for Imaging, Radiotherapy and Care (AXrEM).
They were also invited to become pilot registered stakeholders. In
addition, these industry bodies and the senior directors at NICE started
quarterly meetings during the pilot; these will continue beyond the pilot.
30. Health technology assessment academic community: The support
of the academic community is necessary to ensure the technical
credibility of the programme’s guidance. Academics were mainly
involved in the pilot via the methods WG, which benefited from the
contributions of leading scholars in the field. In addition, some
members of the academic community joined the medtech SRG, and
others were recruited as standing members of the Diagnostics Advisory
Committee.
31. Department of Health: As sponsors of NICE’s new medtech
evaluation activity, Department of Health representatives were actively
involved with the EP pathway board, the medtech SRG and the
methods WG.
32. Technology Strategy Board: During the pilot and establishment of the
early stages of the live programme, strong links were made with the
Technology Strategy Board (TSB). A TSB/NICE industrial engagement
9 of 17
workshop was held in December 2010, with companies participating in
the TSB Detection and Identification of Infectious Agents innovation
platform. NICE is also a partner organisation on TSB’s Stratified
Medicine innovation platform, and contributed presentations at three
launch workshops held late in 2010. Involvement in TSB initiatives is
viewed as an important mechanism for ongoing engagement with
industry for the Diagnostics Assessment Programme.
Evaluation and lessons learned
33. A formal evaluation of the pilot was commissioned. In addition, as the
pilot progressed, the programme team reviewed lessons learned and
implemented them into the evaluation processes of the programme as
quickly as possible.
34. Formal evaluation. An independent review of the pilot project was
commissioned in October 2010 from the Healthcare Innovation and
Technology Evaluation Centre (HITEC), Derby Hospitals NHS
Foundation Trust. HITEC obtained the views of the Diagnostics
Advisory Committee, the methods WG, the EAG, the pilot registered
stakeholders and the medtech SRG. These groups were asked about
the various phases of the pilot project and about process issues such
as communications. We have reviewed the independent report
carefully and our key conclusions, derived from the recommendations
in the evaluation report, are highlighted below:
The pilot topic itself was difficult to assess due to poor evidence
and uncertain post-diagnosis care pathways, but this made it
useful for testing the pilot’s methods and processes.
The prime consideration in selecting future topics for evaluation
should be their importance to the NHS; ease of assessment
should not be the determining factor.
Due to the importance of understanding the post diagnosis care
pathway, the contribution of specialist committee members was
invaluable.
Most participants felt involved in the pilot project and were
pleased with the information they received. However there were
occasional technical and communication issues. NICE has taken
steps to ensure these problems do not recur.
35. The HITEC review is consistent with NICE’s own learning from the
programme. The full report is available here: Diagnostics assessment.
10 of 17
36. Lessons learned. The Diagnostics Assessment Programme team
reviewed the pilot as it progressed and the emerging findings were
incorporated into the processes and methods used for topics in the
substantive programme. The main lessons learned were:
The importance of scoping the topic so that the evaluation is
both worthwhile and feasible. This requires a thorough
understanding of the care pathway, particularly in the absence
of acknowledged guidance such as a NICE clinical guideline.
The scoping phase of the live programme now incorporates
more detailed preliminary scoping, involving NICE’s technical
lead for the topic and the Information Services team.
An additional step that has been added to the process is a
review of the draft scope (revised after the scoping workshop)
by an assessment sub group (DAC chair, topic SCMs, two
standing committee members), the topic EAG and programme
staff.
It may be appropriate to expand an evaluation to include more
than one technology. The programme team has developed
appropriate processes for identifying additional technologies and
adding them to the evaluation where appropriate.
Although SCM input is very important in determining and
clarifying the care pathway, there is a potential for a conflict of
interest between members contributing their topic expertise to
the assessment, and members remaining objective enough to
participate in the committee’s decision-making. The programme
team has made the briefing of both SCMs and EAGs clearer,
and this subject is covered in the SCMs’ induction.
The need for increased understanding by DAC of health
economic modelling. The programme team has arranged for a
methodology workshop and briefing sessions, and more time will
be spent at committee meetings outlining the economic model.
That evaluation of diagnostic technologies is difficult, and the
methods for doing so will evolve over time. For instance, based
on the pilot experience, the methodology for live topics has been
designed in a pragmatic way to make the best use of available
evidence in parallel with encouraging the development of robust
data. Following methods WG discussions, the decision was
taken to include a structured information request to the
technology manufacturers, rather than being prescriptive on
11 of 17
evidence requirements and requiring a formal submission.
Evidence expectations for diagnostics assessments are likely to
evolve over an extended period taking account of experience
with early topics and the economic constraints of the diagnostics
industry.
37. The table below highlights how the objectives of the pilot were
achieved:
Objective (from the project Progress
initiation document, June
2009)
To identify issues in relation to The pilot identified and explored
applying the draft methods in specific methodological issues
practice relevant to evaluating diagnostic
technologies. Key methodological
issues identified were discussed
with the methods working group
and the outcome of these
discussions was incorporated into
the development of live projects
and the programme manual
To highlight any omissions from The interim methods statement was
the [interim] methods statement used as a starting point for the draft
programme manual which now also
incorporates the lessons learned
from the pilot
To inform the development of The programme manual is being
the methods guide for the developed using the experience
programme gained from the pilot and
substantial input from the methods
working group
To allow stakeholder input into The methods working group
the development of the methods involved wide stakeholder
design representation and was actively
involved in discussing and
proposing approaches outlined in
12 of 17
the interim methods statement and
the programme manual, including
commenting on drafts of both
documents
To produce an assessment An evidence assessment and
report for consideration by the analysis report was commissioned
diagnostics advisory committee. from ScHARR and considered by
Although the assessment report the Committee when it reviewed
from the pilot will not make the diagnostic technologies chosen
formal recommendations on the for the pilot
product being assessed, it will
be available for review by the
diagnostics advisory committee
who will formulate
recommendations
38. Current information about the programme is available at
www.nice.org.uk/diagnostics.
13 of 17
Appendix: Members of groups involved in pilot project
Diagnostics Methods Working Group
July 2009 – January 2010
Dr Carole Longson (Chair), NICE
Dr Nick Crabb (Vice Chair), NICE
Professor Alan Brennan, School of Health and Related Research, University of
Sheffield (ScHARR)
Professor Andrew Stevens, Unit of Public Health, Epidemiology & Biostatistics,
University of Birmingham
Dr Anne Mackie, UK National Screening Committee
Dr Anthony James, NHS Institute for Innovation and Improvement
Baish Naidoo, NICE
Carole Cohen, Edwards Lifesciences
Dr Chris Hyde, Professor of Public Health and Clinical Epidemiology, Peninsula
College of Medicine & Dentistry, University of Exeter
Dr Craig Ramsey, College of Life Sciences and Medicine, University of Aberdeen
David Owolabi, Roche Diagnostics
Eleanor Donegan, NICE
Dr Elisabeth George, NICE
Frances Nixon, NICE
Francis Ruiz, NICE
Franz Hessel, Abbott Diagnostics
Georgios Lyratzopoulos, NICE
Gurleen Jhuti, NICE
Dr Hanan Bell, NICE
Dr Helen Chung, NICE
Dr Ian Barnes, Department of Health
Jennifer Butt, NICE
Jill Dhell, Department of Health – Research and Development (DH R&D)
Dr Jo Lord, Health Economics Research Group, Brunel University
Professor Jon Deeks, Unit of Public Health, Epidemiology & Biostatistics, University
of Birmingham
Dr Kalipso Chalkidou, NICE
Laura Norburn, NICE
Mark Samuels, Roche Diagnostics
Professor Mark Sculpher, Team for Economic Evaluation and Health Technology
Assessment, University of York
Professor Martin Buxton, Health Economics Research Group, Brunel University
Dr Matt Stevenson, ScHARR
Matthew Stork, AXrEM
Dr Meindert Boysen, NICE
Mirella Marlow, NICE
Dr Myfanwy Lloyd Jones, ScHARR
Professor Paul Glasziou, Department of Primary Healthcare, University of Oxford
Professor Peter Littlejohns, NICE
Dr Phil Alderson, NICE
Rebecca Trowman, NICE
Ravi Chana, Roche Diagnostics
Dr Rod Taylor, Peninsula Technology Assessment Group, University of Exeter
14 of 17
Sandra Lopes, ABHI
Dr Sarah Garner, NICE
Selma Audi, Boston Scientific
Seren Phillips, NICE
Professor Sue Hill, Diagnostics Programme Board, DH
Dr Susanne Ludgate, MHRA
Professor Tom Walley, NIHR Health Technology Assessment programme
Tarang Sharma, NICE
Toni Price, NICE
Victoria Thomas, NICE
Medtech Stakeholder Reference Group
2008 (set up by DH as interim working group) – March 2011
Carole Longson, NICE (Chair)
Mirella Marlow, NICE (Vice Chair)
Lucy Allen, NIHR
Selma Audi, Association of British Healthcare Industries (ABHI)/Boston Scientific
Sarah Baggaley, NICE
Christine Bantock, Department of Health – Medicines, Pharmacy and Industry
(DHMPI)
Peter Barker, Coventry primary care trust (PCT)
Ian Barnes, DH national clinical lead, pathology
Sandra Barrow, DH Rapid Review Panel
Gifford Batstone, DH national clinical lead, Connecting for Health – pathology
Hanan Bell, NICE
Susan Bennett, lay representative
Judy Birch, lay member
Jennifer Butt, NICE
Bruce Campbell, Chair, Medical Technologies Advisory Committee
Mark Campbell, NICE
Richard Carter, DHMPI
Ravi Chana, Roche Diagnostics
Matthew Chapman, Quotec
Sally Chisholm, NHS National Technology Adoption Centre
Kirstie Clegg, NICE
Alison Cook, National Horizon Scanning Centre Nick Crabb, NICE
Paul Cryer, DH Rapid Review Panel
Erika Denton, National Clinical Director for Imaging, DH
Jill Dhell, DHRDD
Bernice Dillon, NICE
Ann Doyle, King’s College London
Sue Dunkerton, TWI
Elizabeth Dymond, Institute of Physics and Engineering in Medicine (IPEM)
Neil Ebenezer, Medicines and Healthcare products Regulatory Agency (MHRA)
John Egan, Health Technologies and Medicines Knowledge Transfer Network
Deirdre Feehan, DH Modernising Pathology lead
Jennifer Field, NICE
Alison Fowlie, Medical director, Derby Hospitals NHS Foundation Trust
Tonya Gillis, NICE
Christopher Gush, DH Rapid Review Panel
Alan Hickman, Managing director, Quotec
15 of 17
Sue Hill, DH Chief Scientific Officer
Simon Hiller, DHMPI
Melissa Hillier, lay member
Marlon Hope, NICE
Peter Houghton, NHS South East Coast
Tracy Huggins, Roche Diagnostics
Anthony James, National Institute for Innovation and Improvement
Claudia Lally, DHMPI
Lizzy Latimer, NICE
Colm Leonard, University Hospital of South Manchester; NICE
Susanne Ludgate, MHRA
Georgios Lyratzopoulos, Cambridge University; NICE
Anne Mackie, Medical director, UK National Screening Committee
Rajan Madhok, Medical director, NHS Manchester
Judith Mellis, ABHI
Mangala Murali, NICE
Jacqui Nettleton, Primary Care Development, West Sussex PCT
Laura Norburn, NICE
Jacqueline O'Callaghan, NICE
Melanie Ogden, Nursing and Quality Directorate, Northwest SHA
David Owolabi, British In Vitro Diagnostics Association (BIVDA)/Director of Medical
and Public Affairs, Roche Diagnostics
Marg Parton, NHS National Technology Adoption Centre
Claire Packer, National Horizon Scanning Centre
Kay Pattison, DH Health Technology Assessment
Suzi Peden, NICE
Imran Rafi, Royal College of General Practitioners (RCGP) Clinical Innovation and
Research Centre
Philip Ranson, NICE Alaster Rutherford, NICE
Rashmi Sarmah, DHMPI Gary Shield, NICE
Margaret Stanton, DH NICE sponsor branch
Matthew Stork, AXrEM/Toshiba
Andy Taylor, ABHI
Mike Wallace, ABHI/Johnson & Johnson
Tom Walley, NIHR
Tony Warriner, NICE
John Warrington, NHS PASA
Doris-Ann Williams, BIVDA
Brian Winn, NHS National Innovation Centre
David Wright, DHMPI
Terry Young, MATCH
Evaluation Pathway Project Board
May 2009 – April 2010
Jennifer Butt, NICE
Bruce Campbell, chair of Medical Technologies Advisory Committee
Richard Carter, DH MPI
Nick Crabb, NICE
Andrew Dillon, NICE (co-chair)
Sarah Garner, NICE
Tonya Gillis, NICE
Marlon Hope, NICE
16 of 17
Carole Longson, NICE
Georgios Lyratzopoulos, NICE
Rajan Madhok, Manchester PCT
Mirella Marlow, NICE
Adrian Newland, chair of Diagnostics Advisory Committee
Mark Samuels, Roche Diagnostics (co-chair)
Colin Warriner, BIVDA
Mike Wallace, Johnson & Johnson Medical Ltd (co-chair)
Glenn Wells, DH R&D
17 of 17