Peer Review Advisory Committee Meeting Structured Critiques

							    SHORTENING THE REVIEW CYCLE

PEER REVIEW ADVISORY COMMITTEE MEETING
         STRUCTURED CRITIQUES

              September 26, 2005

         Hortencia Hornbeak, Ph.D.
            Associate Director for
         Scientific Review and Policy
                  DEA, NIAID
  SHORTENING THE REVIEW CYCLE

 Outline for Today’s Discussion
Multi-pronged approach to shortening review cycle
Principal drivers of structured critiques
Structured critique formats
Structured critiques-award mechanisms
Reviewer guidance
Changing reviewer behavior
Resources needed
  SHORTENING THE REVIEW CYCLE

Features of NIH Peer Review that
     Meet the Test of Time
Management of conflict of interest (core value)
Secure appropriate expertise (core value)
Develop collective expert advice through discussion
Provide guidance/feedback to applicant
   SHORTENING THE REVIEW CYCLE

      Multi-pronged Approach
Electronic submission of applications
Knowledge management solutions in referral/recruitment
Electronic recruitment of reviewers for SEPs
Internet assisted review (IAR)
Structured critiques
Abbreviated summary statements
Council approval independent of scheduled meetings
     SHORTENING THE REVIEW CYCLE

Structured Critiques: Principal Drivers
 Accelerated review for AIDS initiatives (mid 1980s)
 Hyper-accelerated review of Innovation grants for
   HIV/AIDS hyper-accelerated review
 Large increase in funds ($1.5 B) for biodefense
   research (2003)
 Five review cycles (73 days per cycle) per FY
     60 days from receipt to review
       SHORTENING THE REVIEW CYCLE

Structured Critiques: Principal Drivers (cont.)
   Hyper-accelerated review of Bioshield initiatives-
     Project Bioshield Legislation 2004
       No FACA rules
       Outside IMPAC II
    Constraints on FY funding
   Lag time to hire and train staff
   SHORTENING THE REVIEW CYCLE

  Five Review Cycles: FY03-04
73 days per cycle for most reviews
SRA team work
Flexible SRA assignments
Advance electronic recruitment of reviewers for SEPs
  based on letters of intent
Structured critiques
IAR assembly of abbreviated summary statements
Administrative review report to GMB and Program
   SHORTENING THE REVIEW CYCLE

Five Review Cycles: FY03-04 (cont.)
Lessons learned
   Secure appropriate expertise with large numbers
    of potential reviewers in conflict
   Tools developed to manage large reviews
     Structured critiques
     Practical guidance for reviewers
     Administrative workbooks
     Reviewer Support Site
     Publication search macro for conflicts
   SHORTENING THE REVIEW CYCLE

Five Review Cycles: FY03-04 (cont.)
   Effective flexible working teams
   Conflict of interest management
   Development of staff training resources
   Effective communications with stakeholders
   Pre-meeting teleconferences to educate
    reviewers
   Increase use of teleconferences for reviews
   SHORTENING THE REVIEW CYCLE




STRUCTURED CRITIQUE FORMAT
  SHORTENING THE REVIEW CYCLE


  Structured Critique - Benefits
Goal
Improve utility for stakeholders by:
    More focused discussions
    More concise written evaluations
    Facilitated preparation of abbreviated summary
     statements
 SHORTENING THE REVIEW CYCLE


Structured Critique - Challenges
Requires change in behavior of reviewers
May adversely affect “tutorial” aspect
May not capture adequately complexity and/or
  subtleties
May lead to applicant appeal
  SHORTENING THE REVIEW CYCLE
 Structured Critiques: Simple and
      Complex Mechanisms
R03
R21
R01/U01
P54   Regional Centers - two-tiered review
P01/U19
UC6   Biocontainment laboratories
   SHORTENING THE REVIEW CYCLE

  Structured Critique Template
Tailored for each initiative
Initiative-specific review criteria
Use short phrases to describe each review criteria
Based on the five NIH review criteria as a scaffold for
  initiative specific criteria
   SHORTENING THE REVIEW CYCLE
    Critique Structured Format
Significance:
   Strength # 1 [strongest]    (1-2 lines)
     Justification/explanation (2-4 lines)
   Strength # 2                (1-2 lines)
     Justification/explanation (2-4 lines)
   Strength # X                (1-2 lines)
     Justification/explanation (2-4 lines)
   SHORTENING THE REVIEW CYCLE

Critique Structured Format (cont.)
   Weakness # 1 [most serious weakness] (1-2 lines)
     Justification/explanation          (2-4 lines)
   Weakness # 2                          (1-2 lines)
     Justification/explanation           (2-4 lines)
   Weakness # X                          (1-2 lines)
     Justification/explanation           (2-4 lines)
     SHORTENING THE REVIEW CYCLE

Critique Structured Format (cont.)
Continue with format in previous slides for:
    Approach

    Innovation

    Investigator

    Environment Criteria
SHORTENING THE REVIEW CYCLE




   EXAMPLE OF A
STRUCTURED CRITIQUE
        SHORTENING THE REVIEW CYCLE

    Example of Structured Critique
RESUME AND SUMMARY OF DISCUSSION: This (adjectival
rating) application entitled “             " was submitted
by (organization), Principal Investigator. The applicants propose to (one
sentence summary of specific aim(s). The principal strengths identified by
reviewers include              ,        ,          , and          . The
major weaknesses are              ,            ,         , and          .
After discussion, the panel concluded that                      , and
scored the application in the      range.
       SHORTENING THE REVIEW CYCLE

Example of Structured Critique (cont.)
OVERALL EVALUATION

This is an excellent application in response to the RFA…... It is based on
solid hypotheses and preliminary evidence that XX inhibitors can protect
from toxin-mediated pathogenicity of B. anthracis. The team is highly
experienced and has worked together to establish the background for the
project. Goals are clearly laid out, experiments are generally well
described, and there is a high probability of significant advances in
therapy of anthrax-related toxicity in patients.
       SHORTENING THE REVIEW CYCLE

Example of Structured Critique (cont.)
CRITIQUE
 Significance
    Strength 1 - novel approach to block enzymatic conversion of a
     bacterial XX protoxin in several bacteria, including B. anthracis
    Strength 2 - excellent likelihood of successful development of lead
     compounds
    Strength 3 - excellent opportunity to test lead compounds directly in
     toxin-treated animals
    Weakness 1 - unknown toxicity related to furin inhibitors
     SHORTENING THE REVIEW CYCLE

Example of Structured Critique (cont.)
Approach
   Strength 1 - solid rationale and preliminary evidence for
    Pseudomonas-related toxin XX, including protection of toxin-
    induced death in mice
   Strength 2 - lead compound XXX has potent binding to target
   Strength 3 - excellent team to conduct studies
      SHORTENING THE REVIEW CYCLE

Example of Structured Critique (cont.)
Approach (cont.)
   Weakness 1 - Is furin block in vivo sufficient to cure infection
    or only protect from XX-mediated toxicity?
   Weakness 2 - in preliminary results, the PI concludes that
    “XXX actually enters cells”, but the data only demonstrate
    that “biotinylated XX” enters cells - they are quite different
    substances. To further understand the apparent lack of
    cytotoxicity of XXX, permeabilized cells should be used.
      SHORTENING THE REVIEW CYCLE

Example of Structured Critique (cont.)
Innovation
   Strength 1 - detailed mechanistic study of XXX effect on XX
    toxicity to RAW cells
   Strength 2 - study of kinetics and time course of protection
    from XX and cell-surface bound iodinated XX
   Strength 3 - design of XX analogs including synthetic
    peptides, polyamines and related peptidomimetics
   Strength 4 - in vivo studies examining efficacy of XXX on
    TNF production and on XX induced toxicity in animal models
   Weakness 1 –the approach not innovative and could benefit
    from collaborative expertise for these studies.
      SHORTENING THE REVIEW CYCLE

Example of Structured Critique (cont.)
Investigator
  The PI is experienced in peptide hormone studies and peptide
  processing. Collaborator Dr. XX is experienced in peptide
  synthesis. The team has successfully worked together in the
  discovery of polyarginine furin inhibitors. The application would
  be strengthen by additional collaborative expertise for XX studies

Environment
  Lab facilities of the PI are adequate for the studies. Presumably
  those of Dr. XX’s institution are also adequate.
      SHORTENING THE REVIEW CYCLE

Example of Structured Critique (cont.)

Budget
 Appropriate for the level of effort proposed.

Animal Welfare
 Comments: A letter of approval of an animal protocol was
 included. It does not indicate if both mice and rats are approved
 for the study. The PI does not address the “five questions” about
 animal usage. Nature of anesthesia is not indicated.
   SHORTENING THE REVIEW CYCLE




   STRUCTURED CRITIQUES:
MECHANISM-SPECIFIC EXAMPLES
     SHORTENING THE REVIEW CYCLE

         Structured Critiques:
      Simple to Complex Initiatives
R21- Rapid Response Grant Program on Bioterrorism-
  Related Research
U01/U19 RFA Cooperative Research for the
  Development of Vaccines, Adjuvants, Therapeutics,
  Immunotherapeutics, and Diagnostics for Biodefense
Biodefense Countermeasure Development
     SHORTENING THE REVIEW CYCLE

          Structured Critiques:
        Benefits – All Mechanisms
More focused assessments of the applications
Increased time for discussion of competitive
  applications
Enabled committee to manage larger number of
  applications per SEP.
Faster SS completion for awarded grants
   SHORTENING THE REVIEW CYCLE

      Structured Critiques - R21
RFA Rapid Response Grant Program on Bioterrorism-
  Related Research (2002)
   Mechanism                      R21
     Number of applications       303
     Review committees            3 SEPs
     Number of members/SEP        25
     Streamlined applications     33 percent
 SHORTENING THE REVIEW CYCLE


Structured Critiques - R21 (cont.)

  Electronic critique templates
  3 SRAs assigned
  Resume and summary of discussion templates
  Faster SS completion – 4 weeks
      SHORTENING THE REVIEW CYCLE

  Structured Critiques - U01s and U19s
 RFA Cooperative Research for the Development of
   Vaccines, Adjuvants, Therapeutics, Immunotherapeutics,
   and Diagnostics for Biodefense (2003)
    Mechanism                          U01, U19
       Number of applications               186
       Review committees                    3 SEPs
       Number of members/SEP                40
       Streamlined applications             33 percent
       Electronic critique templates
     SHORTENING THE REVIEW CYCLE

Structured Critiques - U01s and U19s (cont.)

      Resume and summary of discussion templates
      Templates for resume and summary of discussion
      3 SRAs assigned
      Faster SS completion - 3 weeks
   SHORTENING THE REVIEW CYCLE

Structure Critiques - Project Bioshield
Biodefense Countermeasure Development
  Hyper-accelerated Review (2004)
   Mechanism                       N/A (small grants)
     Number of applications        66
     Letter of intent/program
          Pre-approval
     Review committees             1 SEP
     Number of members/SEP         23
     Streamlined applications      60 percent
     SHORTENING THE REVIEW CYCLE

Structure Critique - Project Bioshield (cont.)

       Teleconference review
       Receipt to award                    60 day
       Electronic critique templates
       Enabled committee to manage the large workload in
         a hyper-accelerated time frame
       Faster SS completion - 2 weeks
          Not standard summary statements
   SHORTENING THE REVIEW CYCLE

Guidance Provided to Reviewers
Orientation teleconference with members from
  multiple committee for standardization
Electronic critique template tailored to the initiative.
Instructions for Reviewer Manual tailored for each
  initiative or SEP
Overlapping reviewers among the review panels
    SHORTENING THE REVIEW CYCLE

 Changing Behavior of Reviewer
Clearly stated expectations
Focused rational rationale and objectives
Required buy-in from all stake holders
    NIH
    NIAID
    reviewers
Make it easy across the board
    SHORTENING THE REVIEW CYCLE

Resources Needed to Decrease Time
  Sufficient numbers of trained staff
  Effective SRA/GC Work teams
  Flexible staff assignments
  Versatile and state of the art IT support
    SHORTENING THE REVIEW CYCLE

                  Summary
Preserve features that make NIH Peer Review work
  with any changes adapted
   Manage conflict of interest (core value)
   Secure appropriate expertise (core value)
   Develop collective expert advice through
     discussion
   Provide guidance/feedback to applicant

Structured critiques are applicable to many award
  mechanisms: simple to complex
   SHORTENING THE REVIEW CYCLE

           Summary (Cont.)
Reviewers must be properly oriented in pre-review
  teleconferences
Reviewers do adapt to change
Structured critiques decrease time for summary
  statement preparation
Resulting product (SS) more useful to
  stakeholders
     SHORTENING THE REVIEW CYCLE

               Summary (cont.)
Structured critiques along with other innovations
   can shorten the review cycle
    Electronic submission of applications
    Knowledge management solutions in referral
    Electronic recruitment of reviewers for SEPs
    Internet assisted review (IAR)
    Abbreviated summary statements
    Council approval independent of scheduled meetings
SHORTENING THE REVIEW CYCLE


       Questions?