"Review Outcomes for Rs in CSR"
Review Outcomes of R03s in CSR Valerie L. Durrant, Ph.D. SRA, Health of the Population IRG Center for Scientific Review Key features of R03s • Small research projects – pilot/feasibility studies – secondary analysis – small projects – development of research methods/technologies • Budget: Up to 2 years; $50K/year. Modular. • 10 page research plan • Must be submitted in response to a PA – PA 06-180 (replaced 03-108) is the parent PA Other features of R03s • SF424—all electronic submission (6/1/2006) • 2 revisions allowed; 1 page introduction • Preliminary data not required • No competing continuations • Appendix limitations (only graphics, survey questionnaires) • Used for a variety of research projects and goals (e.g. Fogarty international cooperative projects, data archiving, etc). Number of R03s reviewed in CSR and ICs, 2001-2006 4500 4000 3500 34% Number of applications 44% 3000 30% 2500 25% 21% CSR Review 2000 IC Review 1500 1000 500 0 2001 2002 2003 2004 2005 Fiscal year of review R03s are a relatively small percent of CSR-reviewed applications Percent of applications reviewed by CSR by mechanism (January 2006-October 2006 council rounds) Other (D, K, P, S ,T, U, & other Rs) R41/42/43/44 Fs 2% 10% 7% R21 17% R15 2% R03 3% R01 59% N=50,801 Percent of R03s (CSR-reviewed) increased from 2.1% to 3.2% between 2000 and 2006. CSR-reviewed R03s are concentrated in DCPS and specific IRGs Distribution of R03s applications by Division of review (Oct 2005 – May 2006 councils) DPP IDM 9% DBBD 11% 21% DMCM 25% BDA 13% DCPS 43% HOP 16% RPHB 10% R03 applications have other unique characteristics • More likely to • Less likely to be have new resubmitted investigator as PI 70 70 60 60 50 50 40 R01 40 R01 30 R03 30 R03 20 20 10 10 0 0 % new investigators (Type 1 apps., OER % of unfunded applications resubmitted as data, FY2004) A1 (CSR apps received in 2002, OER data) Most R03 applications are reviewed in standing study sections % of R03 applications reviewed in different types of review groups (CSR review only; n =1660) 74.20% 80% 60% 40% 14.00% 20% 5.10% 3.00% 2.30% 1.30% 0% Standing Special topics RFAs and Member and Small business Overflow and (Chartered, (90s) PARs SRA conflicts other quorum SEPs) R03 review in CSR •Review guidelines instruct reviewers to focus on conceptual framework and general approach, place less emphasis on methods •Review challenges Avoiding R01 expectations; keeping reviewers focused on mechanism Keeping budget out of review When there are few R03s: how to determine a fair review Who are ―peers‖ of R03s? Review Outcomes •Does the score distribution of R03s differ from that of R01s? •Does the score distribution of R03s differ when they are reviewed in different types of review groups? Description of analysis • Limited to Type 1 applications • Looking at raw score distributions – 5 categories: <=150, 151-175, 175-200, 200+, unscored – Many R03s do not receive a percentile • Comparison group—Type 1 R01s in standing study sections (excluding study sections with a primary focus on a non-R01 mechanism) • Excludes ICP1 study section (which only reviews Fogarty International Research Collaboration R03s) R03 review outcomes are similar to Type 1 R01 applications Percent of Type 1 R01 and R03 applications reviewed in standing study sections* in score categories (October 2005-January 2006 council rounds) 60 50 R01 40 (n=12,409) 30 R03 20 (n=1,073) 10 0 <151 151-175 176-200 >200 Unscored *Excludes a few study sections with a primary focus on a non-R01 mechanism R03 review outcomes are similar in different review forums Percent of Type 1 R01 and R03 applications reviewed in standing study sections and R03 applications reviewed in small mechanism SEPs in score categories (Oct 2005-Jan 2006 council rounds) 60 R01 (n=12,409) 50 40 R03s reviewed in standing 30 study sections (n=1,073) R03s reviewed 20 in small mechanism 10 SEPs (n=209) 0 <151 151-175 176-200 >200 Unscored Conclusions • Review outcomes of R03s are similar to Type 1 R01s • No systematic differences in score distributions for R03s reviewed in different review venues • Shows robustness of peer review – No differences in outcomes despite differences in application characteristics, PI characteristics, and low overall numbers of R03s – Reviewers following guidelines; SRAs keeping reviewers focused on mechanism Acknowledgements •Teresa Lindquist, Program Analyst, CSR and OER data analysts •Andre Premen, Assistant Director, Division of Receipt and Referral, CSR •Elliot Postow, Director, Division of Biological Basis of Disease, CSR Additional slides Where are R03s reviewed? • Depends on Institute, substantive topic, and the PA to which an application responds. Parent R03 CSR: AG, AI, EB, ES, HG, NR, NS, and AIDS- (PA 06-180) related applications CSR/IC: AA, DA, MH (depending on science) IC: DE, HD, LM PARs/RFAs Usually reviewed in the institute unless special arrangements are made with CSR Other current Depends on PA FOAs Note: EY, MD currently do not accept R03s. Review criteria same as for R01s, with some guidance provided. ―The NIH R03 small grant is a mechanism for supporting discrete, well- defined projects that realistically can be completed in two years and that require limited levels of funding. Because the research plan is restricted to 10 pages, an R03 grant application will not have the same level of detail or extensive discussion found in an R01 application. Accordingly, reviewers should evaluate the conceptual framework and general approach to the problem, placing less emphasis on methodological details and certain indicators traditionally used in evaluating the scientific merit of R01 applications including supportive preliminary data. Appropriate justification for the proposed work can be provided through literature citations, data from other sources, or from investigator-generated data. Preliminary data are not required, particularly in applications proposing pilot or feasibility studies.‖ (PA 06-180) *Few exceptions such as PA 05-072& 05-073 Majority of CSR-reviewed R03 applications come in response to PA 03-108 (recently replaced by PA 06-180). Percent of CSR-reviewed application by mechanism (October 2005-January 2006 Council rounds) 2.6% PA 03-108 3.2% Other PA 3.7% 11.0% Fogarty (02-057/PAR05- 072/PAR05-073) PAR 16.1% 63.4% No PA referenced PAS R03 applications have other unique characteristics • More likely to have human subjects 70 60 50 40 R01 30 R03 20 10 0 % with human subjects (CSR-reviewed applications only, fiscal year 2006)