Writing an Excellent Project Plan by vuQT1oz

VIEWS: 13 PAGES: 82

									Midwest Area OSQR Training

• 8:30 - 12:00 Presentations by Associate
               Area Director & Program
               Analyst & Statistician
• 12:00 - 1:00 Lunch
• 1:00 - 3:00 Writing & Critiquing of
               Project Plan
• 3:00         Adjourn
Why OSQR Review?
                 It’s not our fault!



     1998 Farm Bill
    ARS research peer-
     reviewed every 5
     years
    Most review panelists
     external to ARS
    Satisfactory review
     before beginning
     research
Importance to ARS
Congressional intent was to enhance
 research.
OMB Analyses. ARS must report statistics for
 review quarterly. OMB is looking for
 average score improvement…
Provides for Agency Accountability
  …especially to Congress
Enhances ARS Image

          Improving peer review success
    could mean more support for ARS research
 What is OSQR Review?

A Dialogue
And an…
   External Review
   Scientific Review
   Prospective Review
   Peer Review
   Competitive Review
Review Panels
Panelists are your colleagues.
  They read your peer-reviewed papers.

Panelists are active scientists.
  Most are academics (per the Farm Bill).

Panelist often know your work.
  And are often familiar with your excellence.

Panelists take their task very seriously.
  The devote many hours to each review.

     They don’t want to give low scores!
How a Panel is Selected

OSQR Receives suggestions/nominations
 from NPS, Areas, others.
All potential chairs are screened for COIs.
SQRO Interviews potential candidates.
SQRO selects and invites Chairs.
Chairs work with OSQR to develop a
 balanced, proposed list of panel members.
SQRO reviews and approves final list of
 panelists.
Writing an Excellent Project
           Plan
   Explaining your research to
            everyone
      What is a Project Plan?

• Presentation of your science to a group of
  peers for an evaluation of quality
• How is quality defined?
  – This is where the differences among panels
    often occurs because they view a plan
    through their own expectations and not a
    quantitative scale
• Therefore
              Project Plans

• Should
  – Be a comprehensive description of an
    approach to address a problem
  – Showcase your capabilities as a project team
  – Provide confidence to the panel that this
    project will advance and impact science
  – Build the case for the research you want to
    do!
 Your Plan is a Marketing Tool
    …and the reviewers are your customers
By page 5 reviewers should know:
     The subject of your research
     Why it is important
     What it will produce
     How you are going to get there

The rest of the document will “flesh out” this but if
  the reviewers don’t know it by page 5, they
  probably won’t get it easily from the rest of the
  document.
                                    this means…
• The project summary should present the overall
  thread of your work, the major gaps being
  addressed, how you will address them, and what
  you will produce…in no more than a dozen
  sentences.

• The Objectives should be followed by one to two
  paragraphs that explain how they are integrated
  and how they address the overall goals of the
  work. A chart illustrating this and showing where
  individuals on the project team fit is most
  helpful.
    What is an Excellent Plan?
Doesn’t just extend past work
Is important and significant
Breaks new ground
Pushes the frontiers of knowledge
Builds on new knowledge
Displays original ideas
Risks breakthroughs

Presented in a logical, clear, easy-to-follow, form.
Don’t “tack” varying bits from different authors together…write
a consistent, smooth, narrative from those pieces.
Get to the SCIENCE in the plan, that is what
 the panel is looking for in their reading
Why do Project Plans Receive Low
            Scores?
    What do Reviewers Want?
Reviewers need to see a coherent story…
     What are the problems you are researching?
     Why are they important?
     What do you hypothesize?
      (or why a hypothesis is not appropriate)
     How will you test your hypothesis?
     Why are you the best group to do this?
Don’t let reviewers get another perspective…
     “My reputation should be enough”
     “This process isn’t important”
     “Who are these reviewers anyway!”
     “What’s the minimum I can do to pass?”
     “I’m not really taking this process seriously”
      What Causes the Problems
•   Lack of clarity in project plan
•   Lack of integration in the overall plan
•   Lack of details in the experimental plan
•   Lack of explanation of the role of project team
    members
•   Lack of explanation of the knowledge gaps and
    how project will fill them
•   Lack of confidence in the ability of the project
    team to accomplish the objectives
Where are the problems?
                             (Science, Data, Writing)



           Science            Missing Information                 Writing or Organization


          3.00
                 The problem is poor
          2.50   or inadequate writing!

          2.00
Score/n




          1.50    Little difference

          1.00

          0.50

          0.00
                    No         Minor      Moderate     Major           Not
                 Rev ision    Rev ision   Rev ision   Rev ision      Feasible

                                       Action Class
Don’t hide
your
details and
expect the
panel to
guess your
path!
Reviewers want to know…

               What is the problem?
                Why is it important?
          Where are you going with it?
       How are you going to get there?
 And how will you know you have arrived?

      Don’t make them hunt for this!
 Poor organization can wear their patience…and reduce your score.
Action Class Ratings
                        No Revision
       Excellent project. No changes or additions are required.

                     Minor Revision
           Approach sound. Some minor changes required

                  Moderate Revision
Some change to an approach required but project is generally feasible.

                      Major revision
   Requires significant revision. Major gaps in plan or information.

                        Not feasible
    Major flaws, omissions. Unfeasible or not possible to assess.
              Comments



• A sampling of comments from previous
 panel reviews provides enlightenment
 about how panels look at these plans.
         Comment Sampling
                 “No Revision”


…studies will fill rather substantial knowledge gaps.
         …well written and comprehensive
             …well written project plan
     …well organized, cogently prioritized and
         comprehensive research plan…
       Comment Sampling
            “Minor Revision”

…studies will fill rather substantial knowledge
                       gaps.
      …well written and comprehensive
          …well written project plan
  …well organized, cogently prioritized and
         comprehensive research plan…
 …well written, well organized, and easy to
                      follow.
      Comment Sampling
        “Moderate Revision”
           …one of the better written
       …excellent discussion of technology
…large research project plan but poorly organized.
  …concerns that some of the objectives can be
              completed by this team.
   …work and ability described as "adequate"
          …missing current information
…hypotheses and plan well conceived, approaches
                    appropriate
          Comment Sampling
                “Major Revision”
…a heroic course that…will not achieve the stated objectives
         …serious flaws in experimental designs…
  General Lack of focus. Plan is so broad that it leaves out
    important details…not clear data will be relevant or
                       interpretable.
            …lacks cohesion and clear direction
    …writing style, quality, and organization significantly
                           detract…
  Short on details. Not well planned. Need to take process
                         seriously.
           Comment Sampling
                  “Not Feasible”
 …approaches highly flawed, lack detail, many experiments
                       duplicative
…poorly written…difficult to discern what the authors want to
                              do.
 …proposed research contradicts intention of the proposal.
   …exaggerations, incomplete reviews…, and misleading
   comments, as well as poor grammar and organization.
      …lack of knowledge…incomplete understanding
             …fundamental misunderstanding…
          Work does not address stated objectives.
          Causes of Low Scores
• Presentation of the information, not necessarily
    the scientific questions or the project team
•   Can not see the logical path of science toward
    the goal
•   Not an integrated plan but a collection of parts,
    plan looks and reads like it was assembled by a
    committee but not read for its “flow” once
    assembled.
Attributes of a Good (Great) Plan

• Explains the problem, the current state of
    knowledge, your expertise in the problem
    area, your approach to solving the problem,
    and the products from the research program
•   Convinces the review panel that you are the
    best team to conduct this research.
•   Presents your plan as a narrative from front to
    back
•   Showcases your ability as a project team to
    conduct this research and solve a problem
  These are not competitive
     grants but think about
     preparing this plan to
 demonstrate to ONP and AO
   that your funding for this
  effort is a great investment
and the return you provide will
benefit ARS and the American
              Public
                   How Full Is the Glass?

The Good News:
        80+ percent of plans receive “passing” scores on
        initial review.
        Most of the rest receive “passing” scores after re-
        review.
The Bad News:
       Nearly a fifth of plans do not fare well in review…

        A number of plans fail to adequately address reviewer concerns
        and fail re-review.

      Our intent is to move more up to the Good News.
            Why do we require
          three ad-hoc reviews?
• Area Director: entomologist (4 years away from
    research)
•   Associate Area Director: poultry diseases (7
    years away from research)
•   Assistant Area Director: vacant (multiple
    detailees with diverse research backgrounds)

We are not qualified to pass judgment on the full
   range of scientific plans that we must approve.
It is in the project team’s best interests to get
   plans reviewed before the OSQR panel gets
   involved.
                  Format


• Lori Wilson Voss is your guide and can
  give you a lot of help.
• Do not deviate from any aspect of the
  official OSQR formatting instructions.
• Take Lori’s suggestions and advice
  seriously – she can save you a lot of grief!
                Basic Information
• The PDRAM is the end of a conversation, not the
    beginning of a negotiation, with ONP.
•   These items should match the PDRAM:
    –   Project Title
    –   National Program
    –   SY time assignments
    –   Project Objectives (and sub-objectives in some cases)
• You MUST document ONP concurrence with any
    changes.
Now let’s look at the “nuts and bolts” of a
                   plan...

                Remember
      You are composing a narrative…
         …not completing a form!

  Your plan is not reviewed by bureaucrats.
 Reviewers are active scientists in your field.
    They often know you and your work.

Reviewers devote, on average, 6 to 8 hours
reviewing each plan BEFORE the panel meets.
       Document Outline
Title and Investigator(s)………….page 1
Signature Page……………...........page 2
Table of Contents……….………….page 3
Project summary (250 words)...page 4
Objectives...…………..................page 5
Need for Research (1-2 p)                 15 – 30
Scientific Background (5-7 p)             pages
Approach & Procedures (6-15 p)
Physical & Human Resources (1/3–1/2 p)
Project Mngmt & Eval (1/3–1/2 p)
Milestone Table (1-3 p)
Literature Cited
Past Accomplishments of Project Team Members
Issues of Concern statements
Appendices (letters plus other material)
          Project Summary


• Marketing plan for your research plan that
  explains the research plan and its
  potential impact
• Understandable by the non-technical
  reader
       Project Plan Components
         Project Summary – 250 words
                    The Appetizer!
Write this in active voice.
State the essential problem and why it is important.
What have you done to date (1-2 sentences)?
How will you address the issue?
Why is this important?


    This is where you capture the interest of the
             reviewer. Make it compelling!
               Objectives


• Don’t just state the objectives but explain
  the goals that are being addressed.
• Link the objectives together by using a
  diagram that shows the project team and
  the products (A picture is worth a 1000
  words).
              Objectives

• Must match PDRAM
• Cannot be changed without documented
  concurrence from ONP
• A diagram showing relationships among
  objectives and their outputs may be
  useful.
   Need for Research & Scientific
            Background
• Why is a solution to this problem
  important?
• How does it fit into the NP action plan?
• What are the current knowledge gaps?
• What preliminary data exists from your
  program (graphs and tables help)?
• This section doesn’t have to be long but
  set the stage for this research
     Need for Research & Scientific
         Background (Hints)
• Your literature review doesn’t have to cover all
    of the information known on this topic.
•   Present related projects as more than a list from
    the CRIS search, show the linkages.
•   What the panel wants to see is what gaps exist
    in the current literature and how your project
    plan fills those gaps.
•   Most of the comments state that this is “great”
    literature review but it doesn’t tell us why the
    problem is important.
              Potential Benefits
             Anticipated Products
                  Customers
• Separate but related sections under Need for
    Research
•   Should be logically linked and related to
    Objectives
•   Tell a story that can be related to the ARS
    mission
    – Problem solving
    – Decision support
    – Risk reduction
         What kind of research?
• Understanding research – the basic physical,
    chemical, and/or biological characteristics and
    processes of the system.
•   Predicting research – once you think you
    understand the system, develop ability to predict
    how the system adjusts to changes, stresses,
    etc.
•   Decision-support research – once you get really
    good at predicting, embody that ability into a
    tool or other product that allows the end user to
    solve a problem, support a decision, reduce risk
                    Customers
• Individuals (agricultural producers or processors)
    or organizations directly using ARS developed
    knowledge, technologies, or services
•   Who is the first user of your work? Who cares?
•   Why should they care?
•   Must be plausible
    – Consumers, taxpayers – not specific enough
    – Who will be the first users of the results?
    – Does someone else need to take the information to
      move it along to decision support ? Who?
    – How will next users get the information? What do
      they need to do with it before the ultimate
      beneficiaries?
                 Products

• Tangible outputs of the research
  – Manuscripts, new varieties, vaccines, models,
    datasets, decision-support tools, practices
  – Things that can be used by the “customers”
               Benefits

• Outcomes (not outputs) of your research
• The ultimate impact
• Be specific
• Be plausible
• How will this be realized? (Few farmers or
 consumers read the genomics literature)
         Scientific Background
• Relevant literature
  – Be brief – save the pages for describing what you
    plan to do.
  – Show that you are aware of the latest relevant
    developments in your field
  – A gap analysis, not a comprehensive lit review --
    What information or technology gaps need to be
    filled? Your project will seek to fill them.
  – This section should support your research plan in the
    eyes of the reviewers, NOT be the centerpiece of it.
      Previous Accomplishments
• Major accomplishments for this project up to now.
• The impact of these accomplishments (Who cares, and
    why?)
•   How these accomplishments are the foundation of what
    you are about to propose.
•   Demonstrate why your team is the best group to
    propose this research.
•   Underscore your team’s expertise, skills, background
•   NOT a comprehensive, unfocused, encyclopedia of the
    team’s collective accomplishments.
•   NOT an unselective publication list.
Our MWA Statistician –
Deb Palmquist
• Will say a few words…
          Research that is not
           hypothesis driven

• Some types of engineering research
• Model development
• May still have an aspect that can be
 studied by testing an hypothesis
  – Inclusion of variable X will increase the
    precision of a model by at least 10%.
    Approach and Procedures


• Adequate detail on experimental
  procedures
• Data to be collected
• Approach for the data analysis
• Who is responsible for the conduct of the
  research?
         Project Plan Components
        Approaches & Procedures: 6-12 pages
             How are you going to get there?
Set out your Experimental design.
  Formulate REAL, testable, hypotheses!
Describe approaches and methods any why they are appropriate.

Discuss advantages and limitations (important if methods are “risky”).

Illustrate how objectives can be achieved.
   Who will do what, how, and when (including collaborators and SCAs!)

Describe nature and extent of collaborations, including SCAs
  Letters in Appendix need to confirm what you say!
  For SCAs, a copy of the agreement is sufficient.
Include management, evaluation and contingencies.
  What is your path to success? How will you monitor it?
    Approach and Procedures (Hints)

• If you are using a new technique show that your
    project team has the expertise to use this
    method.
•   If you have a vacancy describe how this SY will
    fit into the project plan and if the objective is
    totally the responsibility of the vacant SY then
    give scope of what this SY will do and have a 6-
    month milestone that their job will be to develop
    an experimental plan.
Approach and Procedures (Hints)

• Link the experiments back to the
  knowledge gaps and conclude by how
  these studies fill these gaps.
• Lay the foundation for the research
  program by showing what you will be
  doing in the first set of experiments.
• Link to the contingency section
             Contingencies

• This is not the whining section!
• Show how you will use the results from
  the initial studies to determine your next
  steps in the project plan.
• If you are using a new technology, state
  how you will evaluate it’s effectiveness
  against other methods.
                 Contingencies
• Choices of research direction as you get results
    – If there are several possible outcomes of a planned
      experiment, how would each affect what I will do
      next?
    – What result would make me re-examine my sub-
      objectives or overall plan? What will I do if that
      happens?
• NOT a place to describe experiments you will do
    if you get that grant you applied for.
•   NOT a place to describe what you will do if hail
    trashes your field plots.
              Milestones

• Use OSQR’s table format
• Be specific as to what and when
• Decision points along the 5 years
• Should be understandable to an interested
  outside party.
• Nobody should have to guess whether you
  achieve a milestone.
            Explicitly Linking
      Contingencies and Milestones
•   Can be an effective planning tool
•   Demonstrates in-depth thinking and planning
•   Milestones are decision points
•   Contingencies are the decisions

Now that I have achieved X (milestone), I can
  choose among these possible directions for the
  research (contingency).
       Good ideas for milestones
• Complete a data base on….
• Determine the accuracy and bounds of
    uncertainty of a model…
•   Complete all experiments for a paper on…
•   Complete the second year of a two-year
    experiment on….
•   Complete all laboratory analyses on field
    samples collected over the previous summer….
•   Deliver a complete data set from resistance trials
    to a breeder who will….
       Bad ideas for milestones
•   Continue studies on….
•   Develop understanding of….
•   Plan a study that…
•   Initiate an experiment on…

Cannot tell what determines success, or represents
  an ongoing activity, not a specific
  accomplishment at a specific point in time.
            More on milestones
• Used to determine if progress is being made on
    achieving the plan’s objectives
•   CAN BE CHANGED during the 5-year life of the
    plan
    – Unexpected good result sends research in a new
      direction
    – Unplanned vacancy or adverse event legitimately
      prevents achieving a milestone
    – Budget does not support the needed research,
      despite honest planning
• A basis for ARMPS packages and discussions
    with the Area Office
             Collaborations
• Don’t merely list the collaborators but
  explain what they “bring” to the project
  plan.
• Use the collaborator letters to provide
  evidence of their involvement.
• If you have a SCA with a group provide a
  copy of the agreement in the appendix
  with a summary in the Collaboration
  section.
        Collaborations (Hints)

• If there is a project within the location or
  area that is within the same NP show how
  these are linked or complement each
  other.
• Don’t overstate collaborations for
  “window-dressing” but show viable
  collaborations.
     Project Plan Cohesiveness
• Coherent, integrated plan conducted by a team?
• A set of independent objectives, each addressed
  by a different member of the team?

Your plan should reflect how all the work will come
  together to accomplish the overall goals and
  objectives of the project.
Conclusions
The reviewers need to see the logical “thread”
  through your work.
Don’t make readers “search” for what you are
  doing!
Be clear, accurate, and correct.
Don’t assume reviewers know you and your
  work…(a poor plan may not be saved even if
  they do!)
  A NONTECHNICAL reader should be able to understand
  the general goals of your research.

  A TECHNICAL reader should be confident you
  understand the problem and can accomplish the work.
        Project Management

• How is the project team going to evaluate
 the progress of the project?

• If you have collaborators how will the
 overall project team assess progress?
         Project Management

• What is the role of each SY and collaborator?

• What kind of technical support is available?

• What highly specialized equipment or facilities is
  needed, and how will they be made available?

Project plans do NOT contain budget information.
  Human & Physical Resources
• Provide the confidence that you have the
  resources needed to conduct the research.
  – This is not the place to ask for more.
• Human Resources
  – Vacancy- explain the vacancy and expected date it
    will be filled.
  – Training- explain any training needed on new
    methods and how this will be obtained (classes, time
    in other laboratories).
• Physical Resources
  – Provide confidence that you have the resources
    required to answer the problems.
           Milestone Table

• Use this table as a summary of your
  project.
• Include the products and milestones that
  you expect to deliver over the life-cycle.
• Remember these are only the starting
 points and will be changed each year
 through the life-cycle as the project
 progresses.
    Previous Accomplishments

• Summary of each investigator in a related
  project objectives
• Pertinent publications to the project
  objectives
• Remember, the review team is probably
 aware of your research, but they still like
 to read about your assessment of your
 accomplishments.
       Supporting Materials

• Letters from collaborators

• Other material that helps support your
 ability to conduct the research
              General Thoughts
• “You have the right to remain silent.”
    – Just because you have a certain page limit doesn’t
      mean you have to “talk” that much.
• Emphasize tight, precise language….like a
    journal manuscript.
•   When page limits start to be a problem
    – Edit ruthlessly for wordiness
    – Emphasize your plans for future work
    – Minimize the past (tighten the literature review onto a
      gap analysis, and limit prior accomplishments to the
      most relevant ones).
      Separating the Good from the
                  Great
• Prepare an integrated narrative that provides
    the detail and instills confidence about the
    project team approach to addressing the
    problem.
•   Shows that the project plan will fill important
    gaps needed for science and the National
    Program Action Plan.
•   Shows that outcomes will address problems.
                   Hints

• Read your plan with a fresh set of eyes
 after you have assembled the plan.
• The plan should read as an integrated
 effort to solve a problem.
• Use the plan as a reflection of the project
  teams ability to conduct problem-solving
  science.
• Proofread your plan!
  What if I have several SYs?
  Everyone has a contribution to make
  but don’t follow this recipe for failure…
       1.      Ask for a portion from each
       2.      Assemble those in order
       3.      Consider the plan done…(it’s just the start!)

Use the information to write a flowing, logical, and consistent narrative.
            Settle scientific differences…then write!
            You may not like a colleague…don’t let the panel know that!
            You may not like some of the work…don’t let the panel see that!

Alternatively, you may wish to write an initial draft and edit in others’
additions…keep the writing consistent throughout.
          Parting Thoughts

• Don’t expect anyone else other than the
  Project Team to add quality to the project
  plan
• Be proud of your research
  accomplishments and your approach to
  address this problem
• Producing a quality plan takes time. Start
  early and seek the input of colleagues.
             Parting Thoughts

• Average ARS CRIS project budget is
    approximately $900,000 to $1,000,000 per year.
•   A five-year project plan represents a $5 million
    investment by the American taxpayers.
•   Your team is being handed $5 million for a five-
    year project – NON-COMPETITIVELY
•   The reviewers are mostly academics.
•   Justify your planning to the reviewers and the
    taxpayers.
How much effort would you
   spend on a competitive
   grant application if you
 knew it was for 5 years and
         $5 million?
                 Just to Reiterate
• This is your product – Sell it to the reviewers
• Scientific Background – Keep it short, show the
   research gaps
• Approaches and Procedures – this is the meat of
  the project. Tie it together, show your expertise!
• Appendix – Remember you use this to put in
  materials or methods.
• Milestones and Expected Outcomes: Don’t
   shortchange this.
• The Three Reviews will really help.
• The MWA Office Program Analyst is your Friend!
What Happens After
Review?
No, Minor or Moderate Revision
 Lead Scientist responds to comments. Scientific Quality Review Officer
 certifies compliance with recommendations.

Major Revision or Not Feasible
 Lead Scientist revises and responds to comments.
 Panel performs a second review assessing response to their comments
 and assigns a new Action Class Score. If still Major or Not Feasible,
 project is returned for administrative action. No further review.

      Projects are reviewed no more than two times

         (There are no page limits for revised plans)
Can I disagree with the panel?
 This is a dialogue
 If you really disagree…put it away for a few days!
 Then…
 Honestly consider panel opinions.
 Be polite but if you disagree say why
 DON’T skip changes to plan
 DON’T insult or impugn panelists
 DO provide justification for your alternative
    view

 Panels are NOT perfect…they are fellow scientists
Last Words

            Proofread
           Seek Review
              then
 proofread and seek more review
             And lastly
  Proofread and Seek Review

								
To top