4070 Page 1 of 9 FOREST SERVICE MANUAL ROCKY MOUNTAIN RESEARCH STATION (RMRS) FORT COLLINS, CO FSM 4000 - RESEARCH AND DEVELOPMENT CHAPTER 4070 - RESEARCH PROGRAM FORMULATION AND DOCUMENTATION Supplement No.: 4000-2008-1 Effective Date: June 2, 2008 Duration: This supplement is effective until superseded or removed. Approved: GEORGE S. FOSTER Date Approved: 06/02/2008 Station Director Posting Instructions: Supplements are numbered consecutively by Title and calendar year. Print on blue paper and post by document name. Retain this transmittal as the first page of this document. This supplement replaces all RM and INT text to Title 4000, Chapter 4070. The last supplements to this title were RM No. 38 and INT 4000-90-2. New Document(s): 4070 9 Pages Superseded Document(s) RM SUPP No. 38 (April 1987) 8 Pages by Issuance Number and INT 4000-90-2 (September 6, 1990) 2 Pages Effective Date Digest of Changes: 4070.2 – Incorporates the Station’s Quality Assurance (QA) Plan (May 25, 2007) into the directives system and describes the objectives of the QA Plan. 4070.3 – Outlines the Station’s QA policy and implements two forms: RMRS-4000-1, ―Study Plan Review, Approval, and Distribution,‖ and RMRS-1600-1, ―Manuscript Approval Certification.‖ 4070.4 – Defines QA responsibilities for various RMRS employees and staffs, and delegates Program Managers as QA Responsible Officials (4070.42, 4070.42a, 4070.42b, and 4070.44). 4072 – Provides information about data archiving and resources necessary to implement the Station’s QA policy. Directs scientists to prepare and maintain study plans within the mission and problem areas of Program Charters, and includes information about required content and the review process (4072.1, 4072.3, 4072.32, and 4072.33). RMRS SUPPLEMENT: 4000-2008-1 4070 EFFECTIVE DATE: 06/02/2008 Page 2 of 9 DURATION: Effective until superseded or removed FSM 4000 – RESEARCH AND DEVELOPMENT CHAPTER 4070 – RESEARCH PROGRAM FORMULATION AND DOCUMENTATION 4070.2 - Objectives The Quality Assurance (QA) Plan for the Rocky Mountain Research Station (RMRS) emphasizes flexibility and reflects the diverse activities of this complex Station. Our goal is a concise, informative, and realistic plan to promote quality and to facilitate implementation of QA procedures and to avoid creating hurdles that might delay science delivery. The Plan builds upon the substantial foundation that already exists for high quality research at the Station. Quality is an integral part of any research organization and protects our investment in science. Achieving quality is the responsibility of every individual at every level of the organization, and in research done collaboratively or with extramural cooperators. In 1995, the policy document ―Quality Assurance in Forest Service Research and Development‖ established the following goal: ―…to ensure that all research data collected, synthesized, utilized, and reported by or for the Forest Service are scientifically sound, of known quality, and thoroughly documented.‖ Our findings must withstand the most rigorous scientific critique when used as the basis for policy and decisions that affect forest and rangeland health, and public values. In 1996, the Forest Service Research and Development (FSR&D) Executive Team approved desired future conditions upon recommendation by the Quality Assurance Advisory Team. These conditions are included in these major groups: 1) The Stations, FPL, and IITF have current and functioning QA Management Plans; 2) Studies conducted as part of FSR&D have current, written, reviewed, and approved study plans; and 3) Data collected as part of FSR&D studies are archived with the appropriate descriptive information. This Station supplement fulfills these desired conditions and serves as a guide for employees, collaborators, and cooperators to ensure quality research results. The Washington Office FSM 4070, Research Program Formulation and Documentation, is the ultimate source for regulations on procedure. FSM 1630, Publishing and Related Activities, also presents important QA information for our research program. Our goal is to integrate quality assurance activities seamlessly into the research process for maximum efficiency. By doing this, we are better serving our customers and ourselves. DEFINITIONS: Quality Assurance (QA) is a process to produce research data and results with known precision, completeness, representativeness, comparability, and where appropriate, accuracy. QA encompasses the planning, design, and implementation of research studies, as well as analysis and reporting of data. Quality Control (QC) is the routine application of prescribed field or laboratory procedures (e.g., periodic calibration, instrument maintenance) to reduce random and systematic errors and to ensure that data are analyzed, interpreted, synthesized, communicated, and used within known and acceptable performance limits. QC also involves use of qualified personnel, appropriate laboratory practices, and adherence to recommended operating procedures. RMRS SUPPLEMENT: 4000-2008-1 4070 EFFECTIVE DATE: 06/02/2008 Page 3 of 9 DURATION: Effective until superseded or removed FSM 4000 – RESEARCH AND DEVELOPMENT CHAPTER 4070 – RESEARCH PROGRAM FORMULATION AND DOCUMENTATION 4070.3 - Policy Station Policy is that Quality Assurance (QA) shall be integral to all research processes, both internal and external. Cooperators must agree to accept QA procedures that are required of Station scientists. The Station’s QA Plan emphasizes flexibility, giving latitude to the QA Responsible Officials for determining appropriate levels of review. Program Managers are ultimately responsible for implementing this Quality Assurance Plan and serve as Station QA Responsible Officials. Documentation of consistent high quality methods and research products has become even more important with the Data Quality Act—our methods, results, and credibility are continually challenged in the courts by those trying to stop management practices that are based on our studies, by other research organizations that are competing for limited research funding, and by the public seeking detailed knowledge of our research. Policy regarding stewardship and archiving of Station research information was established in 1994. This QA Plan thus prescribes archiving at the Program level, and also offers consistency and management Station-wide by access to a Station Archive. Continued emphasis on QA will make us more competitive and credible. This Station Policy and associated implementation procedures adhere to the General Guidelines for the Quality of Information (http://www.ocio.usda.gov/qi_guide/index.html) established by the USDA. Often, scientists are placed in new and challenging roles with significant impacts on time devoted to research. Scientist involvement has the positive benefit of incorporating the latest knowledge more directly into decisions but may be subject to careful scrutiny, especially with the implementation of the Data Quality Act in 2003. This further supports the need for strict adherence to Station and national Research & Development (R&D) quality assurance policy. Quality Assurance affects all aspects of our work and contributes to goals by documenting and validating scientific processes and outcomes. QA is a component of all basic and applied research programs, technology transfer, and other research related activities in RMRS. Emphasis is placed on use of study plans, technical reviews, and archiving. QA is equally applicable to both internal RMRS research activities and to extramural activities where RMRS scientists collaborate or cooperate with other researchers and organizations. This plan covers development of all products by RMRS employees, regardless of funding source. Products may be in the form of internal or external publications, web-based models or publications, presentations, and other outlets of knowledge and technology transfer. When RMRS employees are co- authors, regardless of order of authorship, products must adhere to the RMRS QA Plan unless another FSR&D Station is taking the lead in QA reviews for publication at that Station. RMRS Forms: Study plans ~ review and approval shall be documented on form RMRS-4000-1, ―Study Plan Review, Approval, and Distribution‖ (http://fsweb.rmrs.fs.fed.us/forms/rmrs_4000_1.doc). Manuscripts and other research products ~ review and approval shall be documented on form RMRS-1600-1, ―Manuscript Approval Certification‖ (http://fsweb.rmrs.fs.fed.us/forms/rmrs_1600_1.doc). RMRS SUPPLEMENT: 4000-2008-1 4070 EFFECTIVE DATE: 06/02/2008 Page 4 of 9 DURATION: Effective until superseded or removed FSM 4000 – RESEARCH AND DEVELOPMENT CHAPTER 4070 – RESEARCH PROGRAM FORMULATION AND DOCUMENTATION 4070.4 - Responsibility 4070.42 - Stations Quality assurance will be implemented through the current infrastructure; no permanent committee will be formed. All levels of the organization shall participate in QA activities so RMRS research products meet high standards of quality. Supervisors must ensure that all personnel understand their QA responsibilities in meeting the Agency’s goal. The Station Leadership Team shall provide leadership and advocacy for QA among employees and external partners. Scientists shall apply QA to work through technical support and through collaborators and cooperators. Monitoring is critical to successful QA management. QA Responsible Officials must enforce and maintain records of compliance with the QA Plan. The Deputy Station Director shall hold QA Responsible Officials accountable during performance appraisals, Business Management Reviews, and at other times such as when Freedom of Information Act or Data Quality Act requests are received. The Station Statistician shall summarize for the Director’s Executive Team the number of research documents (study plans, research proposals, and manuscripts) submitted for review to the Statistics Unit each year. Employees are held accountable through the Performance Management System (FSH 6109.13); performance elements should reflect QA responsibilities. 4070.42a - Station Directors The Director’s Executive Team (DET) is responsible for the Quality Assurance Plan for the Station and provides leadership and advocacy among all employees. The DET shall assign QA responsibilities and ensure that they are understood, set performance expectations for implementation, and monitor and assess documentation of compliance. The DET must allocate resources to facilitate quality research, to implement the Quality Assurance Plan, and to inform employees. 4070.42b - Deputy Station Directors The Deputy Station Director has overall responsibility for this Quality Assurance Plan in the Rocky Mountain Research Station. The Deputy Station Director is responsible for research QA issues, and performance standards should reflect QA responsibilities and oversight. 4070.44 - Program Managers Program Managers serving as Station QA Responsible Officials shall implement this Quality Assurance Plan. They must advocate QA to employees and external partners for all research products. The QA Responsible Official must 1) require study plans for all studies, and 2) determine appropriate levels of review required for study plans and research products to assure quality, 3) document QA activities such as review of study plans and manuscripts, and 4) establish and maintain a Program data archive (or utilize the Station Archive). RMRS SUPPLEMENT: 4000-2008-1 4070 EFFECTIVE DATE: 06/02/2008 Page 5 of 9 DURATION: Effective until superseded or removed FSM 4000 – RESEARCH AND DEVELOPMENT CHAPTER 4070 – RESEARCH PROGRAM FORMULATION AND DOCUMENTATION The QA Responsible Official has considerable latitude in determining the level of reviews required. Programs develop problem areas within their charters. These problems are studied through rigorous problem analyses and use of study plans that have been reviewed for statistical and technical validity. Authors must respond to reviewers’ comments in writing. QA Responsible Officials must assess response of authors to peer reviews and assure that statistical review has been completed and addressed. If a Program is terminated, either the Station Archive or another Program shall be designated to maintain the terminated Program’s archive. Other Individuals with QA Responsibilities: Scientists: Scientists are responsible for data quality, documentation, and interpretation. As authors, they are responsible for preparing factually accurate, readable, concise, and timely reports that can be produced as publications. As stewards of research information, scientists must maintain updated data and study files until they are turned over to the Program archive or the Station Archive upon completion of the study. Scientists are responsible for monitoring data collection for precision and accuracy and for preparing documentation. They must provide training opportunities to employees responsible for data collection and for equipment maintenance and calibration. Guidance on Good Laboratory Practices can be found in FSH 4090.13. Other guidance on quality control procedures can be found on the Internet or at various federal agencies (e.g., http://fia.fs.fed.us/library/fact-sheets/data-collections/QA.pdf). Scientists are responsible for developing agreements with external research partners that ensure that Station QA procedures will be followed and documented. Scientists must arrange for appropriate peer and statistical reviews of study plans and research products and document responses to reviewers’ concerns and recommendations. All study plans and manuscripts with any interpretation of data must be submitted for review by the RMRS Statistics Unit regardless of authors’ expertise in statistical analysis. However, certain study plans or publications not based on data interpretation may forego statistical review, e.g. scholarly or theoretical studies, literature reviews and syntheses, and reviews. The QA Responsible Official for each Program determines the scope of review appropriate in each situation. Technical Staff: Technical and professional employees must understand research concepts and objectives in Study Plans along with QA and QC procedures to be used. Employees must seek and complete training necessary to carry out duties. They must maintain current and complete quality control documentation, and they must properly maintain and calibrate equipment. Employees must assist in developing and implementing Standard Operating Procedures (SOP). Statistics Unit: The Station Statistics Unit provides statistical expertise for all phases of research and support for the Station Archive, and is available to advise Station scientists on experimental and sampling design, data analysis and interpretation, and data archiving. Unit statisticians may also assist in conducting statistical analyses and they review study plans, grant proposals, and research products for appropriateness of statistical methods and inference from analyses. They also provide RMRS SUPPLEMENT: 4000-2008-1 4070 EFFECTIVE DATE: 06/02/2008 Page 6 of 9 DURATION: Effective until superseded or removed FSM 4000 – RESEARCH AND DEVELOPMENT CHAPTER 4070 – RESEARCH PROGRAM FORMULATION AND DOCUMENTATION administrative and technical support for SAS statistical software. RMRS personnel, cooperators, and collaborators may consult with Unit statisticians as well as other statisticians they may have access to, but only Unit statisticians can perform statistical reviews required by the Station QA Plan. Communications and Marketing Services Staffs: Publishing Services ensures high quality editorial review of approved manuscripts and other research products and facilitates their publication. Publications are made available online and are archived in Treesearch. Guidance is provided through developing and disseminating publication procedures to scientists and Leadership. The Library facilitates dissemination of research products. The Public Affairs Office facilitates dissemination of information to the public, other agencies and departments, and to cooperators, ensuring that appropriate employees become involved. Grants and Agreements Staffs: The Grants and Agreements staff provides expertise to Station scientists and cooperators to ensure that QA is addressed in extramural grants and agreements. 4072 - RESEARCH AND DEVELOPMENT PROGRAM DOCUMENTATION Information about data archiving and the resources necessary for implementation of the Station’s QA policy are included in Exhibit 01 at the end of this supplement. 4072.1 - Research Work Unit Descriptions and Program Charters Study Plans are written within the bounds of the mission and problem areas of the Program Charter. Mission statements clearly declare the objectives and rationale for the work to be undertaken. Program reviews conducted periodically are important for involving stakeholders in recommending research needs and in documenting satisfaction of clients and research partners with our research quality. 4072.3 - Study Plans Scientists shall prepare and maintain study plans for all individual studies (including administrative studies) approved by the Program Manager. Study plans should be concise and include the information described in http://fsweb.rmrs.fs.fed.us/statistics/studyplan/RMRSstudyplan_outline.htm. Training for Data Collection Personnel: Employees who collect data must possess technical skills necessary for making quality measurements. Study plans describe the level of experience or training necessary for data collection. The scientist shall document any training in the study and describe the level of experience or training received. Equipment Maintenance and Calibration: Scientists and data collection personnel shall ensure that all field and laboratory measurements and recording devices perform at the level that meets measurement quality objectives described in the study plan or study file. Qualified persons who may be Forest Service employees, manufacturer representatives, contractors, or independent data collection personnel will complete maintenance and calibration. RMRS SUPPLEMENT: 4000-2008-1 4070 EFFECTIVE DATE: 06/02/2008 Page 7 of 9 DURATION: Effective until superseded or removed FSM 4000 – RESEARCH AND DEVELOPMENT CHAPTER 4070 – RESEARCH PROGRAM FORMULATION AND DOCUMENTATION 4072.32 - Content Study plans document the study, describing QA procedures, study design, analytical procedures, peer and statistical review procedures, and other information. They require the scientist to clearly define objectives and methods and to carefully plan details of the studies. Changes to study plans after initiation of the study must be carefully documented and must become part of the original study plan. Major changes must be reviewed and approved. Study plans may be in the form of FSR&D study plans, plans for extramural cooperative research agreements, or research grant applications submitted to outside funding organizations. Contents should include the items described in the WO FSM 4070 and in http://fsweb.rmrs.fs.fed.us/statistics/studyplan/RMRSstudyplan_outline.htm. 4072.33 - Review Technical reviews are the cornerstone of the QA process. Appropriate peer and statistical reviews are critical for evaluating study plans and research products. Reviews should address clarity of stated objectives, formulation of testable hypotheses, sufficient statistical power in sampling and design, validity of statistical methods, and other issues. Study Plans (FSM 4072.3) and problem analysis (FSM 4072.2) define the research problem, identify existing knowledge and knowledge gaps, and specify the information required for solving the problem. Scientists must arrange for appropriate peer and statistical review of study plans, publications, and other research products as determined by the Station QA Responsible Official. Statistical reviews must be provided by the Station Statistics Unit. Detailed guidance for review and approval of publications is available in FSM 1631.15 and in the RMRS author’s guidelines (http://www.fs.fed.us/rm/publications/authors_corner/). RMRS Publishing Services also edits and processes journal articles and assists with editing, proofreading, and other services. The Problem Analysis requirement may be waived for mid-career and senior scientists at the discretion of the QA Responsible Official. Data quality is a measure of the usefulness of data to address research questions, and is a function of how carefully and precisely the measurement and sampling methods are implemented. Methods can span a broad range from quantitative repetitive laboratory analyses to qualitative text analyses, with the level of quality assurance depending on the objectives and desired outcome. Data quality assessment applies to activities where equipment or human performance may influence data accuracy or precision. For each activity, data quality can be assessed in performance reviews or audits. Such audits are independent verifications of the results of documenting the effect of equipment, procedure, or human error on data quality. Examples are use of laboratory standards, use of duplicate or blind samples, independent laboratory verification of analyses, calibration of equipment and sensors against standards, and data entry verification. Good Laboratory Practices (FSH 4090.13) and other references should be used where appropriate to ensure data quality. RMRS SUPPLEMENT: 4000-2008-1 4070 EFFECTIVE DATE: 06/02/2008 Page 8 of 9 DURATION: Effective until superseded or removed FSM 4000 – RESEARCH AND DEVELOPMENT CHAPTER 4070 – RESEARCH PROGRAM FORMULATION AND DOCUMENTATION 4072 - Exhibit 01 Data Archiving Archiving of research information facilitates long-term research, supports published results, and maintains a historical record. Requests for information must be submitted through the Station Archivist who will work with Programs as needed. Confidentiality and privacy concerns shall be addressed by the Station Archivist and the Programs. Study files serve as a permanent record of an individual study and include information needed to reconstruct the study or to ensure the usefulness and validity of the data. Manuscript approval forms and reconciliation of reviewers’ comments should be included in the study files. Study files should be retained as long as desired by the scientists in each Program, then should be submitted to the Station Archive. The Station Archive (http://www.fs.fed.us/rm/data_archive/) is an electronic warehouse for any RMRS-funded research data products. Each data product contains electronic data along with complete electronic documentation. Data products may also contain other electronic information deemed important to the study such as: study plans, manuscripts (along with reviewer comments and author response), and maps. Descriptions of available archiving assistance and the procedure for submitting data products to the Station Archive are found at: http://fsweb.rmrs.fs.fed.us/statistics/archiving/. A Program Archive is a warehouse for research data products developed within a Program, which is maintained by individual scientists or Program Managers. These archives should contain data along with complete documentation, both electronic if possible. These archives should also include any other information deemed important to the study, preferably electronic, such as: study plans, manuscripts (along with reviewer comments and author response), maps, and physical specimens. Research Information: Information that a scientist deems important to a study should be archived. This may include maps, charts, photographs, notes, the study plan, publications, and reports. Research information developed as part of Station research activities or in the National Forest planning process will be managed as part of the Station Archive or in the Program archives. Research information that originates from outside sources and is used with little modification would not be included in the Station Archive or the Program archives. Observational or historical research information not restricted to an active study should be archived. Examples include stream flow data, weather observations, and long-term growth measurements. Individual scientists are responsible for maintaining their own study files as long as they desire to. At any time when a Program decides to discontinue storage of any electronic portion of its archive, it should be transferred to the Station Archive. Samples: Voucher specimens of physical samples such as plant, insect, soil, or chemical samples may be preserved and deposited in a reference collection for future validation of identification, and qualitative and quantitative measurements, at the discretion of the Program. RMRS SUPPLEMENT: 4000-2008-1 4070 EFFECTIVE DATE: 06/02/2008 Page 9 of 9 DURATION: Effective until superseded or removed FSM 4000 – RESEARCH AND DEVELOPMENT CHAPTER 4070 – RESEARCH PROGRAM FORMULATION AND DOCUMENTATION 4072 - Exhibit 01 (continued) Data Documentation: Archived data are usable only if thoroughly documented. The study plan under which data are developed is a primary source of documentation and should be included in the archives. Documentation standards for various data types can be found at the following web site: http://fsweb.rmrs.fs.fed.us/statistics/archiving/. Resources Necessary for Implementation Statistical Expertise: The Station recognizes the importance to scientists of having access to statistical and QA expertise. The scientist, statistician, and QA Responsible Official shall determine the appropriate level of collaboration with the Statistics Unit. Quality Assurance and Quality Control Expertise: QA and QC implementation procedures vary at different levels of a research organization. Tools must be readily available to employees. Expertise may be developed or enhanced in many ways, including: 1) in-house training by employees knowledgeable or experienced in QA/QC activities; 2) formal training by contract consultants; 3) Internet websites; 4) software packages with QA and/or QC applications; and 5) documentation available from many federal agencies (e.g., DOE, EPA) and USDA programs (e.g., NCLAN, NAPAP, EMAP, SAFIS). Product Editing: Publishing Services provides many important services to Station employees. These include editing, proofreading, artwork, creation of figures, layout and design, and multimedia products. Taking advantage of these professional services saves the authors much time and effort. Training: Successful implementation of the QA Plan requires that all participants have the knowledge and skills necessary to play their roles and meet responsibilities. Upon adoption or change, the RMRS Quality Assurance Management Plan shall be presented to all employees. QA policies and procedures shall be included in new employee orientation. Scientists and data collection personnel should obtain training related to laboratory, greenhouse, and field QA techniques, statistical and analytical methods related to QA, data archiving procedures, and data standards. Data Archiving: Maintaining Program archives is necessary for ensuring that important research information is preserved and accessible for potential long-term needs. The Station Archivist position functions within the Statistics Unit. The Station Archivist develops and maintains the Station-level archive. Programs will have the option to transfer responsibility for Program archives to the Station Archive at any time.