RQF Trial 2005 Guidelines FINAL by gegeshandong




                                       JULY 2005

1.       Participating Institutions

The proposed RQF Trial is being conducted under the auspices of the Australian
Technology Network of universities:

        Curtin University of Technology
        RMIT University
        Queensland University of Technology
        University of South Australia
        University of Technology, Sydney

Murdoch University has also agreed to participate in the RQF Trial.

2.       Objectives of the RQF Trial

The objectives of the RQF Trial are:

        To obtain valuable information and data on research quality across key
         research groups in participating universities;
        To explore methods for assessment of research impact appropriate to the
         mission and objectives of the participating universities;
        To develop and test a robust and efficient mechanism for the assessment of
         research quality and impact across the participating universities and one which
         is informed by relevant national and international approaches;
        To assist research groups in meeting the challenges of the RQF which is
         planned for implementation nationally towards the end of 2007.

Given the limited available resources across the participating universities for the
critical review and assessment of submissions it has been agreed that a maximum of
100 researchers per university will be included in all research clusters submitted by
each university. Thus some submissions received by each institution may not be
subsequently subjected to external assessment. It is envisaged that each participating
university will instigate an internal selection process based on local strategic and
developmental considerations to determine those research clusters submitted for
assessment. Each participating university is responsible for ensuring that feedback is
provided to each research cluster that submits a portfolio regardless of whether the
submission is subjected to external review.

3.       Timetable for the RQF Trial

The RQF Trial will be conducted during the period July to October 2005. The
following key dates should be noted:

Key Dates                     Description
Friday July 15, 2005          Guidelines released to participating universities
Friday August 26, 2005        Final date for receipt of submissions from selected
                              research clusters in participating universities (in
                              accordance with these guidelines)
Early September 2005          Submissions subject to internal review by participating
                              universities. Review of selected submissions by PVC(R)
Friday October 14, 2005       Final date for receipt of evaluations from Assessment
Mid November 2005             Feedback provided to research clusters in participating
                              universities including summary and comparative data as

4.        Assessment Panels

The participating universities have agreed to establish a small number of Assessment
Panels based on current ARC and NHMRC discipline clusters (and utilising proposed
RAE2008 assessment panel information):

     Panel      Broad Discipline Coverage
      A         Mathematics, Information and Communications Sciences
       B        Biological Sciences and Biotechnology
      C         Social, Behavioural and Economic Sciences
      D         Humanities and the Creative Arts
       E        Engineering and Environmental Sciences
       F        Physics, Chemistry and Geoscience
      G         Clinical Studies - Cardiovascular Medicine, Cancer Studies, Infection
                and Immunology, Neuroscience and Clinical Psychology
      H         Public Health – Epidemiology, Health Services, Primary Care
      I         Allied Health Professions - Nursing, Allied Health Studies, Pharmacy

5.        Guidelines for Submissions to the RQF Trial

The RQF Trial has been designed to generate valuable information and data based on
currently available national and international approaches. It has particularly been
designed to avoid costly overheads in the collection of data, development of
submissions and in the assessment of quality and impact portfolios. These guidelines
are definitive for the purposes of the trial and must be adhered to in all respects
particularly those relating to page limits and the submission of evidence. For any
additional guidance or advice research clusters should contact the Research Office in
their home institution.

The unit of assessment for this RQF Trial is an institutionally defined research

The period of assessment for this RQF Trial shall be the period from January 2002 to
August 2005 (inclusive).

Each research cluster should contain at least 5 (and preferably 10 or more) FTE of
research active staff (excluding adjunct staff). Research students should not be
included in this count.

Each research cluster must submit a research overview of no more than six A4 pages
(at no less than 10 point font size). Each submission should nominate a Cluster
Leader or up to three Co-Leaders. The overview should contain the following
information regarding the aggregate performance of the research cluster:

      Brief overview/description of the research program(s) within the cluster
       including comments on synergies, themes and interdisciplinary research
       activity. (1 page)
      Strategic direction of the research cluster and relevant developmental context
       including reference to the development of ECRs and future research leaders.
       Information on major national and international collaborations should be
       included. (1 page)
      Summary performance data aggregated for the cluster including total
       research income by source of funding, total publications/output by type and
       year, total HDR student load (EFTSL) by year and type of award, total HDR
       completions by type and year. Citation data and impact factors should be
       included where available. (1 page)
      Analysis of up to three (3) examples of research impact directly attributable
       to the research activities of the cluster where the impact has occurred within
       the period of assessment. Examples provided must include details of the types
       of evidence available to support all claims along with information on the value
       of the impact. See Attachment 1 to these guidelines for more guidance on
       measuring impact and for relevant web sites to secure additional information.
       (3 pages)
      Self-assessment of performance by the research cluster using the rating scales
       provided in these guidelines (additional to the 6 page limit).
      Designation of the relevant assessment panel by which the submission is to
       be primarily evaluated (additional to the 6 page limit). If a submission is to be
       designated as cross-disciplinary then this should be clearly stated along with
       a list of the relevant assessment panels.
      List all members of the research cluster (additional to 6 page limit) by
       category of appointment (academic staff – continuing, academic staff – fixed
       term, research only appointments, postdoctoral researchers, adjunct
       appointments, HDR students etc). All ECRs should be indicated clearly on the
       list of members. For the purposes of this trial researchers can only be
       nominated in one research cluster.

Each submission should also include a two-page research quality profile for each
member of the research cluster (optional for HDR students and adjunct faculty
members). This profile should follow the attached template and must include the
following information:

      Full bibliographic details of best four publications/outputs that have been
       published or have been accepted for publication during the period of
       assessment. Citation data and journal impact factors should be included where
       available and appropriate. Justification of each claim must be included to
       provide assessors with a context in which to judge the ‘quality’ of each

         This should be brief and quantitative (eg impact factor, rejection rates, citation
         data etc) where possible. Total publications/outputs in major DEST
         publications categories over the period of assessment should also be included.
        Summary of research income by DEST category and Grant Scheme source
         pro-rated for joint awards.
        Summary of HDR completions for which the submitting researcher was
         principal supervisor or co-supervisor (for a period of at least one year during
         candidature). Do not include HDR completions for which the researcher was
         an associate supervisor only. Where available include information on current
         appointment of completed students.
        Summary of high level esteem factors including major prizes, key note
         addresses at major international conferences, participation in major national
         committees, Fellowship of a national Academy or similar international
         organisation, significant role in a major national or international review,
         editorship of a major journal, etc. Only high-level esteem factors should be
         included and limited to no more than five items.

6.       Guidelines for Assessment of RQF Trial Submissions

Submissions will be assessed by an Assessment Panel comprising a Chairperson
(senior research manager from a participating university), at least three senior
researchers with relevant experience/background from within the group of
participating universities and at least one external expert able to judge submissions
from a relevant end-user perspective.

Submissions will be assessed for research quality and research impact using the
rating scales below. The descriptors in each case have been developed using
approaches proposed recently by the AVCC, DEST (through the RQF Advanced
Approaches Paper, see p. 20) and the CSIRO who are gratefully acknowledged.

Research Quality Scale

Rating          Descriptor
5               Outstanding performance/pioneering – groundbreaking research that
                is world-renowned. High level of recognition in the international
                research community. Sustained evidence of highly innovative research
                consistently published in the most prestigious journals and presented as
                groundbreaking in major international conferences or exhibitions.
4               International standard – very strong performance, research
                considered truly internationally competitive and making a major
                contribution to the advancement of knowledge. Sustained contribution
                to setting and maintaining new directions within the international
                research community.
3               Exceeding the national standard – strong performance, research of a
                high standard making a solid incremental contribution to knowledge,
                not considered at the leading-edge. Possibly considered a leader but
                only in a developing niche rather than mainstream area of research
2               Average national standard – moderate performance, research of
                average/moderate standard but not at the level of significantly
                advancing knowledge in the field. Not able to set or sustain independent
                research directions other than possibly in a local or regional context.

1             Below national standard – below average performance, research of
              low standard and/or no reasonable level of contribution to knowledge
              demonstrated. Possible declining quality of research output compared
              to other research groups. May have a useful tactical or short term

Research Impact Scale

Rating        Descriptor
5             Outstanding impact directly demonstrated from most research
              activities, research outcomes set the direction or are regularly used for
              commercial, environmental, community or policy development. Strong
              recognition by end-users for the value and impact of research outcomes.
              Strong record of knowledge transfer to end-users at a sophisticated
              level and/or substantial and sustained impact on an academic discipline.
4             Very high impact directly demonstrated from most research activities,
              research outcomes are able to be used and have been used by end-users.
              Record of knowledge transfer at a high level and/or discernable impact
              on an academic discipline.
3             High impact directly or indirectly demonstrated from selected research
              activities, research outcomes are able to be used by end-users to
              enhance relative position or improve delivery to clients/stakeholders
              with some evidence of use in selected or niche cases.
2             Average impact from selected research activities, research outcomes
              that are can be used by end-users to maintain relative position or
              influence delivery to clients/stakeholders. No significant evidence of
              major impact.
1             Low impact or little evidence of impact of research activities, research
              outcomes may not be useable and limited evidence of knowledge
              transfer or end-user engagement.

Note that the Assessment Panels will NOT have access to actual publications or
evidence of impact. All claims must be verifiable and participating universities
reserve the right to audit all claims made within any submission.

7.       Template for Submissions

Research clusters are required to use the templates available on the Grants and
Information Website. Clusters must adhere to the designated page limits in all
sections. Any additional pages will be disregarded.

Once completed, Clusters are to send both a hard copy and an electronic copy, clearly
marked ―RQF Trial 2005‖, to the Grants and Information Office for the attention of
Kellie O‘Toole. The hard copy of the submission should include the signed
certification page. The electronic version of the completed submission is to be sent
to K.O‘Toole@mudoch.edu.au. (It is not necessary to send the certification page

Submissions are to be delivered to the Grants and Information Office nater than
Friday 26 August 2005.


The RQF Trial has been designed to provide information to the participating
universities in relation to possible approaches to the measurement of research impact
complementing the measures of research quality. The guidelines are deliberately
brief in relation to how research clusters should approach the issue of impact as it is
considered vital that a variety of possible approaches are tested through the trial,
particularly those that are relevant to the mission and goals of the participating
universities. One particularly important aspect of this trial is that research quality and
impact will be assessed using different descriptors based on the premise that many
areas of research activity are likely to score differentially and we should seek to refine
the most appropriate robust and efficient approaches at the conclusion of the trial.

It is vital for all research clusters to carefully consider the various approaches to
measuring impact that have been widely canvassed during the present national debate
on RQF. In determining a response to this aspect of the trial, research clusters should
be guided by the need for robustness and evidence in determining possible responses
in support of claims for impact. This will be a very contentious aspect of the RQF but
the ATN and IRUA groups of universities are committed to ensuring that both quality
and impact appear in the final RQF to be implemented in late 2007 across Australia.

In order to assist research clusters in preparing the ‗research impact‘ section of the
RQF trial submission the following information has been extracted from the RQF
Issues Paper released by DEST earlier this year. Tables 1 and 2 provide a broad list of
possible outputs that could be measured in the context of determining ‗research
impact‘. Table 3 provides a possible framework for determining impact that takes
account of different modes of generating and using knowledge. Finally a list of web
sites to access key documents and further information has been provided.

Table 1: Allen Consulting Group’s examples of possible metrics differentiated
for two broad discipline groups

 Discipline grouping           Quality indicators relating to impact           Knowledge diffusion indicators relating to
Physical and biological   • the number of highly cited papers published;     • presentations given at industry sponsored
sciences                  • the number of papers in high quality journals;       conferences;
                          • the number of highly cited patents;              • presentations given at academic conferences;
                          • success rates in securing competitively          • presentations given at learned societies;
                              allocated research grants;                     • distribution of research newsletters to industry
                          • invitations received to act on government            stakeholders;
                              advisory boards;                               • meetings attended with potential industry
                          • the value of research funding donations from         adopters of research;
                              industry; and                                  • results from surveys of government and
                          • the value of research funding contracts from         industry research managers as to who they
                              industry.                                          regard as ‗high impact‘ academic researchers;
                                                                             • the number of research students that are
                                                                                 subsequently employed within industry.
Social sciences and       • the number of books published with high          • presentations given at learned societies;
humanities                    quality publishers;                            • presentations given at academic conferences;
                          • the number of highly cited papers published;     • submissions made to government inquiries;
                          • the number of papers in high quality journals;   • citations of research in government policy
                          • success rates in securing competitively              publications;
                              allocated research grants;                     • citations of research in court judgements;
                                                                             • articles published in the popular press;
                          • invitations received to act on government        • research cited in articles in the popular press;

                                 advisory boards;                                • results from surveys of heads of policy sections
                             • the value of research contracts received from         in government departments as to whom they
                                 industry and government; and                        regard as ‗high impact‘ academic researchers;
                             • the number of citations of papers within papers       and
                                 published in high quality journals.             • the number of research students that are
                                                                                     subsequently employed within government
                                                                                     departments, Ministerial offices and industry.
Source: Allen Consulting Group (2005), Measuring the Impact of Publicly Funded Research, DEST: Canberra,
        pp. 41-43.

Table 2: Examples of possible research impact outputs

    Publication outputs
    Editorship of international journals
    Involvement in international learned societies
    Invited international lectures/other international collaborations
    Peer recognition: Academy membership
    Research income: Competitive Grants Schemes
    Research income: Other Grants Schemes (incl. International)
    Other research income (other contract research)
    International linkages/collaborations/Memoranda of Understanding
    Visiting scholars and/or postgraduate students
    Publication citations
    Incorporation of research results into international/national policies, codes and/or practices
    Research graduates employed in industry
    Industry-funded research places
    Academic-Industry Staff exchanges
    Research students industry placements
    The holding of exhibitions and performances
    Audience/attendances at exhibitions/performances
    Media presence through articles, debates, coverage
    Expert advice/submissions/panel membership at government inquiries
    Patents
    Commercial licences
    Commercial uptake
    Spin-off companies
    Consultancies
    Other?

Source: DEST, 2005.

Table 3: Example of Possible Frameworks for Measuring Impact

Element                   Description                                           Possible Metrics
    Knowledge            Universities and research organisations                   Communication activities
     Diffusion            generating useful economic and social outcomes            Capacity building activities
                          via encouraging the broad industry-wide                   Extension and education activities
                          adoption of research findings through                     Standard setting activities
                          communication, building capacity within                   Industry output data
                          industry through extension, education and
                          training, creating standards relating to
                          production and distribution.
   Knowledge             Universities and research organisations                   Academic publication activities
    Production            generating useful economic and social outcomes            Patenting and licensing activities
                          by selling or licensing the results of research in        Income streams relating to the above
                          the form of commodified knowledge—directly                Spin-off company formation activities
                          exploiting ‗knowledge products‘ embedded in
                          intellectual property and other explicitly codified
                          formats. This is a ‗standard‘ model of research
   Knowledge             Universities and research organisations                   Contract research and consultancy
    Relationships         generating useful economic outcomes by                     activities
                          providing services that indirectly exploit broad          Income streams
                          IP platforms consisting of trade secrets, know-           Staff and students working on
                          how and other forms of tacit knowledge. This               interchange with industry
                          approach centres on cooperation, collaboration,           Industry research staff with sessional
                          joint ventures and partnerships.                           and adjunct appointments in universities
                                                                                    University appointed ‗visitors‘ from
   Knowledge             Universities and research organisations                   Participation in non-academic
    Engagement            generating useful economic outcomes as a by-               community and economic activities
                          product of shared interests and concerns that             Jointly owned and operated technology
                          transcend the boundaries of the university per se.         property infrastructure—technology and
                                                                                     research parks, buildings, equipment,
                                                                                     instruments, etc
                                                                                    University organised events for
                                                                                     community and regional economic and
                                                                                     social benefit (workshops, seminars, etc)
                                                                                    University facilities available for non-
                                                                                     academic purposes (eg, libraries, cultural
                                                                                     centres, sports grounds)

Source: Howard, J. (2005), The Emerging Business of Knowledge Transfer: Creating Value From Intellectual
        Products and Services, DEST: Canberra [Forthcoming].

RQF Issues Paper
Advanced Approaches Paper
Measuring the Impact of Publicly Funded Research
Web of Knowledge


To top