DRAFT GCA RESPONSE TO DEST LETTER

KEY ELEMENTS OF STANDARD GDS, CEQ AND PREQ QUESTIONNAIRES AND PRINCIPLE ASPECTS OF THE STANDARD RECOMMENDED METHODOLOGY Introduction................................................................................................................................... 1 Key Elements of standard national AGS instruments.................................................................. 2 Principle Aspects of the Standard Recommended Methodology................................................. 4 Other Implications for Institutions................................................................................................. 6 Introduction On 5 July 2007, a letter (with the subject line ‘Data Requirements for the Learning and Teaching Performance Fund’) was sent to Vice-Chancellors by the Department of Education, Science and Training. The letter advised that “only data that has been collected via the standard GDS and CEQ questionnaires, using GCA’s standard recommended methodology, will be eligible for inclusion in the [Learning and Teaching Performance] fund’s assessment process”. A later email dated 24 July 2007 was sent to clarify some issues raised in that letter. The intention of this document is to provide a summary of the definition of a standard GDS+CEQ or GDS+PREQ questionnaire and to outline the principle aspects of the GCA / AV-CC’s Standard Recommended Methodology (SRM) for the Australian Graduate Survey (AGS). It should be noted that: • The SRM to be used for the collection of the 2008 survey data is the current methodology (i.e. the document issued jointly by GCA and the AV-CC (as it was at that time) dated May 2005 – attached). Some issues covered in the SRM have been updated via GCA’s annual Australian Graduate Survey (AGS) Manual. The 2007 AGS Manual is attached for information and this will be updated in August to reflect these guidelines. GCA will expand the existing Method Assurance form that is currently signed by institutions at the time of data submission into an AGS Institutional Method Checklist. This checklist will assist institutions to ensure that their administration of the AGS aligns with the SRM and will provide GCA with information about the processes adopted by each institution in respect of the conduct of the AGS. o The AGS Institutional Method Checklist will need to be ‘signed-off’ by a person within the institution with the appropriate level of authority (the designated Senior Survey Contact or higher). o Once completed and returned to GCA, these checklists will be compiled, summarised and published. They will serve to advise the sector of the ways in which each institution, and the sector as a whole, has undertaken the AGS collection process. • Graduate Careers Australia, August 2007. Key Elements of standard national AGS instruments The standard GCA AGS questionnaire has paper, online (oAGS, oCEQ, oPREQ) and telephone interview versions (the telephone script is currently being finalised), and was developed and released as a result of the 2005 GDS Enhancement Project. The standard GCA national AGS questionnaire is defined as follows: • An AGS hard-copy survey questionnaire designed by GCA where the order, wording and layout of the questions are uniform for all institutions. The printing of this questionnaire can be undertaken by either GCA or institutions as set out below: o Any questionnaires to be returned to GCA for any level of processing (e.g. coding or scanning) must be either printed by GCA, or printed by institutions in strict accordance with GCA-provided printing specifications regarding paper size, quality, scale and colour. Universities will be charged any additional costs incurred if they elect to print their own forms and these are not subsequently scannable by GCA. Institutions that plan to process their own questionnaires (i.e. all coding and data file preparation work is undertaken by the institution) may either obtain printed copies of their respective GCA questionnaires via GCA or obtain electronic versions of their respective questionnaires and arrange to have the forms printed themselves. If the latter, the final printed forms must still adhere to GCA colour and paper size specifications. o Institutions may wish to consider the section of this document titled “Other Implications for Institutions” related to costs of printing when considering their options in this regard. • • The GCA online versions of the survey questionnaire - the oAGS, oCEQ and oPREQ The GCA-scripted telephone version of the GDS (to be released shortly) Points to Note: • Current GCA methodology (updated via the 2007 AGS Manual) specifies that CEQ and PREQ responses collected by telephone will not be used in any GCA analyses, but can be included in institutions’ data files. Such cases must be flagged using the relevant variables (refer to ‘AGS 2007 Data File Specifications 250707.xls’ for details of the relevant variables). Institutions can use such responses in their internal analyses. Permissible variations within the standard GCA national AGS questionnaires are: • • • Use of the individual institutional logo at the appropriate place. The selection of optional CEQ scales. However, CEQ questions should be limited to one page of the survey questionnaire, (including the qualitative comments) and the selection of scales should reflect this requirement. The use of the four optional variables (other1, other2, other3 and other4) in the coding process. These four variables have an open format. Institutions can use these to code variables of their choice, such as the respondent’s degree, course or faculty, campus etc. Graduate Careers Australia, August 2007. • • • The inclusion or otherwise of the ‘your contact details’ section on the final page of the questionnaire. The inclusion of the request for a long-term contact email address on the questionnaire. (this is only optional on the hard-copy form – it is included as standard on the oAGS at this time) The use of the words ‘course’ or ‘program’ as applicable to the institution concerned. (this is only optional on the hard-copy form – ‘course’ is used as standard on the oAGS at this time). Variations from practice which will be regarded as ‘non-standard’ • • • The use of non-GCA survey instruments, i.e. survey questionnaires which are not designed by GCA, or non-GCA online survey instruments. The inclusion of additional questions on the AGS survey questionnaires or distribution of other questionnaires with AGS survey questionnaires. Such other questionnaires should be administered by institutions in separate surveys. Deletion of questions from the questionnaire. This includes the deletion of known demographic information, even if certain fields can be populated from student records. If such information is available, it should be used only as a checking mechanism against the responses submitted by students. Any other additions to, or deletions of, any other elements to/from the questionnaires. • Graduate Careers Australia, August 2007. Principle Aspects of the Standard Recommended Methodology The existing Standard Recommended Methodology contains both prescriptive and non-prescriptive elements. The AGS Institutional Method Checklist will cover principle aspects of the SRM. These are summarised below for your information. • Your survey population must be as defined on page 7 of the SRM, being “the group of people who had completed requirements for any higher education qualification in the previous calendar year, including overseas or international students”. Institutions should refer to the 2007 AGS Manual for further detail. o Specifically included in the survey population are: Graduates with bachelor degrees, three-year diplomas, graduate or postgraduate diplomas, graduate certificates or higher degrees Students who have completed a pass bachelor degree but are continuing with an additional year of study in an honours course. Graduates completing two-year associate degrees or diplomas, or 2-year undergraduate certificate courses Students completing and graduating from the first or second half of a combined or double degree Specifically excluded from the survey population are: Off-shore students (people who studied at an off-shore campus). o • Any institution that has students completing requirements for a higher education qualification in the first half of a calendar year should conduct two survey rounds. Students who complete the requirements in the first half of a calendar year should receive a questionnaire in the October round, while those who complete requirements in the second half are surveyed as part of the following April round. Graduates with a postgraduate research degree (a PhD or a masters degree by research) should receive a GDS+PREQ survey questionnaire. All other graduates should receive a GDS+CEQ questionnaire. Coursework masters graduates who had a research component should get the GDS+CEQ questionnaire. Imputed responses are allowed in line with the current SRM guidelines. However, please note that the draft new AGS model proposes the cessation of such response imputation. Survey questionnaires must be coded using the relevant AGS Manual and related documents. Universities capturing their own data must ensure that the file returned to GCA conforms to the format outlined in the AGS Manual and related documents. Universities returning hard-copy survey questionnaires or data files to GCA for processing must ensure that they are delivered by the due dates given in the relevant AGS Manual. The AGS should be conducted within the ethical guidelines laid out in the National Statement on Ethical Guidelines in Research Involving Humans. • • • • • • Graduate Careers Australia, August 2007. Areas where variations in institutional practice are acceptable: • • Optional variables: whether (and how) the institution will use the optional code variables ‘other1’ to ‘other4’. Distribution and reminders: all students should receive at least one initial distribution of survey material. GCA recommends at least two subsequent distributions or reminders. The timing and mode of such distributions are currently at the discretion of the institution. The use of incentives. The existing SRM stipulates that “while these can sometimes be helpful in raising response rates, their use should be considered with regard to research ethics”. Coding and data preparation: institutions may undertake these tasks internally if desired, as long as such work is done in accordance with the AGS Manual. Response rates: response rate issues are covered in the GCA / AV-CC Code of Practice (attached), which stipulates: “a total institutional response rate of at least 70% is desirable and achievable for the GDS and the CEQ. Any data which are disclosed publicly should be accompanied by information on the number surveyed and the response rate. Any GDS or CEQ survey data with an overall institutional response rate below 50% should not be disclosed publicly”. • • • Graduate Careers Australia, August 2007. Other Implications for Institutions Costs GCA anticipates a significant increase in the number of institutions requiring GCA to produce/print/distribute/process their survey questionnaires. The overall impact on GCA in relation to the costs of these activities and the resourcing required will not be known until order forms for the October 2007 survey round are returned to GCA. As a result, and in the interests of fairness and equity for all institutions, GCA will be passing these costs back to institutions on a ‘cost-recovery’ basis. Indicative costs are as follows: • • • Production and setup costs: $ 150 per institution (pre GST). Distribution: as incurred for each distribution location. Printing: o o GDS/CEQ questionnaires: GDS/PREQ questionnaires: 12¢ - 20¢ per questionnaire 60¢ - $1.00 per questionnaire These printing cost variations depend on the number of institutions requiring GCA hardcopy questionnaires and on the total number of questionnaires to be printed by GCA. Institutions may therefore need to take these costs into account when determining how to produce their hard-copy questionnaires. Please contact GCA if you require additional information or clarification related to these charges. Any institutional costs associated with GCA processing hardcopy questionnaires will determined at a later point. These again may be subject to demand. Cindy Tilbrook Executive Director 6 August 2007 Attachments SRM May 2005 2007 AGS Manual and related coding documents Code of Practice Graduate Careers Australia, August 2007.