WHOLESALE AND RETAIL SETA Skills Development for Economic Growth ETQA Monitoring andAuditingPolicy WHOLESALE AND RETAIL SETA ETQA PROVIDER MONITORING AND AUDITING POLICY Effective Date: 00/00/2007 Issued by ETQA Rev: A Pg. 2 of 15 Distribution List ETQA Executive Manager R. Lechet Regional Quality Assurors K. Ntimbela E. Kleingeld E. Olivier Revision History Version Date Author Description 1.0 24/08/2007 K. Ntimbela First Draft Document Information Location Name W&RSETA ETQA : Monitoring and Auditing Policy Status 1st Draft / Review Last Changed 2003 Approval History Date Name Designation Approved by: Executive Manager: ETQA Date of Issue: 24/08/2007 Date of Next Review: 24/08/2008 PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 3 of 15 CONTENTS PAGES 1. STATEMENT OF POLICY INTENT 1.1. Legal / Regulatory and Policy Reference 1.2 Definitions 1.3 Application and Scope 1.4 Relationship to other ETQA policies 1.5 Accountability and Responsibility 2. STATEMENT OF POLICY INTENT 2.1 Purpose and content 2.2 Principles 2.3 General Monitoring and Audit requirements 2.4 Monitoring specific requirements 2.5 Monitoring process 2.5.1 Annual Monitoring Plan 2.5.2 Desktop Review and or Evaluation 2.5.3 On-Site Monitoring / Verification Visits 2.5.4 Monitoring Reports 2.5.5 Findings 2.5.6 Decision on Review 2.5.7 Reporting on Monitoring 2.6 Frequency of Monitoring Intervention 2.7 Auditing 2.7.1 Selection of audit team and audit team leader 2.7.2 Qualification of auditors 2.7.3 Initiation of an audit 2.7.4 Audit report and Auditee Response 2.7.5 Action on audit reports 2.7.6 Conflict of interest 2.7.7 Confidentiality of audit reports 2.7.8 Appeal of Audit Committee Action 2.7.9 Annual Report and Plan 2.7.10 Self-Audits 6.8 Frequency of Audit interventions PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 4 of 15 1. Background 1.1. Legal / Regulatory and Policy Reference Subsection 2(1) of the ETQA Regulations, 1998 (No. R 1127 of 1998), promulgated under section 14 of the South African Qualification Authority Act, 1995 (Act No. 58 of 1995) provides that an ETQA “shall be accredited for the purpose of monitoring and auditing achievements in terms of national standards or qualifications”. This legal mandate is further supported by the NSDS, in particular principles 4 and 5: • Support, monitor and evaluate the delivery and quality assurance necessary for the implementation of the NSDS • Advance the culture of excellence in skills development and life long learning And objectives 2 and 5: • Promote and accelerate quality training for all in the workplace • Improve the quality and relevance of provision W&RSETA ETQA therefore acknowledges its legal right and duty to monitor and audit providers as per the South African Qualifications Authority. In this way W&RSETA ETQA fulfill the quality assurance responsibility of learning provision in the sector. Summary of legal and policy reference documents • Constitution of the Republic of South Africa (Act No 108 of 1996) • Promotion of Administrative Justice Act (Act No. 3 of 2000) • South African Qualifications Authority Act (Act No. 58 of 1995) • Education and Training Quality Assurance Bodies Regulations (No. R 1127 of 1998) • SAQA 2001 Quality Management Systems for Education and Training Providers http://www.saqa.org.za/>Documents>Criteria and Guidelines>All Documents • SAQA 2001 Criteria and Guidelines for Providers http://www.saqa.org.za/>Documents>Criteria and Guidelines>All Documents • SAQA 2002. Criteria and Guidelines for the Registration of Assessors http://www.saqa.org.za/>Documents>Criteria and Guidelines>All Documents PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 5 of 15 1.2 Definitions The definitions of the terms below apply to their use in this policy: o "accreditation" means the certification, usually for a particular period of time, of a person, a body or an institution as having the capacity to fulfill a particular function in the quality assurance system set up by the South African Qualifications Authority in terms of the SAQA Act o “accreditation scope” means the list of qualification(s) and/or unit standard(s) for which a body is accredited for a defined purpose o "assessor" means the person who is registered by the relevant Education and Training Quality Assurance Body in accordance with criteria established for this purpose by a Standards Generating Body to measure the achievement of specified National Qualifications Framework standards or qualifications, and "constituent assessor" has a corresponding meaning o "Education and Training Quality Assurance Body" (ETQA) means a body accredited in terms of section 5(1)(a)(ii) of the SAQA Act, responsible for monitoring and auditing achievements in terms of national standards or qualifications, and to which specific functions relating to the monitoring and auditing of national standards or qualifications have been assigned in terms of section 5(1)(b)(i) of the SAQA Act o "moderation" means the process which ensures that assessment of the outcomes described in National Qualifications Framework standards or qualifications is fair, valid and reliable o “moderator” means someone who is competent to conduct a moderation process o "provider" means a body that delivers learning programmes which culminate in specified National Qualifications Framework standards or qualifications and manages the assessment thereof o "quality assurance" means the process of ensuring that the degree of excellence specified is achieved o "registered standards" means unit standards or qualifications registered on the National Qualifications Framework o “registration scope” means the list of registered standards for which an assessor or moderator is registered with the SETA as an assessor or moderator o “Audit” systematic, independent and document process for obtaining audit evidence and evaluating it objectively to determine the extent to which the audit criteria are fulfilled. o “Audit conclusion” outcome of an audit, provided by the audit team after consideration of the audit objectives and all audit findings. o “Audit evidence” means records, statement of facts or other information, which are relevant to the audit criteria and verifiable. o “Audit findings” means the non-compliant results that have been identified as a result of the evaluation and collection of evidence against audit criteria. Findings are also known as non-conformities PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 6 of 15 o “Monitoring / audit plan” means a description of the activities and arrangements for an audit o “Audit scope” means an extent and boundaries of an audit. o “Audit team” means one or more auditors conducting an audit supported if need by technical experts. o “Auditee” means an organization / company being audited o “Auditor” means a person with the competence to conduct an audit. o Competence” means demonstrated personal attributes and demonstrated ability to apply knowledge and skills o “Process/ Procedure” means a procedure or process describes the steps necessary to complete a process. It may be a text document, a flow chart or any other format that is fit for purpose. The terms” process” and” procedure” are used interchangeably in this guideline o Review” activity undertaken to determine the suitability, adequacy and effectiveness of the subject matter to achieve established objectives o “Corrective action” means an action taken to eliminate the course of a detected no-conformity or other undesirable situation. Corrective action is reacting to a situation to avoid a recurrence. o “Document” information relating to a policy, procedure or work instruction, and its supporting medium. This may include checklists and blank forms. Note that the medium may be paper, magnetic, electronic or optical computer discs, master samples, photographs, or combination thereof. o “Evidence” means data supporting the existence or verify something. Data gathering as part of the assessment process is referred to as evidence of learning competence o “Non-conformity” means non-fulfillment of a requirement. This is also known as findings o Preventive action” means an action taken to eliminate the cause of a potential non-conformity or other potential undesirable situation. Preventive action is pro- actively taken to avoid an occurrence. o Records” means a document stating results achieved or providing evidence of activities performed. Note the records may be stored as any of the document mediums. PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 7 of 15 1.3 Application and Scope The policy applies to all practices and processes conducted by SETA staff, constituent providers, and verifiers and moderating bodies ensuring the monitoring and auditing activities within W&RSETA sector. 1.4 Relationship to other ETQA policies Accreditation Policy Assessor and Moderator Registration Policy Assessment and Verification Policy Programme validation Policy Certification Policy 1.5 Accountability and Responsibility The ETQA staff and or its representatives are responsible and accountable for managing and overseeing the implementation, maintenance, review and improvement of the W&RSETA validation policy and procedures. The responsibility for monitoring and evaluating quality of learning provision rests with the regional Quality Assurer, whilst auditing of W&RSETA ETQA providers` lies with the ETQA Audit Committee under the leadership of the ETQA Executive Manager. The ETQA Audit Committee’s authority includes arranging for audits of W&RSETA ETQA accredited providers, conducting and or overseeing of the audits, and transmitting audit findings and recommendations to the ETQA Executive Manager. The ETQA Executive Manager’s authority includes determining which W&RSETA ETQA providers shall be audited, reviewing audit reports and recommendations received from the ETQA Audit Committee confirming adherence to the criteria for accreditation, and confirming that the procedures and practices of accredited providers continue to be consistent with current accreditation requirements. The ETQA Executive Manager is also responsible for taking any necessary action based on its audit findings. Further, appeals related to such an audit are made to the ETQA Executive Manager. PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 8 of 15 2. Statement of Policy Intent 2.1 Purpose and content The purpose of this policy is to: Provide ETQA monitoring and auditing framework to exercise its function as contemplated in section 1 above here in this policy document. Inform providers of the ETQA approach in monitoring and auditing practice and process of provider learning provision. Clearly outline systems that assure quality of learning provisions verify quality of assessments by internal and external providers verify learner achievement and recognition of prior learning 2.2 Principles W&RSETA ETQA has identified the following principles which will underpin the monitoring, evaluation and review of the quality of learning provision and assessment: a) W&RSETA ETQA will adopt a reliable and comprehensive system of monitoring that will ensure the quality and credibility of programmes leading to achievement of unit standards and or qualifications b) The primary engine of improvement will be providers’ own systems for monitoring and evaluation of programmes and learning provisions. c) Monitoring processes will be open to scrutiny by stakeholders d) Procedures for monitoring will be consistent with the functions set out for ETQA in the SAQA Act 1995 and appropriate for each of the diverse range of providers within the wholesale and retail sector. e) Monitoring will be designed to encourage and facilitate ongoing Improvement in the quality of programmes and services offered to learners f) Compliance to accreditation requirements g) Identify options for development and continuous quality improvements Adherence to procedural fairness as required by the Promotion of Justice Act (Act No. 3 of 2003) in terms of: reasonable notice (nature and purpose), a chance to make representations, a clear statement of the administrative action, advise as to review or internal appeal, and Advise as to the right to request reasons. 2.3 General Monitoring and Audit requirements PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 9 of 15 • The W&RSETA ETQA will ensure quality of provision through monitoring & auditing of providers within its scope of provision as set out in the rest of this document. • The W&RSETA ETQA will ensure the credibility of learner achievements through its monitoring of providers, validated programmes and the attainment of national standards. • The W&RSETA ETQA recognizes provider’s quality assurance as the main engine of quality improvement and will monitor its effectiveness in maintaining and improving the quality of programmes. • An annual monitoring plan will be produced identifying priorities for monitoring in terms of provider types, validated programmes and learning achievements. • The application of monitoring resources will be focused and will maximize the use of available information sources, particularly reports arising from provider self evaluation (internal verification mechanisms), authentication of assessment and learner attainment of national standards. • In order to assure itself of the quality of programmes, W&RSETA ETQA may also visit training delivery and or assessment sites to monitor current practice. • The information generated by monitoring processes will be used by W&RSETA ETQA to recognize and disseminate good practice and to recommend quality improvements where necessary. 2.4 Monitoring specific requirements The W&RSETA ETQA will monitor accredited providers by collecting and disseminating qualitative and quantitative information about provider’s quality of learning provisions on an ongoing basis for the purpose of assuring and improving the quality of learning in the sector. The W&RSETA ETQA will conduct monitoring for assuring learners, stakeholders and others of the quality of provision in the sector and assist providers in implementing their own quality systems. 2.5 Monitoring Process The W&RSETA ETQA`s monitoring process will include some or all as appropriate of the following stages: 2.5.1 Annual Monitoring Plan Given the size and complexity of the further education and training sector, monitoring the activity within needs to be done in a systematic and planned fashion which complements providers’ own quality assurance systems and makes best use of available resources. The W&RSETA ETQA will devise an Annual Monitoring Plan to ensure comprehensive coverage of the accredited providers in this sector. It will ensure that the effectiveness of providers’ quality assurance systems and the outcomes and quality of validated programmes are monitored. It will also allow new trends or issues arising identified by desk monitoring to be further investigated as appropriate. PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 10 of 15 The Monitoring Plan will identify priorities for the year in terms of: a) Provider Quality Assurance systems b) Validated Programmes or learning programmes requiring specific attention c) Provider capacity to manage assessments and moderation thereof, and d) Quality of learner achievement of national standards, i.e. consistency of outcomes across programmes of different providers leading to the same qualifications. 2.5.2 Desktop Review and or Evaluation This is monitoring carried out off-site through analysis of documented evidence of the provider’s strategic, operational and ETD practitioners are submitted for review and evaluated against set criteria within the ETQA Desk monitoring and or evaluation will identify and or validate provider legal status, quality assurance, self evaluation, programmes and approve ETD practitioners required to be registered. It will also identify if and when site visits is it appropriate to conduct such visits for the purpose of gathering more information for accreditation or ascertain further assurance required for the accreditation of providers. All of this information will be required to be at the exposure of the Regional ETQA for detailed analysis. It will liaise with providers in respect of any issues arising pertaining to their quality assurance. 2.5.3 On-Site Monitoring / Verification Visits The regional ETQA`s will conduct visits of providers on-site in accordance with the Annual Monitoring Plan. Evidence submitted for desk-top review is validated against actual practice as this is the first visit within a period of 6 - 12 months after provider accreditation. It is important to note that on-site monitoring and or verification will focus on the following aspects: a) focus on quality assurance system review, b) programme validation, c) management of assessment and moderation, d) validation of assessment design and instruments, e) learner Results / achievements and uploads on SMS f) Provider Self Evaluation Reports and Improvement Plans g) Issues Arising When monitoring a provider with multiple sites, the W&RSETA ETQA will select a sample of delivery and assessment site to visit. However, prior to a visit, the W&RSETA ETQA on-site verification visit will be arranged in consultation with the provider and sites concerned. A primary source of preparatory PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 11 of 15 material will be the provider’s own self evaluation report(s) and or development plans, where available. During a site visit, the W&RSETA ETQA will gather information through a combination of activities including meetings with management, staff and learners, checking of records and evidence, observation of facilities and resources. 2.5.4 Monitoring Reports W&RSETA ETQA will evaluate information gathered through the various monitoring activities and produce a report on the provider / programme(s) monitored. Referencing criteria established and the report will list examples of good practice and recommendations for improvement identified by the monitor(s). The report will be drafted by the ETQA member involved, using a template provided by W&RSETA ETQA, and sent to the provider(s) concerned for comment. The provider may request amendments to any factual inaccuracies and may also have other comments appended to the report. The report will then be sent to W&RSETA ETQA and will be reviewed to ensure consistency of its recommendations, a part of the overall quality assurance of the process. The finalized report will be made available to the provider concerned. 2.5.5 Findings The W&RSETA ETQA will require the provider concerned to implement any or all of the recommendations identified in a monitoring report. The provider’s implementation of the recommendations will become one of the foci of subsequent monitoring. Reports of improvements / initiatives arising from the W&RSETA ETQA on-site monitoring visits in response to W&RSETA ETQA recommendations will have to be adhered to within agreed specified timeframes. 2.5.6 Decision on Review The SAQA Act mandates the review, in consultation with a provider, of programme validation and of the effectiveness of a provider’s quality assurance system. In each case, monitoring will form the basis for the review and any subsequent decisions by the ETQA. PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 12 of 15 Following consultation with a provider, a monitoring report on a provider’s quality assurance of programmes and assessment may recommend one of the following: a) Continuation of the existing provider’s quality assurance system. b) Review mechanisms of the provider’s quality assurance system. c) Withdrawal by ETQA of the quality assurance system and a requirement of the provider to resubmit its quality assurance system for evaluation. This recommendation would only be made following sustained monitoring activity evidencing poor quality assurance with continuing failure by the provider to address issues raised. A decision on withdrawal will only be made by ETQA representative and will be subject to appeal by the provider, ultimately to the Audit Committee. Following consultation with a provider, a monitoring report on a provider’s validated programme may recommend one of the following: a) continuation of the programme validation b) Withdrawal by ETQA representative of the validation. This recommendation would only be made following sustained monitoring activity evidencing that the programme is, for whatever reason, no longer valid and continuing failure by the provider to address issues raised. A decision on withdrawal will only be made by ETQA representative and will be subject to appeal by the provider, ultimately to the Audit Committee. Following consultation with a provider, a monitoring report on a provider’s practitioners may recommend one or all of the following: a) Continuation of the existing registered assessors and moderators. b) Renewal by ETQA representative of the practitioner’s scope of assessment and moderation for re-registration for another three years and a requirement of the practitioners to resubmit application for evaluation. c) Withdrawal / de-registraion by ETQA representative of the assessor and or moderator of the provider. This recommendation would only be made following sustained monitoring activity evidencing poor quality assurance with continuing failure by the provider to address issues raised. A decision on withdrawal will only be made by ETQA representative and will be subject to appeal by the provider, ultimately to the ETQA Committee. PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 13 of 15 2.5.7 Report on Monitoring The ETQA representative is required by this policy to report to the ETQA Committee in ‘such form and manner as the ETQA Committee sees fit, on the effectiveness of providers’ quality assurance. It is proposed that ETQA Representative will, with the agreement of the ETQA Committee, devise and publish an annual monitoring report on the extent and outcomes of its monitoring activities. 2.6 Frequency of Monitoring Interventions The frequency of Monitoring / Validation interventions is scheduled every 6 – 12 months, at ETQA discretion and on the basis of complaints and appeals received. 2.7 Auditing 2.7.1 Selection of audit team and audit team leader The ETQA Executive Manager shall appoint an audit team and audit team leader for each audit of a selected accredited provider (the “auditee”). Audit teams will consist of one, two, or three ETQA representatives, selected from an organized pool of available and qualified people. 2.7.2 Qualification of auditors ETQA representatives selected to serve as auditors shall have the following qualifications: a) experience in, and knowledge of, the requirements for W&RSETA ETQA accreditation, including SAQA criteria and guidelines for training provider accreditation; b) general knowledge of auditing principles and methods obtained through any combination of experience, education, or Quality Auditing training; c) the ability to act objectively and independently; and d) the ability to analyze information and to express findings clearly, concisely, and in a timely manner. 2.7.3 Initiation of an audit The ETQA Executive Manager shall advise the proposed auditee of the identities of the audit team members. The auditee shall advise ETQA Executive Manager within ten (10) working days of any concerns regarding conflict of interest of the assigned audit team members, and the auditee shall be advised of the ETQA Executive Manager’s response within ten (10) working days of the receipt of such concerns. The audit team leader shall attempt to schedule the audit to be conducted within thirty (30) working days from the appointment of the audit team. The audit team leader shall provide the auditee with a list of particulars that the audit team intends to verify or examine during the audit and any other information that the audit team believes will help the auditee prepare for the audit. PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 14 of 15 2.7.4 Audit report and Auditee Response The completed audit report shall be forwarded by the audit team leader to the ETQA Executive Manager within thirty (30) working days after the completion of the audit. The ETQA Executive Manager / Lead Auditor / Representative shall transmit a copy of the report to the auditee with a request that the auditee submit any comments to ETQA Executive Manager / Lead Auditor / Representative within thirty (30) working days of receipt of the report. The auditee comments may include plans and a timetable for corrective action relating to any recommendations contained in the audit report. 2.7.5 Action on audit reports The ETQA Executive Manager shall review the audit material transmitted to it by the Lead Auditor, take appropriate action, and notify the auditee of its action. The action taken may include a finding that the conditions upon which accreditation was granted has been satisfactorily maintained. If the action taken includes a finding that the conditions upon which accreditation was granted has not been satisfactorily maintained, the auditee shall be requested to take defined corrective action within three months period. 2.7.6 Conflict of interest a) Audit Team: The ETQA Executive Manager shall not appoint auditors to an audit team who have a known conflict of interest that may affect their ability to perform an unbiased audit. Appointed auditors shall notify the ETQA Executive Manager of any real or apparent conflict of interest as soon as practicable after notification of their appointment. b) W&RSETA ETQA / Reviewing Body: A member (or his/her employer) of a reviewing body having a conflict of interest with the auditee shall not be allowed to receive or review a copy of the audit report and will not be allowed to participate in the discussion of the audit unless otherwise agreed to by the auditee. The auditee shall be provided with a list of names and affiliations of the members of any reviewing bodies prior to their review of the audit information, and be given the opportunity to advise ETQA Executive Manager of a potential conflict of interest on the part of one or more members of a reviewing body. If the auditee or other relevant party asserts that it believes that a member of the Audit Committee reviewing body has a conflict of interest, that auditee or party is required to state the reason(s) for its belief. That information shall then be forwarded to the member of the reviewing body identified as having a possible conflict for that person’s response. If that member disagrees with the assertion, then the Chairman of the Audit Committee (ETQA Executive Manager) reviewing body shall make a final determination as to whether a conflict of interest exists. PROVIDER MONITORING AND AUDITING POLICY ETQA DEPARTMENT Doc ID: Rev.: A Pg. 15 of 15 2.7.7 Confidentiality of Audit Reports All audit information and audit reports shall remain confidential and shall not be disclosed to any person other than the auditee, appropriate ETQA staff, the auditors, and, as appropriate, members of the reviewing bodies. The auditee may provide the audit information and reports received to whomever it deems appropriate. 2.7.8 Appeal of Audit Committee Action Final action by the Audit Committee may be appealed to the ETQA Executive Manger in accordance with the appeals procedures. 2.7.9 Annual Report and Plan The Audit Committee member shall prepare an annual report summarizing the audit activities for the year. The report shall be transmitted to the ETQA Executive Manager and to the ETQA Committee by December 1 of each year. 2.7.10 Self-audits The W&RSETA ETQA accredited providers may find it useful to conduct self-audits. The purpose of self-audits is to permit an accredited provider to identify areas where improvement and increased efficiency are possible. They also provide an opportunity to identify any areas where policies or procedures may not be in conformance with the W&RSETA Accreditation requirements. The results of any self-audit are required to be submitted to the ETQA for review and advise for continuous improvement actions. 6.8 Frequency of Auditing Interventions The frequency of Audits is conducted for re-accreditation before the five year accreditation period has elapsed. However, the ETQA Executive Manager may use his or her discretion to conduct provider audits or on the basis of discrepancies, complaints and appeals.