Presentation Title Will Appear Here

Document Sample
Presentation Title Will Appear Here Powered By Docstoc
					                       Revisiting CB Use of
                             AS9101 Rev D

                                         Tim Lee
                                   12 March 2012

Company Confidential                               1

AS9101 Rev D Fundamentals
• 9101 defines requirements for the
  preparation and execution of the audit
• It defines the content and composition for the
  audit reporting of conformity and process
  effectiveness, based on
  – 9100-series standards requirements
  – the organization’s quality management system
    documentation, and
  – customer/regulatory requirements
• It builds on and mandates requirements and
  guidelines in ISO/IEC 17021 and ISO 19011
AS9101 Rev D:
• CBs have now been using AS9101 Rev D for
 some time and for all 91XX:2009 transition
• We have learned that auditing practices and
 use of 9101 is inconsistent
• We need to improve the effectiveness of CBs
 auditing and audit reporting to
  ‒ Detect and stop nonconforming practices before
    they become normalised
  ‒ Ensure that CB clients and their clients gain benefit
    from the audits and the audit reports
  ‒ Build and retain confidence airworthiness and other
    authorities confidence in the ICOP scheme
Review of CB Audit Reports:

• The IAQG OPMT carried out a review of sample
 reports in October 2011
• ABs have also been reporting findings made as
 a result of witnessed assessments and review
 of CB audit reports
• Following slides provide information on those
 findings and will help prepare you for
 assessments you will carry out
• There will be IAQG OPMT material in future
 that focuses on positive models for

General 9101 Report Findings:
• Established audit objectives are not always in
 line with the example in 9101 instructions e.g.
 ‘Upgrading of certification to 9100:2009’
• Audit Plans and OERs do not always
 demonstrate that an interview with top
 management has taken place
• Annex E audit conclusions are weak in relation
 to overall performance, effectiveness and
 compliance of the QMS of the client
 organisation or in respect of the audit
 objectives often providing stories of the client

General 9101 Report Findings:
• Forms are not being filled accurately from
 9101 instructions
• Stage 1 form is incorrectly populated with
 regard to Aviation, Space and Defence
 business and associated employees
• Forms other than the OER are being modified
• Previous version 9101 forms are being used
 on occasions e.g. for NCRs
• Annex E not being completed until after all of
 the NCRs raised were closed out i.e. completed
 at final situation not situation at closing
General 9101 Report Findings:
• Recommendations being made incorrectly:
  ‒ Recommendation for certification made even though
   NCRs are raised   (i.e. wrong recommendation box ticked)

• Virtually no use of Annex G for complex clients
 with more than one site
• Lack of understanding that a second Annex E is
 required after a visit to close out NCRs
• Decision-making is not finding and resolving
 issues in audit reports
  ‒ No specific requirement for decision-makers to attend
   AATT or similar to understand 91XX:2009 and AS9101
   Rev D requirements on auditors

9101 Report Findings – QMS Matrix:

• The QMS Matrix does not always contain all
 processes of the client
• QMS Matrix does not contain sub-processes or
 valued-added activities
• QMS Matrix completed with X rather than the
 required C or N

9101 Report Findings - PEARs:
• Processes that contain clause 7 of 91XX:2009
 requirements do not have an associated PEAR
 in all cases
• PEAR forms not always contain a description of
 the process and interactions that is sufficient
 to understand the client’s process (or meet
• Clauses recorded on the PEAR are not always
 consistent with QMS matrix or listed as ‘see
 QMS Matrix’
• PEAR process measures do not always contain
9101 Report Findings – PEARs:
• PEAR process measures accepted are not
 always consistent or applicable to the control
 of the process
• Actual process performance is not always
• Improvement actions against performance
 below target do not in all cases result in
 process achieving targets but process
 described as ‘Level 3’
• Improvement actions are not described just
 recorded as ‘in place’

9101 Report Findings – PEARs:
• Several processes effectiveness measures
 described but actual performance data not
 reported against all them
• Process effectiveness not always recorded on
 the PEAR
• Audit conclusions and process effectiveness
 determination contain information that
 appears to be nonconformities but no
 conformities raised.
• PEARs do not always contain the detail of the
 audit trail to confirm the process is deployed,
 working and effective as described by the
9101 Report Findings – NCRs:
• Incorrect classification of nonconformities
 against the 9101 definitions (previous pre-
 2009 version definitions are still being used)
• Containment, root cause and corrective actions
 being accepted by auditors even if they don’t
 address or fully address the nonconformity
• Details of the verification activity are not
 always described on the NCR form

General Report Findings - OERs:
• OER forms are not being fully populated –
 many boxes have no objective evidence (or
 reference to a PEAR)
• OER does not have columns correctly
 completed with required C or NCR
• PEAR forms reference the OER however no
 objective evidence recorded in the OER at the
 reference point

Additional Feedback:
• Because a stage 1 was not mandated for
 transition insufficient planning is taking place
 ahead of the on-site transition audit to fully
 inform the audit team
• Performance measures set by the company are
 not traced back to the customer or other
 requirements to confirm validity of target
• Auditors are guiding or prompting the client as
 to the ‘correct’ number of processes
• Auditors are seeking to minimise the number
 of PEARs to be raised by compressing
 processes together at a higher level e.g.
Oversight and 9101
• During oversight you will see 9101 in use:
  – As part of the audit process requirements during
    witness assessments
  – As part of the overall set of records held by the CB
    to demonstrate the audit process
• In both cases you will be looking to see that
  9101 requirements have been followed but
  also that CB audit teams
  – Audit processes and process performance
  – Follow and demonstrate the required focus elements
  – Conduct a competent audit that demonstrates the
    overall QMS performance and effectiveness
  – Audit conclusions reflect the audit trail and findings
Oversight and 9101
• Think about how the OEM will use the audit
  report, to understand:
  – How their supplier works
  – How effectively the client is turning customer
    requirements into deliverables
  – If the supplier will deliver conforming products on-
  – Where the weaknesses in the suppliers management
    system exists
  – How effectively the supplier is fixing problems and
    improving to deliver conforming products on time
  – If further assessment of the supplier is required

Approach to oversight
• Use the QMS Matrix to quickly understand the
  established processes
• Use the PEARs to gain an understanding of
  – What each process does – inputs to outputs
  – How the process works - activities
  – How the process is measured and monitored
  – If the measures are appropriate to the process
  – How the process is performing against targets
  – If not performing effectively what the supplier is
    doing to improve and if that improvement is working
• If you can’t determine these items then the
  PEAR is not complete or effective
Approach to oversight
• Where the PEAR identifies nonconformity
  – Follow the nonconformity
• Nonconformities:
  – What were they?
  – Is the statement of nonconformity actually a
    statement of nonconformity?
  – Is the nonconformity consistent with the issue
  – Was it correctly graded?
  – What caused it?
  – How was it contained and corrected?
  – What was used to verify the correction?
  – Is the process now conforming and effective?
Approach to oversight
• What are the other nonconformities?
  – Should they have been inside the PEAR?
  – Are they correctly graded?
• Review the OER if you need detail on a
  specific item or from a PEAR prompt
• Review the audit report
  – Does it reflect the audit results?
  – Does it draw appropriate conclusions?
  – Are the conclusions in line with the results and
    the audit objectives?
• Is the recommendation appropriate given
  everything else?
Approach to oversight
• The certification decision
  – Who took the decision?
  – Were they competent?
  – Were they effective?
• Was the complete report uploaded to OASIS?
  – Correctly?
  – Everything uploaded that was supposed to be?
  – Uploaded before previous certificate expired?
  – Uploaded within correct timescales?
    » Last day of surveillance + 90 days
    » Certification decision + 30 days

Forms Focus – QMS Matrix                  (Annex D)

• QMS Matrix:
  – To be completed by the audit team for each visited
    site to demonstrate which processes and quality
    management system clauses have been audited
  – Relates the Suppliers’ processes to clauses within
    the 91XX:2009 standard
• Why useful?
  – Identifies which processes contain clauses that fall
    within clause 7 of 91XX:2009
  – Each product realisation process is to be recorded
    on a PEAR (see 9101 clause
  – Quick cross check to determine if the CB has raised
    sufficient PEAR (see 9101 clause 3.7) forms.

Forms Focus – PEAR                (Annex C)

• Process Effectiveness Assessment Report:
  – Defined as ‘A document stating results and providing
    evidence of determination on the effectiveness of a
  – Each product realisation process is to be recorded
  – Content is to reflect 9101 clauses and
    regarding process management and process
    performance and effectiveness
  – 9101 Clause NOTE 2:
    » If clause 7 of the 9100-series standards and its’ sub-
      clauses are associated with processes that are
      addressed by PEARs, the objective evidence is only
      required to be documented on the PEAR. In such case,
      the PEAR number should be referenced in the OER.

Forms Focus – PEAR                 (Annex C)

• Process Effectiveness Assessment Report:
  – 9101 Clause NOTE 4:
    » At the discretion of the auditing organization, other
      processes can be recorded on a PEAR
  – The results of effectiveness shall be recorded on the
    PEAR (see Annex C) for each audited product
    realisation process.
  – The level of effectiveness for each recorded process
    shall be recorded on the PEAR (statement of
    effectiveness level).
  – The level of effectiveness for each recorded process
    has been classified as a ‘2’ or a ‘1’, this shall result in
    a nonconformity being issued against 9100-series
    standards clauses 4.1.c and f (see clause 4.2.4).

Forms Focus – PEAR         (Annex C)

• Statement of Effectiveness:

• Instructions:

Forms Focus – PEAR                (Annex C)

• Turn to PEAR form and instructions page
• Review form and associated instructions and
  note the following
  – Box 4 – Requires the OASIS OIN number
  – Box 9 - Process details, including associated process
    » Summarise the process activities, inputs, and outputs;
      including the identification of associated process
  – Box 10 – Applicable AQMS Clause(s)
    » Identify the applicable primary 9100/9110/9120
      clause(s) for this process.

Forms Focus – PEAR                (Annex C)

• Review form and associated instructions and
  note the following
  – Box 11 - Organization’s method for determining
    process effectiveness
    » Describe the method used by the organization to
      determine process effectiveness
    » [e.g., identification of Key Performance Indicators
      (KPIs) and associated targets, process capability data].
  – Box 12 - Auditor observations and comments
    supporting process effectiveness determination
    » Annotate relevant objective evidence, observed
      conditions, data, information, comments, etc. to support
      the auditor's statement of effectiveness or
      ineffectiveness, as indicated in Box #13.
    » This evidence can replace the evidence in the OER
Forms Focus – PEAR                                                                                                                      (Annex C)

• Example PEAR: Purchasing Process
  – Example has three parts; bad example, good example
    and the form instructions as below:
                  Example B:

                       Super CB                        Process Effectiveness                                                                 Example A:
                                                        Assessment Report
                      Organization: A Customer Limited             4
                                                                       Site: Somewhere              4
                                                                                                                                                  Super CB                        Process Effectiveness
                                                                                                                                                                                   Assessment Report
                  5                                       6                                         7                                                                                         4
                      PEAR Number: 02                         Audit Report Number: #1                   Issue Date: 25/12/2011               3                                                    Site: Somewhere Site           4
                                                                                                                                                 Organization: A Customer Limited                                                    OIN: 6123456789
                                                                                                                                                                                                   Wisteria Lane, Somewhere
                      Process Name: Purchasing
                                                                                                                                             5                                       6                                           7
                                                                                                                                                 PEAR Number: 02                         Audit Report Number: #1                     Issue Date: 25/12/2011
                                                                                                                                                 Process Name: Purchasing
                  Process details, including associated process interfaces:
                  Purchase of parts for use by production from approved suppliers
                                                                                                                                                 Process details, including associated process interfaces:
                                                                                                                                                 Inputs: Approved quotation, engineering definition/drawing, purchase requisitions – computer or manually
                                                                                                                                                 generated, manufacturing schedule, delivery schedule, customer sourcing requirements.
                    Applicable 9100/9110/9120 clause(s): 4.1, 4.2.3, 4.2.4, 5.3, 5.4.1, 5.5.1, 5.6, 6.2, 6.3, 6.4, 7.1.2, 7.1.3, 7.4,
                  7.4.1, 7.4.2, 7.4.3, 7.5.1, 7.5.2, 7.5.3, 7.5.4, 7.5.5, 7.6, 8.2.1, 8.2.3, 8.3, 8.4, 8.5.1, 8.5.2, 8.5.3                       Process activities: Maintain the Approved Supplier Listing, Create Purchase Orders, Manage subcontract
                                                                                                                                                 services, generate pick tickets/pick stock/issue stock, receive and inspect purchases and issuing, packaging
                                                                                                                                                 and delivery of finished products.
                       Organization’s method for determining process effectiveness:

                      1) Materials/services received by “Required Date”                                                                          Outputs: Approved suppliers, purchased materials and parts, completed subcontract services, stores stock for
                      2) On-time delivery                                                                                                        manufacturing, stock issued to manufacturing, records of: goods received (including C of C and Mill
                                                                                                                                                 Certificates), stock and completed purchase orders, shipped items delivery notes / records, C of Cs and EASA
                                                                                                                                                 Form 1s (8130s), supplier performance data.
                       Auditor observations and comments supporting process effectiveness determination:
                                                                                                                                                 Process interfaces include: Engineering – Production Engineers, Contracts Department – Sales personnel
                      The process was evaluated and found to be conforming. Samples taken were all found to be compliant with                    Suppliers (sales, quality, transport), Manufacturing - Production Manager/personnel, Accounts – sales and
                      process and 9100 requirements. On time delivery from suppliers a problem but actions are planned to be                     purchasing ledger staff, Customer – Goods receipt / transport Internally within process Buyer, Goods
                      taken to improve the situation. On time delivery to the customer is mainly good but with occasional variations.            receiving staff, Shipping staff, Inspection, transport personnel
                      It is known that there are improvement plans in place by management to improve the situation.
                                                                                                                                                 Other Observations: The risk management model indicates Purchasing is a “high” risk process with a “high”
                  13                                                                                                                             impact and a “medium” likelihood.
                       Statement of Effectiveness Level:                                                                                     10
                                                                                                                                                  Applicable 9100/9110/9120 clause(s):
                       The process is:
                                                                                                                                                 6.3, 6.4, 7.1.3, 7.4 (including 7.4.1, 7.4.2, 7.4.3), 7.5.3, 7.5.4, 7.5.5, 8.2.3, 8.3, 8.4, 8.5.2
                          1. Not implemented; planned results are not achieved.
                          2. Implemented; planned results are not achieved and appropriate actions not taken.
                          3. Implemented; planned results are not achieved, but appropriate actions being taken.                                  Organization’s method for determining process effectiveness:
                          4. Implemented; planned results are achieved.
                                                                                                                                                 A Customer utilises the following metrics to determine process effectiveness:
                  14                                                        15
                       Auditor Name(s):                                          Auditee Representative Acknowledgement
                                                                                 Name:                                                           1)   Materials/services received by “Required Date” – Target 98% schedule adherence
                                      Joe T Auditor                                                                                              2)   Approved Suppliers must have performance scores in Acceptable range –Target 95% in Acceptable range
                                                                                                   A N Auditee                                   3)   Manufacturing Orders picked and issued to schedule – Target 98% schedule adherence
                       Signature(s):                                             Signature(s):                                                   4)   On-time in full delivery to the customer - ” – Target 100% schedule adherence
                                       Joe T Auditor                                              A N Auditee

  – Discuss the examples

Forms Focus – OER               (Annex A)

• Objective Evidence Record:
  – A document recording objective evidence of the audit
    findings, including reference to the reviewed or
    observed procedures, records, products, processes,
    and associated NCRs and opportunities for
  – 9101 Clause
    » The audit team shall record detailed objective evidence
      (e.g., reviewed procedures, shop orders, training
      records, products, verification records).
    » The objective evidence shall be on a standardised form
      [i.e., the OER (see Annex A)], or on the CBs own
      documentation. In this case, the CB document shall
      meet the intent of the OER.
    » The completed forms shall be included in the audit
      records maintained by the CB.
Forms Focus – OER               (Annex A)

• Objective Evidence Record:
  – 9101 Clause continued …
    » NOTE 1 Population of the OER may start during the
      Stage 1 audit to record the documents reviewed.
    » NOTE 2 If clause 7 of the 9100-series standards and its’
      sub-clauses are associated with processes that are
      addressed by PEARs, the objective evidence is only
      required to be documented on the PEAR. In such case,
      the PEAR number should be referenced in the OER.
  – No other form can have an equivalent or
    alternative developed by the CB for its own use
  – Discussion Point:
    » What would you expect a CBs own documentation
      to look like that meets the INTENT of the OER?

Forms Focus – NCR              (Annex B)

• Nonconformity Report:
  – The NCR (see Annex B) shall be used to record
  – Each NCR shall contain only one nonconformity.
  – When nonconformities are identified, the audit team
    shall categorize the nonconformity as ‘major’ or
    ’minor’, according to the definitions provided in
    section 3 of 9101.
    » Please read 3.2 – Major Nonconformity
    » Please read 3.3 – Minor Nonconformity
    » Discussion Point:
    » What has changed in these definitions from the previous
      9101 document?

Forms Focus – NCR             (Annex B)

• Nonconformity Report:
  – The need for containment in accordance with the
    organization’s corrective action process shall be
    reviewed by the audit team.
    » Please read 9101 clause 3.1 Containment
  – Recurrence of the same or similar nonconformity
    found during consecutive audits at a particular
    site/location shall be considered as a failure of the
    corrective action process (see 9100-series standards
    clause 8.5.2) and shall result in a major
    nonconformity being issued.

Forms Focus – NCR               (Annex B)

• Nonconformity Management             (9101 clause 4.2.4)   :
 – Turn to 9101 clause 4.2.4 and read the clause
 – Note the timescales set out in this section:
   » … 7 calendar days after the audit when the nature of the
     nonconformity needs immediate containment
   » … next 14 calendar days to reach agreement with the
     audit for the effectiveness of the action taken
   » a maximum of 30 days from the end of the on-site audit
     to agree corrective action(s) and corrective action plans
 – Discussion Point:
   » According to ISO 17021 or 9101, what is the maximum
     time a nonconformity report can remain open before
     verification activities are required to be completed?
   » Consider Initial audit, Surveillance, Recertification

AS9101 Rev D FAQs

• Answers to Frequently Asked Questions have
  been developed
• Available on the IAQG Website at:

91XX:2009 Questions?

• Refer to Deployment Support Material i.e.
  FAQs, Clarifications, Changes and rational
  and Auditor Guidance Material
• Available on the IAQG Website at:



Shared By: