Company Confidential

1
Revisiting CB Use of
AS9101 Rev D
Tim Lee
12 March 2012
2
AS9101 Rev D Fundamentals
9101 defines requirements for the
preparation and execution of the audit
process
It defines the content and composition for the
audit reporting of conformity and process
effectiveness, based on
9100-series standards requirements
the organizations quality management system
documentation, and
customer/regulatory requirements
It builds on and mandates requirements and
guidelines in ISO/IEC 17021 and ISO 19011
Reminder
3
AS9101 Rev D:

CBs have now been using AS9101 Rev D for
some time and for all 91XX:2009 transition
audits
We have learned that auditing practices and
use of 9101 is inconsistent
We need to improve the effectiveness of CBs
auditing and audit reporting to
Detect and stop nonconforming practices before
they become normalised
Ensure that CB clients and their clients gain benefit
from the audits and the audit reports
Build and retain confidence airworthiness and other
authorities confidence in the ICOP scheme
4
Review of CB Audit Reports:

The IAQG OPMT carried out a review of sample
reports in October 2011
ABs have also been reporting findings made as
a result of witnessed assessments and review
of CB audit reports
Following slides provide information on those
findings and will help prepare you for
assessments you will carry out
There will be IAQG OPMT material in future
that focuses on positive models for
improvement
5
General 9101 Report Findings:

Established audit objectives are not always in
line with the example in 9101 instructions e.g.
Upgrading of certification to 9100:2009
Audit Plans and OERs do not always
demonstrate that an interview with top
management has taken place
Annex E audit conclusions are weak in relation
to overall performance, effectiveness and
compliance of the QMS of the client
organisation or in respect of the audit
objectives often providing stories of the client
history.

6
General 9101 Report Findings:

Forms are not being filled accurately from
9101 instructions
Stage 1 form is incorrectly populated with
regard to Aviation, Space and Defence
business and associated employees
Forms other than the OER are being modified
Previous version 9101 forms are being used
on occasions e.g. for NCRs
Annex E not being completed until after all of
the NCRs raised were closed out i.e. completed
at final situation not situation at closing
meeting

7
General 9101 Report Findings:

Recommendations being made incorrectly:
Recommendation for certification made even though
NCRs are raised (i.e. wrong recommendation box ticked)
Virtually no use of Annex G for complex clients
with more than one site
Lack of understanding that a second Annex E is
required after a visit to close out NCRs
Decision-making is not finding and resolving
issues in audit reports
No specific requirement for decision-makers to attend
AATT or similar to understand 91XX:2009 and AS9101
Rev D requirements on auditors


8
9101 Report Findings QMS Matrix:

The QMS Matrix does not always contain all
processes of the client
QMS Matrix does not contain sub-processes or
valued-added activities
QMS Matrix completed with X rather than the
required C or N



9
9101 Report Findings - PEARs:

Processes that contain clause 7 of 91XX:2009
requirements do not have an associated PEAR
in all cases
PEAR forms not always contain a description of
the process and interactions that is sufficient
to understand the clients process (or meet
requirements)
Clauses recorded on the PEAR are not always
consistent with QMS matrix or listed as see
QMS Matrix
PEAR process measures do not always contain
targets



10
9101 Report Findings PEARs:

PEAR process measures accepted are not
always consistent or applicable to the control
of the process
Actual process performance is not always
reported
Improvement actions against performance
below target do not in all cases result in
process achieving targets but process
described as Level 3
Improvement actions are not described just
recorded as in place



11
9101 Report Findings PEARs:

Several processes effectiveness measures
described but actual performance data not
reported against all them
Process effectiveness not always recorded on
the PEAR
Audit conclusions and process effectiveness
determination contain information that
appears to be nonconformities but no
conformities raised.
PEARs do not always contain the detail of the
audit trail to confirm the process is deployed,
working and effective as described by the
client



12
9101 Report Findings NCRs:

Incorrect classification of nonconformities
against the 9101 definitions (previous pre-
2009 version definitions are still being used)
Containment, root cause and corrective actions
being accepted by auditors even if they dont
address or fully address the nonconformity
raised
Details of the verification activity are not
always described on the NCR form


13
General Report Findings - OERs:

OER forms are not being fully populated
many boxes have no objective evidence (or
reference to a PEAR)
OER does not have columns correctly
completed with required C or NCR
PEAR forms reference the OER however no
objective evidence recorded in the OER at the
reference point



14
Additional Feedback:

Because a stage 1 was not mandated for
transition insufficient planning is taking place
ahead of the on-site transition audit to fully
inform the audit team
Performance measures set by the company are
not traced back to the customer or other
requirements to confirm validity of target
Auditors are guiding or prompting the client as
to the correct number of processes
Auditors are seeking to minimise the number
of PEARs to be raised by compressing
processes together at a higher level e.g.
Manufacturing



15
Oversight and 9101
During oversight you will see 9101 in use:
As part of the audit process requirements during
witness assessments
As part of the overall set of records held by the CB
to demonstrate the audit process
In both cases you will be looking to see that
9101 requirements have been followed but
also that CB audit teams
Audit processes and process performance
Follow and demonstrate the required focus elements
Conduct a competent audit that demonstrates the
overall QMS performance and effectiveness
Audit conclusions reflect the audit trail and findings




16
Oversight and 9101
Think about how the OEM will use the audit
report, to understand:
How their supplier works
How effectively the client is turning customer
requirements into deliverables
If the supplier will deliver conforming products on-
time
Where the weaknesses in the suppliers management
system exists
How effectively the supplier is fixing problems and
improving to deliver conforming products on time
If further assessment of the supplier is required


17
Approach to oversight
Use the QMS Matrix to quickly understand the
established processes
Use the PEARs to gain an understanding of
What each process does inputs to outputs
How the process works - activities
How the process is measured and monitored
If the measures are appropriate to the process
How the process is performing against targets
If not performing effectively what the supplier is
doing to improve and if that improvement is working
If you cant determine these items then the
PEAR is not complete or effective


18
Approach to oversight
Where the PEAR identifies nonconformity
Follow the nonconformity
Nonconformities:
What were they?
Is the statement of nonconformity actually a
statement of nonconformity?
Is the nonconformity consistent with the issue
identified?
Was it correctly graded?
What caused it?
How was it contained and corrected?
What was used to verify the correction?
Is the process now conforming and effective?



19
Approach to oversight
What are the other nonconformities?
Should they have been inside the PEAR?
Are they correctly graded?
Review the OER if you need detail on a
specific item or from a PEAR prompt
Review the audit report
Does it reflect the audit results?
Does it draw appropriate conclusions?
Are the conclusions in line with the results and
the audit objectives?
Is the recommendation appropriate given
everything else?



20
Approach to oversight
The certification decision
Who took the decision?
Were they competent?
Were they effective?
Was the complete report uploaded to OASIS?
Correctly?
Everything uploaded that was supposed to be?
Uploaded before previous certificate expired?
Uploaded within correct timescales?
Last day of surveillance + 90 days
Certification decision + 30 days



21
Forms Focus QMS Matrix (Annex D)
QMS Matrix:
To be completed by the audit team for each visited
site to demonstrate which processes and quality
management system clauses have been audited
Relates the Suppliers processes to clauses within
the 91XX:2009 standard
Why useful?
Identifies which processes contain clauses that fall
within clause 7 of 91XX:2009
Each product realisation process is to be recorded
on a PEAR (see 9101 clause 4.2.2.5)
Quick cross check to determine if the CB has raised
sufficient PEAR (see 9101 clause 3.7) forms.




22
Forms Focus PEAR (Annex C)
Process Effectiveness Assessment Report:
Defined as A document stating results and providing
evidence of determination on the effectiveness of a
process
Each product realisation process is to be recorded
Content is to reflect 9101 clauses 4.1.2.4 and 4.1.2.5
regarding process management and process
performance and effectiveness
9101 Clause 4.2.2.5 NOTE 2:
If clause 7 of the 9100-series standards and its sub-
clauses are associated with processes that are
addressed by PEARs, the objective evidence is only
required to be documented on the PEAR. In such case,
the PEAR number should be referenced in the OER.




23
Forms Focus PEAR (Annex C)
Process Effectiveness Assessment Report:
9101 Clause 4.2.2.5 NOTE 4:
At the discretion of the auditing organization, other
processes can be recorded on a PEAR
The results of effectiveness shall be recorded on the
PEAR (see Annex C) for each audited product
realisation process.
The level of effectiveness for each recorded process
shall be recorded on the PEAR (statement of
effectiveness level).
The level of effectiveness for each recorded process
has been classified as a 2 or a 1, this shall result in
a nonconformity being issued against 9100-series
standards clauses 4.1.c and f (see clause 4.2.4).






24
Forms Focus PEAR (Annex C)
Statement of Effectiveness:




Instructions:





25
Forms Focus PEAR (Annex C)
Turn to PEAR form and instructions page
Review form and associated instructions and
note the following
Box 4 Requires the OASIS OIN number
Box 9 - Process details, including associated process
interfaces
Summarise the process activities, inputs, and outputs;
including the identification of associated process
interfaces
Box 10 Applicable AQMS Clause(s)
Identify the applicable primary 9100/9110/9120
clause(s) for this process.





26
Forms Focus PEAR (Annex C)
Review form and associated instructions and
note the following
Box 11 - Organizations method for determining
process effectiveness
Describe the method used by the organization to
determine process effectiveness
[e.g., identification of Key Performance Indicators
(KPIs) and associated targets, process capability data].
Box 12 - Auditor observations and comments
supporting process effectiveness determination
Annotate relevant objective evidence, observed
conditions, data, information, comments, etc. to support
the auditor's statement of effectiveness or
ineffectiveness, as indicated in Box #13.
This evidence can replace the evidence in the OER






27
Forms Focus PEAR (Annex C)
Example PEAR: Purchasing Process
Example has three parts; bad example, good example
and the form instructions as below:








Discuss the examples





ExampleA:
Super CB

Process Effectiveness
Assessment Report
3
Organization: A Customer Limited
4
Site: Somewhere Site
Wisteria Lane, Somewhere
4
OIN: 6123456789
5
PEAR Number: 02
6
Audit Report Number: #1
7
Issue Date: 25/12/2011
8
Process Name: Purchasing


9
Process details, includingassociated process interfaces:
Inputs: Approved quotation, engineering definition/drawing, purchase requisitions computer or manually
generated, manufacturing schedule, delivery schedule, customer sourcing requirements.

Process activities: Maintainthe Approved Supplier Listing, Create Purchase Orders, Manage subcontract
services, generate pick tickets/pick stock/issuestock, receive and inspect purchases and issuing, packaging
and delivery of finished products.

Outputs: Approved suppliers, purchased materials and parts, completedsubcontract services, stores stock for
manufacturing, stock issued to manufacturing, records of: goods received (including C of Cand Mill
Certificates), stock and completed purchase orders, shipped items delivery notes / records, Cof Cs and EASA
Form 1s (8130s), supplier performance data.

Process interfaces include: Engineering Production Engineers, Contracts Department Sales personnel
Suppliers (sales, quality, transport), Manufacturing - Production Manager/personnel, Accounts sales and
purchasing ledger staff, Customer Goods receipt / transport Internally within process Buyer, Goods
receiving staff, Shippingstaff, Inspection, transport personnel

Other Observations: The risk management model indicates Purchasing is a high risk process with a high
impact and a medium likelihood.
10
Applicable 9100/9110/9120 clause(s):
6.3, 6.4, 7.1.3, 7.4 (including 7.4.1, 7.4.2, 7.4.3), 7.5.3, 7.5.4, 7.5.5, 8.2.3, 8.3, 8.4, 8.5.2

11
Organizations method for determining process effectiveness:

A Customer utilises the following metrics to determine process effectiveness:

1) Materials/services received by Required Date Target 98% schedule adherence
2) Approved Suppliers must have performance scores in Acceptable range Target 95% in Acceptable range
3) Manufacturing Orders picked and issued to schedule Target 98% schedule adherence
4) On-time in full delivery to the customer - Target 100% schedule adherence
ExampleB:

Super CB

Process Effectiveness
Assessment Report
3
Organization: A Customer Limited
4
Site: Somewhere
4
OIN:
5
PEAR Number: 02
6
Audit Report Number: #1
7
Issue Date: 25/12/2011
8
Process Name: Purchasing


9
Process details, includingassociated process interfaces:
Purchase of parts for use by production from approved suppliers
10
Applicable 9100/9110/9120 clause(s): 4.1, 4.2.3, 4.2.4, 5.3, 5.4.1, 5.5.1, 5.6, 6.2, 6.3, 6.4, 7.1.2, 7.1.3, 7.4,
7.4.1, 7.4.2, 7.4.3, 7.5.1, 7.5.2, 7.5.3, 7.5.4, 7.5.5, 7.6, 8.2.1, 8.2.3, 8.3, 8.4, 8.5.1, 8.5.2, 8.5.3

11
Organizations method for determining process effectiveness:

1) Materials/services received by Required Date
2) On-time delivery
12
Auditor observationsand comments supporting process effectiveness determination:

The process was evaluated and found to beconforming. Samples taken were all found to becompliant with
process and 9100 requirements. On time delivery fromsuppliers a problembut actions are planned to be
taken to improve the situation. On time delivery to thecustomer is mainly good but with occasional variations.
It is known that there are improvement plans in place by management to improve the situation.
13
Statement of Effectiveness Level:
The process is:
1. Not implemented; planned results are not achieved.
2. Implemented; planned results are not achieved and appropriate actions not taken.
3. Implemented; planned results are not achieved, but appropriate actions being taken.
4. Implemented; planned results are achieved.

14
Auditor Name(s):
Joe T Auditor

15
Auditee Representative Acknowledgement
Name:
A NAuditee
14
Signature(s):
Joe T Auditor
15
Signature(s):
A N Auditee

28
Forms Focus OER (Annex A)
Objective Evidence Record:
A document recording objective evidence of the audit
findings, including reference to the reviewed or
observed procedures, records, products, processes,
and associated NCRs and opportunities for
improvement.
9101 Clause 4.2.2.5
The audit team shall record detailed objective evidence
(e.g., reviewed procedures, shop orders, training
records, products, verification records).
The objective evidence shall be on a standardised form
[i.e., the OER (see Annex A)], or on the CBs own
documentation. In this case, the CB document shall
meet the intent of the OER.
The completed forms shall be included in the audit
records maintained by the CB.







29
Forms Focus OER (Annex A)
Objective Evidence Record:
9101 Clause 4.2.2.5 continued
NOTE 1 Population of the OER may start during the
Stage 1 audit to record the documents reviewed.
NOTE 2 If clause 7 of the 9100-series standards and its
sub-clauses are associated with processes that are
addressed by PEARs, the objective evidence is only
required to be documented on the PEAR. In such case,
the PEAR number should be referenced in the OER.
No other form can have an equivalent or
alternative developed by the CB for its own use
Discussion Point:
What would you expect a CBs own documentation
to look like that meets the INTENT of the OER?







30
Forms Focus NCR (Annex B)
Nonconformity Report:
The NCR (see Annex B) shall be used to record
nonconformities;
Each NCR shall contain only one nonconformity.
When nonconformities are identified, the audit team
shall categorize the nonconformity as major or
minor, according to the definitions provided in
section 3 of 9101.
Please read 3.2 Major Nonconformity
Please read 3.3 Minor Nonconformity
Discussion Point:
What has changed in these definitions from the previous
9101 document?






31
Forms Focus NCR (Annex B)
Nonconformity Report:
The need for containment in accordance with the
organizations corrective action process shall be
reviewed by the audit team.
Please read 9101 clause 3.1 Containment
Recurrence of the same or similar nonconformity
found during consecutive audits at a particular
site/location shall be considered as a failure of the
corrective action process (see 9100-series standards
clause 8.5.2) and shall result in a major
nonconformity being issued.







32
Forms Focus NCR (Annex B)
Nonconformity Management (9101 clause 4.2.4):
Turn to 9101 clause 4.2.4 and read the clause
Note the timescales set out in this section:
7 calendar days after the audit when the nature of the
nonconformity needs immediate containment
next 14 calendar days to reach agreement with the
audit for the effectiveness of the action taken
a maximum of 30 days from the end of the on-site audit
to agree corrective action(s) and corrective action plans
Discussion Point:
According to ISO 17021 or 9101, what is the maximum
time a nonconformity report can remain open before
verification activities are required to be completed?
Consider Initial audit, Surveillance, Recertification







33
AS9101 Rev D FAQs
Answers to Frequently Asked Questions have
been developed
Available on the IAQG Website at:
http://www.sae.org/iaqg/organization/req
uirements.htm













34
91XX:2009 Questions?
Refer to Deployment Support Material i.e.
FAQs, Clarifications, Changes and rational
and Auditor Guidance Material
Available on the IAQG Website at:
http://www.sae.org/iaqg/organization/req
uirements.htm













Questions?
35