QUALITY BI mQIP CDRR & CRPM

Oct, 2016

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

QDI High Level Architecture

C3
FAQMS

FAQMS
C3

Quality
Manufacturing
Information
Portal

C3BL
QDI
EDWTD

Autotest
Test & Repair

ODS

CG1PRD

CDETS

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

Quality: Overall Architecture

Oracle Analytics

CRPM Exclusion

Transactional
Applications

(Tl9K,PV2,etc)
(Tl9K,PV2,etc)

Data Processing

CG1PRD/
CG1PRD/
ODSPROD
ODSPROD

Downstream
Downstream

Quality Business Layer/

Quality Data Mart
(QBL/QDM)
Customers,RMA,SR,Shipments,
Products,Install base,etc

EDWTD
EDWTD

MDF
MDF

CDETS
CDETS

SPRIT
SPRIT

Excel
Excel
Uploads
Uploads

AGILE
AGILE

Data Capture/
Source
Systems

C3/C3BL
C3/C3BL
(SETPRD)
(SETPRD)

IR Exclusions

CMNPROD

Test, and Repair Data

2010 Cisco and/or its affiliates. All rights reserved.

Data
W/H

Quality
Quality Business
Business Layer
Layer

Date Of
Manufacturing
(DOM)

AUTOTEST
AUTOTEST
(SQL
(SQL
SERVER)
SERVER)

Reporting

CEC/CCIX Security
Security
CEC/CCIX

Internal / External Users

Java Apps

Cisco Confidential

Quality Reporting Data Flow

Hardware track
C3, OM
C3, OM
C3
C3
ESMPRD
ESMPRD

CG1PRD
CG1PRD

CG1PRD
CG1PRD

EDWTD
Hierarchies
EDWTD
Hierarchies
C3BL

Products, Location and Customer data

RMA, Service Request, Customer/Party Data

C3BL
CDB

Purge, MCN, PCN, LECO, GECO, ECO,CPAT

CDB
ODS

Product Reliability Class, Serial Numbers

Shipment

ODS
DOM

Test & Repair Data

DOM

GPI Mapping And HW filters,

Excel upload

Quality Data
Quality
Data
Mart
Mart
Field
Quality

Production Yield
CRPM

Field
Quality
Repair
Metrics
Repair
Metrics
Eng. Change
Eng.
Change
Prod.
Metrics
Prod.Board
Metrics
Yield

QE Contact, NHFF, Customer Acquisition

(CE Russia,etc),QDI Publishing

CPAT Analysis,
Failure
CPATSupplier Quality
Supplier
Quality
Customer
H/w

Quality H/w
Customer
Quality
Product Hierarchy
Product
Time Hierarchy
Hierarchy
Time
Hierarchy
Contract

Manufacturer
Contract
Manufacturer
Suppliers
Suppliers
Customer Hierarchy
Customer Hierarchy

TLK900
QMS
TLK900
QMS
Platform support only
CED
CED
2010 Cisco and/or its affiliates. All rights reserved.

Failure Analysis
Production Yield

CRPM
DRT Field
DRT Field
Mfg Qlty Change
Management
Mfg Qlty Change
Management

Board
Component
Yield
DPPM
Component
DPPM
Failure Analysis,

ODS
ODS

CED
CED
Failure Analysis

TL9000

TL9000

Oracle
Analytics
Oracle
Analytics

Supplier DPPM
Supplier DPPM
Supplier Quality
Supplier Quality
CM Mfg.
Performance
CM Mfg.
(DOM)
Performance
(DOM)
SQEM
SQEM

Adhoc
Reporting
Adhoc

HW Quality
Reporting
Process,
HW QE
Quality
Field HW
QE Process,
FieldProduct
HW
Quality
Product
Quality

Cisco Confidential

Data
Sources

Hardware(QDM) Data Flow

QUALITY DATA MART

Extraction Transformation and Loading

EDW
EDW

C3BL
C3BL

Staging
Staging
Layer
Layer

ODS
ODS

Dimension
Dimension
Layer
Layer
Normalize
Normalize
Layer
Layer

(Maintains
(Maintains History
History for
for
RMA
RMA
/Products/Shipment)
/Products/Shipment)

Aggregate
Aggregate
Layer
Layer

RRL
RRL
Layer
Layer

Fact
Fact
Layer
Layer

Downstream
Systems(TL9K,PV2,KBS,etc)
2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

UpStream Data Sources

C3

IR Exclusion Tool

RMA Exclusion Tool

Java Applications

ODS

EDWT
D

Excel
Upload

QDI

CRPM & CDRR

FAQMS

FAQMS

Compliance Test
TL9K or Targa

TL9K

Auto Test & Repair

Oracle Database (CMNPROD)
Data Sources C3 SRs for RMA (Return Material Authorization)
ODS Shipped Data transactions
EDW Product details
CDETS - SW related Bugs
Excel Upload Customer related networks, Install Base etc.
For example Cisco Voip given to JPMC (Here JPMC is
customer & how many Voips are shipped will derive the Install
Base)
2010 Cisco and/or its affiliates. All rights reserved.

CDETS

DOM
OBIEE Dashboards

Cisco Confidential

Start

GIMS Tool

QME

Month End
FME

Oct Month

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

CRPM Dashboard
Provides a measure that indicates how a customer's individual hardware products (PIDs) are performing within
their network. The product performance is measured by comparing the customer's calculated field
performance or Field MTBF to the products Predicted MTBF. Utilizing Cisco Manufacturing's established
Reliability Ratio methodology to monitor performance
Key Metrics: Reliability Ratio, Predicted MTBF (Mean Time Between Failures), Field MTBF (Mean time
Between Failures, Operation Hours,ISF (In-Service Failure) , FA Codes in Admin,AFR (Annualized Failure
Rate)
Biz uses this reports in customer facing mtg to share HW perf , identify problems and take corrective action.

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

CRPM Dashboard Definitions

GPID: Key Products IDs that are determined by the customer and account team to monitor in CRPM.

Installed base
Actual no. of PID units operating in customers network
Does not include spares, etc. in customer inventory

In Service Failures (ISF)

A scrubbed monthly count of HW returns for each monitored PID
Only actual in service HW failures are counted for ISF. Returns due to
other reasons, including proactive returns, are not included

Operating Hours
Operating hours accumulated by the Installed Base for each PID by calendar month
No. of hours in a month x Installed Base

Field Mean Time Between Failures (fMTBF)

Indicates the actual reliability performance for a PID in the field
Calculated monthly using a 3-month rolling window
Is equal to 3 mo. total of operating hours / 3 mo. total of ISF
2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

CRPM Dashboard Definitions

Predicted Mean Time Between Failures (pMTBF)
Statistical prediction of the duration between failures of a PID
Statistically sums up the FIT (Failures in Time) values for all the components of a PID and generates the overall MTBF
value as an inverse of the total FIT value for PID

Reliability Ratio (RR)

Ratio of the Field MTBF (actual reliability) to pMTBF (predicted reliability). Reliability
Ratio = # Field MTBF Hours / # Predicted MTBF Hours
Higher RR (>1.0) implies field product performance is better than projected
RR < 1.0 or downward trend in RR triggers proactive Cisco action

Customer RR
RR calculated for customer's installed base and In-service HW failures. Customer Reliability Ratio = Customer # Field
MTBF Hours / # Predicted MTBF Hours

Cisco RR
RR calculated across Ciscos entire installed base and RMA authorizations. Cisco Reliability Ratio = Entire Customer #
Field MTBF Hours / # Predicted MTBF Hours

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

10

CRPM Calculation Details

Field MTBF Calculation:
# Field MTBF Hours = # 3Mon Op Hours / # 3Mon In-service Failures
# 3Mon Op Hours = ([Month1 Installbase] * Hours in Month1) + ([Month2
Installbase] * Hours in Month2) + ([Month3 Installbase] * Hours in Month3)
(Note: 4 weeks = 672 hours; 5 weeks = 840 hours)
Example: Month1 IB= 200, Month2 IB = 300, Month3 IB = 400
Op Hours: (200 * 672) + (300 * 672) + (400 * 840) = 672,000 hours

# 3Mon In-service Failures = Month1 In-service failures + Month2 In- service failures + Month3 Inservice failures
Example: Month 1 in-service failure = 1, Month 2 = 2, Month 3 = 0
In-service Failures: 1 + 2 + 0 = 3
# Field MTBF hours: 672,000 / 3 = 224,000 hours

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

11

CRPM Calculation Details

Field MTBF Calculation:
# Field MTBF Hours = # 3Mon Op Hours / # 3Mon In-service Failures
# 3Mon Op Hours = ([Month1 Installbase] * Hours in Month1) + ([Month2
Installbase] * Hours in Month2) + ([Month3 Installbase] * Hours in Month3)
(Note: 4 weeks = 672 hours; 5 weeks = 840 hours)
Example: Month1 IB= 200, Month2 IB = 300, Month3 IB = 400
Op Hours: (200 * 672) + (300 * 672) + (400 * 840) = 672,000 hours

# 3Mon In-service Failures = Month1 In-service failures + Month2 In- service failures + Month3 Inservice failures
Example: Month 1 in-service failure = 1, Month 2 = 2, Month 3 = 0
In-service Failures: 1 + 2 + 0 = 3
# Field MTBF hours: 672,000 / 3 = 224,000 hours

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

12

CRPM Calculation Details

Predicted MTBF Methodology:
o MTBF is calculated in accordance to Telcordia SR-332
o Telcordia SR-332 (Formerly Bellcore TR-332) is the Telecommunications Industry Standard
Reliability Prediction Methodology and is a widely accepted prediction model
o Based upon the Mil-HDBK 217
o

Modified to better represent the environments and failure rates of Telecommunications products

o MTBF Hours = Operation Hours / Failures

o Inversely Proportional to the Failure Rate
o Often expressed as FIT Failure in Time:
o

Failures per 10^9 hours = FIT = 10^9/MTBF

MTBF = 10^9 / FIT

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

13

CDRR Dashboard
CDRR represents a new way to target field performance. Previously Ciscos field reliability targets were based
solely on the output of a standard model, Telcordia SR-332. Though this model represents and industry
standard method of predicting failures rates, a method that our customers accept, it is not necessarily an
adequate targets for all products. CDRR targets were developed to address the limitations of the current field
reliability target setting process.
Key Metrics: CDRR Global Reliability Ratio, CDRR Reliability Ratio by Theatre, CDRR sensing Alarm &
CDRR sensing Alarm by theatre.
Biz uses these reports to address the below : Reflect customer expectations,
Respond to competitive pressures,
Explicitly support to network availability targets, nor
Address historical process capability

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

14

Business Case for CDRR

After a product begins shipping, the CDRR and PRR will be computed and reported in mQIP. FP-QEs are responsible for
monitoring the CDRR as the core metric. The CDRR indicator for a PID will not be green when its value is less than 1.0
that is when the fMTBF is less than the CDRT. When investigating poor performing PIDs (red or yellow CDRRs), different
actions may be required depending on the status of the corresponding PRR for the same PID (when the PRR is green, yellow
or red). The PRR status for a PID with a red or yellow CDRR will determine if there is a manufacturing concern or design
related concern.
Design concern & Mfg concern (case 1): A Design concern and Mfg concern are indicated when both the CDRR and PRR
are red. This is a more serious situation since there is both a risk to meeting customer expectations and Ciscos ability to
cost effectively manufactures the PID. All of the following actions should be followed by both MO-QE and FP-QE.
Design concern (case 2): A red CDRR and green or yellow PRR indicate a potential inability of this PID to meet its ideal
customer target reliability (CDRT) even though it is meeting its given design (pMTBF). Actions in this case are limited to a
review of other PIDs in the same product reliability class (PRC). An inordinate number of other PIDs with a red CDRR in
this same class may require a reevaluation of the CDRT for that class by the Central Quality team. If there are very few or no
other PIDs with a red CDRR in the same class, then this may require a reevaluation of the suitability of this PID belonging to
the class. There is no other recourse beyond these actions for addressing a red CDRR in this situation, other than this
information should be taken into consideration for future product revisions and next generation development.

1. Design & Mfg Concern

2. Design Concern
3. Mfg Concern

Mfg. concern (case 3): A red PRR corresponding to a green or yellow CDRR indicates a potential inability for the PID to
meet its given design (pMTBF), although its performance is sufficient enough to meet customer expectations. The first
priority is given to understanding the potential issues in the factory (MO-QE) or in the field (FP-QE). This case will not
normally be reported.

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

15

Calculations
Customer Driven Reliability Ratio = # Field MTBF Hours / # Customer Target (Hours) for a GPID Class.

Field MTBF Hours = # 3Mon Op Hours / # 3Mon RR Failures (In-service failures)

o

# 3Mon Op Hours = (Month1 Installbase * Hours in Month1) + (Month2 Installbase * Hours in Month2) + (Month3 Installbase * Hours
in Month3)
o #3Mon RR Failures = #13wk HW RMA Units - #13wk Raw IR Authorizations * 1-% Final Non HW Failure
Customer Target (Hours) for a GPID Class = Customer Target for a GPID Class

CDRR targets are determined by taking into consideration historical predicted MTBFs, field MTBFs, Ciscos High
Availability (HA) guidelines, customer input and competitive information.

Algorithm:
The following algorithm is used to determine customer driven reliability targets for reliability classes, or groupings of PIDs.

f(x) = Min[Max{HA, pMTBF, fMTBF}, Cap]

The Customer Driven reliability target is the larger of the three values HA, pMTBF and fMTBF, but not to exceed the
Cap, as defined below.

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

16

CDRR calculations
CDRR PRODUCT CLASSES AND TARGET SETTING PROCESS
CDRR targets are determined by taking into consideration historical predicted MTBFs, field MTBFs,

Ciscos High Availability (HA) guidelines, customer input and competitive information.
a)

HA Support of High Availability baseline

CDRR must support the HA baseline, even though the HA was initially architected for the high end platforms, most customer expectations fall into that category. The same
argument could be made for low-end products, but based on earlier studies, our conclusion is that they normally fall into less complex architecture, hence higher predicted MTBFs
and high availability (see Appendix Platform Availability using Predicted, Field and HA minimum MTBF).
b) pMTBF Predicted MTBF
Predicted MTBF is what is quoted in Marketing claims, data sheets, RFPs and SLAs. CDRR must take into account Predicted MTBF also because of historical consistency. The 5 th
percentile pMTBF is computed for all current PIDs in each reliability class to determine the target value.

c) fMTBF Field MTBF

CDRR must take into account Field MTBF because of historical performance. Customers expect future releases or PIDs of same functionality to
perform at least same or better than their predecessors. The 5 th percentile fMTBF is computed for all current PIDs in each reliability class to
determine the target value.
d) Cap Customer Ideal
The Customer expectations, as discusses earlier, were translated into MTBF hours. The customer Cap serves as the goal for high end PIDs where
high complexity drives low pMTBFs, and it is the maximum value, or cap, for low end PIDs where low complexity drives pMTBF that are
significantly beyond customer expectations.

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

17

CDRR Definitions
Reliability Ratio
Reliability Ratio represents the ratio of estimated Field Mean Time Between Failures (fMTBF) over Target MTBF for a Generic Product ID. The ratio is calculated against two targets, the Customer Driven
MTBF target or the Telcordia SR-332 based target.
Reliability Ratio= fMTBF/Target MTBF
RR = [0,1) indicates the product appears to be performing worse than targeted
RR = 1 The aggregate field performance estimate is exactly as targeted
RR > 1 Aggregate field performance for the GPI appears better than targeted

Reliability Ratio
Reliability Ratio provides a comparison of field failure rate to expectations. Reliability Ratio allows ranking products by performance within a product family or other grouping to determine the worst performing
products in that group. These are primary metrics for reporting Field HW Reliability

Reliability Ratio Formula

Product Reliability Ratio = # Field MTBF Hours / # Predicted MTBF Hours
Customer Driven Reliability Ratio = # Field MTBF Hours / # Predicted MTBF Hours
Field MTBF Hours = # 3Mon Op Hours / # 3Mon RR Failures (In-service failures) # 3Mon Op Hours = ([Month1 Installbase] * Hours in Month1) + ([Month2 Installbase] * Hours in Month2) + ([Month3 Installbase] *
Hours in Month3)

MTBF
MTBF represents a reciprocal measure of failure rate. Similar to how Americans measure automobile fuel consumption rate in Miles Per Gallon, where high MPG implies a low fuel consumption rate,
MTBF represents an analogous measure of failure rate defined as Operation Hours per failure:
MTBF = Operation Hours / Failure

Operation Hours
Operation Hours are calculated using a three month rolling average of Ship Base (IB)
OH= IB*24*7*Weeks in Fiscal Month
Note that Operation Hours estimate assumes that products are installed immediately upon shipment, and does not account for products going out of service without an RMA. Since Ship Base represents
an overestimate of the true returnable population, Operation Hours represents an overestimate of the actual field hours accumulated.

Failures
Failures are calculated using a three month sum of In-Service RMA Authorizations, prorated by the calculated Non-Hardware Failure Factor. In-Service Failures represent a subset of Hardware Failure
coded RMAs.
NHFF represents a specific definition of a No Trouble Found Factor based on FA
In-Service RMA represent a subset of the typical Authorized RMA Qty in that all returns coded as raw Immediate Returns (DOAs and Visual Defects) are removed from the failure count.
Failures = 3 months In-Service RMA * (1-NHFF Factor)

Target MTBF
Target MTBF can be either the predicted MTBF (pMTBF) according to Ciscos implementation of the Telcordia SR-332 prediction standard, or the Customer Driver Reliability Ratio target as described
here

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

18

CDRR Measure Goals

Reliability Ratio status is classified with one of the 4 colors.
RR>1: Green. Exceeds Target
RR<1:

Red. Needs Immediate Attention. Data @>90% CL indicates field performance fails to meet target

Yellow: Pointwize data indicates RR<1, but not statistically significantly

Grey: Pointwize data indicates RR<1, but less than 10 RMA
Business Rules:
1) Products that qualify for the following conditions will be excluded from this metric:
- Products with no Target MTBF assigned (OPUS assigns CDRR Class, STAR for pMTBF)
- Products that are non-Hardware (cables, software, documentation etc.), memory products and Bundle
Product IDs.
- Products displaying Negative Ship Base (addressed by BFRU initiative)
2) Hardware Failure RMA includes Authorizations with Failure Codes OFHW, OFSW, CUUD, CUCD, HWW,
HWP, CRQ, FPQ or any code beginning in OHW.
3) % Final Non HW Failure generated very month is frozen as of the month and to be used for historical
data (trending)
4) 3month Raw Immediate Return Quantity is eliminated from Total Authorization quantity to determine
In-Service Failures
2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

19

Reports
CDRR Report by Time Buckets

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

20

CDRR Reports

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

21

Thank you.