SOS Services

SM

Caterpillar SOS Services Evaluation of the Analex PQ Ferrous Debris Monitor Introduction:
The Analex PQ is designed to measure ferrous debris in lubricating oils. The PQ instrument exposes an oil sample to a magnetic field. If there are magnetic particles in the oil, the PQ records the change in the magnetic field. The result, of this measurement, is called the PQ index. The PQ index is proportional to the mass of ferrous particles in the oil. A small percentage of Caterpillar dealers have purchased the Analex PQ instrument, and use them regularly as part of their oil analysis program. Also, one of the Cat SOS Services regional laboratories provides PQ results. Some of these dealers use the PQ as a screening device to trigger more in-depth testing. This in-dept testing could be microscope analysis or filter patch analysis of particulates. The PQ has not been used as a replacement for an optical particle counter. Instead, the PQ has been used as a supplement to optical particle counting. Optical particle counters are ineffective when used for samples with: high levels of water contamination, dark oxidized oils, dark dyed oils, and engine oils with high soot content. Some oils contain so many particles that they exceed the measuring capacity optical particle counters. And, most labs will not use an optical particle counter on samples that contain visible particles. The PQ can be used in all of these cases. In order to evaluate the value of PQ analysis, Caterpillar SOS Services launched a 6 Sigma project. The 6 Sigma project evaluated PQ in three key areas: 1. Interpretation of PQ results 2. Calibration, Accuracy, and Repeatability of the PQ instrument 3. Worldwide instrument service capability Each of these three areas will be summarized in this report, and a final recommendation will be provided.

Interpretation of PQ results:
To understand how PQ results are used, the 6 Sigma team evaluated interpretations from four Cat dealers. Three of the dealers are located in North America, and one of the dealers is in Europe. These dealers have been using PQ results for two or more years.

SOS Services
SM

Interpretations were selected from the following machine models: 777D, D8R, 330C, D6R, and 980G. These machines were selected because of their high population density and their wide range of applications. A total of 17,302 oil samples were evaluated from these machines. The oil samples were evaluated by individual compartment for each of the five models. The objectives for this part of the project were as follows: 1. Determine the best way to identify abnormal PQ results. 2. Determine how PQ is used when analyzing sample results. 3. Determine if PQ results add a needed dimension of information about the sample. Identifying Abnormal PQ Results Based on studies done by Caterpillar dealers, a set of guidelines have emerged for the PQ. These guidelines were developed from samples taken exclusively on Cat equipment. The guidelines were developed using the bottle method. The bottle method measures the PQ of the sample in the original sample bottle. A pot method is also available. When the pot method is used, 2 ml of the sample is decanted into a standard container (pot) supplied by Analex. The pot method produces lower PQ values. PQ Guidelines Bottle Method Average 14 14 22 154 108 297

Compartment Engine Hydraulic System Power Shift Transmission Axles/Differentials Final Drives Tandems

Pot Method - estimated 5 5 7 51 36 99

The team discovered that these guidelines should only be used if there is no historic PQ data from the compartment. In some cases the guideline is too low and there would be a very high percentage of samples flagged as abnormal. In other cases the guideline is too high, and some abnormal samples would be missed. Instead of using these somewhat generic guidelines, the team found that statistical guidelines could be developed, by model, for each compartment. The statistical guideline is the average value plus three times the standard deviation, (Average + 3Sigma). This statistical guideline flagged approximately 1.4% of the PQ samples as abnormal. The team concluded that statistical guidelines would be needed, by model, for each compartment. These statistical guidelines could be calculated with a tool

SOS Services
SM

that is similar to the Wear Table generator. The statistical guidelines could then be written into SOS Services Manager and used to flag abnormal results. If statistical guidelines are not available, the best way to identify abnormal results is through trend analysis. How Interpreters use PQ To determine how the PQ results are used, the team closely investigated the 1.4% of the samples flagged as abnormal for PQ. The team looked at the Overall Evaluation that was associated with these abnormal samples and they also looked at the way PQ was used in the interpretation. 24% of the abnormal samples produced a Red Alert on the SOS Services report. The percentage of Green Alerts and Yellow Alerts can be seen in the chart below.
Alert Level % Produced by Abnormal Samples All Compartments

24% 41%

35% This result was somewhat surprising since the PQ value for all these samples exceeded the mean plus 3 sigma of the compartment population. Even though the PQ values were quite high, for all of these samples, the interpreters did not assign a Red Alert to the report. After further investigation, the team discovered that many of the Yellow Alert and Green Alert samples did not have other abnormal test results. Without collaborating evidence, the interpreters were reluctant to assign a Red Alert evaluation to the sample. The team also broke down this data by compartment. Each major machine compartment was evaluated for Red alerts, when the PQ was at abnormal levels. The results, shown in the chart below, show that the interpreters found the most value for PQ in evaluating geared compartments. Abnormal PQ values for final drive oils generated the most Red Alerts. This chart produced another interesting result. Abnormal PQ values had the least significance in the interpretation of engine oils. Since engine oils cannot be evaluated with optical particle counters, the team thought that abnormal PQ values would be very important to engine oil interpretations. Apparently, many of

SOS Services
SM

these abnormal engine PQ values were not supported by other abnormal test results. Therefore, the interpreters did not apply a Red Alert to the overall evaluation.
% Red Alerts Produced by Abnormal PQ Values
45% 40% 35% 30% 25% 20% 15% 10% 5% 0% Engine Hydraulic System Powershift Transmission Differential Final Drive

The Value of PQ in Red Alerts The final part of this section was to evaluate the red alert samples and determine how the interpreters used abnormal PQ results. For this evaluation, 58 interpretations were studied. All of these interpretations had an abnormal PQ results, and they were assigned a Red Alert by a dealer interpreter. The team found that 41% of these Red Alert samples absolutely needed the PQ analysis to make an accurate interpretation. In many of these cases, the abnormal PQ value was accompanied by visible metal in the sample. The PQ analysis helped by placing a quantitative value to the visible metal.
% of Red Alert samples that needed PQ to make an accurate interpretation

No 39%

Yes 41%

Maybe 20%

SOS Services
SM

39% of the samples did not need an abnormal PQ value to make an accurate interpretation. These samples had other results that alone would have led to a Red Alert evaluation. The remaining 20% of the samples may have needed the abnormal PQ for an accurate interpretation. Interpretation Summary In this part of the project, the 6 Sigma team discovered that trend analysis or specific PQ guidelines, by compartment, are needed to evaluate PQ results. General guidelines are not acceptable. The team also discovered that PQ provides the most value in the evaluation of Final Drive samples. Interestingly, PQ provided the least value in the interpretation of engine oil samples. Finally the team found that about 41% of the Red Alert samples needed the PQ results for an accurate interpretation. When compared to the 17,302 samples evaluated, PQ was needed for an accurate interpretation of 0.13% of these samples.

Calibration, Accuracy, and Repeatability of the PQ instrument:

Calibration Analex PQ instruments are factory calibrated to a PQ value of 3000. A ten-point calibration curve is used. The calibration process produces an R2 value of 0.9969. For laboratory calibrations, Analex provides a single calibration standard with a PQ value of 750. The calibration procedure uses zero as the starting point and then measures the value of the standard. Calibration checks must be within 1% of the PQ 750 standard. If the instrument is not within 1% of the PQ 750 standard, the instrument will recalibrate to the PQ 750 standard. The 6 Sigma team noted several shortcomings with this calibration procedure. Laboratory best practices recommend that calibration procedures contain more than two data points. However, Analex does not provide additional calibration standards with this instrument. Additional calibration standards can be custom made and purchased from Analex. As an alternative to full calibration standards, Analex can provide check standards. Check standards are less costly than full calibration standards. These check standards are made to be within a PQ range. Check standards can be used during calibration checks; however, they are not part of the instrument calibration process. Another shortcoming is the material composition of the calibration standard. The PQ 750 standard is made in an epoxy matrix, not oil. Again, laboratory best practices recommend that calibration be done in a material that is similar to the material under evaluation. Because of issues involving particle distribution and settling, Analex does not provide calibration standards in oil. The SOS Services lab in Malaga uses check standards and the PQ 750 standard to check calibration. Calibration checks are run four times a day. The results in

SOS Services
SM

the chart below show the repeatability of those checks. The last column in the chart shows how the results compare to the specified repeatability of the instrument. The Analex PQ data sheet states the repeatability as +/- 4 PQ or 1% of average reading, whichever is greater.
PQ Standards Results - Malaga Lab - June 2006 PQ Standard value or (PQ Range) value (0-30) (90-150) (625-685) 750 Samples that Meet Analex Repeatability 37% 35% 73% 88%

Standard Type check standard check standard check standard calibration standard

Average 20.5 141.3 567.9 751.4

Sigma 6 8.4 7.5 5.3

Analex Repeatability: +/-4 PQ or 1% of average reading, whichever is greater

This chart shows that this PQ instrument had difficulty meeting advertised repeatability in the Cat laboratory environment. Repeatability was improved at the higher PQ values. The repeatability of the instrument will be further discussed later in this document. Accuracy The Analex brochure indicates that the PQ number is proportional to the mass of ferrous (magnetic) wear debris. However, Analex does not provide the correlation between the PQ number and the mass of ferrous wear debris. To investigate the relationship between PQ number and Fe, the team plotted PQ verses Fe for typical compartments. The Fe values were generated by a spectrophotometer; therefore, the particle sizes are less than ten micron. Examples of these plots are shown below. This series of plots are from D8R tractors. The same relationships existed for other Cat models in this study.
D8R Engines
140 120 R = 0.05
2

Fe - ppm

100 80 60 40 20 0 0 10 20 30 40 50 60

PQ

SOS Services
SM

D8R Hydraulic System

50 40 R = 0.08
2

Fe - ppm

30 20 10 0 0 10 20 30 40

PQ

D8R Powertrain
100 80 R = 0.36
2

Fe - ppm

60 40 20 0 0 10 20 30 40 50 60 70

PQ

D8R Final Drives

1200 1000 R = 0.58
2

Fe - ppm

800 600 400 200 0 0 50 100 150 200 PQ 250 300 350 400

SOS Services
SM

The plots show no correlation between the PQ number and Fe for the engine and hydraulic system compartments, as shown by the very low R2 values. The powertrain compartment begins to show some correlation with an R2 value of 0.36. The final drives show the best correlation to Fe, with an R2 value of 0.58. The differences seen in these correlations could have been caused by particle size differences. It is conceivable that an engine or hydraulic system would produce smaller particles than a powershift transmission or a final drive. This data indicates that the accuracy of the PQ number increases with higher PQ values. It also may indicate that PQ has a better correlation to Fe when there are potentially larger particles in the oil. Repeatability The repeatability of the PQ was evaluated by a series of controlled tests done with the PQ instrument at the Malaga lab. 41 samples were selected from the samples received for analysis. Compartment totals are shown in the chart below.
Compartment Engine Final Drive Powershift Trans Hydraulic System Total Number of Samples 10 14 8 9 41

The repeatability evaluation was done on the bottle method and on the pot method. 2 ml from each sample was decanted into an Analex sample pot for the comparisons. Each sample was then measured three times on the PQ instrument. The team set a repeatability target for this instrument at +/- 10% of the average.
PQ values within +/- 10% of Average based on 3 tests per sample
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Final Drives Transmissions Hydraluics Engines Bottles Pots

SOS Services
SM

The chart shows that the PQ results from the bottle method are more repeatable than the PQ results from the pot method. The chart also shows the best repeatability from final drive and powershift transmission compartments. The relationship between the average PQ values in bottles and pots is compared in the chart below. This slope of this line specifies that the bottle method will produce values that are 1.3 times higher than the pot method.
Bottles vs Pots - All Compartments
5000

PQ in Bottles

4000 3000 2000 1000 0 0

y = 1.28x + 44.06 2 R = 0.98

500

1000

1500

2000

2500

3000

3500

4000

PQ in Pots

The correlation between the two methods appears to be very good, with the R2 value of 0.98. However, the data that strongly influenced this correlation is from results where the PQ values exceeded 100. The red circle shows a grouping of PQ results that are lest than 100. The chart below shows the data from the red circle area. This grouping of data, with PQ values less than 100, shows no correlation.
Bottles vs Pots with PQ < 100 - All Compartments
120 100

PQ in Bottles

y = 1.11x + 44.77 2 R = 0.06

80 60 40 20 0 0 5 10 15 20 25 30 35 40

PQ in Pots

The team concluded that this poor correlation was caused by the repeatability of the instrument at low PQ values.

SOS Services
SM

Summary of Calibration, Accuracy, and Repeatability The team concluded that Analex provides a thorough instrument calibration at the factory. However, the laboratory calibration procedure does not contain enough data points. Additional calibration standards would be needed to perform a better laboratory calibration. The accuracy of the instrument cannot be validated since Analex does not provide the correlation between PQ number and the mass of ferrous wear debris. The instrument does show a reasonable correlation to spectrophotometer Fe ppm for final drive samples. The final drive samples have the potential to contain larger size particles and they generally generate higher PQ values. The repeatability of the instrument is better for the bottle method than for the pot method. The bottle method produces PQ values that are about 1.3 times higher than the pot method. The repeatability is only acceptable with final drive samples.

Worldwide Instrument Sales & Service Capability

Caterpillar SOS Services laboratories are located all over the world. Therefore, there must be adequate sales and service capability for instruments used in these labs. Analex PQ instruments are sold, distributed, and serviced by Kittiwake. The 6 Sigma team discussed the sales and service capabilities with Kittiwake representatives. Kittiwake maintains: 1. There is no major maintenance associated with the PQ instrument itself. The sensor area should be kept clean. 2. All sales and service are done from a central location in the United Kingdom. If a PQ instrument requires service, it must be sent back to the UK. 3. If service is required on an instrument, Kittiwake does not provide a temporary instrument. This level of service and support may not be acceptable to a large portion of the SOS Services labs.

SOS Services
SM

Summary of Results
The following list summarizes the results obtained by the 6 Sigma team regarding the evaluation of the Analex PQ instrument. Interpretation of PQ Results 1. Trend analysis or specific PQ guidelines, by compartment, are needed to evaluate PQ results. General guidelines are not acceptable. 2. PQ provides the most value in the evaluation of final drive samples. 3. PQ provided the least value in the interpretation of engine oil samples. Calibration, Accuracy, and Repeatability 4. Analex provides a thorough instrument calibration at the factory. 5. The laboratory calibration procedure does not contain enough data points for high level of confidence. 6. Analex does not provide data to check accuracy. 7. Accuracy is indirectly demonstrated by a reasonable correlation to spectrophotometer Fe ppm, for final drive samples. 8. The repeatability of the instrument is better for the bottle method than for the pot method. 9. The bottle method produces PQ values that are about 1.3 times higher than the pot method. 10. The instrument meets SOS Services repeatability guidelines for final drive samples. Service Capability 11. Kittiwake does not maintain a staff of service technicians for this instrument. Therefore, the instrument must be returned to Kittiwake in the UK for repairs. 12. Kittiwake does not provide a temporary instrument, if service is required on your instrument.

SOS Services Recommendations

1. Caterpillar SOS Services does not recommend the Analex PQ for the general analysis and interpretation of used oil samples. The 6 Sigma team sighted the following problems: A. B. C. Interpreters did not consistently utilize abnormal PQ results. The laboratory calibration procedure for the Analex PQ does not contain a sufficient number of data points. The repeatability of the instrument for engines, transmissions, and hydraulic system samples does not meet the +/- 10% guideline established by SOS Services.

SOS Services
SM

D.

The ability of Kittiwake to efficiently support and service an estimated additional 100 Anelex PQ instruments was in question. If an SOS Services laboratory has already purchased an Analex PQ, the instrument should only be used a supplement to the core program.

David S. Nycz Caterpillar SOS Services 30 January 2007