Design of Experiments Introduction and Applications

By: James Hurst

The Testing Process

The planning phase
Select which input variables to change Select which output variables to measure Determining the number of test point that can be executed
Ideally determined by how much information about the system is needed Realistically determined by the budget/schedule limitations of the program

Use DOE processes to select which test points to test

The Testing Process (cont)

The execution phase
Adamantly follow the test plan as laid out in the planning phase
Avoid changing the test plan for convenience as this may reduce the benefits of the DOE process

The reporting phase

Conclude the testing process by properly documenting all results, conclusions, and any interesting nuances about the system that we discovered during the testing A test not reported on is as good as test not done

Types of Testing
One Factor At a Time (OFAT): Change one input variable at a time in order to see how the output variable responds to each change
Simple and straightforward but does not show any compounding/negating effects when two input variables are changed simultaneously

Do what we did last time: When programs are updated or similar systems are developed, it is tempting to simply test the same way the previous system was tested
Experienced, Tried and true Lessons learned from last test can be directly applied to this test Information missed last time will be missed this time

Design of Experiment (DOE): Selection of test points based on statistics and engineering optimization

Design of Experiment (DOE)

The choice of test points to run is rooted in statistics and optimization This technique is most appropriate for large, complex systems or systems with stringent budget and schedule restraints
In complex systems it is impossible or impractical to perform every possible test point. Choosing which to test is crucial In expensive programs (such as flight testing or weapons testing), the cost is the driving force in minimizing the amount of testing necessary

The test must thoroughly assess the system but may have harsh limitations to how much testing can actually be performed DOE processes are critical to selecting which points to test

DOE Historical Background

Dr. James Lind, 1753: Experiment to cure scurvy
Scurvy was a common problem among the British Royal Navy while serving long term missions at sea Dr. Lind took 12 afflicted sailors, split them into groups of two, and gave each pair a different treatment (including garlic, vinegar, lemons, or oranges) Within a week those given citrus fruits were fit for duty

This is the first documented case of a controlled clinical experiment It is now known that scurvy is caused by a deficiency in vitamin C (hence why citrus fruits, high in vitamin C, cured it)

DOE Historical Background (cont)

Sir Ronald A. Fisher (1920s-1940s): Father of the DOE process
Conducted agricultural experiments in Great Britain Varied input parameters on some crops and compared them to control groups of crops Randomly assigned which crops would be in a given experimental group and which would belong to the control group
This allowed him to average out any uncontrollable parameters such as soil richness in a given area of the field

Laid the framework for factorial experiments (to be discussed later)

DOE Historical Background (cont)

George Box (1950s): First use of DOE in industy
Worked at a chemical industry company Wanted to vary input parameters to increase chemical yields Proponent of iterative testing
Testing produces results that will lead to more questions or opportunities to learn more about the system or process. These new questions lead to more testing.

Laid the framework for the response surface method process (to be discussed later)

DOE Historical Background (cont)

Genichi Taguchi (1970s): Applied DOE to quality improvement
Implemented DOE methods for process optimization and quality improvement Ideas and methodology made Toyota an international brand and jump started Japanese quality production Laid the framework for fractional factorial experiments (to be discussed later)

DOE Historical Background (cont)

Modern era: (1990-present): Companies develop specialized DOE processes
Motorola has developed the six sigma program to product optimization Many companies exist with the sole purpose of developing test plans using DOE strategies
These companies have proprietary methods of DOE and are paid by other companies to help optimize their testing process

DOE Basics Principles

Randomization: Randomly assigning which test point to assign to which test item or the order in which the points are conducted
Pioneered by Sir Ronald A. Fisher Eliminates bias by average out any uncontrolled variables For example: if testing ten altered systems and ten unaltered systems for the control group, the order in which they are tested should be random

Replication: Repeating a test point to add confidence to a certain conclusion or to make sure the systems response isnt an anomaly
Important to raise confidence in the test, especially when the results of the test are either controversial or important May be too expensive to replicate test points

DOE Basics Principles (cont)

Blocking: Eliminating uncontrolled variables or other known biases from the experiment
Minimizing uncontrolled variability makes sure the test results are as accurate as possible For example, in flight testing the testers may aim to have the weather as constant as possible to ensure changing weather conditions dont affect the test

Orthogonality: When no compounding or negating effects exist

For example, if an aerially employed munition loses accuracy at higher altitudes or higher Mach numbers but increasing both simultaneously does not compound the effect Is not required or commonly used but simplifies the analysis process

DOE Implementation
The value of DOE methods lies in its ability to help pick test points to conduct Ideally there would be an unlimited budget with unlimited time to conduct all possible test points to completely understand the system
Unfortunately budget and schedule restrictions limit the number of test points that can be completed DOE allows testers to conduct a thorough test even with these limitations

Popular DOE methods include factorial experiments, fractional factorial experiments, response surfaces, and space filling

Factorial Experiments
Invented by Sir R. A. Fisher, all selected input variables are changed orthogonally
For example, if X input variable are selected and two values of each input variable are to be tested, then a cube of X dimensions with 2X test points will be used
If three input variables are used with two values per parameter, then eight test points and a standard 3D cube represents the test points The figure below graphically represents this

Factorial Experiments (cont)

Factorial experimentation is a very systematic, simple approach Any number of input variables and any number of values for these variables can be selected
i.e. a large complex system may have 10 input parameters with two values each for a total of 210 total test points arranged in a cube of 10 dimensions For more precision these ten input variables could have more values for even more total test points

Factorial Experiments Example

Consider a chemical experiment in which the chemists want to test the yield of the reactions when changing temperature and pressure
Temperatures of 50C and 100C and Pressures of 1 MPa, 2 MPa, and 3 MPa Use blocking to the max extent possible to limit the effect of uncontrolled variables Test points in this full factorial experiment are shown in the table below
Test Point 1 2 3 4 5 6 Temperature (C) 50 50 50 100 100 100 Pressure (MPa) 1 2 3 1 2 3

Fractional Factorial Experiment

Factorial experiments can become extremely large and expensive as more input variables and more values for these input parameters are selected
Two input parameters with two values: 4 test points Five input parameters with two values: 32 test points Eight input parameters with two values: 256 test points

Genichi Taguchi developed the fractional factorial experiment in order to lower the number of test points while still keeping the systematic approach of factorial experiments Once the full factorial test points are selected, a geometrically selected subset of these points are conducted

Fractional Factorial Experiment (cont)

Test points from the full factorial experiment are selected to maximize diversity in the test while lowering test points to manageable number
Still has repetition inherently

Figure below shows test point selection

Fractional Factorial Experiment Example

Consider an aerially launched munition
Too expensive to launch missiles for every test point in full factorial test Assume altitude (10,000ft and 20,000ft), Mach number (0.5 and 0.75), and which side of the aircraft the munition is on (left or right) are selected input variables Table on the following page represents the full factorial experiment and highlighted rows represent a fractional factorial experiment

Fractional Factorial Experiment Example (cont)

Test Point 1 2 Altitude 10,000 10,000 Mach number 0.5 0.5 Munition side Left Right

3
4 5 6 7

10,000
10,000 20,000 20,000 20,000

0.75
0.75 0.5 0.5 0.75

Left
Right Left Right Left

20,000

0.75

Right

Notice in full factorial, both altitudes, Mach numbers, and munition locations are tested four time each Fractional factorial experiment tests both the altitudes, Mach numbers, and munition locations twice A smaller fractional factorial expereiment could be performed by using test points 1 and 6

Response Surface Method

Developed by George Box in the 1950s Goal is to derive an analytical expression for the output parameter in terms of the selected input variables
Allows testers to predict the systems response at any input parameter values Analytical expression is easy to maximize or minimize in order to optimize the systems performance

Number of test points depends on number of input variables selected and desired accuracy of the analytical expression

Response Surface Method (cont)

Response surface method allows optimization May require large number of test points as more input variables are selected When multiple output variables are selected it may become complicated Expression is only an approximation (only as accurate as the number of test points allow

Space Filling Method

Meant for systems with little to no variability
Computer programs or simulations Replication is unnecessary because the results will be identical to the original run if the system has no variability

Goal is to spread out the data points as much as possible in the area of interest
Specifically, the goal is to maximize the minimum distance between the test points (see figure on following slide)

Space Filling Method (cont)

Figure shows possible test points

Circles show how data points are spread out evenly Larger circles on the left signify that fewer test points will be run

Optimal Design Procedures

Goal is to minimize cost b intelligently selecting test points
Useful for fractionating large factorial experiments or picking test points for programs with large budget/schedule restraints

Uses statistics to decrease variability in the test points

Allows fewer test points to be selected while still providing the same amount of information

Users to make sure they understand the programs performing these operations rather than using it as a black-box

Optimal Design Procedures (cont)

Programs that perform optimal design procedures are typically proprietary to large companies or companies specializing in DOE
DOE specialized companies are hired by companies that require assistance in designing test programs Contracting out test designing like this may add cost to a program but is hopefully made up through test cost savings and a quality test design

Uses of DOE
DOE is most useful in large or expensive testing such as flight testing or testing where the test item is consumed or destroyed (such as weapons testing)
Flight testing costs ten of thousands in jet fuel, MX/support, etc Minimizing the number of test points can save time and money

Main uses of DOE include Product design/development, process optimization, and tolerance determination

Product design/development
DOE can be used to aid in the design or development of a new product During the design process, decisions must be made about component or process specifications DOE can help show how changing these component specifications (input variables) affect the purpose of the system (out variable)

Product design/development Example

Consider a turbine being developed
Number of blades, geometry of the blades, exit pressure, max temperature inside the turbine, working fluid, etc. can all be selected as input variables Max power or efficiency may be selected as out put variable Building/modeling and testing every possible design is impractical and much too expensive DOE should be used

Product design/development Example (cont)

Response Surface method may work well
Obtaining an analytical expression for the max power output would allow designers to maximize their design Selecting all desired input parameters and perform a full or fractional factorial experiment Using these data points a curve fit for max power can be computed Number of test points depends on desired accuracy of expression and number of input variables selected

Process Optimization
Just as a machine can be optimized, so can a process Parameters of the process can be changed (input variables) to see how the processs outputs are affected Consider a factory assembly line
Management wants to know which employees should be working to maximize the yield of the factory Simplest option is to use every combination of the employees to see which combination makes the most parts
Could take months or longer to run that many shifts

DOE processes can help solve this problem

Process Optimization (cont)

Response Surface methods can apply here
Each employee is an input variable while the number of parts built is the output variable Performing a full or fractional factorial experiment will gather data points to calculate this analytical expression for the number of parts built in terms of which employees are working Enforcing binary constraints on the variables (the employee can either be working or not) and maximizing this expression will answer managements question of the optimal assembly line they can have

System Tolerance Determination

During the designing of a system, physical tolerances are required for all parts to be built Looser tolerances are easier (and cheaper) to build but can cause the system to fail if they are too loose
Goal is to maximize tolerances while still having a system that works to some confidence level

System Tolerance Determination (cont)

Consider the manufacturing of a piston for a car (or any physical part)
The component will be built in a CAD model early on The worst case scenario size of the component would be its maximum tolerance A simulation can be built that operates a model of the system with varying sizes of the component being assessed Using this simulation and the size of the component as the input variable, the space filling DOE method could be used

Summary
Testing is a critical part of the designing and development process Testing gives knowledge about the system and acts to ensure the system will work as designed when used in the operational environment Testing can help in the designing of a system, in the optimization of a process, the determination of tolerances, and more

Summary (cont)
Testing can be extremely difficult and complex
Large/complex or expensive programs may require the testing to be done efficiently Budget or schedule limitations may require less testing than may be desired
Number of test points is often determined by budget and schedule considerations rather than the desired quality of the test plan

DOE methods and procedures should be used for these complex systems to maximize the effectiveness of the test

Summary (cont)
Design of Experiment is a statistical and mathematical approach to test planning
Often lowers the required number of test points while still offering a reasonably thoroughly and informative test Pioneered by individuals like Sir Ronald A. Fisher, George Box, and Genichi Taguchi Uses principles such as randomization, replication, blocking, and orthogonality

Summary (cont)
One of the most important applications of DOE is its uses in choosing which test points to compute. Major DOE methods include:
Factorial Experiments pioneered by Sir. Ronald A Fisher Fractional Factorial Experiments developed by Genichi Taguchi Surface Response methods used by George Box Space filling strategies for systems with low variability Complex techniques used by major companies in industry (such as Motorolas six sigma program) and companies specializing in Design of Experiment

Which method to use depends on the goals of the test and the complexity of the program

Summary (cont)
Implementing DEO processes in a program takes experience and should be tailored specifically to the program While it may be complex to implement, Design of Experience can be an invaluable tool in the testing of more complex and expensive systems