Software Testing

Practical Skills for Testing Practitioners

2009

Configuration Testing and Compatibility Testing

Yue Zhao Consulting Scientist Institute for Software Research Carnegie Mellon University

Objectives
Configuration testing and compatibility testing are two testing processes that are often ignored in many project teams. In this lecture we will explore:
What is configuration testing Configuration testing steps What is compatibility testing Compatibility testing steps

2009, CMU-ISR

What is Configuration Testing(1)

Definition
The process of checking the operation of the tested software with all various types of hardware. There are various configuration possibilities provided by different manufacturers that your software may run on. The important thing to remember is your software may run smoothly on one kind of configuration, but often crashes on another.

Why configuration testing is necessary

2009, CMU-ISR

What is Configuration Testing(2)

Why configuration testing is necessary.
Different software may tie to different configuration areas, for example: 1. 2. 3. A highly graphical computer game requires lots of attention to the video and sound areas. A greeting card program will be especially vulnerable to printer issues. A communication program will need to be tested with numerous modems and network configuration.

2009, CMU-ISR

Configuration Testing Steps

When to test configuration:
In general, we schedule configuration testing after the baseline of function testing in the system testing phase. Otherwise, it is much harder to determine whether the defect lies in software or hardware. Identify hardware combination Design the test cases to run on each configuration Conduct the tests on each configuration Fix defects and verify modifications Report results

Steps

2009, CMU-ISR

Identify Hardware Configuration(1)

The core task of configuration testing is to reduce hardware configuration combinations because:
It is not practical to perform a complete configuration testing A windows online game program support 350 possible display card 200 sound card, 800 modems 1200 printers

there should be 350*200*800*1200 combinations You may conduct a certain number of test cases on these different configurations. The number of total test cases could be larger.

2009, CMU-ISR

Identify Hardware Configuration(2)

Steps to size up your configuration testing:
Determine the types of hardware you need. Determine what hardware brands, models and device drivers are available. Determine which hardware features, modes and options are possible. Write down the identified hardware configuration. Identify your unique software features that work with the hardware configuration.

2009, CMU-ISR

Identify Hardware Configuration(3)

Determine the types of hardware you need by:
Asking yourself what kind of hardware you need. Identifying the critical hardware to your software. Examining your software feature carefully to ensure everything is included.

Determine what hardware brands, models and device drivers are available.
Determine proper devices Identify equivalent devices Determine device drivers included with:
1. Operation system 2. The device 3. Latest drivers available for the hardware or operating system

Hardware that your customers have

2009, CMU-ISR

Identify Hardware Configuration(4)

Determine which hardware features, modes and options are possible.
Each device has different options. Not all the device options should be supported by your software. Minimum configuration should be tested.

Write down the identified hardware configuration.

1. Classify the identified hardware by applying equivalence partitions. 2. Write down configuration information into a table, such as popularity, type, age, manufacturer, model, device driver version, options and so on. 3. Fill in the table with the hardware you get in step 1. 4. If needed, use decision-making processes to get a smaller set. 5. Review the table and decide which configuration best fits your project.
2009, CMU-ISR 9

Identify Hardware Configuration(5)

Identify your unique software features that work with the hardware configuration.
Dont try to test every software feature on each hardware configuration. In many cases it is a hard work to identify unique features, you may: 1. Walk through the product using the black-box method and pull out obvious ones. 2. Talk with project members, especially, the programmer.

2009, CMU-ISR

10

How to Obtain the Hardware

Getting all the identified hardware is hard work
It is expensive to buy everything. Sometimes the hardware will be used only once. It is hard to maintain different models, versions, drivers of hardware. Only buy the most frequently used configuration. Borrow or rent from the hardware manufacturers. Ask other project teams in your company for the needed hardware. If it is possible, build a configuration and a compatibility testing lab.

Approach

2009, CMU-ISR

11

Suggestions
You may adopt the following:
Identifying hardware standards to help you get deep insight about the hardware so that you can make more informed equivalence partition decisions. Answering the following questions may help you when performing configuration testing. 1. What external hardware will operate with this software? 2. What models and versions of that hardware are available? 3. What features or options does that hardware support?

2009, CMU-ISR

12

Design Test Cases

Design test cases to run on each configuration.
According to previous steps, you may design the details of test cases: 1. Configuration and features to be tested 2. Preconditions 3. Operation steps 4. Expected result

2009, CMU-ISR

13

Conduct the Configuration Tests

Identify configuration defects
Configuration defects are hard to identify because: 1. As a software engineer, you may believe your hardware vendor provides perfect products and the defect only lies in your program. 2. In most cases, it is hard for a programmer to find hardware problems because of the lack of domain knowledge. 3. You may be disappointed by huge amounts of configuration possibility combinations when trying to identify configuration defects.

Approach
Perform the exact same operation that caused the problem, step by step on another computer with a different hardware setup. If the defect doesnt occur, it may be that you have a configuration problem.
2009, CMU-ISR 14

Fix Defects And Verify Modifications

Who fixes the configuration defects
The configuration problems may occur because: 1. Your software may have a defect that appears under a broad class of configurations or only to one particular configuration. 2. The hardware device or its device drivers may have a defect that only your software reveals or can be seen with lots of other software.

Solution
Under the first situation, it is surely your responsibility to fix the defect. Under the second situation: The hardware vendor fixes the defect, or Provides a software version to get around the hardware defect.
2009, CMU-ISR 15

What is Compatibility Testing

Definition
The process of checking how the software interacts and shares information correctly with other software. The interaction could occur between:
1. Two programs simultaneously running on the same computer. 2. Two programs running on different computers connected through a network. 3. Simple as saving data to a floppy disk and hand-carrying it to another computer.

Compatibility testing is a test process much like configuration testing. It takes care of interactions with compatible software which should be defined in the specification. The software may interact with each other. If the interaction fails, of course, the function will also fail.

Why compatibility testing is necessary:

2009, CMU-ISR

16

Compatibility Testing Steps

When to test compatibility:
Like configuration testing, in general, we schedule compatibility testing after the baseline of function testing in the system testing phase. Otherwise, it is much harder to identify the compatibility defects. Most project teams combine configuration testing with compatibility testing and call it compatibility testing. Identify compatible software Design test cases for each compatible software Execute the tests Fix defects and verify modifications Report results

Steps

2009, CMU-ISR

17

Identify Compatible Software (1)

Approach
Analyze the specifications. Discuss with the technical stakeholders. Consider the following: 1. Platform and application versions: What platform, such as operating system, web browser, and other application is the software designed to be compatible with? 2. Standards and guidelines: What compatibility standards or guidelines should be followed that define how your software should interact with other software? 3. Data sharing: What types of data will your software use to interact and share information with other platforms and software?

2009, CMU-ISR

18

Identify Compatible Software (2)

Platform and Versions
The compatible applications and their versions can be found in the specifications and should be defined by managers or marketing members. Back and Forward compatibility: Dont forget the software itself may be back compatible or forward compatible or both. 1. Back compatible means the software will work with its previous versions. 2. Forward compatible means the software will work with its future versions. Multiple version: Like configuration testing, in order to reduce the number of compatible software to test, you must determine which ones are the most important to test: 1. Classify the identified software by applying an equivalence partition. (to be continued)
2009, CMU-ISR 19

Identify Compatible Software (3)

Platform and Versions
2. Write down compatible information into a table, such as popularity, type, age, manufacturer, version, options and so on. 3. Fill in the table with the software you get in step 1. 4. If needed, use a decision-making process to get a smaller set. 5. Review the table and decide which compatible software fits your project best.

Standards and Guidelines

There are two levels: 1. High level standards and guidelines: the ones that guide your products general operation such as its look and feel, supported features and so on. 2. Low-Level standards and guidelines: details of your products such as file formats, communication protocols and so on.

2009, CMU-ISR

20

Identify Compatible Software (4)

Data Sharing Compatibility
If data is shared among applications, you should test the data format according to the standards and protocols, and the following should also be taken into account: 1. File save and file load: data format of the files needs to meet standards. 2. File export and file import: use equivalence partition to create test documents in each compatible file format. 3. Cut, copy and paste: you should make sure that the application data can be properly copied in and out of clipboard to other programs. 4. DDE, COM and OLE are the windows methods to transfer data between two applications. These methods transfer data in real time and can occur automatically.

2009, CMU-ISR

21

Other Steps
About the remaining steps:
Design test cases for each compatible software. Conduct the tests. Fix defects and verify modifications. Report results. These steps are much like that of configuration testing. You can refer to previous slides.

2009, CMU-ISR

22

Summary
Configuration testing and compatibility testing are two similar testing processes. Configuration testing supports hardware while compatibility testing supports software. Many project teams combine configuration testing with compatibility testing. The common steps of the two testing processes are: 1. Identify needed hardware/software 2. Design test cases 3. Run test cases 4. Fix defects and verify modifications 5. Report results

2009, CMU-ISR

23