Testing CS3202 Slides

CS3202: Software Engineering Project
Quality Assurance, Testing Suriya Priya Asaithambi, ISS, NUS
LN Set #5
CS3202 Set#5 QA Testing
Why Do we Test?
Reflect & list top three reasons for testing:
Topics
1. Quality Assurance 2. Reviews 3. Test Planning 4. Test Strategies 5. Test Case Design 6. Test Automation
Software Quality Assurance

Goal: to achieve high-quality software product Quality definition:
Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that expected of professionally developed software.
Three import aspects of quality measurement:

Use requirements as the foundation Use specified standards as the criteria Considering implicit requirements
Assurance vs Control
Quality Assurance (Process Oriented)
A set of activities designed to ensure that a process is defined and appropriate Focus is on the process elements assuring that the required level of detail is present
Quality Control (Product Oriented)

A set of activities designed to evaluate a developed work product. Focus is on finding defects
Testing
Executing a system with the intent of finding defects. Testing is more of a quality control activity
Software Quality Assurance

Test & Quality Assurance starts early Each step in validated before the next Do it Right the first time ! approach.
TEST PLANNING & DESIGN
ANALYSIS
DESIGN
CODING
TEST
IMPLEMENT
QUALITY ASSURANCE ACTIVITIES
Spectacular Software Failures

LA Airport Flights Grounded (2007) A single faulty piece of embedded software, on a network card, sends out faulty data on the United States Customs and Border Protection network, bringing the entire system to a halt. Nobody is able to leave or enter the U.S. from the LA Airport for over eight hours. Result - Over 17,000 planes grounded for the duration of the outage Car Manufacturing (2011) 50 500 cars recalled after airbag-related software glitch. A glitch in the automaker's software design and testing approach, which meant that airbags for passengers in the right rear seat during a crash may not be deployed. Result - recall of 47 401 vehicles in the US and a further 3099 in Canada and Mexico. Errors in medical software have caused deaths. Details in B.W. Boehm, "Software and its Impact: A Quantitative Assessment, Datamation, 19(5), 4859(1973).
More on http://www.sereferences.com/software-failure-list.php and http://www.wired.com/software/coolapps/news/2005/11/69355?currentPage=all

Testing
To ensure that software meets requirements To find and eliminate software errors Testing may take 40% of development time! Have I done enough testing? Testing can only show the presence of errors testing is never complete and no amount of testing can prove program correctness
9
Testing Definition & Purpose

Software Testing
Is the process of executing a program or system with the intent of finding errors. [Myers 79] Involves any activity aimed at evaluating an attribute or capability of a program or system against its required results. [Hetzel 88]
Purpose of software testing

Detect faults Establish confidence in software Evaluate properties of software (such as reliability, performance, memory usage, security and usability)
10
Test Automation
Version Control
Software Delivery
Continuous Integration
Testing in the SW Life Cycle

Requirements Acceptance Test Planning System Specification High-Level Design System Test Planning Integration Test Planning Unit Testing Planning Acceptance Testing System Testing Integration Testing Unit Testing
Low-Level Design
Coding
13
Testing Lifecycle
1. Test plan identifies:
The scope of testing effort, The methodology and tools used in testing
2. Test case design 3. Execution of test cases and evaluation of test results
program requirements design spec's code

test case design
library of test cases

test case expected results statistics
debug
run test case
evaluate test resultstest results error
code
14
Black-box and white box testing

Black box testing:
Testing based on specifications, without looking into code
Does a program meet requirements? Are all the required functions provided and fully operational according to specs?
White box testing:

Testing the way code is implemented
Have all program modules been called? Have all statements been executed? Have all braches and control paths been covered?
15
Phases of testing
What and when should I test?
When to test parts of a program? When to test the entire program?
Testing from inside out:

Individual modules - unit testing Some modules together -integration testing The whole system validation/system testing
module m1 unit testing integration testing
complete system tested module
validation/system testing
module m2
unit testing
validated system
16
Why system testing is not enough?

System testing tells you about functions that failed Tracing the source of error from function that failed to detailed code is difficult
May not be clear where buggy code is By now you forgot how this code works
Check these links:

http://www.youtube.com/watch?v=Pq6LHFM4JvE http://www.codeproject.com/KB/architecture/onunittesting.aspx In CodeProject: End-to-end unitt esting is what we call system testing
17
The Testing Matrix

Support Development Critique Product
Per Functionality
Business Facing Technology Facing
Cross Functional
Usability Test Exploratory Test Property Test
Special Tools Manual
Customer Test Component Test Unit Test
Automated
Ref: The Testing Matrix, Refactoring Test Code (Book)
Four Test Quadrants

Business Facing
Q2: Automated & Manual Q3: Manual
Exploratory Testing Scenarios Usability Testing User Acceptance Testing Alpha/Beta
Support Development
Functional Test Examples Story Tests Prototypes Simulations
Q1: Automated
Unit Tests Component Tests
Q4: Tools
Performance Testing Load Testing Security Testing All the -lity Tests
Q1 - Technologyfacing tests that support the team Q2 - Businessfacing tests that support the team Q3 - Businessfacing tests that critique the product Q4 - Technologyfacing tests that critique the product
Critique Product
Technology Facing
Ref: Agile Testing (Book)
Software Testing Principles

1. Make sure product meets the needs of customer 2. Choose a subset of tests that can uncover the maximum number of errors 3. Every phase of SDLC should catch and correct defects at that phase, without letting them seep to next stage 4. Understand the rationale behind every test 5. Make sure that the tests themselves are not faulty 6. Testware need constant revising, with the intent of identifying new defects 7. The number of defects yet to be found increases with the number of defects uncovered. 8. Balance defect prevention and defect detection activities 9. Automation requires careful planning, evaluation, and training. It may not produce immediate returns.
20
Unit testing
The goal of unit testing is to take the smallest piece of testable software in the application, isolate it from the remainder of the code, and determine whether it behaves exactly as you expect A unit test is a piece of code that exercises another piece of code Each unit is tested separately before integrating them into modules to test the interfaces between modules. Unit testing has proven its value in that a large percentage of defects are identified during its use. You may need drivers and stubs to execute unit tests. The driver simulates a calling unit and the stub simulates a called unit. Even though the drivers and stubs cost time and money, unit testing provides some undeniable advantages. It allows for automation of the testing process, reduces difficulties of discovering errors contained in more complex pieces of the application, and test coverage is often enhanced because attention is given to each unit.
Adapted from http://www.codeproject.com/KB/architecture/onunittesting.aspx
21
Unit testing (cppUnit)

Units are functions and class methods White-box tests, written by developer
Before, during or just after writing code Is program logic correct in respect to specs?
Force program to execute:

each statement each branch (edge in CFG) important control flow paths (CFG) do loops terminate properly?
22
Test legal and illegal inputs (exceptions)
Test coverage
2 4
CFG
1 3 5
One test case can cover all statements: 1, 2, 3, 4, 5 Two test cases cover all branches: 1, 2, 3, 4, 5 1, 3, 5 Four test cases cover all control paths: 1, 2, 3, 4, 5 1, 3, 5 1, 2, 3, 5
1, 3, 4, 5
23
Test coverage
We cannot cover all the paths during testing:
No. of iterations 0 1 2 .... 10 N
No. of control paths 2 4 8 .... 2048 2**(N+1)
Try to identify and cover essential paths

24
Covering independent paths

Start with a test case for typical execution path Add test cases as long as they traverse at least one node or edge not traversed before
(1) a = 1; b = 1; (2) while (i < 100) { (3) c = i + 1; z = a; if (y < 0){ (4) a =2; b = 2; else { (5) (6) i = i + 1; y = y + c; } (7) z= a + b
1 F 2 T 3 T 4 6 7 F 5
a = 3; b = 3; }
25
Sample test cases

Test case 1, purpose: skip the loop inputs: i >= 100; any value for y expected results: a = 1; b = 1 Path covered: 1, 2, 7 Test case 2, purpose: execute loop with y < 0 inputs: i < 100; y < 0 expected results: (a = 2 or a = 3) and (b = 2 or b = 3) Path covered: 1, 2, 3, 4, 6, 2, ..., 7 Test case 3, purpose: execute loop with y >= 0 inputs: i < 100; y >= 0 expected results: (a = 2 or a = 3) and (b = 2 or b = 3) Path covered: 1, 2, 3, 5, 6, 2, ..., 7
26
Unit testing, cont.

When to (re-)run unit tests?
As soon as you can compile code of a unit After changes done to a unit
What if changes of unit A may affect other units B, C, ?

Have test cases whose execution invokes A, B, C
Defensive programming helps in testing
27
Test Case Design Methods

Functional Analysis Equivalence Partitioning Boundary Value Analysis Cause Effect Graphing Conditions Testing Loop Testing Variable Coverage Testing User Scenario Testing
28
Functional Analysis
Test the system based on external specification Based on simple concepts
What is the system expected to do How do we determine that it is doing it
Requires for detailed analysis

Understanding of the system objectives Business processes System users
Develop Test Cases for

Valid input conditions Invalid and unexpected conditions
29
Equivalence Partitioning
Divides the input of a program into equivalence classes of data An Equivalence Class is a set of test cases such that any member of the class is representative of any other member of the class Guidelines for Equivalence Partitioning
If an input condition
Requires a value, then at least one valid (correct format) and perhaps two invalid equivalence classes (value absent, or present but incorrect format) are defined and/or Specifies as range, then at least one valid (within range) and two invalid (above and below range) equivalence classes are defined and/or Specifies a member of a set, one valid (within set) and one invalid (outside set) equivalence classes are defined [Note: there above not exhaustive there can be other additional classes e.g.. based on black box / white box logic]
30
Boundary Value Partitioning

Test at the extreme boundaries of both input and output equivalence classes. Guidelines for Boundary Value Analysis If an input/output condition specifies a range bounded by values a and b, design test cases with values a and b, just above and just below a and b. 3 additional test case to test random low value less than a, random mid-range value in between a and b and a random high value above b. If an input/output condition specifies a number of values, test cases should be developed to exercise the minimum and maximum numbers. values below the minimum Values above the maximum For internal data structures design test cases to exercise the data structure at its boundary.
31
Decision Tables
A decision table identifies a set of actions corresponding to a set of decisions. Each column in the decision table is a testable condition. Decision tables are appropriate when
The requirements are presented in a form that can easily be mapped to decision tables. The resulting decision table are not very big Each column in the table is a unique condition independent of the other column
Make sure all permutations are covered

If only Y/N decisions, then one way is to treat each combination as a binary number from zero up to max for the number of conditions e.g.. if 2 conditions, then 00, 01, 10, 11 (i.e.. NN, NY, YN, YY)
32
Cause and Effect Graphing

When a code unit is executed, its action may depend on
input data hidden state within the unit visible state outside the unit - the environment
Likewise, the action of the unit might affect

result returned hidden state external environment
The complete set of ingoing dependencies is the cause scope of the unit, and the outgoing dependencies are the effect scope. Cause-effect graphs connect input causes to output effects
33
Conditions/Variable Coverage
Conditional Testing Aims to exercise all logical conditions in a program Execute every branch at least once Test all combinations of simple or complex conditions Test that every value in a case statement is used Test that both results of a Boolean function are generated Variable Coverage Testing Tests that every variable is accessed All global variables are used by somebody All local variables are used Variables are set or initialized properly Output parameters are set before function exit This is the most difficult and expensive form of coverage analysis, and often gives false alarms
34
Loop Testing
Simple Loops of size n Skip loop entirely Only one pass through loop Two passes through loop m passes through loop where m<n (n-1), n and (n+1) passes through the loop Nested Loops Start with inner loop. Set all other loops to minimum values Conduct simple testing on inner loop Work outwards Continue until all loop tested Concatenated Loops If independent loops, use simple loop testing if two loops are concatenated and the loop for loop 1 is used as initial value for loop2 then the loops are not independent. Use the nested loop testing approach Unstructured Loops If possible redesign into one of the above categories
35
User Scenario Testing

Develop test case based on the perspective of a typical user
Think through how a typical user will be using the system day by day to perform his/her job functions
To develop the test cases address the following:

Who are the users? If there are multiple users to a system, is there a clear understanding of which feature will be used by which user? What kinds of scenarios and situations do they typically encounter on the job? How will the users really use the system? What features, displays, transactions and reports are the users likely to use? What are the common hassles which the users are likely to encounter in using the system?
36
Overall Test Strategy

Combine the methods you have learnt to come up with a overall effective test strategy An example strategy might be
Start with cause effect graphing, if specification contains several combinations input conditions Incorporate boundary value analysis Identify equivalence class both valid and invalid Examine programs logic using decision coverage and conditional coverage
Note: This is ONLY AN EXAMPLE!!!

37
Integration testing
Modules A and B were implemented separately
Will A and B work together ok?
Caller
Parser
Design Extractor
Query Evaluator CFG
Affects Which vars are modified in CFG node? Modifies
AST
Modifies
38
Planning integration testing

Have I tested all important module interactions?
Use sequence diagrams to plan integration tests

Refine sequence diagrams given in the Handbook
39
Running test cases

We must have a program that compiles
inputs testing
expected results real results evaluation test results
program
40
Test Execution Process

Retrieve Test Case Understand Test Case Evaluate Test Result Capture Test Result Monitor Interim Result Failure? Yes Set up Test Write Problem Report Retrieve Data Execute Test Document Test Result
No
41
Test Execution Criteria

Test Start Criteria
The system is delivered to the Test Group The testers are ready The Test Environment is ready
Criteria for Test Completion

Passes have been logged for all test cases If some cases have not passed the consequence of failing is insignificant The important defects have been fixed and retest has passed A comprehensive regression test has been executed successfully Test productivity falls off with further testing No or very few errors found on testing further
Test Suspension & Resumption Criteria

Suspension Criteria specifies the circumstance under which any further testing is paused Too many defects Environment not reliable Severe failure Resumption criteria specifies when testing may resume after a suspension
42
Testing incomplete program

To test, we must compile and run a program
Caller
incomplete program
If A and B are not available, write test stubs to simulate the behavior of A and B If Caller is not available, write test driver
43
Validation/System Testing
Testing a complete system In SPA, different types of queries form system tests
Inputs: source in SIMPLE, query Output: query results
Auto-Tester automates system testing

Design tests once - execute many times Quick validation after changes
Accumulate test cases and execute them often throughout a project

--- The End ---
44
TA Defined (IEEE)
Use of software tools to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Automated testability is the degree to which the application under test facilitates the implementation, execution and maintenance of automated testing Automated testability is about interfaces: Between software under test and test software Between requirements and implemented features Between developers and testers
45
Coping With Development

Development time Version 1.0 5000 days Version 2.0 2500 days Version 3.0 2000 days
Test time Version 1.0 5000 days Version 2.0 + 1.0 * 2500 + 2500 days Version 3.0 + (2.0 + 1.0) * 2000+ 1250 + 2000 days
* Impact Analysis testing

46
When Should a Test Be Automated?

Testing is done through an external interface ("black box testing")
Design the test first and the decide if it needs to be automated. Consider time and life cycle factors before modeling The organization has fixed maturity level of automation support and libraries.
Complete Automated Testing to Complete Manual testing is a spectrum.

A choice is usually governed by requirements of the test case Both automation and manual testing are plausible
47
Automation of testing
Get frequent feedback Get feedback asap after code has been written
First, unit tests find problems early in the development cycle. The common wisdom in software development is that the earlier problems are found, the cheaper it is to fix them. An automated unit test suite finds problems effectively as early as possible, long before the software reaches a customer, and even before it reaches the QA team. Most of the problems in new code are already uncovered before the developer checks the code into source control. Second, an automated unit test suite watches over your code in two dimensions: time and space. It watches over your code in the time dimension because once youve written a unit test, it guarantees that the code you wrote works now and in the future. It watches over your code in space dimension because unit tests written for other features guarantee that your new code did not break them; likewise it guarantees that code written for other features does not adversely affect the code you wrote for this feature. Third, developers will be less afraid to change existing code. Over time, software systems become more and more change resistant because developers are reluctant to change old code. This is natural because when changing old code, there is always the risk of breaking it or some other part of the system through a side-effect.
48
Regression testing
Execute test cases after program modifications to see if no new errors introduced
Source program
Modify program
Update test library Autotester: Perform regression testing Library of test cases
49
Regression Test Selection

Coverage Retest All Strategy Criteria Minimization Risk/Safety Regression Test Selection Unit Sequence Integration System/Acceptance
Regression Test Selection
Test Case Prioritization
Execution Reports
Random Ordering Criteria Coverage Ordering Fault Exposing Potential
Hybrid Approach
50
Autotester regression testing tool

Runs your program Feeds the proper input into it Checks the actual results against the expected results Prints a log of test cases that failed You can Autotester overnight and check results in the morning
51
Documenting test cases

The purpose of a test case and description Required inputs to a program Expected results produced by a program Any other requirements for running a test case
We want to run the same test cases throughout the project:

Sometimes we modify a program to fix an error and at the same time introduce other errors!
52
Test Documentation
Typical Test Documentation consists of
Test Plans Test Specification/Test Case Test Logs Test Status Report Error/Problem Reports Test Case Documentation
Test case ID Test Case Description Pre-condition required for test Test Input Data Expected Result Test Procedure/ Test Script Steps to execute a test case
Test Execution Log

Test case ID Test Date & Time Test Run # Tested By Actual Result Observed Test Result Evaluated By Test Result (Pass/Fail) Severity Level Test Time & date Test Decision (Suspend/Continue/Resume)
53
Recommendations
the developer checks in the source code - including the unit test code - to the source version control system. It is vital that the unit test code is checked in to the version control too. This allows integrating the unit tests with the build. It also allows other developers to check out the unit tests on their local machines and run them at will. before a developer checks in any code into the main source control repository, she should get the latest versions of the unit tests from the version control on her development machine and run all the tests locally. Only after all unit tests pass should the developer check in new code. This ensures that the source code in the version control system remains healthy at all times and can be used to rebuild the system at any time. to get the full benefit, test should be run as part of the build process - this is what is meant by automation of testing The build along with the tests should be scheduled to run automatically once or twice a day. If you have tests but do not run them as part of a scheduled build, then you are not getting the full benefit from the tests. By running the tests as part of a scheduled build, you test early and often.
54
Test planning an example

1. Team members will be assigned various testing roles and responsibilities 2. All test cases will be documented and stored in test case library for ease of execution repeatedly throughout the project 3. The source code and the test code will be checked in to the source version control system 4. All (or selected types of) test cases will be executed as part of the build process 5. An SPA build will be produced with or without test cases based on parameter setting 6. The build with tests will be scheduled to run automatically daily
55
Test planning, cont.

7. Extensive unit testing will be performed during and just after development of each module
Stubs will be developed when necessary
8. Overall planning of integration testing: to be done before module implementation

Sequence Diagrams to be used to decide which groups of interacting modules should be tested together
Notice that these decisions should be considered in project development planning
8. Design of integration test cases: to be done before module implementation 9. Execution of integration test cases: to be done as soon as groups of interacting modules are available and re-run daily
56
Test planning, cont.

10.System test case design
Will be an on-going effort during the project All team members will contribute test cases Test cases will be classified so that relevant test cases can be selectively extracted for execution when appropriate
E.g., we should be able to test Follows and Parent separately from other relationships
11.System test case execution

Selected test cases to be automatically executed and validated throughout the project
12. Tools: cppUnit, Autotester, Sequence Diagrams, ..

57
Testing is expensive
How to reduce testing effort? Quality Assurance is not all about testing: Quality of design and code Code reviews
58
Test Automation Process

Plan Software Test Automation Design Test Automation Strategies & Solutions Select and Evaluate Available Software Testing Tools
Develop & Implement Test Automation Solutions Introduce and Deploy Test Automation Solutions Review and Evaluate Software Test Automation
59
Roles in Test Automation

Business Analyst Requirements Software Analyst Software Developer System Under Test Test System Test
Automation Engineer Automation Architecture Design Create Function Libraries Create Object Repositories Create Business Components
Analyst Test Developer
Subject Matter Expert
Drag Components to create tests in QC Test Engineer Configure Input/Output parameters Debug tests in QC Test Plan module Add tests to test set in Test Lab module
or
60
Strategies Guidelines
Test automation is a fulltime effort, not a sideline The test design and the test framework are totally separate entities
The test framework should be applicationindependent The test framework must be easy to expand, maintain, and perpetuate The test strategy/design vocabulary should be framework independent The test strategy/design should remove most testers from the complexities of the test framework
61
Types Of Frameworks
Test Script Modularity
requires the creation of small, independent scripts that represent modules, sections, and functions of the application-under-test
Test Library Architecture

divides the application-under-test into procedures and functions instead of scripts
Data-Driven Testing
test input and output values are read from data files and are loaded into variables in captured or manually coded scripts
Keyword-Driven or Table-Driven Testing

requires the development of data tables and keywords, independent of the test automation tool
Hybrid Test Automation

combination of all of the above techniques
62
Best Practices Summary
SUMMARY
63
Best Practices
People
Testers need to be involved early in the process Objective reviews (formal or informal) greatly complements testing Close coupling of developer and tester have significantly improved quality of software Program owners cannot be the testers themselves
Process
Risk assessment based testing yields high ROI Subset of tests that have a high probability of finding bugs are selected Incremental integration testing adds value than big-bang approaches It is important to break application under test (AUT) into smaller functional modules and map related tests. This will aid in effective regression testing Automate areas that are less prone to changes
Technology
Fully integrated test case DB, SCM and defect repository is quite useful Tool support for metric calculations are helpful
64
SUMMARY
Testing is the process of executing a program with the intent of finding errors Different types of tests are done at different phases of the Life Cycle The questions to consider before deciding on a strategy are
What is the test objective? Which technique will be best for attaining the objective? Am I comfortable in applying the technique?
Good test planning and management is an integral part of project management

Testware must be adequately reviewed and kept under proper configuration control
Test automation requires a collaborative effort. Costeffective test automation calls for automated testability
65
Practical Testing
Some Tips & Pointers Suria R Asai (Email: suria@nus.edu.sg), ISS, NUS
Slide 1 of 15
SOC CS3202 SEP NUS
2/12/2012
White Box Testing (UT, IT & ST)

CPPUnit, TestRunner Vs Google C++ Testing Framework
Slide 2 of 15
SOC CS3202 SEP NUS
2/12/2012
Unit Testing Frameworks Available:
According to wikipedia:

http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks# C.2B.2B 50+ Frameworks http://www.opensourcetesting.org/unit_c.php 38 Frameworks Noel (2004): http://gamesfromwithin.com/exploring-the-c-unittesting-framework-jungle Alex Ott (2011): http://alexott.net/en/cpp/CppTestingIntro.html
SOC CS3202 SEP NUS 2/12/2012
According to open source testing tools:

According to expert blogs:

Slide 3 of 15
Test Automation Option A
Option A: CPPUnit & TestRunner

Author Micheal Features
http://www.objectmentor.com/omTeam/feathers_m.html http://cppunit.sourceforge.net/doc/lastest/cppunit_cookbook.html Advantage: clarity, less time to learn Disadvantage: lengthy fixtures Unit Tests Integration Samples Selective Testing (TestRunner)
SOC CS3202 SEP NUS 2/12/2012
Easy to Use; Cook Book Available
Similar to JUnit;

Sample Demo

Slide 4 of 15
CPPUnit Framework Components

Assertions

Conditions to be checked Combine assertions; Based on some common functionality At least positive, negative and zero test case Combines several tests Logically related to each other Provides setup of data or state, needed to run test Defines cleanup of state and/or data after test is executed Types: per-test case, per-test suite, and/or global fixtures Monitors: Displays how tests are executed, and collect data about failed test TestRunner: Useful to run test macros or suites or monitors
Test cases:
Test suites:

Fixtures:

Frameworks:

NOTE: Understand the difference between failure and error

Slide 5 of 15 SOC CS3202 SEP NUS 2/12/2012
Test Automation Option B
Option B: Google C++ Unit Testing Framework

Another google initiative
http://code.google.com/p/googletest/ http://code.google.com/p/googletest/wiki/Primer Set up video: http://www.youtube.com/watch?v=mzSzwQOmMRs
Getting Started Support

For Natural Extension for CPPUnit; Whole stack for Unit, Integration and System Test Big Community Followers and Help Sample Demo

Advantages: finer control; skips tests failed and continues; easy set up & tear down; more Configurations and platform support Disadvantage: more things to learn; set up little tricky; MSVC++ has bugs and hence static dll based projects suffer[read bug reports]
SOC CS3202 SEP NUS 2/12/2012
Slide 6 of 15
Google C++ Testing Framework Components
Assertions

Conditions to be checked success, nonfatal failure, or fatal failure Uses an assertion Combine one or more tests; Based on some common functionality At least positive, negative and zero test case share common objects and subroutines for test cases Defines cleanup of state and/or data after test is executed
Test

Test cases:
Test Fixtures

Test Program: Can contain multiple test cases Advanced Facilities
Advanced assertions, death tests, test result formatting, parameterized tests, advanced command line test selection, repeat, shuffle and format
NOTE: Understand the difference between failure and error

Slide 7 of 15 SOC CS3202 SEP NUS 2/12/2012
Comparison Chart
Characteristics Work smoothly in VS2010 External Dependency UT, IT and ST Support Assertion Macro Support Test Selectable from command line Output formatting Handling Failure Advanced Features
Slide 8 of 15
CPPUnit Yes; Simple project settings None Yes; Complete Complete Set No Text & UI Aborts current test and continues with rest of test TestRunner; Fixtures;
SOC CS3202 SEP NUS
Google C++ Testing Framework Yes; Caution regarding static DLL projects None Yes; Complete Exhaustive Set Yes; Customizable Text, UI,XML & HTML Non fatal assertions can be used to test even after failure Fixtures; Mock; Templates; Specialized data types;
2/12/2012
Black Box Testing (Acceptance Testing)
Slide 9 of 15
SOC CS3202 SEP NUS
2/12/2012
Acceptance/System Test
Option A - AutoTester from NUS SE Labs

record/playback user actions Simple programs run inside VS
Option B - AutoTester from Google

record/playback user actions on webpages and windows applications Simple programs run inside
Slide 10 of 15
SOC CS3202 SEP NUS
2/12/2012
Test Management
Slide 11 of 15
SOC CS3202 SEP NUS
2/12/2012
Example Tools

JIRA (Enterprise is Paid) BUGZILLA (GNU) TEAM FOUNDATION SERVER (MS) XSTUDIO (Free) QMETRY (PAID) MOCK FRAMEWORKS

OPMOCK bMOCK Googlemock (for Google Testing)
Slide 12 of 15
SOC CS3202 SEP NUS
2/12/2012
Best Practices For Student Project
Slide 13 of 15
SOC CS3202 SEP NUS
2/12/2012
Best Practices: Testware and Process
Create One Solution & Three Projects

For the project (Say XXXXProject) For White Box testing (Say XXXXProjectWBTests)
Includes unit and integration tests Test fixtures (per test case or suite) are designed Appropriate sequencing is designed for scenarios Framework options may be CPPUnit or GCTF Includes system and/or acceptance tests Use AutoTester Note: Technically all tests can be in one project, but separating WB and BB gives more clarity during regression testing
For Black Box testing (Say XXXXProjectBBTests)

Testing Process

Keep documentation simple; Use tools to help not complicate; Minimum you need high level TC design (aka excel)
SOC CS3202 SEP NUS 2/12/2012
Slide 14 of 15
Best Practices: White Box Testing(UT&IT)

Unit Tests are independent, clear, short, unique & automated While testing complex classes use mock objects
Example: To test class A you need class B
Solution: Create a fixture or mock class B
Setup and Teardown

setUp() create the describe procedure tearDown() remove it (zero residual side effects) Test for things that may be null Test for empty strings Test all but the simplest properties http://googletesting.blogspot.com/ http://code.google.com/p/googletest/wiki/FAQ
SOC CS3202 SEP NUS 2/12/2012
Test for right things

Read more at:

Slide 15 of 15
Best Practices: Sequence Of Testing.

Create ample test cases and related fixtures Start connecting them into sequences using Testrunner or Google test While creating the module (first time):

Run all unit tests; upon successful completion proceed Run all integration Use bottom up or left-right integration approach Selected affected test cases and code new suites Run unit tests first followed be relevant integration suites Run the relevant system tests
While regression or subsequent changes in iteration:

Finally after every iteration
Upon successful final release - package and release the project

ALWAYS: Unit tests before Integration ALWAYS: Run the System/Acceptance Tests as last step
Slide 16 of 15 SOC CS3202 SEP NUS
2/12/2012

Testing CS3202 Slides

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Testing CS3202 Slides

Diunggah oleh

Hak Cipta:

Format Tersedia

CS3202: Software Engineering Project

Quality Assurance, Testing Suriya Priya Asaithambi, ISS, NUS

CS3202 Set#5 QA Testing

Software Quality Assurance

Three import aspects of quality measurement:

Quality Control (Product Oriented)

Software Quality Assurance

QUALITY ASSURANCE ACTIVITIES

CS3202 Set#5 QA Testing

Spectacular Software Failures

More on http://www.sereferences.com/software-failure-list.php and http://www.wired.com/software/coolapps/news/2005/11/69355?currentPage=all

Testing Definition & Purpose

Purpose of software testing

Testing in the SW Life Cycle

CS3202 Set#5 QA Testing

program requirements design spec's code

library of test cases

run test case

evaluate test resultstest results error

CS3202 Set#5 QA Testing

Black-box and white box testing

White box testing:

Testing from inside out:

complete system tested module

Why system testing is not enough?

Check these links:

The Testing Matrix

Customer Test Component Test Unit Test

Ref: The Testing Matrix, Refactoring Test Code (Book)

Four Test Quadrants

Functional Test Examples Story Tests Prototypes Simulations

Software Testing Principles

CS3202 Set#5 QA Testing

Unit testing (cppUnit)

Force program to execute:

Test legal and illegal inputs (exceptions)

No. of iterations 0 1 2 .... 10 N

No. of control paths 2 4 8 .... 2048 2**(N+1)

Try to identify and cover essential paths

Covering independent paths

Sample test cases

Unit testing, cont.

What if changes of unit A may affect other units B, C, ?

Defensive programming helps in testing

CS3202 Set#5 QA Testing

Test Case Design Methods

CS3202 Set#5 QA Testing

Requires for detailed analysis

Develop Test Cases for

CS3202 Set#5 QA Testing

Boundary Value Partitioning

CS3202 Set#5 QA Testing

Make sure all permutations are covered

Cause and Effect Graphing

Likewise, the action of the unit might affect

CS3202 Set#5 QA Testing

CS3202 Set#5 QA Testing

User Scenario Testing

To develop the test cases address the following:

Overall Test Strategy

Note: This is ONLY AN EXAMPLE!!!

Query Evaluator CFG

Affects Which vars are modified in CFG node? Modifies

CS3202 Set#5 QA Testing