Anda di halaman 1dari 6

Software Structural

Testing Methods
By George N. Brower
Analex Corporation

S
tructural testing encompasses levels discussed below.
three critical phases of soft- ❝Structural Structural testing should be done
ware development and test- at the unit, integration, and system
ing; yet, one or more of these phas- testing assures the levels of testing.2 (As used herein, a
es is often deliberately bypassed, “unit” is the smallest separately
overlooked, or performed in a less
program’s compilable – or equivalent – ele-
than rigorous manner because either statements and ment of code, such as a procedure,
the technical advantages are not subroutine, class, method, or data-
fully considered or, more often, the decisions are base table.) Structural testing
cost and schedule benefits are not assures the program’s statements
appreciated. While structural test- fully exercised and decisions are fully exercised by
ing is required by the FDA for code execution. For example, it
medical devices of moderate and by code confirms that program loop con-
major level of concern, this testing execution.❞ structs behave as expected at their
1

should be done for all software, data boundaries. For configurable


regardless of the level of concern. software, the integrity of the data in
Also, it should be noted that there is no fundamental configuration tables are evaluated for their impact
difference between structural testing of software on program behavior. At the unit level, structural
used in a medical device and that used in a manu- testing also includes the identification of “dead
facturing process or a manufacturer’s quality system code,” which is code that cannot be reached for exe-
(or, for that matter, in any other software). An under- cution by any code pathway.
standing of these considerations and, thus, the Integration structural testing should be performed
importance of performing structural testing, are dis- after all verification testing of the units involved and
cussed below. before system-level structural testing. Figure 1 illus-
trates the general relationship between software ver-
Definition of Software Structural Testing ification and validation. Software verification con-
firms that the output of each phase of software
Software structural testing is meant to challenge development is true to (i.e., is consistent with) the
the decisions made by the program with test cases inputs to that phase. Performance qualification test-
based on the structure and logic of the design and ing confirms the final software product, running in
source code. Complete structural testing exercises its intended hardware and environment, is consistent
the program’s data structures (such as configuration with the intended product as defined in the product
tables) and its control and procedural logic at the test specifications and software requirements.

Journal of Validation Technology 37


George N. Brower

Figure 1 Figure 2
Software Verification Versus Configuration for a
Software Validation Testing Structural Unit Test

Driver Simulating
Concept “Compute Y”

(Unit-Under-
Product Test)
Specifications Get Formatted
ABC
Verification

Software
Requirements
Stub Simulating
Verification Stub Simulating “Notify Operator of
“Format ABC” Invalid Data”
Design Performance
Qualification

Verification Unit-level structural tests can be conducted on


Source Executable the actual target hardware, on an emulator or sim-
Hardware
Code Code ulator of the actual hardware, or, if the situation
Verification
requires, on a totally different processor. The latter
case may occur, e.g., if the actual hardware has no
(Test) provision for determining the results of a unit’s
tests, but where the code is written in a higher
order language. Thus, the higher order source code
(such as C or C++) can be compiled and linked to
Maintenance run on another computer that supports reading the
test results where the target computer (for exam-
ple, an embedded microprocessor) could not sup-
Unit-Level Structural Testing port the tests.
Structural testing (a.k.a. white-box testing) is per-
Figure 2 illustrates a typical configuration for a formed with the item-under-test, in this case the unit,
structural unit test. The unit is compiled and linked being viewed internally for purposes of determining
with a driver and stubs, as needed. The driver is a how the item should behave – for example, in deter-
substitute for any actual unit that will eventually call mining all possible code branches. The primary pur-
the unit-under-test, and if the driver passes data to pose of unit-level structural testing is to verify the
the unit-under-test, it is set up to pass test case vari- code complies with the design, including logic,
able values such as maximum, minimum, and other algorithms’ correctness, and accuracy (versus paral-
nominal and “stress-test” values. The stubs are sub- lel code or hand calculations), and the correctness of
stitutes for any units called by the unit-under-test. As the data’s engineering units. This requires, for each
with the driver, if the stubs return data to the unit- unit, complete branch tests, complete code tests
under-test, they also pass “stress test” and nominal (including verifying there is no dead code), and
data values, as appropriate. The interface of the driv- stress tests (to detect, e.g., overflow and underflow
ers and stubs, including their names, are the same as conditions as well as nominal and maximum loop-
the true units’ interfaces, allowing the set of units to control data values). The detailed design is used to
be linked without altering the unit-under-test. develop the acceptance criteria.

38 Journal of Validation Technology


George N. Brower

The Environment for Both Following completion of each functional package’s


Integration-Level and System-Level test, the next functional package may be either sepa-
Structural Tests rately tested or added to (i.e., linked with) the previ-
ously tested package(s). Regression testing (i.e., run-
It is best to set up the integration and system ning a selected subset of previous, successfully run
structural tests using the actual hardware and envi- test cases) must be performed on the previously test-
ronment to the extent practical. There are several ed packages to confirm they are not adversely affect-
reasons for this, but the two most significant are (a) ed by the newly introduced functional package.
the software may have subtle conditions, both good Figure 3 illustrates a building-block, or incre-
and bad, that will only show up when running on the mental, approach to structural testing. While the fig-
actual hardware, and (b) the final computerized sys- ure’s illustration is related to a structured design, the
tem, including the intended hardware and software, same approach is used for all other design methods,
must be qualified running on that hardware, and the including flowchart and object-oriented design. In
structural tests should advance the software devel- Figure 3, the first function needs two stubs, “Get
opment towards that end. However, there are also Formatted ABC” and “Output Y to Device X,”
good and sufficient reasons to perform structural (where a stub is a simple software dummy needed to
tests partially or wholly in a simulated environment. link successfully, but is not part of the software
In considering establishing simulation capabilities, being tested). The stubs in the figure have no calls to
the two most common configurations are to either additional units. The second and third functions
emulate the computer and simulate the environment require no driver (because they use the actual
(used most often when the actual computer is an “Compute Y”) and they require no stubs (because
embedded microprocessor and it is difficult to stim- they use the actual “Output Error Messages” unit).
ulate known inputs and/or read the outputs of a test)
or to simulate both the computer and the environ- The Incremental Approach
ment (used, e.g., if an emulator of the target com-
puter is not available). The principal advantages, The incremental approach to integration-level
then, in using a simulation of the environment and, structural tests is the best for software developers (as
at times, the computer include the following: (a) the opposed to third party, validation testers – see below),
ability to set up absolutely known input values, such especially if the program is large or complex. In this
that the results can be predetermined to establish the approach, selected small, functionally cohesive por-
acceptance criteria of each test; (b) a simulator tions of the software are compiled, linked, and test-
makes it easy to establish inputs that are over, under, ed. This approach is used regardless of the software
and at the exact limits of critical data values; (c) it is life cycle development method being employed,
easy to set up illegal inputs to test all error and fail- including any of the following three methods. In the
ure conditions; and, finally, (d) the results of each waterfall method, all of the requirements are devel-
test can be readily seen. oped, then the design is completed, and, finally,
selected “threads” are coded and structurally tested.
Integration-Level Structural Testing In the spiral method, a major element of the software
system is discussed and then the requirements,
Integration structural testing combines functional- design, and code are developed, and the element’s
ly cohesive units of verified code (which includes structural test is performed prior to going on to dis-
unit-level structurally tested code) by compiling cuss and develop the next major element. Finally, in
and/or assembling and linking the code, along with the incremental software development method, all of
any drivers and stubs needed. The structure is then the specifications and requirements may be devel-
loaded into the actual or simulated environment for oped, but the design and implementation are devel-
execution. This allows the tester to focus on that one oped one function at a time.
functional package to confirm its correct operation, In any case, if the operating system was unique-
including all internal and external interfaces. ly developed for the system-under-test, that operat-

Journal of Validation Technology 39


George N. Brower

Figure 3
Incremental Integration Structural Testing

Compute Y

Get Formatted Output Y to


Initialize Data
ABC Device X

Notify Operator
Open Files if Device X
Format Notify Operator is Off-Line
ABC of Invalid Data
Notify Operator
2nd Function 3rd Function
of Any Errors

1st Function

Output Error
Messages

ing system should be structurally tested first. This utilities for the next structural test, and so on. All
portion of the code itself should be broken down previously tested functions should be regression
into functionally cohesive packages if it is large tested, as appropriate.
and/or complex; otherwise, it can be structurally
tested as an entity. The second portion of the code The Nonincremental Approach
to be tested is normally the unique input/output
section. If there are diverse input/output devices, As used herein, a nonincremental approach is one
these may be structurally tested separately. But it is in which the complete software package, as it is
often best to select a thread that includes both the intended to be deployed, is totally built and then
ability to input data and to see the resulting output structurally tested. The nonincremental approach to
for each structural test. The third step is to select integration-level structural tests may reasonably be
and structurally test a functionally cohesive portion used for small programs and is often the best ap-
of the application and the utilities needed to sup- proach for third party, independent testers. Although
port that application. (Remember: The units select- in most circumstances integration-level structural
ed for integration structural testing, and this is testing is the purview of the software developers and
especially true for the utilities, should be verifica- not, for example, that of the computer system vali-
tion tested first – which includes unit testing – prior dation team, at times these third-party testers are
to any integration-level structural testing.) Then, asked to support a program in this manner. An
select the next functionally cohesive portion of the immediate problem that most often surfaces, howev-
application software and associated application er, is due to budget, schedule, and/or personnel con-

40 Journal of Validation Technology


George N. Brower

straints, testers independent of the software develop- both using the product specifications (which in turn
ment team cannot, in a cost-effective manner, select, may include the user’s manual and other labeling)
compile, and link individual software functions. and software requirements for the test development
However, they can build and load the fully integrat- and the acceptance criteria. Although, at the system
ed program, select and test a function (ignoring the level, a knowledge of the design may be necessary
remainder of program while maintaining a structur- to determine how to set up a test for a specific func-
al-tester’s point-of-view), and then select the next tionality (for example, an output error message that
function for test. One advantage to this approach is a file cannot be found versus an output error mes-
since a majority of the code may be executing dur- sage indicating there is invalid data in a file), that
ing each test, the need for regression testing of the design knowledge should not be used in construct-
previous functions is diminished. A disadvantage to ing the acceptance criteria of the tests – such as the
this nonincremental approach is if a function is boundaries of legitimate data. That information
found to not work properly, finding the cause of the should be derived from the specifications and
problem may be more difficult precisely because so requirements.
much of the code is executing.
At the integration level, structural testing uses the The Importance of Each Testing Phase
architectural high-level design to develop the test
cases and to derive the acceptance criteria. (During As with each verification, integration, and qual-
the integration test phase, functional, black-box test- ification activity, each structural testing phase finds
ing should also be performed.) unique problems that otherwise may be either very
difficult to find at best or, at worst, may be left
System-Level Structural Testing undetected during pre-release testing, waiting for
the error to surface in the field under actual use.
The system-level structural tests must be run on Using a step-wise approach to testing finds prob-
the fully integrated software package. However, lems and solutions faster and with a greater guar-
whether the software functions for the integration antee of producing a solid software product than if
structural tests discussed above are built incremen- any phase is skipped or not performed with rigor.
tally or all at once prior to conducting the integration In this context, rigor means (a) planning all tests
structural tests, the system-level structural tests must and reviewing the plans for completeness, correct-
still be run. The primary differences between inte- ness, and consistency; (b) writing the procedures
gration level and system level structural tests are the sufficiently well so they may be reviewed and,
tester’s frame of reference for developing the test when necessary, rerun (e.g., following software
inputs and determining the acceptance criteria of the updates); and (c) maintaining a written record of
test outputs. An important secondary difference in the tests performed and the results so those may
integration structural testing is that the code may be also be reviewed for confirmation that all tests
altered (e.g., to create test drivers and stubs) or the have been completed. 3 In performing the above
normal code execution flow may be altered (e.g., three tasks, program management is supported in
running the software in a de-bug compiler mode that all of the activities, from planning through
where the execution can be halted, a data value execution and reporting, can be scheduled, budget-
changed, and the execution restarted, allowing for a ed, and, therefore, managed.
variable’s maximum or minimum value, or a failure Figure 4 illustrates, for each structural test phase,
or other perturbation to be inserted). In system struc- the documentation against which the software is test-
tural testing, the code must not be changed, and the ed.1 In that figure, software development implementa-
program execution flow must not be altered. Rather, tion is shown on the left side, going from product
all test inputs must be accomplished by control of specifications to code implementation. Software
the actual or simulated environment. (structural) test is shown on the right side, going from
At the system level, structural testing is also best unit test to system test. The documentation for each
performed along with functional, black-box testing, test phase is illustrated by the horizontal arrows. This

Journal of Validation Technology 41


George N. Brower

general approach is followed for any software devel- of hidden, difficult-to-find software problems in the
opment and testing life cycle approach, be it waterfall, succeeding test phases that can easily disrupt any
spiral, or iterative. For example, if the spiral approach schedule and budget. ❏
is used, a piece of the product specification is deter-
mined, the software requirements for that piece are
developed, the high-level and detailed design, and About the Author
then the code for that piece are developed, and then the
testing is accomplished starting at the unit level, work- George N. Brower has published several articles
ing through the integration level, and finally working and has given presentations at numerous confer -
up to the system level. ences on his areas of expertise: software develop -
ment and software Independent Verification and
Validation (IV&V) testing. For the past 11 years, he
Figure 4
has served as Deputy Director of the Denver office
Structural Test Verification of Analex Corporation, providing software engineer -
Documentation ing services for computer systems validation and
outsource software development. Analex-Denver is
Product Specifications System-Level an ISO 9001 4 registered company that is the recipi -
and Structural ent of the 1995 Supplier of the Year Award for IV&V
Software Requirements Test testing and the 1996 recipient of the James S.
Cogswell Industrial Achievement Award. In 1998
Analex received a second award for IV&V testing of
Integration-Level critical software. Brower can be reached by phone
Top-Level Design
Structural Test toll free at 1-888-262-5391, by fax at 303-730-2057,
or by e-mail at brower@analex.com.

Unit-Level References
Detailed Design
Structural Test
1. Guidance for FDA Reviewers and Industry, Guidance for the
Content of Premarket Submissions for Software Contained in
Medical Devices, CDRH, May 29, 1998.
2. Guidance for Industry, General Principles of Software Val-
Implementation idation (Draft Guidance) Version 1.1, June 9, 1997.
(Coding) 3. FDA Current Good Manufacturing Practice (cGMP) Final
Rule; Quality System Regulation, 21 CFR Part 820.30, Design
Control, fully effective June 1, 1998.
4. International Organization for Standardization, 1994. (Analex
is registered for independent software verification and valida-
Again, the preceding discussion, as well as all tion, software development, systems analysis, and hardware
aspects of this paper, applies to all software applica- prototyping.)
tions, whether the software is for a process control
system, medical device, software supporting a qual-
ity system, or any other software, such as that used
for simulators or even business systems.

Summary

Structural testing allows a step-by-step, rigorous


test approach that easily uncovers problems at each
phase that may otherwise be very difficult to detect
during other software verification and validation
activities. Therefore, structural testing not only adds
value from a technical point of view, but it also sup-
ports programmatic issues, mitigating the possibility

42 Journal of Validation Technology