SOFTWARE QUALITY ENGINEERING versus SOFTWARE TESTING PROCESS

Lazi, Lj. 1, Medan, M.1 SIEMENS d.o.o, Radoja Dakia 7, 11070 Beograd, Serbia&Montenegro, ljubomir.lazic/milan.medan@siemens.com, http://www.siemens.co.yu
1. INTRODUCTION Software Quality Engineering (SQE) is a comprehensive lifecycle approach concerned with every aspect of the software product development process (SDP). An SQE program includes a comprehensive set of quality objectives, measurable quality attributes (quality metrics) to assess progress towards those objectives, and quantitative certification targets for all component software development processes. 2.OVERVIEW OF SOFTWARE QUALITY ENGINEERING Takes into account customer product requirements, customer quality requirements, and corporate quality requirements, SQE is integrated assurance approach to three attributes of SDP: Quality, Process and Management, leading to these SQE components: Quality Assurance (QA) Configuration Management (CM) Verification & Validation (V&V) Test & Evaluation (T&E). These SQE components use many methods and engineering techniques to perform the process and product assurance functions. Software Quality Assurance (SQA) is a group of related activities employed throughout the software life cycle to positively influence and quantify the quality of the delivered software. SQA is not exclusively associated with any major software development activity, but spans the entire software life cycle. It consists of both process and product assurance. Its methods include assessment activities such as ISO 9000 and CBA IPI (CMM-Based Appraisal for Internal Process Improvement), analysis functions such as reliability prediction and causal analysis, and direct application of defect detection methods such as formal inspection and testing. 2.1 PROCESS ASSURANCE Schulmeyer defines software quality assurance as . . . the set of systematic activities providing the evidence of the ability of the software process to produce a software product that is fit for use.[1]. SQA oversight provides management with unbiased feedback on process compliance so process lapses can be addressed in a timely fashion. It provides management with an early warning of risks to product quality and can provide recommendations to address the situation [2]. It is essential that the SQA personnel have a reporting path, which is independent of the management responsible for the activities audited, and the associated daily conflicts generated by schedule and budget. Independent oversight, through various methods, encourages adherence to the official process. Locating an appropriate level of management where SQA will have frequent access, active support, and be above the conflicts of interest may be a difficult but necessary step [2]. The methods typically used to accomplish process assurance include SQA audit and reporting, assessment and statistical process control analysis. 2.2 PRODUCT ASSURANCE Assurance that the product performs as specified is the role of product assurance. This includes in process, or embedded, product assurance, as well as some methods that involve independent oversight. The purpose of embedded SQA processes is product quality assurance. The activities are part of the development life cycle which will build-in the desired product quality. This focus allows identification and elimination of defects as early in the life cycle as possible, thus reducing maintenance and test costs. Embedded SQA methods include formal inspection, reviews, and testing [3]. An independent test function or testing which is witnessed by an independent entity such as SQA is one method of providing product assurance. Other options include tests witnessed by customers, expert review of test results, or audits of the product [2]. Audits are used to examine the conformance of a development process to procedures and of products to standards. Embedded SQA activities, which have the purpose of detecting and removing errors, take a variety of forms, including inspection and testing. Analysis techniques, such as causal analysis, reliability prediction and statistical process control, help ensure both process and product conformance. 3. SQE MAIN ISSUES Motivation for High-Yield (HY) i.e. cost-effective SQE is fulfillment of better, chipper and faster SDP mantra. Giving answer to these, among the others, questions, can do it: What is Software Quality? What is Software problem? Where do Software problems come from? What are appropriate strategic responses? Some software product and SDP quality are: A software product that has integrity is one that: fulfills customer needs, can be easily and completely traced through its life cycle, meets specified performance criteria, meets cost expectations, and meets delivery expectations. Zero Defects is High Quality: to users whose work is disturbed by those defects, to managers who are criticized for those defects. Having Lots of Features is High Quality: to users whose work can profit from those features - if they

know about them, to marketers who believe that features sell products. Elegant Coding is High Quality: to developers who place a high value on the opinions of their peers, to professors of computer science who enjoy elegance. High Performance is High Quality: to users whose work taxes the capacity of their machines, to sales people who have to submit their products to benchmarks. Low Development Cost is High Quality: to customers who wish to buy thousands of copies of the software, to project managers who are on tight budgets. Rapid Development is High Quality: to users whose work is waiting for the software to marketers who want to colonize a market before the competitors can get it. The typical SDP is a human intensive activity and as such, it is usually unproductive, error prone, and often inadequately done. Because software engineering processes and products include elements of human participants (e.g., designers, testers), information (e.g., design diagrams, test results), and tasks (e.g. design an object-oriented system model, or execute regression test suite), uncertainty occurs in most if not all of these elements. There are at list four domains of software engineering where uncertainty is evident: Uncertainty in requirements analysis, Uncertainty in the transition from system requirements to design and code, Uncertainty in software re-engineering and Uncertainty in software reuse [4]. With the pervasive use of software systems in modern society, the negative impact of software defects is also increasing. Consequently, one central activity for quality assurance (QA) is to ensure that few, if any, defects remain in the software when it is delivered to its customers or released to the market. Most modern software systems beyond limited personal use have become progressively larger and more complex because of the increased need for automation, functions, features, and services. It is nearly impossible to completely prevent or eliminate defects in such large complex systems. Instead, various QA alternatives and related techniques can be used in a concerted effort to effectively and efficiently assure their quality. Close examination of how different QA alternatives deal with defects can help one better use them for specific applications. How Do We Develop a High-Yield Process? Possible answer could be to: Assure High Software quality Emphasize results of successful inspections and tests, namely defects and failures. Assess cost-effectiveness quantitatively Identify best practices Continuously improve. It can be done by effective and efficient i.e. HY SQA and management based on these best principles and practices: Key engineering principles (Pragmatic Iterative and Incremental Process, Client Orientation Core functionality, High-Yield and Coverage-Driven Design Inspection, HighYield Testing Process, Well-Integrated Team of highly qualified experts, Monitoring+Integrated Management etc.) Setting Management objectives (in Quality, Productivity and Resources)

Integrate Measurement Process (define and measure Process and Product Metrics) Predict, Assess and Monitor Cost of Quality Apply Constant Process Improvement (CPI) based on Optimization techniques of: Product quality, Development cost, Time to market, and Development Resources (personnel and facility). 4. DEFECTS AND GENERIC WAYS TO DEAL WITH DEFECTS This section clarifies various meanings of the term defect, and then examines the generic ways to deal with defects. 4.1 DEFECT-RELATED DEFINITIONS The term defect generally refers to some problem with the software, either with its external behavior or with its internal characteristics. The IEEE Standard 610.12 [5] defines the following terms related to defects: Failure: The inability of a system or component to perform its required functions within specified performance requirements Fault: An incorrect step, process, or data definition in a computer program Error: A human action that produces an incorrect result The term failure refers to a behavioral deviation from the user requirement or the product specification; fault refers to an underlying condition within software that causes certain failure(s) to occur; error refers to a missing or incorrect human action resulting in certain fault(s) being injected into software. Sometimes error is also used to refer to human misconceptions or other misunderstandings or ambiguities that are the root cause for the missing or incorrect actions. With these definitions, one can see that failures, faults, and errors are different aspects of defects. A causal relation exists among these three aspects; that is, errors may cause faults to be injected into the software, and faults may cause failures when the software is executed. This relationship is not necessarily 1-to-1. A single error may cause many faults, such as when a wrong algorithm is applied in multiple modules and causes multiple faults, and a single fault may cause many failures in repeated executions. Conversely, the same failure may be caused by several faults, such as an interface or interaction failure involving multiple modules, and the same fault may be there because of different errors. 4.2 DEALING WITH DEFECTS With the previous definitions, one can view different QA activities as attempting to prevent, eliminate, reduce, or contain various problems associated with different aspects of defects. One can classify these QA alternatives into the following three generic categories: Defect prevention through error removal. These QA activities prevent certain types of faults from being injected into the software, which can be done in two generic ways: 1. Eliminating certain error sources, 2. Fault prevention, or breaking the causal relation between error sources and faults. Defect reduction through fault detection and removal. In fact, most traditional QA activities fall into this category. For example, inspection directly detects and removes faults in the software, while testing removes faults based on related failure observations. Defect containment through failure prevention. A related

extension is containment measures to avoid catastrophic consequences in case of failures. These QA activities are forming a series of barriers used to remove or block defect sources and prevent undesirable consequences. The authors next survey these alternatives and examine how they deal with defects in their specific ways. 4.3. ROOT-CAUSE ANALYSIS FOR DEFECT PREVENTION Notice that many of the error-removal activities described previously implicitly assume that there are known error sources or missing/incorrect actions that result in fault injections, as follows: if human misconceptions are the error sources, education and training should be part of the solution. if imprecise designs and implementations that deviate from product specifications or design intentions are the causes for faults, formal methods should be part of the solution. if nonconformance to selected processes or standards is the problem that leads to fault injections, then process conformance or standard enforcement should be part of the solution. if there is empirical or logical evidence that certain tools or technologies can reduce fault injections under similar environments, these tools or technologies should be adopted. Therefore, root-cause analyses are needed to establish these preconditions, so that appropriate defect prevention activities can be applied for error removal. Statistical analysis is based on empirical evidence collected either locally or from other similar projects. These data can be fed to various models to establish the predictive relations between causes and effects. Once such causal relations are established, appropriate QA activities can then be selected and applied for error removal. 4.4 DEFECT REDUCTION THROUGH FAULT DETECTION AND REMOVAL i.e. SOFTWARE TESTING TECHNIQUES For most large software systems in use today, it is unrealistic to expect that error-removal or defect prevention activities can be 100 percent effective in preventing accidental fault injections. Therefore, there is a need for effective techniques to remove as many of the injected faults as possible under project constraints. Various individual testing activities and techniques can be classified using various criteria, as discussed next, with a special attention paid to how they deal with defects. Because testing is an execution-based QA activity, a prerequisite to actual testing is the existence of the implemented software units, components, or system to be tested, although preparation for testing can be carried out in earlier phases of software development. As a result, actual testing can be divided into various sub phases starting from the coding phase up to post-release product support, including: unit testing, component testing, integration testing, system testing, acceptance testing, beta testing, and so on. The observation of failures can be associated with these sub phases, and the identification and removal of related faults can be associated with corresponding individual units, components, or the complete system. Other Techniques for Fault Detection and Removal Inspection is the most commonly used static technique for defect detection and removal. Various other static techniques are available, including various formal model-based analyses

such as algorithm analysis, decision-table analysis, boundary value analysis, finite-state machine and Petri-net modeling, control and data-flow analyses, software fault trees, and so on. For example, symbolic execution, simulation, and prototyping can help one detect and remove defects early in the software development process, before large-scale testing becomes a viable alternative. On the other hand, in-field measurement and related analyses, such as timing and performance monitoring and analysis for real-time systems, and accident reconstruction using software event trees for safety-critical systems, can also help one locate and remove related defects. A comprehensive survey of techniques for fault detection and removal, including those mentioned previously, can be found in [6]. 5. A FRAMEWORK ELEMENTS OF INTEGRATED AND OPTIMIZED SOFTWARE TESTING PROCESS In order to achieve Testing Maturity Model level 5 we see software test and evaluation process as multi disciplinary engineering integrated solution as depicted in Figure 1, called Integrated and Optimized Software Testing Process (IOSTP). The our framework of IOSTP is based on a foundation of operations research, experimental design, mathematical optimization, statistics, analyses, and validation, verification, control theory and accreditation techniques. Application of M&S and Design of Experiments dramatically minimized Test Case generation through Scenario Black-Box Testing [11]. For every company there exist optimized testing scenario composed of the testing process, methods, techniques, strategies and tools to be used, and the necessary training program for the people directly involved in this process. In practice, the software development methodologies employ a combination of several software testing techniques designated in our Figure 1 as Test Methods Pool. That there is no silver bullet testing approach and that no single software testing technique alone is satisfactory has been pointed out by many leading researchers. There is need to evaluate strengths and weakness of various software test methods concerning their effectiveness, efficiency, range, robustness and scalability, combine them to build optimized test scenario for particular System/Software Under Test (SUT) based on their strengths but avoid the problems associated with either approach. In order to significantly improve software testing efficiency and effectiveness for the detection and removal of requirements and design defects we apply Model-Based Testing activities through Simulation (M&S) because building the model is testing and vice versa. The USA DoD has developed a framework [9] called Simulation, Test and Evaluation Process (STEP) to integrate Modeling and Simulation (M&S) into the Test and Evaluation (T&E) process of the SUT. STEP is a move beyond the test, fix, test approach to a model-simulate-fixtest-iterate approach with problems fixed as they are discovered. This approach, (model first; simulate; test; fixing after each step and then iterate the test results back into the model), is reiterated throughout system development. When a need to fix is discovered, the time for each fix can be much shorter when the fix can be verified in the model in hours or days, as opposed to a field test, which can take weeks or months to verify a fix. Main task is development of a versatile Optimization Model (OM) for assessing the cost and effectiveness of alternative test, simulation, and evaluation strategies [9,10]. Optimize plays out dynamic balanced test

scenario among the test activities effectiveness and joint minimum of time and test resources needed to deliver the required software features at the expected level of quality.

we applied Model-Based Testing activities through Simulation (M&S) and Design of Experiment because building the model is testing and vice versa. It produced a set of scenarios used to test target software implementation, fund bugs, serves as test oracles in all test phases and track software development progress. In order to assure stable i.e. controllable and predictable software testing process we apply Six Sigma for software methodology. Six Sigma insists on: active management engagement and involment, on financial business case for every improvement, on focusing on only the most important business problems, and it provides clearly defined methodology, tools, role definitions, and metrics to ensure success. REFERENCES [1] Schulmeyer, G. at all, The Handbook of Software Quality Assurance, Upper Saddle River, NJ: Prentice Hall PTR, p. 9, 1998. [2] Humphrey, W., Managing the Software Process, The SEI Series in Software Engineering, Reading, MA: AddisonWesley Publishing, Inc., p. 140, 1990. [3] Peng, W., and Wallace, D., Software Error Analysis, NIST Special Publication 500-209, Gaithersburg, MD: National Institute of Standards and Technology, pp. 7-8, 910, March 1993. [4] Ziv, H. and Richardson, D., Constructing Bayesiannetwork Models of Software Testing and Maintenance Uncertainties, International Conference on Software Maintenance, Bari, Italy, September 1997. [5] IEEE Standard 610.12. IEEE standard glossary of software engineering terminology. 1990. New York: Institute of Electrical and Electronics Engineers. [6] Wallace, D., Ippolito, M., and Cuthill, B., Reference Information for the Software Verification and Validation Process, NIST Special Publication 500-234, 1996. [7] Boehm, B., Software risks management, IEEE Computer Society Press, Los Alamitos, California, 1989. [8] Burstein I. at all. Developing a testing maturity model, Part II, Illinois Institute of Technology, Chicago,1996. [9] http://acq.osd.mil/te/programs/final_STEP.pdf [10] Kai-Yuan Cai., Optimal software testing and adaptive software testing in the context of software cybernetics, Information and Software Technology, Vol. 44, pp.841-855, 2002,. [11] Lazi, Lj., Velasevi, D., Mastorakis, N., A framework of integrated and optimized software testing process, WSEAS TRANSACTIONS on COMPUTERS, Issue 1, Volume 2, 15-23, January 2003. Abstract: This paper presents a validation-oriented approach to the design and development of high quality software products. It is shown haw to develop a cost-effective Software Quality Engineering strategy based on a selection from current and developing new verification, validation and Software Testing Techniques (STT). The relationship between software process quality and product quality versus software testing process is described. High-Yield SQE i.e. cost-effective can be employed only by optimal-yield STT in an Integrated and Optimized Software Testing Process, which is described in this paper.

Figure 1. A Framework elements of Integrated and Optimized Software Testing Process The SUT and corresponding testing strategy-scenario make up a closed-loop feedback control system [10]. At the beginning of software testing our knowledge of the software under test is limited. As the system/software testing proceeds, more testing data are collected and our understanding of the software under test is improved. Software testing parameters (e.g. software defect detection rates, defect containment, cost etc.) of concern may be estimated and updated, and the software testing strategy is accordingly adjusted on-line. In order to assure stable i.e. controllable and predictable software testing process we apply Six Sigma for software methodology called DMAIC for Define, Measure, Analyze, Improve, and Control because it organizes the intelligent control and improvement of existing software test process. Six Sigma is a mantra that many of the most successful organizations in the world swear by and the trend is getting hotter by the day. Six Sigma insists on active management engagement and involment, it insists on financial business case for every improvement, it insists on focusing on only the most important business problems, and it provides clearly defined methodology, tools, role definitions, and metrics to ensure success. 6. CONCLUSIONS AND PERSPECTIVES In general, a concerted effort is necessary with many different QA activities to be used in an integrated fashion to effectively and efficiently deal with defects and ensure product quality. According to the different ways these QA alternatives deal with defects, they can be classified into three categories: defect prevention, defect reduction, and defect containment. The survey and classification of different QA alternatives in this article bring together information from diverse sources to offer a common starting point and information base for software quality professionals. In order to significantly improve software testing efficiency and effectiveness for the detection and removal of requirements and design defects in our framework of IOSTP