Software Quality and standards (IDY0204)

Course and arrangement (31.08.2011) Jaak Tepandi, Stanislav Vassiljev

The latest version of this material is on jaak.tepandi.ee (or http://www.ttu.ee/users/tepandi/wi-tk-k.html)

TABLE OF CONTENTS 1. CONTENT, MATERIALS, AND PREREQUISITES .................................................................. 1 2. PREREQUISITES, DEADLINES, AND EVALUATION ............................................................ 2 3. COMMUNICATION ....................................................................................................................... 3 4. HOMEWORKS: TYPES AND STRUCTURE.............................................................................. 3 4.1. STANDARD AND NON-STANDARD HOMEWORKS ........................................................................... 3 4.2. STRUCTURE OF STANDARD HOMEWORKS..................................................................................... 3 4.2.1. First homework ................................................................................................................... 4 4.2.2. Second homework ............................................................................................................... 5 5. REVIEW ........................................................................................................................................... 6 6. PRESENTATION ............................................................................................................................ 7 7. EXAMINATION .............................................................................................................................. 7 8. APPENDIX. HOMEWORK / PROJECT TITLE PAGE ............................................................. 9

1.

Content, materials, and prerequisites

The objective of this subject is to provide an overview of the problems and methods of software verification and validation and quality management. The course introduces different testing methods and principles of testing management, and gives an overview of planning, performing, and documentation of testing. The course has been divided into two parts. The first part (approximately 60% of the volume) addresses the methods and management of software verification and validation: testing (the principles of software testing, white box, functional, other methods), static methods (inspections, reviews and walkthroughs, program correctness proving), and the frameworks for testing management. The second part (approximately 40% of the volume) provides an overview of the software quality management concepts and standards: the principles of quality management and standardization, software process quality, and software quality criteria and metrics. Introduction and basic concepts are provided on the slides on the course website (jaak.tepandi.ee). In addition, extensive independent reading is required. For an overview, materials from the "Guide to the Software Engineering Body of Knowledge (SWEBOK)" are recommended in Section 7 (Examination). An example list of materials is provided below. Materials

Software Quality and standards (IDY0204) - course and arrangement. 31.08.2011, Jaak Tepandi, TTU

W. Perry. Effective Methods of Software Testing, 3rd edition. John Wiley & Sons, Incorporated ANSI/IEEE Std 829 Standard of software test documentation ISO/IEC (9126). ISO/IEC 9126, Information Technology - Software Product Evaluation Quality Characteristics and Guidelines for their Use Guide to the Software Engineering Body http://www.computer.org/portal/web/swebok of Knowledge (SWEBOK), IEEE,

Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering, ACM / IEEE, http://www.acm.org/education/curricula-recommendations Certified Tester Foundation Level Syllabus. Version 2007. International Software Testing Qualifications Board, http://www.istqb.org/download.htm. +Other materials from this site. Books including software quality, testing, and standards issues as significant portions of the text, edition after 2000 preferred - please discuss with the advisor. Examples: I. Sommerville, Software Engineering, Addison-Wesley; R.S. Pressman, Software Engineering, McGraw Hill; G. OReagan, A Practical Approach to Software Quality, Springer General books on software engineering, management and standards as a complementary material CMMI - http://www.sei.cmu.edu/cmm/cmms/cmms.html References given in the slides of the http://deepthought.ttu.ee/users/tepandi/wi-tk-k.html) course, see jaak.tepandi.ee (or

Prerequisites The prerequisite for this course is familiarity with basic concepts of software engineering and ability to program in at least one programming language.

Note: this material will evolve during the course presentation.

2.

Prerequisites, deadlines, and evaluation

A prerequisite for taking the course is ability to program at a basic level. A prerequisite for passing an examination is working during the semester. To pass an examination, min 25 points should be received from the examination, min 25 - from homeworks and other activities. Examination and homeworks are mandatory; all the other activities are voluntary. The activities are evaluated as follows. Activity 1.homework Project (two homeworks together) Reviews made Time Week 1-7 Week 8 14 Week 8 14 Week 8 14 Week 1-16 Week 1-16 Max points 10 30 Remarks Bonus for presenting until week 7. NB! To enable reviewing and improving, the projects should be uploaded before week 14. Improvements are possible until week 14 incl. Evaluated until examination. Full projects can be reviewed. Review is individual. One student may prepare max 3 reviews, each review gives its author max 2 points. One good review gives max 2 points to all the project authors, three reviews are taken into account, total - max 6 p. One presentation gives max 10 p, and so do all presentations in total. Active participation in discussions.

Reviews received Presentation Participation

6 10 10

Software Quality and standards (IDY0204) - course and arrangement. 31.08.2011, Jaak Tepandi, TTU

Examination

Session

50

Grading: 50...59 points = grade "1", 60...69 = grade "2", ..., 90 and more = grade "5" ("5" is maximum). Other rules of TTU apply.

3.

Communication

Homeworks. To enable identification of authors, homeworks, and files, please indicate in uploading the name of the first author, the group, the course identifier, and topic (an example: "Tnu Tamm, IVSM11, IDY0204, project. Sports Tracker software for Seawolf Spa"). Please indicate the same information (except the topic) in the filenames of all files sent. Example filenames: "Tqnu_Tamm_ IVSM11_IDY0204_project_2011.doc" " Tqnu_Tamm_ IVSM11_IDY0204_project_2011_tests.zip"

Example student code: 092543IABM. Example group ID: IVSM11. Please check the content (next section) and the title list (last section). Sending of other materials. Similar principles apply - please enable identification of persons and topics in a large array of e-mails and files, identifying the author, the group, the course identifier, and topic in the Subject line. E-mails from "Tom <tom112@hotmail.com>" with subject "question" may be indicated as spam by the spam filter. Please send e-mails to jaak@tepandi.ee. Deadlines. Please do not keep to the last date, there may be problems with workloads, computers, Internet etc.

4.

Homeworks: types and structure

The homework team may include 13 persons; a team of two persons is recommended. Please see last section for a homework title page.

4.1.

Standard and non-standard homeworks

The project comprises two homeworks, which collectively represent a snapshot of software life cycle activities. As much as possible they should be presented as project documentation, not as student homework (for example, the task descriptions from the current material should be excluded). One exception from this rule is the homework title page given in the last section. The homeworks may have a non-standard or a standard structure. The non-standard homeworks deal with topics relevant to the course and are agreed beforehand by email with the supervisor. Very often their structure may be similar to that of a standard project (e.g., one has to identify stakeholders and the project, analyze requirements, etc). In case a non-standard project does not involve some practical components the course (such as development of white or black box tests, using test tools etc), experience in these should be demonstrated separately. The next sections deal with standard homeworks.

4.2.

Structure of standard homeworks

The project (two home works together) gives an excerpt of software acquisition and development activities associated with the course. The systems / programs used in the project must exist and testing must be real. If an organisation is not available, it can be modelled, imagining as realistically as possible an organisation which needs a system and has requirements to this system. The project should be as real as possible; this means, for example, that both the organisation and the system should be nontrivial, otherwise testing and other activities undertaken are not justified. 3

Software Quality and standards (IDY0204) - course and arrangement. 31.08.2011, Jaak Tepandi, TTU

The same organization and system is considered throughout the project, with the only possible exception for program based tests (see below). The project comprises requirements definition, testing, and acceptance, but not coding. Typically it is not possible to develop software during the project, so software should exist in some (perhaps intermediate) form. A structure of the standard homeworks is given below. Numerical values (e.g., number of requirements) are given for the two-person project; for a one-person project, they should be divided by 2 (and proportionally increased for projects done by more than 2 persons).

4.2.1. First homework

The structure of the first homework is as follows. 1. Acquirer's view (excerpt) 1.1. Description of the project. Components: 1.1.1. Organisation (and the system to be tested). The project should include at least the following components: organisation and the system to be procured; value of the system for the organisation; system properties that are critical for the organisation (with justifications); stakeholders and their expectations; restrictions on the cost. Remark 1. Both the organisation and the project must be important (a "homework" would not justify testing etc) and assume non-functional requirements. Remark 2. In case there is no real organization available, it can be modelled (this should be realistic, see above - please select an organization which you are familiar with). The system to be tested (and organisation). The project should include at least the following components: users and usage of the system (what is the difference between a stakeholder and a user?); expected architecture; components to be procured (for example, code, documentation, etc); rights to be procured. Remark 1. The system as a whole should be identifiable and important for the organisation (not a module which is not visible to the user). Remark 2. Description of the system does not need to be extensive it should just enable the reader to understand the high-level architecture of the system (components/applications and the definition of the interfaces between those components and applications); non-obligatory examples: distributed or local? How many layers? How many locations? Communication protocols? Underlying software platforms? Etc. Remark 3. The system and its testing must be real - there are many systems freely available for download; or freely available for some period sufficient for testing once the project has been prepared.

1.1.2.

1.2. Requirements. Requirements are presented as use cases. Each use case includes at least its ID (to be associated with tests), the use case name (goal), actors, preconditions, postconditions, primary scenario. Requirements are based on quality attributes from ISO/IEC 9126 and must be testable. Components: 1.2.1. Functional requirements. At least 10 functional requirements are presented. What is the realistic number of requirements for this system, how much of the system is covered? Remark. Evaluation of the realistic number of requirements does not need to be detailed and may be based, for example, on suggesting the approximate overall number of functions and evaluating an average number of requirements per function. Non-functional requirements. At least 10 non-functional requirements are presented, including at least one requirement specifying system load properties. What is the realistic number of requirements for this system, how much of the system is covered?

1.2.2.

1.3. Acquisition activities (excerpt). Components: 1.3.1. 1.3.2. Participation of the procurer. How much time should procurer's employees plan to this procurement? What equipment, infrastructure, other components will be needed? Change management for requirements. Who and how proposes, discusses, decides, manages changes? What happens with the project schedule and cost? 4

Software Quality and standards (IDY0204) - course and arrangement. 31.08.2011, Jaak Tepandi, TTU

1.3.3.

Acceptance plan. How is the system accepted? In which organization (e.g., customer or supplier)? What is the time schedule? Who performs the tests? Under which criteria is the system accepted? Remark. The time schedule evaluation does not need to be detailed and may be based on the amount of testing needed (number of requirements), the resources available, and the organisation deadlines.

1.4. Risk based acceptance tests. At least 20 risk based acceptance tests are designed, including at least 8 tests for non-functional requirements. The tests must include at least one load test. The tests should be inferred from the requirements and risk analysis, refer to the corresponding requirements and risks, cover the requirements as much as possible, and be sufficiently detailed so that it is possible to execute them based on the test descriptions. Every test should include at least the test identifier, idea (referring to the requirement tested), specific given inputs, and specific expected outputs. For documentation the ANSI/IEEE Std 829 may be used. 1.5. Saving the tests using a test automation tool. Tests are saved using appropriate tools, such as Apache JMeter for load tests; Canoo WebTest or Selenium for other tests; in case the source code is available, also JUnit is an alternative, but in this case testing must still be on acceptance testing level (not on level of testing the program logic). Remark 1. In case the system under test is not web based (or the proposed tools cannot be used for some other reason), other testing tools may be used. Remark 2. In case the system under test is not web based and other testing tools cannot be found, the tests may be performed manually. In this case some knowledge of the testing tools presented above should be demonstrated during the course exercises; alternatively, additional test cases for a freely selected web application may be added to the homework: 10 test cases using jMeter and 10 test cases using some other tool (Canoo WebTest, Selenium, JUnit).

4.2.2. Second homework

The structure of the second homework is as follows. 2. Developer's view (excerpt). 2.1. Implementation processes. An overview of the main implementation processes is presented. 2.2. Walkthroughs. Walkthroughs (or other static V&V methods) in the project are planned according to the problem statement. Who will perform them, how often, who will participate, how will the results be used? At least one walkthrough is performed. The object, participants, activities and results of this walkthrough are presented. The walkthrough type is selected by the author according to the organisation and the testing task. Remark. The system must be sufficiently important and critical to justify performing at least one walkthrough. 2.3. Functional tests. Components: 2.3.1. The component to be functionally tested and detailed requirements. The subcomponent of the system to be functionally tested is presented. Detailed requirements used in functional testing are added as necessary. Functional tests. At least 20 functional tests for the selected sub-component are designed (for a two-person project). For each test, the requirement used for testing, the equivalence classes and/or boundary situations stemming from the requirement, test data, grouping the data into tests, and the tests themselves are presented. The description level should allow for following the test design logic, starting from the requirement and finishing with the test. The tests may be presented similarly to acceptance tests. Remark 1. In case these functional tests do not cover all equivalence classes for the selected subcomponent, it is possible to explain the situation and give a short characterization of coverage achieved. Remark 2. The acceptance tests should not coincide with functional tests (in a realistic system the number of tests far exceeds the number of tests given in the homeworks therefore it is possible to select these tests in a different way). Saving and running the tests using a test automation tool. Tests are saved and run using appropriate tools as for acceptance testing. To document the results, the tool output

2.3.2.

2.3.3.

Software Quality and standards (IDY0204) - course and arrangement. 31.08.2011, Jaak Tepandi, TTU

(in case it gives readable results), or ANSI/IEE Std 829 Test-Incident Report may be used. 2.4. Program based tests. Components: 2.4.1. Present the program. Select a program (at least two pages + 8 if-statements + 4 iterations) and add its text to the homework. Remark 1. It is recommended to select a program from the system to be tested (a system may include hundreds of program units). In case this is not possible, please explain the situation, select an independent program (e.g., searching from the Internet), and present briefly the requirements for this program. Remark 2. The selected program may include several interconnected methods / components. Coverage tests. Justify and select a coverage criterion for testing, design tests, present as in the previous section. Remark: the number of tests is not pre-determined (why?). Test, evaluate. Perform the designed tests, document the results, evaluate the quality of the program and adequacy of testing. Test tools may be used, for example JUnit.

2.4.2. 2.4.3. 3.

Acquirer's and developer's view 3.1. Implementing ISKE or ITIL. A short overview of applying either ISKE or ITIL is presented. 3.2. Acceptance testing, system evaluation, retrospective analysis. Components: 3.2.1. Performing the acceptance tests. The acceptance tests designed above are executed and documented similarly to the functional tests. Remark 1. In case the load testing requirements are high and it is not known how the software or site owner will react to high loads, the load testing may be performed with lower actual load, explaining the reason and results in the documentation. Remark 2. In case some non-functional tests cannot be performed in the ways given above (e.g. due to some facilities of nonfunctional testing not available to the testers), it is possible to give an explanation of the situation and present an analysis of the expected results (including the conclusion of whether the author expects the test to pass or not). Evaluation, risk analysis, acceptance. The decision for the project, e.g.: accept, develop further, reject, start from the beginning, postpone etc. What are the risks? Give justifications for the decision. Retrospective analysis. Benefits, lessons, drawbacks form the project. Things to be done another way in the next project how?

3.2.2.

3.2.3.

5.

Review

Each student may analyse another student's project (two homeworks together) from the same year. This gives feedback to the author and enables improving the work (in case of a timely review, see above). The reviewer has a chance to experience values and problems of an additional project. An analysis has one author (analyses and homework project teams are not related). A recommended structure of analysis is as follows. 1. The analyser's data (Author, student code, group ID) 2. The project data

Author, student code, group ID Project title, short overview Project status and date Analysis inputs (e.g. two homeworks, program, documentation, access to the system etc)

Software Quality and standards (IDY0204) - course and arrangement. 31.08.2011, Jaak Tepandi, TTU

3. Evaluation, taking into account requirements and the structure of the homeworks presented above 4. Recommendation for grade points, max 30

6.
Topics

Presentation

Presentation can be on topics related to the course. It is assumed that the author and listeners have participated in the classroom sessions so repetition of the course material is not accepted. A well known topic might be suitable in case the author is an expert on this topic and can present personal practical experience (pluses-minuses-problems etc). Presentation can also be on interesting quality aspect(s) of a specific software application. The software should be neither too widespread (everybody knows the problems) nor too exotic (nobody is interested). It should rely both on author's experience and overview of references. There should be multiple references discussing various viewpoints (only manufacturer's references are not sufficient). The references should be stated in the presentation. Presentation can be on one's homework (two works together, presented as an integrated project). Materials and schedule Presentations are given typically during regular lecture / seminar time. They should be agreed beforehand by e-mail. Materials should be sent to the same address as homeworks at least one day before the presentation. There can be 2...3 presentations during one seminar. In case of insufficient time, the presentations are made in the order of their sending; the first presentation of each author has a priority. In case of a large number of presentations, special sessions can be agreed beforehand. Authors and duration As a rule, a presentation has one author. For joint works, the authors may present different aspects of the work. The duration of a presentation is usually 10...15 min + discussion. Evaluation The evaluation of the presentation depends on the quality of the presentation (content, talk, materials) and the number of listeners (5+ listeners besides the teacher 100%, 2...4 listeners - 60%; 0...1 listeners 30%). It is recommended to make presentations during the semester, as it may be difficult to gather people afterwards. Recommendations Recommendations for the presentation: Present yourself and give the author's name and group ID also on the slides Create structure (e.g. content slide) Give some background and overview Give sources and references Use demos if possible Speak loudly and clearly so people in back seats can hear Watch time

7.

Examination

Arrangement and materials 7

Software Quality and standards (IDY0204) - course and arrangement. 31.08.2011, Jaak Tepandi, TTU

Examination is oral, 5 short questions or problems may be asked (no tickets or materials). Each answer gives max 10 points (total max 50 points, see Section 2). The main topics are given in the course slides on jaak.tepandi.ee (or http://deepthought.ttu.ee/users/tepandi/wi-tk-k.html). In addition to the slides, independent reading on these topics is required. An example list of materials is provided in Section 1. For an overview of the material, the following sections from the "Guide to the Software Engineering Body of Knowledge (SWEBOK)" may be useful: Chapter 5: Software testing Chapter 9, Software Engineering process the notions of software development life cycle models (examples: waterfall, spiral, XP, V-model) and (software) process frameworks (examples: ISO/ IEC 12207, CMMI, ITIL, also Cobit) Chapter 11: Software quality

Different viewpoints are accepted, but the concepts given in the course should be known and compared to other approaches. Short questions for the examination Each slide in the course presentations can be viewed as a source for an examination question. Example: "what is the difference between testable and non-testable requirements?" (based on the slides from Jaak_Tepandi_Sw_Q&S_00.pdf) For each method (standard, framework, etc) discussed in the course, its features may be a topic for a question, examples: idea of the method, preconditions, advantages, disadvantages, results, relationship to other methods, evaluation, tools. Example: "what are the disadvantages of equivalence partitioning?".

The course basics The following list presents some main concepts of the course. Please check that they are known. Quality. Product. Requirements. Process Software product quality factors and criteria. ISO/IEC 9126 Functional / non-functional + Testable / nontestable requirements Software life cycle model examples. ISO/IEC 12207. An overview of models and frameworks Error. Fault. Failure. Verification. Validation. Certification Testing. Test. Testing principles Specification based (black box) testing. Equivalence partitioning and boundary values Testing based on the program text (white box). Coverage criteria, their comparison Risk based testing Other methods of testing. Exploratory testing. Experience-based testing. Non-functional testing. Usability testing. Random testing. Fault seeding Testing volume and termination. Price effectiveness of testing methods ANSI/IEE Std 829 Standard for Software Test Documentation Static methods. Walkthrough/Inspection/Review. Questionnaires. Programmer`s evaluation. ANSI/IEEE Std 1028-1988. IEEE Standard for Software Reviews and Audits Requirements for critical software. N-version programming. Fault tree analysis. Formal verification. Cleanroom software engineering. Common Criteria for Information Technology Security Evaluation Need for testing management. System and software. Testing with V-model. RUP and testing. XP and testing. Test Process Improvement (TPI). IEEE 829 (documentation of tests). Testing automation. Types, advantages, disadvantages of testing tools 8

Software Quality and standards (IDY0204) - course and arrangement. 31.08.2011, Jaak Tepandi, TTU

Quality management: concepts. Informal QM techniques. Quality awards. ISO 9000. Software development, testing and quality management. Quality management first steps in company CMMI. ITIL / ISO/IEC 20000. ISO/IEC 27000 series. Agile and ISO/IEC 12207, CMMI, COBIT IT standards and standardization bodies. ISO, IEC, ISO / IEC JTC1 Software metrics. Software development effort and cost prediction. COCOMO IT audit. ISACA, COBIT, CISA

8.

Appendix. Homework / project title page

The next page presents a homework title page. The title reflects the content (not "homework" etc). Names and data of all authors are presented. Please change the fields as necessary.

Software Quality and standards (IDY0204) - course and arrangement. 31.08.2011, Jaak Tepandi, TTU

TALLINN UNIVERSITY OF TECHNOLOGY DEPARTMENT OF INFORMATICS

Testing of Open Source Accounting Software "GnuCash"

Acquirer's view (excerpt) First homework in subject "Software Quality and standards" (IDY0204)

Author, student code, group ID: ........................................................ ........................................................ ........................................................ Presented: ................... Supervisor: ...................................

TALLINN 2011

10