ISO/IEC 29119 Parts 1 and 4 ISO/IEC 29119 Part 1, Concepts and definitions Ossi Taipale, Lappeenranta University of Technology Part 1: Concepts and definitions This standard defines a risk-based approach to testing. Risk-based testing is a recommended approach to strategizing and managing testing that allows testing to be prioritized and focused. The ISO/IEC/IEEE 29119 software testing series of standards contain three standards where conformance may be claimed: test processes; ISO/IEC/IEEE 29119-2 test documentation; ISO/IEC/IEEE 29119-3 test techniques; ISO/IEC/IEEE 29119-4. Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 2 Software testing is necessary because: Information on the quality characteristics of the test item(s) is required by decision makers; The test item(s) being tested does not always do what it is expected to do; The test item(s) being tested needs to be verified (Verification ensures that software correctly implements a specific function and it answers the question: are we building the product right?). Basic verification methods include inspections, walkthroughs, and technical reviews; The test item(s) being tested needs to be validated (Validation ensures that software that has been built is traceable to customer requirements. It answers the question: are we building the right product?). Validation activities can be divided into unit testing, integration testing, usability testing, function testing, system testing, and acceptance testing. Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 3 Introduction to software testing Product quality has many aspects, including conformance to specifications, absence of defects, and fitness in meeting the requirements of the product's users. ISO/IEC 25010, System and software quality models defines eight quality characteristics that can be measured or assessed in testing. Testing is a process. A process is a set of interrelated or interacting activities that transforms inputs into outputs. The objective of this standard is to present and describe generic testing processes (ISO/IEC/IEEE 29119-2 Test processes). Testing processes and sub-processes can be applied to any phase or level of testing (e.g. system testing) or type of testing (e.g. performance testing). Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 4 Introduction to software testing Testing can be carried out on a product without executing the product on a computer. This is called static testing in this standard and in many areas of the industry, although other standards (e.g. IEEE 1028) may more specifically call this reviews, walkthroughs or inspections. For static testing this standard acknowledges and identifies the role of the tester in these activities even though they may be performed by other groups within a project or defined by other non-testing standards. This is because the static testing activities are considered highly important for complete life cycle testing and test involvement has been shown to be critical for early defect detection, reduced overall project costs and an improved ability to meet schedule demands. Static testing may also include the use of static analysis tools which find defects in the code or documents without the code executing (e.g. a compiler, a cyclomatic complexity analyzer, or a security analyzer for Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo code). 5 Introduction to software testing Dynamic testing consists of more than "just" running executable test items; it also includes both preparation activities and follow-up activities. The Dynamic Test Processes described in ISO/IEC/IEEE 29119-2 cover each of the activities to be performed in dynamic testing. This standard addresses testing; it does not address the other activities of validation and verification (e.g. V&V analysis, formal methods). Other standards, e.g. ISO/IEC 12207 and IEEE 1012, address also other verification or validation activities. Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 6 Annex A: The role of testing in verification and validation Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 7 Generic test process The relationship between the generic test process, generic test subprocesses, test levels/phases and test types is described in more detail in Figure 2. The generic test sub-process can be applied in the following ways: As a test level or phase (e.g. component test phase, acceptance test level); As a test type (e.g. performance testing, usability testing). The diagram 2 also makes explicit the relationship between test types and quality characteristics (as defined in ISO/IEC 25010) ; A test sub-process associated with a test level may contain more than one test type sub-process (e.g. functional and performance testing as part of system testing); and The project test process may be comprised of a sequence of test subprocesses (e.g. component testing, integration testing, system testing and acceptance testing test sub-processes). Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 8 Figure 2 — The relationship between the generic test subprocess, test levels and test types Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 9 Annex C: Testing in different life cycle models The development activities performed are more or less the same in all life cycle models; the main differences lie in the definition of their scopes, the amount and nature of documentation produced, and the frequency with which they are repeated during the course of the development life cycle. Agile Evolutionary Sequential (i.e. the waterfall model) Scrum is not a development methodology (i.e. it does not tell you the best method for describing requirements or how to write code) and is better described as a project management framework within which an agile methodology is applied by the developers (XP is often used). A Scrum project comprises a number of iterations called sprints, with each sprint normally resulting in new functionality that can be delivered to the users (Figure). Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 10 29119 Part 4, Test techniques Ossi Taipale, Lappeenranta University of Technology Part 4, Test techniques: Introduction The purpose of ISO/IEC 29119-4 Test Techniques is to provide an International Standard that defines software test design techniques (also known as test case design techniques or test methods) that can be used during the test design and implementation process of ISO/IEC 29119-2 Test Processes. The test design techniques presented in ISO/IEC 29119-4 can be used to derive test cases that, when executed, can be used to collect evidence that test item requirements have been met and/or that defects are present in a test item (i.e. that requirements have not been met). Activities: Derive Test Conditions (TD2), Derive Test Coverage Items (TD3), and Derive Test Cases (TD4). Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 12 Introduction: Figure 1, ISO/IEC 29119-2 Test Design and Implementation Process Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 13 Introduction Experience-based testing practices like exploratory testing and other test practices such as model-based testing are not defined in ISO/IEC 29119-4 because this standard only describes techniques for designing test cases. Test practices such as exploratory testing are described in ISO/IEC 29119-1. Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 14 Test Design Techniques ISO/IEC 29119-4: In specification-based testing, the test basis (e.g. requirements, specifications, models or user needs) is used as the main source of information to design test cases - "black-box testing“. In structure-based testing, the structure of the test item (e.g. source code or the structure of a model) is used as the primary source of information to design test cases - "white-box testing" (or "clear-box testing"). Both - "grey-box testing" In experience-based testing, the knowledge and experience of the tester is used as the primary source of information during test case design. In practice they can be used interchangeably. Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 15 Test design techniques presented in ISO/IEC 29119-4 Figure 2, page 7 Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 16 Equivalence Partitioning Derive Test Conditions (TD2) Equivalence partitioning (BS 7925-2:1998; Myers 1979) uses a model of the test item that partitions the inputs and outputs of the test item into equivalence partitions (also called “partitions” or “equivalence classes”), where each equivalence partition shall be defined as a test condition. These equivalence partitions shall be derived from the test basis, where each partition is chosen such that all values within the equivalence partition can reasonably be expected to be treated similarly (i.e. they may be considered "equivalent") by the test item. Equivalence partitions may be derived for both valid and invalid inputs and outputs. Derive Test Coverage Items (TD3) Each equivalence partition shall also be identified as a test coverage item Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 17 Equivalence Partitioning Derive Test Cases (TD4) Test cases shall be derived to exercise the test coverage items (i.e. the equivalence partitions). Approach for selecting test coverage items: One-to-one, in which each test case is derived to cover a specific equivalence partition. The number of test cases derived using one-to-one equivalence partitioning equals the number of test coverage items. Minimized, in which equivalence partitions are covered by test cases such that the minimum number of test cases derived covers all equivalence partitions at least once. Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 18 Example: Equivalence Partitioning Test basis: The component receives as input an exam mark (out of 75) and a coursework (c/w) mark (out of 25), from which it outputs a grade for the course in the range 'A' to 'D'. The grade is calculated from the overall mark, which is the sum of the exam and c/w marks, as follows: greater than or equal to 70 - 'A' greater than or equal to 50, but less than 70 - 'B' greater than or equal to 30, but less than 50 - 'C' less than 30 - 'D' Where a mark is outside its expected range then a fault message ('FM') is generated. All inputs are passed as integers. Page 42 Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 19 Annex A1: Testing Quality Characteristics This annex contains examples of how quality characteristics defined in ISO/IEC 25010 (ISO/IEC 25010:2011) can be mapped to the test design techniques defined in ISO/IEC 29119-4. Ohjelmistotestauksen standardi ISO/IEC 29119 kurssi 12-13.2.2014, Innopoli 2, Espoo 20