经典原版书库

软件测试：原理与实践

（英文版第2版）

Software Testing：Principles and Practice

（Second Edition）

（爱尔兰）斯蒂芬·布朗(Stephen Brown)

（爱尔兰）乔·蒂莫尼（Joe Timoney）

         （爱尔兰）范氏钗（Thoa Pham）            著

（爱尔兰）汤姆·莱萨特（Tom Lysaght）

（中国）叶德仕（Deshi Ye）

Preface

This book is based on a series of lectures given at Maynooth University and Zhejiang University. It provides a textbook for a number of courses, describing the fundamentals of software testing. The material has been developed over the past ten years, and reflects both  the  experience  of  20  years  in  industry  from  one  of  the  authors,  and  the  authors’ joint experience in lecturing.

   There is no one standard textbook on software testing,  and this book is the result of many years of extracting and interpreting test techniques from a wide and varied number of sources. These include testing classics such as The Art of Software Testing by Myers, Software Testing by Roper, Testing Object- Oriented Systems by Binder, and Introduction to Software Testing by Ammann and Offutt; standard software engineering textbooks such as Software Engineering by Pressman , and Software Engineering by Sommerville; software process books such as Software Testing in the Real World by Kit and extreme Programming explained by Beck; and ISO and IEEE standards related to software quality and testing.

   Software testing is a challenging task–it is as important for businesses and government as it is for research institutions. It is still as much an art as a science: there are no accepted standards or norms for applying the various techniques, and interpretation  is required. There is no well established research on the effectiveness of the different approaches. The techniques are easy to understand, but generally difficult to apply to real-world software. By providing extensive worked examples, this book aims to provide a solid basis for both understanding, and applying, various test techniques

   In the second edition, many of the examples have been updated, and the sections on Integration Testing and System Testing have been expanded. The examples have been updated to work with the latest version of the tools used, and a companion website is provided (www.softwaretestingbook.org).

   The software tools used in this book are merely representative–they have not been selected as the best example of each, but rather as good examples of the range of capabilities that such tools exhibit. The key goal of the book is that the reader should understand the principles of software testing,  and be able to apply them in practice.   The book does not endorse or recommend any particular tool.   Only a subset of the   tool features are covered in this book: the reader should refer to the tool-specific documentation for more details.

前　　言

本书的内容基于爱尔兰国立大学梅努斯分校和浙江大学的一系列课程讲稿。该书覆盖了软件测试的基本原理，可以作为许多课程的参考教材。本书的内容历经十年发展，既反映了其一位作者二十年的工业界经验，也融合了多位作者在教学方面的共同经验。

在软件测试领域，目前还没有统一的标准教科书，而本书是通过对各种不同的软件测试技术进行多年的提炼、阐释而形成。这些测试技术的来源包括一些经典的测试书籍如Myers的《软件测试的艺术》、Roper的《软件测试》、Binder的《面向对象系统测试》、Ammann和Offutt的《软件测试基础》；标准的软件工程教材如Pressman的《软件工程：实践者的研究方法》、Sommerville的《软件工程》；软件过程类书籍如Kit的《现实世界中的软件测试》、Beck的《极限编程》以及软件质量及测试相关的ISO和IEEE标准。

软件测试是一项具有挑战性的任务，它对科研机构、企业及政府具有同等的重要性。软件测试既是门科学也是门艺术。针对如何应用不同的软件测试技术，目前还没有广泛认同的标准，因此对各种软件测试技术的阐释是必须的。针对各种不同技术的有效性，目前还没有成熟的研究成果。总的来说，各种测试技术容易理解，但如何将其应用到软件产品中是困难的。本书旨在通过广泛丰富的实例，为读者对理解和如何应用各种不同的软件测试技术提供一个坚实的基础。

第2版对第1版的内容进行了全面更新和修订，新增和扩充了集成测试和系统测试两章。示例已更新，以使用新版本的工具，并提供了一个配套网站（www.soft-wareetestingbook.org）。

本书中使用的软件工具仅仅是这些工具中的代表—我们选择这些工具并不是因为它们是最佳工具，而是这些工具能够较好地展示我们想要说明的功能。本书的主要目标是让读者了解软件测试的原理，并能在实践中应用。本书不支持或推荐任何特定的工具。书中只介绍了工具的一部分特性：读者应该参考特定工具的详细文档。

Contents

Preface ii

1 Introduction   1

1.1 The Software Industry    1

1.1.1 Software Testing and Quality   2

1.1.2 Software Testing and Risk Management      3

1.2 Mistakes, Faults and Failures 3

1.2.1 Software Faults      4

1.2.2 Software Failures   5

1.2.3 Need for Testing   6

1.3 The Role of Specifications     8

1.4 Overview of Testing 9

1.4.1 Testing in the Development Process     9

1.4.2 Test Automation   9

1.5 The Theory of Testing     10

1.5.1 Exhaustive Testing Example      11

1.5.2 Implications    11

1.5.3 When To Finish Testing      12

1.6 Book Structure   13

1.6.1 Order of Testing    14

1.6.2 Programming Language    14

1.6.3 Level of Detail 14

1.6.4 Code Examples     14

1.7 Notes on Terminology   14

2 Principles  of Software Testing     15

2.1 Static Verification     15

2.1.1 Walk-throughs      16

2.1.2 Code Reviews/Inspections 16

2.1.3 Formal Methods and Testing    18

2.2 Dynamic Verification      19

2.3 Black-Box and White-Box Testing     19

2.3.1 Errors of “Omission” and “Commission”      21

2.4 Test Approaches       22

2.4.1 Black-Box Testing 22

2.4.2 White-Box Testing       23

2.4.3 Fault Insertion 24

2.5 Test Design Process 24

2.5.1     Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   25

2.5.2     Generating Test Cases . . . . . . . . . . . . . . . . . . . . . . . . .         25

2.5.3     Generating Test Data . . . . . . . . . . . . . . . . . . . . . . . . .   26

2.5.4     Implementing Tests . . . . . . . . . . . . . . . . . . . . . . . . . .    26

2.6 Analy    sis of Software Specifications . . . . . . . . . . . . . . . . . . . . . .   27

2.6.1     Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .        27

2.6.2     Parameter Ranges . . . . . . . . . . . . . . . . . . . . . . . . . . .    27

2.6.3     Equivalence Partitions . . . . . . . . . . . . . . . . . . . . . . . . .         28

2.6.4     Boundary Values . . . . . . . . . . . . . . . . . . . . . . . . . . . .     29

2.6.5     Combinations of Values . . . . . . . . . . . . . . . . . . . . . . . .        30

2.6.6     Sequences of Values . . . . . . . . . . . . . . . . . . . . . . . . . .   38

2.7 Analy    sis of Software Components . . . . . . . . . . . . . . . . . . . . . .    40

2.7.1     Control Flow Graphs . . . . . . . . . . . . . . . . . . . . . . . . . .   40

2.7.2     Decisions and Conditions . . . . . . . . . . . . . . . . . . . . . . .       43

2.7.3     Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .     44

2.7.4     Data Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .         47

2.8 Analy    sis of Targets for Fault Insertion . . . . . . . . . . . . . . . . . . . .   47

2.8.1     Offutt’s 5 Sufficient Mutations   . . . . . . . . . . . . . . .  . . . . .     47

2.9 Test       Artefacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .         48

3 Unit Testing   50

3.1 Techniques 50

3.2 Usage   51

3.3 Black-Box Techniques    52

3.3.1 Equivalence Partitioning (EP)    52

3.3.2 Boundary Value Analysis (BVA) 53

3.3.3 Testing Combinations of Inputs      55

3.3.4 Testing Sequences of Inputs/State-Based Testing    56

3.3.5  Testing with Random Data . . . . . . . . . . . . . . . . . . . . . .  56

3.3.6 Error Guessing/Expert Opinion . . . . . . . . . . . . . . . . . . . .  58

3.4 White-Box Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  59

3.4.1Statement Coverage (SC) . . . . . . . . . . . . . . . . . . . . . . .  60

3.4.2 Branch Coverage (BC) . . . . . . . . . . . . . . . . . . . . . . . . .  61

3.4.3 Condition Coverage (CC) . . . . . . . . . . .  . . . . . . . . . . .  62

3.4.4 Decision Condition Coverage (DCC) . . . . . . . . . . . . . . . . .  64

3.4.5 Multiple Condition Coverage (MCC) . . . . . . . . . . . . . . . . .65

3.4.6 Modified Condition Decision Coverage (MCDC) . . . . . . . . . .66

3.4.7 Path Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  67

3.4.8 Dataflow Coverage (DU Pairs) . . . . . . . . . . . . . . . . . . . .  68

3.5 Fault Insertion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  69

3.5.1 Strong Mutation Testing . . . . . . . . . . . . . . . . . . . . . . . .  69

3.6 Test Ranking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  70

4 Unit Testing Examples 71

4.1 Example One: fits() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  71

4.1.1     Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  71

4.1.2     Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  72

4.1.3     Note on Black-Box Testing . . . . . . . . . . . . . . . . . . . . . .  72

4.1.4     Equivalence Partitioning . . . . . . . . . . . . . . . . . . . . . . . .  72

4.1.5Boundary Value Analysis . . . . . . . . . . . . . . . . . . . . . . .  77

4.1.6 Combinational Testing . . . . . . . . . . . . . . . . . . . . . . . . .  78

4.1.7 Random Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . .  81

4.1.8 Testing Sequences of Inputs . . . . . . . . . . . . . . . . . . . . . .  82

4.1.9 Elimination of Duplicate Black-Box Tests . . . . . . . . . . . . . .82

4.1.10 Important Note on White-Box Testing . . . . . . . . . . . . . . . .83

4.1.11 Source Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  84

4.1.12 Statement Coverage . . . . . . . . . . . . . . . . . . . . . . . . . .  84

4.1.13 Branch Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . .  85

4.1.14 Condition Coverage . . . . . . . . . . . . . . . . . . . . . . . . . .  87

4.1.15 Decision/Condition Coverage . . . . . . . . . . . . . . . . . . . . .  88

4.1.16 Multiple Condition Coverage . . . . . . . . . . . . . . . . . . . . .  90

4.1.17 Modified Condition Decision Coverage . . . . . . . . . . . . . . . .90

4.1.18 Path Coverage  . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  92

4.1.19 Dataflow Coverage (DU Pairs) . . . . . . . . . . . . . . . . . . . .  93

4.1.20 Elimination of Duplicate Tests . . . . . . . . . . . . . . . . . . . .  95

4.1.21 Test Implementation . . . . . . . . . . . . . . . . . . . . . . . . . .  96

4.2 Example Two: premium()      97

4.2.1 Description     97

4.2.2 Specification   97

4.2.3 Equivalence Partitioning     97

4.2.4 Boundary Value Analysis   101

4.2.5 Combinational Testing . . . . . . . . . . . . . . . . . . . . . . . . .  102

4.2.6 Random Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . .  105

4.2.7 Elimination of Duplicate Black-Box Tests . . . . . . . . . . . . . .105

4.2.8 Source Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  107

4.2.9 Statement Coverage . . . . . . . . . . . . . . . . . . . . . . . . . .  107

4.2.10 Branch Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . .  108

4.2.11 Condition Coverage . . . . . . . . . . . . . . . . . . . . . . . . . .  110

4.2.12 Decision/Condition Coverage . . . . . . . . . . . . . . . . . . . . .  111

4.2.13 Multiple Condition Coverage . . . . . . . . . . . . . . . . . . . . .  113

4.2.14 Modified Condition Decision Coverage . . . . . . . . . . . . . . . .116

4.2.15 Path Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  117

4.2.16 Dataflow Coverage (DU Pairs) . . . . . . . . . . . . . . . . . . . .   119

4.2.17 Elimination of Duplicate Tests . . . . . . . . . . . . . . . . . . . .  121

4.2.18 Test Implementation . . . . . . . . . . . . . . . . . . . . . . . . . .  121

5 Unit  Testing Object-Oriented Software    123

5.1 Characteristics of Object-Oriented Software  124

5.2 Effects of OO on Testing       124

5.3 Object-Oriented Testing Models 125

5.3.1 Conventional Models  125

5.3.2 Combinational Models       126

5.3.3 State Machine Models 127

5.3.4 Specification & Design Models 129

5.3.5 Built-In-Test   130

5.4 Example 1   131

5.4.1 Class CarTax    131

5.4.2 Black-Box Testing in Class Context 131

5.4.3 Equivalence Partitioning     132

5.4.4 Boundary Value Analysis   134

5.4.5 Combinational Testing 134

5.4.6 White-Box Testing in  Class Context    134

5.4.7 Combinational Testing at the Class Level     135

5.4.8 State-Machine Testing 137

5.4.9 Specification/Design Testing    141

5.4.10 Built-In Testing    141

5.5 Example 2   143

5.5.1 Classes Shape and Square 143

5.5.2 Specification/Design Testing    143

6 Integration Testing      145

6.1 Principles . . . . . . . . . . . . . . . . .    . . . . . . . . . . . . . . . . . . . 145

6.1.1     Analysis . . . . . . . . . . . . . .      . . . . . . . . . . . . . . . . . . . 146

6.1.2     Test Cases . . . . . . . . . . . . .   . . . . . . . . . . . . . . . . . . . 147

6.1.3     Test Data . . . . . . . . . . . . .     . . . . . . . . . . . . . . . . . . . 147

6.1.4     Test Implementation . . . . . . .     . . . . . . . . . . . . . . . . . . . 147

6.2 Example . . . . . . . . . . . . . . . . .      . . . . . . . . . . . . . . . . . . . 147

6.2.1     The Premium System . . . . . .      . . . . . . . . . . . . . . . . . . . 148

6.2.2     Sequence Diagram . . . . . . . .      . . . . . . . . . . . . . . . . . . . 149

6.2.3     Source Code . . . . . . . . . . .   . . . . . . . . . . . . . . . . . . . 150

6.2.4     Analysis . . . . . . . . . . . . . .      . . . . . . . . . . . . . . . . . . . 150

6.2.5     Test Cases . . . . . . . . . . . . .   . . . . . . . . . . . . . . . . . . . 152

6.2.6     Test Data . . . . . . . . . . . . .     . . . . . . . . . . . . . . . . . . . 152

6.2.7     Test Implementation . . . . . . .     . . . . . . . . . . . . . . . . . . . 153

6.2.8     Interpretation of Test Results . .        . . . . . . . . . . . . . . . . . . . 153

6.3External Integration Testing . . . . . .   . . . . . . . . . . . . . . . . . . . 153

6.4 Unit Testing during Software Integration 154

6.4.1 Drivers and Stubs  154

6.4.2 Top-Down Integration 154

6.4.3 Bottom-Up Integration      155

6.4.4 Incremental Feature/Continuous Integration      155

7 System Testing     156

7.1 Introduction       156

7.1.1 System Test Categories      156

7.2 Usage   158

7.2.1 System Level Functional Testing      158

7.2.2 Test Cases       160

7.3 GUI Testing Example      160

7.3.1 Example System    161

7.3.2 Testing the Interface Behaviour       167

7.3.3 Testing the Interface Navigation     168

7.3.4 Testing Software Features  170

7.3.5 Testing Use Cases and Scenarios    173

7.3.6 Testing a Web-Based GUI  174

7.4 Field Testing and Acceptance Testing      175

7.4.1 Field Testing   175

7.4.2 Acceptance Testing     175

8 Software Test Automation 176

8.1 Unit Test Automation     177

8.1.1 TestNG     177

8.1.2 Basic TestNG Features 177

8.1.3 Other TestNG features 179

8.1.4 Documenting Automated Tests      181

8.1.5 Automated Unit Testing in an IDE   182

8.2 Automated Unit Test Example: fits()  182

8.2.1 Testing fits() with Inline Data    182

8.2.2 Parameterised Tests for fits()    183

8.3 Coverage Measurement Automation 186

8.3.1 EclEmma Example 186

8.3.2 Lazy Evaluation      187

8.4 Automated Object-Oriented Unit Testing      188

8.4.1 CarTax Example     188

8.4.2 Automating Inheritance Testing      188

8.5 Automated System Testing   190

8.5.1 Swing-Based GUI Example 191

8.5.2 Web-Based GUI Example   197

8.6 Selenium IDE (Integrated Development Environment) . . . . . . . . . . .205

8.7 Other Test Automation Tools and Techniques . . . . . . . . . . . . . . . .205

8.7.1 Customising the Test Runner    205

8.7.2 Stress Testing 206

8.7.3 Mutation Testing   207

9 Testing  in the Software Process   209

9.1 Test Planning     210

9.2 Software Development Life Cycle      211

9.3 The Waterfall Model       211

9.4 The V-Model .   212

9.5 Incremental and Agile Development 213

9.5.1 Incremental Development  214

9.5.2 Extreme Programming 215

9.5.3 SCRUM     217

9.5.4 DevOps    218

9.5.5 Process-Related Quality Standards and Models 219

9.6 Repair-Based Testing     219

9.6.1 Specific Repair Test      219

9.6.2 Generic Repair Test      219

9.6.3 Abstracted Repair Test 220

9.6.4 Example   220

9.6.5 Repair-Based Test Suites   220

10 Advanced Testing Issues 221

10.1 Philosophy of Testing   221

10.2 Test Technique Selection    222

10.3 Design For Testability (DFT)       222

10.4 Overlapping  and Discontinuous Partitions 223

10.4.1 Overlapping Input Partitions  223

10.4.2 Overlapping Output Partitions      223

10.4.3 Inband Error Reporting     224

10.5 Handling Relative Values    225

10.5.1 Classic Triangle Problem  226

10.6 Pairwise Testing     227

10.7 Modified Combinational Testing      228

10.8 Improved Condition Coverage  229

10.9 Testing with Floating Point Numbers     229

10.10 Testing with Complex Data Structures  231

10.11 State-Based  or Sequential Testing     231

10.12 Inserting Data Faults  235

10.13 Automated Statement and Branch Testing 235

10.14 Automated Random Testing   235

10.15 OO Dataflow Testing  (DU Pairs)  237

10.16 Verifying State in  Object-Oriented Testing    239

10.17 Advanced OO Testing       240

10.17.1 Multiplicity in UML Class Diagrams   241

10.18 Text-Based Value Representation  242

10.19 Unit Testing  of GUI Components       243

10.20 Some issues in Testing Web-Based Applications    244

10.21 Responsive UI Design Testing 244

10.22 Testing  Concurrent and Parallel Software       244

10.22.1 Unit Testing 245

10.22.2 System Testing  245

10.22.3 Static Analysis   245

10.22.4 Tools     245

10.23 Testing Embedded Software    246

10.24 Testing  Network Protocol Processing       248

10.24.1 Text-Based Protocols     248

10.24.2 Binary Protocols       249

10.24.3 Protocol Stacks and DFT 249

10.25 Including Testing  In The Build Procedure 249

10.26 Research Directions    250

APPENDICES

A Terminology  252

B Software Testing Tools 254

C Exercises  255

C.1 TestNG and Eclipse . . . . . . . . . . . . . . .  . . . . . . . . . . . . . . . 255

C.2 Unit Test - Exercise 1 . . . . . . . . . . . . . .  . . . . . . . . . . . . . . . 257

C.3 Unit Test - Exercise 2 . . . . . . . . . . . . . .  . . . . . . . . . . . . . . . 258

C.4 Unit Test - Exercise 3 . . . . . . . . . . . . . .  . . . . . . . . . . . . . . . 259

C.5 Unit Test - Exercise 4 . . . . . . . . . . . . . .  . . . . . . . . . . . . . . . 259

C.6 Unit Test - Exercise 5      260

C.7 Unit Test - Exercise 6      261

C.8 Unit Test - Exercise 7      262

C.9 Unit Test - Exercise 8      262

C.10 Unit Test - Exercise 9   263

C.11 Exercise 10 - Test Projects  264

D Source Code 265

D.1 Chapter 4 - Unit Testing Examples   265

D.1.1 CharterFlight.java 265

D.1.2 CharterFlightTest.java 266

D.1.3 Insurance.java 267

D.1.4 InsuranceTest.java 268

D.2 Chapter 5 - OO Testing 269

D.2.1 CarTax.java     269

D.2.2 CarTaxTest.java     270

D.2.3 CarTaxWithAssertions.java 271

D.2.4 CarTaxWithAssertionsTest.java 273

D.2.5 Shape.java      273

D.2.6 Square.java     274

D.2.7 ShapeTest.java      274

D.2.8 SquareTest.java     274

D.3 Chapter 6 - Integration Testing  275

D.3.1 PremiumCalculator.java     275

D.3.2 Validator.java 276

D.3.3 PremiumCalculatorIntegrationTest.java 276

D.4 Chapter 7 - System Testing 277

D.4.1 FlightBooker.java  277

D.4.2 SimpleSwingTestFramework.java    279

D.4.3 FlightBookerBehaviourTest.java      282

D.4.4 FlightBookerNavTest.java  283

D.4.5 FlightBookerFeatureTest.java   285

D.4.6 FlightBookerScenarioTest.java 286

D.4.7 FlightBookerWebTest.java 288

D.5 Chapter 8 - Software Test Automation    292

D.5.1 Demo.java      292

D.5.2 DemoTest.java      293

D.5.3 DemoTestM.java   293

D.5.4 DemoTestA.java    294

D.5.5 DemoTestB.java    294

D.5.6 Demo2.java    295

D.5.7 Demo2Test.java    295

D.5.8 testng-1.xml  295

D.5.9 CharterFlightTestInline.java      296

D.5.10 CustomRunner.java   296

D.6 Chapter 10 - Advanced Issues   298

D.6.1 Lamp1.java     298

D.6.2 Lamp2.java     298

D.6.3 Lamp1Test.java     298

D.6.4 Lamp2Test.java     299

Select Bibliography 300