Manage software testing .


Table Of Contents and List of Figures and Tables of the book "Manage software testing ", by Peter Farrell-Vinay., 2008


Introduction............................................................. 1
1.1 Conclusions.........................................................  1
1.2 What Has This Got to Do With Software? .............................. 1
1.3 If It Isn’t Planned, It Won’t Happen ................................ 2
1.4 Let’s Not Test....................................................... 2
1.5 Something to Do When You Really Hit Opposition....................... 5
1.6 Conventions ......................................................... 7
1.7 Apologies ........................................................... 8
1.8 Some Basics.......................................................... 8
1.9 The Players ......................................................... 11
1.10 Quality Assurance and Quality Control .............................. 14
1.11 Processes, Tools, Products, and Standards ...........................15
2
The Big Questions You Need Answers To ....................................17
2.1 Why Do We Test?...................................................... 17
2.2 How Do We Test? ......................................................18
2.3 When Do We Start Testing? ............................................18
2.4 When Do We Stop Testing? .............................................20
2.5 What Do We Test it Against? ..........................................21
2.6 What Do We Test it With? .............................................22
2.7 Are We Getting Better or Worse at Finding Bugs?.......................22
2.8 How Do We Measure Our Test Coverage? .................................26
2.9 Are There Any More Embarrassing Questions? ...........................28
3
Risk Management ..........................................................29
3.1 How to Manage Risk ...................................................30
3.2 A Worked Example......................................................35
3.3 Hazard Analyses and Testing ..........................................37
4
Test Planning and Management..............................................39
4.1 Testing Principles....................................................39
4.2 Laws .................................................................41
4.3 Test Management Principles ...........................................42
4.4 Life-Cycles and Four-Wheel-Drive Karmas...............................43
4.5 Starting from Where You Are ..........................................56
4.6 Arguments You Need to Win.............................................59
vi
Manage Software Testing
5
Testing and the Web.......................................................65
5.1 How Websites Work ....................................................66
5.2 Website Risks and Problems ...........................................67
5.3 Web Test Planning ....................................................69
5.4 Web Performance.......................................................71
5.5 Web Tests.............................................................72
5.6 Monitoring Issues ....................................................82
6
The Overall Test Life-Cycle...............................................83
6.1 Initiation Phase......................................................83
6.2 Requirements Analysis Phase...........................................85
6.3 Architectural Definition Phase .......................................87
6.4 Coding and Unit Testing...............................................88
6.5 Software/Hardware Integration ........................................88
6.6 Test Readiness Review ................................................89
6.7 System-Testing Phase..................................................89
6.8 Beta Testing .........................................................90
6.9 Certification- or Acceptance-Testing Phase ...........................90
6.10 Documentation, Help, and Training Phase .............................91
6.11 Install, Coexist, Cutover, Support, and Evolution Phase .............91
7
Testing Processes and Infrastructure......................................93
7.1 Initiation............................................................93
7.2 Keeping the Configuration Management System in Order..................94
7.3 Test Environment......................................................96
7.4 Test Automation.......................................................97
7.5 Monitoring Test Progress .............................................101
7.6 Bug Management Process and Tool ......................................101
7.7 Operational Profiles..................................................109
7.8 Static Analyzers .....................................................115
7.9 Dynamic Analyzers ....................................................119
8
Test Documents ...........................................................125
8.1 General ..............................................................125
8.2 Requirements Specification............................................125
8.3 Test Strategy Document ...............................................130
8.4 Interface Specification ..............................................132
8.5 Test Plan ............................................................133
8.6 Test Monitoring Document .............................................142
8.7 Test Design Specification ............................................148
8.8 Test Case Specification ..............................................149
8.9 Test Procedure Specification .........................................151
8.10 Release Note ........................................................152
8.11 Test Log ............................................................153
8.12 Bug Report ..........................................................154
8.13 Test Summary ........................................................156
8.14 Usability Test Summary ..............................................157
8.15 Acceptance Certificate ..............................................158
8.16 Security Risk Analysis Report .......................................158
8.17 Risk Log.............................................................160
8.18 Daily Test Report ...................................................160
8.19 Reporting Using TestDirector™ (now Quality CentER) ..................162
8.20 Choosing a Set of Test Documents ....................................166
8.21 Quality Plan ........................................................167
9
The Test Team and Its Context.............................................171
9.1 Recruiting the Test Team..............................................171
Contents
vii
9.2 Test Qualifications ..................................................175
9.3 Test Team Structures .................................................175
9.4 Assessing the Test Team ..............................................176
9.5 Motivating the Test Team..............................................177
9.6 Career Progression....................................................177
9.7 Relations ............................................................178
9.8 The Politics of Testing ..............................................185
10
Outsourcing ..............................................................187
10.1 Dangers of Outsourcing...............................................187
10.2 Process of Outsourcing ..............................................187
10.3 Backsourcing Considerations..........................................202
10.4 Testing Proprietary Software.........................................203
11
Test Techniques: Functional (Black-Box) ..................................205
11.1 Equivalence Partitioning ............................................205
11.2 Boundary-Value Analysis .............................................205
11.3 Cause–Effect Graphing................................................206
11.4 Random Data Selection................................................207
11.5 Feature Test Process.................................................208
12
Test Techniques: Structural (White-Box) ..................................209
12.1 Statement Coverage ..................................................209
12.2 Decision (Branch) Coverage...........................................209
12.3 Condition Coverage...................................................210
12.4 Path Analysis .......................................................211
12.5 All-DU-Paths Coverage ...............................................213
12.6 Execution-Time and Resource-Use Analysis ............................213
12.7 Algorithm Analysis ..................................................213
12.8 Linear Code Sequence and Jump (LCSAJ) ...............................214
12.9 Dynamic Analysis Techniques .........................................215
12.10 Derivative Techniques ..............................................217
13
Test Techniques: Static Analysis..........................................219
13.1 Static Dataflow Analysis ............................................220
13.2 Dataflow Bugs .......................................................221
13.3 Control-Flow Analysis ...............................................221
13.4 Function Value Analysis .............................................222
13.5 Symbolic Execution ..................................................223
13.6 Mutation Testing.....................................................222
13.7 Mutation Testing Problems............................................223
13.8 Fault Injection......................................................224
13.9 Test Data Generators ................................................228
13.10 Failure Mode Effects and Criticality Analysis.......................229
14
Unit Test Process.........................................................231
14.1 Managing Unit Testing ...............................................231
14.2 Purpose of Unit Testing .............................................231
14.3 Identify the Test Baseline ..........................................232
14.4 Identify Unit Characteristics........................................233
14.5 Unit Test Development ...............................................235
14.6 Why the System Test Group Should (Not) Unit Test ....................237
14.7 What We Have to Do to Unit Test .....................................238
14.8 Unit Test Evaluation ................................................239
14.9 Unit-Test Tool Criteria .............................................240
viii
Manage Software Testing
15
System and Acceptance Testing ............................................243
15.1 System Test Process (Figure 15.1) ...................................243
15.2 Plan the System Tests ...............................................246
15.3 Design and Code the Tests............................................250
15.4 System Test Coverage ................................................254
15.5 Is It Worth All This Work?...........................................256
15.6 Test Support Software and Tools......................................260
15.7 Regression Testing...................................................260
15.8 Localization Testing ................................................261
15.9 System Integration and Operations Testing ...........................262
15.10 Non-Functional Testing .............................................262
15.11 Conduct a Test Readiness Review ....................................272
15.12 Beta Testing Process................................................273
15.13 Acceptance Testing..................................................274
15.14 Test a Build........................................................275
15.15 Monitor the Test Process............................................277
15.16 Report .............................................................280
15.17 Agree That the Release is Ready ....................................280
16
Performance and Stress Testing............................................281
16.1 General .............................................................281
16.2 Service Level Agreements.............................................281
16.3 Performance Testing: Tool Use .......................................282
16.4 Performance Testing: Requirements....................................283
16.5 What is the Baseline for Performance Tests? .........................283
16.6 Performance Testing: Process.........................................284
16.7 Stress Testing ......................................................285
16.8 Reliability Testing..................................................287
17
Usability Test Process ...................................................289
17.1 Why We Should Usability Test.........................................289
17.2 When Should We Usability Test?.......................................289
17.3 What is the Baseline for UI Tests?...................................290
17.4 How to Usability Test ...............................................290
17.5 Readability Indices..................................................296
17.6 Task Analysis .......................................................297
17.7 Who Is Involved in Usability Testing?................................298
17.8 How Will We Know if We’ve Usability-Tested Successfully? ............299
17.9 What It’ll Cost Us to Usability Test.................................300
17.10 Why We Shouldn’t Usability Test ....................................301
17.11 Usability-Testing Deliverables......................................301
17.12 Usability Test Specification .......................................302
17.13 Usability Test Warnings.............................................304
18
Metrics...................................................................305
18.1 Functionality Metrics ...............................................305
18.2 Testability..........................................................309
18.3 Performance Metrics .................................................312
18.4 Resilience (Robustness) Metrics .....................................316
18.5 Usability Metrics ...................................................320
18.6 Reliability..........................................................324
18.7 Availability.........................................................327
18.8 Maintainability Metrics .............................................329
18.9 Process Metrics .....................................................331
18.10 Estimates of Bugs in the System.....................................338
18.11 Function Point Metric...............................................351
18.12 Test Case Estimation ...............................................354
18.13 Cost of Finding and Fixing a Coding Bug ............................358
Contents
ix
18.14 Some Metrics from the “Real” World .................................360
18.15 Multiple Regression Analysis .......................................361
A
Examples .................................................................365
A.1 Case Study: Getting Test Objectives Out of a Use Case ................365
A.2 Case Study: Withdraw Cash Use Case ...................................365
A.3 Case Study: Getting Test Objectives Out of Business Rules.............370
A.4 Case Study: Test Coverage.............................................373
A.5 Example: Requirements Traceability Using TestDirector™................379
A.6 Example: How to Get a Report Out of TestDirector .....................379
A.7 Case Study: Performance Testing (Throughput) .........................386
A.8 Case Study: Performance Testing (Bottlenecks and Critical Variables)..389
A.9 Example: End-of-System-Test Report ...................................399
A.10 Example: Test Function Report .......................................404
A.11 Case Study: Testing a Formal Requirements Specification (TransitCard Ticketing System) 406
B
Checklists ...............................................................417
B.1 Test Manager’s Terms of Reference ....................................417
B.2 What Test Managers Do Checklist ......................................420
B.3 Communication, Team Compatibility, and Motivation Checklist...........424
B.4 Bid Review Checklist .................................................424
B.5 Requirements Analysis Checklist.......................................424
B.6 Risk Management Checklist.............................................426
B.7 Test Planning Checklists..............................................427
B.8 Quality Review Checklist .............................................429
B.9 Localization Checklist ...............................................429
B.10 Safety Testing Checklist............................................ 430
B.11 Security Testing Checklist ......................................... 433
B.12 Usability Checklist ................................................ 437
B.13 Web Issues Checklist........................................... .....440
B.14 Performance Checklists ......................................... ....442
B.15 Coding and Unit Testing Review Checklist ........................... 446
B.16 Software–Software Integration Test Review Checklist..................447
B.17 System Test Review Checklist ........................................448
B.18 Operations Acceptance Checklist .....................................448
B.19 Metrics Checklist ...................................................451
B.20 Very Wonderful New Approaches Checklist..............................451
C
Category and Priority Classifications for Bug Reporting...................453
C.1 Purpose...............................................................453
C.2 Classification by Category............................................454
C.3 Classification by Severity Approach 1.................................454
C.4 Classification by Severity Approach 2.................................455
C.5 Beizer’s Taxonomy ....................................................455
C.6 Bug Severity Calculation Matrix ......................................456
D
Tools ....................................................................457
D.1 Automated Testing Tools...............................................457
D.2 Dynamic Analysis Tools ...............................................457
D.3 Executable
UML
Tools ....................................................................458
D.4 Formal Methods’ Tools.................................................458
D.5 Localization Links....................................................458
D.6 Metrics Tools.........................................................459
D.7 Performance-Testing Tools.............................................459
D.8 Risk Tools ...........................................................460
D.9 Security Tools........................................................461
D.10 Source Code Comprehension and Analysis Tools.........................461
x
Manage Software Testing
D.11 Static Analysis Tools ...............................................461
D.12 Unit Testing.........................................................462
D.13 Web Tools ...........................................................462
D.14 Web Usability Test Tools and Facilities .............................463
D.15 Web Security ........................................................463
D.16 Other Tools..........................................................463
Glossary..................................................................467
Abbreviations Used .......................................................467
Terms Used ...............................................................470
References................................................................513
Books and Papers .........................................................513
Related Standards.........................................................520
Index.....................................................................527



List of Figures Figure 1.1 Found and fixed bug detection chart ...........................6 Figure 1.2 Overall bug detection profile..................................6 Figure 1.3 Project roles and relations ...................................14 Figure 2.1 Which is the buggiest feature?.................................25 Figure 3.1 A risk management process......................................30 Figure 3.2 Hazard/consequence decision tree ..............................34 Figure 3.3 Hazard analysis process........................................38 Figure 4.1 The waterfall model ...........................................43 Figure 4.2 Cooper’s stage-gate process model .............................44 Figure 4.3 Boehm’s spiral life-cycle model ...............................46 Figure 4.4 UP phases and models ..........................................49 Figure 4.5 Standard UML models and supporting diagrams ...........................................50 Figure 4.6 The “Display Options” point of the “Withdraw Cash” use case....51 Figure 4.7 Executable UML (x UML ) process model A.........................................................52 Figure 4.8 Executable UML (x UML ) process model B ........................................................52 Figure 4.9 Executable UML (x UML ) process model C ........................................................53 Figure 4.10 Formal methods process model .................................55 Figure 4.11 A bug detection curve of a project which is a long way from release 62 Figure 4.12 Release readiness shown in terms of a bug curve...............62 Figure 5.1 A typical web architecture ....................................66 Figure 5.2 Web interactions...............................................66 Figure 5.3 Web test architecture..........................................72 Figure 5.4 Payment processing.............................................76 Figure 5.5 Simple reliability block diagram ..............................79 Figure 5.6 Probabilities of failure.......................................80 Figure 6.1 A test process ................................................84 Figure 6.2 A PERT chart of the system definition phase ...................85 Figure 6.3 An untestable requirement (just what contains what?) ..........87 Figure 7.1 Item acceptance review process.................................94 Figure 7.2 Bug management process ........................................102 Figure 7.3 Cumulative bugs chart .........................................104 Figure 7.4 Chart showing cumulative bugs and fixes........................105 Figure 7.5 Chart showing cumulative bugs and fixes with the addition of extra effort from day 18...............................................................105 xii Manage Software Testing Figure 7.6 Chart showing cumulative bugs found, split out by priorities...105 Figure 7.7 Chart showing bugs found and bugs predicted....................106 Figure 7.8 Proportions of system tests run on 5 features .................106 Figure 7.9 Proportion of units executed with and without bugs ............106 Figure 7.10 Quality gate effectiveness....................................108 Figure 7.11 Bug detection effectiveness ..................................109 Figure 7.12 Bug injection.................................................109 Figure 7.13 Product use process model.....................................111 Figure 7.14 Key input variable matrix ....................................113 Figure 7.15 Example of a test tool reference listing .....................115 Figure 7.16 Example of a tool management summary .........................116 Figure 7.17 Extract from a test path analysis listing showing an LCSAJ....116 Figure 7.18 Faulty decision coverage evaluation...........................116 Figure 7.19 Structured program verification definition (ADA) .............117 Figure 7.20 Example of complexity analysis output.........................117 Figure 7.21 Example of a procedure call summary...........................118 Figure 7.22 Example of dataflow analysis error messages ..................118 Figure 7.23 Example of parameter analysis output .........................119 Figure 7.24 Example of the output of a cross-referencer...................120 Figure 7.25 Example of statement coverage summary (TER 1).................121 Figure 7.26 Example of branch execution profile...........................121 Figure 7.27 Example of branch execution summary...........................121 Figure 7.28 Example of test path execution profile........................122 Figure 7.29 Example of test path execution summary........................122 Figure 8.1 Example of a user manual ......................................126 Figure 8.2 Example of a requirements specification .......................127 Figure 8.3 Example of where to insert new requirements ...................127 Figure 8.4 Requirements specification table of contents...................129 Figure 8.5 Alternative boilerplate test plan specification of suspension criteria (severity)136 Figure 8.6 Alternative boilerplate test plan specification of suspension criteria (type) ...137 Figure 8.7 Boilerplate test plan specification of resumption criteria ......................137 Figure 8.8 Boilerplate test plan specification of system test ending criteria ..............137 Figure 8.9 Boilerplate test plan specification of handover meeting and release note contents139 Figure 8.10 The raster display controller block diagram ..................139 Figure 8.11 A HOOD diagram for a raster display controller................140 Figure 8.12 An integration tree ..........................................141 Figure 8.13 Key to the preceding figure ..................................142 Figure 8.14 Document status tab ..........................................142 Figure 8.15 RFC list tab .................................................143 Figure 8.16 Test status tab ..............................................143 Figure 8.17 Test events log tab ..........................................144 Figure 8.18 Crash log tab.................................................144 Figure 8.19 Weekly bug report count tab...................................144 Figure 8.20 Test execution status ........................................145 Figure 8.21 Overall bug detection profile.................................146 Figure 8.22 Open and closed critical bug reports..........................146 Figure 8.23 Build quality profile expressed as an absolute ...............147 List of Figures xiii Figure 8.24 Build quality profile expressed as a proportion...............147 Figure 8.25 Feature stability profile ....................................148 Figure 8.26 Kiviat diagram of task/bug status per release ................157 Figure 8.27 Acceptance test certificate ..................................158 Figure 8.28 Bug detection chart ..........................................161 Figure 8.29 Bug sources ..................................................162 Figure 10.1 Identify outsource project process model A....................189 Figure 10.2 Identify outsource project process model B....................190 Figure 10.3 Find the supplier process model A ............................191 Figure 10.4 Find the supplier process model B ............................192 Figure 10.5 Find the supplier process model C ............................193 Figure 10.6 Tell them what you want process model ............................................................195 Figure 10.7 Make a release................................................198 Figure 10.8 Supplier creates and tests a build............................200 Figure 10.9 Customer accepts a release ...................................201 Figure 10.10 Plan structure ..............................................203 Figure 11.1 Data ranges ..................................................205 Figure 11.2 Cause–effect graph constructs.................................206 Figure 11.3 Cause–effect graph example ...................................207 Figure 11.4 Cause–effect identification ..................................207 Figure 11.5 A cause-and-effect graph......................................207 Figure 11.6 Example of a cause–effect decision chart......................208 Figure 12.1 A simple program .............................................211 Figure 12.2 A program tree ...............................................211 Figure 12.3 Logically infeasible paths ...................................212 Figure 12.4 A control control-flow diagram................................212 Figure 12.5 Example of instrumented code .................................216 Figure 12.6 Example of expanded, instrumented code........................217 Figure 13.1 Static control-flowgraph .....................................220 Figure 13.2 Dynamic control-flowgraph ....................................220 Figure 13.3 ADA program listing...........................................221 Figure 13.4 Code fragment.................................................222 Figure 13.5 Code fragment rewritten ......................................222 Figure 13.6 Code before mutation..........................................223 Figure 13.7 Code with a mutation check ...................................223 Figure 13.8 Bug propagation ..............................................224 Figure 13.9 Assertion demonstrating a bug ................................225 Figure 13.10 Confidence-level equation ...................................225 Figure 13.11 Fault-injection process .....................................227 Figure 13.12 A TAN function...............................................229 Figure 13.13 Failure mode effects and criticality analysis form...........229 Figure 14.1 Classes and subclasses .......................................233 Figure 15.1 System test process ..........................................244 Figure 15.2 A raw assertion in a requirements specification ..............252 Figure 15.3 An analyzed assertion ........................................252 Figure 15.4 A not-very-useful test objective .............................253 Figure 15.5 A system test objective ......................................253 xiv Manage Software Testing Figure 15.6 System test definition details ...............................254 Figure 15.7 System test operator actions..................................254 Figure 15.8 System test pass–fail criteria ...............................254 Figure 15.9 “Open” screen ................................................258 Figure 15.10 “Open” screen rendered on a spreadsheet......................258 Figure 15.11 Feature/feature interaction example .........................259 Figure 15.12 Feature/feature interaction matrix example...................259 Figure 15.13 eCommerce architecture example...............................263 Figure 15.14 Task/objective matrix example ...............................272 Figure 15.15 Bugs found in three projects expressed as percentages........279 Figure 16.1 Performance testing environment...............................283 Figure 16.2 Performance test process model................................286 Figure 16.3 Relationship between response time and throughput.............287 Figure 17.1 Exploration phase process ....................................291 Figure 17.2 Inception phase process ......................................293 Figure 17.3 Execution phase process ......................................295 Figure 17.4 Evaluation phase process .....................................296 Figure 18.1 Performance influence table: data entry.......................313 Figure 18.2 Performance influence table: determining the variable outcomes313 Figure 18.3 Performance influence table: determining the instance factors.314 Figure 18.4 Performance influence table: determining the primary variables315 Figure 18.5 Performance influence table: identifying the most important primary variable316 Figure 18.6 A system modeled as a Markov chain ...........................318 Figure 18.7 A system modeled as a Markov chain with probabilities.........318 Figure 18.8 A system modeled as a Markov chain with probabilities after 4 iterations319 Figure 18.9 The iterations converge the values............................320 Figure 18.10 Example of a maintainability comparison of three applications; none of them is improving, although efforts are sporadically made .....................329 Figure 18.11 Bugs found and fixed overall.................................332 Figure 18.12 Mean age of bugs ............................................333 Figure 18.13 Mean age of open bugs example chart .........................333 Figure 18.14 Code turmoil from August to November.........................336 Figure 18.15 Bug data showing the phase in which each is created and found337 Figure 18.16 Bug data showing how the bug removal effectiveness of each phase is calculated337 Figure 18.17 Bug detection effectiveness chart ...........................338 Figure 18.18 What happens to an out-of-control project 1..................340 Figure 18.19 What happens to an out-of-control project 2..................340 Figure 18.20 Bug prediction spreadsheet ..................................342 Figure 18.21 Adding the raw bug data .....................................348 Figure 18.22 Echoing and preparing the raw bug data ......................348 Figure 18.23 Preparing the predicted and actual data .....................349 Figure 18.24 The Weibull curve (also known as the probability density function)350 Figure 18.25 Bug severity distribution....................................350 Figure 18.26 Bug severity distribution source data .......................351 Figure 18.27 Features and tests ..........................................354 Figure 18.28 A typical unit ..............................................355 Figure 18.29 Number of executable paths...................................356 List of Figures xv Figure 18.30 Multiple regression: data input .............................361 Figure 18.31 Multiple regression: calculations............................362 Figure 18.32 Multiple regression: equations ..............................362 Figure 18.33 Multiple regression: substitution............................363 Figure 18.34 Linear multiple correlation coefficients.....................363 Figure 18.35 Multiple determination coefficient of bugs...................363 Figure A.1 Use-case diagram for the Withdraw Cash use case ...............365 Figure A.2 Withdraw Cash activity diagram.................................366 Figure A.3 UML Sequence diagram example “Withdraw Cash”..................................367 Figure A.4 UML Sequence diagram example “Handle Authentication Failures” ................368 Figure A.5 UML Sequence diagram example “Handle the Bank Refusing the Withdrawal”........369 Figure A.6 A business rule in a hierarchy ................................371 Figure A.7 A simple set of use cases for Word ............................374 Figure A.8 Example: Word “File” Pulldown…is transformed into: ............374 Figure A.9 Example: Word “File” Pulldown, specified…and the resulting screen:375 Figure A.10 Example: Word “Project Gallery” screen…is also specified: ....375 Figure A.11 Example: Word “Project Gallery” screen, specified.............375 Figure A.12 Word top-level menus and contents shown as a UML activity diagram........................376 Figure A.13 State transition diagram showing user-initiated events within the Project Gallery screen ...................................................376 Figure A.14 Example of a scenario/object cross-reference .................377 Figure A.15 TestDirector™ Test Schedule pane .............................378 Figure A.16 Tests and Scenarios worksheet ................................378 Figure A.17 TestDirector Test Steps pane..................................378 Figure A.18 Scenario prerequisites and end states ........................379 Figure A.19 TestDirector™ Test Details pane ..............................379 Figure A.20 TestDirector™ Design Steps pane ..............................380 Figure A.21 TestDirector™ Test Schedule pane .............................380 Figure A.22 TestDirector™ Test Runs pane .................................380 Figure A.23 TestDirector™ (Test Plan tab) ................................381 Figure A.24 TestDirector™ (Filter window) ................................381 Figure A.25 TestDirector™ (Filter Condition window).......................381 Figure A.26 TestDirector™ (Save Filter window)............................382 Figure A.27 TestDirector™ (Reports pull-downs)............................382 Figure A.28 TestDirector™ (Planning Reports window) ......................382 Figure A.29 TestDirector™ (Subject Tree Report window) ...................383 Figure A.30 TestDirector™ (Tests with Design Steps Report window) ........383 Figure A.31 TestDirector™ (Tests with Covered Requirements window) .......383 Figure A.32 TestDirector™ (Tests with Associated Defects window) .........384 Figure A.33 TestDirector™ (Report Configuration button)...................384 Figure A.34 TestDirector™ (Custom Fields button) .........................384 Figure A.35 TestDirector™ (Custom Fields window) .........................385 Figure A.36 TestDirector™ (Generate a Report button)......................385 Figure A.37 TestDirector™ (Save All Pages of a Report pull-down) .........385 Figure A.38 TestDirector™ (Save window)...................................385 Figure A.39 Scenario lists ...............................................386 Figure A.40 Weekly averages ..............................................387 xvi Manage Software Testing Figure A.41 Daily transaction profile ....................................387 Figure A.42 Performance test environment .................................388 Figure A.43 Performance test report ......................................388 Figure A.44 Performance test report ......................................389 Figure A.45 BizRool architecture .........................................391 Figure A.46 BREng5 increase the number of variables.......................394 Figure A.47 Performance testing results...................................395 Figure A.48 BizRool performance glossary..................................396 Figure A.49 Test environment 1............................................397 Figure A.50 Test environment 2............................................397 Figure A.51 Test environment 3............................................397 Figure A.52 Bugs raised by severity.......................................400 Figure A.53 Bugs raised by features.......................................400 Figure A.54 Bugs raised and closed overall................................401 Figure A.55 Feature quality ..............................................404 Figure A.56 Sources of bugs...............................................404 Figure A.57 Release history ..............................................405 Figure A.58 Release interval analysis ....................................405 Figure A.59 Bug-finding ability of test environments .....................405 Figure A.60 Bug resolution ...............................................406 Figure A.61 Ticketing subsystem documentation structure ..................407 Figure A.62 TransitCard Charger...........................................407 Figure A.63 2 Access controller and card reader unit system diagram.......408 Figure A.64 2 Access controller and card reader unit functional diagram...408 Figure A.65 State machine view of the TransitCard.........................411 Figure B.1 FTA symbols ...................................................433 Figure C.1 Bug severity calculation matrix................................456

Buy now

Search Now:
In Association with Amazon.ca

List of Tables Table 2.1 The use of testing methods .....................................19 Table 2.2 Late-found bug list ............................................23 Table 2.3 Identifying whether testing or development is affecting quality.25 Table 3.1 Project risks and how the test team can help mitigate them......33 Table 3.2 Hazard/consequence..............................................33 Table 3.3 Simple risk ....................................................36 Table 3.4 Cost of risk mitigation.........................................37 Table 4.1 Complaints made about a failing test function ..................58 Table 5.1 Web test approaches.............................................68 Table 7.1 Where the bugs spring from......................................108 Table 7.2 Where the bugs were caught .....................................109 Table 7.3 Where the bugs were created and caught..........................109 Table 7.4 Sample user profile.............................................110 Table 7.5 Sample system-mode profile .....................................111 Table 7.6 Sample explicit functional profile .............................112 Table 7.7 Scenario probability of occurrence for an employment agency.....113 Table 8.1 A requirements notation of GUI objects .........................128 Table 8.2 Boilerplate test plan introduction .............................134 Table 8.3 Boilerplate test plan specification of suspension criteria .....136 Table 8.4 A bug report ...................................................155 Table 8.5 Risk log example................................................161 Table 10.1 Outsourcing threats and mitigations ...........................188 Table 12.1 Output of a path analysis tool ................................213 Table 13.1 Confidence levels .............................................226 Table 14.1 Average coefficients of object-oriented metrics ...............234 Table 14.2 Unit-testing arguments ........................................237 Table 15.1 Analysis of effort on six major projects ......................247 Table 15.2 Effort expended on the CFC project ............................248 Table 15.3 Bugs per unit reported by Endres...............................248 Table 15.4 Bug detection rates on the CFC project ........................249 Table 15.5 Faults discovered in each phase reported by Lipow..............249 Table 15.6 Relative cost of fixing faults having found them in later phases of software development ..............................................................250 xviii Manage Software Testing Table 15.7 Example chart plotting the system attributes against the various system tests which most exercise that attribute .......................................255 Table 15.8 Example chart plotting the system features against the various system tests256 Table 15.9 Regression test candidate examples ............................261 Table 15.10 Web testing combinations spreadsheet..........................264 Table 15.11 A project status table........................................277 Table 17.1 Cost of usability testing example .............................300 Table 17.2 RoI on usability calculation...................................301 Table 17.3 Usability RoI calculation......................................301 Table 17.4 Usability specification .......................................303 Table 18.1 MTTF Example ..................................................317 Table 18.2 The iterations converge the values.............................319 Table 18.3 Coexistence metrics............................................328 Table 18.4 Example of a coexistence metric: some interference between applications B and D is evident........................................................328 Table 18.5 Code turmoil data key .........................................335 Table 18.6 Code turmoil data .............................................336 Table 18.7 Bug density of four releases...................................339 Table 18.8 IBM data on bug types .........................................345 Table 18.9 ICL data on bug types .........................................345 Table 18.10 Binomial distribution example.................................346 Table 18.11 Albrecht weightings...........................................353 Table 18.12 McCormack’s project statistics ...............................361 Table A.1 Withdraw cash test .............................................369 Table A.2 Use case glossary ..............................................370 Table A.3 Transaction mix.................................................398 Table A.4 Test application characteristics................................398 Table A.5 Number of users/license ........................................398 Table A.6 Current bug status..............................................401 Table A.7 Overall bugs predicted and found ...............................401 Table A.8 Bug levels in some previous releases............................401 Table A.9 Functional Specifications for Wave 1............................403 Table A.10 Interfaces of the ticketing system.............................409 Table A.11 Constraints....................................................409 Table A.12 System test of “Enter A Station” ..............................416 Table B.1 BS ISO/IEC 17799: 2000 Information Technology Code of Practice for Information Security Management Test checklist .......................437 Table B.2 Operations acceptance checklist.................................449


Find more on our page - Software testing books recomendations

On this page I put Table Of Contents and List of Figures and Tables of the book "Manage software testing ", by Peter Farrell-Vinay., 2008



Software Testing Main Page
© January 2006 Alex Samurin http://www.geocities.com/xtremetesting/