Software Testing

eXtreme Testing Is Elementary, My Dear Watson.

Getting started with Xtreme testing
Be not astonished at new ideas; for it is well known to you that a thing
does not therefore cease to be true because it is not accepted by many.
Baruch Spinoza (1632-1677) 

The following recommendations highlight the main ideas which black/gray box tester needs to know and implement during transition to Extreme Testing. I presume that you have already read an article or a book on Extreme Programming (XP). If not, you can read about it on the Web at and in Extreme Programming Explained by Kent Beck. Please pay attention on the main two practices of XP "write tests first" and "pair programming." Excellent abstract about XP you can find in the article " Programming at the Edge of Chaos" by Brian Foote and Laura Hill. They say about designers of XP: "They simply had to remove a few roadblocks, and allow it to unfold in front of them. They didn't as much have to craft it, as make sure they got out of its way. They simply had to observe what worked, and what didn't, and take pieces of what worked, and fit them together. The brilliance was in simply having had the boldness and brashness to do so. This is far from a simple thing." The concept of XP is the brightest set of guidelines in software development. Xtreme testing must follow the same rules as XP (when these rules make sense from testing point of view) and take the best from research and practical experience. The concept of Xtreme testing must be based on the fact that there is no single approach that will work for all companies. Nor should it expect that all the pieces from one methodology are the best of the breed. The right mix of good practices will differ for each individual project or company. But some of them (like write test first) come along with any methodology. I prefer "design test first". Good test must be designed like a good software. Xtreme testing must eliminate, as much as possible, the unnecessary elements of building software. Remember: a relatively small increase in methodology size or density adds a relatively large amount to the project cost. XP is the less complicated in implementation methodology comparing to others. It shows better results if you ask for return on investment (ROI). Any increase in methodology has to show some proven ROI before company even begins to consider it. You have to create a set of guidelines for implementing Xtreme testing. Some of these guidelines may reflect other methodologies. Recommendation for implementing Xtreme Testing Remember an old Chinese proverb: "There are many mountains up to heaven and many pathways up each mountain." My opinion on this topic is little different from common. 1. Define what is considered to be Quality for your system and then begin to build Xtreme testing process to assure you meet the standard you set. My suggestion is that quality for Xtreme testing project should be defined as achieving the target, increasing the testability and minimizing the variability of the system under the test. The fact that minimizing the variability of the system under the test increases its quality has been proved long ago. Designing unit tests first forces you to build modular, unit testable objects (see the vision page) and is only the first step for minimizing the system variability and increasing testability. Read Taguchi if you would like to find more about minimizing the variability of the system under the test. It is possible to minimize variability of the system during requirements defining; architectural design, GUI design and coding . Minimizing variability during test cases design helps to save time in a future maintenance of the test suite. Remember that: * Xtreme testing is a major source of quality feedback * Xtreme testing is a key to customer satisfaction. 2. XP suggests to works in pairs. Are two brains better than one? Sure. However, is productivity increased by two? Instead of working in pairs as XP suggests for developers, use rotation team (even for this activity you need to schedule additional time for switching) where a less experienced tester starts the cycle. Rotation can be based for example on the iteration cycles. 3. Xtreme testing embodies communication with focus on simplicity: use person-to-person communication instead of written documents wherever possible. Cooperation between developers and testers through all development life cycle is essential. One of the rules is: Add document, or process only when you really need them, not in anticipation of need. If you still have to create a document - it should contain essentials only. * Are you afraid that a tester will leave in the middle of the project? Schedule an additional time for him to prepare an extensive documentation. 4. In fact extreme programming/testing provides better risk management than many other methods. Let us begin Xtreme testing (set of guidelines): "The whole is more than the sum of its parts." Aristotle, in Metaphysics (384 - 322 B.C.) * Define the target. * Define the minimum necessary documentation for testing that must be created. * Design test first. ( Try to specify as many test cases as possible before the code is written.) * Let developers write unit tests according to XP. Define 100% line coverage of code for unit testing as a must. This means only that developers' work is finished, not that code is bugs free. * Try to eliminate redundancy of TC for unit and system testing. * Try to avoid frequent builds for bugs fixes and changes after promotion. An every night build is the good practice only during development process. Try to avoid (as much as you can) supporting different versions of applications in production.. * Try to cover with test cases all layers of the system. (define TC related to the system architecture and environment) * Try to make all testing processes more robust. * Allow working together developers and testers (deep integration.) I mean here not pairs of developers or testers but developers and testers working in one group. And try to make some innovations like: * Using a rotation team. * Implementing a gray box testing. * Participating of testers on all architecture related meetings and walkthrough. * Developers reviewing TC running by testers and vice versa. * Testers involving in the business analysis and requirements review. This can help to increase testability and minimize the variability of the system under test. * Use UML model and diagrams for defining TC: This is not the spirit of XP, but from my experience it helps to create better TC. Unfortunately, special diagrams and notations for testers do not exist yet. I believe that easily understandable graphical TC is the only right way for improvement the testing process. Class diagram for unit test cases Sequence diagram for integration test cases Use Case and Activity diagram for system test cases * Use orthogonal array technique to reduce the number of TC. * Next step will be creating a built-in diagnostic for any of: External system Third party libraries Database Configuration parameters Built-in verification and calculation of a control example for the end-to-end scenario - is a good decision for some systems. Allow developers to concentrate on: * Developing new and running existing unit test cases. * Applying analysis tools such as code coverage directly into your JUnit tests * Generating automatically unit tests from class definitions and UML models. Allow tester (i.e. black box tester) to concentrate on: * Integration testing of the system. * Business cycle testing (system behavior in certain time frames). * Concurrency testing. * Integration testing of any third party components. * Specification based testing, scenario testing. * Database testing using CRUD matrix. (gray box testing method) * Other testing types like: load, performance, installation, volume, user documentation, disaster recovery etc. Conclusion: "Trust, but verify" (the old Russian proverb). Ronald Reagan, speaking to Mikhail Gorbachev. We know that any good idea potentially can be misused. If we ruled out ideas because of their potential misuse, we wouldn't advance anywhere. Many different development and testing models have been proposed in books, journals and on conferences, each one having its own assumptions. The concept of XP is the brightest of them. My experience shows that "design test first" practice not only helps to create better software but also allows manage changes quickly. So let us be careful, trust in Xtreme testing and believe that competence and attitude of individual testers are the most important components of a successful project. Recommended reading: R. Jeffries, "Extreme Testing," Software Testing and Quality Eng., Mar./Apr. 1999, pp. 23-26. Planning Extreme Programming by Kent Beck, Martin Fowler, 2000 Useful sites: Extreme programming practices, extreme programming software methodology, agile development OK, these are my two cents of input. Good luck. Alex

P.S. If you're thinking of implementing some extreme testing ideas from the above article -
read a review by Phlip, from the Agile Software Testing Forum first.

Software Testing Main Page
© 2002 Alex Samurin