Software bug life cycle and defect management.



Every software tester must be trying to prove that after fixing one bug in an application that has X number of bugs, the number of remaining bugs in the same application will be X+1.
Summary:

The purpose of this tutorial is to put together all the information that may be useful for software testers about software bugs, bug life cycle in software testing, bug management during software development life cycle (SDLC), types of bug tracking systems and tools, customization of bug tracking systems and tools, as well as a little bit about communication with the developer, fixing bugs and regression testing.

Tutorial table of contents:
  1. What is a bug in software testing?
  2. Bug life cycle in software testing
  3. Activities for tracking all bugs found through SDLC.
  4. Types of bug tracking software systems and tools.
  5. Customization of bug tracking system
  6. Description of a bug in the software testing.
  7. The probability of locating more errors in an application under testing.
  8. Regression testing and reproducible test cases
  9. Communication between software testers and developers, as well as developers’ view on bug fixing
  10. Conclusion (Murphy's Laws about bugs )
  11. Draft of the requirements for a custom developed bug tracking system

1. What is a bug in software testing?
Let us begin our tutorial with some popular definitions related to software defects /bugs:
The golden rules of software testing defined by Glenford J. Myers, [The Art of Software Testing, 1979]
  • Testing: run program with intent to find an error
  • Synonyms of Bug : glitch, error, goof, slip, fault, blunder, boner, howler, oversight, botch, delusion, elision. [B. Beizer, 1990], defect, issue, problem, variance

    Defect. The difference between the functional specification (including user documentation) and actual program text (source code and data). Often reported as problem and stored in defect-tracking and problem-management system

    Defect. Also called a fault or a bug, a defect is an incorrect part of code that is caused by an error. An error of commission causes a defect of wrong or extra code. An error of omission results in a defect of missing code. A defect may cause one or more failures.[Robert M. Poston, 1996.]

    I think that there is no need to explain the above definitions and so we move on.

    [Bug Management Back to Top]


    2. Bug life cycle in software testing
    1. When a tester finds a variance from requirement, he/she needs to report the problem and try to describe it so, that the defect will be easy to reproduce and understand.
    Open new log in XXX system (Can be Problem Originator like Tester or Technical Support person)
  • Try to reproduce the problem. If the problem is not reproducible, specify in the problem log.
  • Verify whether any duplicates of the same problem is already in the system
  • Enter full description of the problem
  • If necessary print the log & attach any supporting printouts
  • Assign the Priority an Severity of the Problem
  • Assign the Owner as a responsible person to look after the problem resolution (may be Development team lead or product manager/coordinator)
  • In the case when Sales Person receives the problem description from the client, he/she supplies the problem description to the Technical Support with the names of the client and the Sales Person the problem came from. The Technical support will be acting as Problem Originator.

    2. Review the Problem (Owner may be Development Leader) Review the Problem Priority and Severity. If disagree – change the Priority and Severity and specify the reason why the severity or/and Priority was changed:

    3. If this is a bug, but it will not be corrected at this time due to the low Priority/Severity rating, time or resources limitation:
  • Escalate for decision/agreement
  • Set the problem type as appropriate
  • Annotate the log with recommended solution
  • Set status to pending
  • The Development Leader remains as a problem owner until the problem will be re-assign for resolution, corrected, send to training or closed by the management decision with a Pending status assigned

    4. If this is an environmental issue, initiate the environment correction process by assigning to the appropriate person

    5. If this is an Originator’s error:
  • Annotate the problem log with explanation
  • Change problem type to Non a problem, Duplicate or Change Request
  • Get a Problem Originator’s agreement Set status to Void

    6. If the problem will not be corrected, but it was reported by the Technical support or Sales as a client complain:
  • Change problem type to User Education
  • Annotate the problem log with explanation on the workaround
  • Get a Problem Originator’s agreement
  • Assign the Ownership to Product Manager
  • Set status to Deferred
  • Consider the problem correction in the next release

    7. Fix and Unit test a corrected problem (Owner Developer)
  • Update the problem log with resolution information
  • Set the status to Fixed
    Pass back to problem Development Leader for Promotion.
    Notify Technical Writer with Problem Resolution.

    8. Promote any modified programs to the next release, and update the problem status to Promoted. (Owner - Development Leader)

    9. Retest the fix (Originator)
  • Update the problem log
  • Change status to Closed or Re-open Annotate the test execution history with retest information

    10. If the problem is fixed:
  • Change the problem status to Closed
    Originator remains as a problem owner after he/she closes the problem

    11. If the problem is not fixed or other problems were created as a result of the fix:
  • Change status of the problem to Re-open
  • Annotate the test execution history with retest information
    Redirect the ownership to Development Team Leader

    NOTE:

    Priority 1 problems will require immediate attention, because they will hold up the testing. If Priority 1 problems will be discovered, the problem resolution will follow the process that will be structured as follows:

  • Problem will be logged into the XXX system
  • The notification with request for immediate problem resolution will be sent to Development Team Leader
  • The problem fix will be done in the Test environment and promoted if necessary into the development after the fix the retest in test environment is done.

    [Bug Management Back to Top]

    3. Activities for tracking all bugs found through SDLC.
    The process must be defined and approved for tracking all bugs in a bug tracking system through SDLC.

    Every bug must have a unique name that will help find it in the future. The bug name must describe the main details of the defect as well. By correctly tracking we mean that all necessary fields in the bug tracking system must be used correctly. Correct combination of the fields helps in uniquely identifying the bug reported by the tester and for finding duplication in future.It is important to reopen the old, fixed bug instead creating a new bug with the same description.


    [Bug Management Back to Top]

    4. Types of bug tracking software systems and tools.
    For bug tracking, use Bugzilla open-source free bug tracking software , with a web-based interface. Written in Perl, with MySQL database back-end, you can download Bugzilla from http://www.bugzilla.org/ Bugzilla is the best choice if the number of users is greater than 20. Otherwise look for something simpler.

    Remember that the use of expensive QA tools does not inherently lead to the creation of a bug-free environment. The brittle, buggy and costly systems can be built with any methodology, technology and software tools for automation testing. For example, try to compare one of the most expensive defect and change tracking systems ClearQuest, with Bugzilla free open-source bug tracking software, and you will find that there are no big differences.


    And remember: As Voltaire once said, “better is the enemy of good.” If you search the web, you will find that current experience shows that open source software like Bugzilla is the greatest insurance policy a QA lab can have.

    [Bug Management Back to Top]

    5. Fields selection and customization for bug tracking tools .

    Recommendation on bug tracking software customization.

    A variety of free open source, commercial, problem-tracking and bug management software systems and tools is available. These systems and tools allow tracking of all necessary information during software development and the bug life cycle. A detailed description of a bug in this system will give complete information to all involved personnel. See a list of available tools and their comparison on: http://www.aptest.com/bugtrack.html


    In order to select the fields that we will use in our bug tracking system, let us first identify all possible field choices.
    Bug Components -Bug tracking criteria:
    1) Bug # (primary key)
    1) Short Description – (defect name)
    2) Author (tester or person found a bug)
    2) Date bug discovered
    3) Bug Status/state (Open, Pending, Verified, Fixed, in test, Closed,)
    4) Product/Project Name and version # (Found in Build Number)
    5) Software module / Section (if a project has several different code modules)
    6) Operating System— if applicable
    7) Web Browser— if applicable
    8) Test Case name (test case used when bug was discovered)
    9) Priority (1-5, or better using definition High, Medium, Low) Crashes must be P1; Data loss must be P1
    10) Originator or Test Engineer name (tester who discovered the bug or assign to it)
    11) Software Engineer (person assigned to repair bug)
    12) Developer Resolution— (general text entry)
    13) Fixed in Build Number
    14) Keywords 1 & 2 (To describe bugs for future database queries)
    15) Summary of bug (general text entry)
    16) Steps to reproduce (general text entry)
    17) Comments (general text entry)
    18) Attachments— screenshots, files,
    Additional functionality of any bug tracking system:
  • Customizable fields
  • Auto-populate fields
  • Submit by email (automatic Email notification )
  • Attachments

    Let us drill down and add some description to some of the attributes:
    Bug priority (Priority is Business):
  • Must be fixed as soon as possible. Bug is blocking further testing or progress in this area.
  • Should be fixed soon, before product release.
  • Fix if time allows; somewhat trivial. This defect may be postponed. (better create special maintenance release where all this types of defects will be fixed)

    Bug severity can be categorized as follows (Severity is Technical):
  • Cosmetic (Bug contains typos, unclear wording or non descriptive error messages, low visibility fields or difficult not friendly navigation.)
  • Inconvenience (Bug causes minor functionality problems, may affect "fit and finish".)
  • Loss of Function (Bug causes loss of major functionality or other severe problems in the product.)
  • System crash or hang (Bug causes system crashes in obscure cases.)
  • Loss of Data (Bug causes system data loss)

    Defect can be found in different functions of the system:
  • Entry of a data
  • Front-end
  • Backend
  • Database of other types of storage
  • Output
  • Process or logic

    Resolution field (Possible situations)
  • Need More Information
  • Not a Bug
  • Not Reproducible
  • No Plan to Fix
  • User Error

    Choose only the necessary fields from the above list!

    Remember: Adding additional fields to the bug tracking system increases the amount of work to create and maintain the bug. The project cost rises as well. Keep in mind that the same rules apply to the test suite and a test log document.


    [Bug Management Back to Top]

    6. Description of a bug in the software testing.
    The following are sample requirements for the description of a bug in software testing.
    Let’s switch from theory to real world testing. Read the bug description example.
    Note that the main attributes of the defect description in the bug tracking system are:
    1. The description allows easy reproduction of the problem.
    2. The description narrows down the possibilities of recreating the problem.
    3. The description eliminates things that can't be related to the problem.
    4. The description must help to easily find root causes of a failure.
    5. The description must have all critical input values that cause the defect.


    Example:
    Following the sequence below has unexpected result.
    1. open revise applet
    2. enter client number for revise
    3. go to debtor screen, add new debtor with name and surname (P)
    4. go to defendant screen, add a blank record
    5. go to creditor screen, add a blank record
    6. go to judgment screen, add a blank record
    7. click revise button
    8. close error message
    9. remove blank record from defendant screen
    10. click revise button
    11. close error message
    12. remove blank record from creditor screen
    13. click revise button Unexpectedly nothing appears to have happened. No pop-up message of any Sort was displayed. The revise button was simply disabled and the application remained on creditor screen. When go to judgment screen, one can still see the blank record there.

    Revise -difficult navigation from judgment to creditor to defendant and debtor.
    How to reproduce: open revise, go to judgement screen. enter some data, click revise you will return to basic screen with message enter mandatory fields. However, judgment is still highlighted on the tree panel. After entering client number you cannot return to judgement panel by clicking on judgment on the tree panel. You must click on another panel first and then click on judgment.


    [Bug Management Back to Top]

    7. The probability of locating more errors in a module is directly proportional to the number of errors already found in that module.

    At first glance, this may seem surprising. However, it has been shown that if certain modules or sections of code contain a high number of errors, subsequent testing will discover more errors in that particular section than in other sections.

    Consider a program that consists of two modules, A and B. If testing reveals five errors in module A and only one error in module B, module A will likely display more errors than module B in any subsequent tests.

    Why is this so? There is no definitive explanation, but it is probably due to the fact that the error prone module is inherently complex or was badly programmed. By identifying the most “bug prone” modules, the tester can concentrate efforts there (if this module has high business risk factor) and achieve a higher rate of error detection than if all portions of the system were given equal attention.


    [Bug Management Back to Top]

    8. Regression testing and reproducible test cases.

    Reproducible test cases become important later when a program is revised, due to the discovery of bugs or because the user requests new options. In such cases, the revised program can be put through the same extensive tests that were used for the original version. (Extensive testing of the system after modifications have been made is referred to as regression testing.) Without saved test cases, the temptation is strong to test only the logic handled by the modifications. This is unsatisfactory because changes, which fix one, problem often, create a host of other apparently unrelated problems elsewhere in the system. As considerable time and effort are spent in creating meaningful tests, tests which are not documented or cannot be duplicated should be avoided.


    [Bug Management Back to Top]

    9. Communication between software testers and developers, as well as developers’ view on bug fixing

    There is a different view between developers and testers on bug fixing. (Tips for Beginners)

    Sometimes bug fixing really only introduces additional bugs and does not even fix the existing problem. In a complex system, it is sometimes easier to rewrite all modules / classes than it is to fix the existing defect. There are some cases when it is impossible to fix the bug without rewriting the application. However, developers never like bug fixing and almost never confirm that it is their mistake in the code. Some fixes are very simple, like changing one number (e.g., if boundary conditions do not match the requirements.)

    Finding a bug is sometimes the same or requires even more of the art of thinking than does fixing that bug.

    The main philosophy is that the competence and attitude of individual testers and developers is the single most important component of a successful project. Sometimes bugs (defect, issue, problem, error) are valid but don’t really hold any importance from the user’s point of view and sometimes only enhance the application’s behaviour in comparison to the requirements. If coding was done by an experienced developer who understands the users’ needs well, he or she can suggest a better, or more user friendly, solution. However, it is still formally known as a bug and in this case the requirement document must be fixed and sometimes approved to match the real behaviour of the system.

    Sometimes one bug in a software application may cost millions of dollars for banks or other financial institutions and sometimes it can put human life in danger (e.g. if the software is used in the medical or aircraft industry.) When you define severity and priority to the bug you must take into account the influence of many different factors such as technical risk, business risk, and the frequency of use of the current function.

    As an example, for a common financial application, rounding is the main possible source of bugs.
    Many bugs can be called problematic, particularly when you are running a usability test. What is obvious navigation through the application from one person’s point of view may be difficult to understand for another person with a different mentality.
    Lets switch from theory to the real world testing and be prepared receive the following replies from developers after they review your bug description.

    Top 24 replies by programmers when their programs don't work :

    24. "It works fine on MY computer"
    23. "Who did you login as ?"
    22. "It's a feature"
    21. "It's WAD (Working As Designed)"
    20. "That's weird..."
    19. "It's never done that before."
    18. "It worked yesterday."
    17. "How is that possible?"
    16. "It must be a hardware problem."
    15. "What did you type in wrong to get it to crash?"
    14. "There is something funky in your data."
    13. "I haven't touched that module in weeks!"
    12. "You must have the wrong version."
    11. "It's just some unlucky coincidence."
    10. "I can't test everything!"
    9. "THIS can't be the source of THAT."
    8. "It works, but it's not been tested."
    7. "Somebody must have changed my code."
    6. "Did you check for a virus on your system?"
    5. "Even though it doesn't work, how does it feel?"
    4. "You can't use that version on your system."
    3. "Why do you want to do it that way?"
    2. "Where were you when the program blew up?"
    1. "I thought I fixed that."

    [Bug Management Back to Top]

    10. Murphy's Laws about bugs
    **********************
  • Any non-trivial program contains at least one bug
  • There's always one more bug.

  • Every program has at least one bug.

  • Programmers also know that every program has at least one line of unnecessary source code. By combining these two rules and using logical induction, it's a simple matter to prove that any program could be reduced to a single line of code with a bug.

  • Undetectable errors are infinite in variety, in contrast to detectable errors, which by definition are limited.

  • In any program, any error which can creep in will eventually do so.

  • Not until the program has been in production for at least six months will the most harmful error be discovered.


    [Bug Management Back to Top]

    11. If you decided to design you own bug tracking system you can use the following document as a draft of Requirements.
    Introduction

    Bug tracking system.

    The purpose of this document is to describe Requirements for the selection/creation of the Bug Tracking on-line system that will support basic requirements. Bug Tracking Procedure must support internal as well as possible remote locations.


    Bug Tracking System - Project Overview.

    Describe specific to your company requirements to bug tacking software..

    General Bug Tracking Process for the XXX program.

    All problems found during the testing will be logged in the XXX Bug tracking system, using a single database for all participating systems. Everyone who will be testing, fixing bugs, communicating with a clients, or managing teams doing either activity, will be given “write” access to the database.

    Several different kinds of reports and graphs can be produced in XXX using its standard versions, or using the included report writer to create custom reports. Either of these can be combined with XXX’s filtering capabilities to report on selection of data.

    During the testing activities all promotions to the test environment must be associated with a problem log, and agreed with a Testing Team. To avoid destabilizing the test environment unnecessarily, promotions may be scheduled to include several changes, except for problems classed as high priority (urgent), because they will hold up the testing.


    Updating & Routing Bug Logs
    When you route or update a bug log, you should always:
  • Annotate the log with information about what you did, and/or with the reason you are routing it to someone else
  • Change the status (substate) to reflect the log’s new status
  • Change the owner to the person you are routing the log to Please do not enter any of the statuses which belong to the state “closed”. These will be entered by the test team or test manager as appropriate. The restricted statuses are:
  • Void (use “pending” and the problem type “non-problem” if you think a log should be voided
  • Deferred (use “pending” and enter the reason you want to defer the problem in the log)
  • Closed (entered by the test team when a fixed and promoted problem has been retested and confirmed to be fixed) The Project Management will be review “pending” logs periodically, and decide whether they should be worked on, voided or deferred.

    User Requirements

    Requirement 1

    For the New Open Problem the selected bug tracking tool should support the login function with following information attached:

  • Problem ID (generated)
  • Problem Originator Name (editable)
  • Problem Submitter Name (selected)
  • Priority (selected)
  • Severity (selected)
  • Version where Problem was discovered (selected)
  • Problem Status (selected)
  • Problem Type
  • Problem Owner (selected)
  • Brief Problem description (editable)
  • Detail Problem Description (editable)
  • Attachment (browse)
  • Functional area where the problem was discovered (selected)
  • Platform where the problem was discovered (selected)
  • Type of testing where the problem was discovered
  • Repeatable or not. (selected)
  • Date problem originated (generated by the system)
  • Date problem was modified (generated by the system)

    Requirement 2
    For management activities the selected tool should support the following:
  • Specify the Estimate Fix time (editable)
  • Specify the Actual Fix time (editable)
  • Specify the in what Version of the software the fixed problem will be Promoted

    Requirement 3
    For corrective actions the selected tool should support the following:
  • The problem corrector name (selected)
  • Fixed on Platform (selected)
  • Detail of the Fix information (editable)

    Requirement 4
    For problem verification activities the selected tool should support the following:
  • Verified in version (selected)
  • Verified on platform (selected)

    Requirement 5
    The selected tool should support the “New Log” function:
  • It should have a possibility to call a new screen (clear all field), except the name of the Submitter
  • It should have a possibility the keep predefined data in any selected fields.

    Requirement 6
    The selected tool should support the Save function:
  • The last problem log save function
  • All changed problem logs save function

    Requirement 7
    The selected tool should support the Cancel function:
  • The last problem log Cancel function
  • All changed problem logs Cancel function

    Requirement 8
    The selected tool should have Viewing capabilities. The User should be able to select (filter) problems he/she wants to be displayed. The User should be able to select fields that will be in the Viewing screen.

    Requirement 9
    From the Requirement 8: The filtering capabilities should allow select multiple criteria. For example, list (1) Open Problems where (2) I am an Owner and from the (3) functional Area – Opportunity with (4) word in the Brief Description – Contact.

    Requirement 10
    The selected tool should allow to call selected Problem Log, using
  • Problem Log number
  • Double clicking on the Problem Log from the selected list of problem logs. See requirement 8

    Requirements 11
    The selected tool should allow to edit any field of the problem logs. It should allow saving and canceling changes.

    Requirement 12
    The selected tool should maintain the problem log history with the following information:
  • Originator and Submitter information
  • When the problem was discovered
  • List of the people who were involved in the problem resolution (?)

    Requirement 13
    The selected tool should allow handling multiple projects.

    Requirement 14
    The selected tool should allow working with remote groups. For example: testing in India and development in Canada.

    Requirement 15
    The selected tool should handle concurrent Users. Number of required concurrent users. TBD.

    Requirement 16
    The selected tool should have a reporting capability with filtering (see requirement 8). Report format should be editable (i.e. fields are selected).

    Requirement 17

    The selected tool should have a printing capability:
  • possibility to print selected problem
  • possibility to print selected problem list (requirements 8 and 9)
  • possibility to print a report (requirement 17)

    Requirement 18

    The selected tool should have a Login screen with User Name and User password. The User Name should automatically display in the Submitter field and must not be editable.

    Requirement 19
    The System admin should have a complete control over
  • Users access, by creating a New User, Deleting the User, assigning the User to a User Group (TBD – We should need it to assign the users to a different projects. The rest can be the same for everybody, except Administrative tasks), clearing the User password
  • Template creating, including possibility to create mandatory fields
  • Data in the selectable fields
  • Only Admin can delete a problem log.
    Requirement 20
    The selected tool should have a possibility to export the historical data after the project is done into Excel spreadsheet for archiving purpose. Note: if we will think longer we could need Importing function for sure.
    Requirement 21 Security requirements for the selected tool are the following:
  • All User groups should be allowed to Select, Update, Create and Delete
     - their own customized Reports
     - create and change their own View queries
     - password change window to create and change their passwords
     - new logged and fix information problem screen
     - set the default data
     - view a project they are assigned to
  • All Users group should be allowed to Select only
     - Predefined Reports
     - Predefined View queries
     - All projects
  • Management group should allowed to Select, Update, Create and Delete
     - Predefined Reports
     - Estimated fix time from requirement 2
     - Specify the in what Version of the software the fixed problem will be Promoted

    Technical Requirements
    Requirement 22
    The selected bug tracking tool must have data resided in unlocked non proprietary database.

    Requirement 23
    The selected bug tracking tool has be compatible to Windows and Linux.

    Requirement 24
    The selected bug tracking tool should ideally be Microsoft Exchange based.

    [Bug Management Back to Top]



    Find more on bug tracking tools

    On this page I put some information about Software bug life cycle and defect management for QA and testers.



    Software Testing Main Page
    © January 2006 Alex Samurin http://www.geocities.com/xtremetesting/