I have yet to see a process where bug regression is also part of bug verification. It is not uncommon to sometimes see that new bugs are created as a result of a bug fix. Sometimes the change in the code inadvertently causes the existing functionality to break. As we have found a few cases where the existing functionality was no longer working, we felt the need to do bug regression. I will discuss more about bug regression in the following paragraphs, but I will limit the discussion specific to our process.
Test & Fix
In order to keep this discussion brief and simple, I will not go into the details of the current development environment, but will simply state that it is a Quasi- Agile environment. The application is built by working on multiple components at the same time over a series of repetitive processes. There is fairly good documentation of the user stories and design specs, but they do change throughout the course of development. As code is written, reviewed and unit tested by the developers, testers are creating and reviewing test scenarios, writing and reviewing (time permitting) test cases. In the Test & Fix phase, testing will primarily focus on the positive test scenarios (happy path). Here is an example,
The subsequent test scenarios will probably not be tested from the metric for simple reasons such as, the code for the Program Editor may not be in yet or, we don’t know the rules of interactions between the DC console and the Program Editor or the DC console and the change of page layouts. And so, it makes sense to have a shorter test cycle for Test & Fix. The next phase is ready to begin, as all the different components are built and integrated.
Integration
Since the bulk of the changes are out of the way, there is almost a static target to take an aim at. It is very difficult to shoot down a moving target when user stories are continuously defined and redefined and the code is constantly changing to reflect the user stories. In some cases, new user stories are added and some existing ones are removed. At this phase of testing expectations are such that changes are mostly done, but it is not always the case. The hope is that there will not be new user stories and very few new definitions of any existing one at this point. De-scoping of a user story is fine as it doesn’t require any new work and rework from our part. As the target is fairly immobile at this stage, we are now poised to attack. For the same user story (DC4.1.4.1.1), the following test scenarios will be tested.
DC console will not disappear when:
- Page layouts are changed
- Applications are added and deleted within each page layout
- A program or function is opened
- A program or function is viewed
- A program or function is edited
- A program or function is being executed
- Pages are dragged and dropped in tree view
- Pages are copied and pasted in tree view
- Pages are copied and pasted in page sorter view
DC console will disappear when:
- A new problem is added
- Existing problem is deleted
Since a significantly large number of test cases are executed during integration, this is perhaps the longest of all test cycles. As a result, a large number of fairly complex and difficult bugs are generated during this test cycle. As integration nears the end, the bug reporting frenzy flattens out gradually. The fact that fewer and fewer bugs are found near the end of the test cycle tells me a couple of things. Either most of the bugs have already been found or, the testers have simply run out of ways to find more bugs. The thought of running out of ideas for finding more bugs worries me the most. It is often a tester’s desire to improve on the depth and breadth of test coverage. But, which way does one go? Right or left to add the breadth or up or down to add the depth? How far left or right and up or down does a tester need to go? There is not an easy answer to these very difficult questions. However, he answer to some of these questions just might be in bug regression. Here is how it works.
Bug Regression
Once a new bug is logged and triaged, to make sure that it is indeed a bug, a test case is written (if we don’t have one already) to address the bug. But more needs to be done such as:
1) Regress Around the Bug:
A set of test scenarios, and a number of test cases based on those scenarios, needs to be created around the bug. So, next time when the bug is verified, the test cases associated with the bug will also be executed. Let’s take this bug (a true bug) for instance.
Bug Summary: DC console disappears when opening a program or function in the Program Editor.
Test Scenario: DC console will not disappear when a program or function is opened.
In this case, there is already a test case based on the test scenarios from the metric. As a matter of fact, the bug is found by executing the very test case that illustrates the test scenario. The cause of the bug is known from the bug tracking tool. It turns out that, the function GoToWidgetOnPage() does not handle the moving console between pages very well. The solution to this bug was to add code to this function to better handle the moving console between pages. In order to regress this bug, test cases need to run with the following test scenarios (see Test Coverage metric).
DC console will not disappear when:
- Page layouts are changed
- Applications are added and deleted within each page layout
- Applications are swapped within each page layout
- A program or function is opened
- A program or function is viewed
- A program or function is edited
- A program or function is being executed
- Pages are dragged and dropped in tree view
- Pages are copied and pasted in tree view
- Pages are copied and pasted in page sorter view
Since test cases already exist, there is not a need to create any new test scenarios or write new test cases. The current test case provides good coverage for this bug.
2) Follow the Code:
Since changes are made to function GoToWidgetOnPage(), it is important to know if the same function is being used in any other part of the application. The premise of going after other areas of the application has to do with the notion that they are guilty by association. This information has to come from the developers and will be tracked through a bug tracking tool. Bug tracking tools can be configured such that the developers can document the affected areas (GUI and functional) in the tool. It can be a simple text field where the developers will list all the areas affected by the change. Once this information is obtained a tester will either use the existing test cases, (if there are any), create new ones or do both to test the areas that use this function. As a result, next time this bug is verified as part of the regression, a tester will also test the parts of the application guilty of using the same code thus making sure that fixing this bug does not cause any unpleasant occurrences in any other areas of the application. It is possible that some bug fixes are strictly isolated and have no affect on any other areas. But for the bugs with shared code (most are, since codes are reusable), this approach of bug regression will significantly increase the depth and width of our test coverage in all directions.
Exploratory: The exploratory sessions usually start after the integration. This form of unscripted, unstructured and anything goes type of testing is actually very refreshing. The only scripted part is a small document with a few lines of text which explains the areas of the application which are to be tested. Exploratory sessions are like Easter egg hunting; a race to find as many bugs as possible with as many participants as possible. Sometimes very strange bugs come out of these exploratory sessions. Here is a true bug:
Bug Summary: Cursor position changes
Bug description:
- Open the application n Create a program in Program Editor
- Enter a line of text
- Go to the end of the line by pressing the right arrow on the keyboard
- Do a mouse click on the current cursor position
Result: Cursor jumps to a different position.
This is probably a very minor issue and chances are the user will always use the mouse instead of the keyboard to change the cursor position. Since this is a bug, we may want to create a couple of more test scenarios around it such as:
- Going to the beginning of the line by pressing the left arrow on the keyboard
- Going to the middle of the line by pressing the right arrow on the keyboard
It is comforting to know that we usually don’t find too many show stoppers from the exploratory sessions. At this stage of the test cycle we don’t anticipate any critical issues but, we do find one or two every now and then.
In Conclusion
It is important to point out some of the difficulties of practicing bug regression. They are mainly two-fold:
- “Follow the code” part of bug regression requires configuring the bug tracking tool. It also requires participation from development and thus needs a coherent effort from both testers and developers for putting this in place.
- Adding more on to a tester’s plate can also be a tough sell.
Fortunately, we were able to see some reduction on bug-recurrences by regressing around the bug. Over time when all the pieces are in place, it will benefit both QA and development and the outcome will be a better product.
About the Author
Mohammad Alam I started as a mainframe programmer and gradually migrated to c/s environment in mid 90’s with the intention to become a software developer. In order to make a smooth transition from mainframe programming to software development, I decided to start as a Software QA Engineer and thought I would work my way to be a developer. Because of my programming experience, I was asked to be a part of test automation which gave me an opportunity to write code. As I learned more about the world of test automation and QA, I realized that this is what I want to do and I have been doing software QA ever since in various capacities, such as tester, test lead, automation engineer, automation lead and coordinator for offshore test teams. Occasionally, I do share my experience (some are more painful than others) with the QA community hoping that others may benefit from it. But ,on a serious note, I would love to travel around the world and taste the local delicacies and if I could get paid for doing this (like that dude on Food Network), it would be awesome!