Regression Testing Guidelines- Without Automation

1.    Introduction

Regression testing is full or partial selection of already executed test cases which are re-executed to ensure existing functionalities work fine.

1.1             Purpose:

The purpose of regression testing is to confirm that a recent program or code change has not adversely affected existing features. This testing is done to make sure that new code changes should not have side effects on the existing functionalities.

1.2             Scope:

The scope of testing is based on the how large fixes or enhancements of features. The scope of regression testing increases when the bug fixes have impact on larger area of the application. To analyze this tester should get the correct inputs from developer to decide the scope.

1.3             References:

Most of the references can be taken from the Test case document, which are failed and fixed in new code design.

1.4             Goals:

The goals of regression testing are:

·         To check the defect fixes of the previous build work fine and also to ensure that the defect fixes or enhancements have not caused any unintended defects in same or other modules of the build in test.

·         Help determine the extent to which the system is ready for launch.

·         Report these defects to development teams so that the associated defects can be fixed.

·         Provide input to the defect trend analysis effort.

2.    Regression Testing

Regression testing is a verification method & mainly done in system testing but it applicable to all levels of software testing like Unit testing, Integration testing, System testing and Acceptance testing. Good Regression tester need to aware of what all features to be focus on changes in the software application.

Regression Testing is required when there is a

• Change in requirements and code is modified according to the requirement
• New feature is added to the software
• Defect fixing

2.1 Types of Regression Testing:

·         Corrective Regression Testing: Corrective regression testing applies when specifications are unmodified and test cases can be reused.

·         Progressive Regression Testing: Progressive regression testing applies when specifications are modified and new test cases must be designed.

·         The Retest-All Strategy: The retest-all strategy reuses all tests, but this strategy may waste time and resources due to execution of unnecessary tests. When the change to a system is minor, this strategy would be rather wasteful.

·         The Selective Strategy: The selective strategy uses a subset of the existing test cases to reduce the retesting cost. In this strategy, a test unit must be rerun if and only if any of the program entities, e.g. functions, variables etc., it covers have been changed. The challenge is to identify the dependencies between a test case and the program entities it covers.

·         Defect Regression: Retest a specific defect that has been fixed.

·         Localization testing: The program has been modified to present its user interface in different languages and/or following different set of cultural rules.

·         Conversion or port testing: The program has ported to a new platform and a subset of regression test suite is run to determine whether the port was successful. Here the main and important topics to cover will be the new platform, rather than the code changes.

·         Configuration testing: This testing is like port testing except that the underlying code hasn’t been changed, only the external components that the software under test must interact with.

Regression testing helps in

·         Increasing chances of detection of bugs in passed scenarios.

·         Effectively planning regression cycles to ensure critical elements are always tested.

·         Regression testing also detects undesirable side caused always by changing the operating environment. 

2.2 Identifying the type of regression test for the build:

There are two main ways to identify the types that are processed:

·        Full and Final Regression testing:

The above test is conducted when the build contains only defect fixes and no enhancements. The product must be continuously exercised over a defined period of duration. Some of the test cases are subjected to repeat tests to find out if there are failures in the final product that will reach the client.

Exploratory testing and valid possible test scenarios can be tested in this defined period of duration to make sure the build is perfect without any high severity defects. All the defect fixes for the release are required to be made available in the build used for final regression test cycle.

The final regression test cycle is more critical than any other type or phase of testing which ensures that the same build reaches to the client which was tested.

·        Partial Regression testing:

The above test is conducted to verify the defect fixes in the interim builds. However unchanged build is highly recommended for each cycle of regression testing.

3.    Factors and Strategies to include

·         Test fixed bugs promptly. The programmer might have handled the symptoms but not have gotten to the underlying cause.

·         Watch for side effects of fixes. The bug itself might be fixed but the fix might create other bugs.

·         Write a regression test for each bug fixed.

·         If two or more tests are similar, determine which is less effective and get rid of it.

·         Identify tests that the program consistently passes and archive them.

·         Focus on functional issues, not those related to design.

·         Trace the effects of the changes on program memory.

3.1             Pre- Conditions to start:

·         The test team is adequately trained and staffed in functional and regression testing.

·         The relevant software components have passed unit testing and one cycle of functional testing.

·         The defects for the earlier builds have been posted and acted up on.

·         The test environment is ready.

·         The regression test suite as identified during planning or test case authoring phase is ready.

·         The defect reporting procedure and communication protocol is ready.

4.     Process flow

4.1 Selection of Test cases

In many cases last min bug reporting and fixing the bug due to technical or functional issues will lead to difficulty in regression testing process. Estimating side effects for small bug fixes will be crucial and time based analysis.

Effective Regression Tests can be done by selecting following test cases:

·         Test cases which have frequent defects.

·         Functionalities which are more visible to the users

·         Test cases which verify core features of the product

·         Test cases of Functionalities which has undergone more and recent changes

·         All Complex Test Cases

·         Sample of Successful test cases

·         Sample of Failure test cases

·         Requires knowledge on defect fixes and how it affects the system

Selection of test cases depends predominantly on the nature of change and its impact. It is also recommended that the regular test cycles before regression testing should have the right mix of both failed and passed test cases.

Failed or negative test cases are those which are introduced with intent to break the system.

4.2 Prioritizing Test cases

Based on the defect fixes, which are affecting the major functionalities of the application, test cases can be differentiated related to the bug fix.

Prioritize the test cases depending on business impact, critical & frequently used functionalities. Selection of test cases based on priority will greatly reduce the regression test suite.

The prioritization can be categorized as below:

            Priority 0: Sanity test cases which checks basic functionality and run for pre-system acceptance and when the product goes to the major change in terms of major functionality included or a critical defect fix provided. These test cases will deliver a very high value to the project as well as client.

Priority 1: These test cases which check for major defect fixes which will deliver high value to the project and client.

Priority 2: These test cases will deliver moderate value to the project. These are executed as a part of System Test cycle and selected for regression testing on need basis.

4.3 Test suite preparation

Prioritize the test cases depending on business impact, critical & frequently used functionalities. Selection of test cases based on priority will greatly reduce the regression test suite.

Executing the assorted test suite with the related test cases will affect the testing procedure and time to identify the effect of bugs which are being fixed in the application.

5.    Major Risks in Regression testing:

Identifying software changes and its impacts in a systematic way is really a technical approach to execute regression testing process.

5.1 Identifying software changes:

Software changes related to Requirement specification changes or Environment related changes can be easily located and can be tested.

A technical approach to follow in identifying software changes:

·         Requirement specification changes.

·         Design specification changes.

·         User manual changes.

·         Environment or technology changes.

5.2 Identifying impacts of software changes:

Identifying impacts based on the changes done in related section will make the process easy to test the application.

6.    Concluding the results of regression testing

Regression testing uses only one build for testing (if not, it is strongly recommended). It is expected that all 100% of those test cases pass using the same build. In situations where the pass % is not 100, the test manager can look at the previous results of the test case to conclude the expected result;

Ø  If the result of a particular test case was PASSING using the previous builds and FAIL in the current build, then regression failed. We need to get a new build and start the testing from scratch after resetting the test cases.

Ø  If the result of a particular test case was a FAIL using the previous builds and a PASS in the current build, then it is easy to assume the bug fixes worked.

Ø  If the result of a particular test case was a FAIL using the previous builds and a FAIL in the current build and if there are no bug fixes for this particular test case, it may mean that the result of this test case shouldn't be considered for the pass %. This may also mean that such test cases shouldn't be selected for regression.

6.1          Completion Criteria:

Regression testing is complete when the following post conditions hold:

·         All the defects have been fixed and have been checked and found that they have not created any further defects in same or other modules.

·         All the test cases of regression test have passed.

·         Defects from the failed test cases, from the regression test have been properly posted to the client.


Note: Automation tools and automation regression tests will be available soon.