So, a bunch of software developers discussed about getting the software development process to be Lean. One of the things that was glaringly slowing down their pipeline (as per them) and their hurry to take the software development to the next level was Human Regression Testing. They wanted no interruptions by humans for the Regression Testing in Pipeline.
So they decided to divide and conquer. They divided regression testing into “Manual Regression Testing” and “Automated Regression Testing”. They glorified “Automated Regression Testing” as a best practice, and vilified regression testing done by humans as error-prone, repetitive, time-consuming, and against the Lean and Agile Principles. They still do. Organisations and individuals who don’t understand the depth of Software Testing, organisations that would like to make a quick release (especially startups which don’t want to invest in Software Testing but rely on their end-users to do the testing for them), and organisations who wanted to reduce testing workforce to improve margins bought that story and went ahead and fired testers or not hired them at the first place. In addition to that, there was always this fancy thing of “100% automated tests” which is a dream of many test managers to show process efficiencies and get their next promotion.
Now let’s look at how to do regression testing. Yesterday, I wrote about what regression testing is – its meaning and purpose. If you want to thoroughly test an area that has a regression fix, you cannot simply run a test suite of automated checks to show them as passed and say your fix worked. In fact, it’s the same with scripted checks done by humans by hand. Checkmarks are not sufficient.
“Isn’t that why we have ‘exploratory testing’?”, you may ask.
Well, the whole horror story is because of boxing and seperating out exploration and automation!
Here’s a method I would recommend for regression testing:
- Start with human(s) testing the fix
- Explore, experience, and experiment (as some Testing leaders call it) to see if the fix not only addresses the regression defect for which it was written but also making sure that the area as well as the other areas are not affected (in an unwanted fashion) because of the fix
- Automation? When you test, use automated checks (along with data preparation, scenario and infrastructure setup, etc.) as diligently as required. Automated checks are okay, but they should be interspersed with human effort orchestrating the flow of the test. There should never be a goal of 100% automated regression test. Even if you achieve that, it’s the human(s) who will oversee the process and test where the checks are not covering the areas thoroughly or where more exploration is needed
Anyone who advocates bringing down the importance of Software Testing post-development is incorrect. Your automated checks are not a fool-proof shield against defects getting to production, and we cannot assure anyone that the product is bug-free because we got all the checks automated. In fact, we have a few test colleagues who pride themselves with maintaining millions of lines of code of test automation. Good luck to them.
Well, what about complex scenarios where there are thousands of scripts running for regression testing in pipeline and we cannot possibly have human intervention in the middle of the pipeline? Well, again, good luck to you. But don’t forget that human oversight and orchestration is a must. If you blindly run the scripts, you are doomed to failure at some point of time. I would be extra careful as such companies with big efforts affect the lives of millions of people.
To summarise, have a unified human-machine concerted testing effort for a regression fix. Don’t believe anyone who talks about the evils of “manual regression testing”. They are not in your best interests of having superior product quality.
Feel free to get in touch to discuss regression testing in pipeline!