A while back, I wrote about how automation checks cannot validate automation checks, or at least why that is not 100% reliable and places a burden on the Software Tester who juggles between the mutation test code, automation test code, and product source code. In this blog, let us look at the perspective of whether using a mutation testing tool would work.
When I wrote about mutation test code last time, it implicitly meant mutation tools also. Mutation test tools look cool, but they are also developed by humans and are susceptible to defects.
Let’s take an example of a statement where you want to check if the value of variable C is higher than the product of two other variables A and B [ if (C > A * B) ]. The mutation test tool would replace the operators ‘>’ and ‘*’ with other operators in the product source code to check if the automation tests detect those incorrect scenarios because the automation checks would have been written specifically for ‘>’ and ‘*’.
In this case, we trust that the mutation test tool would do the right thing and not have ‘>’ and ‘*’ and something else. But this is not necessarily completely reliable. There could be a defect in the mutation testing tool itself which might make it check against ‘>’ and ‘*’ themselves, in which case the test would pass, but that is not the intent of the mutation test.
Any software written by humans is prone to defects, and the chances of defects happening multiply if automated checks are used to test automation checks. You will ask then what is the way out and how are we going to check any code written through automation – be it source code or automation code. The trick is to have the checks closest to the source. TDD is a natural choice as the code is written test-first, provided, the human writing the source code by employing TDD does it in the correct fashion and does not introduce defects! Here again, the checks done could be incorrect! But TDD does take away the need for mutation testing and its associated complexities of handling three layers of code.
So, as you can see, even if we use TDD, the software is prone to defects, and hence, there is no way out other than to test after the product has been developed. Here again, the quality of testing depends on the software testing professional involved and various other factors. A reasonable balance seems to be TDD plus post-code testing (not just automation).
Hope that gives some clarity on mutation testing tool. Feel free to chat with me on Software Testing strategies.