Whenever there’s a query raised about assessing the quality of automation code, the immediate answer that comes up these days is “Mutation testing!” So I thought, “Wow! This should be a great out-of-the-box approach that’s gonna look into the scripts somehow and find out if the automation scripts are faulty. Interesting!” and then I started reading a blog off the net to understand what it is.
Here’s the essence of what I found out. It’s purposefully making a mistake in the code to check if the automation check detects it. If it does not, then the check is considered ineffective. Essentially, yet another check to check if the check worked.
I could imagine at least one point in time where my head would spin trying to figure out if the issue is with the code or the automation, in this scheme. Yes, I know that the valuable mutation code is going to be stored in a very separate repository, not to be mixed up with the actual code, but as an engineer working on these three sets of code looking at them back and forth to verify what’s going right and what’s going wrong is a lot of work. Oh yes, I agree that some shell scripts are going to narrow down the checks that are not effective, but it would still be a human effort to go through all these failures and fix them. One source tells me that they found thousands of mutants (as the automation check failures are popularly known as) in a couple of files. Go figure.
Figuratively, to me, this scheme looks like a movie climax where the villain and the hero point the guns at each others’ forehead and watch for who is gonna blink first. Let’s all hold our breaths to figure out if the problem is in the code, or in the automation check, or Nature forbid, the mutation test itself!
So, who is going to test the tests that test the tests that test the tests? Short answer: I would let an SDET do it.
If you are hosting a Software Testing conference on Mutation Testing, maybe I’ll buy a ticket.
Happy Testing! Feel free to chat with me.