Automation Testing: Are you doing the right thing?

Test Automation, as it is popularly called, is extensively used to replace the human efforts because of its showcased benefits – quicker, faster, error-free, no-fatigue, and available 24×7. Testing of products under several industries today use automation for testing, and there are several tools competing to provide the benefits of automation testing to the businesses. While this is great, we need to look deeply into the ways it is getting deployed and whether those ways are in the best spirits of quality. We need to indeed cut the hype and get to the core of quality so that the several issues that we face with automation testing today can be addressed and the industry can move on with better implementations.

There are several incorrect ways people look at automation for testing. Here is a list. Text in the braces are my thoughts.

• I want all my test cases to fail, because testing is supposed to expose all the defects! I like Fail test cases than the Pass test cases, because Fail test cases tell me more about the system under test (are your test cases comprehensive?)
• Pass/Fail (assertion) is a great way to assess quality of a software system (how many of the combinations have you written the ‘tests’ for?)
• You absolutely want 100% pass rate for a releasable product (does the 100% pass mean your product is of superior quality?)
• Failing tests convey a lot more than just deviation from expected functionality. (True! But how many of us are willing to explore more about the context when a test fails? We quickly fix the failure and make the test pass!)
• A failing test gives me the confidence that the automated framework is finding bugs. I would be more worried if it never failed! (is your objective only to find bugs?)

… and so on.

Do you see what is worrying about the above? The focus of automation testing is on the ‘Pass/Fail’. Pass/Fail unfortunately has become the only indicator for product quality. It is even more terrifying when the quality is assessed by the ‘no. of pass percentage’ irrespective of what those tests do!

Testing tools have been built for so long now based on the philosophy of pass/fail. Test tools are marketed to be a replacement of human testing and let the product be tested hands-free without human intervention and generate a report of pass/fail. Fair enough, the failures can be analyzed and see where the system is broken, but how much do the passing tests tell you about the system?

The problem is in making automation CONCLUDE about the pass/fail.

When you think about testing a system, you should think like a person entering a room and observing it. A diligent person would take note of all that is in the room in utmost detail. Your automation should do the same thing. It should, by the way of its parameters, take note of the state or the context in which the system is in, so that the human who is doing the test can learn and make conclusions about the state of the system, and orient further tests that need to be done on the system. Your automation should not make those conclusions!

Venkat Ramakrishnan, 24th June 2023.

I was telling someone on Twitter that if the focus of the automation changes from “Pass/Fail” to taking note of the context/state of the system, the world would be a much better place!

It’s time we move away from the tentacles of automated “Pass/Fail” and look at product quality from the eyes of context. Let’s hope it happens.

For detailed consultancy on Software Testing and Quality for your organisation, feel free to reach out to me.