These days, there’s a lot of talk about Software Testing in production, observability, test automation, continuous delivery, etc, attributing them to something innovative. There was one article that I recently read in Medium which was one such post.
When I read this, I was reminded of what we used to do before the word Agile came into being:
– We automated tests, and we figured automation is diminishing-return stuff
– We tried to reproduce production environments with our own labs – and we struggled
– We tried to observe behavior of software in production – at customer premises
Those are the key things that are mentioned in that article, except of course, the continuous delivery part, which is about committing changes as and when required instead of waiting for 6 months or an year to get customer feedback on a version and then releasing the next version with the changes.
We did automate tests and we came to the same conclusion that adding more automation or having 100% automation is not going to solve the issue.
The second point above about reproducing production environments reminded me of the struggles I personally had with trying to simulate a real production scenario of huge number of cable modems connected to a big router (some 100,000 devices). The cable modems individually flip-flop, go into error states, firmware in them being upgrade, and transition between different service levels. We always struggled with this problem even if we had software simulation systems (as we could not procure/have so many number of devices and make them do things randomly!) because software simulation systems have their devices do things in batch and not individually.
We also observed in production sitting in the customer premises and debugging and resolving issues. The only thing that probably would change is the cloud deployment which might not require visits to customer premises, but for physical access to hardware devices, even that might be required some times (not always).
I wonder what’s new about all this? I have heard Agile leaders talk about feedback being a main component of Agile, and that’s missing in that article. If I don’t take the learnings from the production environment and incorporate those in my next revision of software, am I being Agile?
I figure we need to move on, and start working more on how to:
– spend less time on developing and maintaining automation scripts
– make Software Testing done in pre-production more valuable to the customers (just the fact that we can commit as and when we like does not mean the Quality of Software can be taken for granted)
– not gamify proper Software Testing as ‘exploratory testing’ and reduce its importance
I think these will be valuable for the quality of software. We don’t want any plane crashes or mass database outages that affect millions of people and then trying to triage in those situations and call it as ‘testing in production’!
