Tracy Bannon brought up an interesting point in LinkedIn while highlighting the importance of human oversight when building software with the help of GenAI. Let’s unpack this aspect a bit in this article when we talk about humans in SDLC.
I have no qualms about using GenAI where it is applicable. As someone mentioned, it is an extension of auto-fill functionality for a large amount of data. The main functionality of GenAI is to generate based on the data corpus it has. It is supposed to generate text, data, or code, based on the rules and constraints given to it in the form of documents, programming language grammar, or a data corpus with restrictions. But we have all seen that GenAI does not strictly comply with those rules and generates wrong data, code, and factually incorrect text. Some blame this to inaccurate data corpus, but it could not be always the case because even in specific domain scenarios, this is seen.
I added a comment to Tracy’s post saying that human oversight is indeed needed throughout the software development life cycle. More importantly, just because you could generate unit, integration, and system tests, or even write them by humans, it does not mean human testing can be avoided in all these cases. Add solution testing to that list, which is systems of systems, and the importance of human testing goes up more.
Again, I am not saying we should not use automation or GenAI. All I am saying is that the automation has limitations and they do what they have been told to and cannot think and reason based on context. Even if you argue that you can feed context to an AI so that it can do the test generation for you based on context, effectively feeding context to AI depends on many factors like ability to generate effective prompts, data completeness, model versatility, etc. One cannot be hundred percent sure that an AI can generate tests or recommend tests based on the context that it has. Human oversight and testing is of course needed.
If we go back to our intention of introducing computers and automation, we did that to relieving humans from mundane, repetitive tasks. We also argued that humans are error-prone and can make mistakes in computation, verification, and checking because of fatigue. But automation proved to be limited because it can implement and check only a certain portion of the vast scope of possibilities. Non-deterministic things like GenAI, and probability-driven machine learning algorithms add complexity to the scene instead of making the computation easier because of the way they work. Again, humans’ involvement becomes necessary. This defeats the whole purpose of delegating things to machines, because humans need to check in every step of the way in the software development life cycle.
The key here is finding that fine balance. It needs deep understanding of GenAI systems work. The learning curve for that is steep, and also, the algorithms are evolving at a rapid case for anyone to keep up. Unfortunately, the metrics that are thrown out as measuring effectiveness of new models are not very detailed about how those measurements are made, and on what basis. So, for someone to take a model and make it as the basis of the functionality that they want to build is a challenge, because the metrics and their measurements are hearsay.
I remember the times when machine learning was introduced afresh and people were confused about to what extent they should learn and implement machine learning in their software systems. Personally I had one rule that I followed – keep it simple and use it for very easy, mundane tasks, whose results you can cross-check and verify. If you need to build a system whose accuracy is very important with business rules, go for RPA and not machine learning. I guess the same applies for GenAI too. Even with RAGs and agents, the accuracy might suffer, and we have to consider the efficiency, sustainability, and cost aspects too.
If you are looking for inputs for assessing if you need to have GenAI-based solutions in your organisation and the quality implications of doing so, please feel free to reach out to me. Glad to help!