Heya folks, did you get your computers out of that blue screen? 😉 (Psst…I wanted to emphasize the importance of testing!)
As they say, “Life happens when you are busy making plans!”
Here I was, meticulously planning to address some deep challenges in the ‘most spoken about’ (er…’hyped about’) LLM security, and bang, comes in the news that businesses and airlines are facing ‘technical difficulties’ worldwide because of a bug in Windows machines.
The deeper analysis by many awesome folks, some from the kernel communities (both working as well as retired), made the picture clear to all of us on why 8.5 million Windows computers around the world have blue screens.
It was impressed upon me (and I don’t know to what extent this is a fact), that the issue was to do with CrowdStrike releasing a device driver update with a sys config file that had a p-code, where the function that was called was NULL. Sigh.
Several questions were raised by the community, and the most interesting ones that would concern to a testing or a quality person are the following:
- Why was the p-code NULL?
- Why didn’t the device driver check the arguments passed to it? If it had, it should have detected the NULL pointer for the function. How did Microsoft certify a device driver that didn’t check the arguments?
- Why was no testing done on the update when it was as simple as using a spare debugging machine to check if the upgrade worked as expected?
Testers, do you see the importance of testing in this picture? It does not matter if a developer ‘wrote’ the tests or if a tester did the testing by hand, the issue would have been detected and rectified IF it had been done. BUT, it wasn’t.
There are several things that can be / could have been done to prevent this from happening, some of which excluding testing are:
- Given the speed in which updates are released in device drivers, should Microsoft increase the quickness with which the device drivers are certified?
- Should Microsoft make its certification process more versatile to make sure that the device driver is doing the thing that it is expected to do?
- Shouldn’t Microsoft subject sys files, p-code, configuration files, etc. to certification process, and not just the device driver code?
Those who say ‘Test is dead’, ‘Testing is dead’, or ‘testing is dud’ are again and again woken up from their deep ‘Agile’ slumber that it is none of those. The relevance of testing is more and more emphasized.
Where are we going with LLMs if we don’t address the basic software quality issues?
Reach out to me if your organisation needs a quality / testing process review. Glad to help!