I am a fan of cutting down the “hand-offs” and “silos”, so I love the concept of DevOps. DevOps implementation performance survey is being done as DORA State Of DevOps yearly reports. I go through each year’s report and see how the organisational and operational performances are being surveyed and presented to the development community. Also, being interested from Software Testing and Software Quality perspective, I go through those yearly reports through that prism too. This would be the first blog in the series of blogs which look at DORA Report 2022.
DORA reports have been coming out for eight years now, and it is claimed that around 33,000 practitioners contribute to the survey. It is certainly a very extensive and painstaking work. At the same time, it is necessary for the community to take a critical look at what is being measured and whether they make sense. I will cover Software Supply Chain Security and Cloud adoption in this blog.
Software Supply Chain Security
The mention of software supply chain security, SLSA, etc. is appreciated. A couple of months BEFORE log4j vulnerability happened, I happened to take Linux Foundation’s certification on Software Supply Chain, and I was happy that I took it. I learnt the basics, and I am continuing to educate myself more on that. Software Supply Chain security is so important, and I’m glad that 2022 DORA report thought of mentioning it. I also liked the mention about the potential correlation between adoption of security practices and public cloud usage – the reason probably being cloud service providers insist on following supply chain security practices. Not sure about what extent though. It would be worthwhile to point to some data related to that. At the same time, I’m skeptical about the mention of ‘automating builds and deployments’ being the direct reason for improved security. It should rather be ‘including security checks in the integration and pre-deployment automation’.
I would also like to touch upon the mention of NIST as the reference for non-technical aspects and attitudes of software supply chain security. While it is great that a standard from an important country liked United States is taken for reference, it would make the analysis and the study stronger to include standard bodies from other countries and regions (EU, India, etc.). There’s a mention about teams not following some Software Supply Chain security practices increasing the “odds” of “higher burnout” by 1.4x. Statements like these without proper reasoning and supporting data highlighted in the relevant sections will decrease the credibility of DORA Report.
Cloud and Organisational Performance
There’s a section on cloud usage and its correlation to organisational performance. I am skeptical (in fact, I would disagree) about the assertion that usage of cloud increases organisational performance. I am also very skeptical using multiple clouds is 1.4x likely to have above average organisational performance. These kind of assertions should be qualified with additional comments and notes from the people who are surveyed about why it is so, showing the reasoning. Otherwise, these kind of ‘correlations’ will water down the authenticity of an important survey like DORA metrics.
That’s to start with the review of Software Supply Chain security and cloud. I will elaborate on this later as required as I go through the rest of the report. I will also write about other sections and considerations in the next few blogs.
If you would like to chat with me about my thoughts on 2022 DORA Metrics report, please feel free to contact me.