Shifting the Audit: A New DevOps Direction for Engineers and Auditors
Software engineering teams are accelerating application development and delivery with DevOps, but others are being left behind. A new “DevOps love letter” written specifically to auditors says it’s time to change.
“With all this growth, we made a mistake, we forgot to bring you along for the ride. That is totally our bad, but we want to make it right,” key members of the DevOps community confess in the letter. An associated DevOps Risk and Controls Matrix outlines risks engineering teams should plan to manage by performing specific activities that fulfill specific audit controls.
It’s a giant first step for engineers in redressing the oversight of excluding auditors from DevOps, a philosophy of which inclusion is an underpinning principle. But the void between these groups isn’t new, says Scott Prugh, who is Chief Architect and VP of Software Development at CSG International and a DevOps thought leader.
“In my experience, there isn’t a relationship between engineering and auditors. The relationship usually comes when something goes wrong,” Prugh said. “Audit is an afterthought and it is additionally very mysterious. These folks report to the CEO and only appear when something goes bad, so of course they’re associated with bad things.”
That is, intimidating and tedious investigations into Engineering’s compliance with regulations. This is an important business concern, though, ensuring compliance can be sufficiently and efficiently met. With an expanding checklist of regulations to abide by—ISO, SOC, GDPR, HIPPA, PCI and so forth—organizations can’t risk bungling compliance for unhealthy relationships between auditors and engineers.
Shifting the Audit
To solve this, Prugh says auditors must shift left and educate engineering teams on why controls are important for risk mitigation. Engineers must come to understand controls and build systems designed to support them.
The DevOps Risk and Controls Matrix should help guide organizations towards this higher order of collaboration between auditors and engineers, away from the traditional processes they’ve relied upon according to Sam Guckenheimer, Product Owner of Visual Studio Team Services at Microsoft and a principle author of the letter to auditors.
“Our goal is to move from a world where audit compliance is a huge tax to one where it is free,” Guckenheimer said. “The traditional thing is that you have this big ceremony with internal audit folks to prep for when a particular regulatory auditor comes. Then the auditor comes, tries to do sufficient statistical sampling of cases to find evidence of something bad or that there’s nothing bad to be found.”
Guckenheimer says this is like substituting a newspaper poll for an election when we’re fully capable of counting the votes—we don’t need to statistically sample for indefinite results. Relying upon change control boards to approve thousands, tens of thousands, of lines of code they are unfamiliar with is an issue.
The change of pace in business is another issue. Ten years ago, auditors managed far fewer deployment events on a quarterly basis. Now, they must manage several deployments a day.
“To apply what worked before to a case where you have thousands of deployments just doesn’t work for auditors. It’s unreliable and it is extremely painful for everyone,” Guckenheimer said.
“Today, we can literally get a view of all the changes and actions that have happened over the last year. What you need to do is make sure the process and pipeline through which those changes flow provide sufficient security and satisfy compliance objectives.”
Prugh adds that this traditional big-batch compliance theater is disingenuous to what compliance is really going to do.
“If you only do it once a year, there are 364 other days you aren’t doing what you’re supposed to be doing,” he said.
Instead, Guckenheimer says, organizations should follow an iterative DevOps approach to auditing, completing code reviews of five, maybe ten lines of changes on every pull request several times a day.
“This builds a collective knowledge of the code and does a better job satisfying the intent of the compliance objective, which is to have a knowledgeable review that ensures the right stuff is in the code,” Guckenheimer said.
“Audit standards exist to assure that meaningful risks are mitigated. In the modern world, daily engineering practices should mitigate risk and provide evidence by default. In that way, audit compliance is a natural by-product. This eliminates the wasteful dance in which engineering teams (re)construct historical evidence for auditors without adding value to their own work.”
Prugh followed this approach with his teams to manage PCI checks, first bringing the education of engineers in compliance to the front of the pipeline, then designing systems with the correct inbuilt automation of compliance controls for continuous checks every 15 minutes. Engineers now extract a daily report that is provided to auditors.
This follows a core proposition of the DevOps Risk and Controls Matrix for managing the risk of “production breaks due to human error or untested/insecure code.” But Prugh warns automation for automation’s sake can produce undesirable results.
“If you automate a bad process, you make that bad process faster, so you have to think of how to reengineer parts of the pipeline to make the automation truly efficient. You need to change the process to have the automation make sense,” he said.
Empathy, Education, Empowerment
Changing processes also requires changing behaviors. Prugh has found success focusing on three critical paths to better outcomes:
- Empathy and building relationships between auditors and engineers is important because so oftentimes they only communicate through email and when things are bad.
- Compliance education and training gives engineers a basis of understanding, but it also shows auditors and security teams these areas of the organization are valued and worthy of investment.
- Moving the pain to where it will be fixed is necessary for solving critical problems, as in Prugh’s case of making it an engineering priority to re-architect their traditional method for creating compliance reports into a DevOps process supported with automated dashboard reporting on compliance.
Prugh is also changing behaviors by providing DevOps knowledge resources to his audit and product management teams, who are currently running a book club around The Phoenix Project, the seminal novel by Gene Kim, Kevin Behr and George Spafford about IT, DevOps and helping your business win, as its front cover highlights.
“I want the compliance folks to understand everything that happens in The Phoenix Project is a business failure that can violate what the COSO Cube in audit talks about,” Prugh said, in reference to the Committee of Sponsoring Organizations of the Treadway Commission’s model for evaluating internal controls.
“I want the product managers to understand all technology should be treated like a product, but also that we need to make strategic investments in things like risk reduction and technical debt, things that are often missed because product managers are always focused on building features,” he said.
But even following Prugh’s effective methods of empathy, education and empowerment, tools are absolutely necessary for enabling sustained change with DevOps. The key is knowing how to strike a delicate balance between technology and tact.
“There’s a virtuous cycle between the platform and the culture change. You cannot move the culture a lot if you continue to have all of these old impediments in how people work; and you cannot put in all of this great technology if you continue to work as though it didn’t exist,” Guckenheimer said.
There’s also a virtuous cycle between teams of auditors and teams of engineers. Despite rapid progress in DevOps, visible primarily in the acceleration of engineering, the philosophy cannot continue to transform within organizations at such a rate without involving other key groups.
Fostering empathy between engineers and auditors, enabling mutual education and giving them projects and tools through which a higher order of collaboration can manifest will make way for a new DevOps future.
Latest posts by Mike Siemasz (see all)
- GSE UK Conference 2018: Join Us to Learn About Agile Ops and Batch Automation - November 5, 2018
- Where Are You on Your Mainframe DevOps Journey? - October 18, 2018
- Don’t Miss It: DevOps Fireside Chat with Gene Kim and Chris O’Malley - October 15, 2018