Red Hat Helps Organizations Make Agile a Reality (Sponsored Content)

Talk about taking on a big job: Red Hat wants to bring agile software production practices — and the company’s OpenShift application development platform — into the Department of Defense with their decades of constraints, habits and bureaucracy.

And the task is made more daunting still, because making an organization agile isn’t just a matter of buying stuff.

“Agile is not bought, it’s taught,” said Red Hat Public Sector Lead Transformation Specialist Chuck Svoboda, “And it needs to be taught by seasoned practitioners.”

Svoboda dismisses the idea that an organization can become agile simply by buying the right technology—like the lightweight and easily replicable software called containers.

“There’s this mistaken idea [in the market] that just by adopting new technology like containers, or [container management software] Kubernetes or even an application development platform itself, you solve the problem,” he explained. In reality, he added, “If that’s all you do, you really haven’t done anything much but add complexity.”

To become truly agile, Svoboda said, “You need a different culture to drive, agile methods and practices. ... It needs to become a part of the organization’s DNA.”

Being agile vs doing agile

The principles laid out in the Agile Manifesto in 2001 outline an iterative, flexible and responsive approach to software development—where requirements can change right up until delivery and applications are updated continuously after deployment. It’s often contrasted with the Defense Department’s traditional waterfall approach, in which requirements are gathered at the beginning of the process and then the software is written to fulfill them.

But the years that are often spent gathering and refining requirements can make waterfall development worse than useless in the face of today’s rapidly evolving threats and the quantum speed of technological development, Svoboda said. “In that period of time, the threats may have changed, technology has changed, everything’s changed—but waterfall development doesn’t let you go back and adjust... So I might release something that’s inferior to current technology and completely out of need from a requirements standpoint.”

The dangers to the warfighter of being equipped with out-of-date weapons or equipment that can’t help them meet current mission requirements are clear, he pointed out; and our enemies take advantage of this deficiency.

Implementing agile methodologies for software development means a radical change of culture for the Defense Department. And even organizations that understand that agile has to be learned rather than bought often come at it the wrong way, Svoboda said.

“There’s a world of difference between just being agile and actually doing agile … Being agile is where you have an organization that goes out and gets a bunch of certifications … and then says, ‘OK now I’m agile.’”

Barriers to agile

But organizational cultures are resilient, Svoboda points out, and to truly reap the benefits of agile methodologies, organizations need to commit to the transformation — and work to implement it every day.

“Doing agile is actually living it—living the spirit of the agile philosophy through adherence to agile principles, by implementing more and more agile methodologies, by executing on more and more agile practices—all while ensuring that you don’t fall back into a waterfall-type mentality.”

“Taking theory and making it practice is very difficult,” he concluded.

That difficulty can make it hard to get buy-in from the top of an organization. There’s sometimes leadership reluctance to start a transformation because they want to see someone else do it first; “It’s the classic ‘second mouse [into the trap] gets the cheese’ thing,” said Svoboda.

“I get it,” he continued. “Software is hard. Agile transformation is hard... and it’s expensive, but you must be willing to take a leap of faith and just start—with a partner who has done this before like Red Hat.”

One way to help overcome that reluctance is through the use of metrics to provide assurance the transformation is working. “During a transformation, you must beware of hiding behind agile mysticism … You have to measure success the right way.” He suggests starting with the quantitative assessment scorecard developed by DevOps Research and Assessment, or DORA.

The Red Hat way

To begin the transformation, it’s best to start small, he said. “You don’t do a big bang, you do it small. It’s evolution not revolution, because revolutions are bloody and they destroy. On the other hand, evolutions create; this is what you want.”

Red Hat agile transformations generally start with a residency at the company’s Open Innovation Labs for a small group from the agency. “It’s a two-pizza team of 6-8 people,” he said. “We take them away from their day to day distractions, into a separate environment—one where it’s easier and safer to innovate. And we build something with them.”

It’s a “shock and awe” approach, designed to inculcate developers in the agile methodologies the company calls the Red Hat way—and learn from them about their own organizations.

“We don’t want any of these agencies to be exactly like Red Hat. We want to give our Red Hat DNA to them so they can take our best practices, the way we build software—we’ve been doing it for over 20 years at scale—and make it their own.”

The idea is to turn the team into a kind of “patient zero” that can infect an organization with agile methods and practices.

“What we’re doing is we’re taking them away from bad habits and we are infecting them with the Red Hat way,” Svoboda said. At the end of the two month residency, “We assess and figure out how do we scale this out” in that particular organization, based on what has been learned from the small team.

The dojo phase

After the initial lab residency, comes what Svoboda calls “the dojo phase,” when Red Hat staff begin using the same immersive techniques to train a cadre of leaders.

During the dojo phase, “We are teaching not only developers and operators and program managers, but we’re also teaching the leadership of those teams to help train and build up agile teams themselves—it’s training the trainer,” he said.

The dojo phase typically starts to bear fruit after six months, said Svoboda, by which time “the customer typically has development team leadership that is able to continue to scale out and transform [successive] teams to agile.”

In one engagement Svoboda highlighted, the initial two pizza team had scaled out to more than 100 strong within a year. By contrast, the number Red Hat staff on the engagement had only doubled. “It’s very nonlinear,” he said, “Our people are force multipliers, not staff augmenters.”

Trusted tools and guardrails: Keeping developers honest

The cultural transformation is accomplished in part through technology: The tools that Red Hat OpenShift and the company’s partners provide for developers enforce best practices—defined differently for each customer organization.

“Tools keep developers honest,” said Svoboda. Automated control processes in OpenShift tools enforce processes like code tests and quality audits.

A common hazard agencies face are containers. “The promise of containers is developers can put whatever it is they want in there and toss it over the fence [into a production environment] and the operators can run it wherever it’s needed.”

“But actually,” he points out, “You don’t want developers putting whatever the heck they want in containers because you lose control. … You can’t determine whether there might be security vulnerabilities or other badness in code.”

The OpenShift application development platform leverages a Continuous Integration/Continuous Deployment, or CI/CD pipeline, “To shepherd the developers in a prescriptive fashion—meaning we provide the developers with accredited and hardened tools, and an accredited and hardened development environment … and force them to adhere to certain minimum requirements.”

Unlike just telling developers they can’t deploy code until it meets certain minimum quality requirements, the tools enforce compliance, Svoboda explained.

“We can tell people to use agile methods and processes like test driven development … But how do I enforce them? How do I enforce security best practices?” Svoboda asked.

“The chaining of CI/CD tools with specific configurations and quality gates enables us to not just urge people to use agile best practices, we can enforce that automatically and programmatically,” he said.

Svoboda compares the limits set by Red Hat tools to the guardrails and other features of a highway—essential for an organization trying to negotiate agile methods and practices. “Would you want to drive fast down a dirt road? No! That’s why you want a highway, with lane markers and guardrails and lights. The way a lot of organizations approach agile—just dumping containers or Kubernetes or an app development platform—is asking developers to move fast, but with no structure.”

The trusted software supply chain

These guardrails form what Svoboda calls a Trusted Software Supply Chain (TSSC). And like an industrial supply chain, it’s all about producing quality software out of quality parts, in an indefinitely repeatable fashion. A major benefit of the TSSC, according to Svoboda, is the audit and trust built in at every step, “which provides what I call automated provenance [for code] which is traceability to the origin every step of the way, enforced by automation of the platform.”

“The trusted software supply chain, in our definition, is the enforced sequence of processes involved in the development and deployment of software,” he concluded.

And the exact parameters of those guardrails can be set for each organization, depending on their mission needs—although the customization tends to fall into a few categories.

“Every agency thinks they’re a special snowflake, but I like to remind agencies: All snowflakes are made of water and water comes in three phases. The trick is to figure out which of the three phases an agency is—a solid a liquid or a gas. The point is we build these guardrails based off the needs of the customer and the nature of the mission, including the unique constraints such as security and accreditation of the DOD.”

Agile: A never ending journey

The agile philosophy is all about continuous improvement, so it’s perhaps not surprising that Svoboda doesn’t want to put a timeline on the agile transformation of DOD.

“Agile transformation, like software, is like mowing your lawn,” he said. “It’s never done because when you cut a blade of grass, it starts growing again.”

Once the dojo phase is complete and the organization has a cadre of leaders ready to continue the work of agile transformation, Red Hat staff transition to a “trusted advisor” role.

But agile is a journey, not a destination. “You’re never full agile. If you ever hear someone say, ‘Oh we’re agile now we’re done with our agile transformation,’ they don’t get it and you should be skeptical.”

“Agile is a journey and I believe it’s an indefinite one.”

For more information, go to https://www.redhat.com/en/services/consulting/open-innovation-labs