Demystifying the Agile, Waterfall, and Lean Conundrum
In an era where many organizations are navigating a “Double S” curve of innovation, transitioning from agile methodologies to enterprise agility, persistent confusion surrounds the concepts of agile, waterfall, and lean.
I recently stumbled upon an intriguing piece in the Harvard Business Review titled “It’s Time to End the Battle Between Waterfall and Agile” (you can find it here: https://hbr.org/2023/10/its-time-to-end-the-battle-between-waterfall-and-agile). The notion of putting an end to conflicts has always been enticing, and this article piqued my interest. It champions the idea of finding a middle ground through hybrid project management methodologies, and I’m all for it.
But while I’m a staunch supporter of the concept of a hybrid project or product management methodology, there are some aspects of the article that need some clarification, particularly from the perspective of an agile enthusiast. Having dedicated a significant part of my adult life to accelerating the delivery of customer value through lean and agile principles, I’d like to offer a few insights.
Clarification #1: “The Agile Methodology” is a Misnomer
Formalized back in 2001 through the Manifesto for Agile Software Development, agile methods are a diverse set, encompassing team-based and scaled approaches. We’re talking about well-known team-based agile methods like Scrum, Kanban, and eXtreme Programming (XP).
The term “Agile” includes a number of team- and scaled-agile methods
There’s also a treasure trove of lesser-known team-based agile methods like Adaptive Software Development, Dynamic System Development Methodology (DSDM), and Crystal, all of which deserve a seat at the discussion table.
Over the last decade, the Scaled Agile Framework for the Enterprise (SAFe) has taken center stage as the most recognizable scaled agile method. But we mustn’t forget its lesser-known scaled agile counterparts, such as Large Scale Scrum (LeSS), Nexus, and Scrum at Scale. Each of these bring their own unique capabilities to the equation. For example, Scrum at Scale recommends an Executive Action Team (EAT) to coordinate the agile elements of an organization, and to interface with its non-agile elements. From the Scrum at Scale Guide, https://www.scrumatscale.com/scrum-at-scale-guide-online/:
Sample Diagram showing an EAT coordinating 5 groupings of 25 teams
Here’s the bottom line: Any approach to project (or product) management, whether agile or not, should be generative, not prescriptive. It should be tailored to the specific circumstances an organization faces at a given moment. So, when we talk about “Agile,” it’s not just one thing. It’s a diverse realm, making a strong case for a hybrid, “fit-for-purpose” methodology.
Clarification #2: Waterfall is Iterative and based on the PDCA Cycle
Let’s take a trip down memory lane to Winston Royce’s 1970 paper, “Managing the development of large software systems” (you can read it here: https://www.praxisframework.org/files/royce1970.pdf).
Iteration between phases in the Royce’s Methodology
This paper underscores the inherently iterative basis of Waterfall, rooted in the Deming-Shewhart iterative Plan-Do-Check-Act (PDCA) cycle. And guess what? This alignment with agile methods is crystal clear because agile methods themselves are built on this same iterative PDCA cycle.
Incidentally, there are no substantial differences in the iterative approach represented in Scaled Agile methods like SAFe and Royce’s large systems methodology.
Both SAFe and Royce’s Method for Large System Development Incorporate Iteration
Now, when it comes to documentation, practices differ within agile methods. For instance, DSDM, developed by a business consortium, takes a more meticulous approach to documentation compared to Scrum as detailed in the official Scrum Guide. What’s intriguing about DSDM is that it’s pragmatic and generative, capable of working alongside existing standards and approaches, be it Prince2, ITIL, or ISO. It even covers the entire project lifecycle, from feasibility to deployment, enveloped by pre- and post-project phases.
See https://www.agilebusiness.org/dsdm-project-framework/product.html.
DSDM is a rigorous, governance-based agile methodology
So, here’s the key takeaway: Waterfall shouldn’t be demonized. The Agile Manifesto emerged as a response to what its authors referred to as “monumental methods” – derivatives of Royce’s approach that stripped away its iterative basis, converting it into a rigid, sequential, plan-driven nightmare that led to numerous project fiascos.
Clarification #3: Lean Management is Timeless and Pervasive
Born in the Toyota production system and popularized as Lean in the West, this management approach forms the bedrock for many project and product management methodologies. Lean principles have transcended their automotive roots, influencing a wide range of industries.
Lean principles found their way into Eric Ries’ “The Lean Startup,” a concept that extends to business model generation and product-market fit. By fusing eXtreme Programming with Steve Blank’s “4 Steps to the Epiphany,” the Lean Startup approach champions iterative business model development and product success in dynamic markets.
In the realm of agile product and project management, Lean principles provide the essentials: a laser focus on the customer, the seamless flow of value, small batch sizes, minimal work in progress, and, perhaps most importantly, respect for people.
Notably, Lean stresses the importance of breaking work into smaller increments, dynamic prioritization based on direct customer input, and speedy delivery of these increments to customers.
What’s crucial to remember is that this emphasis on Lean doesn’t negate the value of documentation, planning, design, or other quintessential agile components. Instead, Lean promotes a “just-enough, just-in-time” approach to these elements, a philosophy ingrained in methodologies like Adaptive Software Development, DSDM, and Crystal.
Following the classic Lean approach, Alistair Cockburn’s Crystal methods call for varying levels of rigor and discipline, depending on the unique situation at hand.
(Diagram from Alistair Cockburn, Agile Software Development, 2006)
Crystal’s Family of Methods is Ideal for Hybrid Projects
At the end of the day, the most effective hybrid product or project management approach, when operating at scale, is a Lean, “fit-for-purpose” method. It’s rooted in the needs of the customer, adaptable, equipped with essential governance elements, and capable of delivering tangible business outcomes. With Lean as the enduring foundation, there’s no inherent conflict between agile and other methodologies.
