Why do companies still use Waterfall today?

waterfall

For many years, people have touted the advantages of an iterative development model versus a sequential waterfall model. While waterfall is a laggard as depicted in the adoption curve below, there are still countless companies who still use waterfall today. This blog discusses iterative model advantages compared with a sequential model.

Requirements change

In a waterfall model, teams typically go through at least 4 distinct, sequential and non-overlapping phases: requirements analysis, design, code and test. A team does not proceed into the next phase until the current phase is “done”. So, a team would not start design until the requirements are done. That, in and of itself, poses an issue since requirements often continuously change. Unless the entire waterfall cycle is very short then it is often not feasible to disallow changing requirements. Yet, many waterfall projects I have seen somehow introduce new and changing requirements in any phase while the process does not smoothly allow for this.

Kanban and scrum are both iterative

Iterative models include both timeboxed iteration models such as Scrum or continuous flow models such as Kanban. While Kanban has several advantages when compared to Scrum (See https://kanbanmentor.com/how-kanban-streamlines-your-process/) , either of these iterative models will yield benefits compared to issues seen in a sequential waterfall model.  This blog specifically addresses the following issues: lack of continuous feedback, lack of ability to apply technical lessons learned and big bang integration issues.

Continuous feedback and early deployment

When building complex systems there are lots of unknowns including both requirements/market/business unknowns and technical unknowns. Assuming we are building something new and different, we need to continue to build a little, get feedback that we are heading in the right direction, perhaps modify what we just built and then build some more. While there is a lot of change that occurs, iteratively changing is a lot less costly in the long run versus waiting until the end only to find we built the wrong thing. The other inherent benefit of this approach is the possibility that we may deploy software early. We do not need to wait until everything is built before making revenue on the subset that has been  built.

Technical unknowns

Getting early visibility into technical unknowns is another benefit of iterative development. In complex systems it is often nearly impossible to know what technical problems will occur just by doing design. It is only after the code is written, tested and integrated that we know how it will perform. Even if the software performs as expected with respect to the requirements, there can still be issues with performance.

Big bang integration

Big bang integration can be a nightmare. I recall when I started my career as a software engineer back in the 1980’s. Our team worked on huge complex projects for a defense contractor which took about 4 years to get through the waterfall lifecyle. We were allowed do nothing but requirements for an entire year. We created reams of documentation. Following several days of  grueling meetings to prove to the customer that we understood the requirements, we were allowed to proceed to the design phase. After producing even more documentation over the next year, and again following a series of meetings, we were given permission to code. After one year of coding, we finally attempted to integrate and test everything together at the end.

At the end, we could not even get the software to build, let alone integrate and test. Moreover, while we had design contracts between teams depicting exactly how they should integrate, there were still communication mismatches. It can be extremely difficult to try and integrate lots of software together at the tale end of a large project. The fact that these projects were 4 years in duration magnified the issues. But even small waterfall projects can have big bang integration issues.

Change can be difficult but worth it

These are just some of the issues when building software using a sequential model. It is a surprise to see companies still using this model today. I suspect most are doing so since that is the way it has always been done in the past and change is difficult. While that may be true, the benefits of an iterative model outweigh the pains of getting there.

Waterfall tendencies within Scrum

When I initially wrote this waterfall blog, I ended it with the preceding paragraph about change being difficult. However, I recently saw some posts and comments in user group forums that compelled me to add just one more paragraph. There are many teams who are doing Scrum who came from a Waterfall background. Some of these teams tend to still want to do lots of upfront planning along with timelines and roadmaps and then more detailed planning each and every sprint.

One of the common problems with this sort of detailed planning is that it gives a false sense of security to managers that the plan must be correct because it is so detailed. One of two things typically happen as a result. Either:

  1. The team will stick to their detailed plan since they spent so many hours creating it up front and they do not want to tell management it needs to change or
  2. as they get feedback from stakeholders and learn more, requirements and plans change so they are constantly updating the detailed plan and consequently wasting time in the process (with no added value)

While these teams are still doing iterative development, it feels like a waterfall model disguised as Scrum. Having said that, this may still be a good stepping stone for very traditional organizations to migrate to an iterative model. I would recommend they choose option 2 and not stick with their detailed plan (the better of the 2 evils). As the team and organization matures, they can remove the waste associated with all this detailed planning.