The inherent difficulty in long term projects
The inherent difficulty in long term projects
One of the more interesting parts of being a software developer is balancing the needs of the executive or management team against the…
One of the more interesting parts of being a software developer is balancing the needs of the executive or management team against the constraints of software development itself, and of the software development team. This takes many forms and is constantly a process of give and take, but one that reached out to me in a particularly emphatic way is balancing a “project” with “agile software development”.
Agile vs Project
It’s perhaps best to start on common ground, and define what both “Agile” and “Project” style development might be.
Agile
From Wikpedia:
Agile software development is an approach to software development under which requirements and solutions evolve through the collaborative effort of self-organizing and cross-functional teams and their customer(s)/end user(s).It advocates adaptive planning, evolutionary development, early delivery, and continual improvement, and it encourages rapid and flexible response to change.
— https://en.wikipedia.org/wiki/Agile_software_development
Typically for me, this involves:
A list of tickets that are added to a “backlog”; a list of work items that need to be completed in future.
A periodic “grooming” session with all stakeholders to review each of these items getting them into a shape where they can be “planned”
A periodic “planning” session where those tickets are estimated, and taken into a “sprint” for delivery
Working on the sprint, trying to get all of the promised tickets delivered by the end of that sprint
A periodic “review” session to determine if the work is functioning as expected
This sits in tight loops, usually about 1–2 weeks long. Distinctly from projects, we do not plan timelines longer than this 1–2 week block; work is broken down until it fits into that block, and then completed.
Projects
By comparison, a “project” is much more similar to the “waterfall” style of development. From Wikipedia:
The waterfall model is a relatively linear sequential design approach for certain areas of engineering design. In software development, it tends to be among the less iterative and flexible approaches, as progress flows in largely one direction (“downwards” like a waterfall) through the phases of conception, initiation, analysis, design, construction, testing, deployment and maintenance.
— https://en.wikipedia.org/wiki/Waterfall_model
Projects are perhaps not so complete, and retain only the “design”, “construction”, “testing” and “deployment” phases.
Essentially, a larger set of requirements is drafted and that specification handed to a given developer for estimation. That developer comes up with a figure and returns it to the executive team for consideration. If the executive team likes the figure, the project is approved for development.
A developer, or a team of developers is then allocated to work on that project until its completion, where they integrate it with the rest of the software and release it.
Projects are usually 4–6 weeks long, but can be months or even years long.
Projects: The theory
It is my opinion that the “agile” development workflow is a far superior approach to software development. However, that is not to say that the project style of software development does not ostensibly provide benefits that make life much simpler and easier for the executive team.
Fixed budget
In perhaps the most motivating factor, a project defines a set of features that should be delivered for a given budget. That budget can be allocated ahead of time, and the return on investment calculated. That goes into larger budget considerations, and makes for a predictable business case.
Fixed set of features
All businesses are becoming software businesses in some capacity or other. In this way, adding a new business unit, product or any other organisational capacity has a component that is usually reflected in software.
The effects of these business units are compounding; a well run IT department can provide services to marketing, and marking can take advantage of a well organised product department. The executive team can steer all other teams to take advantage of shifts in the general market.
All of this is predicated on the business units behaving predictably, so they can rely on each other, especially over time. By defining the software features ahead of time, business can start to plan how those features can be trained for and integrated with other business processes.
Fixed timeline
Software does not operate in a world devoid of other stakeholders. It must adhere to the same budgets as all other business units, so many units can coordinate a product release, marketing campaign or new new business region.
Projects offer the allure of a fixed timeline that allows coordinating many teams together.
Projects: The practice
The aforementioned guarantees are excellent from a business point of view. With their roots in other business processes such as manufacturing and construction they’re easy to reason about, predictable and fit in with our conceptual models of how a business should run.
Unfortunately they’re extraordinarily difficult to deliver; especially for an organisation that is used to working in the much more dynamic agile delivery process.
Developers: Sometimes we don’t know what we’re doing either
It’s easy to consider software among the other time-for-labour based disciplines:
Construction
Hardware Engineering
Manufacturing
Marketing
However, software is unique in the cost to reproduce well developed work is nearly zero.
Throughout my career I have built many different sets of business logic; appointment management, coaching software, exercise planners, websites, shipping, carts, checkouts, search and currency systems. However, by and large I only ever build those things 1x. They might be refined subsequently, but the vast majority of all labour is completed a single time and then reused indefinitely.
That means all work falls into two categories:
Trivial: I can reuse work I (or someone else) did previously, dramatically reducing the amount of time required.
Unknown: This particular set of circumstances is partially new to me, and I have to guess how large parts of it should work.
There is, broadly speaking, no middle ground of “yes I’ve done this many times before”. This is a good thing, as it allows us to accomplish an extraordinary amount in a short amount of time — at the cost of such an extreme difference in our capacities.
Normally we manage this risk by plotting work with limited guarantees, over a period of ~2 weeks. It might be that we can accomplish releasing to France in 3 days, but that it takes us 5 to work out a particular quirk of CSS. To create an new, production ready store in 90 seconds but spend 3 months developing a design for that same store.
In the case of these large projects, these unknowns quickly compound. What might appear to be a simple problem turns out to be complex, and what appears complex can (with some discussion) be made simple again. Accordingly, the larger a project is, the less accurately we can predict how long it will take.
Large problems are innately unknowable problems
There is some difficulty in determining how much investigation to do for a large project. More investigation reduces the risk of the estimation, but investigation in and of itself accomplishes nothing. Indeed, the vast majority of a developers time is not spend writing code, but rather reading it.
Large projects are designed to solve large problems. Things like:
Redesigning an entire site
Transferring between one system and another
Designing and implementing a custom currency system
Completely rebuilding a new e-commerce system within a large business
There are a plethora of questions that occur immediately. However, it is not the questions that occur that hurts — it’s the ones that don’t. To quote Rumsfeld:
…There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns — the ones we don’t know we don’t know. … It is the latter category that tend to be the difficult ones.
— https://en.wikipedia.org/wiki/There_are_known_knowns
In a recent example, we had a project that required moving from one set of infrastructure to another set of infrastructure. This is a task that we understood well, and had completed many times previously. However, we discovered once we started implementing that there was a hard requirement on “no cloud” — tooling that we had spent years investing in perfecting. Our previous estimates were now worthless, and our previously acquired knowledge did not map well onto the task. The project went significantly over budget.
That project and the project team did all in their power to learn what was possible up front, and this project has served as a learning experience of what information to look for in future. However, the nature of large problems is they will always have “unknown unknowns”, and that the larger the project, the more “unknown unknowns” there will be.
Large projects are shockingly expensive
Software development, for better or worse, is still prohibitively expensive. A typical 8 week project could be calculated as follows:
8 weeks * 5 days = 40 days
40 days + (40 * .2 PM overhead * .1 QA overhead) = 54 days
54 days * 1500 EU / Day = 81000 EU81000 EU is a very large number to recoup for a return on investment, and many balk at realising how expensive such a project would be.
This creates a set of “downward pressure” on the implementing team. Questions like:
“Are you sure it will take that long?”
“Is there not a faster way of doing it?”
“How much safety time have you built into your estimates?”
Are always asked. While the correct answer to these questions are always “the estimate is fixed”, less experienced or secure developers may start to question their own numbers, revising them to a lower figure. In doing this, the developers hope that nothing will go wrong, falling into the scarcity trap.
Naturally, the project is much more likely to exceed this bound.
Projects usually do not exist in isolation
While building a room on a house, it is not usually common to build two rooms independently of each other. Rather, there is a single builder who coordinates the entire house, bringing in specialists to complete each room.
In the case of software development it is less clear. As the analogy goes, it is like both software developers are building independent rooms, and while there are rules that indicate how rooms should be built and the developers are very practiced, each developer might need to make some small foundational changes to the house.
The cost of coordinating these changes is significant, especially if the work is not “completed”. Each developer must both simultaneously develop for the new room, and for the new room not to exist, as each project might be released before the other. Or, worse, one project relies on the other project, and the delay in that project pushes out all projects.
The most efficient way developers can coordinate their changes is by making sure their respective rooms look enough like each other that there will be no difficulties; they “merge” those two rooms together into a single room with both of the developers ideas.
However, longer term projects mean more changes to each room before that room is compared with the other. Over time, the rooms get further and further apart, and when they are eventually reunited it is an expensive, uncoordinated mess.
Different ideas of the same concept make for mismatched expectations
Given any project, large or small, we need to form a shared understanding of how the required functionality should work. However, at the best of times it is difficult to be completely in agreement as to how things should work.
An executive team member may give a directive to a department to “push car sales more”. That department then decides the best way to do that is to make cars easier to understand, and requests a configurator — a tool to help choose a given make and model of car. The developer understands this, and then takes the work of a previous pen configurator they implemented and adjusts it for cars.
The problem is, the pen configurator is not what the department had in mind. And a configurator is not what the executive team member had in mind. Everyone is at best a little dissatisfied, or at worst, considers their colleagues stupid. However, it is not that the colleagues are stupid, but rather were making the decision to their understanding of the initial idea.
Large projects mean longer periods of time without ensuring that an idea is developing as it should. It could be many weeks to create a new system, and when that new system is delivered, it might not be what the project owner had in mind. The project owner will request changes which were not budgeted in the initial design. If there is budget remaining the developer will implement those changes, but if there is not, the project owner will need to pay more and accept more delays to have their project in the shape they intended.
Scarcity Economics
Large projects combine two particularly perverse economic requirements:
An apparently large period of time to complete the project
That time evaporating, and having too much to complete in a short period of time
This effect is well defined in the scarcity economics book, and essentially functions as a series of the following steps:
The project appears to have a large time to be completed, and thus ambitious goals are set. These are usually referred to as “Refactoring”, “Upgrades”, “Paying back technical debt”.
There are problems with these ambitious goals, dragging out their completion.
Suddenly, the project has not enough time and large amounts of the normal processes are sacrificed.
The project is delivered in a state that resembles complete well enough, but without the “normal” safeguards.
While the project started (and may moderately succeed) at it’s largely ambitious goals, it often fails them and leaves the project in a dissatisfying state that future “maintenance” must continue to clean up.
Compounding failure
Large projects take these problems that are inherent in software development and compound them, without any of the safeguards that agile software development otherwise provides.
It is not that large projects are inherently destined for failure — it is simply that failure is statistically more likely, and the failures left unchecked stack up.
This stacking up ruins any of the guarantees that the project model is supposed to provide. The project is not:
On budget
Delivering what it’s supposed to
On time
Despite being exactly what the project model was intended to provide.
A predictable development model
The problems of the project model of software development have long been known and understood, and a good model to address them already exists — the agile development model.
This model surfaces these risks directly by refusing these longer term, failure ridden structures in favour of structures that surface risks directly.
It addresses:
Developer lack of knowledge by making knowledge heavy tasks too expensive for the sprint
Larger, unknowable problems knowable by making them smaller and easier to understand
Project expense as all backlog tasks are estimated, and small but effective tasks can be easily identified and the project run much more efficiently.
Cooperative work with a regular release process that forces developers to synchronise their ideas
Verifying ideas by releasing them in small chunks, correcting them as they deviate from the required model
Forcing developers to focus on the delivery of their code, keeping the economic trade offs better aligned with business goals.
It is extremely tempting to fall back into the project model of development; it seems such a simple solution to the risks inherent in planning software over the long term. However, practically speaking it does not work out as it seems it should and dooms teams to broken promises and frustration.