Falling into the “technical debt” poverty trap
Falling into the “technical debt” poverty trap
Modelling technical debit as scarcity
Modelling technical debit as scarcity
At the minute I’m taking a couple of weeks off to catch my breath, see some family and friends and generally collect myself. As part of this mental cleanse I’m reading material of a non-technical nature that I otherwise do not. Among the purchases I picked up in this period was the excellent “Scarcity” book.
Written in a similar style as “Freakanomics” or “Thinking Fast and Slow” it defines the idea of “Scarcity” and then explores its impact in a number of interesting settings. The book notes:
“The definition of scarcity is inherently subjective. One person with a lot of wealth but many desires can in principle experience the same scarcity as another with less wealth (and fewer desires). This subjective definition of scarcity is essential for understanding the psychology.” — Scarcity, pp 234
So, if I may paraphrase it is the perception of having “not enough” to live up to expectations that are placed upon our lives.
The implications laid out in the book were remarkable, and provide a number of useful metaphors to make predictions about how people will react given a specific set of circumstances. However, when reading material I tend to try and consider against the projects, tools and processes that I work with on a daily basis and determine where these things may be improved given the insights that such literature might impart. In this case scarcity was particularly reminiscent of another common metaphor that we have in software engineering — “technical debt”.
Modelling technical debt as a scarcity
To understand how our understandings of technical debt might be improved by the study of scarcity it is useful to provide a parallel concept similar to that we have likely faced in our own lives.
Payday loans
Wikipedia defines a payday loan as:
a small, short-term unsecured loan, “regardless of whether repayment of loans is linked to a borrower’s payday.” The loans are also sometimes referred to as “cash advances,” though that term can also refer to cash provided against a prearranged line of credit such as a credit card.
It is perhaps most useful to provide one of the litany of examples from my own life where I have hit circumstances where such a payday loan was required.
Due to various unfortunate circumstances, I struggled to make enough cash to support myself well through university. I had various commitments:
Rent
Food
Transport
Books, equipement
Social Commitments
Work Commitments
In the end, I had a small amount of money saved, and I was able to make it through university for a period edging slowly into these savings. However, over time, the savings ran out but the commitments remained steady. I sacrificed where I thought possible, dropping out of social commitments and reducing my food diet in almost comical ways. However, this was still not enough; I would need more.
I am perhaps fortunate here in that I was able to borrow money from family, with the promise to pay it back in the next pay packet. However, my family was also not cash-rich — they depended on that cash being returned.
I was essentially drawing on “family backed” payday loans.
Technical Debt
Once again referring to Wikipedia:
Technical debt is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy solution now instead of using a better approach that would take longer.
It’s perhaps not simple to see how this relates to payday loans, so let’s once again explore this with a concrete example.
In the current project I have there are several components but the main two are:
An e-commerce store, used to sell goods
a content management system (CMS), used to host articles
They looked like:
These two applications form the bounds of the customer facing service, “foobar.com”.
One of the requirements of this project was to take coupon codes that were managed by the “e-commerce” application and display them in the “content management” system. This is is a fairly standard requirement; applications need to talk to each other all the time. This website has visible on your device because the application “browser” talked to the application “webserver” over the language “HTTP”!
However, making two applications that do not ordinarily talk to each other talk to each other can be expensive. We must:
Make one application open to conversation (a “listener” or “server”)
Make one application able to ask questions (a “client”)
Agree on a standard language to converse in (or “protocol”)
Ensure that this conversation is secure
The mechanisms to do this are fairly straight forward — define an application programming interface using REST (ideally via OpenAPI or Swagger), implement TLS server or mutual authentication and add code to ask and answer the questions:
The problem is one of time. As software engineers we’re always under pressure to deliver as much as possible in terms of feature set for as little time as possible. This problem is not unique to software; all construction is more profitable if we’re able to deliver more for less effort.
Accordingly, while optimising we found another solution: we can simply start one application from within the other application! This was possible because:
The applications were in an interpreted language (PHP)
The applications were on the same file system, and
The applications were deployed together.
This more like:
Hah! Brilliant! We have the same result, but for far less effort. Except, it turns out this is not such a good idea.
Shortly following this work, other new feature requests came in. Requests like:
Upgrade PHP
Ensure that the CMS does not have access to credit card data
Ensure users can log in to the CMS via single sign on (SSO).
Make the CMS faster
Immediately the gains made in time by taking the “simpler” solution and simply bootstrapping one application within another were incompatible with future tasks. Specifically,
PHP could not be upgraded on a per-application basis, dramatically increasing the risk of deployment and thus testing that must be completed prior to.
If the CMS bootstraps the shop, it has access to all shop data
If the CMS can sign in via SSO, then the shop also has access to this SSO data even though it should not
The CMS bootstraps an entire secondary application for a series of requests making it much, much slower.
This means that these tasks now take longer to do, and we lose any benefit of the shortcut we took earlier.
Furthermore, in this case as in many others, the work needed to be redone completely to resolve the issue. Our time implementing the feature in the first round was essentially wasted. Thus, all future features took a much larger budget than they necessarily needed to.
Commonalities
In both the Payday case and the technical debt case decisions were made that alleviated pressure in the short term, but increased the cost over the longer term.
Incurring debt
In both cases the intention was not to set out to take on a certain amount of debt to complete the work, rather a decision made as circumstances reduced the alternatives. Each task seemed at least ostensibly possible — both living with a modest wage and attending university, and implementing the coupon component of applications. However, both cases similarly fail to capture an inherent tendency of human life — disaster.
It is impossible to factor in planning for specific disasters. Each disaster is unique, unplanned and destructive in its own, unique way. However, it is a truism that there will *always be* disasters. I cannot think of a single period in which my life has been “disaster free”, and even if that were true it would be because my life has been reduced to such a heavy level of risk mitigation I would accomplish nothing. The same is true of projects — there are always unplanned events that delay and extend the cost of future work.
In order to retain our current quality of life or project schedule we must spend to resolve the disaster. If we are lucky and not overstressed we can solve this by dipping into our savings, effectively buying our way out of the disaster to keep things on track. However, if we find ourselves in a disaster position and we do not have the savings to fall back on we must incur some debt in order to continue.
This is evident in the payday loan example borrowing money from family. Should I have accrued more savings, asked for a raise or any number of other financial investments I would have been able to continue university, perhaps continuing to graduate. This can be additionally found in the project example; should we have implemented the coupon code in a way that did not require applications to intermingle we would have been much faster and cheaper implementing PHP upgrades or other system wide changes.
In all cases the common thread is the optimistic budgeting of either time or project architecture, failing to factor in life’s inevitable problems.
The poverty trap
It is perhaps the most nefarious property of the debt cycle that it is extremely difficult to break out of. The problem is at least twofold.
First, and perhaps most straight forward, debt has a cost associated with it. In the case of pay day loans the cost is financial — a high interest rate on the loaned cash leading to less cash being available even once the loan is paid back. In the case of a project the costs are perhaps harder to spot, but still present. Each new implementation is now slightly more expensive and each bug more complex. So, no matter what incurring debt will put a burden on a project already so burdened it had to incur debt.
Second, once debt is incurred we tend to break into a cycle of short term decision making, becoming “tunnel visioned”. Once we have solved our immediate problem we are focussed only on solving the new problems our compensation introduced; paying back the debt. In the pay day loan example this manifests as thinking only until the next pay check so that the loan can be repaid, after which “normal” planning can resume. With the project it’s much the same; the intent is to return to fix the introduced difficulties “at some point”. However, this implies an *even more optimistic* set of plans. We have now only not planned beyond resolving our first issue, we are completely unable to see that another new disaster will strike!
Invariably the disaster does strike. This puts a new strain on either the project or the pay check. The new disaster requires new compensations; another loan to pay off the first plus a bit more to help solve this new disaster or hacking in even wilder changes to the application. This leads in turn to shorter term decision making and even more optimistic budgeting, more disasters and so on.
Eventually we are trapped in a cycle of debt, with expenses far beyond our initial disaster.
Staying out of the trap
Broadly, in order to stay out of this cycle of debt we need to retain a safety net and refill and grow this safety net over time as our risks increase. In other words, we need some “slack” in which disasters can happen but we can quickly recover without incurring additional debt.
Consider the recent scenario in which my wife and I went on holiday to London. Unfortunately we forgot a card that allowed access to one of our bank accounts. In the scenario in which we did not have much “slack”, this would have significantly affected our holidays. We would be stuck in a foreign country, unable to participate in family traditions and we would spend our otherwise relaxed period ensuring that we can still afford what we need to. However, we are fortunate in that we had worked hard to retain both some savings and a larger income than those years before had left us. The result was a much less interesting decision to use one account now and transfer the difference across when we returned. We used the “slack” of the additional bank balance to avert what would have otherwise been a significant problem.
The same is true of projects. Consider another project that has failed critically an inopportune moment. All services are unavailable and customers are becoming very unhappy, costing the merchant significant cash. If we should not have invested in tooling that gives us visibility into this failing application and its surrounding system the investigation would take much longer to restore services to the system if it was possible to return at all. Additionally, we would have no insight into how the issue occurred. However, because we had implemented this tooling the investigation was short, the issue discovered and fixed and the system rapidly restored to working state.
Accordingly, to stay out of the debt cycle we must accept some level of “inefficiency”; some equity that is not spent on our immediate needs but rather retained to mitigate the cost of the inevitable disaster. For loans it is simply cash. For projects it is investments in testing, monitoring and delivery.
Paying down debt
All projects, and I would hazard a guess all people retain some level of debt. At least with regards to projects it happens that projects will come to us only once they have become a disaster with their previous provider, and we must find a way to cost effectively reduce this debt while still retaining enough forward momentum to make investment reasonable.
There are several ways to pay down debt. If the debt is simply too large that it cannot be repaid, it is perhaps reasonable to declare bankruptcy. However, this has several negative implications:
Losing control of capital
Unable to borrow any further credit
Having assets carved up and sold
In project teams, bankruptcy is “replatforming”. The move from Magento to Magento 2, or to Shopify, Shopware or any other software package.
However, it is often possible to implement some level of austerity to pay down debt but still retain forward momentum. In terms of the previous pay day loan story, I left university in another city and went to work in my home town. It was there I perfected other skills that lead me to my current vocation, and I was able to break free of the debt cycle before it landed too deep.
In terms of a project, this is best described as refusing to implement any further technical debt, and progressively paying down (refactoring) that debt over time with other work. It can mean reducing or deleting features, software and managing expectations for the pace of future development.
In both cases it is the shifting of circumstances to find a more sustainable pace to proceed.
Conclusion
In projects, as with life, it is sometimes a necessity that we take on some level of debt. However, debt can quickly force us into a cycle of short term decision making. Further, it might be that we become so tunnel visioned that we cannot even see the problems with our decision making, setting us up for failure with future disasters and ever increasing debt. Therefore, decisions to incur debt need to be made cognisant of the risks of falling in to the debt cycle. More generally I try to avoid debt wherever possible, and where I do have it pay it back as quickly as possible.
Thanks
Notes
The technical debt example was made up and the university example embellished. They are, however, rooted in similar true stores.