Yearly Archives: 2022

Technical debt – the truth

.NET, Architecture, DevOps

Rationale

Within software engineering we often talk about Technical Debt. It was defined by elder programmer and agilist Ward Cunningham and likens trade offs made when designing and developing software to credit you can take on, but that has to be repaid eventually. The comparison further correctly implies that you have to service your credit every time you make new changes to the code, and that the interest compounds over time. Some companies literally go bankrupt over unmanageable technical debt – because the cost of development goes up as speed of delivery plummets, and whatever use the software was intended to have is eventually completely covered by a competitor yet unburden by technical debt.

But what do you mean technical debt?

The decision to take a shortcut can be a couple of things. Usually amount of features, or time spent per feature. It could mean postponing required features to meet a deadline/milestone, despite it taking longer to circle back and do the work later in the process. If there is no cost to this scheduling change, it’s just good planning. For it to be defined as technical debt there has to have been a cost associated with the rescheduling.

It is also possible to sacrifice quality to meet a deadline. “Let’s not apply test driven development because it takes longer, we can write tests after the feature code instead”. That would mean that instead of iteratively writing a failing tests first followed by the feature code that makes that test pass, we will get into a state of flow and churn out code as we solve the problem in varying levels of abstraction and retrofit tests as we deem necessary. Feels fast, but the tests get big, incomplete, unwieldy and brittle compared to the plentiful small all-encompassing, and specific tests TDD bring. A debt you are taking on to be paid later.

A third kind of technical debt – which I suspect is the most common – is also the one that fits the comparison to financial debt the least. A common way to cut corners is to not continuously maintain your code as it evolves over time. It’s more akin to the cost of not looking after your house as it is attacked by weather and nature, more dereliction than anything else really.

Let’s say your business had a physical product it would sell back when a certain piece of software was written. Now the product sold is essentially a digital license of some kind, but in the source code you still have inventory, shipping et cetera that has been modified to handle selling a digital product in a way that kind of works, but every time you introduce a new type of digital product you have to write further hacks to make it appear like a physical product as far as the system knows.

The correct way to deal with this would have been to make a more fundamental change the first time digital products were introduced. Maybe copy the physical process at first and cut things out that don’t make sense whilst you determine how digital products work, gradually refactoring the code as you learn.

Interest

What does compound interest mean in the context of technical debt? Let’s say you have created a piece of software, your initial tech debt in this story is you are thin on unit tests but have tried to compensate by making more elaborate integration tests. So let’s say the time comes to add an integration, let’s say a json payload needs to be posted to a third party service over HTTP with a bespoke authentication behaviour.

If you had applied TDD, you would most likely have a fairly solid abstraction over the rest payload, so that an integration test could be simple and small.

But in our hypothetical you have less than ideal test coverage, so you need to write a fairly elaborate integration test that needs to verify parts of the surrounding feature along with the integration itself to truly know the integration works.

Like with some credit cards, you have two options on your hypothetical tech debt statement, either build the elaborate new integration test at a significant cost – a day? Three? Or you avert your eyes and choose the second – smaller- amount and increase your tech debt principal by not writing an automated test at all and vow to test this area of code by hand every time you make a change. The technical debt equivalent of a payday loan.

Critique

So what’s wrong with this perfect description of engineering trade offs? We addressed above how a common type of debt doesn’t fit the debt model very neatly, which is one issue, but I think the bigger problem is – to the business we just sound like cowboy builders.

Would you accept that a builder under-specified a steel beam for an extension you are having built? “It’s cheaper and although it is not up to code, it’ll still take the weight of the side of the house and your kids and a few of their friends. Don’t worry about it.“ No, right? Or an electrician getting creative with the earthing of the power shower as it’s Friday afternoon, and he had promised to be done by now. Heck no, yes?

The difference of course is that within programming there is no equivalent of a GasSafe registry, no NICEIC et cetera. There are no safety regulations for how you write code, yet.

This means some people will offer harmful ways of cutting corners to people that don’t have the context to know the true cost of the technical debt involved.

We will complain that product owners are unwilling to spend budget on necessary technical work, so as to blame product rather than take some responsibility. The business expects us to flag up if there are problems. Refactoring as we go, upgrading third party dependencies as we go should not be something the business has to care about. Just add it to the tickets, cost of doing business.

Sure there are big singular incidents such as a form of authentication being decommissioned or a framework being sunset that will require big coordinated change involving product, but usually those changes aren’t that hard to sell to the business. It is unpleasant but the business can understand this type of work being necessary.

The stuff that is hard to sell is bunched up refactorings you should have done along the way over time, but you didn’t- and now you want to do them because it’s starting to hurt. Tech debt amortisation is very hard to sell, because things are not totally broken now, why do we have to eat the cost of this massive ticket when everything works and is making money? Are you sure you aren’t just trying to gold plate something just out of vanity? The budget is finite and product has other things on their mind to deal with. Leave it for now, we’ll come back to it (when it’s already fallen over).

The business expects you to write code that is reliable, performant and maintainable. Even if you warn them you are offering to cut corners at the expense of future speed of execution, a non-developer may have no idea of the scale of the implications of what you are offering .

If they spent a big chunk out of their budget one year – the equivalent of a new house in a good neighbourhood – so that a bunch of people could build a piece of software with the hope that this brand new widget in a website or new line-of-business app will bring increased profits over the coming years, they don’t want to hear roughly the same group of people refer to it as “legacy code” already at the end of the following financial year.

Alternative

Think of your practices as regulations that you simply cannot violate. Stop offering solutions that involve sacrificing quality! Please even.

We are told that making an elaborate big-design-upfront is waterfall and bad – but how about some-design-upfront? Just enough thinking ahead to decide where to extend existing functionality and where to instead put a fork in the road and begin a more separate flow in the code, that you then develop iteratively.

If you have to make the bottom line more appealing to the stakeholders for them to dare invest in making new product through you and not through dubious shadow-IT, try and figure out a way to start smaller and deliver value sooner rather than tricking yourself into accepting work that you cannot possibly deliver safely and responsibly.

Size matters

Architecture

When I started out, I had no idea about lean, agile or TDD. I did read a lot about Object Oriented Programming. I even thought I was practicing it. My first decade of programming was not what would pass for professional grade by modern standards, but even my poor standard back then was – and still would be – not the worst. Sadly.

Eventually I read some pamphlets about TDD, tried some katas, read books about lean software development and the book the Phoenix Project and was subjected to Fred George‘s Object Bootcamp with my co-workers a couple of employers ago.

Then I understood. All of it. Object oriented programming, TDD, lean. All of it. Because of how steeped in my old bad habits I am it still takes conscious effort to practice all these things, and I fail to enforce it sometimes, but at least at this point I know what good looks like. If I had stronger discipline I could refactor towards it as often as I like.

Like, if you make small enough focused classes, object orientation makes sense and you find yourself binning and replacing classes that are no longer fit-for-purpose rather than rewriting them. Open-Closed principle. There are all of a sudden more value objects and composable domain classes that cleanly describe domain behaviour and you come away from cascading changes that you thought were inherent in OO. But is it easy? No. It requires constant effort to maintain.

The most important thing is size. Make the smallest possible change to make the test go green. People – me, you – we? – overcomplicate things and fail to comprehend just how small small should be and in the refactor step, we rarely refactor enough and leave classes too big. Now I find this really hard, the commit-by-commit design step where you are supposed to refactor code into proper shape, reduce classes down to their most composable form, their bare essence. On the other hand, “It’s your only job, Rachel!” like Jimmy Carr says on Eight out of Ten Cats Does Countdown when she can’t solve a numbers game right away. We are paid not only to sit in front of our computers and type but also to think. If we do a bit more of the thinking and a little bit less of the typing, nobody is going to get upset.

Allen Holub talks about user stories on Twitter, reminding us that a story isn’t a neologism for “requirement” and that a user story literally means that, a software user telling us a story about a domain problem they are having. Not “as a user I want to authenticate so that I can have access to the system” but like actual meat on the bone. “As an underwriter I want to bind a quote as I have agreed terms with the broker”. Requirements for authentication will come as part of some story, but start stupid simple with a .htaccess file or similar. For some subsystems that requirement will never come, or can be trivially covered with infrastructure, and you just saved yourself a bunch of maintenance. Code is not an asset, it’s a liability.

Allen Holub also uses TDD when designing systems, as in uses Java and jUnit to TDD integrations between microservices like a less-trendy jupyter notebook proving his overall system design, even ending up with some code that could describe integration points. Start smaller. No, even smaller than that.

Then we have the No Estimates crowd. Seems insane to people used to the widespread non-story user stories with large blocks of detailed requirements that necessitates complex up-front engineering. How could teams take on these big stories without estimating how much they will take to build – what if we spend three months and we don’t even achieve what we want?!

That’s right. But – do you remember Agile? Yea, see the point was to build the smallest possible implementation that can prove viability. If your stories are sized consistently and correctly – i.e. smaller than you think possible – the consistent implementation work will give you the foresight you need to coordinate and plan as necessary.

We can then add features incrementally as users examine the existing product and realise the potential and get ideas based on their experience as domain experts for what they could do to meet their customers’ needs, or what experiments could be run to determine customer desires. In small increments still.

Even if you insist on keeping creating your estimates, you will find that making stories smaller will improve delivery quality and adherence to timescales. As soon as stories get too big, they are harder to reason about and you may even forget acceptance criteria you thought you had clear in your head. You will never regret making stories smaller.

SD&D 2022

.NET, Architecture

I had the chance to attend the Software Design & Development conference at the Barbican Centre this week. It was my first conference since the plague so it was a new experience. I will here coalesce some of the main points that I gathered. I may go into specifics in a couple of topics that stood out to me, but this is mostly so that I have some notes to aid my memory.

Overall

The conference is very well organised, you can tell it’s not their first rodeo. From a practicality point of view the Barbican is equally easy/cumbersome to get to from all directions, which is about as good as you can get in London where usually you will end up favouring proximity to a subset of main line termini, thus making the location cumbersome to get to for at least half the population (since there are airports in every direction out from the city). There were a number of timeslots where you truly had FOMO for choosing one track over another, which is a design goal of a program committee – so, well done SD&D! There was some unfortunate setting in the AV equipment which interfered with shortcuts in Visual Studio in interesting ways, but surely that can be addressed somehow.

Kevlin Henney

The first law of developer conferences states Always Catch a Talk by Kevlin Henney if Available. It doesn’t teach you anything about a specific new thing, but it puts the entire universe into context, and always inspires you to go ahead and look up papers from the olden days that describe modern phenomena.

C# features

My biggest bafflement came not from the talks that showcased new C# features, such as the latest iteration of pattern matching and the like – as I usually get introduced to them when they show up as options in ReSharper – but by the old abomination that is default implementations on interfaces in C#8. This talk by Jeremy Clark was an eye opener. It is so jank you wonder how it could ever be released into production. My guess is that the compatibility argument from MAUI was the big reason, and it makes sense, but basically my instinct to stay away from it was right, but my guess is that you will have weird bugs because of it at some point in the future.

I also caught the C# Channels talk. I seem to recall the gestation of Channels, as if I remember correctly it was publicly brought into being through discussions on David Fowler’s twitter account (unless I misremember). Anyway, it is a highly civilised way of communicating between two async tasks with back pressure. Intuitive to use and safe. Like a concurrent queue but a lot nicer.

Micro services

Allen Holub had a number of talks on Microservices and if you can I’m sure you should see more of them, due to the abundance of other brilliant talks I only caught his test driven architecture talk which I will mention below. Other than him though there were talks by Juval Löwy and Neal Ford that covered architecture and micro services. I caught Software architecture foundations: identifying characteristics by Neal Ford and Sander Hoogendoorns’s talk on migrating to microservices in small steps, both worth watching.

Security

There were a couple of security talks focussing on automating security as well as the updated OWASP top 10 chart of threats as well as a couple of talks by Scott Brady about – I am guessing – – Identity Server. The dizzying array of choices meant I didn’t go to the identity server talks, but they are definitely on my to watch-list if there is ever video from this.

The security automation ones though Continuous Security by Kim van Wilgen and Add Security into your Agile Process by Cecilia Wirén took you through what you need to really improve the security posture of your development process. Kim v Wilgen was more in-depth on tooling whilst Cecilia Wiren was more about the whole process and what to consider where. The only sustainable way forward is to automate things like dependency vetting/tracking and as much static code analyisis you can, to bring these concerns as far left as you can, menaing early in the process. Rewriting a function even before you commit it is easy. Having an automated fuzzing tool discover you have a buffer overrun vulnerability when you thought you were done is worse from a cost-of-remediation point of view, but of course letting a bad vulnerability out into the wild is an order of magnitude worse, so whilst catching them in the dev cycle is preferable, attempting to catch stuff late through more heavy handed automation that is too time consuming to run on every commit is still worth considering running periodically as it’s better than the alternative.

Tests drive everything

How to stop testing and break your codebase by Clare Sudbery was an amazing talk that really hit home. It was like an experiment of what if I just skip test-first for a bit and see what happens? brought out of time crunch, tiredness and a sense that it would be a safe trade-off because I know what I’m doing and – in her case – I have acceptance tests . Like I have noticed in various side projects, when you let go of the discipline the drawbacks come at you hard and fast – almost cartoonishly so. It was one of the most relatable talks I have ever attended.

Allen Holub’s DbC (Design by Coding):applying TDD principles to architecture was fascinating. It started out by being a bit “old man yells at clouds” about how real agile is index cards on a board, not Jira, and although yes, index cards or post-it’s on a physical board is preferable to an electronic board, the electronic board prevents me from having to commute four days out of the five, so – no. The points raised about authoring million detailed tickets ahead of time being a waste though, hard agree, and the rest of the diatribe against big design up-front I was all aboard with and probably to some extent already doing. The interesting bit was to come though.
He presented a hypothetical technical problem that needed architecting. Instead of drawing a diagram or writing specs, he whipped out an editor with his favourite version of junit and wrote some java code through TDD – all in one file – that implemented tests and classes that symbolised microservices and their endpoints, as well as the interactions between them. Light weight, easy to read for developers, pleasant tooling and at least as useful as a diagram. In both cases you start writing the code from scratch, but you have a design document that makes sense. I’m not 100% sold on the concept but it’s worth considering.

Various highlights

Tuesday morning keynote was about quantum computing, and reluctantly I must concede that it probably is the future, and the community seems to be looking for converts, but I just can’t go from quantum entanglement to taking data from a web page and shoving it into a database, so I guess I have to wait those three years before it hits the mainstream and it will become digestable for the likes of me. If you are into cryptography as in encryption, not the various ponzi schemes, you should probably get into it now.

I caught a Kate Gregory talk – on naming – after only being a fan off of her YouTube talks on C++ vs C, definitely worth seeing. She proposed the strategy of “just mash the keyboard if you can’t think of a name and go back to it later” was also brought forward elsewhere on the conference, the point being: don’t get stuck trying to come up with a name, start writing the code and as you start talking about what the thing you just wrote is and what it is responsible for, a great name will eventually become evident and then you use that. The effort of coming up with a good name is worth it, and with refactoring tools it’s worth just moving past the instant rather than making a bad decision. Once you have finished the feature and proceeded, the caveats and difference between the name and the actual implementation will fade into obscurity and when you come back to the same code in three weeks you will have forgotten all about it and can be misled by bad names as easily as if you hadn’t written the code yourself.

Conclusion

I really enjoyed this conference. Again, with a past in a program committee I really admired the work they put in to cause so much anxiety when picking talks. Obviously the QE2 conference centre in Westminster is newer so NDC London benefits from that, and if you want to see Troy Hunt and the asp.net core guys Fowler and Edwards you would be better off over to NDC, but SD&D had all the core things right and a wider array of breakout sessions, and infrastructure like the food was a lot less chaotic at SD&D than it usually is at NDC. Compared to BuildStuff and Øredev – those conferences I only attended as visitor to the city, so I didn’t have to commute, meaning of course that’s nice.

If you like the environment you’ll be pleased that there was no shilling or abundance of obscure t-shirts handed out that would have drained natural resources, but I suspect that SD&D would have enjoyed more sponsorships and an expo floor which allegedly they have had before . Since this conference was actually SD&D2020 postponed several times over the course of two years, it is possible that SD&D 2023 will have pre-pandemic levels of shilling. All I know is I enjoy free t-shirts to clothe the child. Regardless I strongly recommend attending this conference.

Life is friction

.NET

Life is just people and things working together to make things difficult for you. Like on a rainy windy day where you can just lean into the wall of oncoming air and water and just push through.

Most of these things you cannot really do anything about, and there is no point to complaining about it, but then there are small wins, like going around the corner of a big building and it taking a few seconds for the wind to change direction and blast you in the face again. Those few seconds are golden.

Anyway – one of those breaks in the rain is that I’ve switched off comments on my blog. There are two people on average that read a post, and rarely do they want anything from me. A handful of posts have over the decades accumulated hundreds of views. Among humans my writing has the attention it deserves.

The bots though are big, unrelenting fans and have an insatiable appetite for communicating all kinds of offers through commenting on my posts (that they can’t have read according to the page statistics).

I pay for a service that is supposed to deal with my popularly in the bot scene. An inbox zero-as-a-service, basically. Well those guys were annoyed that I sent too much traffic. Again two (2) readers per day generates enough spam bots that I either have to get an even more ludicrously expensive anti spam tier, buy a higher tier blog hosting to be allowed to add a captcha, or lastly self-host with expenses of both money and time.

I don’t want to do any of those as they cost money, and if you have seen the rest of the blog you’ll see why I’d rather not be spending any money on it. So I’m shutting the comments. I know this may lead to reduced “engagement” but the thing is, people that reach this page know how to reach me, so nothing is really lost, except friction.

I get that brief respite from the rain that you get at a large building site where the hoarding and scaffolding are overbuilt into a luxurious chip board arcade with strip lights and trip hazard warning tape everywhere. You get in out of the direct rain, but big drops from 70m up the scaffolding hit you directly on your skull through a gap in the chip board instead. It’s a win, but you’re never allowed to be too elated.

Anyway, if you need me, you know where to find me.

Transformers

Architecture, DevOps

What are the genuinely difficult aspects of transforming your software function?

It seems everybody intuitively understands what brings speed and short time to market, and how that in turn automatically allows for better innovation. Also people seem to get that in the current stale market, with fast enough delivery you could even forego smarts and just brute force innovation launching new concepts and tweaks until profits go up and then declare a win as if you knew what you were doing that whole time. Secretly people also know that although you could rinse and repeat doing the naïve approach until retirement, optionally you could exert minimum effort and measure a bit better so that you know what you are doing so that you can focus your efforts.

So why aren’t everyone moving on this?

When you get a bunch of people in the same organisation you want to achieve some economies of scale and solve common problems once rather than once per team.

This means you delegate some functions into separate teams. Undoing this, or at least mitigating this, is difficult politically. Some people – with some cause – fear for their jobs when reorgs happen.

Sudden unexpected cost runaway is the biggest recurring nightmare of middle managers. Controls are therefore in place to prevent developer cloud spend to balloon.

Taken together however, this means teams are prevented from innovating independently as they cannot construct the virtual infrastructure as needed because of cost not being authorised, and they cannot play with new pieces of virtual infrastructure because they haven’t been approved by the central tech authority yet.