The bygone much-derided Software Craftsmanship debate brought forward and addressed some of the same topics we now see bubble up from the large democratisation of software construction as more – previously only casually acquainted – people are introduced to the challenges of building software.
If you recall, the software craftsmanship was about telling developers to stop acting like labourers and start acting like professionals. I.e. instead of blaming tools, managers or customers, they would start taking responsibility for the quality of the software product. Not offering to introduce poor constructs as a way of achieving short term time savings, when as a supposed professional you know what the risks are. There were proposals of self-regulating the industry, introducing something akin to a bar association and licensing, so that you had a professional license you were risking if you turned a blind eye to data security or similar. The thesis was, if we do not do it ourselves, it will be imposed upon us by governmental regulation as soon as the loss of life becomes too egregious.
The reason for concern at the time of the Software Craftsmanship debate was the exponential growth of the programmer workforce, with a majority of active software developers having less than three years experience, combined with the way software was permeating all aspects of life, getting into cars, fridges and toasters et c.
Software bugs took the first lives back in the mid 1980s, and with people vibe coding apps or running AI agents without enough experience, we will see drastic increases in cost, hopefully only economical.
I myself likened it to plumbers and electricians having certifications within the trades, where they could risk exclusion if they did not adhere to agreed standards.
If the explosive growth of the programming professions were a problem in the mid 2010s, what do we think is happening now?
I liken the new sets of AI tools to the proliferation of power tools that are available to the general public. It does not have to be a disaster to do your own plumbing or electricity in your house, but if you ask any plumber or electrician they seem to have a kaleidoscope of experience of DIY installations they have encountered that were interesting / dangerous.
There is a reason why the council might send a man round to see what you are up to when undertaking construction on your house.
The point of this is not to broadly accuse people of being the software equivalent of a cowboy builder, the point is – I think the software developers that will keep their jobs going forwards are the ones that abandon the role of labourer and at minimum become a foreman over their AI agents. Occasionally lift your gaze above the line-by-line and examine architecture, question the overall structure of what is being built. Along with ensuring that the code remains sound, we additionally need more eyes on the bigger picture. If you have an organisation above you that make strategic decisions about software architecture, ask good questions, bring contrary opinions – maybe your observations are correct, maybe they are not – but the discussion is healthy. Blindly typing some code because the boss says so, that job will be done faster and cheaper by an AI agent.
Boomers gave us the internet. Gen X would like to claim we were the only ones to know an online world and a “real world”. We got our mobile phones as adults, basically. Millennials now claim to be the first online people, giving younger generations a lot of snark. I think Gen X and Millennials will be like drivers from the 1970s, sure you can drive, but you can also adjust a carburettor and replace points in the distributor at a push – because you needed to be able to just to get to work back in the day. Same thing with Gen X and Millennials, we will like someone said – be the only generations that can rotate a PDF. Sorry for the digression – but the points is boomers gave us the Internet. Sure, Tim Berners-Lee gave us the web, but the internet of today is largely the product of some geniuses working for the US military or Big Telephone in the US, some honest-to-goodness hippies at UC Berkeley, a guy from Finland (Swedish speaking minority, mind you) and of course, that British guy from CERN, the aforementioned father-of-the-web – Tim Berners-Lee.
Now, a lot of the innovation that came from American academia came from a group of people that could afford to buy a house, that were tenured professors and otherwise never had to worry about money. They were all way up on the Maslow hierarchy of needs.
Their main concern was evil corporations writing sinister things in code and taking people’s freedoms away by stealth. Much better to make the source free for everyone to see, because now that Big Telephone has invented the high level language C, everybody can read code, right? I mean, history has proved their point about evil corporations – and evil government bodies – 100% correct, so – well, fair play.
Freedom versus Trade Secrets
Meanwhile, a lot of software development started happening in various companies around the world. People doing clever things in computer games or in business, where they wanted the algorithm itself to be kept as a trade secret.
Like everywhere else in life, a religious struggle ensued.
The gilded elites in California proclaimed that software should be free. People heard free-as-in-beer, but the gilded elites meant free-as-in-speech.
The free-as-in-speech crowd wants there to exist a corpus of free software that you can use, but not abuse. You are free to use it in any way you like, but you need to contribute back and you cannot distribute it as your own – roughly – there are a couple of different licenses.
The free-as-in-beer crowd heard “free” and thought it was a great idea to use free software to then make money on top of by creating products. Of course this strikes against the core ideal of the free-as-in-speech crowd, so that led to the creation of Open Source, which is more freely available to be exploited commercially – also driving the creation of additional popular licenses.
Then there is of course the third “I just need to pay rent” crowd that writes whatever code the employer asks for, that shamelessly exploit the work that the either free software crowd put out there, only of course abiding by the license agreement, since even SMEs have a legal department.
I am part of the third group obviously, so my description will not be entirely accurate, but this is my blog, roast me in the comments.
The toll
Now – contributing to open source or free software projects require a lot of guts. You submit your code for review by people that probably aren’t peers, but rather your elders. All dynamics that come into play when humans interact over limited bandwidth – i.e. usually text only – come into play, and the amount of toxicity that appear in free software and open source communities is staggering. Also, the more “core” a project is, the more edge cases you have to think about, especially since a lot of older libraries maintain compatibility with obsolete hardware. As far as I can tell best case the vibes are like in a Stack Overflow thread.
Now – with the advent of package managers on various platforms it has become super easy to take on a dependency on a clever piece of code you find on the internet. All of a sudden some student that wrote something over summer gets pinged with Github issues in the middle of the night because some bank’s deployments are failing and a couple of people are afraid of getting fired. No compensation to the maintainer of course.
The maintainer’s job is thankless and never-ending.
At some point a popular open source project will negatively affect the maintainer’s life, and they need to start making hard choices about whether or not to continue. If they dare to choose to segregate their library into free versus paid versions, i.e. partially closing the source, all three groups of people rage in protest.
By the way – citing Linus Torvalds as a success story to counter the greater narrative is like saying buying lottery tickets is a valid strategy to put food on the table because a few people do win.
The maintainer of core-js had to move home to Russia to afford to live. The whole internet hinged upon his package, but he had such a small income he had to take his chances getting drafted into a war, because if he wanted to generate enough money to afford to eat in a Western country the whole internet told him to do one.
Because of the interpersonal dynamics that sometimes crop up in these communities, a lot of effort is spent making welcoming spaces for beginners, such as creating issues that are classified as “great for beginners” because the interdependencies are contained and thus the blast radius is small, so you can allow a beginner to get their feet wet, open a dodgy PR and be guided through making it perfect. Obviously this mentorship time also comes out of the maintainer’s limited actual hours of life, so this is yet another thing that burdens the maintainers.
Maybe you saw the story of an AI agent that supposedly autonomously grabbed a ticket from an open source project tagged “great for beginners”, implemented and submitted a PR and then wrote a slanderous blog post roasting the maintainer by name when the ticket was rejected specifically because the ticket was there for humans to learn and contribute. I doubt the story is that clear-cut and there was no human in the loop there, but if not- the AI has fully adopted the toxicity of an entitled user of open source software. Yay, I guess.
The problematic volume of entitled users that the introduction of package managers generated seems to have been multiplied yet again, when open source projects that could rely on support contracts to pay employees to maintain the open codebase all of a sudden are starved of their income as AI agents steal their lunch. If your framework becomes the favourite framework proposed by Open AI or Anthropic and the various agents that use their models, you will get a gigantic user base over night, but no revenue. If you put a price on your framework, you lose the entire user base. Fundamentally – pick your poison.
The proposal
Just like when websites started to put ads everywhere in ever more elaborate configurations so that they could pay “journalists” to create listicles – I started thinking… There must be a better way. Surely, we should be able to create conditions where journalists can eat without infecting my computer with malware.
Yes, I am pro paywall. If a “newspaper” produce great writing, I pay a subscription – like in the olden days, hoping that there is collectively enough money for them to avoid chasing clicks Buzzfeed-style.
I therefore think, that conversely, we need to figure out a way where there are some coins falling into the developer’s wallet when they publish some code that others end up using. Perhaps a subscription to an entire catalog of libraries like photo library subscriptions or the music libraries YouTubers use to safely put music in videos. Subscription based on allowance of access (so that a giant company pays more than one dude in his attic), like an app store.
That way, if your AI agent starts loving a certain library, that maintainer will be able to take days off work to review PRs, or sell it to someone who has the time (because the expected revenue is possible to estimate, and it is greater than 0) – and the liability for any rogue takeovers (like the axios hack) can stay with the app store, their “cut” can pay the necessary liability insurance.
Time’s Arrow. Energy can only change phase in one direction without needing physical work. I.e a cup of coffee cools down when left in room temperature. You will need to actually heat it up, perform work, for the cup to be hotter. A tidy room left alone will get messy until you apply work to organise it, et cetera.
I think we see the same thing happening more broadly in society. Like the iPhone killing the luxury phone market. Perhaps you do not recall it, but there briefly existed niche providers selling phones and laptops covered in gold and diamonds, aimed at the Gucci customer base to ostentatiously signal wealth, but that market is dead, and the flashiest you can get is an iPhone Pro Max Mega Ultra or whatever the latest is – which are far cheaper than those old blinged-out flip phones were.
Music was immensely valuable back in the day, and of course – a proper experience like being in the room with a chamber orchestra is pretty full on, no CD or MP3 can ever compete. Even the much less refined form of music, getting a couple of lads in a room and attempting to play an old song at roughly the same time is an incredibly rewarding experience.
Now people stream all kinds of music with ease. You start getting things like “functional” music for concentration, workouts or relaxation. Music that people don’t really care about, it’s like morning TV was, just a bit of pleasant noise in the background that fulfils a specific purpose.
Just like with the iPhone 1 – it being “good enough” for a lot of people, destroyed an entire market – I think this is what is happening with music, video, and software with the advent of AI.
Why get a tool for something when you can just tell Cursor to sort you out? It’ll push back and argue cost of maintenance versus a more high-level script, but it’ll write things for you. Python, Rust or something else? You decide. Whatever you say, the sycophantic AI will tell you it was a brilliant choice. If you have a well defined problem space and clear acceptance criteria, you can create code you only have passive knowledge of – i.e. can read and understand, but not a solid awareness of how to construct idiomatic code in it – without the perhaps hundreds of hours of googling for documentation you used to have to go through. For a singular problem ( e.g. “I need to test a development website for 123 specific scenarios listed in an excel file…”) it makes perfect sense, and after you’ve validated your results, you can keep the evidence and throw the code away.
Entropy.
Things devalue as tooling becomes more competent. Like people don’t need to know how to adjust carburettors anymore just to own a car.
The lowest bar gets higher, but also the highest bar gets lower.
Us old folks reminisce about the olden days when you would hit a button and the corresponding character would appear on the screen immediately. A lot of old custom built ERP or POS software would be impenetrable for a new user with incredibly cluttered text based user interface BUT there would be no latency. You cannot get that snappiness anymore, despite modern computers being several orders of magnitude faster.
Hardware
Back in the day, the keyboard was attached to a dedicated DIN port, most definitely not plug and play. The processor would be yanked out of whatever it was doing to tend to keyboard input whenever signals came in on the serial port. Today there is a Universal Serial Bus that contains a lot more distributed decision making and plug & play. Devices announce their presence, and there is a whole ceremony to ensure that the correct drivers are in place, the processor gets to deal with incoming traffic when it chooses to. Great for the overall smoothness of the experience using the computer, but for keyboards you usually get latency. Gaming mouse and keyboard manufacturers work to reduce this, but fundamentally USB means latency.
Operating systems
Back in the day, the software would run on a DOS machine, an OS so lightweight it barely qualified as an operating system, you mostly talk directly to the hardware as an application developer. These days, your process is waiting to be told by the OS when a user has typed text in your app. The process not only sounds complicated, it IS complex. The operating system will first have to make sure the correct thread is currently running, and then deliver the keystroke.
Applications
In the before times, the application would receive the keystroke, confer with its internal state (am I in a menu or am I in a text editor? Was this a special key, like a function key or similar?) and then immediately render the character on the screen if appropriate.
Today, all kinds of things can happen. Are you editing a word document on Sharepoint – or using Google Docs? In that case your keystrokes go to the cloud first. Bonus points, no save button, but also – massive legacy on an order of magnitude greater than that USB malarkey. Also – either app, sometimes even text boxes in the OS – will spell check words if you press the space bar or stop typing for a bit.
As developers we are aware of intellisense, I.e. predictive text for developers. Yet another order of magnitude of latency, because the developer tool has to sort of recompile large parts of the app underneath your fingers. Even though the app tries to be clever about doing as little work as possible, you can imagine how insanely much more work that is, compared to writing a character to a screen in text mode.
A possibly redeeming factor is that while personal computers in the olden days had a single thread of execution, literally doing one thing at the time, in the modern world these distractions can happen literally simultaneously to your typing, so the latency is not quite as bad as it could have been.
What to do?
I suggest you go ask the people most keenly interested in low latency, I.e. gaming. There will be tests online you can peruse before picking a keyboard. Your operating system may offer you tweaks to prioritise UI responsiveness in its scheduling, and you can switch off interactive features. You can run Linux in text mode with tmux. Or, you can just accept that the days of snappy UIs are over and let the computer go off and do its thing like an otherwise faithful dog that only listens to a subset of commands.
I know in the labour market of today, suggesting someone pick up programming from scratch is akin to suggesting someone dedicate their life to being a cooper. Sure, in very specific places, that is a very sought after skill that will earn you a good living, but compared to its heyday the labour market has shrunk considerably.
Getting into the biz
How do people get into this business? As with I suspect most things, there has to be a lot of initial positive reinforcement. Like – you do not get to be a great athlete without several thousands of hours of effort rain or shine, whether you enjoy it or not – but the reason some people with “talent” end up succeeding is that they have enough early success to catch the “bug” and stick at it when things inevitably get difficult and sacrifices have to be made.
I think the same applies here, but beyond e-sports and streamer fame, it has always been more of an internal motivation, the feeling of “I’m a genius!” when you acquire new knowledge and see things working. It used to help to have literally nothing else going on in life that was more rewarding, because just like that fleeting sensation of understanding the very fibre of the universe, there is also the catastrophic feeling of being a fraud and the worst person in the world once you stumble upon something beyond your understanding, so if you had anything else to occupy yourself with, the temptation to just chuck it in must be incredibly strong.
Until recently – software development was seen as a fairly secure career choice, so people has a financial motivator to get into it – but still, anecdotally it seems people many times got into software development by accident. Had to edit a web page, and discovered javascript and PHP – or , had to do programming as part of some lab at university and quite enjoyed it et c. Some were trying to become real engineers but had to settle for software development, some were actuaries in insurance and ended up programming python for a living.
I worry that as the economic prospects of getting into the industry as a junior developer is eaten up by AI budgets, we will see a drop-off of those that accidentally end up in software development and we will be left with only the ones with what we could kindly call a “calling”, or what I would say “has no other marketable skills” like back in my day.
Dwindling power of coercion
Microsoft of course is the enemy of any right thinking 1337 h4xx0r, but there has been quite a while where if you wanted a Good Job, learning .NET and working for a large corporation on a Lenovo Thinkpad was the IT equivalent of working at a factory in the 1960s. Not super joyous but, a Good Job. You learned .NET 4.5 and you pretended to like it. WCF, BizTalk and all. The economic power was unrelenting.
Then the crazy web 2.0 happened and the cool kids were using Ruby on Rails. If you wanted to start using ruby, it was super easy. It was like back in my day, but instead of typing ABC80 basic – see below- they used the read evaluate print loop in ruby. Super friendly way of feeling like a genius and gradually increase the level of difficulty.
10 PRINT "BAD WORD ";
20 GOTO 10
Meanwhile legacy Java and C# were very verbose, you had to explain things like static, class, void, include, static not to mention braces and semicolons et c to people before they could create a loop of a bad word filling the terminal.
People would rather still learn PHP or Ruby, because they saw no value in those old stodgy languages.
Oracle were too busy being in court suing people to notice, but on the JVM there were other attempts at creating some things less verbose – Scala and eventually Kotlin happened.
Eventually Microsoft noticed what was going on, and as the cool kids jumped ship from Ruby onto NodeJS, Microsoft were determined to not miss the boat this time, so they threw away the .NET Framework, or “threw away” – as much as Microsoft have ever broken with legacy, but still fairly backward compatible, and started from scratch with .NET Core and a renewed focus on performance and lowered barriers to entry.
The pressure really came as data science folks rediscovered Python. It too has super low barrier to entry, except there is a pipeline to data science, and Microsoft really failed to break into that market due to the continuous mismanagent of F#, except they attacked it form the Azure side and get the money that way – depite people writing python.
Their new ASP.NET Core web stack stole borrowed concepts like minimal API from Sinatra and Nancy, and they introduced top level statements to allow people to immediately get the satisfaction of creating a script that loops and emits rude words using only two lines of code
while (true) Console.Write("Rude word ");
But still, the canonical way of writing this code was to install Visual Studio and create a New Project – Console App, and when you save that to disk you have a whole bunch of extra nonsense there (a csproj file, a bunch of editor metadata stuff that you do not want to have to explain to a n00b et cetera), which is not beginner friendly enough.
This past Wednesday, Microsoft introduced .NET 10 and Visual Studio 2026. In it, they have introduced file based apps, where you can write one file that can reference NuGet packages or other C# projects, import namespaces and declare build-time variables inline. It seems like an evolution of scriptcs, but slightly more complete. You can now give people a link to the SDK installer and then give them this to put in a file called file.cs:
#!/usr/bin/env dotnet run
while (true)
Console.Write("rude word ");
Then, like in most programming tutorials out there you can tell them to do sudo chmod +x file.cs if they are running a unix like OS. In that case, the final step is ./file.cs and your rude word will fill the screen…
If you are safely on Windows, or if you don’t feel comfortable with chmod, you can just type dotnet file.cs and see the screen fill with creativity.
Conclusion
Is the bar low enough?
Well, if they are competing with PHP, yes, you can give half a page length’s instruction and get people going with C#, which is roughly what it takes to get going with any other language on Linux or Mac and definitely easier than setting up PHP. The difficulty with C# and with Python as well is that they are old. Googling will give you C# constructs from ages ago that may not translate well to a file based project world. Googling for help with Python will give you a mix of python 2 and python 3, and with python it is really hard to know what is a pip thing and what is an elaborate hoax due to the naming standards. The conclusion is therefore, dotnet is now in the same ballpark as the other ones in terms of complexity, but it depends on what resources remain available. Python has a whole gigantic world out there of “how to get started from 0”, whilst C# has a legacy of really bad code from the ASP.NET WebForms days. Microsoft have historically been excellent at providing documentation, so we shall see if their MVP/RD network flood the market with intro pages.
At the same time, Microsoft is going through yet another upheaval with Windows 10 going out of support and Microsoft tightening the noose around needing to have a Microsoft Account to run Windows 11, and at the same time Steam have released the Steam Console running Windows software on Linux, meaning people will have less forced exposure to Windows even to game, whilst Google own the school market. Microsoft will still have corporate environments that are locked to Windows for a while longer, but they are far from the situation they used to be in.
I don’t know if C# is now easy enough to adopt that people that are curious about learning programming would install it over anything else on their mac or linux box.
High or low bar, should people even learn to code?
Yes, some people are going to have to learn programming in the future. AGI is not happening, and new models can only train on what is out there. Today’s generative AI can do loads of things, but in order to develop the necessary skills to leverage it responsibly, you need to be familiar with all the baggage underneath or else you risk releasing software that is incredibly insecure or that will destroy customer data. Like Bjarne Stoustrup said “C makes it easy to shoot yourself in the foot; C++ makes it harder, but when you do it blows your whole leg off” – this can apply to AI generated code as well.
In popular parlance you have two categories of code: your own, freshly written code, which is the best code, code that never will be problematic – and then there is legacy code, which is someone else’s code, untested, undocumented and awful. Code gradually goes from good to legacy in some ways that appear mystical, and in the end you change jobs or they bring in new guys to do a Great Rewrite with mixed results.
So, to paraphrase Baldrick from Blackadder Goes Forth: “The way I see it, these days there’s a [legacy code mess], right? and, ages ago, there wasn’t a [legacy code mess], right? So, there must have been a moment when there not being a [legacy code mess] went away, right? and there being a [legacy code mess] came along. So, what I want to know is: How did we get from the one case of affairs to the other case of affairs”
The hungry ostrich
Why does code start to deteriorate? What precipitates the degradation that eventually leads to terminal decline? What is the first bubble of rust appearing by the wheel arches? This is hard to generally state, but the causes I have personally seen over the years boil down to being prevented from making changes in a defensible amount of time.
Coupling via schema – explicit or not
E.g. it could be that you have another system accessing your storage directly. Doesn’t matter if you are using schemaless storage or not, as long as two different codebases need to make sense of the same data, you have a schema whether you admit it or not, at some point those systems will need to coordinate their changes to not break functionality.
Fundamentally – as soon as you start going “neah, I won’t remove/rename/change type of that old column because I have no idea who still uses it” you are in trouble. Each storage must have one service in front of it that owns it, so that it can safely manage schema migrations, and anyone wanting to access that data needs to use a well defined API to do so. The service maintainers can thereafter be held responsible to maintain this API in perpetuity, and easily so since the dependency is explicit and documented. If the other service just queried the storage directly, the maintainer is completely unaware (yes, this goes for BI teams as well).
Barnacles settling
If every feature request leads to functions and classes growing as new code is added like barnacles without regular refactoring to more effective patterns, the code gradually gets harder to change. This is commonly a side-effect of high turnover or outsourcing. The developers do not feel empowered to make structural changes, or perhaps have not had enough time to get acquainted with the architecture as it was intended at some point. Make sure that whomever maintains your legacy code is fully aware of their responsibility to refactor as they go along.
Test after
When interviewing engineers it is very common that they say they “practice TDD, but…”, meaning they test after. At least to me the difference in test quality is obviously different if I write the tests first versus if I get into the zone and write the feature first and then try to retrofit tests afterwards. Hint: there is usually a lot less mocking if you test first. As the tests get more complex, adding new code to a class under tests gets harder, and if the developer does not feel empowered to refactor first, the tests are likely to not cover the added functionality properly , so perhaps a complex integration test is modified to validate the new code, maybe the change is tested manually…
Failure to accept Conway’s law
The reason people got hyped about micro services was the idea that you could deploy individual features independently of the rest of the organisation and the rest of the code. This is lovely, as long as you do it right. You can also go too granular, but in my experience that rarely happens. The problem that does happen is that separate teams have interests in the same code and modify the same bits, and releases can’t go out without a lot of coordination. If you also have poor automation test coverage you will get a manual verification burden that further slows down releases. At your earliest convenience you must spend time restructuring your code or at least the ownership of it so that teams afully own all aspects of the thing they are responsible for and they can release code independently, and with any remaining cross-team dependencies made explicit and automatically verifiable.
Casual attitude towards breaking changes
If you have a monolith that is providing core features to your estate, and you have a publicly accessible API Operation, assume it is being used by somebody. Basically, if you must change its required parameters or its output, create a new versioned endpoint or one by a different name. Does this make things less messy? No, but at least you don’t break a consumer you don’t know about. Tech leads will hope that you message around to try and identify who uses it and coordinate a good outcome, but historically that seems too much to ask. We are only human after all.
Until you have PACT tests for everything, and solid coverage, never break a public method.
Outside of support horizon
Initially it does not seem that bad to be stuck with a slightly unsupported version of a library, but as time moves on, all of a sudden you get stuck for a week with a zero day bug that you can’t patch because three other libraries are out of date and contain breaking changes. It is much better if you are ready to make changes as you go along. One breaking change usually lets you have options, but when you are already exposed with a potential security breach, you have to make bad decisions due to lack of time.
Complex releases
Finally, it is worth mentioning that you want to avoid manual steps in your releases. Today there is really no excuse for making a release more complex than one button click. Ideally abstract away configuration so that there is no file.prod.config template that is separate from file.uat.config, or else that prod template file is almost guaranteed to break the release, much like the grille was the only thing rusting on the Rover 400 that was almost completely a Honda, (except for the grille).
Stopping Princip
So how do we avoid the decline, the rot? As with shifting quality and security left, it is much cheaper to address these problems the earlier you spot them, so if you find yourself in any of the situations above, address them with haste.
Avoid engaging “maintenance developers”, their remit may explicitly mean they cannot do major refactoring even when necessary
Keep assigning resources to keep dependencies updated. Use SAST to validate that your dependencies are not vulnerable.
Disallow and remove integration-by-database at any cost. This is hard to fix, but worth it. This alone solves 90% of niggling small problems you are continuously having as you can fix your data structure to fit your current problems rather than the ones you had 15 years ago. If you cannot create a true data platform for reporting data, at least define agreed views/ indexes that can act like an interface for external consumers. That way you have a layer of abstraction between external consumers and yourself and stay free to refactor as long as you make sure that the views still work.
Make dependencies explicit. Ideally PACT tests, but if not that, at least integration tests. This way you avoid needing shared integration environments where teams are shocked to first find out that their changes they have been working on for two weeks breaks some other piece of software they didn’t know existed.
What are cookies? Why do they exist? Why on earth would I ever NOT want to accept delicious cookies? What is statelessness? All that and more in this treatise on cookies and privacy.
Requests
The original humble website (info.cern.ch) looked very different from the sites that currently power banking, commerce and even interactions with government. Fundamentally a web server is a program that lets you create, request, update, or delete resources. It tells you some information about the content, what type it is, how big it is, when it was last modified, if you are supposed to cache it or not, among other things. These metadata are returned as headers, i.e. bits of content before the main content, so to speak.
To over-simplify the process, the client, e.g. the browser, simply breaks down the address in the address bar into the scheme – usually http or https, the host name – info.cern.ch, and the path – /. If the scheme is http and not port number was explicitly given, the browser will contact info.cern.ch on port 80, and then send the command GET /. The browser will send information in headers, such as User-Agent, i.e. it tells the web server which browser it is, and it can tell you referrer as well, i.e. which website linked to this one. These headers are sent by the browser, but they are not mandatory, and any low level http client can set their own referrer and user-agent headers, so it is important to realise that these headers are not guaranteed to be correct. The server too will offer up information in headers. Sometimes the server will – as headers in addition to the content it serves you – announce what type of web server it is (software and platform) which is something you should ideally disable, because that information is only helpful for targeting malware, with no real valid use case.
Why this technical mumbo jumbo? Well, the thing you didn’t see in the above avalanche of tech stuff was the browser authenticating the user to the server in any way. The browser rocked up, asked for something and received it, at no point was a session created, or credentials exchanged. Now, info.cern.ch is a very simple page, but it does have a favicon, i.e. the little picture that adorns the top left of the browser tab, so when the page is requested, it actually makes two calls to the Swiss web server. One for the HTML content, and one for the picture. Now with modern HTTP protocol versions this is changing somewhat, but let’s ignore that for now, the point is – the server does not keep session state, it does not know if you are the same browser refreshing the page, or if you are someone completely new that requests the page and the favicon.
There was no mechanism to “log in”, to start a session, there was no way to know if it was the same user coming back that you already knew because no such facility existed within the protocol. From fairly early on you could have the server return status code 401 to say “you need to log in”, and there was a provision for the browser to then supply some credentials using a header called Authorize, but you had to supply that header for every request or else it wouldn’t work. This is how APIs work still, each request is a new world, you authenticate with every call.
The solution, the way to log into a website, to exchange credentials once and then create a session that knows who you are whilst you are on a website, was using cookies.
Taking the biscuit
What is a cookie? Well, it is a small file. It is stored by the browser somewhere in the user’s local files.
The server returns a header called Set-Cookie where the server tells the browser to remember some data, basically name & value and possibly a domain.
Once that has happened. there is a gentleman’s agreement that that browser will always send along those cookies when a subsequent call is made to that same server, and the normal flow is that the server will set a cookie like “cool-session-id= a234f32d” and the server will then upon subsequent requests read the cookie cool-session-id and know which session this request belongs to “a234f32d, ah long time no see – carry on”. Some cookies live for a very long time “don’t ask again”, and some, the session ones, live for 5 minutes or similar. When the cookies expire, the browser will no longer send them along with requests to the server, and you willl have to log in again, or similar.
How the cookie crumbles
What could possibly go wrong, these cookies seem perfect with no downsides whatsoever? Yes, and no. A HTML page, a hypertext document, contains text, images, and links. Usually you build up a web page using text content and images that you host on your own machine, so the browser keeps talking to the same server to get all the necessary content, but sometimes you use content from somewhere else, like under-construction.gif that was popular back in my day. That means that the server where under-construction.gif is hosted can set cookies as well, because the call to its server to download that picture is the same type of thing that the call to my server where the HTML lives, those calls work the same way. If the person hosting under-construction.gif wanted to, they could use those cookies to figure out which pages each person visits. If it was 1995. then under-construction.gif could be referenced from 1000 websites, and by setting cookies, the host of under-construction.gif could start keeping a list of the times when the same cookie showed up on requests for under-construction.gif from different websites. The combination of Referrer header and the cookie set in each browser would allow interesting statistics to be kept.
Let’s say this isn’t under-construction.gif, but rather a Paypal Donate button, a Ko-Fi button, a Facebook button or a Google script manager, and you start seeing the problem. These third party cookies are sometimes called tracking cookies, because, well – that’s what they do.
Why the sweet tooth?
Why do people allow content on their website that they know will track their users? Well, for the plebs, like this blog here, I suspect the main thing is the site creator cannot be bothered to clear house. You use some pre-built tool, like WordPress, and accept that it will drop cookies like a medieval fairytale, you can’t be arsed to wade in PHP on your spare time to stop the site from doing so. Then there’s the naive greed, like if I add a Paypal Donate button, or an Amazon affiliate link, I could make enough money to buy several 4 packs of coke zero, infinite money glitch !!1one.
For companies and commercial websites, I am fairly convinced that Google Analytics is the biggest culprit. Even if you have zero interest in monetising the website itself, and you never intend to place ads at any time, Google Analytics is a popular tool to track how users use your application. You can tag up buttons with identifiers and see what features either are not discovered or are too complex, i.e. users abandon multi-step processes half way through. From a product design perspective these seem like useful signals, but form a pure engineering perspective it allows you to build realistic monitoring and performance tests because you have reasonably accurate evidence of how real world users use your website. The noble goal of making the world a better place aside, the fact is that you are still using that third party cookie from Google, and they use it for whatever purposes they have, the only thing is you get to use some of that data too.
Achieving the same level of insight about how your users use your app by using an analytics tool you built in-house would take a herculean effort, and for most companies, that cost would not be defensible. You see a similar problem happen after Sales develops a load-bearing excel template, and you realise that building a line-of-business web app to replace that template would be astronomically expensive and still miss out on some features Excel has built-in.
Consent is fundamental
As you can tell the technical difference between a marketing cookie and a cookie used for improving the app or monitoring quality is nonexistent. It is all about intent. The General Data Protection Regulation was an attempt at safeguarding people’s data by requiring companies to be upfront about the intent and scope of the information it keeps, and to keep them accountable in case they suffer data breaches. One of the most visible aspects of the regulation is the cookie consent popup that quckly became ubiquitous over the whole of the internet.
Now, this quickly became an industry of its own where companies buy third party javascript apps that allow you to switch off optional cookies and present comprehensive descriptions about what the purpose is around each cookie. I personally think it is a bit of a racket preying on the internal fear of the Compliance department in large corporations, but still – these apps do provide a service. The only problem is that you as a site maintainer gets to define if a cookie is mandatory or not. You can designate a tracking cookie as required, and it will basically be up to the courts to decide if you are in violation. Some sites like spammy news aggregators do this upfront, they designate their tracking cookies as mandatory.
Conclusion
So, are cookies always harmful, or can you indulge in the odd one now and then without feeling bad? The simple answer is, it depends. Every time you approve of a third party cookie, know that you are traced across websites. You may not mind, because it’s your favourite oligopoly Apple, or you might mind because it’s ads.doubleclick.net – it is up to you. And if you are building a website with a limited budget that does not include also building a bespoke analytics platform, you may hold your nose and add google analytics, knowing full well that a lot of people will block that cookie anyway, reducing your feedback in potentially statistically significant ways. Fundamentally it is about choice. At least this way you can stay informed.
All work in software engineering is about abstractions.
Abstractions
All models are wrong, but some are useful
George Box
It began with Assembly language, when people were tired of writing large-for-its-time programs in raw binary instructions, they made a language that basically mapped each binary instruction to a text value, and then there was an app that would translate that to raw binary and print punch cards. Not a huge abstraction, but it started there. Then came high level languages and off we went. Now we can conjure virtual hardware out of thin air with regular programming languages.
The magic of abstractions, it really gives you an amazing leverage, but at the same time you sacrifice actual knowledge of the implementation details, meaning you often get exposed to obscure errors that you either have no idea what they mean, or even worse- understand exactly what’s wrong but you don’t have access to make that change because the source is just a machine translated piece of Go, and there is no way to fix the translated C# directly, just to take one example.
Granularity and collaboration across an organisation
Abstractions in code
Starting small
Most systems start small, solving a specific problem. This is done well, and the requirements grow, people begin to understand what is possible and features accrue. A monolith is built, and it is useful. For a while things will be excellent and features will be added at great speed, and developers might be added along the way.
A complex system that works is invariably found to have evolved from a simple system that worked
John Gall
Things take a turn
Usually, at some point some things go wrong – or auditors get involved because regulatory compliance – and you prevent developers from deploying to production, hiring gate keepers to protect the company from the developers. In the olden days – hopefully not anymore – you hire testers to do manual testing to cover a shortfall in automated testing. Now you have a couple of hand-offs within the team, meaning people write code, give it to testers who find bugs, work goes the wrong way – backwards – for developers to clean up their own mess and to try again. Eventually something will be available to release, and the gate keepers will grudgingly allow a change to happen, at some point.
This leads to a slow down in the feature factory, some old design choices may cause problems that further slow down the pace of change, or – if you’re lucky – you just have too many developers in one team, and you somehow have to split them up in different teams, which means comms deteriorate and collaborating in one codebase becomes even harder. With the existing change prevention, struggles with quality and now poor cross-team communication, something has to be done to clear a path so that the two groups of people can collaborate effectively.
Separation of concerns
So what do we do? Well, every change needs to be covered by some kind of automated test, if only to at first guarantee that you aren’t making things worse. This way you can now refactor the codebase to a point where the two groups can have separate responsibilities, and collaborate over well defined API boundaries, for instance. Separate deployable units, so that teams are free to deploy according to their own schedule.
If we can get better collaboration with early test designs and front-load test automation, and befriend the ops gatekeepers to wire in monitoring so that teams are fully wired in to how their products behave in the live environment, we would be close to optimum.
Unfortunately – this is very difficult. Taking a pile of software and making sense of it, deciding how to split it up between teams, gradually separating out features can be too daunting to really get started. You don’t want to break anything, and if you – as many are won’t to do, especially if you are new in an organisation – decide to start over from scratch, you may run into one or more of the problems that occur when attempting a rewrite. One example being where you end up in a competition against a moving target. The same team has to own a feature in both the old and the new codebase, in that case, to stop that competition. For some companies it is simply worth the risk, they are aware they are wasting enormous sums of money, but they still accept the cost. You would have to be very brave.
Abstractions in Infrastructure
From FTP-from-within-the-editor to Cloud native IaC
When software is being deployed – and I am ignoring native apps now, largely, and focusing on web applications and APIs- there are a number of things that are actually happening that are at this point completely obscured by layers of abstraction.
The metal
The hardware needs to exist. This used to be a very physical thing, a brand new HP ProLiant howling in the corner of the office onto which you installed a server OS and set up networking so that you could deploy software on it, before plugging it into a rack somewhere, probably a cupboard – hopefully with cooling and UPS. Then VM hosts became a thing, so you provisioned apps using VMWare or similar and got to be surprised at how expensive enterprise storage is per GB compared to commodity hardware. This could be done via VMWare CLI, but most likely an ops person pointed and clicked.
Deploying software
Once the VM was provisioned, things like Ansible, Chef and Puppet began to become a thing, abstracting away the stopping of websites, the copying of zip files, the unzipping, the rewriting configuration and the restarting of the web app into a neat little script. Already here you are seeing problems where “normal” problems, like a file being locked by a running process, show up as a very cryptic error message that the developer might not understand. You start to see cargo cult where people blindly copy things from one app to another because you think two services are the same, and people don’t understand the details. Most of the time that’s fine, but it can also be problematic with a bit of bad luck.
Somebody else’s computer
Then cloud came, and all of a sudden you did not need to buy a server up front and instead rent as much server as you need. Initially, all you had was VMs, so your Chef/Puppet/Ansible worked pretty much the same as before, and each cloud provider offered a different was of provisioning virtual hardware before you came to the point where the software deployment mechanism came into play. More abstractions to fundamentally do the same thing. Harder to analyse any failures, you some times have to dig out a virtual console to just see why/how an app is failing because it’s not even writing logs. Abstractions may exist, but they often leak.
Works on my machine-as-a-service
Just like the London Pool and the Docklands were rendered derelict by containerisation, a lot of people’s accumulated skills in Chef and Ansible have been rendered obsolete as app deployments have become smaller, each app simply unzipped on top of a brand new Linux OS sprinkled with some configuration answer, and then have the image pushed to a registry somewhere. On one hand, it’s very easy. If you can build the image and run the container locally, it will work in the cloud (provided the correct access is provisioned, but at least AWS offer a fake service that let’s you dry run the app on your own machine and test various role assignments to make sure IAM is also correctly set up. On the other hand, somehow the “metal” is locked away even further and you cannot really access a console anymore, just a focused log viewer that let’s you see only events related to your ECS task, for instance.
Abstractions in Organisations
The above tales of ops vs test vs dev illustrates the problem of structuring an organisation incorrectly. If you structure it per function you get warring tribes and very little progress because one team doesn’t want any change at all in order to maintain stability, the other one gets held responsible for every problem customers encounter and the third one just wants to add features. If you structured the organisation for business outcome, everyone would be on the same team working towards the same goals with different skill sets, so the way you think of the boxes in an org chart can have a massive impact on real world performance.
There are no solutions, only trade-offs, so consider the effects of sprinkling people of various background across the organisation, if instead of being kept in the cellar as usual you start proliferating your developers among the general population of the organisation, how do you ensure that every team follows the agreed best practices, that no corners are cut even when a non-technical manager is demanding answers. How do you manage performance of developers you have to go out of your way to see? I argue such things are solvable problems, but do ask your doctor if reverse Conway is right for you.
Conclusion
What is a good abstraction?
Coupling vs Cohesion
If a team can do all of their day-to-day work without waiting for another team to deliver something or approve something, if there are no hand-offs, then they have good cohesion. All the things needed are to hand. If the rest of the organisation understands what this team does and there is no confusion about which team to go to with this type of work, then you have high cohesion. It is a good thing.
If however, one team constantly is worrying about what another team is doing, where certain tickets are in their sprint in order to schedule their own work, then you have high coupling and time is wasted. Some work has to be moved between teams or the interface between the teams has to be made more explicit in order to reduce this interdependency.
In Infrastructure, you want the virtual resources associated with one application to be managed within the same repository/area to offer locality and ease of change for the team.
Single Responsibility Principle
While dangerous to over-apply within software development (you get more coupling than cohesion if you are too zealous), this principle is generally useful within architecture and infrastructure.
Originally meaning that one class / method should only do one thing – an extrapolation of the UNIX principles – it can more generally be said to mean that on that layer of abstraction, a team, infrastructure pipe, app, program, class […] should have one responsibility. This usually mean a couple of things happen, but they conceptually belong together. They have the same reason to change.
What – if any – pitfalls exist ?
The major weakness of most abstractions is when they fall apart, when they leak. Not having access to a physical computer is fine, as long as the deployment pipeline is working, as long as the observability is wired up correctly, but when it falls down, you still need to be able to see console output, you need to understand how networking works, to some extent, you need to understand what obscure operating system errors mean. Basically when things go really wrong you are needed to have already learned to run that app in that operating system before, so you recognise the error messages and have some troubleshooting steps memorised. So although we try and save our colleagues from the cognitive load of having to know everything we were forced to learn over the decades, to spare them the heartache, they still need to know. All of it. So yes, the danger with the proliferation of layers of abstraction is to pick the correct ones, and to try and keep the total bundle of layers as lean as possible because otherwise someone will want to simplify or clarify these abstractions by adding another layer on top, and the cycle begins again.
The results of the latest StackOverflow Developer Survey just came out, showing – among other interesting things – that Windows is dying as a developer OS. Not one to abandon ship any time soon I’d still like to offer up some suggestions.
TL;DR
Make the commandline deterministic.
Copying files across the network cannot be a lottery.
Stop rebooting UI-frameworks
Make F# the flagship language
Back in the day, Microsoft through VB and Visual C++ overcame some of the hurdles of developing software for Windows – then the only, effectively, desktop OS in the enterprise. Developers, and their managers, rallied behind these products and several million kilometres of code was written over a couple of decades.
The hurdles that were overcome were related to the boilerplate needed to register window classes, creating a window and responding to the basic window messages required to show the window in Windows and have the program behave as expected vis-a-vis the expectations a Windows user might have. Nowhere in VB6 samples was anybody discussing how to write tests or how, really, to write good code. In fact, sample code, simplified on purpose to only showcase one feature at a time, would not contain any distractions such as test code.
When Classic ASP was created, a lot of this philosophy came a cross to the web, and Microsoft managed to create something as horrible as PHP, but with less features, telling a bunch of people that it’s OK to be a cowboy.
When the .NET framework was created as a response to Java, a lot of VB6 and ASP.NET programmers came across and I think Microsoft started to see what they had created. Things like Patterns & Practices came out and the certification programmes were taking software design and testing into consideration. Sadly, however, they tended to give poor advice that was only marginally better than what was out there in the wild.
Missed the boat on civilised software development
It was a shock to the system when the ALT.NET movement came out and started to bring in things that were completely mainstream in the Java community but almost esoteric in .NET. Continuous integration – unit testing – TDD – DDD. Microsoft tried to keep up by creating TFS that apart from source code version in had ALM tools to manage bugs and features as well as a built-in build server but it became clear to more and more developers that Microsoft really didn’t understand the whole thing about testing first or how lean software development needs to happen.
While Apple had used their iron fist to force people to dump Mac OS for the completely different, Unix-based operating system OS X (with large bits of NextStep brought across, like the API and InterfaceBuilder) – Microsoft were considering their enterprise customers and never made a clean break with Gdi32. Longhorn was supposed to solve everything, making WPF native and super fast, obsoleting the old BitBlt malarkey and instead ushering in a brighter future.
As you are probably aware, this never happened. .NET code in the kernel was a horrible idea and the OS division banned .NET from anything ever being shipped with Windows, salvaged whatever they could duct tape together – and the result of that was Vista. Yes, .NET was banned from Windows and stayed banned up until Powershell became mainstream a long, long time later. Now, with Universal Windows Apps, a potentially viable combo of C++ code and vector UI has finally been introduced, but since it is the fifth complete UI stack reboot since Longhorn folded, it is probably too little too late and too many previously enthusiastic Silverlight or WPF people have already fallen by the wayside. Oh and many of the new APIs are still really hard to write tests around, and it is easy finding yourself in a situation where you need to install Visual Studio and some SDK on a build server, because the dependency relies on the Registry or the GAC rather than things that come with the source.
Automation
As Jeffrey Snover mentions in several talks, Windows wasn’t really designed with automation in mind. OLE Automation possibly, but scripting? Nooo. Now, with more grown-up ways of developing software – automation becomes more critical. The Windows world has developed alternate ways of deploying software to end-user machines than work quite well, but for things like automated integration tests and build automation you should still be able to rely on scripting to set things up.
This is where Windows really lets the developer community down. Simple operations in Windows aren’t deterministic. For a large majority of things you call on the command-line – you are the only one responsible for determining if the command ran successfully. The program you called from the command-line may very well have failed despite it returning a 0 exit code. The execution just might not have finished despite the process having ended, so some files may still be locked. For a while, you never know. Oh, and mounting network drives is magic and often fails for no reason.
End result
Some people leave for Mac because everything just works, if you can live with bad security practices and sometimes a long delay before you get some things like Java updates. Some people leave for Linux because if you script everything, you don’t really mind all those times you have to reinstall because thing like a change in screen resolution or a security update killed the OS to the point you can’t log in anymore, you just throw away the partition and rerun the scripts. Also, from a developer standpoint, everything just works, in terms of available tools and frameworks.
What to do about it
If Microsoft wants to keep making developer tools and frameworks, they need to start listening to the developers that engage whenever Microsoft open sources things. They most likely have valuable input into how things are used by your serious users – beyond the tutorials.
Stop spending resources duplicating things already existing for Windows or .NET as that strikes precisely at the enthusiasts that Microsoft needs in order to stop hemorrhaging developers.
What is .NET Core – really? Stop rewriting the same things over and over. At least solve the problems the rewrite was supposed to address first before adding fluff. Also – giving people the ability to work cross-platform means people will, so you are sabotaging yourselves while building some good-will, admittedly.
Most importantly – treat F# like Apple treats Swift. Something like – we don’t hate C# – there is a lot of legacy there but F# is new, trendier and better. F# is far better than Swift and has been used in high spec applications for nine years already. Still Microsoft after years of beta testing still manages to release a JITer that has broken tail call optimisation (a cornerstone of functional runtimes as it lets you do recursion effectively). That is simply UNACCEPTABLE and I would have publicly shamed then fired so many managers for letting that happen. Microsoft needs to take F# seriously – ensure it gets the best possible performance, tooling and templating. It is a golden opportunity to separate professional developers from the morons you find if you google “asp.net login form” or similar.
In other words – there are many simple things Microsoft could do to turn the tide, but Im not sure they will manage, despite the huge strides taken of late. It is also evident that developers hold a grudge for ages.
Having grown up in a society evolved beyond the confines of 7-bit ASCII and lived through the nightmare of codepages as well as cursed the illiterate that have so little to say they can manage with 26 letters in their alphabet, I was pleased when I read Joel Spolsky’s tutorial on Unicode. It was a relief – finally somebody understood.
Years passed and I thought I knew now how to do things right. And then I had to do Windows C in anger and was lost in the jungle of wchar_t and TCHAR and didn’t know where to turn.
Don’t use wchar_t or TCHAR or any of the associated macros. Always assume std::string and char * are UTF-8. Call Wide windows APIs and use boost nowide or similar to widen the characters going in and narrowing them coming out.
Never produce any text that isn’t UTF-8.
Do note however that the observations that Windows would not support true UTF-16 are incorrect as this was fixed before Windows 7.
