Category Archives: Evil

Abstractions, abstractions everywhere

Architecture, DevOps, Evil
X, X everywhere meme template, licensed by imgflip.com

All work in software engineering is about abstractions.

Abstractions

All models are wrong, but some are useful

George Box

It began with Assembly language, when people were tired of writing large-for-its-time programs in raw binary instructions, they made a language that basically mapped each binary instruction to a text value, and then there was an app that would translate that to raw binary and print punch cards. Not a huge abstraction, but it started there. Then came high level languages and off we went. Now we can conjure virtual hardware out of thin air with regular programming languages.

The magic of abstractions, it really gives you an amazing leverage, but at the same time you sacrifice actual knowledge of the implementation details, meaning you often get exposed to obscure errors that you either have no idea what they mean, or even worse- understand exactly what’s wrong but you don’t have access to make that change because the source is just a machine translated piece of Go, and there is no way to fix the translated C# directly, just to take one example.

Granularity and collaboration across an organisation

Abstractions in code

Starting small

Most systems start small, solving a specific problem. This is done well, and the requirements grow, people begin to understand what is possible and features accrue. A monolith is built, and it is useful. For a while things will be excellent and features will be added at great speed, and developers might be added along the way.

A complex system that works is invariably found to have evolved from a simple system that worked

John Gall

Things take a turn

Usually, at some point some things go wrong – or auditors get involved because regulatory compliance – and you prevent developers from deploying to production, hiring gate keepers to protect the company from the developers. In the olden days – hopefully not anymore – you hire testers to do manual testing to cover a shortfall in automated testing. Now you have a couple of hand-offs within the team, meaning people write code, give it to testers who find bugs, work goes the wrong way – backwards – for developers to clean up their own mess and to try again. Eventually something will be available to release, and the gate keepers will grudgingly allow a change to happen, at some point.

This leads to a slow down in the feature factory, some old design choices may cause problems that further slow down the pace of change, or – if you’re lucky – you just have too many developers in one team, and you somehow have to split them up in different teams, which means comms deteriorate and collaborating in one codebase becomes even harder. With the existing change prevention, struggles with quality and now poor cross-team communication, something has to be done to clear a path so that the two groups of people can collaborate effectively.

Separation of concerns

So what do we do? Well, every change needs to be covered by some kind of automated test, if only to at first guarantee that you aren’t making things worse. This way you can now refactor the codebase to a point where the two groups can have separate responsibilities, and collaborate over well defined API boundaries, for instance. Separate deployable units, so that teams are free to deploy according to their own schedule.

If we can get better collaboration with early test designs and front-load test automation, and befriend the ops gatekeepers to wire in monitoring so that teams are fully wired in to how their products behave in the live environment, we would be close to optimum.

Unfortunately – this is very difficult. Taking a pile of software and making sense of it, deciding how to split it up between teams, gradually separating out features can be too daunting to really get started. You don’t want to break anything, and if you – as many are won’t to do, especially if you are new in an organisation – decide to start over from scratch, you may run into one or more of the problems that occur when attempting a rewrite. One example being where you end up in a competition against a moving target. The same team has to own a feature in both the old and the new codebase, in that case, to stop that competition. For some companies it is simply worth the risk, they are aware they are wasting enormous sums of money, but they still accept the cost. You would have to be very brave.

Abstractions in Infrastructure

From FTP-from-within-the-editor to Cloud native IaC

When software is being deployed – and I am ignoring native apps now, largely, and focusing on web applications and APIs- there are a number of things that are actually happening that are at this point completely obscured by layers of abstraction.

The metal

The hardware needs to exist. This used to be a very physical thing, a brand new HP ProLiant howling in the corner of the office onto which you installed a server OS and set up networking so that you could deploy software on it, before plugging it into a rack somewhere, probably a cupboard – hopefully with cooling and UPS. Then VM hosts became a thing, so you provisioned apps using VMWare or similar and got to be surprised at how expensive enterprise storage is per GB compared to commodity hardware. This could be done via VMWare CLI, but most likely an ops person pointed and clicked.

Deploying software

Once the VM was provisioned, things like Ansible, Chef and Puppet began to become a thing, abstracting away the stopping of websites, the copying of zip files, the unzipping, the rewriting configuration and the restarting of the web app into a neat little script. Already here you are seeing problems where “normal” problems, like a file being locked by a running process, show up as a very cryptic error message that the developer might not understand. You start to see cargo cult where people blindly copy things from one app to another because you think two services are the same, and people don’t understand the details. Most of the time that’s fine, but it can also be problematic with a bit of bad luck.

Somebody else’s computer

Then cloud came, and all of a sudden you did not need to buy a server up front and instead rent as much server as you need. Initially, all you had was VMs, so your Chef/Puppet/Ansible worked pretty much the same as before, and each cloud provider offered a different was of provisioning virtual hardware before you came to the point where the software deployment mechanism came into play. More abstractions to fundamentally do the same thing. Harder to analyse any failures, you some times have to dig out a virtual console to just see why/how an app is failing because it’s not even writing logs. Abstractions may exist, but they often leak.

Works on my machine-as-a-service

Just like the London Pool and the Docklands were rendered derelict by containerisation, a lot of people’s accumulated skills in Chef and Ansible have been rendered obsolete as app deployments have become smaller, each app simply unzipped on top of a brand new Linux OS sprinkled with some configuration answer, and then have the image pushed to a registry somewhere. On one hand, it’s very easy. If you can build the image and run the container locally, it will work in the cloud (provided the correct access is provisioned, but at least AWS offer a fake service that let’s you dry run the app on your own machine and test various role assignments to make sure IAM is also correctly set up. On the other hand, somehow the “metal” is locked away even further and you cannot really access a console anymore, just a focused log viewer that let’s you see only events related to your ECS task, for instance.

Abstractions in Organisations

The above tales of ops vs test vs dev illustrates the problem of structuring an organisation incorrectly. If you structure it per function you get warring tribes and very little progress because one team doesn’t want any change at all in order to maintain stability, the other one gets held responsible for every problem customers encounter and the third one just wants to add features. If you structured the organisation for business outcome, everyone would be on the same team working towards the same goals with different skill sets, so the way you think of the boxes in an org chart can have a massive impact on real world performance.

There are no solutions, only trade-offs, so consider the effects of sprinkling people of various background across the organisation, if instead of being kept in the cellar as usual you start proliferating your developers among the general population of the organisation, how do you ensure that every team follows the agreed best practices, that no corners are cut even when a non-technical manager is demanding answers. How do you manage performance of developers you have to go out of your way to see? I argue such things are solvable problems, but do ask your doctor if reverse Conway is right for you.

Conclusion

What is a good abstraction?

Coupling vs Cohesion

If a team can do all of their day-to-day work without waiting for another team to deliver something or approve something, if there are no hand-offs, then they have good cohesion. All the things needed are to hand. If the rest of the organisation understands what this team does and there is no confusion about which team to go to with this type of work, then you have high cohesion. It is a good thing.

If however, one team constantly is worrying about what another team is doing, where certain tickets are in their sprint in order to schedule their own work, then you have high coupling and time is wasted. Some work has to be moved between teams or the interface between the teams has to be made more explicit in order to reduce this interdependency.

In Infrastructure, you want the virtual resources associated with one application to be managed within the same repository/area to offer locality and ease of change for the team.

Single Responsibility Principle

While dangerous to over-apply within software development (you get more coupling than cohesion if you are too zealous), this principle is generally useful within architecture and infrastructure.

Originally meaning that one class / method should only do one thing – an extrapolation of the UNIX principles – it can more generally be said to mean that on that layer of abstraction, a team, infrastructure pipe, app, program, class […] should have one responsibility. This usually mean a couple of things happen, but they conceptually belong together. They have the same reason to change.

What – if any – pitfalls exist ?

The major weakness of most abstractions is when they fall apart, when they leak. Not having access to a physical computer is fine, as long as the deployment pipeline is working, as long as the observability is wired up correctly, but when it falls down, you still need to be able to see console output, you need to understand how networking works, to some extent, you need to understand what obscure operating system errors mean. Basically when things go really wrong you are needed to have already learned to run that app in that operating system before, so you recognise the error messages and have some troubleshooting steps memorised.
So although we try and save our colleagues from the cognitive load of having to know everything we were forced to learn over the decades, to spare them the heartache, they still need to know. All of it. So yes, the danger with the proliferation of layers of abstraction is to pick the correct ones, and to try and keep the total bundle of layers as lean as possible because otherwise someone will want to simplify or clarify these abstractions by adding another layer on top, and the cycle begins again.

Desktop OS for developers

.NET, C#, Evil, F#

The results of the latest StackOverflow Developer Survey just came out, showing – among other interesting things – that Windows is dying as a developer OS. Not one to abandon ship any time soon I’d still like to offer up some suggestions.

TL;DR

  • Make the commandline deterministic.
  • Copying files across the network cannot be a lottery.
  • Stop rebooting UI-frameworks
  • Make F# the flagship language

Back in the day, Microsoft through VB and Visual C++ overcame some of the hurdles of developing software for Windows – then the only, effectively, desktop OS in the enterprise. Developers, and their managers, rallied behind these products and several million kilometres of code was written over a couple of decades.

The hurdles that were overcome were related to the boilerplate needed to register window classes, creating a window and responding to the basic window messages required to show the window in Windows and have the program behave as expected vis-a-vis the expectations a Windows user might have. Nowhere in VB6 samples was anybody discussing how to write tests or how, really, to write good code. In fact, sample code, simplified on purpose to only showcase one feature at a time, would not contain any distractions such as test code.

When Classic ASP was created, a lot of this philosophy came a cross to the web, and Microsoft managed to create something as horrible as PHP, but with less features, telling a bunch of people that it’s OK to be a cowboy.

When the .NET framework was created as a response to Java, a lot of VB6 and ASP.NET  programmers came across and I think Microsoft started to see what they had created. Things like Patterns & Practices came out and the certification programmes were taking software design and testing into consideration. Sadly, however, they tended to give poor advice that was only marginally better than what was out there in the wild.

Missed the boat on civilised software development

It was a shock to the system when the ALT.NET movement came out and started to bring in things that were completely mainstream in the Java community but almost esoteric in .NET. Continuous integration – unit testing – TDD – DDD. Microsoft tried to keep up by creating TFS that apart from source code version in had ALM tools to manage bugs and features as well as a built-in build server but it became clear to more and more developers that Microsoft really didn’t understand the whole thing about testing first or how lean software development needs to happen.

While Apple had used their iron fist to force people to dump Mac OS for the completely different, Unix-based operating system OS X (with large bits of NextStep brought across, like the API and InterfaceBuilder) – Microsoft were considering their enterprise customers and never made a clean break with Gdi32. Longhorn was supposed to solve everything, making WPF native and super fast, obsoleting the old BitBlt malarkey and instead ushering in a brighter future.

As you are probably aware, this never happened. .NET code in the kernel was a horrible idea and the OS division banned .NET from anything ever being shipped with Windows, salvaged whatever they could duct tape together – and the result of that was Vista. Yes, .NET was banned from Windows and stayed banned up until Powershell became mainstream a long, long time later. Now, with Universal Windows Apps, a potentially viable combo of C++ code and vector UI has finally been introduced, but since it is the fifth complete UI stack reboot since Longhorn folded, it is probably too little too late and too many previously enthusiastic Silverlight or WPF people have already fallen by the wayside. Oh and many of the new APIs are still really hard to write tests around, and it is easy finding yourself in a situation where you need to install Visual Studio and some SDK on a build server, because the dependency relies on the Registry or the GAC rather than things that come with the source.

Automation

As Jeffrey Snover mentions in several talks, Windows wasn’t really designed with automation in mind. OLE Automation possibly, but scripting? Nooo. Now, with more grown-up ways of developing software – automation becomes more critical. The Windows world has developed alternate ways of deploying software to end-user machines than work quite well, but for things like automated integration tests and build automation you should still be able to rely on scripting to set things up.

This is where Windows really lets the developer community down. Simple operations in Windows aren’t deterministic. For a large majority of things you call on the command-line  – you are the only one responsible for determining if the command ran successfully. The program you called from the command-line may very well have failed despite it returning a 0 exit code. The execution just might not have finished despite the process having ended, so some files may still be locked. For a while, you never know. Oh, and mounting network drives is magic and often fails for no reason.

End result

Some people leave for Mac because everything just works, if you can live with bad security practices  and sometimes a long delay before you get some things like Java updates. Some people leave for Linux because if you script everything, you don’t really mind all those times you have to reinstall because thing like a change in screen resolution or a security update killed the OS to the point you can’t log in anymore, you just throw away the partition and rerun the scripts. Also, from a developer standpoint, everything just works, in terms of available tools and frameworks.

What to do about it

If Microsoft wants to keep making developer tools and frameworks, they need to start listening to the developers that engage whenever Microsoft open sources things. They most likely have valuable input into how things are used by your serious users – beyond the tutorials.

Stop spending resources duplicating things already existing for Windows or .NET as that strikes precisely at the enthusiasts that Microsoft needs in order to stop hemorrhaging developers.

What is .NET Core – really? Stop rewriting the same things over and over. At least solve the problems the rewrite was supposed to address first before adding fluff. Also – giving people the ability to work cross-platform means people will, so you are sabotaging yourselves while building some good-will, admittedly.

Most importantly – treat F# like Apple treats Swift. Something like – we don’t hate C# – there is a lot of legacy there but F# is new, trendier and better. F# is far better than Swift and has been used in high spec applications for nine years already. Still Microsoft after years of beta testing still manages to release a JITer that has broken tail call optimisation (a cornerstone of functional runtimes as it lets you do recursion effectively). That is simply UNACCEPTABLE and I would have publicly shamed then fired so many managers for letting that happen. Microsoft needs to take F# seriously – ensure it gets the best possible performance, tooling and templating. It is a golden opportunity to separate professional developers from the morons you find if you google “asp.net login form” or similar.

In other words – there are many simple things Microsoft could do to turn the tide, but Im not sure they will manage, despite the huge strides taken of late. It is also evident that developers hold a grudge for ages.

UTF-8

Evil, General, Win32

Having grown up in a society evolved beyond the confines of 7-bit ASCII and lived through the nightmare of codepages as well as cursed the illiterate that have so little to say they can manage with 26 letters in their alphabet, I was pleased when I read Joel Spolsky’s tutorial on Unicode. It was a relief – finally somebody understood.

Years passed and I thought I knew now how to do things right. And then I had to do Windows C in anger and was lost in the jungle of wchar_t and TCHAR and didn’t know where to turn.

Finally I found this resource here:

http://utf8everywhere.org/

And the strategies outlined to deal with UTF-8 in Windows are clear:

  • Define UNICODE and _UNICODE
  • Don’t use wchar_t or TCHAR or any of the associated macros. Always assume std::string and char * are UTF-8. Call Wide windows APIs and use boost nowide or similar to widen the characters going in and narrowing them coming out.
  • Never produce any text that isn’t UTF-8.

Do note however that the observations that Windows would not support true UTF-16 are incorrect as this was fixed before Windows 7.

Mail server and collaborative calendar

.NET, Evil, General

I was thinking, in light of the fact that all our data, including this blog, is being scrutinized by foreign (to me and most internet users) powers, that maybe one should try to replace Google Apps and Outlook.com, well Google Apps really. The problem is that Google Apps is pretty damn useful.

So what does one need? I have no money to spend, so it has to be free.

One needs a full featured SMTP server, a good web interface with simple design that also renders well on mobile in which you can search through (and find!) email and appointments. Some would argue that you need chat and video conferencing as well, and I guess one where neither Chinese nor US military is also on the call would be preferable, but I can live without it.

I cannot, however, live without civilized charset support. As in, working iso-8859-1 or something proper that can display the letters of honor and might, aka åäö. It would also not seem like a grown-up solution if it wasn’t trivial to add encryption and/or signing of e-mails.

You also need shared calendars and the option of reserving common resources. Not perhaps in the home, but the very first thing you do when you use shared calendars in a company is that you start booking things, such as conference rooms, portable projectors et c.

The user catalog needs to be easily managed. You need folders/labels of some kind and filters to manage the flow of e-mail.

I also, probably, need a way to access my e-mail via IMAP.

So the question is, how much of this do I have to build? How much exists in Linux? Does it look decent or does it look Linuxy? By decent I mean Web 2.0-ish, as opposed to linuxy or Windowsy, such as outlook web access.

I’d like any suggestion. I am prepared to code, albeit in C#/mono, but I would love to use as much Linux OSS available as I probably suck at writing this kind of stuff compared to those who have already done so.

So far I have found the following:
http://www.howtoforge.com/perfect-server-ubuntu-13.04-nginx-bind-dovecot-ispconfig-3
But instead of an ISP tool a custom website? Perhaps solr or lucene indexing each user’s maildir. But what about a calendar?

Why not just install Citadel?
This requires testing.

Windows Phone 6.1

BOOL, Evil

I managed to lose my employer’s Nokia N82 and as punishment by my boss he stuck me with an HTC Touch Cruise Windows Phone Classic 6.1 phone that nobody had wanted to use since 2008. I have tweeted about my findings with the hashtag #punishmentphone.

In short, the experience has been mixed. Synchronization with Google Apps works like a charm with e-mail, contacts and calendar and the messaging function is quite OK in the way e-mail works and the SMS part has conversations just like the iPhone. Sadly, though, the Windows Mobile general feel remains with very bad tactile feedback from the touch interface and a borderline unusable virtual keyboard and having a Windows interace on a phone means that user stories like “Create new SMS” or “Make a phone call” be at least a few clicks too far away for comfort. Oh, and another pet peeve: When the phone boots, it throws the SIM-card PIN-code dialog at me first, but that gets hidden by the WinMo desktop and I have to go in to the comm manager and disable the phone and reenable it to get the PIN dialog to a place where I can actually punch the numbers in. WinMo has improved since before, though as the phone has only died on me once so far for no reason, which is vastly better than a QTek S100 I wrestled with years before.

The Dark Side

.NET, Evil

Back in the nineties when I trained as an apprentice coder at the University of Umeå, I was first exposed to the Dark Side. It was very seductive with an intuitive TCP/IP stack, simple signal management conventions, concise UI:s, lean config files, powerful scripts, nifty daemons and of course the Bible: Advanced Programming in the UNIX Environment by W. Richard Stevens. I marveled at the distributed GUI, broken and deeply insecure though as it was, I admired that multi-user security concerns had been addressed back in the ‘70s already, not as an afterthought in 1989. Then I was brought out in the real world and began appreciating GUI conventions,  speed of implementation, solving customer problems and so on and came and joined the Just and Fair in the Microsoft camp and haven’t strayed since. Much.

But

But now and then the urge to complicate things gets stronger and stronger and with members of my immediate family having been lost to the Others for years now, the lure became overwhelming. After years of propaganda from my father I finally decided to install OpenSolaris on a virtual machine. Of course Solaris will not install on Windows Virtual PC Beta. It does not recognize any of the virtual SATA devices and thus cannot install. What to do? Well I gave up of course.

For a week or two until I thought I should make a serious attempt and actually dowloaded Sun’s VirtualBox.

VirtualBox vs Windows Virtual PC Beta

I really wanted to hate it, it not being made by the Just and Fair folks at Redmond. However, it is difficult to argue with 64 bit CPU high performance virtualization with USB, disk and network integration. It sucks the life out of the host machine, but in return both guest and host perform acceptably, lest you allocate too much memory to the guest “computer”, unlike with Windows Virtual PC where you get low performance (but better than Virtual PC 2007) on the handful of platforms it does handle.

Of course, I popped in the ISO in an empty Virtual machine (ironically using the same empty VHD Windows Virtual PC Beta created but couldn’t make available to Solaris) and of course the setup just chugged along event free. A vast improvement over previous UNIXes I installed back in the day. It refused to give me any options but to fill my VHD with the One Solaris partition to bind them all. None of that allocating partitions to /root, /home, /usr/local et c-business that used to be 90% of the fun of trying to set up a Unix style system.

Stereo? Not for me

Of course, once everything is set up you want to do stuff. Back in the day I would have set up FTP and the Apache httpd and created logins for friends and acquaintances and set up the firewall to allow for SSH:ing in to my computer. My computer in this case just being nothing but a figment of my laptop’s imagination, that would be even more pointless than it was back in the day. So: What to do? Weill, of course: Develop in .NET! After all, if I am to achieve global supremacy, I need to be able to code on Solaris as well, at some point.

This is where it all came back to me. Because it is a proper UNIX and not some humble Linux, allegedly, they haven’t gotten mono to run consistently on Solaris. The hours of gunzipping and ./configuring and make believing. compile errors upon compile errors. Downloaded various other package managers to appease the evildoers. Nothing I did could make mono build, despite gigabytes of source code. Ah well… Now I gave up properly.

OpenSUSE

Until I tried OpenSUSE, that is. Similar scenario, clean VM in VirtualBox, popped a physical DVD in my drive and shared it to VirtualBox, the empty VM booted and started the Open SUSE setup. I got to fiddle with my partitions (yay!) but I chose not to. Again, nothing fun happened during install, it just worked. A couple of reboots later I was able to login on my new machine, open the package installer, select everything mono-related and click install. After 20 minutes or se everything was downloaded and installed, and I just went to the Start menu(or what do you guys call it? The non-Start menu?) –> Applications-Development-Integrated Development Environment (monodevelop) and start to code. Very painless. Press play, off you go and you have ASP.NET being hosted on a Linux machine.

Then I noticed that my window was a bit on the smallish side, I would have preferred a higher resolution, like the one I had on the Solaris VM or so, so I go into the yast2 thingy and change display settings to something I consider appropriate and save the settings. I am instructed to restart the window manager, so I reboot the computer (yes I know. but it involved less interaction) and login again. Everything looks fine until after 5 seconds when the display gets garbled and I’m thrown out back to the login screen. I try all kinds of failsafe settings but to no avail. Of course I could manually edit a conf file somewhere to solve the issue, but Google has yet to reaffirm its friendship with me by coming up with a link to a Q&A forum where the question has been answered,

So?

Back to the good side it is, where stuff just works. Of course you can also end up in situations where stuff just doesn’t work as well (oi! CRM4! There’s this new thing, Windows Server 2008. It’s been out for a good two years now!), but resources are more plentiful and as an added bonus: you feel better cursing big successful company than makers of free software.