Yearly Archives: 2021

Why is everybody waterfalling “Agile”?

Just like that rebrand cycle years ago when RUP consultants transitioned over to scrum masters through staged re-titling on LinkedIn and liberal use of search / replace in their CV, scaled agile frameworks and certified project managers attempt to apply the agile manifesto to large organisations by bringing in processes and tools to manage the individuals and interactions, comprehensive documentation of the working software, to negotiate contracts to manage customer collaboration and make plans for how to respond to changes. You start seeing concepts like the Agile Release Train, which are – well – absurd.

Why? Do they not see what they are doing? Are they Evil?

No – I think it’s simple – and really really hard, at the same time.

You cannot respond to change quickly if you have delays in the system. These delays could be things like manual regression testing due to lack of automated test coverage, insufficient or badly configured tooling around the release or having a test stack that is an inverted pyramid, where you rely on a big stack of UI tests to cover the entire feature set of the application because faster, lower level tests aren’t covering all the features and you have undeniable experience of users finding bugs for you.

Obviously, if these tests are all you have, you need to run them before releasing or you would alienate customers. If your software stack is highly coupled, it would be irresponsible to not coordinate carefully when making changes on shared components with less-than-stellar contract test coverage. You are stuck with this, and it is easy to just give up. The actual solution is to first shorten the time it takes from deciding you have all the features you want to release until the software is actually live. This means automate everything that isn’t automated (the release itself, the necessary tests, et c) which could very well be a “let’s just stop developing features and focus all our attention on this until this is in place” type investment, the gains are so great. After this initial push you need to make an investment into decoupling software components into bits that can be released independently. This can be done incrementally whilst normal business is progressing.

Once you have reached the minimum bar of being able to release whatever you want at any time you want and be confident that each change is small enough that you can roll them back in the unlikely event that the automated tests missed something, then you are in a position to talk about an agile process, because now teams are empowered and independent enough that you only need to coordinate in very special cases, where you can bring in ad hoc product and technical leadership, but in the day to day, product and engineering together will make very quick progress in independent teams without holding each other up.

When you can release small changes, you can all of a sudden see the value in delivering features in smaller chunks with feature flags, because you can understand the value in making 20 small changes in trunk (main for you zoomers) rather than a massive feature branch, as releases go live several times a day, and the benefit of your colleagues seeing your feature flagged changes start appearing from beginning to end, they can work with your massive refactor rather than be surprised when you open a 100 file PR at 16:45 on a Friday.

Auto-login after signup

If you have a website that uses Open ID Connect for login, you may want to allow the user to be logged in directly after having validated their e-mail address and having created their password.

If you are using IdentityServer 4 you may be confused by the hits you get on the interwebs. I was, so I shall – mostly for my own sake – write down what is what, should I stumble upon this again.

OIDC login flow primer

There are several Open ID authentication flows depending on if you are protecting an API, a mobile native app or a browser-based web app. Most flows basically work in such a way that you navigate to the site that you need to be logged in to access. It discovers that you aren’t logged in (most often – you don’t have the cookie set) and redirects you to its STS, IdentityServer4 in this case, and with this request it tells identityserver4 what site it is (client_id), the scopes it wants and how it wants to receive the tokens. IdentityServer4 will either just return the token (the user was already logged in elsewhere) or get the information it needs from the end user (username, password, biometrics, whatever you want to support) and eventually if this authentication is successful, the IdentityServer will return some tokens and the original website will happily set an authentication token and let you in.

The point is – you have to first go where you want, you can’t just navigate to the login screen, you need the context of having been redirected from the app you want to use for the login flow to work. As a sidenote, this means your end users can wreak havoc unto themselves with favourites/ bookmarks capturing login context that has long expired.

Registration

You want to give users a simple on-boarding procedure, a few textboxes where they can type in email and password, or maybe invite people via e-mail and let them set up their password and then become logged in. How do we make that work with the above flows?

The canonical blog post on this topic seems to be this one: https://benfoster.io/blog/identity-server-post-registration-sign-in/. Although brilliant, it is only partially helpful as it covers IdentityServer3, and the newer one is a lot different. Based on ASP.NET Core, for instance.

  1. The core idea is sound – generate a cryptographically random one-time access code and map against the user after the user has been created in the registration page. (In IdentityServer4)
  2. Create an anonymous endpoint in a controller in one of the apps the user will be allowed to use, in it, ascertain that you have been sent one of those codes, then Challenge the OIDC authentication flow, adding this code as an AcrValue as the request goes back to the IdentityServer4
  3. Extend the authentication system to allow these temporary codes to log you in.

To address the IdentityServer3-ness, people have tried all over the internet, here is somebody who get’s it sorted: https://stackoverflow.com/questions/51457213/identity-server-4-auto-login-after-registration-not-working

Concretely you need a few things – the function that creates OTACs, which you can lift from Ben Foster’s blog post. A sidenote, do remember that if you use a cooler password hashing algorithm you have to use special validators rather than rely on applying the hash onto the the same plaintext to validate. I e, you need to fetch the hash from whatever storage you use and use the specific methods the library offers to validate that the hashes are equivalent.

After the OTAC is created, you need to redirect to a controller action in one of the protected websites, passing the OTAC along.

The next job is therefore to create the action.

        [AllowAnonymous]
        public async Task LogIn(string otac)
        {
            if (otac is null) Response.Redirect("/Home/Index");
            var properties = new AuthenticationProperties
            {
                Items = { new KeyValuePair<string, string>("otac", otac) },
                RedirectUri = Url.Action("Index", "Home", null, Request.Scheme)
            };

            await Request.HttpContext.ChallengeAsync(ClassLibrary.Middleware.AuthenticationScheme.Oidc, properties);
        } 

After storing the OTAC in the HttpContext, it’s time to actually send the code over the wire, and to do that you need to intercept the calls when the authentication middleware is about to send the request over to IdentityServer. This is done where the call to AddOpenIdConnect happens (maybe yours is in Startup.cs?), where you get to configure options, among which are some event handlers.

OnRedirectToIdentityProvider = async n =>{
    n.ProtocolMessage.RedirectUri = redirectUri;
    if ((n.ProtocolMessage.RequestType == OpenIdConnectRequestType.Authentication) && n.Properties.Items.ContainsKey("otac"))
    {
        // Trying to autologin after registration
        n.ProtocolMessage.AcrValues = n.Properties.Items["otac"];
    }
    await Task.FromResult(0);
}

After this – you need to override the AuthorizeInteractionResponseGenerator, get the AcrValues from the request, and – if successful – log the user in, and respond accordingly. Register this class using services.AddAuthorizeInteractionResponseGenerator(); in Startup.cs

Unfortunately, I was still mystified as to how to log things in, in IdentityServer4 as I could not find a SignIn manager used widely in the source code, but then I found this blog post:
https://stackoverflow.com/questions/56216001/login-after-signup-in-identity-server4, and it became clear that using an IHttpContextAccessor was “acceptable”.

    public override async Task<InteractionResponse> ProcessInteractionAsync(ValidatedAuthorizeRequest request, ConsentResponse consent = null)
    {
        var acrValues = request.GetAcrValues().ToList();
        var otac = acrValues.SingleOrDefault();

        if (otac != null && request.ClientId == "client")
        {
            var user = await _userStore.FindByOtac(otac, CancellationToken.None);

            if (user is object)
            {
                await _userStore.ClearOtac(user.Guid);
                var svr = new IdentityServerUser(user.SubjectId)
                {
                    AuthenticationTime = _clock.UtcNow.DateTime

                };
                var claimsPrincipal = svr.CreatePrincipal();
                request.Subject = claimsPrincipal;

                request.RemovePrompt();

                await _httpContextAccessor.HttpContext.SignInAsync(claimsPrincipal);

                return new InteractionResponse
                {
                    IsLogin = false,
                    IsConsent = false,
                };
            }
        }

        return await base.ProcessInteractionAsync(request, consent);
    }

Anyway, after ironing out the kinks the perceived inconvenience of the flow was greatly reduced. Happy coding!

WSL 2 in anger

I have previously written about the Windows Subsystem for Linux. As a recap, it comes in two flavours- one built on the concept of pico processes, marshalling the Linux ABI into Win32 API calls (WSL1) and an actual Linux kernel hosted in a lightweight Hyper-V installation (WSL2). Both types have file system integration and fairly transparent command line interface to run Linux commands from Windows and Windows executables from the Linux command line. But, beyond the headline stuff, how does it work in real life?

Of course with WSL1, there are compatibility issues, but the biggest problem is horrifyingly slow Linux file system performance because of it being Windows NTFS pretending to be EXT4. Since NTFS is slow on small files, you can imagine an operating system whose main feature is being a immense collection of small files working together would run slowly on top of it a filesystem with those characteristics.

With WSL2, obviously kernel compatibility is 100%, as, well it’s a Linux kernel, and the Linux file system stuff Just Works, as the file system is managed natively (although over hypervisor), but ironically, the /mnt filesystem with the Windows drives mounted are prohibitively slow. It has been said to be a bug that has been allegedly fixed, but given that we are – at the end of the day – talking about accessing local PCIe gen 4 NVME storage, managing to make file I/O this slow, betrays plenty of room for improvement. To summarise – if you want to do Linuxy in Linux under Windows, use WSL2 , if you want to do Windowsy things in Linux under Windows, use WSL1. Do with that what you will. WSL2 being based on a proper VM means despite huge efforts, the networking story is not super smooth, no proper mechanism exists to make things easier for you and no hits on Google will actually address the fundamental problem.

That is to say, I can run a website I have built in docker in WSL2, but l need to do a lot of digging to figure out what IP the site got, and do a lot of firewall stuff to be able to reach it. Also, running X Window with the excellent X410 server requires a lot of bespoke scripting because there is no way of setting up the networking to just work on start-up. You would seriously think that a sensible bridging default could have been brought in to make things a lot more palatable? After all, all I want to do is road test my .NET Core APIs and apps in docker before pushing them. Doesn’t seem too extreme of a use case.

To clarify – running or debugging a .NET Core Linux website from Visual Studio Code (with the WSL2 backend) works seamlessly, absolutely seamlessly. My only gripe is that because of the networking issue, I cannot really actually verify docker things in WSL2 which I surmised was the point of WSL2 vs WSL1.