F# Actor: An Actor Library

F# Actor: An Actor Library.

This is a follow-up on my post on the Actor Model, as I was very generous in my definition of F# Agents as a full-fledged Actor framework. F#:er s have been wanting something like the Akka framework for Scala. Colin Bull from whom I also reblogged the F# and Raven post below, posted about a framework on his blog that will provide this, and it also explains what the difference is and what the requirements are for an Actor framework.

 

Azure Websites with automatic deployment

Automatic deployment when you commit and push changes to your Github/Bitbucket repo is truly awesome and like running water or electricity that it is so convenient that you incorporate it into the fabric of your life and eventually only really notice it when it’s broken.The amount of reduced headache and increased productivity is amazing.

Overview

However, bringing continuous deployment to legacy web projects and solutions can be a bit challenging until it all works. The moving parts involved favor convention over configuration so in the most cases you don’t need to really do anything, but then there are cases when you all of a sudden notice that you need to do something to make things roll smoothly again.

Windows Azure Websites use a few tricks for the Deploy from Source Repository-feature to work. They put a git repository in your allotted folder on the IIS where your website resides which you can easily see via FTP.  You can access this repo directly if you do a straight git deployment where an “azure” remote is added to your local git repo, otherwise it is used by WAWS to pull your changes from TFS, Codeplex, GitHub or Bitbucket and it is from there the deployment happens. Allegedly, even when you deploy from Dropbox, your changes are first applied to the git repo in your directory in Azure and then the deployment begins from there.

The deployment system is open sourced at https://github.com/projectkudu/and you find further documentation there.

Gotchas

Multiple deployable projects in one repo

Let’s say you have a solution to deploy that has multiple projects that are potentially deployable. Kudu will not be able to guess which one to deploy and the deployment will fail with an error. What to do?

The solution is a deployment configuration file, a .deployments file. Yes, Windows won’t let you right click and create a file called “.deployment”, nor rename an existing file to that name, so you need to save it directly to that name from Visual Studio or just type the following in cmd.exe: (yes, the command means to copy from the Console, as in what you type, so just type the config file and finish with Ctrl+Z which means End of File and press enter again).

C:\Users\H4xx0rD00d\teh_c0dez\awesomerepo>copy con: .deployment
[config]
project = theActualFolderWhereTheProjectThatShouldBeDeployedResides
^Z
        1 file(s) copied.
C:\Users\H4xx0rD00d\teh_c0dez\awesomerepo>

After you add, commit and push that file, Azure will know which project to deploy. Further documentation on the .deployment file is available here.

Compilation fails with missing references

If you are using Nuget to manage your dependencies, you can choose either of two routes, Nuget Package Restore where Kudu will download the relevant libraries for you, or just committing your entire .nuget folder to your Git repository so that all your reference libraries are available for compilation on Azure.

The first option makes your repository more light-weight, but on the other hand you would be hoping that whomever manages the nuget packages on which you depend aren’t cheeky, like Microsoft have been in the past and withdraw the packages from under you, or that the maintainer misses a breaking change thus violating SemVer versioning causing Kudu to supply your app with an incompatible library.

With the second option you get a bulkier Repo, but you are in control of new versions of dependent libraries so that you can find out that they truly work with your system before you deploy. It works almost the same if you manually maintain your dependencies in the form of a Lib folder containing the DLLs that you need. Just ensure your Libs folder has been comitted to your Git repo and Kudu will be able to find them.

Øredev

For the last year and a half I have been a part of the Øredev Conference Program Committee. I should at some point write a post about it, and yes, I have had a draft laying around for a while, so here it goes:

I have always felt at home at Øredev as the conference was founded by Jayway and it is very much held close to our hearts. Of course, having played a part in the creation of the program makes it even more special for me to attend and last year I had the chance to attend Wednesday and also be at the Speakers’ dinner the evening before. This was a great opportunity to meet and greet, and I could have been more efficient there, but I did get to talk to a whole heap of interesting people.

The unrelenting search for quality speakers my colleagues pursued over the course of the year made Øredev 2012 angst-ridden in a good way. Whatever you chose to go see, there was always another talk you wanted to see going on at the same time. This is a good thing, when you are one of the ones putting together the program.

Last year

Last year we brought Mads Torgersen, Program Manager for the C# Language at Microsoft, which is pretty huge for a .NET guy like myself. He had a secret talk he wanted to do, and weeks before the conference, TypeScript was announced and we got to be the first ones to really bring that out there to our attendees.

TypeScript – JavaScript development at Scale
Async in C#5.0 – No More Callbacks!

Scott Barnes is a former Product Manager for Silverlight at Microsoft, now with Riagenic in Brisbane. He used his experience and laid out a plausible explanation to the twists and turns involving UI frameworks at Microsoft over the last year, as well as doing a very popular talk on a Design eye for the Developer Person. A very entertaining speaker that underplays his smarts.

Sebastien Lambla is another favourite of mine who took last year’s meme theme one step farther to combat session narcolepsy by leading the attendees in a Gangnam Style dance. He did an extremely initiated talk on HTTP Caching.

Jim McCarthy’s closing keynote on the Wednesday was mesmerising. I had to leave for the airport but I had a hard time breaking the spell to get up and leave.

Glenn Block is a good friend to the Øredev and a dream to work with. He is a very busy man but has the admirable quality of always wanting to help. In 2012 he talked about Hypermedia and WebApi as well as Node.js on Azure.

I have been a fan of Oren Eini’s since I saw him deliver a talk and demo on NHibernate back in 2009, and he is a very skilled and formidable speaker with a huge following on his blog. This time he delivered Raven DB training and two talks on Advanced RavenDB and an architecture talk entitled Hard Coding.

So what about 2013?

As experience tells us, you never know everything until November, but the program is out  and is looking rather complete. I went for .NET Open Source and F# this year, no point in trying to hide it. Maybe that’s a bit preachy, but so be it. We have great content on ASP.NET for IIS 8.5 and VS 2013 (“blue”) as well, of course and so on, but I really wanted to feature F# and functional in general as well as give props to some really mature OSS that isn’t getting enough praise.

In DDD with Value Objects, with ES and with the concept of Inversion of Control (well, OK, bear with me here), immutability and pure functions have already been shown to be the awesomesauce when trying to reduce complexity and increase parallelizability. Julie Lerman will talk DDD and Entity Framework, by the way.

Jessica Kerr will talk about Functional Principles for Object Oriented developers which explores how you can write better code in OO without going full Functional and Phil Trelford will talk about  F# for C# developers, when you do decide to go full Functional. Once you have decided to bridge the chasm, you go see Rachel Reese talk about the Actor model in F#, as in F# Agents, and about F# Type Providers. If you are ready to put the pieces together, there is Phil Trelford’s talk on F# for Trading, which is an example of why the City of London is crawling with F# developers  or at least recruiters looking for them.

NancyFx and ServiceStack are among the most popular Open Source frameworks out there for .NET and they have large communities, rich featuresets and proven track records and their respective fathers Andreas Håkansson and Demis Bellot will talk about them in greater detail and of course they will be available to be accosted for further questions you may have, which is the awesomest thing with Øredev.

A long time goal of mine was to bring Michele Leroux Bustamante to Malmö and Øredev. I became a fan back with her book Learning WCF which is the best programming book I have ever read, in tough competition with W Richard Stevens’ book Advanced Programming in the UNIX Environment. Since then Michele has created and sold a startup in the Identity field and is a Windows Azure MVP so what she doesn’t know about cloud and identity in the .NET space is probably not worth knowing.

Another important thing that you cannot miss is the excellent Anders Janmyr who with his talk on Habits of a Responsible Programmer managed to overcome the impossible standards you need to keep if you are both a part of Jayway and the Program Committee and still want to speak at the conference. You may think we would hand out speaking positions like candy to “our own”, but it’s rather the other way around as the last thing we would want to be accused of is playing favors and not striving for quality first.

This is just for starters, explore the program and book your tickets now. The conference will be the best ever.

The Actor Model

In the .NET community perhaps the Actor model of computing is not all that familiar. In this post I will briefly describe my understanding of the possibilities it gives the “real life” Enterprise LOB-application developer without going into specifics.


Different abstraction

The Actor model is a different abstraction from the normally more machine-like multithreaded server process model. This model sees the software system as a set of Actors listening for incoming messages. Messages are sent using to mailing addresses, conceptually, as in – not a hard reference to a running piece of code of any kind. An actor can only send messages to other actors that it knows the address of. It will know only of actors it receives messages from or creates itself. Unintuitively, the communication between actors is handled through a concept known as Channels, which would imply something more direct than a mail-drop system. In common implementations today, channels are implemented as queues.

The Actor will react to the incoming message by either sending a finite number of messages, creating a finite number of actors as well as deciding how to handle the next incoming message. The messages contain all the information needed for this processing, so there is no global state to modify or any shared resources to wait for. Each actor has its own execution context. Whether you execute all actors in the same thread in the same process or whether you have tens of thousands of separate machines with a multitude of threads of execution in several processes does not matter to the actor. This leads to a fantastic potential for scalability as there is little or no negative coupling.

Real-world examples?

If you think about it more closely, the Actor model can be said to better model the organizations we are building systems for. If a process requires a certain number of steps to be taken, modelling the steps, or actually the coworkers performing the step, as actors and the piece of paper or file that would travel from desk to desk can be modelled as a message. Just like faceless corporate drones, actors can be stateful, as is evident from the definition above.

In the Actor model data is not encapsulated but rather transmitted as messages between actors that perform actions with that data and then send the information forward. In object oriented programming you still send messages, conceptually, but actually you call methods on the object, but the objects are one with their data and the only way to modify the data is by sending a message to the object and hope the object will allow itself to be modified in the way you want.

There are large practical similarities between classic object oriented programming and the Actor model, and the Actor model frameworks implement actors as “objects” and use the OOP toolbox to implement the infrastructure that the frameworks provide, but the conceptual differences are importan remember when working with this model.

What frameworks exist for .NET?

There are two well known Actor model framework for .NET, Retlang (http://code.google.com/p/retlang/) and Stact. They both provide the basics you can expect from a framework of this kind, only Retlang stray somewhat from the Actor model nomenclature and provide no remote messaging capability, focusing on high performance in-memory messaging. It is unclear how development proceeds with Retlang, but there is support for .NET 4.

Another framework that supports Actor model is Stact https://github.com/phatboyg/Stact) which uses concepts more closely linked to the specification and would be preferable by those who like conceptual clarity.

Recently, F# has introduced the concept of F# Agents, which is essentially actors. They have isolation between instances (actors) and respond asynchronously to incoming statically typed messages.  (http://www.developerfusion.com/article/139804/an-introduction-to-f-agents/)

We will proceed and show a brief example using F# as it is less verbose than other ways of implementing this architecture. The examples are slightly modified from the above link and also ripped straight from the horse’s mouth, i e  Don Syme’s post http://blogs.msdn.com/b/dsyme/archive/2010/02/15/async-and-parallel-design-patterns-in-f-part-3-agents.aspx 

open Microsoft.FSharp.Control
type
Agent<‘T> = MailboxProcessor<‘T>

let agent =
    Agent.Start(
fun inbox ->
        let rec loop n = async
{
           
let!
msg = inbox.Receive()
            printfn
“now seen a total of %d messages”
(n+1)
           
return!
loop (n+1)
        }
        loop 0 )

for i in 1 .. 1000 do
    agent.Post (sprintf “Hello %d” i )

The above agent uses a builtin class called MailboxProcessor that allows you to post messages, passing in the actual Processor function as an argument, “putting the fun back into functional”.

Of course this is a silly example that doesn’t make anybody happy. Let’ s show something more network-related, so that you can imagine the implications.

open System.Net.Sockets

 

/// serve up a stream of quotes

let serveQuoteStream (client: TcpClient) = async {

    let stream = client.GetStream()

    while true do

        do! stream.AsyncWrite( “MSFT 10.38″B )

        do! Async.Sleep 1000.0 // sleep one second

}

 

What is the point of this then? Well, the agents are asynchronous and take no resources while waiting.  You might as well create a bunch of actors that will all lie there waiting for an incoming request and deal with them asynchronously with mindblowing efficiency. Every Actor (or instance of Agent) has its own state completely isolated from the others. See? No problems with shared mutable state. Put the foot on the accelerator and press it all the way to the floor.

Error management in Actors

Below is an example of error management using a supervisor that will take care of your vast pool of actors and deal with their failures. Again, contrived example, but remember you have all of the .NET framework at your disposal, so just imagine the supervisor doing some really clever reporting or cleanup.

open Microsoft.FSharp.Control

type Agent<'T> = MailboxProcessor<'T>

module Agent = 
   let reportErrorsTo (supervisor: Agent<exn>) (agent: Agent<_>) = 
       agent.Error.Add(fun error -> supervisor.Post error); agent

   let start (agent: Agent<_>) = agent.Start(); agent

let supervisor = 
   Agent<int * System.Exception>.Start(fun inbox ->
     async { while true do 
               let! (agentId, err) = inbox.Receive()
               printfn "an error '%s' occurred in agent %d" err.Message agentId })


let agents = 
   [ for agentId in 0 .. 10000 ->
        let agent = 
            new Agent<string> (fun inbox ->
               async { 
                    while true do 
                        let! msg = inbox.Receive()
                        if msg.Contains("agent 99") then 
                            failwith "I don't like that cookie!" }
                ) 
        agent.Error.Add(fun error -> supervisor.Post (agentId, error))
        agent.Start()
        (agentId, agent) ]

for (agentId, agent) in agents do 
   agent.Post (sprintf "message to agent %d" agentId )


Conclusion

The implications that arise from the Actor model and F# Agents in particular is that where you can use messaging (you are free to use F# Type Providers to make your various XML WebService packets become F# Types) you can process them with great efficiency to trigger all sorts of  .NET functionality asynchronously. You could use F# Agents to back your WebApi services to make the code more concise and responsive. There is no end to the possibilities.

For further reading, please don’t hesitate to follow Don Syme at http://blogs.msdn.com/b/dsyme/  and the various blogs and documentation available through there.

So what’s this FP thingy?

People are talking about Scala, F#, Erlang, Haskell and what-have-you. What is the point of functional? What is wrong with OOP? How does it all work?

Here are some videos you might want to look at. Maybe you find them helpful, maybe not.

Jessica Kerr: Functional Concepts for Object Oriented developers

This introduces the concept of immutability and pure functions and how your life gets better if you use those concepts in your everyday life. Of course, in DDD Value Objects and immutability is king as well, for the very reasons Kerr brings up in the video.

Brian Beckman: Don’t fear the Monad

This is a scary attempt at explaining monoids and monads. You may follow it, you may not. Don’t get too distracted or discouraged. Think of it monads in terms of LINQ and composability.

Erik Meijer C# Lecture, Functional Programming and Monads

http://www.youtube.com/watch?v=OrAVS4QbMqo

This talk is affected by Erik Meijers huge brain and is scarily influenced by Haskell, but makes sense if you saw the video above.

F# Creator Don Syme does a Tutorial

Don Syme F# Tutorial Part 1

You should skip the first part unless you want to know from whence F# came. There is very little syntax info in this chapter.

Don Syme F# Tutorial Part 2
Loops, data structures, beginning of pattern matching.

Don Syme F# Tutorial Part 3

Windows Azure Websites: -1

You should have by now – but if you haven’t – do try Windows Azure Websites. It is almost as usable as AppHarbor. Just register and start setting up your websites. For small blogs and wikis you can get away with internal storage and spend no money at all. No need for hosting ever again.

If you, as you should be, are writing your own web apps, you can connect your Bitbucket free private repo, or indeed your almost free GitHub private repo, or indeed a public repo directly to an azure website, so that you get CI whenever a commit is made in that repo. You can differentiate on branches so that the master branch triggers deploys to “production” and the staging branch, for instance, triggers deploys to the staging site. The possibilities are if not endless pretty huge.

The thing is, it works, although I’ve manage to upset the system by trying to do naughty things with weird Silverlight apps, but if auto deploy doesn’t work, manual deployment from Visual Studio still does. Even if auto deploy does work –  having CD from Git to the test site and VS -> Publish to your production site may not be such a bad idea, if you want to find a middle ground. Of course you are free to spend money if you like, but it is not necessary.

www.windowsazure.com

More on groupware

I have been thinking some more about a simple email + calendar service for Linux. I have tried a couple of ready-made groupware solutions but they seem not QUITE right FOR me. I want to rip off the ideas that GMail has shown to be successful. Also, let’s not ignore the Not-invented-here syndrome.

To start with the GMail experience is one of simplicity. Compare and contrast with Outlook Web Access which is essentially a fat client Windows app shoehorned into a website.

Users

User accounts will have to be managed in an LDAP server, but if not in version 1, very soon after, the users and rights management will have to be done in the website for the Simplicity goal to be fulfilled. The LDAP database needs to be complete enough to be meaningful as a source for existing Linux phonebook management clients as well as access control for websites and e-mail.

Calendar

What I want is a simple calendar view, with a possibility to launch a few forms to add and update entries. Here I’d have to find something suitable online, but the issue here mostly is the backend. I’d need to find an iCal compatible server that has a database format that I can figure out and live with.  From here I have to export files to each user’s directory so that they can be indexed as needed.

Email

The e-mail site can be extremely “flattering” to GMail, if imitation is the greatest form of flattery, with folders on the left and the list of e-mails front and center.

Wiki/Intranet

For it to be a solution fit for a smallish company you also need the ability to create content pages and publish them in the system somehow. A Wiki, for sure and also maybe a blog and a general CMS. Orchard or some other .NET CMS should be quite able to handle this, the major issues are to get either of them running on Mono and how to set up a default configuration 

Search

Search has to actually work. This means I will have to find a way to index web content, e-mail and appointments and parse queries and submit them to Lucene or SolR to actually provide great results fast. My whole bitterness against the proliferation of everybody’s personal data is that GMail has removed such large sections of brain that are no longer necessary. Almost whatever it is you have been up to in life you can find it using GMail search as you were bound to have written or received an e-mail about it – and if that didn’t happen you probably wrote an e-mail to yourself about it on purpose in order for it to be retrievable using the GMail search. It is a wonderful way of life, except when you have just handed strangers that same ease of access.

Conclusion

So, I’m thinking:

  • OpenLDAP
  • OpenSSH
  • Dovecot
  • Postfix
  • iCal compatible calendar server, there are several.
  • Solr
  • Mono 3.0.11
  • nginx
  • Jetty (for SolR)
  • PostGreSQL (?)
  • Orchard CMS or DNN or Umbraco. Orchard preferred.
  • My code / stuff
    • Wrapper website that hosts settings, e-mail, calendar and CMS views. Handles sessions and the almighty search bar.
    • Search results view
    • E-mail listing
    • E-mail compose  
    • Calendar view (week, month, days – need to find component that does the rendering.
    • Appointment view
    • Account management – against LDAP
    • Account management – with SSH/certificates
    • External POP3 and SMTP accounts
    • E-mail indexer
    • Calendar indexer
    • Packaging
      • Packaging my code
      • Packaging config of other packages

Mail server and collaborative calendar

I was thinking, in light of the fact that all our data, including this blog, is being scrutinized by foreign (to me and most internet users) powers, that maybe one should try to replace Google Apps and Outlook.com, well Google Apps really. The problem is that Google Apps is pretty damn useful.

So what does one need? I have no money to spend, so it has to be free.

One needs a full featured SMTP server, a good web interface with simple design that also renders well on mobile in which you can search through (and find!) email and appointments. Some would argue that you need chat and video conferencing as well, and I guess one where neither Chinese nor US military is also on the call would be preferable, but I can live without it.

I cannot, however, live without civilized charset support. As in, working iso-8859-1 or something proper that can display the letters of honor and might, aka åäö. It would also not seem like a grown-up solution if it wasn’t trivial to add encryption and/or signing of e-mails.

You also need shared calendars and the option of reserving common resources. Not perhaps in the home, but the very first thing you do when you use shared calendars in a company is that you start booking things, such as conference rooms, portable projectors et c.

The user catalog needs to be easily managed. You need folders/labels of some kind and filters to manage the flow of e-mail.

I also, probably, need a way to access my e-mail via IMAP.

So the question is, how much of this do I have to build? How much exists in Linux? Does it look decent or does it look Linuxy? By decent I mean Web 2.0-ish, as opposed to linuxy or Windowsy, such as outlook web access.

I’d like any suggestion. I am prepared to code, albeit in C#/mono, but I would love to use as much Linux OSS available as I probably suck at writing this kind of stuff compared to those who have already done so.

So far I have found the following:
http://www.howtoforge.com/perfect-server-ubuntu-13.04-nginx-bind-dovecot-ispconfig-3
But instead of an ISP tool a custom website? Perhaps solr or lucene indexing each user’s maildir. But what about a calendar?

Why not just install Citadel?
This requires testing.