TL;DR: I find it useful to have all the (NixOS) machines I administrate know meta information about their currently deployed configuration, which allows monitoring to shout at me if I forget to update one or (worse) have deployed changes I forgot to commit.

This comes up in conversation every now and then, and surprisingly often people are surprised how easily this can be done. Hence I thought I’d write a summary of what I do as a little reference I can point people to in the future.

Getting Git’s Information

This is fairly easy, whether using the (still experimental!) flakes or not. Flakes, by design, have a tighter integration to the underlying git repository (assuming the directory of the flake is a git repository, which it does not have to be) then we get information about it as part of the flake’s self input:

{
  inputs = { ... };
  
  outputs = { self }: { inherit self; };
}

Merely evaluating the self output here gives something like

{ 
  _type = "flake";
  inputs = { };
  lastModified = 1761994565;
  lastModifiedDate = "20251101105605";
  narHash = "sha256-5USBhx5RlZ6YVQiukrj5rsBXDZwO3d1QbNkLoixEFgc=";
  outPath = "/nix/store/z66skpk7izzap6336lsd091wpmcpk1v2-source";
  outputs = { self = «repeated»; };
  rev = "972aa7dd2f313733f551475115c6bb2cc1abdb57";
  revCount = 2;
  self = «repeated»;
  shortRev = "972aa7d";
  sourceInfo = { 
    lastModified = 1761994565;
    lastModifiedDate = "20251101105605";
    narHash = "sha256-5USBhx5RlZ6YVQiukrj5rsBXDZwO3d1QbNkLoixEFgc=";
    outPath = "/nix/store/z66skpk7izzap6336lsd091wpmcpk1v2-source";
    rev = "972aa7dd2f313733f551475115c6bb2cc1abdb57";
    revCount = 2;
    shortRev = "972aa7d";
    submodules = false;
  };
  submodules = false;
}

Entertainingly (as part of what appears to be a fundamental law of all things nix) we even get the same information twice.

Without flakes

What do we do if we don’t use flakes? In that case, the decision to keep our source files in a git repository is decoupled from the fact that they’re a human-appreciable collection of source files: nix does not know or care that they’re in a git repository, and so won’t give us the information unless we ask for it explicitly.

How do we ask? By simply misusing builtins.fetchGit:

builtins.fetchGit ./.

This will, if it’s evaluated in the root file of the git repository, clone it & give use essentially the same information that we saw before:

{
  lastModified = 1761994565;
  lastModifiedDate = "20251101105605";
  narHash = "sha256-5USBhx5RlZ6YVQiukrj5rsBXDZwO3d1QbNkLoixEFgc=";
  outPath = "/nix/store/z66skpk7izzap6336lsd091wpmcpk1v2-source";
  rev = "972aa7dd2f313733f551475115c6bb2cc1abdb57";
  revCount = 2;
  shortRev = "972aa7d";
  submodules = false;
}

(Though disappointingly, this time, we only get it once)

Uncommitted changes

When evaluating a flake in a “dirty” worktree which contains uncommitted changes, nix will produce a warning, and produce a slightly different attribute set:

{ 
  _type = "flake";
  dirtyRev = "9952748d841d18c5434919f457b0cbac4ba53ba7-dirty"; 
  dirtyShortRev = "9952748-dirty";
  inputs = { };
  lastModified = 1761951614;
  lastModifiedDate = "20251031230014";
  narHash = "sha256-5USBhx5RlZ6YVQiukrj5rsBXDZwO3d1QbNkLoixEFgc=";
  outPath = "/nix/store/z66skpk7izzap6336lsd091wpmcpk1v2-source";
  outputs = { self = «repeated»; };
  self = «repeated»;
  sourceInfo = { 
    dirtyRev = "9952748d841d18c5434919f457b0cbac4ba53ba7-dirty";
    dirtyShortRev = "9952748-dirty";
    lastModified = 1761951614;
    lastModifiedDate = "20251031230014";
    narHash = "sha256-5USBhx5RlZ6YVQiukrj5rsBXDZwO3d1QbNkLoixEFgc=";
    outPath = "/nix/store/z66skpk7izzap6336lsd091wpmcpk1v2-source";
    submodules = false; };
  submodules = false;
}

rev and shortRev have been replaced by dirtyRev and dirtyShortRev.

The same is (almost) true for fetchGit:

{
  dirtyRev = "9952748d841d18c5434919f457b0cbac4ba53ba7-dirty";
  dirtyShortRev = "9952748-dirty";
  lastModified = 1761951614;
  lastModifiedDate = "20251031230014";
  narHash = "sha256-5USBhx5RlZ6YVQiukrj5rsBXDZwO3d1QbNkLoixEFgc=";
  outPath = "/nix/store/z66skpk7izzap6336lsd091wpmcpk1v2-source";
  rev = "0000000000000000000000000000000000000000";
  revCount = 0;
  shortRev = "0000000";
  submodules = false;
}

Very annoyingly, the returned attributes differ: unlike with flakes, the attributes rev and shortRev still exist, but now always give a meaningless dummy value! On its own this is not a large problem—all the information is still there—but it’s good to be mindful of this when writing code which consumes this information and is used both with and without flakes, as otherwise you’ll get surprising bugs.

Aside: What is a “Short Rev”, anyways?

Depending on how much experience you have with git, and especially with very large git repositories (say, Nixpkgs), at this point there might be a question burning inside you: What is a “short hash” here, anyways, and exactly how long is it?

Well, if you’re using CppNix, the answer is “it’s exactly the first seven characters of the full-length commit hash”.

Except this is not at all how git behaves when it shortens commit hashes! If you ask it for one (e.g. via git rev-parse --short) it will create a “short” prefix of the full commit hash, but ensure that it is still long enough to uniquely identifies its referent object in the current repository, with seven characters being a minimum that might well grow if the repository is very large.

Why did I specify “CppNix” above? Surely the Nix implementation you choose should not matter? Isn’t this ecosystem all about reproducability? Oh, but what a nice world that would be.

Instead, if you use Lix, you will get a dirtyShortRev that is produced by simply calling out to git — it will always be a unique identifier, exactly as git would like it to be, but its precise value and length might well depend on your git config.

Can you already hear the ugly beast rearing its head? Lix will not give you a guaranteed-unique identifier for the shortRev attribute in the non-dirty case, because that would break reproducability of flakes: the length of a guaranteed-unique short hash depends not just on the locked revision of a git repository input, but also on your git’s config, and worse, on exactly how much other stuff there is in that git repository; if it grows, eventually the short revs will grow, too.

As this is not only unlikely to be fixed, but indeed appears conceptionally impossible to fix at all, it’s probably best to never use the values of shortRev and dirtyShortRev at all, and prefer the full versions in all cases.

Telling the machine where it’s at

First, the standard standard way to include information like this into a system is to put it into /etc/os-release, where it can be picked up by various tools - for example, the boot menu will show it when choosing a generation. Such attributes are easily set by using the system.nixos.label NixOS options.

Second, I want my machines to immediately tell my what state is on them when I log in. NixOS and scriptable motd messages are tricky, so instead I set it from inside Nix:

let
  formatDate = date: with lib.strings;
    let
      year = substring 0 4 date;
      month = substring 4 2 date;
      day = substring 6 2 date;
      hour = substring 8 2 date;
      minute = substring 10 2 date;
      second = substring 12 2 date;
    in
      "${year}-${month}-${day} ${hour}:${minute}:${second} UTC";
in
{
  users.motd = ''
    Welcome to ${config.networking.hostName}, running NixOS ${config.system.nixos.release}!
    Built from ${self.dirtyRev or self.rev}.
    Last commit was at ${formatDate self.lastModifiedDate}.
    ${if self ? dirtyRev then "\nPlease remember to commit your changes!\n" else ""}
  '';
}

(why did CppNix’s designers think it appropriate to provide a “readable” date format as a string, but make contain only digits and nothing else, forcing us to re-parse that inside Nix if we want to have anything readable? I have no clue)

Monitoring

Finally I like to do one more thing as well: Set up some kind of monitoring which will tell me if I forgot to commit changes I have already deployed. This is especially important when a machine is administrated by more than one person: If one of us forgets to commit anything, others are either blocked from working on the machine, or may unwittingly roll back previous changes.

First, create two files inside /etc:

{
  environment.etc.commit.text = self.dirtyRev or self.rev;
  environment.etc.timestamp.text = builtins.toString self.lastModified;
}

Of course we could also simply parse this information out of /etc/os-release — but doing it this way makes it a little easier to write custom scripts, so, eh, why not.

monit

For reasons incomprehensible to most of my acquaintances, I still use monit for a lot of my monitoring. But it makes it easy to set up little checks that should cause notifications if they fail, and it’s trivial to set up, especially compared to more modern cloud-infra monitoring tools I could use on my usually comparatively tiny setups.

(on the other hand, it comes from the pre-systemd ages when it was seen as reasonable to write your monitoring daemons in C and run them as root so they could restart arbitrary startup scripts — your mileage may vary on that in today’s world)

Do we know what the currently deployed state is?

There is one main check:

Is the currently deployed commit the one that’s actually the head of the main branch, as pushed to some forge?

Most git forges offer convenient APIs for retrieving commit hashes of branch heads, so this is easily implemented as a little script:

let
  checkHash = pkgs.writeScriptBin "check-commit-hash" ''
    #!${lib.getExe pkgs.fish}
    set wanted (${lib.getExe pkgs.curl} -s \
                https://git.example.org/api/v1/repos/config/branches/main \
                -H 'accept: application/json' | jq -r .commit.id)

    if test $status != 0
      echo "could not reach git.infra4future.de"
      exit 2
    end

    set actual (cat /etc/commit)
    if test $actual != $wanted
      echo "${config.networking.hostName} was built on $actual, but commit on main is $wanted"
      exit 1
    end
  '';
in { 
  services.monit.config = ''
      check program deployed-commit-on-main path ${lib.getExe checkHash}
            if status == 1 for 64 cycles then alert
            if status == 2 for 3 cycles then alert
  '';
}

Conveniently, this has the by-product of also checking for dirty-ness of the currently deployed commit, as the -dirty at the end of dirtyCommit will cause the check to fail even if the hashes match.

Did we forget to update the host?

let
  checkDeployAge = pkgs.writeScriptBin "check-deploy-age" ''
    #!${lib.getExe pkgs.fish}

    set date (date +%s)
    # we do this indirection here so monit's config won't change on each deploy
    set deploytimestamp (cat /etc/haccfiles-timestamp)
    set age (expr $date - $deploytimestamp)

    if test $age -ge (expr 3600 \* 24 \* 10)
      echo "${config.networking.hostName} has not been deployed since 10 days, perhaps someone should do updates?"
      exit 1
    end
  '';
in {
  services.monit.config = ''
      check program check-deploy-age path ${lib.getExe checkDeployAge}
            if status == 1 then alert
  '';
}

Alternative: include the full config

There’s an alternative to this approach which achieves all the same goals (and then some), but which I found a little more annoying in practice. Simply do:

{
  environment.etc.nixfiles.target = self.outPath;
}

and the entire configuration that has currently been deployed will always be immediately available, no matter what has been messed up or forgotten. We actually ran this on the infra4future.de server for almost a year, and it worked well enough.

So why all this extra effort, if it’s this easy? Mainly, it makes deploys horribly sluggish — there is no good way to cache half a derivation, so each and every deploy (unless it is a complete no-op) will now require re-uploading the entire configuration to the server, and deploy speed is effectively capped at however long that takes on whatever uplink one’s currently using. And even if it’s only a couple kilobytes this can take a while, especially during debugging sessions from a moving train.

Conclusion

That’s it! Overall, I’m pretty satisfied with this approach & it’s been useful more than once the last few years.

There is one thing I wish I could add, tho: adding the whole configuration to the system closure is too costly, but adding a diff to the last commit (especially in dirty states) would be extremely useful, but I’ve never looked into how possible that would be … perhaps one day.

TL;DR: if you use git-annex and have a reMarkable 2 tablet, you might find this special remote useful.

Too many pdfs …

It is a truth universally acknowledged (among some people, at least), that pdfs have an unfortunate tendency to pile up, forming unstructured heaps in the “Downloads” directory where nothing that has once been read can ever be found again. Inevitably, things become much worse once one attempts to sort a few of them into their own directory, renames files, moves computers …

For a couple years now, git-annex has been my way to tackle this issue: I have a single git repository which contains everything I read (and a lot more I don’t); git-annex dutifully takes care of tracking these files, of moving the entire thing to new computers, of checking I have copies on other systems before deleting any local copies, etc — all I have to do is remember to check files into git.

In a wild twist, the thing designed to be a good archival tool turns out to be good at archival work.

Reading

I read a lot, but LCD screens strain my eyes. This is a general problem, but for reading there’s off-the-shelf solutions:

Conveniently, this runs a reasonably ‘standard’ Linux — including familiar busybox and systemd — and it also runs a pre-configured ssh daemon out of the box.

Happily, while the UI software is proprietary, I am also not beholden to the company’s commercial cloud service to sync files to the device — someone has already re-implemented that (only one project among many; people have also written a package manager and even entirely new UIs).

Except – I don’t want any cloud service, be it self-hosted or not! I don’t even connect the thing to the internet very often — surely, if I can have an ssh session via its USB port, that should be enough?

Special remotes

git annex has an obvious way to handle such things: special remotes. If it can store and retrieve objects by a key (an identifier usually based on some hash), git annex can be taught to consider it external storage and push files to it — without caring how it looks on the ‘other side’.

So let’s just make the other side that tablet?

Writing a new special remote is as easy as implementing the dedicated line protocol, which git annex uses to talk to a sub-process via std IO.

For extra fun, I decided to write mine in Rust, without any external crates, i.e. only using things from std (calling a few external program to handle e.g. ssh and uuids). This works surprisingly well: rust makes for a – somewhat verbose – scripting language, too.

Xochitl ipan quixichihua in amatl

File structure

Xochitl, the tablet’s UI, expects documents to be stored under ~/.local/share/remarkable/xochitl as a flat list:

$ ls .local/share/remarkable/xochitl
-rw-r--r-- .local/share/remarkable/xochitl/d1e71a92-3ab8-4455-9233-d84c06f3997a.content
-rw-r--r-- .local/share/remarkable/xochitl/d1e71a92-3ab8-4455-9233-d84c06f3997a.local
-rw-r--r-- .local/share/remarkable/xochitl/d1e71a92-3ab8-4455-9233-d84c06f3997a.metadata
-rw-r--r-- .local/share/remarkable/xochitl/d1e71a92-3ab8-4455-9233-d84c06f3997a.pagedata
-rw------- .local/share/remarkable/xochitl/d1e71a92-3ab8-4455-9233-d84c06f3997a.pdf

One displayed document is several files, grouped by a common UUID — the pdf file itself, metadata, hand-drawn notes for each page of the document, tags, …

To store a new one, the special remote can derive a stable UUIDv5 from the item’s key, namespaced to the special remote itself (conveniently, git annex assigns each special remote instance its own UUID already, which works well for this).

File content

Second step: minimal skeletons of the other files, just enough to make xochitl happy and display our document. It turns out a few fields in .metadata and .content are enough; xochitl will fill in the rest by itself.

{
  "createdTime": "970351200",
  "lastModified": "978303600",
  "lastOpened": "{time}",
  "lastOpenedPage": 0,
  "parent": "",
  "pinned": false,
  "type": "DocumentType",
  "visibleName": "Nahuatl As Written"
}

{
  "coverPageNumber": 0,
  "documentMetadata": {},
  "extraMetadata": {},
  "fileType": "pdf",
  "fontName": "",
  "pageTags": [],
  "tags": [],
  "textAlignment": "justify",
  "textScale": 1,
  "zoomMode": "bestFit"
}

One issue: .metadata sets the document’s name, as shown on the tablet. But a special remote is nothing more than a hash table: it knows items by key, not by any title or name, and won’t be told anything as helpful as a file name.

Document names

So where to get a recognisable name?

I tried extracting the pdf’s embedded title, but it turns out this is too often missing or unhelpful (who needs tens of documents titled “Microsoft Word Document” in their listing?).

At this point I got a little stuck, but luckily, the protocol’s spec has an ‘export/import’ appendix, for this exact case: special remotes which also look like file listings, not only like hash tables.

Its operations are broadly similar to their ‘normal’ store/retrieve siblings, but each also receives a file name to be freely used by the remote. For the tablet, just taking the file name (without extension) seems good enough. All this requires is that the special remote is initialised with exporttree=yes. Problem solved?

… well, mostly. This appendix is only half-done, and only the export operations are specified, testable, and implemented in git annex; the whole ‘import’ section is an unimplemented draft.

I hereby declare getting files back out from the tablet to be a problem for future me.

Pushing to the device

Having set up the file structure, all that’s left is funneling it over ssh. Thankfully, this is as easy as it possibly could be:

With all this done, it’s time to git annex testremote (but do it in a sandbox — or a VM test — else there’s a good chance of leaving litter behind if anything was wrong).

git annex export & git annex wanted

One final issue: using the export operations allows us to use file names, but the usual git annex copy & friends now no longer work: Any special remote initialised with exporttree=yes has to be used with git annex export, which can only operate a git branch or revision (or a subtree thereof), but not on a single file.

Happily, git annex has a concept of filtering, which the export will respect:

git annex metadata can attach little tags to files in git annex’s repository.

git annex wanted lets one specify which files a remote “wants” to store; files that don’t match aren’t exported.

So all I need to do is git annex wanted <specialremote> "metadata=amatl=store" once, and then git annex metadata -s amatl=store <filename> to mark new files for transfer, and then git annex export (or simply git annex sync) to push all new files to the device.

All done?

Well, not quite. Now I can read pdfs, but would it not be nice to also get any notes I draw on them back into the repository?

As mentioned, the “import” section of the protocol spec is still a draft, and unimplemented; there’s some additional complexity in that the tablet’s format for drawings is prorietary, although people have reverse-engineered it.

Perhaps I’ll look at this at some point in the future; I’m not sure if doing this via git annex import wouldn’t be stretching the concept of special remotes a little too far — do I really want to replace the entire content-addressed pdf in git annex’s store every time I draw a new line on it? — but for now, that’s it.

About a month ago now, I spent a couple fun but exhausting days in Milano for a conference. While there I gave a talk, unexpectedly found a yellow soft drink I miss a lot, and of course I also took the tram:

Milano is famous for still having a large number of old trams from the 1920ies running in regular service. And as one of the lines stops directly next to the university campus, I decided to simply get on …

… to quickly discover I had no idea where it was going. At the tram stop, there’d been an information sign for the two lines stopping at it, but only with a list of stations (useless to non-local me, who recognised none of them). So I looked for a map.

Without success! No map at the stop, no map in the tram, and (as I checked later) not even at larger interchange stations. I got off near Cadorna, and then took the metro back — but in the metro station, too, the maps showed only the metro itself and the suburbano lines, not the trams.

À la recherche des cartes perdues

So really, how does one find out which tram to take? Looking around, I did find a couple options:

official

The city’s transit operator ATM has a map displayed at metro stations, but it only shows the metro and (the central parts of) the suburbano train lines. They also have a couple other maps [ATM] listed on their website; confusingly this includes a complete map of night lines, but no equivalent for ‘normal’ service hours.

Unfortunately I didn’t have the time to go bother them at an information desk while I was in the city, so perhaps there are some maps available there — but if so, I’ve not found any of them mentioned online.

As far as I can tell, currently the only official way to navigate the tram (and also the much larger bus) network is via ATM’s mobile apps, or alternatively via its web app [ATMApp]. But no matter how convenient these are for routing, they don’t contain a good map either (and the web app likes to get stuck if one tries to make it display more than a few tram lines at once — not that it would produce a very readable map if it succeeds).

third-party

Finally, there are (of course) various tourist maps of the city’s center and main sightseeing attractions, which sometimes also include the central section of the tram network (although not always in the most readable way), as well as countless third-party web sites and transport wikis containing variously up-to-date information.

Even on WikiMedia (usually a good source of custom-drawn network maps, often much nicer than their official versions) a cursory glance found only a few maps of the tram system. But mostly these are geographical maps first, with extra information squished into them (although looking again I did find this very neat map).

historical

What came before the mobile apps? Surely people must’ve had a way to figure out which tram to take? They did! Fortunately for us, stagniweb [Stagniweb] has preserved an impressive collection of archived historical maps of Milano’s trams, the last of which is dated to 1982:

Unfortunately, I’m unclear on what happened after this map — an urbanfile.org blog post [Montella] mentions a similar map from 1985, but sometime after that, ATM seems to have discontinued this format; I’ve not found any official map of the trams from the years since.

By the time archive.org starts having snapshots of ATM’s website in 2011 (before then, the domain belonged to an online shopping business), they already offered the routing app, and seemingly no full network map of the tram lines were available to download then, either — although the site mentions you could still get free paper maps. But if these still included the trams, I’ve not found any pictures of them online.

If anyone has clues on this (or even lived in or visited Milano during that time), feel free to message me, I’d love to know!

Drawing my own

So there I was, sitting in my hotel room, and wondering how to procrastinate on making the slides for my talk (thankfully scheduled for the last day of the conference, so I had some time). Why not try to make my own map?

By complete coincidence, the weekend before I’d been at BayT_EX, where Aada had given an amazing last-minute talk [Aada] about how to draw line network maps with T_EX, so I had a rough idea of how to go about it.

sketching

Lacking an official map to get any kind of overview, I fought ATM’s web app for a while (it kept getting stuck) until I had a rough sense where each line goes, and where they meet and cross each other:

And having done that I thought, well, really, how hard can it be to do this with TikZ? All I have to do is to re-use Aada’s framework and shapes, surely the rest is doable, too?

tikzpictures

Aada’s maps rely on tikzpictures to do most of the heavy lifting involved in creating pictures.

Depending on your background, you might have a certain intuitive aversion to TikZ — or at least I’ve found that many academics who use T_EX for their papers are wary of it, and its 1300+ page manual has a reputation of its own – but it is (surprisingly?) one of the least bad tools for this kind of task:

• compared to most (vector-)graphics programs, there’s a usable macro system.
• it’s possible to build (albeit leaky) abstractions over the shapes used to draw the map.
• drawings made with TikZ are parametric: nodes can be set relative to other nodes, so the network’s layout stays relatively flexible even with all lines and stations filled in.

A decision I made very early on was to use lines 9 and 10 as “anchors” for the rest of the network, as they form a nice circle around the city’s center. This wasn’t the best idea — it kept getting crowded when I filled in the central stations later — so if you intend to design your own map, I’d advise to grow it “outwards” instead: start with the most complex central stations, and then slowly add the rest around them.

Of course, using TikZ for this sort of thing does have its annoyances:

• most obviously, it involves writing text; results are only visible after calling T_EX.
• there’s thus all the usual problems that come with writing several hundred lines of code in any language; it’ll easily turn into spaghetti. Though if that happens, one can always define more macros …
• the biggest by far: T_EX is fundamentally unsuitable for creating true, non-leaky abstractions (barring some hackery with catcodes, as e.g. expl3 does), and various macros defined by TikZ have strange, unexpected limitations — for example, a \\ occurring inside a \foreach will usually break things, especially if it is text inside a node.

Unfortunately it turns out drawing one of the largest tram networks in Europe is not doable in an evening in a hotel, so I left Milano (and the conference) with a very unfinished map. But over the following weeks I kept coming back to it, fiddling around until the overall layout looked mostly usable.

so many stations …

After that TikZ could really shine: filling in all the other stations was a breeze, and done in only a couple hours (most of which were spent learning just enough expl3 to build a useful macro for nicely setting multi-line labels in odd locations from inside a \foreach).

And well, here it is. A mostly-accurate map of Milano’s trams. While it has a few known defects (some routes take a slightly different route in either direction, and sometimes there was some guessing involved due to [ATMApp] lacking some information), I am overall quite happy with the result:

• The overall layout of the lines is very clear, with geographic convolutions smoothed out in favour of their topology.
• I could entirely avoid rotating text even for very long station names (“Piazzale Principessa Clotilde Ospedale Fatebenefratelli”), which I’ve never found aesthetically pleasing. (I also avoided the common Italian shortenings like “P.le.” for “Piazzale” simply because I discovered I could.)
• Despite some complications, it’s (almost) always easy to see which station a label belongs to, and the labels mostly don’t overlap anything else.

And now?

Some information essential to a usable transit map is still missing: there probably ought to be a legend, stations outside the City of Milano should probably be marked as such, fare zones and interchanges to other modes of transport could be added in … but I don’t really have plans to add these for now; the fun for me was in drawing the network itself.

Perhaps I could now integrate both metro and suburbano into my map as well; they both intersect with the tram network heavily, and are also useful transit modes inside the city’s core. Otoh there’s very little chance I will ever add the bus lines …

can’t we automate this?

Several people, when seeing what I was working on, immediately asked: why have you written 1500 lines of T_EX for this, a task which surely can be automated?

I think there’s two answers to this: first, this kind of ‘planarisation’ isn’t really easy at all. For one, it’s unlikely the network graph is planar, thus choices must be made on where it should overlap itself. But there’s a lot more: how much should it abstract, how true should it be to geography? How and where to place labels, and how to make them fit next to a well-connected interchange? Nor is planarity the same a clarity: even if no lines ever cross over at all, a reader might still easily get ‘lost’ when attempting to understand the network depicted by the map.

Thus, secondly: while we can insist on imagining a transit map as the result of a large optimisation problem, in the end there’s a lot more of art to designing it than there is graph theory. After all, its purpose isn’t to depict a graph, but to communicate helpful information to people who don’t know their way around an unfamiliar city. That it depicts a graph is only incidental.

This is not to say we can’t perhaps make the process of designing such a map a little less onerous. There’s some fun ways of drawing transit lines onto geography [trapp], or even automating the entire thing, but in an interactive way so a person can still decide on at least some of the ‘choices’ made inside the map [Janiak].

conclusion & advice for making your own

Finally, perhaps you feel — for whatever reason — the insatiable need to create a similar map yourself. Perhaps your local city lacks a good map (or any map). Perhaps it does have a map, but it’s outright wrong in how it displays some lines (this is waay more common than I used to think). Regardless, here (in no particular order) are some useful things I learnt along the way:

• Drawing a rough sketch by hand first turns out to be a very good idea. Whatever program you use to draw the graphic later, it’s always clunky to re-organise many lines at once; starting a new, improved rough sketch is easier.
• This is doubly true if you don’t have an existing map to guide you.
• Your final map will look very different than your sketch. Start with the most difficult, most densely clumped section of the network, it’ll change the most (and afterwards you can be glad to have that behind you)
• don’t forget to think about where labels go; compared to lines / stations, they take up a lot of space!
• If you’re using TikZ, it pays to have several “anchor” stations, and place everything else relative to these in an outwards-growing dependency tree.
• Also if using TikZ: read a little of the manual first! Yes, it does have 1300+ pages. But it’s really useful to know about all the fun ways to specify coordinates.
• Learn some expl3 while you’re at it, and try not to use the pgf \foreach unless you really need to, it’s brittle and will break on you later in entirely unexpected ways.

And most importantly, have fun with it! Don’t be afraid to make unexpected design choices and see where they lead you! And if you get stuck, nerdsnipe some of you friends into looking at it; transit maps turn out to make for a fun social activity — I’d especially like to thank io for so many good suggestions for my own map, along with Aada for having the initial idea of drawing these with T_EX.

References

Connection to ICE 4711 today on track 8, on the same platform directly opposite.

How reassuring to hear these words — but often they’re not there. Perhaps there was a surprising last-minute track change, or train staff isn’t sure either, or someone simply forgot, or you’re on a regional line where this kind of announcement is simply never done at all. Or perhaps (my personal favorite) the connection you’re intending to take was never meant to be possible in the first place.

And suddenly you really need to know if tracks 3 and 4 are on the same platform in a station you’ve never been in before.

This post will describe how to solve this problem using data from Open Street Map (OSM), and use it as an excuse to learn some OverpassQL along the way. Since I happened to already run a search engine for German railway stations (who doesn’t?), you can also use the result simply by visiting bahnhof.name to get a list of platforms for your favorite station.

The Shape of the Problem

For the benefit of everyone who does not feel an innate sense of comfort in the liminal spaces which lurk at the edges of large rail yards (are you, perhaps, normal about trains?), I should perhaps explain how this is a problem at all. Because, well, aren’t tracks just numbered?

And indeed, usually they are. Occasionally even sequentially. Except when they’re not – almost all railway stations in Europe are a confusing, organically-grown mess, often more than a century old, and have had tracks removed, added, renamed, jumbled around, re-purposed, or even forgotten about entirely several times already.

So while tracks are usually numbered, sometimes track 3 simply no longer exists (but track 4 does). Or track 3 does exist, but is now only used for trains going through the station without stopping, and no longer has a platform next to it. Maybe there’s an additional track 1a at the far end of the platform with track 1. Or maybe track 1a just means one half of track 1, and the other half is called 1b. Maybe there is a track 108 next to track 8. Or maybe there is a track 101 at the far end of the station, for no obvious reason at all.

Or maybe you’ve stumbled into an unforgiving beast such as Kaiserslautern Hauptbahnhof, which has tracks 1 through 5, tracks 8 and 10, and also 39, 40, 41, 42, 45, and 120. Good luck guessing which of these are at the same platform (or even how the platforms are positioned relative to each other) if you’ve never seen it on a map. The signage at the station can only help so much.

So it seems worthwhile to invest a little time into this problem. Surely it’s possible to find an easier way to answer “are these two tracks next to each other” than fiddling around with maps and signage?

Related Work

The best all-in-one free tool available for navigating stations that I know of is KDE Itinerary. While it focuses on general travel-planning, ticket-bookkeeping, and basically absorbing every use-case of any train operator’s in-house journey app except literally selling you tickets, it also displays station layouts, which it sources from Open Street Map (OSM).

Platform numbers are highlighted, complex multi-level station building layouts are displayed nicely, and whereever it can it integrates real-time APIs of station operators to display things like which elevators are currently out of order. More importantly for our problem, it can also give a list of all tracks in the station; useful for multi-level stations like Berlin Hbf, where not all platforms are immediately visible on a map.

The official bahnhof.de website run by DB Station & Service, who operate the public-facing side of almost all train stations in Germany, likewise offer interactive station maps. These are again heavily based on OSM, but also source some of their information from elsewhere, presumably from DB itself. Notably, for larger stations they often include platform sections, information which is frequently absent in OSM. But more observations on that a little later.

Yet it misses out on some of KDE Itinerary’s nicest features: no live elevator status, no search for a given platform number, and no links to elsewhere. For a few stations, such as Berlin Haupt­bahn­hof, they additionally offer PDF maps, but these are rare.

Confusingly, they also don’t seem to advertise this website very much – multiple people I spoke to had no idea that it exists at all, even when they said it would’ve been very useful for them in the past.

Many other trip planning apps offer similar features, but are usually not nearly as sophisticated about it — e.g. Oeffi simply offers to open the station’s position in any third-party maps app that is installed.

Unfortunately, the only app I know to do actual in-station routing is SBB’s official mobile app, where that feature is limited to stations actually run by SBB itself.

And while all of them can be used to answer the question “are these tracks opposite of each other?”, they all do so as side-product of focusing on adjacent, more general themes. So why not build our own, special-purpose thing?

Getting raw data

Since using OSM data seems to work out nicely for KDE Itinerary, I will here continue with that idea.

There is one pretty obvious alternative: Deutsche Bahn does run its own Open Data portal. It does not contain much, but there is in fact a data set describing platforms [DBStus20a]. Unfortunately, it has last been updated in 2020, and there seems little hope of regular yearly updates returning (my email about this went unanswered); and unlike with OSM, where the platforms are part of a much richer mapping effort, this data stands alone — so if we did use it, there’d be no easy way to connect it to anything else.

Data Model

OSM has three kinds of objects: nodes, ways, and relations, which can all have tags attached to them. Additionally, there is a membership relation between objects. Train stations usually look something like this:

• Individual platforms each get their own object, which have a ref or local_ref tag giving their name (the difference between these two seems a little unclear; both are in frequent use for the same thing). For platforms which have no name (basically all in Germany) this is a semicolon-separated list of track numbers.
• Platform edges are often (but not as universally) also mapped, and are then set as members of the platform.
• Platforms are part of a relation, which collects everything belonging to an entire station (along with walkways, shops, stairs, buildings, etc.).
• Stations also usually have a single “meta” node, on which we can find the station’s name and various kinds of IDs. We will use the railway:ref tag, which contains “the operator’s internal abbreviation” of this station, and is widely mapped in OSM.
• Finally, some stations are more complex than others: they may be split into smaller stations, have a local tram stop associated with them, or any manner of other things, which all result in a deeper nesting of membership relations.

So all we have to is to find a station’s meta node, walk through all the potentially messy forest of objects it’s associated with, and then grab everything that looks like a platform.

OverpassQL

It turns out that there is an entire language which exists solely to query data in OSM, which I had never encountered before.

Some of its design choices feel a little arcane (why does it limit almost all functions to names which are at most three characters long?), but otherwise I found it very pleasant to use.

Writing Queries

A first attempt, using München Haupt­bahn­hof as a test case:

[out:json][timeout:25];
node["railway:ref"="MH"]["operator"~"^DB"];
rel["public_transport"="stop_area"](<);
nwr["railway"="platform"](>>);
out geom;

It’s pretty easy to read, line-by-line: first we define general query parameters, such as the output format. After that, each line defines a statement which selects something.

1. Select any node with railway:ref set to MH whose operator’s name starts with “DB”; this is the station’s meta node. We can filter on tags with either = for simple equality, or ~ to match against a regular expression.
2. Then “walk up” the graph that node’s membership relations by one step (<) to find the station’s meta-relation, which is tagged with "public_transport"="stop_area".
3. “Walk down” membership relations (>>), and select everything that is tagged as a platform anywhere below this relation.

By default, the data “flows” implicitly from one line into the next: each statement assigns its output to the “default variable” ._, and the next statement reads from it. We could also write these explicitely and get a query with the same semantics, like this:

[out:json][timeout:25];
node["railway:ref"="MH"]["operator"~"^DB"] -> ._;
rel["public_transport"="stop_area"](<._) -> ._;
nwr["railway"="platform"](>>._) -> ._;
out geom;

In a bit, we’ll see more complex queries use custom variables.

For running test queries like this, it’s best to use overpass turbo. Note that queries which don’t return any geographical features won’t show up on the map; if your query seems to return an empty result, switch to the data tab on the top right to see its result. On the other hand, should a query result in an error, that is always shown.

For now, our query does not even find everything in München Hbf’s upper floor: both wing stations along with the underground S-Bahn station, are missing.

Wing stations are modelled in one of two ways: there is the railway:ref:parent tag used for for stations which have a clear hierarchy between them: KKDT ‘belongs’ to KKDZ, MH N and MH S ‘belong’ to MH, etc.

Let’s incorporate that:

[out:json][timeout:25];
node[~"railway:ref|railway:ref:parent"~"^MH$"]["operator"~"^DB "];
relation["public_transport"="stop_area"](<);
nwr["railway"="platform"](>>);
out geom;

This does almost the same as before, but in the first statement the simple filter for equality of a tag has been replaced with one which matches tags against a regular expression, marked as such by having a ~ in front.

This now catches both wing stations. But the S-Bahn is still missing.

To handle large, grouped stations, we need to more fully traverse the tree:

[out:json][timeout:25];
node[~"railway:ref|railway:ref:parent"~"^MH$"]["operator"~"^DB "];
relation["public_transport"~"stop_area|stop_area_group"](<<);
nwr["railway"="platform"](>>);
out geom;

The above now uses the << operator, which gives the transitive closure of membership relations upwards. This does indeed now find everything we wanted – but using it is expensive: recursively walking upwards winds up traversing through a lot of things we aren’t at all interested in. Unfortunately, there seems to be no way to give a limit to that operation, nor can it filter out things as it goes along: it first traverses everything, and only then starts applying the filter.

This is bad. The query now takes several (5-15) seconds, even for relatively small stations.

Instead, we can use the rel(bn) and rel(br) functions, which at least limit use to nodes and relations “on the go”, so we won’t needlessly walk along ways (in our case, ‘ways’ have an unfortunate tendency to model entire railway lines, greatly adding to needlessly-traversed data).

[out:json][timeout:25];
node[~"railway:ref|railway:ref:parent"~"^MH$"]["operator"~"^DB "];
rel["public_transport"~"stop_area|stop_area_group"](bn) -> .a;
rel["public_transport"~"stop_area|stop_area_group"](br.a) -> .b;
(.a;.b;);
nwr["railway"="platform"](>>);
out geom;

This still finds the same as above, but takes much less time to run. It now also uses named variables: to get both nodes and relations above the original meta node, we first walk up to nodes using rel(bn) and assign the result to the name .a, then walk up from that to relations using rel(br.a), and give that the name .b.

The (.a;.b;) clause then merges both sets of objects, and assigns it back to the default variable ._ as normal, so the next statement can use it as its implicit input.

Testing

I deployed this version of the query to bahnhof.name almost a month ago now, and since then have received many pointers to stations where it failed to give any reasonable result.

In some cases this could be traced to things not being mapped in OSM at all — but often, there was something I had simply missed:

[out:json][timeout:25];
nwr[~"railway:ref|railway:ref:parent"~"^MH$"][operator~"^(DB|Deutsch)"];
(._;rel["public_transport"~"stop_area|stop_area_group"](bn);) -> .a;
rel["public_transport"~"stop_area|stop_area_group"](br.a) -> .b;
(.a;.b;);
nwr[railway=platform](>>);
out geom;

Two major changes: one, there’s now an additional merge operation to keep the original meta node when using rel(bn); otherwise some information gets lost in a few cases.

The second change is perhaps more significant: the match against the operator tag has become more complex — because although all passenger stations “run by Deutsche Bahn” are operated by DB Netz AG on the railway-infrastructure side and by DB Station & Service on the passenger side, there is no consensus at all how to refer to this situation in OSM. The operator tag thus might contain variants of “DB Netze” or “DB Station & Service”, or merely “DB” or “Deutsche Bahn”, or anything else vaguely along these lines.

Since we use the railway:ref tag to identify stations, and these identifiers are specific to Deutsche Bahn, there isn’t really much we can do about this situation other than attempting to match as many variants as we can.

Catch them all?

It would be good to have some measure of certainty about how well this query works, as an attempt to measure its usefulness. What percentage of stations actually have platforms mapped and are found by our query?

Names are hard

So far, I’ve avoided talking about how we find the stations in the first place: via their Ril100 code, which in OSM is contained in the railway:ref tag. Initially I wrote the queries to work with these because it was convenient — I am familiar with the codes for stations I visit often, and they avoid dealing with the fuzzier problems of station names.

Unfortunately, these codes have a major disadvantage: they are operator-specific, in our case to Deutsche Bahn. Within Germany, this seldom matters — almost all German station either are or historically were operated by Deutsche Bahn, and thus have their own unique Ril100 code.

But OSM is a global project — and figuring out if what is contained in railway:ref is a Ril100 code or something specific to another operator isn’t trivial; limiting the query to stations run by Deutsche Bahn is only a (bad) approximation. Nor can we assume that railway:ref is a Ril100 code for every station in Germany, and for none outside it — for complicated reasons, national rail operators sometimes operate stations outside their ‘home’ country; DB’s best-known example of these is probably Basel Badischer Bahnhof, which is in Switzerland.

Annoyingly, this means that all other stations anywhere else in the world are now automatically out of scope – and even the (rare) stations within Germany not operated by some branch of DB are missing.

So many Betriebsstellen

Even then, attempting to get a hold of the actual stations can be a little fuzzy. There is a complete list of all Ril100 codes [DBNetz21], but it does not match the usual intuitive meaning of “(passenger) railway stations”.

Luckily, DB Station & Service also publishes a list of what they consider “stations”, also using Ril100 codes to identify them [DBStus20b]. This list comes much closer to what we want. But it’s important to keep in mind that it’s still not a complete match — though uncommon, there are still many stations or stopping points in Germany which are not operated by DB Station & Service, which are not included here. Especially smaller, regional stations are thus underrepresented.

Some Results

For the 5392 Ril100 codes contained in [DBStus20b], the query returned a non-empty result containing at least one platform for 3551 stations, of which 3071 contained at least one ref or local_ref tag. So at least we got over half of them — but there’s still a large gap.

It is, of course, somewhat hard to judge what happened with the 1841 stations for which nothing is found at all. At a glance, these skew heavily towards smaller stations — but I’ve not yet had time to go through at least some of them manually, and check whether they are simply not mapped at all in OSM, or mapped in some unexpected way which the query could not find.

More interesting are the 480 stations for which the query did find at least one platform in OSM, but with no ref or local_ref tag. What sort of stations are these?

Well, the majority (383 of them) have only a single platform, so the entire question of which connections are cross-platform becomes rather trivial.

The remaining 97 are a haphazard mix of still very-small stations, which are otherwise well-mapped but, for whatever reason, simply lack track numbers. The largest two, with four platforms each, are Herlasgrün and Großkorbetha, both in Saxony.

Somewhat hilariously, DB Station & Service’s official bahnhof.de website doesn’t have track numbers for these on their maps, either, perhaps suggesting that (at least for smaller stations) they do source their data entirely from OSM and do not have their own maps. On the other hand, we can definitely rule out that these tracks simply lack numbers entirely: on their accessability info, there is a list of tracks for both Herlasgrün and Großkorbetha – just without any information on which track is where.

Bahnhof.name

As mentioned, I happened to already have a small web service for quickly looking up Ril100 codes at bahnhof.name. At first I extended it with platform data by simply adding a static data set containing all platform data I could find in OSM — but it turns out you can’t simply publish a service backed by OSM data without people finding & fixing mistakes in the data it displays. So soon enough, I got people asking, “hey, how long will changes take to show up?”

So it now does dynamic updates instead, and caches results for a week. For no particularly well-explainable reason, I also decided this was a good opportunity to rewrite the entire thing in Haskell (before that and for even less-clear reasons, I’d originally written it in Gleam).

As a result, you can now look up a station’s platforms via e.g.

→ https://bahnhof.name/MH/tracks

for München Hauptbahnhof. If you find a mistake in OSM and decide to fix it, first of all, great! You can use

→ https://bahnhof.name/MH/fetch

to forcibly invalidate the cache and re-fetch platform data from OSM afterwards.

Possible improvements

There are some exciting ways in which this could be extended:

For one thing, lifting the restriction on DB-run stations would be great. This shouldn’t be too hard — if push comes to shove, one can always look up a station by its name — but neither is it entirely trivial. As mentioned, there are more universally applicable station ID standards (in the shape of IFOPT or UIC numbers) — but so far, these are not as widely used in OSM.

Much more complicated (but very useful) would be an attempt to create a kind of inside-station routing engine, akin to that which the SBB’s app already has. As far as I’m aware, this is probably not possible with the data that is (currently) contained in OSM. Perhaps it would be possible to integrate data from the official NeTEx data set — but matching that against the OSM data looks like a daunting task.

Finally, for now there is one thing missing entirely: information on platform sections (in Germany, usually designated A through G, with fewer sections on shorter platforms). These would be especially useful, as many other passenger information systems will tell you in which platform section your carriage will stop — but for now, these are seemingly not modelled in OSM at all, and I don’t even know where I’d begin if I wanted to add them.

EDIT: the above is incorrect! Thanks to trissc̈hen for pointing out to me that the railway:platform:section tag exists, which I’d overlooked before.

Conclusion

Go have fun, and hopefully worry slightly less about your tightly-planned travel itinerary!

I might revisit this some other time, or implement some of the ideas in the section above — but well, we’ll see, and for now I’ll make no promises. In the meantime, if you have ideas or improvements, feel free to poke me. I’ll also gladly accept patches for bahnhof.name’s source code.

Many thanks to many wonderful friends who helped point out things to me on the way, to Moira for patiently answering my questions about what kinds of Betriebstellen exist, to networkException for all their suggestions and for getting me to do live updates, to FireFly and some friendly dragons for listening to all my ramblings about NeTEx, OSM, and obscure stations (and asking helpful questions along the way), to Fynn and everyone else who pointed out mistakes in the initial query’s results, and especially thanks to everyone who has contributed to the station data contained in OSM!

References

This is just a short post explaining how the phone network used by the VOC at 22f3 worked.

Note: this post is provided “as is”, with no assurance of correctness of any kind. Be aware that it was written by someone who, three weeks ago, didn’t know a thing about how any of this stuff worked, and that most of it is the result of a single late-night config-file hacking session. It’s meant to be notes for myself should I need such a setup again as much as it’s meant to be an explanation for others.

Why?

Traditionally, there is a lot of DECT on Congress, run by eventphone / the POC and used by all kinds of beings for all sorts of things. But 22f3 was not Congress (nor was it trying to be); there was no visitor-facing infrastructure for calling each other, and if people in the event’s orga had to talk to someone not in the same room, they mostly used old-school walkie-talkies, which also get the job done (but don’t need any physical infrastructure at the location, and so could be used from early stages of buildup right through the event to the end of teardown).

But seen from the VOC, walkie-talkies have a couple disadvantages:

• even if you turn off all their beeping, walkie-talkies are noisy, which is not a good thing to be if you are trying to be quiet while recording a talk.
• their sound quality tends to be bad, and especially when people try to talk quietly, understanding each other is difficult
• this maybe could’ve been mitigated by using earpieces, but we had none
• they generally cause anxiety, especially if there’s neurodivergent people around (hi!)

Yet since a surprising room plan change the week before the event meant that the lecture halls and backoffice were all pretty far apart, we needed some way of contacting everything that was quicker than running up and down four flights of stairs.

So, phones.

Overview

The standard c3voc room case already contains a snom IP telephone, so we had one for each of the lecture halls where we had a recording setup. I also got two more cheaply (10€) from Ebay Klein­anzeigen.

The VOC’s main use case for these is that if something breaks in a lecture hall and the people there don’t know how to fix it, they can call our backoffice for help.

Less important but still nice to have is the reverse direction: if the backoffice notices that something is off, it can also call the lecture halls. The ring tone on the snoms can be set to a very quiet “ding”-sound — and while we never actually had to call a lecture hall during a talk, it’s still nice to know you could do so if necessary without causing further disturbance.

A telephone server

We used asterisk running on a NixOS machine as an SIP server to let the phones talk to each other.

Asterisk is — in both good and confusing — software that was conceived in the 90ies, so it expects lots of config files in a self-defined format under /etc, and also while using it you can occasionally smell the C. It comes with exhaustive documentation, which is frequently helpful, but in some corners itself notes that it’s partially incomplete, unwritten, or that some passages have probably fallen out of date.

Luckily, NixOS comes with a corresponding module, and the following is enough to start it:

{ config, lib, pkgs, ... }:
{
  services.asterisk = {
    enable = true;
    confFiles = {
      # config files go here
    };
  };
  # we had a sepearte VLAN for this, so *shrug*
  # makes things easier if I don't have to keep track of ports
  networking.firewall.enable = false;
}

If you add the snippet above to your configuration and switch to it asterisk won’t do much yet, but it also won’t fail on startup because NixOS gives it a couple default config files. It will still try to load a lot of unconfigured modules which fail immediately, but these errors are non-fatal and can be safely ignored.

One thing to note: Asterisk can live-reload its own config, and to avoid breaking things NixOS won’t automatically restart asterisk when doing nixos-rebuild switch. This is very sensible, but it does also mean you shouldn’t forget to manually restart asterisk or make it reload its config before wondering why a deploy didn’t do anything.

What’s a phone call?

Asterisk is an amazingly powerful piece of software, and can apparently do everything from interacting with actual, analog telephone hardware to shepherding WebRTC sessions. Unfortunately this generality also means it has no simple, high-level concept of what a “phone call” is.

It does know things called “channels” and “bridges”. I recommend reading the Asterisk Architecture section of the “Getting Started”-Guide before attempting to do anything with it.

PJSIP

Asterisk has (at least) two ways of interacting with SIP: the older chan_sip module, and the newer res_pjsip. The latter is recommended, but although the former is deprecated and will be removed in a future version, often the wiki’s examples still uses the former and only informally note what would change with PJSIP.

So here’s an annotated pjsip.conf example, patched together from the wiki:

; we use UDP for transport
[transport-udp]
type=transport
protocol=udp
bind=0.0.0.0

; Note: this defines a macro, to shorten the config further down
[endpoint_internal](!)
type=endpoint
context=from-internal
disallow=all
allow=ulaw

[auth_userpass](!)
type=auth
auth_type=userpass

[aor_dynamic](!)
type=aor
max_contacts=1


; here come the definitions for our phones, using the macros from above

; lecture hall 1
[saal1](endpoint_internal)
auth=saal1
aors=saal1
[saal1](auth_userpass)
; well, maybe set a better password than this
password=saal1
username=saal1
[saal1](aor_dynamic)

; lecture hall 2
[saal2](endpoint_internal)
auth=saal2
aors=saal2
[saal2](auth_userpass)
password=saal2
username=saal2
[saal2](aor_dynamic)

[backoffice](endpoint_internal)
auth=backoffice
aors=backoffice
[backoffice](auth_userpass)
password=backoffice
username=backoffice
[backoffice](aor_dynamic)

Note that a single phone usually consists of (at least) three things:

endpoint:

defines the “SIP account” and references the other two.

auth:

defines the authentication method, which here is just a password stored in plaintext.

aor:

defines how the server ought to reach the phone. Without further config, this is done dynamically—SIP clients register themselves when they start up, and the server remembers their IP address. But we could also just set a static IP here (useful if you can’t get a SIP client to register correctly — e.g. I didn’t get linphone to work except with static addresses), or do any number of other things.

A phone doesn’t have to have an aor associated with it — but without one, you have a phone that can only place calls, not receive any.

Extensions

This is all nice and fine, but so far we’ve not seen any phone numbers! These go into the extensions.conf config file, and define how asterisk should create bridges between the channels that PJSIP provides:

[from-internal]
; dial the lecture rooms & backoffice
; the syntax is NUMBER,SEQUENCE,FUNCTION
; to call someone do Dial(MODULE/account, timeout)
exten => 1001,1,Dial(PJSIP/saal1,20)
exten => 1002,1,Dial(PJSIP/saal2,20)
exten => 1600,1,Dial(PJSIP/backoffice,20)

; Dial 100 for "hello, world"
; this is useful when configuring/debugging clients (snoms)
exten => 100,1,Answer()
same  =>     n,Wait(1)
same  =>     n,Playback(hello-world)
same  =>     n,Hangup()
; note: "n" is a keyword meaning "the last line's value, plus 1"
; "same" is a keyword referring to the last-defined extension

That’s all we need, and now we can connect our phones!

Snom Snom

These snoms (we had one snom 300 and three snom 320) are suprisingly comfortable, no-fuss devices (though if you’ve bought yours used, you might have to figure out how to reset them first — for some reason, their manual doesn’t mention this). They can also run on PoE, which is especially useful if you’ve run out of power adapters.

On startup, they will display their own IP address (or, of unconfigured, ask you if you want them to use DHCP).

After that it’s easiest to use the web interface to configure them. Set the name, password & server of an identity, and enable that identity. If you expect to make calls across a NAT, also go to the NAT tab & set it to send keepalive packets every second or so.

Make sure to disable any other SIP accounts that might be configured (e.g. from previous events) and which you don’t need.

duut-duut-duut …?

You can ask the server which clients it knows by letting it display the current aors of PJSIP. It’s best to enter the asterisk cli for this:

❯ sudo asterisk -r
Asterisk 19.7.0, Copyright (C) 1999 - 2022, Sangoma Technologies Corporation and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Connected to Asterisk 19.7.0 currently running on televoc (pid = 704)
televoc*CLI> pjsip show aors

      Aor:  <Aor..............................................>  <MaxContact>
    Contact:  <Aor/ContactUri............................> <Hash....> <Status> <RTT(ms)..>
==========================================================================================

      Aor:  saal1                                                1

      Aor:  saal2                                                1

      Aor:  backoffice                                           1
    Contact:  backoffice/sip:backoffice@10.0.73.117:2048;li e3bba38c1f NonQual         nan


Objects found: 3

televoc*CLI> 

If a client has connected, their “Contact” field will include their IP address; in the above example, only the backoffice phone has registered successfully. Note that this does not necessarily mean the other phones can’t place calls, just that the server doesn’t know how to reach them if anyone else tries to call their extension.

There are many more useful things you can do in the asterisk cli interface; poking around it is definitely worth it. Just as an example, you can start ad-hoc calls, like this,

televoc*CLI> channel originate PJSIP/saal1 extension 100@from-internal

after which the saal1 phone should ring.

The wiki gives a few basic examples, but since most asterisk’s functionality is provided by individual modules, the actually useful examples are a little spread out. Most of the time it’s easier to find interesting things in the cli using tab-completion, and then using the built-in help function to find out what they do.

Conversly, with the above config you can test if a client can reach the server by dialing 100 and checking that it answers with the “hello, world”. If you’ve done it wrong, it’ll probably reject your call immediately, and the snom will display an SIP status code and error message on its display. Some of these may be familiar to you from HTTP, but probably not all — wikipedia has a complete list if you need it.

If you dial and nothing at all happens (not even a “dialing” or “hold” sound signal), then you’ve probably run into network issues. Perhaps there’s a NAT or a firewall in the way somewhere, or the snom is attempting to reach an IP where there’s no server that could answer?

Some recommendations:

• give all clients, and especially the server, static IP addresses, or make their DHCP leases static. This isn’t strictly necessary, but it makes fixing problems much easier
• if things don’t work for no apparent reason and were fine earlier, try rebooting the snom in question, or at least make it re-register (there’s a button for that in its web gui)
• send keepalive packets if there’s a NAT somewhere (once every couple seconds seems to work mostly fine), or the server won’t be able to reach the phone if someone attempts to call it. This will hopefully be enough; if not, there are entire subsections of the wiki dedicated to various NAT-related issues

Thinkpads make great servers

This is a sidenote, but: the asterisk server ran on an old T430 that the VOC had lying around, and which we kept in our backoffice the entire time. I can highly recommend doing this over running it on e.g. a single-board computer, simply because if in doubt, it comes with a keyboard and monitor built in, so you can always quickly jump into asterisk’s cli interface.

Just don’t forget to set it so it won’t go to sleep if someone closes its lid (and perhaps set the “Boot on AC Attach” BIOS option).

Future Work

There’s lots of other stuff asterisk can do, like DECT, or a dial-out & dial-in to a larger phone network, automatically redirecting calls if no one picks up, callgroups, …

Maybe I’ll look at some of those the next time we have a surprise telephone setup at some place where there’s no POC.

Conclusion

Overall, this setup worked pretty well. The stationary nature of the phones was seldom an issue, and the overall lack of DECT was less noticable than I’d feared.

We only made a few calls during the event, but they were a great help in resolving issues that came up, and made the overall work of the VOC much less stressful.

You might’ve also noticed that one of the snoms is still unaccounted for — I wrote at the beginning that we had four, and then only mentioned three of them in the config. Well, it turns out that eventphone also runs an SIP server, so we logged the last snom into theirs so that people on other chaos events could call us.

Of course, that does mean there was technically no need to run our own SIP server in the first place; we could’ve just used the eventphone server.

But consider this: it was fun doing so!

This is less a post than a couple of notes to myself; but perhaps they might also be helpful to others when starting out. I may extend or update it later.

Resources

General: along with the official Isabelle Homepage there is isabelle.systems, collecting helpful links to other sites.

Specific to Isabelle/ML:

• the Isabelle/ML Cookbook, which while incomplete (with only sporadic updates) is still the best beginner-friendly source to get started with many things
• Burkhart Wolff’s My Personal, Ecclectic Isabelle Programming Manual, which has much deeper information for a few topics

Basic Isabelle/ML

This is just a collection of helpful functions so I’ll have a place to look them up once I inevitably forget about them again.

How to execute code?

Useful antiquotations:

• ML‹code› executes the ML code; the surrounding theory can be accessed via antiquotations, especially @{context}. Some functions may complain about a “missing local theory context”; use these either from inside an Isabelle command or with local_setup
• local_setup‹code› expects the code to have type Proof.context => Proof.context, i.e. to modify the given theory context. Useful for hacking on functions that should eventually be called from an Isabelle command

Printing things

Printing any value that can reasonably be printed:

writeln (@{make_string} ...)

Pretty-printing terms (color-coding of variables depends on context):

Pretty.writeln (Syntax.pretty_term @{context} intr)

What are types?

Types are a simple ADT:

datatype typ =
  Type  of string * typ list |
  TFree of string * sort |
  TVar  of indexname * sort

• Type (name, args) is an instance of type name (the name is qualified, e.g. Nat.Nat instead of just Nat). If this type is parameterised, then its arguments go into args
• TFree (name, sort) is a type variable (e.g. 'a)
• TVar ((name,index), sort) are schematic variables which may occur e.g. in theorems and can be instantiated. Values with the same name but different index are not considered equal.

Additionally, there’s also Term.dummyT (which is really Type ("dummy", [])). This is used to leave the type unspecified. Terms using this type may lead to errors if passed to functions which do not expect them (but usually they are resolved during type inference).

Note that there is no extra builtin function type; a ⇒ b is represented simply as Type ("fun", [a,b]).

What are terms?

Terms are also simply an ADT implementing Isabelle’s take on the lambda calculus:

datatype term =
  Const of string * typ |
  Free of string * typ |
  Var of indexname * typ |
  Bound of int |
  Abs of string * typ * term |
  $ of term * term

• Const (name, type) are constants defined outside of the term (this may include function symbols, quantifiers, concrete values like true~/~false, datatype constructors, etc.)
• Free (name, type) are free variables (when pretty-printed in jedit, these are printed in blue)
• Var ((name,index),type) are schematic variables that can be instantiated, e.g. when they occur in a theorem (usually printed with a leading question mark); values with the same name but different index are not considered equal
• Bound index is a variable bound by a lambda in the same term, referred to using a deBruijn-index (dangling indices lead to errors). Note that in this case the variables type and name are still recorded, but as part of the lambda abstraction
• Abs (name,type,body) is the expression λname : type. body
• a $ b is function application

Truly a lot of ways to represent a variable!

Note that all names must always be qualified by theory name (antiquatations do this automatically — don’t forget to insert theory names when replacing one with a concrete term!).

What are sorts?

There is one important other detail: sorts. These give an extremely simple meta-logic over types: each sort is just a list of strings, and “subsumes” all lists which contain (at least) the same elements.

There is a “top” sort (the empty list), and apart from that sorts are used to implement things like locales (e.g. in a term a < b, the types of a, b should have a sort that contains "Orderings.org").

Most sorts are just ["HOL.Type"]. There is a \<^sort> antiquotation to write them more easily.

How to find things

To find a specific function (especially if it’s a basic, “obvious” function, like some variation of a map/fold), the best way is often to use ripgrep on Isabelle’s src directory with a likely name/type signature.

The best way to find theories/ML files is likewise with fd.

Use bat (or isabat) to browse these quickly; pass -l sml for files ending in .ML to get correct syntax highlighting.

Basic SML functions may be found in contrib/polyml-x.x/src/basis/*; many additional useful (general) functions are in src/Pure/library.ML.

Interactive exploring of ML files

Isabelle/jedit can provide “jump to definition” and similar features for ML (as it does by default for Isabelle theory files), but only does so for files explicitly loaded by some theory.

So to get these features e.g. in inductive.ML, do this:

ML_file ‹~~/src/HOL/Tools/inductive.ML›

(the double tilde is Isabelle’s home directory, value of $ISABELLE_HOME)

Sometimes this leads to errors (e.g. command defined twice); these are safe to ignore.

Layout of ML files

Most ML files in Pure/HOL define a main signature, and everything contained therein is available qualified with that file’s name, but capitalised; e.g. the result type defined in HOL/Tools/inductive.ML is available as Inductive.result.

Sometimes if an ML file is not part of any theory that is in Main; in that case, import the corresponding theory into your own, and that signature will be available.

aka: likely the worst ‘what is a monad?’-post it is possible to write.

tl;dr: at the very least, they’re tic tac toe-complete!

Introduction

A: I wonder if Nix is Turing-Complete?

B: The Nix Expression language? Sure it is: it has λ, and β-reduction, what more would you need?

A: Well – I guess so. But can we use it for other things than just describing how to build things?

B: Not sure. It’s meant to describe how to build things, not to support any extra language features that could get in the way.

A: But Guix does perfectly fine just using Scheme to define packages, and that’s a general-purpose language!

B: I guess so. Scheme does have things like I/O, and side-effects, and … – people just don’t use those when defining derivations.

A: Does Nix really not have I/O? It can read files, which sounds like input, and it can write files, which sounds a lot like output. It can even print to the console!

B: But all input has to happen when evaluation starts, all at once, I think.

A: Is that a problem?

B: Well, it different from other languages, where you could first print something (say, a question) then wait for more input (let’s hope it’s an answer) and then after that print some more stuff, and so on.

A: And you don’t believe we can do that in Nix?

B: No, I don’t think so. Can we?

Some Facts about Nix

A: Here’s some facts about Nix (the language):

1. It can read any file, from any path which it can construct as a string. Moreover, it can also import that file and evaluate it as an expression.

2. It can write files with any content it can construct, but not to arbitrary locations – only to set paths in the Nix Store.

And here’s some facts about Nix (the interpreter):

1. It cannot freeze the entire file system while it is busy evaluating some expression, nor can it take a snapshot of every file when it starts.

2. Which means that we can force it to read files that were created only after it started evaluating!

I think that’s already anough, actually. Do you see it?

B: Hm … not sure yet. But I did notice that (1) and (2) imply that Nix can interpret itself – there’s an eval function:

let
  pkgs = import <nixpkgs> {};
in
  eval = expression:
    import (pkgs.writeText "eval" expression)

A: Huh, I didn’t think of that. So we can just plug strings into eval and it’ll interpret them as Nix expressions?

nix-repl> eval "1"
[1/0/1 built] building eval1

[1 built]

Uhm …

B: Looks like that’s not what nix repl was designed for. It did work, it’s just glued the resulting 1 right into the middle of the build messages.

A: It’s easier to see for derivations:

nix-repl> eval "(import <nixpkgs> {}).firefox"
[2 built]
«derivation /nix/store/xp2ycak6xn4zhryvdwjdakgz5xmapqdk-firefox-89.0.drv»

Loopings and Undecidability

A: You gave me an idea, though: what happens if we evaluate this?

let loop = a: let file = eval (toString a);
              in if file == 1 then loop a else "done";
in loop 1

[2/3 built] error: stack overflow (possible infinite recursion)

Huh. Looks like Nix doesn’t have tail-call optimisation. And doesn’t it usually detect infinite recursion?

B: Why would need to have tail-call optimisation? And detecting infinite recursion in general is a hard problem, you can’t expect it to always work!

A: Hm.

B: What?

A: if file == 1 ... then loop a. That’s always true. We could put a more complicated condition there.

B: Like what?

A: Like, say, a Turing machine.

B: Why would you even – no, don’t –

A: – too late, already done it: turingmachine.nix

> nix-build turingmachine.nix
building '/nix/store/nm5kf2ybl29dsbj4l1d9bg6assivm9a1-now-state-0-went-left.drv'...
building '/nix/store/fhim7mirjgp3rliajiij9bd60sfxps3w-now-state-1-went-right.drv'...
building '/nix/store/wm930rdavr3vsn1r09zzh2d71yjvbfi5-now-state-2-went-left.drv'...
building '/nix/store/5z5p51a8lgxcgvr176l6gdsb2v0pyxj4-now-state-3-went-left.drv'...
building '/nix/store/xk8lrv8zgbdz20yggz52dx31g0pr3698-now-state-3-went-right.drv'...
building '/nix/store/cr8yl0bxs0v8ypp6w2md7andmky0qi3f-now-state-0-went-right.drv'...
building '/nix/store/x8aj86xjwzi63wv1qgfg473s8lxrfyla-now-state-1-went-right.drv'...
building '/nix/store/bdwi16w3kxil65phad8w6zc1s08si8xd-now-state-0-went-left.drv'...
building '/nix/store/8472d4sl3l9h6h0m1nyis3v2d6jpavrh-now-state-1-went-left.drv'...
building '/nix/store/fh5al936sl8phzpf1a9ycn64kmggzxmr-now-state-0-went-left.drv'...
building '/nix/store/df3mzfxygv50l8p218yxf3f31zqclc4r-now-state-1-went-left.drv'...
building '/nix/store/5jzb29yjy3lldhdh535n24m7mdgi56c7-now-state-0-went-left.drv'...
building '/nix/store/pdc7c4ssslyf14h4cdn99pwayjp55dvk-now-state-1-went-right.drv'...
building '/nix/store/ylvfk029nv6njd9g2kjjrqqrpy990dgh-now-state-2-went-left.drv'...
building '/nix/store/l3qr8rhih8syw1afv3b6sx7z19a2817r-now-state-3-went-left.drv'...
building '/nix/store/pf7znba9ipck4qnacjh3mwxdyl65kpkx-now-state-3-went-right.drv'...
building '/nix/store/s1y3jw3q8jam18g64xpk5f4prqamx76j-now-state-0-went-right.drv'...
building '/nix/store/xigm9h6w2vgbyjx16bs1srv2fkmdx976-now-state-1-went-right.drv'...
building '/nix/store/4bm2f2aqf76w8dckplxc9vx2lm1x3ysc-now-state-2-went-left.drv'...
...

B: Oh god. Well, on the other hand — if you still expect Nix to detect whether or not that thing leads to infinite recursion, you’re literally asking it to solve the halting problem.

basic I/O

B: Anyways, can we please stop this and go back to the actual topic of this post?

A: you mean, I/O in Nix?

B: sigh yes, I guess I do.

A: So, fact (4) says we can read files that were created after Nix already started evaluating. Here’s a particularly boring example:

builtins.readFile (pkgs.writeFile "nix" "Lorem Ipsum")

B: right – this is what’s called “import from derivation”, and it’s also how things like niv work: write files into the nix store (using a fetcher like fetchGit), then import them into Nix.

A: Sure, but actually we can just import any path – we’re not limited to just those that are within the nix store:

builtins.readFile "/tmp/hello-nix.txt"

If we evaluate that, and then very quickly create that file (or force Nix to do some lengthy operation first), then Nix will read something a user put somewhere, after the start of evaluation!

> nix-build -E 'builtins.readFile "/tmp/hello-nix.txt"' \
    & echo hello > /tmp/hello-nix.txt
error: expression does not evaluate to a derivation (or a set or list of those)

See? It complains that the result isn’t a derivation, but it doesn’t complain about a missing file.

B: You sure that it would complain otherwise?

A: Sure. Nix is untyped, how would it know that builtins.readFile doesn’t produce a derivation until it’s done evaluating it?

> nix-build -E 'builtins.readFile "/tmp/does-not-exist"'
error: opening file '/tmp/does-not-exist': No such file or directory

See?

B: Okay, so technically we can create files just when we start Nix. I’m not sure how that is useful for anything, though.

Waiting for Input

A: Well, let’s make Nix wait a little:

let
  pause = idx: pkgs.stdenv.mkDerivation {
    name = "sleep-${seed}";
    src = pkgs.hello; # just some dummy input
    phases = [ "buildPhase" ];
    buildPhase = ''
      # change idx if used more than once in the same file to wait every time
      # ${toString idx}
      sleep 2
      echo waiting 3 ...
      sleep 2
      echo waiting 2 ...
      sleep 2
      echo waiting 1 ...
      # without this, Nix would consider evaluation to have failed
      mkdir -p $out
    '';
  };
  pause_and = idx: code:
    pkgs.stdenv.mkDerivation {
      name = "pause_and${toString idx}";
      buildInputs = [ (pause idx) ];
      phases = [ "buildPhase" ];
      src = pkgs.writeText "code" code;
      buildPhase = ''
        # ${seed}
        cp $src $out
      '';
    };
in import (pause_and 0 "10")

Using that, we can even write a handy little read_input function:

read_input = idx: name: import (pause_and idx ''
  with import <nixpkgs> {};
  # ${seed}
  lib.readFile
    "/tmp/input-${name}"
'');

B: Why’s the seed variable in there? It doesn’t seem to be doing anything …

A: … except we can bind it to a different value each time we run the build, making sure Nix will re-run everything even if nothing else has changed and idx is the same.

Now we can use it like this:

  seed = toString 0;
in
  read_input 0 "test"

> nix-build &
> sleep 4 && echo "hello from the shell" > /tmp/input-test
[1] 21354
building '/nix/store/07l20zp51ghl88d4xrpr8sw744fpi43r-sleep-0.drv'...
building
waiting 3 ...
waiting 2 ...
waiting 1 ...
building '/nix/store/8ibcbgbc4g60xan05xa49qcvncnaz35p-pause_and0.drv'...
building
error: expression does not evaluate to a derivation (or a set or list of those)

[1]+  Exit 1                  nix-build scratch.nix

A: See?

Maximal Sharing

B: Quite. But what happens if you try reading in the same file twice?

A: uhm, well, let’s see …

let seed = toString 1;
in
  (read_input 0 "test") + (read_input 0 "test")

> nix-build
building '/nix/store/k56caykkwwc4dygxvnyq7dsrahz3j7bg-code.drv'...
building '/nix/store/ryh9f942r57ji220ccizscwypgdg8hbj-sleep-2.drv'...
building
waiting 3 ...
waiting 2 ...
waiting 1 ...
building '/nix/store/ldz27rinz91sws3zjlbq7grb42zmz6n7-pause_and1.drv'...
building
error: expression does not evaluate to a derivation (or a set or list of those)

Huh. Looks like both read_inputs waited at the same time.

B: Not quite. Nix implements what it calls maximal lazyness: if you do the same thing twice, it’ll only evaluate it once.

A: Huh? But that was what the idx was for! … oh, I forgot to set it to different values each time. My bad.

B: But even if you did set them to different values, which one would be evaluated first?

A: Well, we can make one of them wait extra long:

let seed = toString 2;
in
  print
    ((read_input 0 "test")
     + (pkgs.lib.readFile (pause_and 2 (read_input 1 "test"))))
    4

> nix-build
building '/nix/store/wva39fi81zr0f3hlav0dz129zpvwcyvm-sleep-4.drv'...
building
waiting 3 ...
waiting 2 ...
waiting 1 ...
building '/nix/store/3l768hhidqxj4qz3wn4xak170gp69j0v-pause_and0.drv'...
building
building '/nix/store/90d3hggp881kapmlpsbqd506rsb9q4h4-sleep-4.drv'...
building
waiting 3 ...
waiting 2 ...
waiting 1 ...
building '/nix/store/gvly9wd9yjlk5bwvy01dhahs2cz8j40d-pause_and1.drv'...
building
building '/nix/store/4fzcwfrhdm3014515frv7mx7ki5wqjvd-sleep-4.drv'...
building
waiting 3 ...
waiting 2 ...
waiting 1 ...
building '/nix/store/s661xcgx66igj3qy4haf22k4xj0fqvi7-pause_and2.drv'...
building
these derivations will be built:
  /nix/store/ki59fdjbihnza1zw7ygvp71ygwqlxp4d-print.drv
building '/nix/store/ki59fdjbihnza1zw7ygvp71ygwqlxp4d-print.drv'...
building
hello from the shell
hello from the shell

/nix/store/akxfzsq8wwi85pzidl8b4s7qi4q9p41w-print

B: … aaand it cached the file after the first read.

A: So no reading in the same file twice, I guess.

B: Maximal sharing can be annoying sometimes.

Chaining

A: I guess we could let read_input count how many times it’s been called by passing some sort of state around, and then read from a different file each time, like this:

let
  read_with_state = prompt: oldstate: {
    result = read_input oldstate.state (toString oldstate.state);
    state = oldstate.state + 1;
  };
in
  ...

And then using it we just have to be careful to always pass that state from one call to the next. It’s kinda like chaining them together, dragging a sequential order into the evaluator’s laziness:

let
  result1 = read_with_state "first" {state = 0;};
  first_input = result1.result;

  result2 = read_with_state "second" result1;
  second_input = result2.result;
in
  first_input + second_input

building '/nix/store/kqrc3nsxcm06f9924nxscvs8z2vsk52f-code.drv'...
building '/nix/store/qwhmkg3r6xmjzmf8yk5cp6z7gs1x0afw-sleep-5.drv'...
building
waiting 3 ...
waiting 2 ...
waiting 1 ...
building '/nix/store/48njzi1al7qgvrcyv4jy3ln5gg39jj1a-pause_and0.drv'...
building
building '/nix/store/qmvq75cls017pkq0bz1z5whxpyxqrv7q-code.drv'...
building '/nix/store/0gh8r26wjzciljxcnyq89bq5mj8ysll1-sleep-5.drv'...
building
waiting 3 ...
waiting 2 ...
waiting 1 ...
building '/nix/store/8jki0snrr21bh08pvy22afq3xgq3axp7-pause_and1.drv'...
building
"Hello from input 0\nHello from Input 1\n"

B: You know, at this point you can probably stop pretending and just admit that what you’re building is a monad.

An IO Monad

A: A what?

B: That pattern of your read_with_state function – the way it always needs to be called with some extra “state” value from the last time it was used, wrapping its value into an extra structure that we don’t really care about by itself? That pattern is what’s called a monad.

A: Huh.

B: See, we can define an operation that takes two of these functions and chains them together (which is usually called bind). For yours, it would look like this:

bind = monadic: operation:
  monadic // (operation monadic.result) monadic

A: Hang on — what happened to the state attribute? It’s not mentioned in your definition of bind.

B: Right, it isn’t — the assumption here is that monadic is one of these attribute sets which has result and state attributes as above, and operation is a function which takes a result, and then also takes the entire state, and then returns another monadic value with the same attributes. If we had types, we could say this function takes a value of a certain type, a function, and then returns another value of the same type.

A: But do we have types?

B: We don’t.

A: We don’t?

B: We don’t.

A: How annoying.

B: Yes. We’ll have to be very careful – if we do anything wrong, things might just blow up and produce errors after half the evaluation!

A: Aaaaa! Why don’t we have types?

B: Because we don’t.

A: Exactly why?

B: Just because.

A: Okay, I guess we don’t. … anyways, where were we?

B: Explaining the bind function.

A: Ah, right. Why does operation take the result value as a first argument? Seems like we could save an argument by simply passing it the entire attribute set only once, since it already contains the result.

B: In theory, we could — but we’ll see in a moment why it’s more convenient to do it this way.

A: Well, okay.

B: So now we can “bind” two reads together:

let
  double_read = bind
    (v: read_with_state "first")
    (v: read_with_state "second")
in
  double_read {idx = 0;}

Notation

A: What if we want to read in three things?

B: Yeah, uhm …

bind
  (v: bind
    (v: read_with_state "first")
    (v: read_with_state "second"))
  (v: read_with_state "third")
  {idx = 0;}

A: Well that’s awkward.

B: We can do better, though:

let do = operations: monadic:
      pkgs.lib.foldl bind monadic operations;
in
do [
  (v: read_with_state "first")
  (v: read_with_state "second")
  (v: read_with_state "third")
] {idx = 0;}

A: Huh. Looks almost a little like an imperative language now.

B: I guess so.

Monad Laws

A: Hang on a minute. If I look up “monad” with a search engine, there’s all this stuff about how it also needs some function called pure, and obey “monad laws”, and …

B: Ah sorry, I skipped over these. Whoops.

A: So what do these do?

B: pure is very simple: it just takes a value, and returns it wrapped into the monad, like this:

let pure = val: state: state // {value = val;}

In fact, it’s often also called just that: return! (and sometimes also unit)

A: Why would we ever need to do that?

B: We can use it to change the value “inside” our monad – that is, change the value that the next function gets:

do [
  (v: read_with_state "hello")
  (v: pure (v + ", world!"))
]

> nix-build &
> echo "hello" > /tmp/input-0
"hello, world!"

A: And the monad laws?

B: There’s three of them. Each is an equation of two expressions that should evaluate to the same thing:

1. Left Identity: bind (pure a) h is the same as just h a
2. Right Identity: bind m pure is the same as just m
3. Associativity: bind (bind m g) h is the same as bind m (v: bind (g v) h)

A: What happens if one of these three doesn’t hold?

B: Weird things.

A: Such as?

B: Well, mostly things won’t behave as you may expect them to — they’re not all that important here, but imagine we wanted an optimising compiler for Nix: perhaps we could use these laws for optimisations?

A: Perhaps. How do we know the laws are in fact true, though, for our definition of bind?

B: Well, if we had a formal semantics for Nix, we could write all the rules down and then prove from those no matter what the variables in the laws are, the laws will always hold.

A: Actually, we do have a formal semantics for Nix: it’s in Eelco Dolstra’s Thesis.

B: Looks out of date, though. Nix as described there only has the weird old-style let — no wonder, it’s from 2006!

A: Yeah, and I can’t really find a newer version, either.

B: It sounds like a lot of work, anyways.

A: What could we do instead?

B: Squint at the functions for a bit, then shrug, move on, and hope that it’s all right?

A: Sure, let’s do that instead!

Let’s play!

A: What we have so far is still a bit awkward to use. Sure, we can read in many things one after another, but it doesn’t seem like we can handle more than one thing at a time. What if we want to combine two inputs?

B: Yeah, that’s true – we can’t bind new variables on the fly, at least not without interrupting the nice do-way of writing things.

A: Hm, let me think … I’m pretty sure we can tweak the state of our monad a bit to simulate something like variables …

B: … I guess?

A: … it’d also be useful if we had more than just sequences of functions. What about loops, and conditions, and all that stuff? I’m sure we can add that to …

B: … I’m not sure that’s a good idea. Do you that’s even doable?

A: Yep! In fact, I’ve done it already: io.nix. I also wrote a version of tic tac toe in it:

...
{seed}:
with import ./io.nix {s = seed;};
do initialWorld [
  (v: assign "grid" emptyGrid)
  (v: assign "turn" "x")

  (while (v: !(hasWon "x" v.grid) && !(hasWon "o" v.grid))
    (ifThenElse (v: v.turn == "o")
      (v: doMonadic [
        (v: read_input "test")
        (ifThenElse (v: v.grid.${stripWhitespace v.input} == " ")
          (v: doMonadic [
            (v: assign "grid"
              (v.grid // { ${stripWhitespace v.input} = "o"; }))
            (v: print (showGrid v.grid))
            (v: assign "turn" "x")
          ])
          (v: print "this is not a legal move, try again!")
        )
      ])
      (v: doMonadic [
        (v: assign "grid" (v.grid // { ${maximalMove v.grid} = "x"; }))
        (v: print (showGrid v.grid))
        (v: assign "turn" "o")
      ])
    )
  )
  (match (v: gameState v.grid) {
    "draw" = v: print "this game ended in a draw";
    "o" = v: print "o won this game";
    "x" = v: print "x won this game";
  })
]

A: Well, that’s only the Monad-part of it; there’s lots of helper functions. The entire thing is in game.nix.

It actually works, too!

B: sigh

A: Just run nix-build -argstr seed 4 on that, then write all of your moves into the tmp files it tells you to. But make sure to never take more than three seconds to decide your next move, or it’ll fail to read in the next file and everything will crash!

On the plus side, if it does crash, you can restart where you left of by just running the same command again – maximal lazyness shold take care of the rest.

I guess this should count as proof that, in addition to being Turing-complete, Nix is also tic-tac-toe complete?

B: … I don’t know why I put up with you.

Addendum

A: So, does all of this actually mean anything?

B: Apart from that Nix kinda has impure functions, if you squint hard enough?

A: Well, we already knew that before – even without impure things like builtins.fetchGit that don’t require hashes, reading in files from outside the Nix store without requiring hashes will always be impure. It’s just so happens to also be very convenient.

B: I guess so. Nix doesn’t usually come with an I/O monad, though.

A: Yeah, that’s true. Blame this post for the monad, if you like.