Adventures with the purely functional OS

NixOS Ecology
tersely

NixOS is a Linux distribution based on the Nix package manager

Nix supports Linux, OS X and FreeBSD

Nix uses the purely functional Nix Expression language to define packages

NixOps uses Nix to deploy NixOS based infrastructure to Amazon Web Services, Google Cloud Platform, VirtualBox, Hetzner and NixOS containers

DisNix uses Nix to deploy distributed services

Hydra is a continuous integration and build cache server for Nix

If I told you with NixOS

For those that come from Windows, there are no software installers

For those from OS X, there is no App Store

For those from Linux, it is not FHS compliant
no /sbin /lib /lib64 /usr/local /usr/lib …
/etc off limits
only /bin/sh and /usr/bin/env

In general you can’t install arbitrary applications or libraries from the internet

You have to learn the Nix Expression language

You may have to learn a little bit about purely functional programming

If you get stuck you can’t just get the answer on AskUbuntu

You may have to add packages to the package manager

You might ask

The answer is

We do these things not because they are easy, but because they are hard

and because it makes the impossibly difficult easily possible

The Nix Package Manager

There is an excellent explanation of what the Nix package manager is on Sander van der Burg’s blog http://sandervanderburg.blogspot.co.za/2012/11/an-alternative-explaination-of-nix.html

and what follows is directly based on that

Tools to automate installing, upgrading, configuring, and removing software packages

It borrows concepts from purely functional programming languages to make deployment reliable, reproducible and efficient

Was initiated by Eelco Dolstra as part of his PhD research

Purely functional programming languages

Functions are found in many programming languages

But they are usually not the same as functions in mathematics

Leibnz’ principal 𝑥 ＝ 𝑦 ⇒ ⨍⟮𝑥⟯ ＝ ⨍⟮𝑦⟯ if two function arguments are identical then two function applications are too

Purely functional programming languages are not

This C code

int v = 0;

int add(int a)
{
    v = v + a;
    return v;
}

int main()
{
    int p = add(1); /*1*/
    int q = add(1); /*2*/
}

does not obey the Leibnz’ principal

because it allows side-effects to be programmed

meaning it lacks referential transparency

so function application cannot be replaced by its result

Purely functional programming languages

Obey the Leibnz’ principal
The result of function application depends only on its definition and its arguments

Side-effects are not allowed to be programmed (they are pure)

No variable, only identifiers to immutable objects

This all means they are referential transparent

Purely functional programming languages

Are often lazy

So expression only get evaluated when needed

Allows tying the knot used for cyclic data structures while retaining purity

Purely functional programming languages which are lazy

Have benefits

Results of evaluation can be cached

Evaluation only happens when the result is required

Referential transparency gives determinism allowing parallel evaluation

Purely functional package management

Treat deployment of packages as a function in a programming language

Conventional package management

In conventional package mangers

A “function” is representative of imperative languages like C

Execution can destructively modify other packages

Files installed in global locations (like /usr/lib), so dependencies could be found without being declared and its difficult for multiple version of a package to co-exist

Purely functional package management

In Nix
A build recipe / derivation is a function that describes how to build an artefact being invoked with its required dependencies as arguments
Build recipes defined with the Nix Expression DSL (which is a lazy purely functional language)

Build recipes are run in isolated environments limiting chances of side effects

Purely functional package management

In Nix

The result of each build is stored in a unique separate location

The storage locations are immutable

The storage identity is a cryptographic hash derived from all the inputs of the function

Unique locations give better guarantees against implicit dependencies sneaking through

Purely functional package management

This means in Nix

You get determinism, the same package is the same where or when ever you install it

You can have multiple versions of packages co-existing

You can safely upgrade any package and always rollback an upgrade

You can in-place upgrade packages

NixOS

NixOS is a GNU/Linux distribution built around the Nix package manager

It builds on Nix using the Nix Expression DSL to declaratively define every component including the kernel and configuration files

It uses systemd for services which are declaratively defined using the Nix Expression DSL.

It uses a source based model with binaries, that is cached binaries are used instead of building from source when possible
NixOS gives you determinism; the same configuration builds the same machine no matter when or where

NixOS is gives you relaible atomic upgrades

NixOS allows you to always rollback any changes

NixOS allows you to safely live upgrade your system

Nix Expression Language

It is a pure, lazy, functional language

Purity means no side-effects (e.g. no variable assignment)

Laziness means function arguments are only evaluated when needed

Functional means functions are first class and can be passed around and manipulated

The language is not full-featured and general purpose

Its aim is describing packages, compositions of packages and variability within packages

Comments

Comments can be single-line, started with a # character, or inline/multi-line, enclosed within /* ... */
# this is a line comment
/* this
is a multiline comment */

Simple Values

  • The null value null
  • Integers 1234
  • Booleans true and false

  • Strings

    "single line string"
''
  First line of a multiline string.
  Second line
''
"this is an ${antiquote} in a string"

    • as a convenience URIs don’t need quotes

      http://example.org/foo.tar.bz2

  • Paths /bin/sh or ./builder.sh

Lists 

[123 ./foo.nix "abc" (f {x = y;}) ]
  • heterogeneous list
  • white space delimited list of values
  • enclosed by square brackets

Sets 

{ x = 123;
  text = "Hello";
  y = f { bla = 456; };
}
  • heterogeneous dictionary
  • list of name value pairs called attributes
  • delimited by semicolon
  • enclosed by curly brackets

  • attributes selected using .

    { a = "Foo"; b = "Bar"; }.a

  • in recursive sets elements can refer to each other

    rec {x = y; y = 123;}.x == 123

Let-expressions

Allows you to introduce local variables for an expression

let
  x = "foo";
  y = "bar";
in x + y

Inheriting attributes

When defining sets its often convenient to copy variable from the surrounding scope

let x = 123; in
{ inherit x;
  y = 456;
}

or from another set

let aset = {x = 123, q = 456}; in
{ inherit (aset) x;
  y = 456;
}

both evaluates to

{ x = 123; y = 456; }

With-expressions 

with e1; e2

introduces the set e1 into the lexical scope of expression e2

let as = { x = "foo"; y = "bar"; };
in with as; x + y

Functions

Functions take a single argument with multiple arguments supported through currying

Its defined as an argument pattern followed by a body pattern: body
Function application is white space f x
let negate = x: !x;
    concat = x: y: x + y;
in if negate true then concat "foo" "bar" else ""

Functions

set patterns are supported

{ x, y, z }: z + y + x

including variable binding, default values for missing keys and partial matches

args@{ x, y ? "default", z, ... }: z + y + x + args.a

Conditionals

Doesn’t really need explaining

if e1 then e2 else e3

Operators 

e.attrpath    # select attribute
e1 e2         # call function
e ? attrpath  # test attribute existance
e1 ++ e2      # list concatenation
e1 + e2       # list or path concatenation
!e1           # boolean negation
e1 // e2      # set union
e1 == e2      # equality
e1 != e2      # inequality
e1 && e2      # logical AND
e1 || e2      # logical OR
e1 -> e2      # logical implication (!e1 || e2)

Default builtin functions

  • There are some default builtin functions
  • The most important being import
    • which loads a Nix Expression from a file
    • and if the expression is a function
    • allows you to call it immediately

Nix Hello World

Nix expressions that actually get built are derivations

Here is the example derivation for building GNU Hello

{stdenv, fetchurl}:

stdenv.mkDerivation {
  name = "hello-2.6";

  src = fetchurl {
    url = ftp://ftp.gnu.org/gnu/hello/hello-2.6.tar.gz;
    sha256 = "1h6fjkkwr7kxv0rl5l61ya0b49imzfaspy7jk9jas1fil31sjykl";
  };

  meta = {
    homepage = http://www.gnu.org/software/hello/manual/;
    license = "GPLv3+";
  };
}
{stdenv, fetchurl}:

stdenv.mkDerivation {
  name = "hello-2.6";

  src = fetchurl {
    url = ftp://ftp.gnu.org/gnu/hello/hello-2.6.tar.gz;
    sha256 = "1h6fjkkwr7kxv0rl5l61ya0b49imzfaspy7jk9jas1fil31sjykl";
  };

  meta = {
    homepage = http://www.gnu.org/software/hello/manual/;
    license = "GPLv3+";
  };
}
  • function taking a set with attributes stdenv and fetchurl
  • returns a derivation
    • specifying the package’s name, source and meta info
    • not specifying build procedure so standard Autotools will be used ./configure; make; make install

to build hello we compose its function with its arguments

rec {
  stdenv = ...;
  fetchurl = import ../build-support/fetchurl {
    inherit stdenv curl;
  };
  hello = import ../applications/misc/hello {
    inherit stdenv fetchurl;
  }
  ...
}
  • The hello attribute is bound to the function defined earlier and invoked with its arguments
  • The dependencies of the function are themselves defined in the set
  • Essentially in Nix one recursive set defines all packages and dependencies in terms of each other

When you install hello the following paths will be added to the Nix store

  • The derivation that describes how to build /nix/store/vh0mxra4c7jrdg2fi5yn83k5kwdflsbx-hello-2.6.drv
  • The source code fetched that was fetched /nix/store/agjbqxl2n6yhqwcarx9shdbsc13dgwk0-hello-2.6.tar.gz.drv
  • The actual final binary /nix/store/1gqj6zr1x0n812qxijy066fzrypgh3im-hello-2.6

Everything is cached so they won’t be downloaded again or build again if something else requires it

Nix Profiles

So if all artefacts live the Nix store won’t it make using it difficult

The answer is Nix profiles and again a very good explanation can be found on Sander’s blog http://sandervanderburg.blogspot.co.za/2013/09/managing-user-environments-with-nix.html

Nix profiles are user environments exposing a generation of installed packages to a user

  • Packages can be installed into profiles using
    nix-env command line utility

    $ nix-env -i zip unzip

  • Installed packages can be run as normal without specifying paths with hash codes

    $ zip --version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.

When you install packages a new profile is created

A profile is a symlink tree synthesizing the contents of the currently installed packages

A user environment / profile is also a Nix package residing in the store

  • A generation symlink is generated (profile-1)
  • There is a default symlink (profile) pointing to the currently active generation
  • The default symlink is also referenced from the user’s home directory through ~/.nix-profile
  • Paths are added to the user’s PATH by e.g. export PATH=~/.nix-profile/bin:$PATH

Deployment actions are actually non destructive

  • if you install or remove packages with nix-env new profiles are created
  • you can always revert to a previous profile using nix-env --rollback
  • to actually remove old profiles
    nix-env --delete-generations old
  • and to actually physically remove files from the store nix-collect-garbage

A minimal NixOS machine

{ config, pkgs, ... }:
{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  # Use the GRUB 2 boot loader.
  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;
  boot.loader.grub.device = "/dev/sda";

  networking.hostName = "example-machine";

  # List packages installed in system profile
  environment.systemPackages = with pkgs; [
    wget
    termite
    nvim
  ];

  # List services that you want to enable:
  services.openssh.enable = true;
  networking.firewall.enable = false;

  # Enable the X11 windowing system.
  services.xserver.enable = true;
  services.xserver.layout = "us";

  # Enable the KDE Desktop Environment.
  services.xserver.displayManager.sddm.enable = true;
  services.xserver.desktopManager.plasma5.enable = true;

  # Define a user account. Don't forget to set a password with ‘passwd’.
  users.extraUsers.a_user = {
    isNormalUser = true;
    uid = 1000;
    home = "/home/a_user";
    extraGroups = ["wheel" "audio" "video" "networkmanager" "postgres" "vboxusers"];
    initialPassword = "a_user";
  };

  # The NixOS release to be compatible with for stateful data such as databases.
  system.stateVersion = "17.09";

  # virtual box related
  boot.initrd.checkJournalingFS = false;
  virtualisation.virtualbox.guest.enable = true;
}
{ config, pkgs, ... }:
{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];
......
  • The configuration is a function that lives in /etc/nixos/configuration.nix
  • It imports the auto generated hardware-configuration.nix
......
  # Use the GRUB 2 boot loader.
  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;
  boot.loader.grub.device = "/dev/sda";

  networking.hostName = "example-machine";

  # List packages installed in system profile
  environment.systemPackages = with pkgs; [
    wget
    termite
    neovim
  ];
......
  • some boot options
  • the machine name
  • some packages to be installed in the root profile
......
  # List services that you want to enable:
  services.openssh.enable = true;
  networking.firewall.enable = false;

  # Enable the X11 windowing system.
  services.xserver.enable = true;
  services.xserver.layout = "us";

  # Enable the KDE Desktop Environment.
  services.xserver.displayManager.sddm.enable = true;
  services.xserver.desktopManager.plasma5.enable = true;
......
  • list some services to have enabled
  • specify the desktop environment
......
  # Define a user account. Don't forget to set a password with ‘passwd’.
  users.extraUsers.a_user = {
    isNormalUser = true;
    uid = 1000;
    home = "/home/a_user";
    extraGroups = ["wheel" "audio" "video" "networkmanager" "postgres" "vboxusers"];
    initialPassword = "a_user";
  };
......

Define the users of the system

  • If you want to try out changes without making it permenant
    sudo nixos-rebuild test
  • If you are happy and want to switch to the new configuration
    sudo nixos-rebuild switch
  • If you want to rollback changes
    sudo nixos-rebuild rollback
  • If you accidentally make your system non-bootable you can still boot into any previous config from the grub menu.

Developing with Nix

When developing software you want environments setup with all your development dependencies present

If you work on different sets of software you don’t want the dependencies to interact

In Nix you use the nix-shell command to enter a development environment with all your dependencies setup.
Usually a software package like hello are defined by a default.nix in a folder named after it

This file defines a function though that needs to be invoked with its environment

This is where the shell.nix file comes in which nix-shell uses by default

The shell.nix file sets up the environment and then calls the default.nix file

  • directory hello

  • default.nix

    {stdenv, fetchurl}:

stdenv.mkDerivation {
  name = "hello-2.6";

  src = fetchurl {
    url = ftp://ftp.gnu.org/gnu/hello/hello-2.6.tar.gz;
    sha256 = "1h6fjkkwr7kxv0rl5l61ya0b49imzfaspy7jk9jas1fil31sjykl";
  };

  meta = {
    homepage = http://www.gnu.org/software/hello/manual/;
    license = "GPLv3+";
  };
}

  • shell.nix

    let
  pkgs = import <nixpkgs> {};
in import ./default.nix {inherit (pkgs) stdenv fetchurl;}
let
  pkgs = import <nixpkgs> {};
in import ./default.nix {inherit (pkgs) stdenv fetchurl;}
  • import the top level nix packages expression
  • import ./default.nix
  • call the function passing in stdenv and fetchurl from the top level nix expression

In the hello directory run nix-shell

  • This drops you in a development shell
  • All your build dependencies are present
  • to unpack the fetched source run unpackPhase
  • then cd into the source directory and run ./configure and ./make

Adhoc Build Environments

Sometimes you just want adhoc build environments in which to experiment
You can easily do this by creating “fake” build recipes and using nix-shell to setup your environments

Haskell Hakyl Environment 

let
  pkgs = import <nixpkgs> {};
  haskellPackages = pkgs.haskellPackages.override {
    overrides = self: super: with pkgs.haskell.lib;{
      ghc-syb-utils = dontCheck super.ghc-syb-utils;
  };};
  ghcWithPackages =
    haskellPackages.ghcWithHoogle (g: with g;
    [classy-prelude hakyll hakyll-favicon hakyll-filestore
     hakyll-ogmarkup hakyll-series base pandoc pandoc-types
    ]);
in with pkgs;
  runCommand "hakyll-env"
    (with ghcWithPackages.haskellPackages;
      rec
        { ghc = ghcWithPackages;
          shellHook = "eval $(egrep ^export ${ghc}/bin/ghc)";
          buildInputs =
            [ ghcWithPackages zsh ghc-mod hindent cabal-helper
              cabal-install codex stylish-haskell hoogle hlint
              align netcat-openbsd hdevtools];})
  "echo success > $out"

Haskell Hakyl Environment 

let
  pkgs = import <nixpkgs> {};
  haskellPackages = pkgs.haskellPackages.override {
    overrides = self: super: with pkgs.haskell.lib;{
      ghc-syb-utils = dontCheck super.ghc-syb-utils;
  };};
...
  • we import the top level nix expression
  • we then override the ghc-syb-utils package skipping its test suite

Haskell Hakyl Environment 

  ghcWithPackages =
    haskellPackages.ghcWithHoogle (g: with g;
    [classy-prelude hakyll hakyll-favicon hakyll-filestore
     hakyll-ogmarkup hakyll-series base pandoc pandoc-types
    ]);

Then we create a special GHC that has the specified libraries available along with a custom Hoogle and documentation

Haskell Hakyl Environment 

in with pkgs;
  runCommand "hakyll-env"
    (with ghcWithPackages.haskellPackages;
      rec
        { ghc = ghcWithPackages;
          shellHook = "eval $(egrep ^export ${ghc}/bin/ghc)";
          buildInputs =
            [ ghcWithPackages zsh ghc-mod hindent cabal-helper
              cabal-install codex stylish-haskell hoogle hlint
              align netcat-openbsd hdevtools];})
  "echo success > $out"
  • we specify our “fake” derivation
  • we setup some environment variables for GHC on shell entry in shellHook
  • we list all the tools we want in our buildInputs attribute

Clang Environment 

let
  pkgs = import <nixpkgs> {};
in with pkgs;
clangStdenv.mkDerivation rec
{ name = "cpp-play-${version}";
  version = "0.0.0";
  buildInputs =
    [clang clang-tools llvmPackages.libcxx llvmPackages.libcxxabi glibc gdb];
  CPLUS_INCLUDE_PATH =
    "${llvmPackages.libcxx}/include/c++/v1:${clangStdenv.cc.libc_dev}/include";
  C_INCLUDE_PATH = "${clangStdenv.cc.libc_dev}/include";
  LD_FLAGS =
    "-L${llvmPackages.libcxx}/lib -L${llvmPackages.libcxxabi}/lib";
}

all attributes are defined as environment variables in the shell

antiquoting a package results in its final stored location

Pegging Build Environment

For production development you want to ensure that all developers use the same top level nix expression

Given file hello/pegged-nix.nix

# helper to fetch nix-expression
let fetchFromGitHub = (import <nixpkgs> {}).fetchFromGitHub;
# fetch the nix expression from github at the specified revision
in import
    ( fetchFromGitHub
      {
        owner = "NixOS";
        repo = "nixpkgs";
        rev = "2839b101f927be5daab7948421de00a6f6c084ae";
        sha256 =
          "0a863cc5462gn1vws87d4qn45zk22m64ri1ip67w0b1a9bmymqdh";
      }
    ) {}

Then hello can be called in hello/shell.nix using the pegged nix packages

let
  pkgs = import ./pegged-nix.nix ;
in import ./default.nix {inherit (pkgs) stdenv fetchurl;}

How We Use It at Work

  • we all run NixOS either natively or in a VM
  • We develop and deploy our Haskell code using Nix
  • We deploy to Heroku using Nix and Hydra
  • We setup our Node.js and Elm development environments using Nix

Final thoughts

  • There is much more, we have only scratched the surface
  • The documentation is good and getting better https://nixos.org
    • You can actually search configuration attributes
    • You can search packages too
    • Arch Linux’s Wiki is really helpful too.
  • Your are not limited to packages build from source, third party binaries are installable
    • using patchelf or
    • emulating FHS
  • It is really nice if you want a slightly unconventional setup (XMonad)
  • It makes it easier to experiment with new hardware like I’ll supported laptops.
    • Easily iterate configurations tills something works without worrying about bricking it
  • When your laptop gets stolen its really nice to get back up and running quickly