What is NixOS ? It's a very different OS, unlike anything you've seen before!

Software
13

Some of this is already documented starting at this post - $500[ish] AI Build Challenge - #18 by Belfry, but I’ll cover the NixOS specific stuff here.

I did restart that machine from scratch using NixOS rather than Debian. I had some initial issues getting the nvidia drivers to work, but I think that more about my own unfamiliarity with NixOS than anything else. My use case for this build is a machine that has no desktop environment, and runs nothing other an ollama and the associated CUDA drivers, with the ollama API to be queried by OpenWebUI on a separate host. The hardware, etc. is discussed in that thread so I won’t cover it here. I also won’t reproduce my entire configuration.nix file below, but here are the few places I’ve deviated from the default post-install configuration.nix, and why:

 # Allow unfree packages
  nixpkgs.config.allowUnfree = true;

This is a tick box during the install. It’s required for the nvidia drivers and can be added if the box wasn’t ticked during install time. Even if open drivers are being used (see below), apparently some of the related components are still non-free.

  #Nvidia stuff
  hardware.graphics.enable = true;
  hardware.nvidia.open = false;
  services.xserver.videoDrivers = [ "nvidia" ];

The top and bottom lines tripped me up initially. hardware.graphics.enable is more about graphics acceleration than graphics themselves. Similarly, it took me a while to realise that the services.xserver.videoDrivers line was necessary as it was loading the nvidia drivers themselves. I had initially omitted this as I was running the machine headless (and therefore “don’t need xserver drivers” - a reasonable but entirely incorrect assumption). Despite the reference to X, this is still necessary to load the nvidia drivers at all. The middle line hardware.nvidia.open will be card specific. Newer cards apparently have quite good open drivers, but I’m using older hardware which need the non-free drivers.

  # Compile all packages with CUDA support
  nixpkgs.config.cudaSupport = true;

I deviated from @techman’s approach here. He has installed ollama-cuda whereas I have flagged that where any package exists that has a cuda variant, the cuda variant should be used system wide. It achieves the same thing (running ollama with cuda acceleration), but if there are any other packages in the chain that also have cuda support lurking somewhere, then those will also be compiled with cuda support.

Packages added were:
ollama (see above regarding ollama vs ollama-cuda)
nvtopPackages.nvidia (nvtop is handy to keep an eye on the card - this was a straight copy and paste from @techman’s config)
btop (unrelated to nvidia or ollama stuff, but btop is my favourite top)

#Ollama cudaArches workaround
  nixpkgs.config.packageOverrides = pkgs:
    { ollama = pkgs.ollama.override {
      cudaArches = [ "61" ];
    };
  };

Discussed further in the $500[ish] AI build thread linked above. Specific to my cards I’m playing with (GTX1050Ti and Tesla P4), and is a workaround for a quirk where NixOS was only compiling CUDA support for a subset of newer nvidia cards rather than all the cards that ollama supports.

  # Ollama config
  services.ollama = {
    enable = true;
    openFirewall = true;
    host = "[::]";
  };

Set up ollama as a service, enable it, open the firewall, and have it listen on all interfaces.

Lastly, I uncommented the line near the bottom of configuration.nix to enable the SSH server (headless machine, after all).

That’s it! So, how did it go in the Z220+GTX1050Ti?

I didn’t time it, but the NixOS install took a few minutes at most, I did the initial reboot, dropped my additional lines into configuration.nix as documented above, did a sudo nixos-rebuild switch, wandered off as everything compiled…

buildPhase completed in 11 minutes 7 seconds
checkPhase completed in 3 minutes 58 seconds

… and it just worked. No reboots, no nothin’. The config I got together on machine #1 (the P330) worked perfectly on machine #2 (the Z220) - same generation of nvidia card requiring the same CUDA/ollama workaround was a bit of a stroke of luck, admittedly. I honestly spent more time digging the Z220 out from under the house and putting the GTX1050Ti back in it than I did in the rest of the process combined.

Absolutely dead easy for someone who hadn’t even picked up NixOS until two days ago, spent an hour or so on a VM going “oh, this seems neat”, and then jumping head first into it this time yesterday.

For the benefit of others, I found this excellent post on NixOS package cleanup when trying to work my way through the nvidia driver and ollama issues. I went through and deleted all my superseded builds using this process last night and it was super easy.

Give NixOS a go, if you haven’t already!

My next steps will be to complete the rest of my assigned “HLB homework” and set up Caddy for a certificate for the Ollama API, set up a NixOS VM to get Tube Archivist going, and then save the config files in a git or fossil repository…

1 Like