Packing up Terraform

Software
1

Inspired by @jdownie and @mcrilly I have had a go at terraform. My use case is that of a base in which to start playing with kubernetes. @jdownie instantiates and tears down his vms with bash scripts but that seemed too hard for me to get going.

Ubuntu server 2404 is frequently used as the base distro for many kubernetes projects and so I went with that. Sam Gabrail has done a couple of recent videos on getting things up and running. He has associated blog posts and most importantly a github repository.

If you’re interested you can read through the blog posts. In retrospect I would be inclined to just use Sam’s packer install of the ubuntu server template “as is”. I mucked around and failed with creating my own user rather that the default (ubuntu: pw - ubuntu) and I used the latest version of ubuntu-24.04.3-live-server-amd64.iso not the version 2 that Sam uses. When instantiating a VM from a template an automatic upgrade runs on each boot so a slightly more recent base install makes little difference (and even less after the first boot).

The nice thing about building a template this way is that you can strip out all unwanted apps and get down to the bare essentials. However you can also add in anything else you may want. I have NFS backend NAS repositories so I added nfs-client.

If you look in the repository at …/http/user-data you will see

packages:

  • qemu-guest-agent
  • sudo
  • openssh-server
  • net-tools
  • perl
  • open-iscsi
    snip
  • nfs-client

The grub boot configuration emulates that of a user at the keyboard. These instructions are pretty specific for each distro and possibly release (although ubuntu-24.04 versions 2 and 3 were the same). It seems a bit fiddly to sort this out and if one wanted to create a template for another distro it would be much better to steal somebody else’s configuration.

boot_command = [
“<esc><wait>”,
“e<wait>”,
“<down><down><down><end>”,
" autoinstall quiet ds=nocloud",
“<f10><wait>”,
“<wait1m>”,
“yes<enter>”
]

This does the following.

  1. Press ESC to access the GRUB menu
  2. Press ‘e’ to edit the boot entry
  3. Navigate to the end of the kernel command line
  4. Add “autoinstall quiet ds=nocloud” to enable autoinstall
  5. Press F10 to boot with these options
  6. Wait for the installer to start
  7. Respond “yes” to the partitioning prompt

Sam installs docker from the docker repository in his script. I pulled that out for k3s. It can be added later via ansible.

Security seems pretty tight with the script verifying the sha256sum of the iso and also requiring an api key to get into the build process on the proxmox server.

The process as outlined on the blog post is:

  1. Initialization: Packer checks and installs required plugins
  2. VM Creation: Packer instructs Proxmox to create a new VM with ID 9001
  3. ISO Mounting: Proxmox attaches the Ubuntu 24.04 ISO and the cloud-init ISO
  4. Installation: The Ubuntu installer runs with our autoinstall configuration
  5. Provisioning: After installation, Packer connects via SSH and runs our provisioners
  6. Template Conversion: Finally, the VM is converted to a template

So at the end of it you have a custom template for your kubernetes cluster. It seems like a lot of hard work but once it’s done, it’s done. The real magic happens in part two when terraform instantiates the cluster.

1 Like
2

I never really understood what “infrastructure as code” meant but had heard that terraform was a great tool to do it. Having now played with it I can spin up and modify virtual machines in an instant but the ability to destroy vms even quicker is a little scary. Thankfully in proxmox you can opt to protect VMs from terraform destruction.

Sam’s script uses Pavel Boldyrev’s Terraform Provider for Proxmox VE. It’s not sanctioned by Proxmox and is a personal initiative of Pavel’s. Thankfully there have been many contributors to the project over the last 4 years and it is not likely to go away in a hurry. In any event “other terraform providers are available”.

Once you have your terraform variables set in the files terraform.tfvars and secrets.tfvars it’s just a matter of running the terraform ./build script. For Sam the packer component took six minutes and the deployment four. I spent a few hours earlier in the week cleaning up my NAS configuration so that I could run all these vms on an SSD cluster. My deployment time was not much different to his and even if it were it would not matter. I am happy to linger over my coffee a little longer as the process runs to completion.

So now I’ve got my cluster up ready to go. k8s is up next.

3

So ansible is a great tool for deploying software on vms. A few months ago I created an ansible script to run an update each morning on my machines and vms. It works on debian / red hat machines and also on Windows. (I did not try the Mac.) As you might expect windows was the trickiest to get going but thankfully that vm has subsequently been terminated and I no longer need to wrestle with it.

Vincent Rabah is the creator of k3s-io/k3s-ansible. He has two versions of his install and both worked flawlessly. (Well, flawlessly, after I stuffed around for quite a while.)

ansible-galaxy collection install git+https://github.com/k3s-io/k3s-ansible.git

or

git clone GitHub - k3s-io/k3s-ansible; cd k3s-ansible

After creating an inventory.yml file it was as simple as

ansible-playbook k3s.orchestration.site -i inventory.yml

or

ansible-playbook playbooks/site.yml -i inventory.yml

I didn’t get his upgrade to work with

ansible-playbook playbooks/upgrade.yml -i inventory.yml

but just reapplying the install script brought me up to the latest version of k3s.

I am now managing the cluster from a separate node using kubectl.

curl -LO “https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256”
echo “$(cat kubectl.sha256) kubectl” | sha256sum --check
=> kubectl: OK

sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

and then copying the ./kube/config from the primary master node to my admin machine.

So at the end of all this I had a working k3s cluster. Somehow it seemed too easy so just to make life more difficult I started again but this time created a k8s cluster from “scratch” using ansible scripts. (When I say I created it I really mean that I stole other’s examples from the web and then used AI to get things up and running.)

Perplexity’s AIs were very helpful … until they weren’t. I spent about two days fighting with kube-vip. Perplexity was unaware of the recent requirements for super-admin.conf in setting up the first master node but I eventually stumbled on a thread in reddit that sent me down the right track.

So … AIs are great but the secret with using them is to know when to cut and run. It is also helps if you know were to run.