Declarative Immutable K8s with Talos OS
It seems like in recent years there has been a shift to immutability. I hear it being the new "buzz" word on podcasts and IT conversations. For a while now I have been brushing off NixOS and Silverblue distros as a fad. I didn't have time to learn them or wasn't realizing the gains from making a large infrastructure switch. Recently, I found myself at the tail end of a Kubernetes upgrade that was using Alma servers to host the control plane and worker nodes. This works great with Terraform (openTofu) and Ansible for provisioning. However, I was looking for something even easier and more stable (cattle not pets).
My searches led me to Talos OS, and I'm glad they did! Talos takes a unique, stable, and secure approach to running Kubernetes. The operating system runs containerized as a bootable image. There is no SSH or way to modify the filesystem and configuration in the conventional means. Instead, everything is done with our beloved YAML manifests. You tell it what Kubernetes version you want, which drive to install, and importantly which image of Talos to install. Talos uses something called image factory to create images that can include extensions on top of the Talos OS. These extensions can be used to install extra packages, like iscsi for Longhorn or open-nvidia for GPU workloads.
The Setup
For stability, I set up 3 master nodes using HAProxy to balance the API calls. This lets us take a master down and still have a quorum and proper operation. From here, I was able to add as may nodes as needed for workloads. Adding a new node is easy. I just need to start the VM and use an online command pointing to my manifest file. Talos reads it and quickly installs the OS reboots, joins the cluster, and starts taking on work. I use the word easy now but I will admit the process had a bit of a learning curve coming from the old imparitve OS mindset.
Most of the best information for getting started can be found on their website, so it's not worth copying the instructions. I do, however, have a knowledgebase that has some of the bits that I used to get this started.