Proxmox with OpenTofu Kubespray and Kubernetes

Overview

I need a Kubernetes cluster, again

I am using a lot of my spare time playing around with my lab exploring different topics. Very much of that time again is spent with Kubernetes. It has been so many times have I deployed a new Kubernetes cluster, then after some time decommissioned it again. They way I have typically done it is using a Ubuntu template I have created, cloned it, manually adjusted all clones as needed, then manually installed Kubernetes. This takes a lot of time overall and it also stops me doing certain tasks as sometimes I just think nah.. not again. Maybe another day and I do something else instead. And, when a human being is set to do a repetive task it is destined to fail at some stage, forgot a setting, why is this one vm not working as the other vms and so on. Now the time has come to automate these tasks with the following goal in mind:

  • Reduce deployment time to a minum
  • Eliminate human errors
  • Consistent outcome every time
  • Make it even more fun to deploy a Kubernetes cluster

I have been running Home Assistant for many years, there I have a bunch of automations automating all kinds of things in my home which just makes my everyday life a bit happier. Most of these automations just work in the background doing their stuff as a good automation should. Automating my lab deployments is something I have been thinking of getting into several times, and now I decided to get this show started. So, as usual, a lot of tinkering, trial and error until I managed to get something working the way I wanted. Probably room for improvement in several areas, that is something I will most likely use my time on "post" this blog post. Speaking of blog post, after using some time on this I had to create a blog post on it. My goal later on is a fully automated lab from VM creation, Kubernetes runtime, and applications. And when they are decommisioned I can easily spin up everything again with persistent data etc. Lets see.

For now, this post will cover what I have done and configured so far to be able to automatically deploy the VMs for my Kubernetes clusters then the provisioning of Kubernetes itself.

My lab

My lab consists of two fairly spec'ed servers with a bunch of CPU cores, a lot of RAM, and a decent amount of SSDs. In regards to network they are using 10GB ethernet. In terms of power usage they are kind of friendly to my electricity bill with my "required" vms running on them, but can potentially ruin that if I throw a lot of stuff at them to chew on.

The total power usage above includes my switch, UPS and some other small devices. So its not that bad considering how much I can get out of it.

But with automation I can easily spin up some resources to be consumed for a certain period and delete again if not needed. My required VMs, that are always on, are things like Bind dns servers, PfSense, Truenas, Frigate, DCS server, a couple of linux "mgmt" vms, Home Assistant, a couple of Kubernetes clusters hosting my Unifi controller, Traefik Proxy, Grafana etc.

For the virtualization layer on my two servers I am using Proxmox, one of the reason is that it supports PCI passthrough of my Coral TPU. I have been running Proxmox for many years and I find it to be a very decent alternative. It does what I want it to do and Proxmox has a great community!

Proxmox has been configured with these two servers as a cluster. I have not configured any VMs in HA, but with a Proxmox cluster I can easily migrate VMs between the hosts, even without shared storage, and single web-ui to manage both servers. To be "quorate" I have a cute little RPi3 with its beefy 32 GB SD card, 2GB RAM and 1GB ethernet as a Qdevice

Info

On that note, one does not necessarily need have them in a cluster to do vm migration. I recently moved a bunch of VMs over two these two new servers and it was easy peasy using this command:

1# executed on the source Proxmox node
2qm remote-migrate <src vm_id> <dst vm_id> 'apitoken=PVEAPIToken=root@pam!migrate=<token>,host=<dst-host-ip>,fingerprint=<thumprint>' --target-bridge vmbr0 --target-storage 'raid10-node01' --online

I found this with the great help of the Proxmox community here

Both Proxmox servers have their own local ZFS storage. In both of them I have created a dedicated zfs pool with identical name which I use for zfs replication for some of the more "critical" VMs, and as a bonus it also reduces the migration time drastically for these VMs when I move them between my servers. The "cluster" network (vmbr1) is directly connected 2x 10GB ethernet (not through my physical switch). The other 2x 10GB interfaces are connected to my switch for all my other network needs like VM network (vmbr0) and Proxmox management.

Throughout this post I will be using a dedicated linux vm for all my commands, interactions. So every time I install something, it is on this linux vm.

But I am digressing, enough of the intro, get on with the automation part you were supposed to write about. Got it.

In this post I will use the following products:

Provision VMs in Proxmox using OpenTofu

Terraform is something I have been involved in many times in my professional work, but never had the chance to actually use it myself other than know about it, and how the solutions I work with can work together with Terraform. I did notice even the Proxmox community had several post around using Terraform, so I decided to just go with Terraform. I already knew that HashiCorp had announced their change of Terraform license from MPL to BSL and that OpenTofu is an OpenSource fork of Terraform. So instead of basing my automations using Terraform, I will be using OpenTofu. Read more about OpenTofu here. For now OpenTofu is Terraform "compatible" and uses Terraform registries, so providers etc for Terraform works with OpenTofu. You will notice that further down, all my configs will be using terraform constructs.

In OpenTofu/Terraform there are several concepts that one needs to know about, I will use some of them in this post like providers, resources, provisioners and variables. It is smart to read about them a bit more here how and why to use them.

To get started with OpenTofu there are some preparations do be done. Lets start with these

Install OpenTofu

To get started with OpenTofu I deployed it on my linux machine using Snap, but several other alternatives is available. See more on the official OpenTofu docs page.

1sudo snap install --classic opentofu

Now OpenTofu is installed and I can start using it. I also installed the bash autocompletion like this:

1tofu -install-autocomplete # restart shell session...

I decided to create a dedicated folder for my "projects" to live in so I have created a folder in my home folder called *proxmox" where I have different subfolders depending on certain tasks or resources I will use OpenTofu for.

1proxmox/
2├── k8s-cluster-02
3├── proxmox-images

OpenTofu Proxmox provider

To be able to use OpenTofu with Proxmox I need a provider that can use the Proxmox API. I did some quick research on the different options out there and landed on this provider: bpg/proxmox. Seems very active and are recently updated (according to the git repo here)

An OpenTofu/Terraform provider is being defined like this, and the example below is configured to install the bpg/proxmox provider I need to interact with Proxmox.

 1terraform {
 2  required_providers {
 3    proxmox = {
 4      source = "bpg/proxmox"
 5      version = "0.43.2"
 6    }
 7  }
 8}
 9
10provider "proxmox" {
11  endpoint  = var.proxmox_api_endpoint
12  api_token = var.proxmox_api_token
13  insecure  = true
14  ssh {
15    agent    = true
16    username = "root"
17  }
18}

I will save this content in a filed called providers.tf

First a short explanation of the two sections above. The terraform sections instructs OpenTofu/Terraform which provider to be downloaded and enabled. The version field defines a specific version to use, not the latest but this version. Using this field will make sure that your automation will not break if there is an update in the provider version that introduces some API changes.

The provider section configures the proxmox provider how it should interact with Proxmox. Instead of using a regular username and password I have opted for using API token. Here I in the endpoint and api-token keys am using a variable value that are defined in another file called variables.tf and a credentials.auto.tfvars. There are also certain tasks in my automation that requires SSH interaction with Proxmox so I have also enabled that by configuring the ssh field.

For more information on the bpg/proxmox provider, head over here

Prepare Proxmox with an API token for the OpenTofu bpg/proxmox provider

To be able to use the above configured provider with Proxmox I need to prepare Proxmox to use API token. I followed the bpg/proxmox provider documentation here

On my "leader" Proxmox node:

 1# Create the user
 2sudo pveum user add terraform@pve
 3# Create a role for the user above
 4sudo pveum role add Terraform -privs "Datastore.Allocate Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit Pool.Allocate Sys.Audit Sys.Console Sys.Modify SDN.Use VM.Allocate VM.Audit VM.Clone VM.Config.CDROM VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Migrate VM.Monitor VM.PowerMgmt User.Modify"
 5# Assign the terraform user to the above role
 6sudo pveum aclmod / -user terraform@pve -role Terraform
 7# Create the token
 8sudo pveum user token add terraform@pve provider --privsep=0
 9
10┌──────────────┬──────────────────────────────────────┐
11│ key          │ value                                │
12╞══════════════╪══════════════════════════════════════╡
13│ full-tokenid │ terraform@pve!provider               │
14├──────────────┼──────────────────────────────────────┤
15│ info         │ {"privsep":"0"}16├──────────────┼──────────────────────────────────────┤
17│ value        │ <token>                               │
18└──────────────┴──────────────────────────────────────┘
19# make a backup of the token
20    

Now I have create the API user to be used with my bpg/proxmox provider. Though it is not sufficient as I also need to define a ssh keypair on my Linux jumphost for passwordless SSH authentication which I will need to copy to my Proxmox nodes. This is done like this:

1andreasm@linuxmgmt01:~$ ssh-copy-id -i id_rsa.pub root@172.18.5.102 # -i pointing to the pub key I want to use and specify the root user

Now, I can log into my Proxmox node using SSH without password from my Linux jumphost. But, when used in combination with opentofu its not sufficient. I need to load the key into the keystore. If I dont do that the automations that require SSH access will fail with this error message:

1Error: failed to open SSH client: unable to authenticate user "root" over SSH to "172.18.5.102:22". Please verify that ssh-agent is correctly loaded with an authorized key via 'ssh-add -L' (NOTE: configurations in ~/.ssh/config are not considered by golang's ssh implementation). The exact error from ssh.Dial: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

Ah, I can just configure the .ssh/config with this:

1Host 172.18.5.102
2    AddKeysToAgent yes
3    IdentityFile ~/.ssh/id_rsa

Nope, cant do.. Look at the error message again:

1(NOTE: configurations in ~/.ssh/config are not considered by golang's ssh implementation)

Ah, I can just do this then:

1andreasm@linuxmgmt01:~$ eval `ssh-agent -s`
2andreasm@linuxmgmt01:~$ ssh-add id_rsa

Yes, but that is not persistent between sessions, so you need to do this every time you log on to your Linux jumphost again. I couldnt figure out how to do this the "correct" way by following the expected Golang approach so instead I went with this approach.

1# I added this in my .bashrc file
2if [ -z "$SSH_AUTH_SOCK" ] ; then
3 eval `ssh-agent -s`
4 ssh-add ~/.ssh/id_rsa
5fi

OpenTofu variables and credentials

Instead of exposing username/tokens and other information directly in my provider/resource *.tf's I can refer to them by declaring them in a variables.tf and credentials in a credentials.auto.tfvars. The variables.tf defines the variables to use, and the credentials.auto.tfvars maps the value to a variable. I am using this content in my variable.tf

1variable "proxmox_api_endpoint" {
2  type = string
3  description = "Proxmox cluster API endpoint https://proxmox-01.my-domain.net:8006"
4}
5
6variable "proxmox_api_token" {
7  type = string
8  description = "Proxmox API token bpg proxmox provider with ID and token"
9}

Then in my credentials.auto.tfvars file I have this content:

1proxmox_api_endpoint = "https://proxmox-01.mu-domain.net:8006"
2proxmox_api_token = "terraform@pve!provider=<token-from-earlier>"

To read more about these files head over here. One can also use the varible.tf to generate prompts if I have created variables being referred to in any of my resources but do not map to a value (credentials.tfvars etc..). Then it will ask you to enter a value when doing a plan or apply.

Prepare a Ubuntu cloud image - opentofu resource

In my automations with OpenTofu I will use a cloud image instead of a VM template in Proxmox.

I have been using Ubuntu as my preferred Linux distribution for many years and dont see any reason to change that now. Ubuntu/Canonical do provide cloud images and they can be found here

But I will not download them using my browser, instead I will be using OpenTofu with the bpg/proxmox provider to download and upload them to my Proxmox nodes.

In my proxmox folder I have a dedicated folder for this project called proxmox-images

1proxmox/
2├── proxmox-images

Inside that folder I will start by creating the following files: provider.tf file, variable.tf and credentials.auto.tfvars with the content described above.

1proxmox-images/
2├── credentials.auto.tfvars
3├── provider.tf
4└── variables.tf
5
60 directories, 3 files

But these files dont to anything other than provides the relevant information to connect and interact with Proxmox. What I need is a OpenTofu resource with a task that needs to be done. So I will prepare a file called ubuntu_cloud_image.tf with the following content:

 1resource "proxmox_virtual_environment_file" "ubuntu_cloud_image" {
 2  content_type = "iso"
 3  datastore_id = "local"
 4  node_name    = "proxmox-02"
 5
 6  source_file {
 7    # you may download this image locally on your workstation and then use the local path instead of the remote URL
 8    path      = "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img"
 9
10    # you may also use the SHA256 checksum of the image to verify its integrity
11    checksum = "1d82c1db56e7e55e75344f1ff875b51706efe171ff55299542c29abba3a20823"
12  }
13}

What this file does is instructing OpenTofu where to grab my Ubuntu cloud image from, then upload it to my Proxmox node 2 and a specific datastore on that node. So I have defined a resource to define this task. More info on this here.

Within this same .tf file I can define several resources to download several cloud images. I could have called the file cloud-images.tf instead and just defined all the images I wanted to download/upload to Proxmox. I have decided to keep this task as a separate project/folder.

When I save this file I will now have 4 files in my proxmox-image folder:

1proxmox-images/
2├── credentials.auto.tfvars
3├── provider.tf
4├── ubuntu_cloud_image.tf
5└── variables.tf
6
70 directories, 4 files

Thats all the files I need to start my first OpenTofu task/automation.

Lets do a quick recap. The provider.tf instructs OpenTofu to download and enable the bpg/proxmox provider in this folder, and configure the provider to interact with my Proxmox nodes.

Then it is the variables.tf and the credentials.auto.tfvars that provides keys and values to be used in the provider.tf. The ubuntu_cloud_image.tf contains the task I want OpenTofu to perform.

So now I am ready to execute the first OpenTofu command tofu init. This command initiates OpenTofu in the respective folder proxmox-images, downloads and enables the provider I have configured. More info here.

So lets try it:

 1andreasm@linuxmgmt01:~/terraform/proxmox/proxmox-images$ tofu init
 2
 3Initializing the backend...
 4
 5Initializing provider plugins...
 6- Finding bpg/proxmox versions matching "0.43.2"...
 7- Installing bpg/proxmox v0.43.2...
 8- Installed bpg/proxmox v0.43.2 (signed, key ID DAA1958557A27403)
 9
10Providers are signed by their developers.
11If you'd like to know more about provider signing, you can read about it here:
12https://opentofu.org/docs/cli/plugins/signing/
13
14OpenTofu has created a lock file .terraform.lock.hcl to record the provider
15selections it made above. Include this file in your version control repository
16so that OpenTofu can guarantee to make the same selections by default when
17you run "tofu init" in the future.
18
19OpenTofu has been successfully initialized!
20
21You may now begin working with OpenTofu. Try running "tofu plan" to see
22any changes that are required for your infrastructure. All OpenTofu commands
23should now work.
24
25If you ever set or change modules or backend configuration for OpenTofu,
26rerun this command to reinitialize your working directory. If you forget, other
27commands will detect it and remind you to do so if necessary.

Cool, now I have initialized my proxmox-image folder. Lets continue with creating a plan.

 1andreasm@linuxmgmt01:~/terraform/proxmox/proxmox-images$ tofu plan -out plan
 2
 3OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
 4  + create
 5
 6OpenTofu will perform the following actions:
 7
 8  # proxmox_virtual_environment_file.ubuntu_cloud_image will be created
 9  + resource "proxmox_virtual_environment_file" "ubuntu_cloud_image" {
10      + content_type           = "iso"
11      + datastore_id           = "local"
12      + file_modification_date = (known after apply)
13      + file_name              = (known after apply)
14      + file_size              = (known after apply)
15      + file_tag               = (known after apply)
16      + id                     = (known after apply)
17      + node_name              = "proxmox-02"
18      + overwrite              = true
19      + timeout_upload         = 1800
20
21      + source_file {
22          + changed  = false
23          + checksum = "1d82c1db56e7e55e75344f1ff875b51706efe171ff55299542c29abba3a20823"
24          + insecure = false
25          + path     = "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img"
26        }
27    }
28
29Plan: 1 to add, 0 to change, 0 to destroy.
30
31──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
32
33Saved the plan to: plan
34
35To perform exactly these actions, run the following command to apply:
36    tofu apply "plan"

I am using the -out plan to just keep a record of the plan. It could be called whatever I wanted. Now OpenTofu tells me what its intention are, and if I am happy with the plan, I just need to apply it.

So lets apply it:

1andreasm@linuxmgmt01:~/terraform/proxmox/proxmox-images$ tofu apply plan
2proxmox_virtual_environment_file.ubuntu_cloud_image: Creating...
3proxmox_virtual_environment_file.ubuntu_cloud_image: Still creating... [10s elapsed]
4proxmox_virtual_environment_file.ubuntu_cloud_image: Creation complete after 20s [id=local:iso/jammy-server-cloudimg-amd64.img]
5
6Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

And after 20 seconds I have my Ubuntu cloud image uploaded to my Proxmox node (jammy-server-cloudimg-amd64.img):

jammy-server-cloud-image

Thats great. Now I have my iso image to use for my VM deployments.

Next up is to deploy a bunch of VMs to be used for a Kubernetes cluster

Deploy VMs using OpenTofu and install Kubernetes using Ansible and Kubespray

In this section I will automate everything from deploying VMs to install Kubernetes on these VMs. This task will involve OpenTofu, Ansible and Kubespray. It will deploy 6 virtual machines, with two different resource configurations. It will deploy 3 vms intended to be used as Kubernetes controlplane nodes, and 3 vms intended to be Kubernetes worker nodes. Thats the task for OpenTofu, as soon as OpenTofu has done its part it will trigger an Ansible playbook to initiate Kubespray to install Kubernetes on all my nodes. This should then end up in a fully automated task from me just executing tofu apply plan to a ready Kubernetes cluster.

I will start by creating another subfolder in my proxmox folder called k8s-cluster-02. In this folder I will reuse the following files:

1k8s-cluster-02/
2├── credentials.auto.tfvars
3├── provider.tf
4└── variables.tf
5
60 directories, 3 files

These files I will just copy from my proxmox-images folder. I will also need define and add a couple of other files. When I am ready with this task I will end up with these files:

1k8s-cluster-02/
2├── ansible.tf
3├── credentials.auto.tfvars
4├── k8s-cluster-02.tf
5├── provider.tf
6├── ubuntu_cloud_config.tf
7└── variables.tf
8
90 directories, 7 files

I will go through all the files one by one. But there is still some preparations to be done to be able to complete this whole task.

Prepare OpenTofu to deploy my VMs

In addition to the three common files I have copied from the previous project, I will need to create two proxmox_virtual_environment_vm resource definitions (one for each VM type). The content of this file will be saved as k8s-cluster-02.tf and looks like this:

  1resource "proxmox_virtual_environment_vm" "k8s-cp-vms-cl02" {
  2  count       = 3
  3  name        = "k8s-cp-vm-${count.index + 1}-cl-02"
  4  description = "Managed by Terraform"
  5  tags        = ["terraform", "ubuntu", "k8s-cp"]
  6
  7  node_name = "proxmox-02"
  8  vm_id     = "100${count.index + 1}"
  9
 10
 11  cpu {
 12    cores = 2
 13    type = "host"
 14  }
 15
 16  memory {
 17    dedicated = 2048
 18  }
 19
 20
 21  agent {
 22    # read 'Qemu guest agent' section, change to true only when ready
 23    enabled = true
 24  }
 25
 26  startup {
 27    order      = "3"
 28    up_delay   = "60"
 29    down_delay = "60"
 30  }
 31
 32  disk {
 33    datastore_id = "raid-10-node02"
 34    file_id      = "local:iso/jammy-server-cloudimg-amd64.img"
 35    interface    = "virtio0"
 36    iothread     = true
 37    discard      = "on"
 38    size         = 40
 39    file_format  = "raw"
 40  }
 41
 42
 43  initialization {
 44    dns {
 45      servers = ["10.100.1.7", "10.100.1.6"]
 46      domain = "my-domain.net"
 47    }
 48    ip_config {
 49      ipv4 {
 50        address = "10.160.1.2${count.index + 1}/24"
 51        gateway = "10.160.1.1"
 52      }
 53    }
 54    datastore_id = "raid-10-node02"
 55
 56    user_data_file_id = proxmox_virtual_environment_file.ubuntu_cloud_init.id
 57  }
 58
 59  network_device {
 60    bridge = "vmbr0"
 61    vlan_id = "216"
 62  }
 63
 64  operating_system {
 65    type = "l26"
 66  }
 67
 68  keyboard_layout = "no"
 69
 70  lifecycle {
 71    ignore_changes = [
 72      network_device,
 73    ]
 74  }
 75
 76
 77}
 78
 79resource "proxmox_virtual_environment_vm" "k8s-worker-vms-cl02" {
 80  count       = 3
 81  name        = "k8s-node-vm-${count.index + 1}-cl-02"
 82  description = "Managed by Terraform"
 83  tags        = ["terraform", "ubuntu", "k8s-node"]
 84
 85  node_name = "proxmox-02"
 86  vm_id     = "100${count.index + 5}"
 87
 88
 89  cpu {
 90    cores = 4
 91    type = "host"
 92  }
 93
 94  memory {
 95    dedicated = 4096
 96  }
 97
 98
 99  agent {
100    # read 'Qemu guest agent' section, change to true only when ready
101    enabled = true
102  }
103
104  startup {
105    order      = "3"
106    up_delay   = "60"
107    down_delay = "60"
108  }
109
110  disk {
111    datastore_id = "raid-10-node02"
112    file_id      = "local:iso/jammy-server-cloudimg-amd64.img"
113    interface    = "virtio0"
114    iothread     = true
115    discard      = "on"
116    size         = 60
117    file_format  = "raw"
118  }
119
120
121  initialization {
122    dns {
123      servers = ["10.100.1.7", "10.100.1.6"]
124      domain = "my-domain.net"
125    }
126    ip_config {
127      ipv4 {
128        address = "10.160.1.2${count.index + 5}/24"
129        gateway = "10.160.1.1"
130      }
131    }
132    datastore_id = "raid-10-node02"
133
134    user_data_file_id = proxmox_virtual_environment_file.ubuntu_cloud_init.id
135  }
136
137  network_device {
138    bridge = "vmbr0"
139    vlan_id = "216"
140  }
141
142  operating_system {
143    type = "l26"
144  }
145
146  keyboard_layout = "no"
147
148  lifecycle {
149    ignore_changes = [
150      network_device,
151    ]
152  }
153
154
155}

In this file I have define two proxmox_virtual_environment_vm resources called k8s-cp-vms-cl02 and k8s-worker-vms-cl02 respectively. I am using count.index in some of the fields where I need to automatically generate an increasing number. Like IP address, VM_ID, Name. It is also referring to my Ubuntu cloud image as installation source, then a user_data_file_id (will be shown next) to do a simple cloud-init on the VMs.

Then I need to configure a proxmox_virtual_environment_file resource called ubuntu_cloud_init to trigger a cloud-init task to configure some basic initial config on my VMs.

This is the content of this file:

 1resource "proxmox_virtual_environment_file" "ubuntu_cloud_init" {
 2  content_type = "snippets"
 3  datastore_id = "local"
 4  node_name    = "proxmox-02"
 5
 6  source_raw {
 7    data = <<EOF
 8#cloud-config
 9chpasswd:
10  list: |
11    ubuntu:ubuntu    
12  expire: false
13packages:
14  - qemu-guest-agent
15timezone: Europe/Oslo
16
17users:
18  - default
19  - name: ubuntu
20    groups: sudo
21    shell: /bin/bash
22    ssh-authorized-keys:
23      - ${trimspace("ssh-rsa <sha356> user@mail.com")}
24    sudo: ALL=(ALL) NOPASSWD:ALL
25
26power_state:
27    delay: now
28    mode: reboot
29    message: Rebooting after cloud-init completion
30    condition: true
31
32EOF
33
34    file_name = "ubuntu.cloud-config.yaml"
35  }
36}

This will be used by all my OpenTofu deployed VMs in this task. It will install the qemu-guest-agent, configure a timezone, copy my Linux jumphost public key so I can log in to them using SSH without password. Then a quick reboot after all task completed to get the qemu-agent to report correctly to Proxmox. This will be uploaded to my Proxmox server as a Snippet. For Snippets to work. One need to enable this here under Datacenter -> Storage

snippets

Now I can go ahead and perform tofu init in this folder, but I am still not ready. I Kubernetes to be deployed also remember. For that I will use Kubespray.

Install and configure Kubespray

If you have not heard about Kubespray before, head over here for more info. I have been following the guides from Kubespray to get it working, and its very well documented.

Kubespray is a really powerful way to deploy Kubernetes with a lot of options and customizations. I just want to highlight Kubespray in this section as it does a really great job in automating Kubernetes deployment.

A quote from the Kubespray pages:

Comparison

Kubespray vs Kops

Kubespray runs on bare metal and most clouds, using Ansible as its substrate for provisioning and orchestration. Kops performs the provisioning and orchestration itself, and as such is less flexible in deployment platforms. For people with familiarity with Ansible, existing Ansible deployments or the desire to run a Kubernetes cluster across multiple platforms, Kubespray is a good choice. Kops, however, is more tightly integrated with the unique features of the clouds it supports so it could be a better choice if you know that you will only be using one platform for the foreseeable future.

Kubespray vs Kubeadm

Kubeadm provides domain Knowledge of Kubernetes clusters' life cycle management, including self-hosted layouts, dynamic discovery services and so on. Had it belonged to the new operators world, it may have been named a "Kubernetes cluster operator". Kubespray however, does generic configuration management tasks from the "OS operators" ansible world, plus some initial K8s clustering (with networking plugins included) and control plane bootstrapping.

Kubespray has started using kubeadm internally for cluster creation since v2.3 in order to consume life cycle management domain knowledge from it and offload generic OS configuration things from it, which hopefully benefits both sides.

On my Linux jumphost I have done the following to prepare for Kubespray

 1# clone the Kubespray repo
 2andreasm@linuxmgmt01:~/terraform/proxmox$ git clone https://github.com/kubernetes-sigs/kubespray.git
 3# kubespray folder created under my proxmox folder
 4andreasm@linuxmgmt01:~/terraform/proxmox$ ls
 5k8s-cluster-02  kubespray  proxmox-images
 6# Python dependencies -  these I am not sure are needed but I installed them anyways.
 7sudo apt install software-properties-common
 8sudo add-apt-repository ppa:deadsnakes/ppa # used this repo to get a newer Python 3 than default repo
 9sudo apt update
10sudo apt install python3.12 python3-pip python3-virtualenv

Don't try to install Ansible in the systemwide, it will not work. Follow the Kubespray documentation to install ansible in a Python Virtual Environment.

1andreasm@linuxmgmt01:~/terraform/proxmox$ VENVDIR=kubespray-venv
2andreasm@linuxmgmt01:~/terraform/proxmox$ KUBESPRAYDIR=kubespray
3andreasm@linuxmgmt01:~/terraform/proxmox$ python3 -m venv $VENVDIR
4andreasm@linuxmgmt01:~/terraform/proxmox$ source $VENVDIR/bin/activate
5(kubespray-venv) andreasm@linuxmgmt01:~/terraform/proxmox$ cd $KUBESPRAYDIR
6(kubespray-venv) andreasm@linuxmgmt01:~/terraform/proxmox/kubespray$ pip install -U -r requirements.txt

To exit out of the python environment type deactivate.

Now I have these subfolders under my proxmox folder:

1
2andreasm@linuxmgmt01:~/terraform/proxmox$ tree -L 1
3.
4├── k8s-cluster-02
5├── kubespray
6├── kubespray-venv
7├── proxmox-images
8
94 directories, 0 files

Next thing I need to do is to copy a sample folder inside the kubespray folder to a folder that reflects the kubernetes cluster name I am planning to deploy.

1andreasm@linuxmgmt01:~/terraform/proxmox/kubespray$ cp -rfp inventory/sample inventory/k8s-cluster-02

This sample folder contains a couple of files and directories. The file I am interested in right now is the inventory.ini file. I need to populate this file with the nodes, including the control plane nodes, that will form my Kubernetes cluster. This is how it looks like now, default:

 1# ## Configure 'ip' variable to bind kubernetes services on a
 2# ## different ip than the default iface
 3# ## We should set etcd_member_name for etcd cluster. The node that is not a etcd member do not need to set the value, or can set the empty string value.
 4[all]
 5# node1 ansible_host=95.54.0.12  # ip=10.3.0.1 etcd_member_name=etcd1
 6# node2 ansible_host=95.54.0.13  # ip=10.3.0.2 etcd_member_name=etcd2
 7# node3 ansible_host=95.54.0.14  # ip=10.3.0.3 etcd_member_name=etcd3
 8# node4 ansible_host=95.54.0.15  # ip=10.3.0.4 etcd_member_name=etcd4
 9# node5 ansible_host=95.54.0.16  # ip=10.3.0.5 etcd_member_name=etcd5
10# node6 ansible_host=95.54.0.17  # ip=10.3.0.6 etcd_member_name=etcd6
11
12# ## configure a bastion host if your nodes are not directly reachable
13# [bastion]
14# bastion ansible_host=x.x.x.x ansible_user=some_user
15
16[kube_control_plane]
17# node1
18# node2
19# node3
20
21[etcd]
22# node1
23# node2
24# node3
25
26[kube_node]
27# node2
28# node3
29# node4
30# node5
31# node6
32
33[calico_rr]
34
35[k8s_cluster:children]
36kube_control_plane
37kube_node
38calico_rr

This is just a sample, but nice to know how it should be defined. I will create a OpenTofu task to create this inventory file for me with the corresponding VMs I am deploying in the same task/project. This will autopopulate this inventory.ini file with all the necessary information. So I can just go ahead and delete the inventory.ini file that has been copied from the sample folder to my new k8s-cluster-02 folder. The other folders and files contains several variables/settings I can adjust to my liking. Like different CNIs, Kubernetes version etc. But I will not cover these here, head over to the official Kubespray docs for more info on that. These are the files/folder:

1andreasm@linuxmgmt01:~/terraform/proxmox/kubespray/inventory/k8s-cluster-02/group_vars$ tree -L 1
2.
3├── all
4├── etcd.yml
5└── k8s_cluster
6
72 directories, 1 file

I will deploy my Kubernetes cluster "stock" so these files are left untouched for now, except the inventory.ini file.

Now Kubespray is ready to execute Ansible to configure my Kubernetes cluster as soon as my OpenTofu provisioned VMs has been deployed. The last step I need to do is to confgiure a new .tf file to create this inventory.ini file and kick of the Ansible command to fire up Kubespray.

Configure OpenTofu to kick off Kubespray

The last .tf file in my fully automated Kubernetes installation is the ansible.tf file that contains this information:

 1# Generate inventory file
 2resource "local_file" "ansible_inventory" {
 3  filename = "/home/andreasm/terraform/proxmox/kubespray/inventory/k8s-cluster-02/inventory.ini"
 4  content = <<-EOF
 5  [all]
 6  ${proxmox_virtual_environment_vm.k8s-cp-vms-cl02[0].name} ansible_host=${proxmox_virtual_environment_vm.k8s-cp-vms-cl02[0].ipv4_addresses[1][0]}
 7  ${proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1].name} ansible_host=${proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1].ipv4_addresses[1][0]}
 8  ${proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2].name} ansible_host=${proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2].ipv4_addresses[1][0]}
 9  ${proxmox_virtual_environment_vm.k8s-worker-vms-cl02[0].name} ansible_host=${proxmox_virtual_environment_vm.k8s-worker-vms-cl02[0].ipv4_addresses[1][0]}
10  ${proxmox_virtual_environment_vm.k8s-worker-vms-cl02[1].name} ansible_host=${proxmox_virtual_environment_vm.k8s-worker-vms-cl02[1].ipv4_addresses[1][0]}
11  ${proxmox_virtual_environment_vm.k8s-worker-vms-cl02[2].name} ansible_host=${proxmox_virtual_environment_vm.k8s-worker-vms-cl02[2].ipv4_addresses[1][0]}
12
13  [kube_control_plane]
14  ${proxmox_virtual_environment_vm.k8s-cp-vms-cl02[0].name}
15  ${proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1].name}
16  ${proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2].name}
17
18  [etcd]
19  ${proxmox_virtual_environment_vm.k8s-cp-vms-cl02[0].name}
20  ${proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1].name}
21  ${proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2].name}
22
23  [kube_node]
24  ${proxmox_virtual_environment_vm.k8s-worker-vms-cl02[0].name}
25  ${proxmox_virtual_environment_vm.k8s-worker-vms-cl02[1].name}
26  ${proxmox_virtual_environment_vm.k8s-worker-vms-cl02[2].name}
27
28  [k8s_cluster:children]
29  kube_node
30  kube_control_plane
31
32  EOF
33}
34
35resource "null_resource" "ansible_command" {
36  provisioner "local-exec" {
37    command = "./kubespray.k8s-cluster-02.sh > k8s-cluster-02/ansible_output.log 2>&1"
38    interpreter = ["/bin/bash", "-c"]
39    working_dir = "/home/andreasm/terraform/proxmox"
40    }
41  depends_on = [proxmox_virtual_environment_vm.k8s-cp-vms-cl02, proxmox_virtual_environment_vm.k8s-worker-vms-cl02, local_file.ansible_inventory]
42  }

This will create the inventory.ini in the /proxmox/kubespray/inventory/k8s-cluster-02/ for Kubespray to use. Then it will fire a command refering to a bash script (more on that further down) to trigger the Ansible command:

1ansible-playbook -i inventory/k8s-cluster-02/inventory.ini --become --become-user=root cluster.yml -u ubuntu

For this to work I had to create a bash script that activated the Kubespray virtual environment:

 1#!/bin/bash
 2
 3# Set working directory
 4WORKING_DIR="/home/andreasm/terraform/proxmox"
 5cd "$WORKING_DIR" || exit
 6
 7# Set virtual environment variables
 8VENVDIR="kubespray-venv"
 9KUBESPRAYDIR="kubespray"
10
11# Activate virtual environment
12source "$VENVDIR/bin/activate" || exit
13
14# Change to Kubespray directory
15cd "$KUBESPRAYDIR" || exit
16
17# Run Ansible playbook
18ansible-playbook -i inventory/k8s-cluster-02/inventory.ini --become --become-user=root cluster.yml -u ubuntu

I will create and save this file in my proxmox folder.

This is now the content of my proxmox folder:

1andreasm@linuxmgmt01:~/terraform/proxmox$ tree -L 1
2.
3├── k8s-cluster-02
4├── kubespray
5├── kubespray-venv
6├── kubespray.k8s-cluster-02.sh
7└── proxmox-images
8
94 directories, 1 file

And this is the content in the k8s-cluster-02 folder where I have my OpenTofu tasks defined:

 1andreasm@linuxmgmt01:~/terraform/proxmox/k8s-cluster-02$ tree -L 1
 2.
 3├── ansible.tf
 4├── credentials.auto.tfvars
 5├── k8s-cluster-02.tf
 6├── provider.tf
 7├── ubuntu_cloud_config.tf
 8└── variables.tf
 9
100 directories, 6 files

Its time to put it all to a test

A fully automated provisioning of Kubernetes on Proxmox using OpenTofu, Ansible and Kubespray

To get this show started, it is the same procedure as it was with my cloud image task. Need to run tofu init, then create a plan, check the output and finally approve it. So lets see how this goes.

From within my ./proxmox/k8s-cluster-02 folder:

 1andreasm@linuxmgmt01:~/terraform/proxmox/k8s-cluster-02$ tofu init
 2
 3Initializing the backend...
 4
 5Initializing provider plugins...
 6- Finding latest version of hashicorp/null...
 7- Finding bpg/proxmox versions matching "0.43.2"...
 8- Finding latest version of hashicorp/local...
 9- Installing hashicorp/null v3.2.2...
10- Installed hashicorp/null v3.2.2 (signed, key ID 0C0AF313E5FD9F80)
11- Installing bpg/proxmox v0.43.2...
12- Installed bpg/proxmox v0.43.2 (signed, key ID DAA1958557A27403)
13- Installing hashicorp/local v2.4.1...
14- Installed hashicorp/local v2.4.1 (signed, key ID 0C0AF313E5FD9F80)
15
16Providers are signed by their developers.
17If you'd like to know more about provider signing, you can read about it here:
18https://opentofu.org/docs/cli/plugins/signing/
19
20OpenTofu has created a lock file .terraform.lock.hcl to record the provider
21selections it made above. Include this file in your version control repository
22so that OpenTofu can guarantee to make the same selections by default when
23you run "tofu init" in the future.
24
25OpenTofu has been successfully initialized!
26
27You may now begin working with OpenTofu. Try running "tofu plan" to see
28any changes that are required for your infrastructure. All OpenTofu commands
29should now work.
30
31If you ever set or change modules or backend configuration for OpenTofu,
32rerun this command to reinitialize your working directory. If you forget, other
33commands will detect it and remind you to do so if necessary.

Then create the plan:

 1OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
 2  + create
 3
 4OpenTofu will perform the following actions:
 5
 6  # local_file.ansible_inventory will be created
 7  + resource "local_file" "ansible_inventory" {
 8      + content              = (known after apply)
 9      + content_base64sha256 = (known after apply)
10      + content_base64sha512 = (known after apply)
11      + content_md5          = (known after apply)
12      + content_sha1         = (known after apply)
13      + content_sha256       = (known after apply)
14      + content_sha512       = (known after apply)
15      + directory_permission = "0777"
16      + file_permission      = "0777"
17      + filename             = "/home/andreasm/terraform/proxmox/kubespray/inventory/k8s-cluster-02/inventory.ini"
18      + id                   = (known after apply)
19    }
20
21  # null_resource.ansible_command will be created
22  + resource "null_resource" "ansible_command" {
23      + id = (known after apply)
24    }
25
26  # proxmox_virtual_environment_file.ubuntu_cloud_init will be created
27  + resource "proxmox_virtual_environment_file" "ubuntu_cloud_init" {
28      + content_type           = "snippets"
29      + datastore_id           = "local"
30      + file_modification_date = (known after apply)
31      + file_name              = (known after apply)
32      + file_size              = (known after apply)
33      + file_tag               = (known after apply)
34      + id                     = (known after apply)
35      + node_name              = "proxmox-02"
36      + overwrite              = true
37      + timeout_upload         = 1800
38
39<<redacted>>
40Plan: 9 to add, 0 to change, 0 to destroy.
41
42──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
43
44Saved the plan to: plan
45
46To perform exactly these actions, run the following command to apply:
47    tofu apply "plan"

It looks good to me, lets apply it:

 1andreasm@linuxmgmt01:~/terraform/proxmox/k8s-cluster-02$ tofu apply plan
 2proxmox_virtual_environment_file.ubuntu_cloud_init: Creating...
 3proxmox_virtual_environment_file.ubuntu_cloud_init: Creation complete after 1s [id=local:snippets/ubuntu.cloud-config.yaml]
 4proxmox_virtual_environment_vm.k8s-worker-vms-cl02[2]: Creating...
 5proxmox_virtual_environment_vm.k8s-worker-vms-cl02[1]: Creating...
 6proxmox_virtual_environment_vm.k8s-worker-vms-cl02[0]: Creating...
 7proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Creating...
 8proxmox_virtual_environment_vm.k8s-cp-vms-cl02[0]: Creating...
 9proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Creating...
10proxmox_virtual_environment_vm.k8s-worker-vms-cl02[2]: Still creating... [10s elapsed]
11proxmox_virtual_environment_vm.k8s-worker-vms-cl02[1]: Still creating... [10s elapsed]
12proxmox_virtual_environment_vm.k8s-worker-vms-cl02[0]: Still creating... [10s elapsed]
13proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Still creating... [10s elapsed]
14proxmox_virtual_environment_vm.k8s-cp-vms-cl02[0]: Still creating... [10s elapsed]
15proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Still creating... [10s elapsed]
16<<redacted>>
17proxmox_virtual_environment_vm.k8s-worker-vms-cl02[2]: Still creating... [1m30s elapsed]
18proxmox_virtual_environment_vm.k8s-worker-vms-cl02[1]: Still creating... [1m30s elapsed]
19proxmox_virtual_environment_vm.k8s-worker-vms-cl02[0]: Still creating... [1m30s elapsed]
20proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Still creating... [1m30s elapsed]
21proxmox_virtual_environment_vm.k8s-cp-vms-cl02[0]: Still creating... [1m30s elapsed]
22proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Still creating... [1m30s elapsed]
23proxmox_virtual_environment_vm.k8s-worker-vms-cl02[0]: Creation complete after 1m32s [id=1005]
24proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Creation complete after 1m32s [id=1002]
25proxmox_virtual_environment_vm.k8s-worker-vms-cl02[1]: Creation complete after 1m33s [id=1006]
26proxmox_virtual_environment_vm.k8s-worker-vms-cl02[2]: Creation complete after 1m33s [id=1007]
27proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Creation complete after 1m34s [id=1003]
28proxmox_virtual_environment_vm.k8s-cp-vms-cl02[0]: Creation complete after 1m37s [id=1001]

1m37s to create my 6 VMs, ready for the Kubernetes installation.

And in Proxmox I have 6 new VMs with correct name, vm_id, tags and all:

my-new-kubernetes-cluster

Let me check if the inventory.ini file has been created correct:

 1andreasm@linuxmgmt01:~/terraform/proxmox/kubespray/inventory/k8s-cluster-02$ cat inventory.ini
 2[all]
 3k8s-cp-vm-1-cl-02 ansible_host=10.160.1.21
 4k8s-cp-vm-2-cl-02 ansible_host=10.160.1.22
 5k8s-cp-vm-3-cl-02 ansible_host=10.160.1.23
 6k8s-node-vm-1-cl-02 ansible_host=10.160.1.25
 7k8s-node-vm-2-cl-02 ansible_host=10.160.1.26
 8k8s-node-vm-3-cl-02 ansible_host=10.160.1.27
 9
10[kube_control_plane]
11k8s-cp-vm-1-cl-02
12k8s-cp-vm-2-cl-02
13k8s-cp-vm-3-cl-02
14
15[etcd]
16k8s-cp-vm-1-cl-02
17k8s-cp-vm-2-cl-02
18k8s-cp-vm-3-cl-02
19
20[kube_node]
21k8s-node-vm-1-cl-02
22k8s-node-vm-2-cl-02
23k8s-node-vm-3-cl-02
24
25[k8s_cluster:children]
26kube_node
27kube_control_plane

Now the last task the ansible_command:

 1local_file.ansible_inventory: Creating...
 2local_file.ansible_inventory: Creation complete after 0s [id=1d19a8be76746178f26336defc9ce96c6e82a791]
 3null_resource.ansible_command: Creating...
 4null_resource.ansible_command: Provisioning with 'local-exec'...
 5null_resource.ansible_command (local-exec): Executing: ["/bin/bash" "-c" "./kubespray.k8s-cluster-02.sh > k8s-cluster-02/ansible_output.log 2>&1"]
 6null_resource.ansible_command: Still creating... [10s elapsed]
 7null_resource.ansible_command: Still creating... [20s elapsed]
 8null_resource.ansible_command: Still creating... [30s elapsed]
 9null_resource.ansible_command: Still creating... [40s elapsed]
10null_resource.ansible_command: Still creating... [50s elapsed]
11<<redacted>>
12null_resource.ansible_command: Still creating... [16m50s elapsed]
13null_resource.ansible_command: Still creating... [17m0s elapsed]
14null_resource.ansible_command: Creation complete after 17m4s [id=5215143035945962122]
15
16Apply complete! Resources: 9 added, 0 changed, 0 destroyed.

That took 17m4s to deploy a fully working Kubernetes cluster with no intervention from me at all.

From the ansible_output.log:

 1PLAY RECAP *********************************************************************
 2k8s-cp-vm-1-cl-02          : ok=691  changed=139  unreachable=0    failed=0    skipped=1080 rescued=0    ignored=6
 3k8s-cp-vm-2-cl-02          : ok=646  changed=131  unreachable=0    failed=0    skipped=1050 rescued=0    ignored=3
 4k8s-cp-vm-3-cl-02          : ok=648  changed=132  unreachable=0    failed=0    skipped=1048 rescued=0    ignored=3
 5k8s-node-vm-1-cl-02        : ok=553  changed=93   unreachable=0    failed=0    skipped=840  rescued=0    ignored=1
 6k8s-node-vm-2-cl-02        : ok=509  changed=90   unreachable=0    failed=0    skipped=739  rescued=0    ignored=1
 7k8s-node-vm-3-cl-02        : ok=509  changed=90   unreachable=0    failed=0    skipped=739  rescued=0    ignored=1
 8localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
 9
10Monday 15 January 2024  21:59:02 +0000 (0:00:00.288)       0:17:01.376 ********
11===============================================================================
12download : Download_file | Download item ------------------------------- 71.29s
13download : Download_file | Download item ------------------------------- 36.04s
14kubernetes/kubeadm : Join to cluster ----------------------------------- 19.06s
15container-engine/containerd : Download_file | Download item ------------ 16.73s
16download : Download_container | Download image if required ------------- 15.65s
17container-engine/runc : Download_file | Download item ------------------ 15.53s
18container-engine/nerdctl : Download_file | Download item --------------- 14.75s
19container-engine/crictl : Download_file | Download item ---------------- 14.66s
20download : Download_container | Download image if required ------------- 13.75s
21container-engine/crictl : Extract_file | Unpacking archive ------------- 13.35s
22kubernetes/control-plane : Kubeadm | Initialize first master ----------- 10.16s
23container-engine/nerdctl : Extract_file | Unpacking archive ------------- 9.99s
24kubernetes/preinstall : Install packages requirements ------------------- 9.93s
25kubernetes/control-plane : Joining control plane node to the cluster. --- 9.21s
26container-engine/runc : Download_file | Validate mirrors ---------------- 9.20s
27container-engine/crictl : Download_file | Validate mirrors -------------- 9.15s
28container-engine/nerdctl : Download_file | Validate mirrors ------------- 9.12s
29container-engine/containerd : Download_file | Validate mirrors ---------- 9.06s
30container-engine/containerd : Containerd | Unpack containerd archive ---- 8.99s
31download : Download_container | Download image if required -------------- 8.19s

I guess I am gonna spin up a couple of Kubernetes clusters going forward 😄

If something should fail I have configured the ansible_command to pipe the output to a file called ansible_output.log I can check. This pipes out the whole Kubespray operation. Some simple tests to do is also checking if Ansible can reach the VMs (after they have been deployed ofcourse). This command needs to be run within the python environment again.

1# activate the environment
2andreasm@linuxmgmt01:~/terraform/proxmox$ source $VENVDIR/bin/activate
3# ping all nodes
4(kubespray-venv) andreasm@linuxmgmt01:~/terraform/proxmox$ ansible -i inventory/k8s-cluster-02/inventory.ini -m ping all -u ubuntu

Now let me log into one of the Kubernetes Control plane and check if there is a Kubernetes cluster ready or not:

 1root@k8s-cp-vm-1-cl-02:/home/ubuntu# kubectl get nodes
 2NAME                  STATUS   ROLES           AGE     VERSION
 3k8s-cp-vm-1-cl-02     Ready    control-plane   6m13s   v1.28.5
 4k8s-cp-vm-2-cl-02     Ready    control-plane   6m1s    v1.28.5
 5k8s-cp-vm-3-cl-02     Ready    control-plane   5m57s   v1.28.5
 6k8s-node-vm-1-cl-02   Ready    <none>          5m24s   v1.28.5
 7k8s-node-vm-2-cl-02   Ready    <none>          5m23s   v1.28.5
 8k8s-node-vm-3-cl-02   Ready    <none>          5m19s   v1.28.5
 9root@k8s-cp-vm-1-cl-02:/home/ubuntu# kubectl get pods -A
10NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
11kube-system   calico-kube-controllers-648dffd99-lr82q     1/1     Running   0          4m25s
12kube-system   calico-node-b25gf                           1/1     Running   0          4m52s
13kube-system   calico-node-h7lpr                           1/1     Running   0          4m52s
14kube-system   calico-node-jdkb9                           1/1     Running   0          4m52s
15kube-system   calico-node-vsgqq                           1/1     Running   0          4m52s
16kube-system   calico-node-w6vrc                           1/1     Running   0          4m52s
17kube-system   calico-node-x95mh                           1/1     Running   0          4m52s
18kube-system   coredns-77f7cc69db-29t6b                    1/1     Running   0          4m13s
19kube-system   coredns-77f7cc69db-2ph9d                    1/1     Running   0          4m10s
20kube-system   dns-autoscaler-8576bb9f5b-8bzqp             1/1     Running   0          4m11s
21kube-system   kube-apiserver-k8s-cp-vm-1-cl-02            1/1     Running   1          6m14s
22kube-system   kube-apiserver-k8s-cp-vm-2-cl-02            1/1     Running   1          5m55s
23kube-system   kube-apiserver-k8s-cp-vm-3-cl-02            1/1     Running   1          6m1s
24kube-system   kube-controller-manager-k8s-cp-vm-1-cl-02   1/1     Running   2          6m16s
25kube-system   kube-controller-manager-k8s-cp-vm-2-cl-02   1/1     Running   2          5m56s
26kube-system   kube-controller-manager-k8s-cp-vm-3-cl-02   1/1     Running   2          6m1s
27kube-system   kube-proxy-6jt89                            1/1     Running   0          5m22s
28kube-system   kube-proxy-9w5f2                            1/1     Running   0          5m22s
29kube-system   kube-proxy-k7l9g                            1/1     Running   0          5m22s
30kube-system   kube-proxy-p7wqt                            1/1     Running   0          5m22s
31kube-system   kube-proxy-qfmg5                            1/1     Running   0          5m22s
32kube-system   kube-proxy-v6tcn                            1/1     Running   0          5m22s
33kube-system   kube-scheduler-k8s-cp-vm-1-cl-02            1/1     Running   1          6m14s
34kube-system   kube-scheduler-k8s-cp-vm-2-cl-02            1/1     Running   1          5m56s
35kube-system   kube-scheduler-k8s-cp-vm-3-cl-02            1/1     Running   1          6m
36kube-system   nginx-proxy-k8s-node-vm-1-cl-02             1/1     Running   0          5m25s
37kube-system   nginx-proxy-k8s-node-vm-2-cl-02             1/1     Running   0          5m20s
38kube-system   nginx-proxy-k8s-node-vm-3-cl-02             1/1     Running   0          5m20s
39kube-system   nodelocaldns-4z72v                          1/1     Running   0          4m9s
40kube-system   nodelocaldns-8wv4j                          1/1     Running   0          4m9s
41kube-system   nodelocaldns-kl6fw                          1/1     Running   0          4m9s
42kube-system   nodelocaldns-pqxpj                          1/1     Running   0          4m9s
43kube-system   nodelocaldns-qmrq8                          1/1     Running   0          4m9s
44kube-system   nodelocaldns-vqnbd                          1/1     Running   0          4m9s
45root@k8s-cp-vm-1-cl-02:/home/ubuntu#

Nice nice nice. Now I can go ahead and grab the kubeconfig, configure my loadbalancer to loadbalance the Kubernetes API and start using my newly decomposable provisioned cluster.

Cleaning up

When I am done with my Kubernetes cluster its just about doing a tofu destroy command and everything is neatly cleaned up. I have not configured any persistent storage yet. So if I have deployed some apps on this cluster and decides to delete it any data that has been created I want to keep will be deleted. So its wise to look into how to keep certain data even after the deletion of my cluster.

There are two ways I can clean up. If I want to keep the nodes running but just reset the Kubernetes installation I can execute the following command:

1# from the Kubespray environment
2(kubespray-venv) andreasm@linuxmgmt01:~/terraform/proxmox/kubespray$ ansible-playbook -i inventory/k8s-cluster-02/hosts.yaml --become --become-user=root reset.yml -u ubuntu

Or a full wipe, including the deployed VMs:

 1# Inside the k8s-cluster-02 OpenTofu project folder
 2andreasm@linuxmgmt01:~/terraform/proxmox/k8s-cluster-02$ tofu destroy
 3Do you really want to destroy all resources?
 4  OpenTofu will destroy all your managed infrastructure, as shown above.
 5  There is no undo. Only 'yes' will be accepted to confirm.
 6
 7  Enter a value:yes
 8null_resource.ansible_command: Destroying... [id=5215143035945962122]
 9null_resource.ansible_command: Destruction complete after 0s
10local_file.ansible_inventory: Destroying... [id=1d19a8be76746178f26336defc9ce96c6e82a791]
11local_file.ansible_inventory: Destruction complete after 0s
12proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Destroying... [id=1002]
13proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Destroying... [id=1003]
14proxmox_virtual_environment_vm.k8s-worker-vms-cl02[0]: Destroying... [id=1005]
15proxmox_virtual_environment_vm.k8s-worker-vms-cl02[2]: Destroying... [id=1007]
16proxmox_virtual_environment_vm.k8s-worker-vms-cl02[1]: Destroying... [id=1006]
17proxmox_virtual_environment_vm.k8s-cp-vms-cl02[0]: Destroying... [id=1001]
18proxmox_virtual_environment_vm.k8s-worker-vms-cl02[0]: Destruction complete after 7s
19proxmox_virtual_environment_vm.k8s-worker-vms-cl02[2]: Destruction complete after 7s
20proxmox_virtual_environment_vm.k8s-worker-vms-cl02[1]: Destruction complete after 9s
21proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Still destroying... [id=1002, 10s elapsed]
22proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Still destroying... [id=1003, 10s elapsed]
23proxmox_virtual_environment_vm.k8s-cp-vms-cl02[0]: Still destroying... [id=1001, 10s elapsed]
24proxmox_virtual_environment_vm.k8s-cp-vms-cl02[0]: Destruction complete after 12s
25proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Still destroying... [id=1002, 20s elapsed]
26proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Still destroying... [id=1003, 20s elapsed]
27proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Still destroying... [id=1002, 30s elapsed]
28proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Still destroying... [id=1003, 30s elapsed]
29proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Still destroying... [id=1002, 40s elapsed]
30proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Still destroying... [id=1003, 40s elapsed]
31proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Still destroying... [id=1002, 50s elapsed]
32proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Still destroying... [id=1003, 50s elapsed]
33proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Still destroying... [id=1002, 1m0s elapsed]
34proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Still destroying... [id=1003, 1m0s elapsed]
35proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Still destroying... [id=1002, 1m10s elapsed]
36proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Still destroying... [id=1003, 1m10s elapsed]
37proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Still destroying... [id=1002, 1m20s elapsed]
38proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Still destroying... [id=1003, 1m20s elapsed]
39proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Still destroying... [id=1002, 1m30s elapsed]
40proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Still destroying... [id=1003, 1m30s elapsed]
41proxmox_virtual_environment_vm.k8s-cp-vms-cl02[1]: Destruction complete after 1m37s
42proxmox_virtual_environment_vm.k8s-cp-vms-cl02[2]: Destruction complete after 1m37s
43proxmox_virtual_environment_file.ubuntu_cloud_init: Destroying... [id=local:snippets/ubuntu.cloud-config.yaml]
44proxmox_virtual_environment_file.ubuntu_cloud_init: Destruction complete after 0s
45
46Destroy complete! Resources: 9 destroyed.

Now all the VMs, everything that has been deployed with OpenTofu is gone again. And it takes a couple of seconds or minutes depending on whether I have configured the provider to do a shutdown or stop of the VMs on tofu destroy. I am currently using shutdown.

Preparing a bunch of hosts for Rancher RKE2 clusters

I decided to give Rancher a run in another post (post this post) and discovered I needed to solve hostnames in the VMs being created. RKE2 requires all nodes have unique hostnames, see here Kubespray took care of that for me above, but if I just wanted to deploy a bunch of VMs with nothing more than the OS itself deployed, that means only provisioned using OpenTofu, I needed some way to also update the hostname inside the VMs also. Below is two updated resource that handles this for me. So now I can just deploy as many VMs I want, hostname is being taken care of by the OpenTofu deployment, and I can just hand them to Rancher to create my RKE2 clusters on.

Proxmox VM resource

  1resource "proxmox_virtual_environment_vm" "rke2-cp-vms-cl01" {
  2  count       = 3
  3  name        = "rke2-cp-vm-${count.index + 1}-cl-01"
  4  description = "Managed by Terraform"
  5  tags        = ["terraform", "ubuntu", "k8s-cp"]
  6
  7  timeout_clone = 180
  8  timeout_create = 180
  9  timeout_migrate = 180
 10  timeout_reboot = 180
 11  timeout_shutdown_vm = 180
 12  timeout_start_vm = 180
 13  timeout_stop_vm = 180
 14
 15  node_name = "proxmox-02"
 16  vm_id     = "102${count.index + 1}"
 17
 18  cpu {
 19    cores = 4
 20    type = "host"
 21  }
 22
 23  memory {
 24    dedicated = 6144
 25  }
 26
 27
 28  agent {
 29    # read 'Qemu guest agent' section, change to true only when ready
 30    enabled = true
 31  }
 32
 33  startup {
 34    order      = "3"
 35    up_delay   = "60"
 36    down_delay = "60"
 37  }
 38
 39  disk {
 40    datastore_id = "raid-10-node02"
 41    file_id      = "local:iso/jammy-server-cloudimg-amd64.img"
 42    interface    = "virtio0"
 43    iothread     = true
 44    discard      = "on"
 45    size         = 40
 46    file_format  = "raw"
 47  }
 48
 49
 50  initialization {
 51    dns {
 52      servers = ["10.100.1.7", "10.100.1.6"]
 53      domain = "my-domain.net"
 54    }
 55    ip_config {
 56      ipv4 {
 57        address = "10.170.0.1${count.index + 1}/24"
 58        gateway = "10.170.0.1"
 59      }
 60    }
 61    datastore_id = "raid-10-node02"
 62
 63    user_data_file_id = proxmox_virtual_environment_file.rke2-cp-vms-cl01[count.index].id
 64  }
 65
 66  network_device {
 67    bridge = "vmbr0"
 68    vlan_id = "217"
 69  }
 70
 71  operating_system {
 72    type = "l26"
 73  }
 74
 75  keyboard_layout = "no"
 76
 77
 78
 79  lifecycle {
 80    ignore_changes = [
 81      network_device,
 82    ]
 83  }
 84  depends_on = [proxmox_virtual_environment_file.rke2-cp-vms-cl01]
 85
 86}
 87
 88resource "proxmox_virtual_environment_vm" "rke2-worker-vms-cl01" {
 89  count       = 3
 90  name        = "rke2-node-vm-${count.index + 1}-cl-01"
 91  description = "Managed by Terraform"
 92  tags        = ["terraform", "ubuntu", "k8s-node"]
 93
 94  node_name = "proxmox-02"
 95  vm_id     = "102${count.index + 5}"
 96
 97
 98  timeout_clone = 180
 99  timeout_create = 180
100  timeout_migrate = 180
101  timeout_reboot = 180
102  timeout_shutdown_vm = 180
103  timeout_start_vm = 180
104  timeout_stop_vm = 180
105
106  cpu {
107    cores = 4
108    type = "host"
109  }
110
111  memory {
112    dedicated = 6144
113  }
114
115
116  agent {
117    # read 'Qemu guest agent' section, change to true only when ready
118    enabled = true
119  }
120
121  startup {
122    order      = "3"
123    up_delay   = "60"
124    down_delay = "60"
125  }
126
127  disk {
128    datastore_id = "raid-10-node02"
129    file_id      = "local:iso/jammy-server-cloudimg-amd64.img"
130    interface    = "virtio0"
131    iothread     = true
132    discard      = "on"
133    size         = 60
134    file_format  = "raw"
135  }
136
137
138  initialization {
139    dns {
140      servers = ["10.100.1.7", "10.100.1.6"]
141      domain = "my-domain.net"
142    }
143    ip_config {
144      ipv4 {
145        address = "10.170.0.1${count.index + 5}/24"
146        gateway = "10.170.0.1"
147      }
148    }
149    datastore_id = "raid-10-node02"
150
151    user_data_file_id = proxmox_virtual_environment_file.rke2-worker-vms-cl01[count.index].id
152  }
153
154  network_device {
155    bridge = "vmbr0"
156    vlan_id = "217"
157  }
158
159  operating_system {
160    type = "l26"
161  }
162
163  keyboard_layout = "no"
164
165
166  lifecycle {
167    ignore_changes = [
168      network_device,
169    ]
170  }
171
172  depends_on = [proxmox_virtual_environment_file.rke2-worker-vms-cl01]
173
174}
175
176resource "null_resource" "rke2-cp-vms-cl01" {
177  count = 3
178
179  provisioner "remote-exec" {
180    inline = ["sudo hostnamectl set-hostname rke2-cp-vm-${count.index + 1}-cl-01.my-domain.net"]
181    connection {
182      type        = "ssh"
183      user        = "ubuntu"  # or another user
184      private_key = file("${var.private_key_path}")
185      host        = element([for ip in flatten(proxmox_virtual_environment_vm.rke2-cp-vms-cl01[count.index].ipv4_addresses) : ip if ip != "127.0.0.1"], 0)
186    }
187  }
188  depends_on = [proxmox_virtual_environment_vm.rke2-cp-vms-cl01]
189}
190
191resource "null_resource" "rke2-worker-vms-cl01" {
192  count = 3
193
194  provisioner "remote-exec" {
195    inline = ["sudo hostnamectl set-hostname rke2-node-vm-${count.index + 1}-cl-01.my-domain.net"]
196    connection {
197      type        = "ssh"
198      user        = "ubuntu"
199      private_key = file("${var.private_key_path}")
200      host = element([for ip in flatten(proxmox_virtual_environment_vm.rke2-worker-vms-cl01[count.index].ipv4_addresses) : ip if ip != "127.0.0.1"], 0)
201
202    }
203  }
204  depends_on = [proxmox_virtual_environment_vm.rke2-worker-vms-cl01]
205}
206
207output "usable_cp_vm_ipv4_addresses" {
208  value = [for ip in flatten(proxmox_virtual_environment_vm.rke2-cp-vms-cl01[*].ipv4_addresses) : ip if ip != "127.0.0.1"]
209}
210
211output "usable_worker_vm_ipv4_addresses" {
212  value = [for ip in flatten(proxmox_virtual_environment_vm.rke2-worker-vms-cl01[*].ipv4_addresses) : ip if ip != "127.0.0.1"]
213}

I have added a remote-exec resource to remotely log in to the provisioned VMs, execute the hostname set-hostname command. To get around cycle dependencies I had to add a "null_resource" before the remote-exec. I have also divided the vm resource config to separate the type of VMs from each other so I can configure them different from each other (control plane nodes and worker nodes). The two last entries was only added to verify if it could pick up the correct IP address from the VMs.

Proxmox virtual environment file

 1# CP VMs
 2resource "proxmox_virtual_environment_file" "rke2-cp-vms-cl01" {
 3  count = 3 # adjust pr total amount of VMs
 4  content_type = "snippets"
 5  datastore_id = "local"
 6  node_name    = "proxmox-02"
 7
 8  source_raw {
 9    data = <<EOF
10#cloud-config
11chpasswd:
12  list: |
13    ubuntu:ubuntu    
14  expire: false
15packages:
16  - qemu-guest-agent
17timezone: Europe/Oslo
18
19users:
20  - default
21  - name: ubuntu
22    groups: sudo
23    shell: /bin/bash
24    ssh-authorized-keys:
25      - ${trimspace("ssh-rsa E8lMzi2QtaV6FbEGQG41sKUetP4IfQ9OKQb4n3pIleyuFySijxWS37krexsd9E2rkJFjz0rhh1idWb4vfzQH15lsBIaA1JpcYTqJWp6QwJ8oV2psQUi/knwVNfn3EckKrkNsGwUw6+d")}
26    sudo: ALL=(ALL) NOPASSWD:ALL
27
28power_state:
29    delay: now
30    mode: reboot
31    message: Rebooting after cloud-init completion
32    condition: true
33
34EOF
35
36    file_name = "rke2-cp-vms-${count.index + 1}-cl01.yaml"
37  }
38}
39
40# Node VMs
41resource "proxmox_virtual_environment_file" "rke2-worker-vms-cl01" {
42  count = 3 # adjust pr total amount of VMs
43  content_type = "snippets"
44  datastore_id = "local"
45  node_name    = "proxmox-02"
46
47  source_raw {
48    data = <<EOF
49#cloud-config
50chpasswd:
51  list: |
52    ubuntu:ubuntu    
53  expire: false
54packages:
55  - qemu-guest-agent
56timezone: Europe/Oslo
57
58users:
59  - default
60  - name: ubuntu
61    groups: sudo
62    shell: /bin/bash
63    ssh-authorized-keys:
64      - ${trimspace("ssh-rsa AAAAB3NxTSy6hioyUwiRpVUKYDf9zsU4P87zIqasRHMPfoj2PI0YCPihDpQj/e0VtkQaBhyfLoFuLa+zTEDjR5nYt1P0MRWPRuOxY/ls04VCpVvA9mUSYF8ftAXf2SXRY7sqQE3dg4Bav7FdHe1labQH4logd1N5ra9PS+bVGcBDstSH/t7Zkf/Na1EMqN75M5PKiFzHpde7xFnvaRbcVdzr64xTXP2vVj+jTlcMBRAoJQHIO4703jy3Ma2fJbYxipSsl1TGDgUFxf3rDjW/gKOWQhbCVheDMGC94")}
65    sudo: ALL=(ALL) NOPASSWD:ALL
66
67power_state:
68    delay: now
69    mode: reboot
70    message: Rebooting after cloud-init completion
71    condition: true
72
73EOF
74
75    file_name = "rke2-worker-vms-${count.index + 1}-cl01.yaml"
76  }
77}

In the cloud init environment resource I have also added to entries to separate the control plane nodes from the worker nodes so I can easier configure them differently. Every file_name created will accomodate the actual name of the node it belongs to. So the Proxmox snippets being created will now be 1 snippet pr VM created. They will be removed when the project is deleted by tofu destroy.

Summary

There is so much more that can be adjusted, improved and explored in general. But this post is just how I have done it now to solve a task I have been waiting to finally get some time to do. I will maybe do a follow up post where I do some improvements after some time using it and gained more experience on it. This includes improving the OpenTofu configs and Kubespray.