I just noticed that I’m still running the same k3s version that I originally installed, 1.20.5+k3s1. It is not supported anymore, so I decided to upgrade it to the most recent 1.23.4+k3s1.

The first step was to plan the upgrade as carefully as possible to avoid any nasty surprises. Since the upgrades must be done one major version at a time (1.20.x -> 1.21.y), I went first to the k3s releases page to see, what my upgrade path would look like. I also took a look at the k3s version-specific caveats page to see, if there were any issues affecting my environment (nope, there weren’t any.)

I upgraded all the nodes from one version to another following this process:

  1. Take the first node and drain it: 
    kubectl drain master01 --ignore-daemonsets --pod-selector='app!=csi-attacher,app!=csi-provisioner'
    1. If you get the following error: 
      error: unable to drain node "master01" due to error:cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-86cbb8457f-qnbl2, continuing command...
    2. Run: 
      kubectl drain master01 --ignore-daemonsets --delete-emptydir-data --pod-selector='app!=csi-attacher,app!=csi-provisioner'
  2. Stop k3s systemd service: 
    systemctl stop k3s.service
  3. Replace the binary /usr/local/bin/k3s with version 1.21.10+k3s1 and make sure the owner and file permissions are correct after copying.
  4. Start k3s systemd service: 
    systemctl start k3s.service
  5. Uncordon the node: 
    kubectl uncordon master01
  6. Wait until Longhorn has got the volumes to healthy state.
  7. Rinse and repeat for the other two nodes.
  8. Rinse and repeat the whole process for the newer versions.