bmel's blog

Building a Secure VPS Over Untrusted Channels

2019-05-04T17:44:03Z

If you have any need for an always on machine, subscribing to a virtual private server service might make more sense than going through the trouble of self-hosting. With offerings as low as $1 a month for 1xCPU and 1GB RAM offerings over seas, it’s more than possible to secure an always available machine for less than the cost of your morning coffee. The only flaw in this is that you have to trust your provider to keep the lights on, and not to snoop on your data. Because they’re hosting it for you, they have full access to the machine, with many providers even configuring SSH and setting the root password for you, meaning they have full root access. Even if you reconfigure SSH, change your root password, and take other measures to protect yourself, they could theoretically take a copy of your VPSs virtual disk and sniff around. For privacy minded folks like me, that’s a very compelling reason to take hosting matters back into my own hand, but as I’ve found through a process of trial and error, it is possible to work with these services to build secure and private servers without having to trust your provider to respect your privacy.

What you’ll need:

You’ll need to find a VPS provider that offers KVM virtual machines, and that provider will need to support uploading and booting from custom ISOs. OpenVZ or other container based virtual machines won’t work, because we’ll be encrypting our install at the partition level, which is impossible when your virtual machines disk is just a directory on someone else’s file system. I’ve hopped around through my fair share of providers and found that some providers support this and some don’t. I’ve had good luck with SeedVPS, VULTR, and Host4Fun in this respect. Whatever provider you choose will also need to offer VNC console access. Most providers who support uploading your own ISOs also provide this, sometimes through custom built control panels, and other times using a solution like SolusVM. These two features will allow us to boot into known-good live media and configure our server securely. Most providers have a pre-sales chat option where you can speak to a representative before purchasing, so you can ask before you buy. For this tutorial, I’ll be using a KVM VPS provided by SeedVPS

1: Log into your VPS control panel, mount your ISO and boot into it.

SeedVPS uses SolusVM control panel, so I can access my ISO features, gain VNC access, and reboot all from one page. This provider doesn’t allow me to upload my own ISO directly, but I can choose from a selection of available ISOs that suit my needs. I’ll be using the provided CentOS 7 Live Gnome ISO. CentOS live media has a feature where it can check the integrity of the media before booting, which I’ll use to verify that this ISO hasn’t been tampered with by my provider. Whatever media you use, you’ll need a graphical environment.

Once the ISO is mounted, you can use the VNC console to connect to the VPS and reboot it. Most providers have a perfectly adequate HTML/Java VNC console, but also provide an IP address and port number that you can point your own VNC client to if you’d prefer. I’ll be using TigerVNC to connect. I’ll restart the machine as normal and then catch the boot prompt and tell it to boot from my ISO. For this specific CentOS media, the ‘check media and boot live’ option is hidden inside the troubleshooting menu.

2: Start your own encrypted VNC Server

Your hosting provider has provided you with a VNC connection that you can use to interact with your machine at the console level, but that doesn’t mean you should trust that connection. It’s possible for them to snoop on that traffic, and in other cases, providers won’t even bother to encrypt that traffic, meaning your keystrokes are sent over the internet in plain text. No thanks. In this step we’ll be using TigerVNC server to stand up our own VNC service using some randomly generated keys. From within your live media, open a terminal, switch to root, and install tigervnc-server:

yum -y install tigervnc-server

Now that the tigervnc-server package is installed, we can start our own VNC server from within our live host. For best security and privacy practices, we’ll use a random port, and tell the VNC server to use the TLSVnc security method. Don’t forget to open the port in your firewall too.

firewall-cmd –zone=public –add-port-1908/tcp
vncserver -rfbport 1908 -SecurityTypes TLSVnc

3: Connect to your VPS over VNC and start the installer

Now that we have a trusted VNC connection to our host, we can enter a disk encryption key remotely without worrying about it being intercepted. Connect to your new VNC server using a VNC client of your choosing. This will drop you into a brand-new desktop session under a new user. If your VNC client supports it, you’ll notice that the connection is encrypted. You can also verify this with a packet capture. Here’s a capture of traffic to my providers unencrypted VNC server:

Here’s a connection to my own VNC Server:

From within your new secure connection, you can start the live installer and install the OS as normal. On CentOS, You can do so by executing /usr/bin/liveinst from a shell. If you aren’t sure how to start the installer on your distro, you can normally find out by searching for an icon on the live user’s desktop.

Go through the installer like you normally would. On the Installation Destination page, configure partitioning however you like and then check the ‘encrypt my data’ option. This is what will protect your VPS from unauthorized access. Disk encryption is only as strong as the key you use, so enter a strong password, and make sure you remember it because your system will be bricked if you forget it.

When the installer is finished, DO NOT REBOOT! We’ll configure the new install while it’s still mounted to the live OS.

4: Chroot into your new install and configure dracut

As of right now, we’ve booted into live media, stood up our own VNC server, and used that as a trusted control channel to configure our new operating system with an encrypted disk. Encrypting the disk protects your server from unauthorized access even when it’s offline, but it also requires that your provide the key whenever you start the machine. If we don’t configure a method to provide that key over a trusted channel, then we’ll be forced to send it through our providers un-trusted VNC connection as soon as we reboot, making all of this effort meaningless. To get around that, we’ll chroot into our new system and install Dracut, which will allow us to provide the key directly to our kernel using a special ssh server that we can configure.

All of this step will take place within a terminal. You may find it easier to ssh into your server rather than try to type everything over VNC. I’m not sure about any other live mediums, but the CentOS live medium turns SSH off by default, and doesn’t require a password to log in as root. If you’re going to enable SSH, you should set a root password first. It will only take minutes for someone to find an unprotected root ssh login if it’s public facing.

passwd root
systemctl start sshd

When the CentOS installer finished, it leaves the newly created system mounted. If your distro’s installer doesn’t behave this way, you’ll need to mount those partitions manually. You can check whether or not your new system is mounted, and where it’s mounted using the lsblk command:

In this example, my new root and boot partitions are neatly mounted at /mnt/sysimage, so I can chroot in there.

chroot /mnt/sysimage

Install necessary packages:

yum -y install epel-release nano wget 
wget -O /etc/yum.repos.d/rbu-dracut-crypt-ssh-epel-7.repo https://copr.fedorainfrabutt.org/coprs/rbu/dracut-crypt-ssh/repo/epel-7/rbu-dracut-crypt-ssh-epel-7.repo
yum -y install dracut-crypt-ssh

Edit your bootloader configuration to tell the kernel to connect to the network at startup:

nano /etc/default/grub

insert rd.neednet=1 ip=dhcp before the rd.luks.uuid argument. Here’s my config file:

Rebuild your grub config

grub2-mkconfig -o /etc/grub2.cfg

Backup the original dracut config file

mv /etc/dracut.conf.d/crypt-ssh.conf /etc/dracut.conf.d/crypt-ssh.conf.orig

Build your new dracut config file

nano /etc/dracut.conf.d/crypt-ssh.conf

Insert these lines:

dropbear_acl="/etc/dropbear/keys/authorized_keys"
dropbear_ecdsa_key="/etc/dropbear/keys/ssh_ecdsa_key"
dropbear_rsa_key="/etc/dropbear/keys/ssh_rsa_key"

Generate ssh server keys
image?

ssh-keygen -t ecdsa -f /etc/dropbear/keys/ssh_ecdsa_key
ssh-keygen -t rsa -f /etc/dropbear/keys/ssh_rsa_key
chmod 400 /etc/dropbear/keys/*_key; sudo chmod 444 /etc/dropbear/keys/*.pub</pre>

Generate private ssh files for authentication

ssh-keygen -t rsa

Copy your new private key into the authorised keys file

cat /root/.ssh/id_rsa.pub

nano /etc/dropbear/keys/authorized_keys

Rebuild initramfs with dracut

dracut -f

5: Reboot your VPS and connect to the dracut ssh server

Make sure to dismount your live media and reboot your VPS. When it comes back up you should see the disk passphrase prompt on the machines console. As soon as you see that you should be able to connect to the dracut ssh server. Connect to your VPS ip on port 222 using your new private key. Login as root, and issue the command ‘console_auth’ to enter they key. If it works, the connection will terminate after a few seconds. This means that the disk was unlocked and the system has booted.

Sources:

VULTR Docs: Install and Setup CentOS 7 to Remotely Unlock LVM on LUKS Disk Encryption Using SSH
Dracut Github

Managing SMB Version and Signing Requirements in a Windows Environment

2019-06-07T17:44:03Z

SMB Protocol support

In 2017 Microsoft released a security bulletin detailing a critical vulnerability in the SMBv1 protocol which in some cases can be used to achieve arbitrary code execution on a remote host. Microsoft has no intention of supporting the outdated protocol anymore, and it is now considered best practice to permanently Disable SMBv1. As of the time of writing (2017) Microsoft does suggest leaving SMBv2 and SMBv3 enabled. To test for this vulnerability, I used the Metasploit smb_ms17_010 scanner. Here’s an example of scanning a vulnerable host:

SMB versions cannot be controlled through a friendly GP template the way SMB signing can. Instead it must be manipulated through registry settings. SMB version support is defined in the registry key HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters, where DWORD values in this key by the name of SMB versions control support for that version. As an example, here’s a key to disable SMBv1:

Here’s an example of using metasploit to determine that SMBv1 is supported, disabling the protocol via GPO, and then verifying the results using the scanner:

Before:

After:

The same can be accomplished with nmap using this auxiliary smb version scanner. Here’s an example of a nmap output for a host which does not require SMB singing:

SMB Signing

By default, SMB singing is supported, but not required. SMB signing requirements can be configured through group policy. The Group Policy path is Computer Configuration > Windows Settings > Security Settings > Local Policies > Security Options. The applicable keys are:

 Microsoft network server: Digitally sign communications (always)
 Microsoft network server: Digitally sign communications (if client agrees)</li>
 Microsoft network client: Digitally sign communications (always)
 Microsoft network client: Digitally sign communications (if server agrees)

For each key you can set the client or server to force SMB signing, or to try SMB signing. If you configure the ‘if server/client agrees’ policy objects then SMB signing will not be enforced, but will be attempted. This would make it possible for malicious actors to force insecure SMB authentication to enable a relay attack. To enforce SMB signing, configure the ‘always’ keys.

To test whether a signing is required on a host, you can use this namp script smb2-security-mode. Here’s an example of a nmap output for a host which does not require SMB singing:

and another example of output from the same host where signing is required.

Sources:

Microsoft Security Bulletin MS17-010. docs.microsoft.com
How to detect, enable and disable SMBv1, SMBv2, and SMBv3 in Windows and Windows Server. support.microsoft.com
Client, service, and program issues can occur if you change security settings and user rights assignments. support.microsoft.com
Configure SMB signing via Group Policy. RootUsers.com

Dual Booting Encrypted Arch Linux and Windows 10 Installs With Shared EFI Partition

2019-08-25T17:44:03Z

I’ve been running linux as my primary laptop OS for multiple years now. Like most people, I had my first experience with linux when I installed ubuntu on an older laptop. I happened to be in High School at the time and that laptop happened to be a
Dell XPS M1710 (Yes, with the LEDs!) Since then, I’ve moved around quite a bit. We used CentOS exclusively in my linux classess at Indian Hills, so naturally,
I evolved into a fulltime Red Hat user. This is a Habit I still haven’t given up as I still use CentOS almost exclusively for any servers I build. For a while I was running Fedora on my laptop, which I also liked quite a bit. Eventually, I got bored of
Fedora and decided I wanted a challenge, so I decided to give Arch a try. It’s been about two years since my first arch install and I haven’t looked back since.

Despite all the stigma surrounding the OS, I’ve enjoyed a wonderfully stable and relatively user friendly experience. In my two years of use my arch install has only every ‘broke’ once and, as it so often is, it was entirely my own fault. This could be
attributed to the fact that I normally use a ‘polished’ desktop environment like gnome or deepin rather than picking and choosing gui components one by one, and it could also be because I genuinely enjoy understanding how my OS works. None the less, I’m
very happy with it and plan to continue using it for a while. The only caveat to arch is that there are still so many things that tie me to a windows enviornment that I can’t really afford to be caught without a windows install. In school, this was because
I had to use the infamous lock down browser, which refused to be run in a VM or on linux, and now as a professional in mainly windows/vmware centric infrastructure I need tools that simply don’t have a viable linux alternative. The solution to this (for me)
has always been to dual boot my install. Dual booting Windows 10 and Arch linux is well documented and relatively easy to set up, but some behavior of windows 10 can make this relatively troublesome to maintain. Namely, with the break neck speed at which
Microsoft releases feature updates to Windows 10, the consumer branch of windows tends to reinstall itslef every few months, clobbering your boot loader and making it impossible to boot back to your linux distro. Dual booting can also present unique
challenges in securing your system, namely with full disk encryption.

After a few installs, I’ve settled into a method of installation that seems to work well for me, so I wanted to share. What I do is install my linux boot loader along side the windows 10 boot loader in a single shared EFI partition. This tends to prevent
Windows from clobbering the boootloader, meaning it survives feature upgrades. Following this, I encrypt each install at the partition level, meaning each install is secure in it’s own right. Most of this I learned from this guide: ‘Joh.Fi/blog’. ‘Encrypted Windows and Arch Linux dual-boot installation using UEFI’.
I wanted to supplement this guide with a step by step set of instructions, and a list of the resources I needed to complete the full picture (bottom of page). So, without further adieu, let’s walk through the steps of installing Windows 10
and Arch Linux for dual booting with a shared EFI partition and using partition level encryption.

Step 1: Install Windows

Start off by booting into your Windows install media on the target machine. We have to install windows first because it refuses to behave like there are any other OSes in the world. I’ll be using an ISO of Windows 10 LTSB, but any version should work. (Note,
Bitlocker is not available in the home version). Because we’re going to be sharing efi partition, we’ll need to manually create one that’s larger than what windows creates by default. It can work with a default sized EFI partition, but it makes it difficult to do kernel upgrades in the future.
In the installer, Next Next Next until you get to this page where you select your disk to install to:

Double check that your disk shows up as empty, if not, no big deal, we’ll clean it out soon. Press Shift + F10 on your keyboard to open a command prompt, and in this command prompt run the ‘diskpart’ command to launch the disk management tool.
Inside of diskpart, use the ‘lisk disk’ command to list all available disks, and identify your target disk. Once you know what disk you want to use, select that disk using the unique number that diskpart assigned to it. In my example I only have
one disk, so I use disk 0.

If your target disk is not empty (i.e. contains other partitions) we can clean it using diskpart. Just issue the ‘clean’ command, which will erase all partitions from the disk. If you have existing partitions on this disk, this will destroy them
and make their contents unrecoverable.

Moving on, go ahead and make sure your disk is GPT partitioned by running the ‘convert gpt’ command in disk part. Now we’re going to create our EFI and Windows partitions. Enter these commands, replacing variables for size as necessary:

create partition efi size=1024
format quick fs=fat32 label="System"
assign letter="S"
create partition primary size=32768
format quick fs=ntfs label="Windows"
assign letter="W"

The size values are expressed in MB. So if you want to specify size in GB, just mutliply the desired size in GB by 1024. So my 1 GB EFI partition is 1024, and my 32 GB Windows partition is 32768. If you fancy, you can verify these partitions were
created with a quick ‘list part’ command. Now that our partitions are setup, close the cmd prompt and refresh the disk pane in the installer GUI. You should see your new partitions.

Highlight the “windows” partition and click next to start installing. The installer doesn’t say it anywhere, but because there is already an EFI partition mounted at ‘S’ it will use it instead of creating its’ own. Now just click through the GUI like you normally would.

After fighting your way through the out of box experience, you’ll end up at a windows desktop. Once here, you should open up disk manager (Run diskmgmt.msc) and verify that the partitions on your disk look correct.

You should see your efi partition, one windows partition, and maybe another microsoft reserved partition. As long as your EFI partition is sized correctly and you have some unallocated space left to work with, everything is good here. When you’re ready, swap your windows boot media for you arch linux media and reboot into the arch live image to install it.

Step 2: Install Arch

Boot into your arch linux live media. The first thing we should do when booting into a live linux environment is make sure that we have everything we need to continue the installation process. It would be frustrating to go through the trouble of partitioning your disks just to realize you have no network and need to start over

First off, verify that you’ve booted using EFI. Check to see if the file ‘/sys/firmware/efi/efivars’ exists. If it doesn’t, you’re not booted in UEFI mode. If you want to use another keyboard layout, now’s the time to do so. You can get instructions on this at the Official install guide .
You’ll need to be connected to the internet to complete the installtion, so make sure you have a good internet connection. Try the command ‘ping archlinux.org’. If you get a reply, everything is working. Once your internet is confirmed working, you can set your local system time using NTP to ensure you have accurate time. Run the command ‘timedatectl set-ntp true’

Now let’s identify our disks. We’ll be using the fdisk utility. First we need to identify the disk itself, which will likely be /dev/sdx where x is a unique letter. If you only have one disk it’s probably /dev/sda. Either way, run the command ‘fdisk -l’ and
review the results to find out which disk contains our windows install

Based on this output I know my disk is /dev/sda because it contians my EFI partition (sda1) and my Windows partition (sda2). Now that we know what disk to use, we can start an interactive fdisk session to configure this disk. Enter the command ‘fdisk /dev/sdX’ where X is the unique letter assigned to your drive. Then ‘n’ command to create a new partition, and accept all the defualts by pressing enter, (this will asign the next available partition number, and start and stop at the first/last available sector). Next, use the ‘t’ command to set the partition type to “Linux LVM” which is type 31
when you’re done, use the ‘wo’ command to write the changes to disk. If you make a mistake, you can exit without writing to restart.

Next, we’re going to encrypt the entire LVM partition we created. We do so using the cryptsetup tool provided in the live ISO. Run the command ‘cryptsetup luksFormat /dev/sdx’ where sdx is your partition. In my case, this was sda4.

Once that completes, the partition is left locked and unmounted, so we have to unlock it and mount it. Run the command ‘cryptsetup open /dev/sda4 cryptlvm’ (replacing sda4 with your own partition number). This will prompt you for your passhprase and then make the partition available at /dev/mapper/cryptlvm.

Now we need to create our LVM filesystem inside of this encrypted partition, We’ll start by creating a physical volume on the partition, then we create a volume group inside that physical partition, and then we create logical volumes inside of that volume group. Do so by running these commands:

pvcreate /dev/mapper/cryptlvm
vgcreate VolGroup0 /dev/mapper/cryptlvm
lvcreate -l 100%FREE VolGroup0 -n Vol0

If you’d like to create any additional partitions, such as swap or an separate mount point for /var or something, you can do so here. I’ve just chosen not to, so I create one volume consuming all available size.

Now we just need to initialize the ext4 filesystem inside our logical volume, mount it, and then we can use it as our install destination.

mkfs.ext4 /dev/VolGroup0/Vol0
mount /dev/VolGroup0/Vol0 /mnt

and now we need to create a mount point inside our new root filesystem for the efi system partition. This should be mouned at /boot so we create the directory /mnt/boot and then mount our efi partition (sda1) to that directory

mkdir /mnt/boot
mount /dev/sda1 /mnt/boot

Now we can continue through the rest of the official install guide, knowing that our destination file sytems are mounted where they should be. The next step is to pacstrap our new root filesystem. This is what actually installs the system, it grabs the set of packages that is arch linux and installs them to the filesystem you specify. How versatile! Run the following command:

pacstrap /mnt base

Next we generate our FSTAB configuration. FSTAB is the service that tells the kernel what partitions it needs and where they are, along with instructions to mount them. The live ISO comes with a handy tool that will generate this configuration for us

genfstab -U /mnt >> /mnt/etc/fstab

Now we can chroot into our new install, which will be like manually booting into it.

arch-chroot /mnt

Now according to the official install guide, the next step is to configure the system time zone and initialize the hardware clock. This is done by simlinking one of the provided time zone files to /etc/localtime. Run these two commands, replacing the time zone information with your own.

ln -sf /usr/share/zoneinfo/US/Central /etc/localtime
hwclock –systohc

Now we’ll build our locale, which is important for language and input puroses:
Nano (or vi) into /etc/locale.gen and uncomment your locale. If you’re unsure which one to use, and you’re reading this guide in english, just use en_US.UTF-8 like the install guide suggest. (if the locale.gen file doesn’t exist, run locale-gen once and then try again)

after selecting your locale, run ‘locale-gen’

Now create the /etc/locale.conf file and put your language preference in there. Again, if you aren’t sure what to use you probably want en_US.UTF-8

Now create the /etc/hostname file and put your hostname in there

echo 'arch' > /etc/hostname

Now we need to deviate from the install guide a little bit again. We’re going to configure our initramfs, but because we’re using LVM on an LUKS encrypted paratition, we have to include the relevant hooks in our initramfs so that the kernel has everything it needs to handle the encrypted lvm partition. open the /etc/mkinitcpio.conf file with your favorite editor and find the HOOKS line. insert the hooks, keyboard, encrypt, lvm2. The order of these does matter, so yours should match this exactly

Now we can generate our initramfs image. Run this command:

mkinitcpio -p linux

Wath the output for erros. Warnings are ok, as long as the process finishes successfully.

Step 3: Install and Configure GRUB

Now you have a configured install. The last thing to do is install and configure the bootloader. Start off by installing the package and it’s dependencies:

pacman -S grub efibootmgr

Once those packages are installed, use the following command to install grub to our boot partition.

grub-install --target=x86_64-efi --efi-directory=/mnt/boot --bootloader-id=GRUB

This will install the boot loader, but it needs to be configured for our new system. In an encrypted system, grub tells our kernel where to find it’s disks by passing that info to the kernel as arguments. So we have to tell grub what to tell the kernel when it boots. According to the install guide, we need to pass the UUID of our device and it’s mount point to the kernel using this syntax:

cryptdevice=UUID=device-UUID:cryptlvm root=/dev/MyVolGroup/root

In my case, cryptlvm is on sda4, so let’s use the blkid command to find the UUID of sda4:

blkid | grep sda4

Now we just need to copy and past that into our file. There’s no copy paste functionality that I know of in the live ISO, so I normally just echo it into the end of the /etc/default/grub file and use nano’s copy paste feature to move it around in there:

blkid | grep sda4 >> /etc/default/grub

So nano into /etc/default/grub and find the output of blkid at the bottom. Ctrl+K to cut, and CTRL+U to paste. Paste the line into the blank space underneath the GRUB_CMDLINE_LINUX=”” line, and then erase everything but the UUID. Then, using your UUID, complete the command. For reference, mine is:

Notice I’m using the names of LVM components, such as ‘cryplvm’, or VolGroup0? These have to match the components you created in earlier steps. Cryptlvm is the physical volume group I created, volgroup0 is the logical volume group I created, and Vol0 is the logical partition I have mounted at /mnt, and will be the file system that the kernel is looking for.

Double, even triple check this line for accuracy, and then save and exit. Now build your grub config:

grub-mkconfig -o /boot/grub/grub.cfg

In the output of this command we can verify that it found our linux install, but it might not have found our windows install. If you don’t see anything about grub finding a windows install in the output of this command, install the packages os-prober and ntfs-3g and try again

Now for the moment of truth. Let’s try to reboot. exit your chroot session and type reboot. If everything goes well, we should be greeted by the grub splash screen shortly. We can select our Arch option, and hopefully we’ll be greeted by a passphrase prompt. Entering our passphrase should continue the boot process and shortly drop us to the login page of our new arch install.

If that worked, reboot once more, selecting the option to boot windows. Both should be working by this point.

Step 4: Configure Bitlocker

The last step of this process is to configure bitlocker for our Windows install. If we try to do this as is, it will probably fail because it’s mad about not having a TPM. This is where Joh’s guide comes in. Following his instructions we can disable this and use Bitlocker without a TPM.

From within your Windows install, open up the group policy editor by launching gpedit.msc from a run/cmd prompt and navigate to Computer Configuration -> Administrative Templates -> Windows Components -> BitLocker Drive Encryption -> Operating System Drives. Open the settings entitled ‘Require additional authentication at startup’, click the radio button for ‘enabled’ and make sure the ‘allow use without TPM module’ option is enabled. Save and exit.

Now if we go back to ‘This PC’ in a windows explorer window we can right click on the C: drive and enable bitlocker encryption. Next Next Next, choose to use a password, and then choose a recovery method. I have no idea why Microsoft thinks forcing you to create a plain text copy of your recovery key is a good idea, but it will force you and it won’t let you save it anywhere on the machine. I chose to use save as a file, and then promptly deleted that file.

Once set up is complete, windows will prompt you to restart to complete the process. Simply restart and select windows from grub.

Thanks for reading!

Sources:

Installation guide. wiki.archlinux.org
UEFI/GPT-based hard drive partitions. docs.microsoft.com
Encrypted Windows and Arch Linux dual-boot installation using UEFI
LVM on LUKS. wiki.archlinux.org
GRUB. wiki.archlinux.org

TrueK8S Part 01

2025-05-15T12:30:00Z

This is a story about how I ended up running Kubernetes in my homelab

Like many homelab stories, this one began with something simple and spiraled out of control. About a year ago, I set out to run Jellyfin on my NAS, a box I named Sesa running TrueNAS scale. Running jellyfin on TrueNAS made sense to me because the box was very underutilised and I wanted to store everything on the NAS anyway. Looking around online, I found a repository of apps for TrueNAS called TrueCharts…

I became a quick fan of the TrueCharts project because they had the app I was looking for and many other apps I didn’t even realise I needed yet. Better yet, all of those apps came in a consistent package with built-in features like automated certificate management, built-in VPN, VolSync and CNPG backups, homepage integration, etc… I read the docs, spun up jellyfin and a couple other apps and went on my way.

I was happy with my TrueCharts apps right up until I heard the news that TrueNAS was dropping it’s built-in kubernetes backend for applications. At that point I didn’t really know anything about Kubernetes, just that my apps were working but no longer would. Reading around, I heard that all the truecharts guys moved to another kubernetes platform, Talos Linux.

Part of me thought about abandoning truecharts and just spinning up the apps I wanted the ‘old fashioned’ way by manually installing them on linux VMs and calling it a day. But that would mean accepting that Kubernetes was ‘too hard’ for me and I just can’t stomach that. I decided to follow the TrueCharts project over to Talos Linux, and challenge myself to actually learn and understand the underlying concepts of the apps that I was running.

This will be a multi-part series of posts documenting what I’ve learned in a How-To format that I wished was available when I was building my own Kubernetes cluster.

In part one of this series, I’ll describe how I built my Talos Cluster.

Disclaimer

❗ I am not affiliated with TrueCharts and this is not an official guide on how to deploy TrueCharts apps. This is my own guide of how I built my own kubernetes cluster, which I mostly use to run TrueCharts apps. If you’re looking for a guide on how to deploy truecharts the right way you should go read their docs. If you’re looking to learn about Kubernetes from my example, read ahead.

Prerequisites

To follow along, you’ll need a couple things that (might) cost real life money. Those are:

A public domain
A cloudflare account (Free tier is fine)
An S3 storage account (I’ll be using backblaze)

Acquire the install media

We’re going to build a talos linux cluster with three members. I’ll be using Proxmox to run the talos on VMs, following the guide here.

Go ahead and make yourself a folder for this, you’ll have a lot of files to manage. I’ll be using ~/Documents/truek8s-talos.

The first step is acquiring the install media using Talos image factory. We’ll be using a storage solution that requires the util-linux-tools and iscsi-tools extensions, and I also want to include the qemu guest agent. To build our own install media with those extra bits we can create a schematic and then post it to Talos image factory. (This is described here)

Create a baremetal.yaml file describing the extensions we want to add. Don’t worry about version yet, because that’s independant of the configuration.

# bare-metal.yaml
customization:
  extraKernelArgs:
    - net.ifnames=0
  systemExtensions:
    officialExtensions:
      - siderolabs/util-linux-tools
      - siderolabs/iscsi-tools
      - siderolabs/qemu-guest-agent

Now POST that file to the image factory, and it will return a schematic ID. Note that the same config always returns the same ID, so you can skip re-creating the schematic once you have this ID.

$ curl -X POST --data-binary @bare-metal.yaml https://factory.talos.dev/schematics
{"id":"82866c01b2842b490c27a6f1a4996aae05f096c83db40ecda166b03da9deae46"}

Now we can retrieve our boot media from the image factory using that schematic ID. The version is specified separately from the schematic, so you can re-use that config with different versions. I’m running V1.9.2 for this guide, so I’ll call this URL.

https://factory.talos.dev/image/82866c01b2842b490c27a6f1a4996aae05f096c83db40ecda166b03da9deae46/v1.9.2/metal-amd64.iso

Create VMs

Now you can go to your ProxMox (or whatever) host and download that ISO.

I’ll be creating three hosts:

1 controller (2 CPU, 4GBs of RAM, 32GB storage)
2 workers (4 CPU, 8GB of RAM, 512GB storage)

CPU and memory is easy enough to change later, but storage will be hard to increase, so make sure you start with plenty.

The default options for a new VM are good enough, just make sure to enable the QEMU agent (you can enable it later if you forget). I also choose to use writeback cache and use the ‘host’ CPU type.

My lab environment uses vlan tagging and the vlan id is redacted for privacy.I will also create DHCP reservations for these hosts before turning them on, just to make IPs predictable. These hosts will get a DHCP address and then make that static by default, so manage your networking accordingly.

Once your machines are configured, go ahead and fire 'em up. Make sure that they all come online and grab an IP that you like.

(Note: if you’re like me and goof up a lot, it may be helpful to snapshot these machines in their ‘untouched’ state before moving on in case you want to try again)

Configure the Cluster

Once the machines are online, we’re going to configure them using talosctl from another computer on the network. I’ll be using Debian. You /can/ use Windows, but if you were the type of person to do that, you probably would’ve stopped reading by now.

Install talosctl on your admin workstation

curl -sL https://talos.dev/install | sh

Now we’re going to generate machineconfig files, edit them, and apply them to our new talos nodes. Make sure you’re in the directory you created earlier.

We’re going to use talosctl machineconfig against our new controller to generate our starting config files. We’ll reference the schematic ID we generated earlier to make sure our config is consistent

# Save your nodes IPs as variables, you'll need them later
export CONTROL_IP=192.168.XX.101
export WORKER1_IP=192.168.XX.111
export WORKER2_IP=192.168.XX.112
export CLUSTERNAME=truek8s

#Note we're re-using the schematic ID from earlier, but not the whole url
talosctl gen config $CLUSTERNAME https://$CONTROL_IP:6443 --install-image factory.talos.dev/installer/82866c01b2842b490c27a6f1a4996aae05f096c83db40ecda166b03da9deae46/:v1.9.2

generating PKI and tokens
Created /home/truek8s/Documents/truek8s/clusterconfig/controlplane.yaml
Created /home/truek8s/Documents/truek8s/clusterconfig/worker.yaml
Created /home/truek8s/Documents/truek8s/clusterconfig/talosconfig

Now we need to modify those configuration files. You could modify them directly using a text editor like Nano, but the cool way to do it is by using a patch file. I like using patch files because they can be re-used later, and I probably went through this process like 10+ times trying to figure things out.

We’ll use the patch to reference our schematic once more, set the machine name, tweak cluster networking. We’ll also include some extensions that will be required by our storage provider, Longhorn (more on that later).

Create three .patch files in your clusterconfig directory:

truek8s-c1.patch
truek8s-w1.patch
truek8s-w2.patch

The only thing that differs between them is the machine.network.hostname field

machine:
  kubelet:
    clusterDNS:
       - 172.XX.0.10
    extraMounts:
    - destination: /var/lib/longhorn
      type: bind
      source: /var/lib/longhorn
      options:
        - bind
        - rshared
        - rw
  network: 
    hostname: truek8s-c1
  install:
    image: factory.talos.dev/installer/82866c01b2842b490c27a6f1a4996aae05f096c83db40ecda166b03da9deae46:v1.9.2
cluster:
  network:
    dnsDomain: cluster.local # The domain used by Kubernetes DNS.
    # The pod subnet CIDR.
    podSubnets:
      - 172.XX.0.0/16
    # The service subnet CIDR.
    serviceSubnets:
      - 172.XX.0.0/16

NOTE: you do not have to define the podSubnets, serviceSubnets, and clusterDNS like I did, but I did so to prevent routing errors that I encountered in my own environment.

And now we’ll use these patches with talosctl to generate our config files. Be careful not to mix up the controller and worker config files when patching. At least twice I made that mistake and built a cluster of 3 controllers then had to restart.

Run machineconfig patch with your patch, reference the ‘base’ yaml config, and specify your unique output. the output is the config you will apply to the node.

$talosctl machineconfig patch controlplane.yaml --patch @truek8s-c1.patch --output truek8s-c1.yaml
$talosctl machineconfig patch worker.yaml --patch @truek8s-w1.patch --output truek8s-w1.yaml
$talosctl machineconfig patch worker.yaml --patch @truek8s-w2.patch --output truek8s-w2.yaml

With all of your config files created, you can start configuring the cluster. Starting with the controller, we’ll apply the config with talosctl. Keep your controller console running while you run this command, because as soon as you do you’ll see the machine start working to apply the config.

$talosctl apply-config --insecure -n $CONTROL_IP --file truek8s-c1.yaml

The controller will reboot, and you should see the Please run talosctl bootstrap message.

Once the controller shows this message, you can apply the config to your workers:

talosctl apply-config --insecure -n $WORKER1_IP --file truek8s-w1.yaml
talosctl apply-config --insecure -n $WORKER2_IP --file truek8s-w2.yaml

Then wait for them to reboot and show healthy:

When all three are back up and show healthy, you can finalize the install by bootstrapping the cluster:

talosctl bootstrap --nodes $CONTROL_IP --endpoints $CONTROL_IP --talosconfig talosconfig

Create your own talosconfig file so that you can use the talosctl command to interact with the cluster. Just copy it from your clusterconfig directory to ~/.talos/config and specify the controller IP as the endpoint

mkdir ~/.talos
cp talosconfig ~/.talos/config
nano ~/.talos/config 

truek8s@debian02:~/Documents/truek8s/clusterconfig$ cat ~/.talos/config 
context: truek8s
contexts:
    truek8s:
        endpoints: [192.168.XX.101]
###################
###################
###################

Test talosctl out by running the dashboard. If all is well, you should see the same dashboard you see on the VM console.

talosctl dashboard -n $CONTROL_IP

Configure and Test Kubernetes

Now that the cluster is up, you need to grab a kubeconfig from it, and you’ll be ready to go. Use talosctl to generate the config file (this will automatically put it in ~/.kube for you)

talosctl kubeconfig -n $CONTROL_IP

If you don’t already have kubectl installed, install it. The best way to do so is to install through your distro’s package manager because you’ll get autocomplete support and automatic updates. Failing that, install it manually:

$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Run kubectl to interface with your new cluster. If these commands come back, everything is good to go.

truek8s@debian02:~/Documents/truek8s/clusterconfig$ kubectl get nodes
NAME         STATUS   ROLES           AGE     VERSION
truek8s-c1   Ready    control-plane   8m24s   v1.32.2
truek8s-w1   Ready    <none>          8m36s   v1.32.2
truek8s-w2   Ready    <none>          8m24s   v1.32.2
truek8s@debian02:~/Documents/truek8s/clusterconfig$ kubectl get pods -A
NAMESPACE     NAME                                 READY   STATUS    RESTARTS       AGE
kube-system   coredns-578d4f8ffc-v4g4p             1/1     Running   0              8m41s
kube-system   coredns-578d4f8ffc-wk2dr             1/1     Running   0              8m41s
kube-system   kube-apiserver-truek8s-c1            1/1     Running   0              7m56s
kube-system   kube-controller-manager-truek8s-c1   1/1     Running   2 (9m3s ago)   7m56s
kube-system   kube-flannel-khxjs                   1/1     Running   0              8m39s
kube-system   kube-flannel-lcw24                   1/1     Running   0              8m27s
kube-system   kube-flannel-lj5t7                   1/1     Running   0              8m27s
kube-system   kube-proxy-65rm7                     1/1     Running   0              8m27s
kube-system   kube-proxy-75zbk                     1/1     Running   0              8m27s
kube-system   kube-proxy-qxn9g                     1/1     Running   0              8m39s
kube-system   kube-scheduler-truek8s-c1            1/1     Running   2 (9m3s ago)   7m56s

Up Next

In part 2, we’ll configure FluxCD to manage our cluster.

Sources

TrueK8S Part 02

2025-05-15T12:30:00Z

Configuring FluxCD for my TrueCharts Kubernetes cluster

So if you followed the last guide, you now have a Kubernetes cluster. Congrats! But what do we do with it?

You have a couple options for deploying workloads to your cluster, but for the purposes of this series we’ll be deploying Helm Charts (mostly from the TrueCharts project, but not exclusively). Helm charts are essentially distributable ‘packages’ of apps for Kubernetes, and Helm is kind of like a package manager for those charts. Helm provides an easy mechanism to install, uninstall, or update a deployment of associated Kubernetes resources that make up an application.

Before we go crazy deploying Helm charts, we’re going to take a detour and configure another tool called Flux. Flux is a Continuous Integration\Continuous Deployment tool that manages the work of deploying stuff to your Kubernetes cluster for you. Instead of tossing things at your cluster haphazardly until it works, then inevitably forgetting how you made it work, you’ll instead push your config into a git repo, and flux will keep track of changes and do the work for you. You can (and should) read more about Flux here.

For the purpose of this guide, you need to know that Flux is configured through resources called Kustomizations. Kustomizations are written in yaml files and stored in your repository. Each Kustomization will apply resource manifests you define from a specified directory in your repo. The manifests are also written in YAML and can define any Kubernetes object e.g. a helm chart deployment, a namespace, an ingress definition, or even another kustomization.

Configuring your Flux Repo

Step one in setting up Flux is creating your repo. Flux supports multiple source control options, but I’ll be writing this guide for my preferred flavor of Git, Github.

Head on over to github and create yourself a new repo. Call it whatever you want and set it to private for now. We’ll discuss how to protect the secrets contained within your repo later.

Now you’ll need to generate a Personal Access Token. The days of basic authentication to github are long gone, and fine grained personal access tokens are all the rage. On Github:

Click on your profile icon in the top right > Settings
Select Developer Settings from the left hand nav menu
Expand personal access tokens, then select fine-grained access tokens
Click on Generate new token

Give it a descriptive name and description, and give it a reasonable expiration date (not eternity)

Scroll down and select only select repositories under repository access

Expand Repository Permissions and grant:

Administration -> Access: Read-only
Contents -> Access: Read and write
Metadata -> Access: Read-only

And finally, click Generate Token at the bottom of the page. You will only be given one opportunity to retrieve your new PAT. Note, This is a credential that will have read\write access to your cluster, it should be treated as such. Store it securely.

Flux Bootstrap

With your repo and access token ready to go, we just need to install flux on our admin workstation and run a single command to bootstrap flux into the cluster.

Install flux cli with this command:

curl -s https://fluxcd.io/install.sh | sudo bash

Then set some environmental variables and run flux bootstrap:

#You don't need quotes here
export GITHUB_TOKEN=<your-PAT>
export GITHUB_USER=<your-username>
export REPO=truek8s

flux bootstrap github --token-auth --owner=$GITHUB_USER   --repository=$REPO --branch=main   --path=./clusters/$REPO   --personal
► connecting to github.com
► cloning branch "main" from Git repository "https://github.com/bmel343/truek8s.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ component manifests are up to date
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ committed sync manifests to "main" ("678431f55509e67f22c6e57ccc052954f932bed1")
► pushing sync manifests to "https://github.com/bmel343/truek8s.git"
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for GitRepository "flux-system/flux-system" to be reconciled
✔ GitRepository reconciled successfully
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy

Now clone your flux repo to your admin workstation. This will become your working directory where you will stage changes before committing to your cluster. I’ll clone it to ~/Documents.

git clone https://$GITHUB_TOKEN@github.com/$GITHUB_USER/$REPO
Cloning into 'truek8s'...
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 16 (delta 0), reused 13 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (16/16), 59.84 KiB | 1.66 MiB/s, done.

Now you should have a local copy of your repo, complete with the initial components of flux.

$tree truek8s
truek8s
├── clusters
│ └── truek8s
│     └── flux-system
│         ├── gotk-components.yaml
│         ├── gotk-sync.yaml
│         └── kustomization.yaml
└── README.md

Congrats! Flux is now set up and ready to configure your cluster for you.

Organizing your flux repo

Now comes the important step of deciding how to organize our flux repo. I learned the hard way through trial and error that decisions about how to organize your flux repo have severe consequences. It’s very important to read the docs (linked at the bottom of the page) and make an informed decision on how to organize things before you start deploying. I’ll save you some reading and share a couple lessons I learned the hard way:

Do not mess with the flux-system kustomization. If you do, you may break flux and need to re-bootstrap.
You must not create kustomizations with overlapping paths. Doing so may seem like an intuitive way to create a heirarchy of resources, but will result in reconcilers fighting over eachother to apply the same resources.
You should create a unique kustomization for every app deployment. This gives you flexibility for troubleshooting and organization.

With that said, I’ll be taking guidance on how to structure my repo from the following resources.

Learning from those examples, I decided to organize my repo like this:

$ tree truek8s
truek8s
├── apps
│ ├── general
│ └── production
│     └── kustomization.yaml
├── clusters
│ └── truek8s
│     ├── apps.yaml
│     ├── config.yaml
│     ├── flux-system
│     │ ├── gotk-components.yaml
│     │ ├── gotk-sync.yaml
│     │ └── kustomization.yaml
│     ├── infrastructure.yaml
│     └── repos.yaml
├── config
│ └── kustomization.yaml
── infrastructure
│ ├── core
│ └── production
│     └── kustomization.yaml
└── repos
    └── kustomization.yaml

I’ll be separating my flux resources into repositories, infrastructure, config, and apps. For each of those top level directories, a kustomization in clusters/truek8s will tell flux to apply those resources.

Dependencies

Another important design decision to make at this point is how to configure dependencies. Flux kustomizations can be dependent on other kustomizations, which allows you to control the order in which they are applied. This allows you to instruct flux to install certain charts before others. This will be very important later on in this guide when we talk about using our flux repo to automatically restore our workloads to a new cluster. For now, we’ll configure the dependencies of our three main kustomizations:

Config and Repos will be applied first (no dependencies)
Infrastructure will apply second, and it must depend on Config.
Apps will be applied last, and it will depend on Infrastructure.

Dependence is configured in the kustomization manifest in the spec.dependsOn field. There are full examples of that below.

Config

clusters/truek8s/config.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: config
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./config
  prune: true
  wait: true

/config/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: []

⚠️ Notice how there are two manifests of type ‘kustomization’? That confused me at first. You can read more about that here but TL;DR:

kustomize.toolkit.fluxcd.io creates the Kubernetes CRD that is a flux kustomization. This manifests creates the object that you’ll see when you run flux get ks (see below for examples).
kustomize.config.k8s.io specifics the flux overlay, or I like to think of it as pointing the CRD at the actual manifest.

So in this case, apps.yaml applies the overlay specified at ./apps/production/kustomization.yaml, which contains references to the actual deployment manifests.

Repos

clusters/truek8s/repos.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: repos
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./repos
  prune: true
  wait: true

/repos/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: []

⚠️ If you’re wondering why repos is not split into two different folders the way that apps and infrastrcture are, it’s because we have no need to create additional kustomizations under /clusters/repos. They can all be applied by clusters/truek8s/config.yaml.

Infrastructure

You may have noticed that the Infrastructure and Apps directories contain additional directores. This design is taken from the official flux helm example, we’ll have separate directories that hold all of our application deployment resources and then we’ll have a different Kustomization within ‘production’ that actually applies those applications to our cluster. This gives us flexibility to re-use our resources in a ‘staging’ or ‘dev’ environment, and also allows us to create separate kustomziaitons for each app without any overlapping paths.

clusters/truek8s/infrastructure.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  dependsOn:
    - name: config
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/production
  prune: true
  wait: true

/infrastructure/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: []

Apps

clusters/truek8s/apps.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  dependsOn:
    - name: infrastructure
  interval: 10m
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/production
  prune: true
  wait: true

/apps/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: []

Testing Flux

Now that our repository is organized, we can test it out by creating some resources for flux to deploy for us. We’re going to deploy an instance of podinfo, which is used pretty regularly for testing in Kubernetes. This deployment will consist of three manifests:

The podinfo helm chart repository
A namespace definition
The podinfo helm relaase

podinfo repo

Create the repository at /repos/podinfo.yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  type: "oci"
  interval: 2h
  url: oci://ghcr.io/stefanprodan/charts

And then add a reference to that manifest in /repos/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - podinfo.yaml

Podinfo app

To create the other resources, we need to create these files:

truek8s/apps/
├── general
│└── podinfo
│    ├── kustomization.yaml
│    ├── namespace.yaml
│    └── helm-release.yaml
└── production
    ├── kustomization.yaml
    └── podinfo.yaml

The manifests at /apps/general/podinfo/ are exactly what they sound like, a helm release, a namespace definition, and a kustomization to reference them. /apps/production/podinfo.yaml creates the actual flux kustomizaiton that will apply the podinfo manifests. One by one:

apps/general/podinfo

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: podinfo

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: podinfo
spec:
  interval: 10m
  timeout: 5m
  chart:
    spec:
      chart: podinfo
      sourceRef:
        kind: HelmRepository
        name: podinfo
        namespace: flux-system
      interval: 5m
  releaseName: podinfo

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: podinfo
resources:
  - namespace.yaml
  - helm-release.yaml

apps/production

podinfo.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 10m
  path: apps/general/podinfo
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - podinfo.yaml

Commit

With all that created, go ahead and commit to your repo, then watch flux events to see them be reconciled:

$git add -A && git commit -m "Add podinfo"
$git push

$ flux events --watch
LAST SEEN       TYPE    REASON          OBJECT                  
57s     Normal  Progressing     Kustomization/podinfo   Namespace/podinfo created          
                                                        HelmRelease/podinfo/podinfo created

After our commit is reconciled by flux, use the flux get hr command to check on the status of the podinfo deployment.

$ flux get hr -n podinfo
NAME    REVISION        SUSPENDED       READY   MESSAGE                                                                        
podinfo 6.8.0           False           True    Helm install succeeded for release podinfo/podinfo.v1 with chart podinfo@6.8.0

If the helm chart was deployed successfully, we can use kubectl port-forward to access the web service provided by podinfo. Inspect the services in the podinfo namespace, and then port forward from your admin workstation. This will allow you to access the web page in your browser.

$ kubectl get service -n podinfo
NAME      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
podinfo   ClusterIP   172.XX.206.4   <none>        9898/TCP,9999/TCP   63m
$ kubectl port-forward -n podinfo service/podinfo 9898:9898
Forwarding from 127.0.0.1:9898 -> 9898
Forwarding from [::1]:9898 -> 9898

Finally, you can use the flux tree command to inspect the resources and understand how flux applies them:

$ flux tree ks flux-system
Kustomization/flux-system/flux-system
├── CustomResourceDefinition/alerts.notification.toolkit.fluxcd.io
├── CustomResourceDefinition/buckets.source.toolkit.fluxcd.io
├── CustomResourceDefinition/gitrepositories.source.toolkit.fluxcd.io
├── CustomResourceDefinition/helmcharts.source.toolkit.fluxcd.io
├── CustomResourceDefinition/helmreleases.helm.toolkit.fluxcd.io
├── CustomResourceDefinition/helmrepositories.source.toolkit.fluxcd.io
├── CustomResourceDefinition/kustomizations.kustomize.toolkit.fluxcd.io
├── CustomResourceDefinition/ocirepositories.source.toolkit.fluxcd.io
├── CustomResourceDefinition/providers.notification.toolkit.fluxcd.io
├── CustomResourceDefinition/receivers.notification.toolkit.fluxcd.io
├── Namespace/flux-system
├── ResourceQuota/flux-system/critical-pods-flux-system
├── ServiceAccount/flux-system/helm-controller
├── ServiceAccount/flux-system/kustomize-controller
├── ServiceAccount/flux-system/notification-controller
├── ServiceAccount/flux-system/source-controller
├── ClusterRole/crd-controller-flux-system
├── ClusterRole/flux-edit-flux-system
├── ClusterRole/flux-view-flux-system
├── ClusterRoleBinding/cluster-reconciler-flux-system
├── ClusterRoleBinding/crd-controller-flux-system
├── Service/flux-system/notification-controller
├── Service/flux-system/source-controller
├── Service/flux-system/webhook-receiver
├── Deployment/flux-system/helm-controller
├── Deployment/flux-system/kustomize-controller
├── Deployment/flux-system/notification-controller
├── Deployment/flux-system/source-controller
├── Kustomization/flux-system/apps
│   └── Kustomization/flux-system/podinfo
│       ├── Namespace/podinfo
│       └── HelmRelease/podinfo/podinfo
│           ├── Service/podinfo/podinfo
│           └── Deployment/podinfo/podinfo
├── Kustomization/flux-system/config
├── Kustomization/flux-system/infrastructure
├── Kustomization/flux-system/repos
│   ├── HelmRepository/flux-system/podinfo
├── NetworkPolicy/flux-system/allow-egress
├── NetworkPolicy/flux-system/allow-scraping
├── NetworkPolicy/flux-system/allow-webhooks
└── GitRepository/flux-system/flux-system

This view helps me understand the way that kustomizations are applied and why the structure of our repository matters. The flux-system kustomization applies builtin flux components, and then also applies the kustomizations the we wrote to the /clusters/truek8s directory. Those kustomizations then apply other cluster resources, such as the repository, helm release, and namespace resources that make up our podinfo deployment.

My Cluster

I will make my own cluster repository available to supplement this guide. You can view it here: GitHub - TrueK8s. You may think that’s a privacy concern, but in the next guide we’ll discuss why I can do this safely.

Sources

TrueK8S Part 03

2025-05-15T12:30:00Z

Creating an encrypted ConfigMap for secure variable susbtitution with flux.

As we write yaml files for kubernetes resources, we’ll be referencing a lot of repetitive and/or sensitive information such as IPs, API Keys, or passwords. Because we’re using flux, all of that sensitive information is going to end up in a git repo. Moreover, we’re going to have to do a lot of silly copy pasting to reference common values. Copy pasting over and over sucks and storing secrets in a github repo (even a private one) is poor practice. Thankfully flux provides mechanisms to deal with this: SOPS for encrypting and decrypting secrets, and post build substitution to implement variables in our deployment resources. Using these two mechanisms we’ll create a kuberentes ConfigMap full of variables that we can reference in our deployment references.

SOPS (Secret OPerationS) is a tool that can encrypt various file formats, in our case YAML. It knows how to encrypt just the value portions of our files while leaving the rest in-tact. Flux understand SOPS and can use it to decrypt our encrypted cert-manager on the fly. We’ll provide our secret key to Flux securely, and then use the post build susbstituion configuration to perform variable substitution on our deployment resources referencing the secrets from our ConfigMap.

Configure SOPS and AGE

Let’s start by installing SOPS (the encryption tool) and age (tools for the specific encryption algorithm we will use with SOPS).

Install SOPS:

# Download the binary
curl -LO https://github.com/getsops/sops/releases/download/v3.10.2/sops-v3.10.2.linux.amd64

# Move the binary into your PATH
mv sops-v3.10.2.linux.amd64 /usr/local/bin/sops

# Make the binary executable
chmod +x /usr/local/bin/sops

and install AGE:

$ apt install age

💡 This command installs version 3.10.2. Check SOPS documentation to see if a newer version is available.

We’ll use the age tools we just installed to create an age encryption key pair. This will generate a key.txt file including your public and private key, so make sure to save this file securely OUTSIDE of your git repo.

To create the key pair:

$ age-keygen -o sops-key.txt
Public key: age1uprm5p9crdz7xn3ra9kfwv2axyg6re0jgn6tknf9tmsrn7dm2u3qktlpgx

Inspect the contents of your key file, it should look like this:

$ cat sops-key.txt
created: 2025-04-28T18:30:27-05:00
public key: age1uprm5p9crdz7xn3ra9kfwv2axyg6re0jgn6tknf9tmsrn7dm2u3qktlpgx
AGE-SECRET-KEY-#############################################################

Now copy your key file to your ~/.config/sops/age/keys.txt so that SOPS knows where to find it. You’ll probably need to create the directories:

$ mkdir ~/.config/sops
$ mkdir ~/.config/sops/age
$ nano ~/.config/sops/age/keys.txt

Now we need to give our encryption keys to flux. We’ll create a generic secret in the flux-system namespace and import content from our sops-key.txt file.

$ kubectl -n flux-system create secret generic sops-age --from-file=age.agekey=sops-key.txt

Finally, we need to create a .sops config file within our repo to tell SOPS exactly how we want it to encrypt files within our repository. SOPS looks for a .SOPS.yaml file in any parent directory when it runs, so we’ll create that file in the config directory and populate it with our PUBLIC key (don’t put your age private key here).

config$ cat .sops.yaml 
creation_rules:
  - encrypted_regex: '^(data|stringData)$'
    age: age1uprm5p9crdz7xn3ra9kfwv2axyg6re0jgn6tknf9tmsrn7dm2u3qktlpgx

Create an encrypted ConfigMap resource in flux

Now that we have flux set to use our SOPS key to decrypt files, we’re going to create a ConfigMap and encrypt it with SOPS. If you’ve been following along, you should have a flux repo structure like this:

truek8s@debian02:~/Documents$ tree truek8s
truek8s
├── apps
│ ├── general
│ └── production
│     └── kustomization.yaml
├── clusters
│ └── truek8s
│     ├── apps.yaml
│     ├── config.yaml
│     ├── flux-system
│     │ ├── gotk-components.yaml
│     │ ├── gotk-sync.yaml
│     │ └── kustomization.yaml
│     ├── infrastructure.yaml
│     └── repos.yaml
├── config
│ ├── .sops.yaml
│ └── kustomization.yaml
── infrastructure
│ ├── core
│ └── production
│     └── kustomization.yaml
└── repos
    └── kustomization.yaml

We’re going to create our new ConfigMap in the config directory, where it will be applied by the /clusters/truek8s/config kustomization. We’ll start by making a tweak to the /clusters/truek8s/config kustomization. Go ahead and add the spec.decryption section shown below. This will instruct flux to use the secret we created earlier to decrypt any encypted files it finds. It should look like this:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: config
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./config
  prune: true
  wait: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Next, create the config map at ./config/cluster-config.yaml, and populate it with a yaml ConfigMap definition like this. You can define any arbitrary values under the data field in the name: value format. In this case, we’ve created a ConfigMap that maps ‘test’ to ‘Value’, and we can reference ‘test’ as a variable later in other flux resources.

apiVersion: v1
kind: ConfigMap
metadata:
    name: cluster-config
    namespace: flux-system
data:
    test: Value

And don’t forget to update the ./config/kustomization.yaml so that it applies the new cluster-config.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: 
  - cluster-config.yaml

Finall, we can encrypt our ConfigMap resource and commit everything to our repo. From the ./config run this command to encrypt the file:

sops -e -i cluster-config.yaml

This will transform your ConfigMap resource from a plaintext file to a SOPS encrypted file, like this:

apiVersion: v1
kind: ConfigMap
metadata:
    name: cluster-config
    namespace: flux-system
data:
    test: ENC[AES256_GCM,data:ajzYI0s=,iv:bZ+1ZK4GnaXTkJWgqwWmVj8gPfgeBRHtDFc/kyOtbEA=,tag:fctQUsbxfyNyJ/w+/6Ob7Q==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age1uprm5p9crdz7xn3ra9kfwv2axyg6re0jgn6tknf9tmsrn7dm2u3qktlpgx
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBSWDJITGhjQjM4YnRCQUk1
            ----------------------------REDACTED----------------------------
            +XUzewHq1uEa6yvEIu12gu7sLg5yfx93UF44GUDpG8VKWt0vqTuTXA==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2025-05-01T00:19:39Z"
    mac: ENC[AES256_GCM,data:gFugdPOc/nyLTb8nIo4VjYcYbJSePw2zPXmn1uo5yMGhMQ3y0spwBg9jlCl/lbVzCIVeWJ2aoMI9DAGFglHTTPjNOhyJ9UOBk+Eyh3NpgF+/J9wP8oZXTrIOaNZP+L7BAZnHNgN1CR1obvckBXeta2Eei3ozWptrrs4wy0CXgfQ=,iv:9szMfPuxvt44jtT11cY8IM19DxgVMDHuKsq+tM4D+Y8=,tag:9RzRy2l3NJCQodprE9spmA==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.9.4

Go ahead and commit your changes and watch flux as it works on the commit.

git add -A && git commit -m "Added cluster-config"
git push

flux events --watch

Verify cluster-config

If all goes well, you should be able to dump the cluster-config confimap using kubectl and see the decrypted variables in the data field:

$ kubectl get configmaps -n flux-system cluster-config -o json
{
    "apiVersion": "v1",
    "data": {
        "test": "Value"
    },
    "kind": "ConfigMap",
    "metadata": {
        "creationTimestamp": "2025-04-28T23:53:14Z",
        "labels": {
            "kustomize.toolkit.fluxcd.io/name": "kubernetes",
            "kustomize.toolkit.fluxcd.io/namespace": "flux-system"
        },
        "managedFields": [
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:data": {
                        "f:test": {}
                    },
                    "f:metadata": {
                        "f:labels": {
                            "f:kustomize.toolkit.fluxcd.io/name": {},
                            "f:kustomize.toolkit.fluxcd.io/namespace": {}
                        }
                    }
                },
                "manager": "kustomize-controller",
                "operation": "Apply",
                "time": "2025-04-28T23:53:14Z"
            }
        ],
        "name": "cluster-config",
        "namespace": "flux-system",
        "resourceVersion": "834225",
        "uid": "3d7344d0-8960-4c77-acc2-0abc52da29be"
    }
}

Configuring Substitution

We now have an encrypted ConfigMap that we can use to securely store secrets in our cluster. To utilize these secrets we need to configure the post build substitution on any kustomizations that we want to reference these values in. Making sure to apply substitution to every single kustomization would be tedious, but lucky for us, we have structured our flux repo in such a way that makes our lives easier. We can apply ‘patches’ to higher level kustomizations that will apply changes to other kustomizations applied further down the hierarchy. In our case, we can apply patches to the clusters/truek8s/apps.yaml and clusters/truek8s/infrastructure.yaml kustomizations, and all other resources will automatically inherit our substitution configuration. This is one of the reasons it’s so important to consider how you’ll structure your repository.

Go ahead and add the following patch definition to the apps and infrastructure kustomizations. I’ll provide apps as an example, but the content is exactly the same for both (or any other kustomizations you wish to apply it to in your own cluster).

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/production
  prune: true
  wait: true
  patches:
      - patch: |-
          apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: not-used
          spec:
            decryption:
              provider: sops
              secretRef:
                name: sops-age
            postBuild:
              substituteFrom:
                - kind: ConfigMap
                  name: cluster-config
        target:
          group: kustomize.toolkit.fluxcd.io
          kind: Kustomization
          labelSelector: substitution.flux.home.arpa/disabled notin (true)

It’s that easy! Don’t forget to commit your changes.

git add -A && git commit -m "patched apps and infrastructure"
git push

flux events --watch

Testing Variable Substitution.

We can verify everything is working together by testing out substitution in a helm release. In the last part of the series we deployed podinfo, and that will be the perfect app to test substitution on. Edit the /apps/general/podinfo/podinfo.yaml file, adding a reference to the value we created in our ConfigMap:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: podinfo
spec:
  interval: 10m
  timeout: 5m
  chart:
    spec:
      chart: podinfo
      sourceRef:
        kind: HelmRepository
        name: podinfo
        namespace: flux-system
      interval: 5m
  releaseName: podinfo
  values:
    extraEnvs:
      - name: MULTIPLE_VALUES
        value: ${test}

Commit your changes, and wait for the podinfo helm release to reconcile. Now check the chart values with helm and observe whether or not the test variable was substituted. It should have been replaced with ‘Value’

$ helm get values -n podinfo podinfo
USER-SUPPLIED VALUES:
extraEnvs:
- name: MULTIPLE_VALUES
  value: Value

Sources

TrueK8S Part 04

2025-05-15T12:30:00Z

Certificate management with TrueCharts ClusterIssuer

In this addition to the series, we’ll finally deploy a chart from the TrueCharts project. We’re still working on configuring the essential services that our cluster will depend on. One of those essential services is ingress, but before we can set that up we need to talk about certificates.

Certificates help make HTTPS happen, and help get rid of those annoying ‘your connection is not secure’ errors you see in your browser. If you’re building a kube cluster, you probably knew that. What you may not know is that TrueCharts has a handy tool named ClusterIssuer that will let deployments in your cluster automagically fetch and manage their own certificates.

All you need to take advantage of this tool is a public domain, and an API key to one of the supported DNS Providers. In this guide we’ll deploy the cert-manager and ClusterIssuer helm charts, and configure them to work with our Cloudflare account.

Cloudflare integration

Our new certificates will be issued by Let’s Encrypt, which utilizes ACME (Automatic Certificate Management Environment) to prove ownership of a domain. We’ll give cert-manager an API key for our Cloudflare account, which will allow it to publish records in our domain to satisfy the ACME challenge and retrieve certificates automatically.

Head on over to the Cloudflare API Tokens page and get yourself a new API token.

From the available templates, select Edit zone DNS
Give your new token a descriptive name, and grant the following permissions:
- Zone - DNS - Edit
- Zone - Zone - Read
- Zone Resources - All zones (or all zones for which you wish to issue certificates)

Review the summary and create the token

Make sure to save your API token securely, you won’t get to see it again.

And now we have a secret that we need to give to our cluster. Sounds like a good use of our cluster-config ConfigMap we created earlier.

Decrypt your ConfigMap

$sops -d -i cluster-config.yaml

and add your token, as well as the email address associated with your Cloudflare account, and the domain name you’d like to use.

apiVersion: v1
kind: ConfigMap
metadata:
    name: cluster-config
    namespace: flux-system
data:
    test: Value
    DOMAIN_0: mydomain.tld
    CLOUDFLARE_TOKEN: token-token-token-token-token
    CLOUDFLARE_EMAIL: email-email-email@domain-domain-domain.tld

And zip your clusterconfig file back up before committing the changes to your repo.

$sops -e -i cluster-config.yaml 
$git add -A && git commit -m 'added ACME info to cluster-config'
$git push

Now we have everything we need to start getting some certs.

Deploying cert-manager and ClusterIssuer

cert-manager is the tool that actually does the ACME challenges for us, and cluster-issuer is the tool that takes cert requests from our cluster and hands them to cert-manager to fulfil. We’ll be deploying both.

Add these resources to your cluster repo:

truek8s
├── infrastructure
│ ├── core
│ │ ├── cert-manager
│ │ │ ├── helm-release.yaml
│ │ │ ├── kustomization.yaml
│ │ │ └── namespace.yaml
│ │ ├── cluster-issuer
│ │ │ ├── helm-release.yaml
│ │ │ ├── kustomization.yaml
│ │ │ └── namespace.yaml
│ └── production
│     ├── cert-manager.yaml
│     └── cluster-issuer.yaml
└── repos
    ├── jetstack.yaml
    └── truecharts.yaml

Repos

jetstack.yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: jetstack
  namespace: flux-system
spec:
  interval: 2h
  url: https://charts.jetstack.io/

truecharts.yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
    name: truecharts
    namespace: flux-system
spec:
    type: oci
    interval: 2h
    url: oci://tccr.io/truecharts

infrastructure/core/cert-manager

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
    name: cert-manager
    namespace: cert-manager
spec:
    interval: 5m
    chart:
        spec:
            chart: cert-manager
            sourceRef:
                kind: HelmRepository
                name: jetstack
                namespace: flux-system
            interval: 5m
    install:
        createNamespace: true
        crds: CreateReplace
        remediation:
            retries: 3
    upgrade:
        crds: CreateReplace
        remediation:
            retries: 3
    values:
      crds:
        enabled: true
      dns01RecursiveNameservers: "1.1.1.1:53,1.0.0.1:53"
      dns01RecursiveNameserversOnly: false
      enableCertificateOwnerRef: true

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - helm-release.yaml

/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: cert-manager

infrastructure/core/cluster-issuer

helm-release.yaml

🚧 See how we’re passing the secrets from our ConfigMap here? Also, pay attention to spec.values.clusterCertificates. This will create a default wildcard certificate that we’ll need later!

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: clusterissuer
  namespace: clusterissuer
spec:
  interval: 15m
  chart:
    spec:
      chart: clusterissuer
      sourceRef:
        kind: HelmRepository
        name: truecharts
        namespace: flux-system
      interval: 15m
  timeout: 20m
  maxHistory: 3
  install:
    createNamespace: true
    remediation:
      retries: 3
  upgrade:
    cleanupOnFail: true
    remediation:
      retries: 3
  uninstall:
    keepHistory: false
  values:
    clusterIssuer:
      selfSigned:
        enabled: true
        name: "selfsigned"
      ## Remove these if you do NOT want to use clusterissuer
      ACME:
        - name: letsencrypt-domain-0
          # Used for both logging in to the DNS provider AND ACME registration
          email: "${CLOUDFLARE_EMAIL}"
          server: 'https://acme-v02.api.letsencrypt.org/directory'
          # Options: HTTP01, cloudflare, route53, akamai, digitalocean, rfc2136, acmedns
          type: "cloudflare"
          # for cloudflare
          cfapitoken: "${CLOUDFLARE_TOKEN}"
    ## Remove these if you do NOT want to use clusterissuer
    clusterCertificates:
      # Namespaces in which the certificates must be available
      # Accepts comma-separated regex expressions
      replicationNamespaces: '.*'
      certificates:
        - name: star-domain-0
          enabled: true
          certificateIssuer: letsencrypt-domain-0
          hosts:
            - ${DOMAIN_0}
            - '*.${DOMAIN_0}'

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - helm-release.yaml

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: clusterissuer

/infrastructure/production/

cert-manager.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: cert-manager
  namespace: flux-system
spec:
  interval: 10m
  path: infrastructure/core/cert-manager
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

cluster-issuer.yaml

Note that cluster-issuer depends on cert-manager

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: cluster-issuer
  namespace: flux-system
spec:
  dependsOn:
    - name: cert-manager
  interval: 10m
  path: infrastructure/core/cluster-issuer
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

And finally, don’t forget to modify existing kustomizations to point to the new kustomizations we’ve created.

/repos/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cert-manager.yaml
  - cluster-issuer.yaml

/infrastructure/production.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - truecharts.yaml
  - podinfo.yaml
  - jetstack.yaml

That is a lot of junk to write (or copy paste), but that is my experience of flux. The payoff of deploying things this way is that we now have every part of our deployment declaratively defined, and independently documented. Every namespace and helm-chart is well defined, and any changes are tracked very accurately. Notice we’re also using the spec.dependsOn field to tell flux to wait until cert-manager is deployed to try and deploy ClusterIssuer.

When you’re done complaining about all the files, commit your changes and watch Flux do its work.

❗ Triple check that cluster-config has been re-encrypted before committing!

git add -A && git commit -m 'added certs stuffs'
git push
flux events --watch

Verifying

If everything worked correctly, we should have two new helm releases, and a cluster certificate. Let’s verify that with flux and kubectl.

#Verify that the charts installed successfully.
$ flux get hr -A
NAMESPACE       NAME            REVISION        SUSPENDED       READY   MESSAGE                                                                                             
cert-manager    cert-manager    v1.17.2         False           True    Helm install succeeded for release cert-manager/cert-manager.v1 with chart cert-manager@v1.17.2    
clusterissuer   clusterissuer   9.6.4           False           True    Helm install succeeded for release clusterissuer/clusterissuer.v1 with chart clusterissuer@9.6.4                     
podinfo         podinfo         6.8.0           False           True    Helm upgrade succeeded for release podinfo/podinfo.v4 with chart podinfo@6.8.0                     
#Verify that the star-domain-0 certificate was created.
$ kubectl get certificate -A
NAMESPACE       NAME                               READY   SECRET                             AGE
clusterissuer   certificate-issuer-star-domain-0   False   certificate-issuer-star-domain-0   52s

$ kubectl describe certificate -n clusterissuer certificate-issuer-star-domain-0 
Name:         certificate-issuer-star-domain-0
Namespace:    clusterissuer
Labels:       app=clusterissuer-9.6.4
              app.kubernetes.io/instance=clusterissuer
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=clusterissuer
              app.kubernetes.io/version=latest
              helm-revision=1
              helm.sh/chart=clusterissuer-9.6.4
              helm.toolkit.fluxcd.io/name=clusterissuer
              helm.toolkit.fluxcd.io/namespace=clusterissuer
              release=clusterissuer
Annotations:  meta.helm.sh/release-name: clusterissuer
              meta.helm.sh/release-namespace: clusterissuer
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2025-05-15T02:24:56Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:nextPrivateKeySecretName:
    Manager:      cert-manager-certificates-key-manager
    Operation:    Update
    Subresource:  status
    Time:         2025-05-15T02:24:56Z
    API Version:  cert-manager.io/v1
    [....]
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    71s   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  71s   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "certificate-issuer-star-domain-0-zwn8g"
  Normal  Requested  71s   cert-manager-certificates-request-manager  Created new CertificateRequest resource "certificate-issuer-star-domain-0-1"

Next Steps

In Part 5, we’ll put that cert to use and configure ingress for our cluster.

Sources

TrueK8S Part 05

2025-05-15T12:30:00Z

Configuring Ingress for my Kubernetes Cluster

Do you remember in part 2 of this series where we used the kubectl command to manually port-forward to a service in our cluster? That sucked, right? Let’s not do that. Let’s deploy an ingress controller to handle the task of routing traffic for us.

We’ll be configuring a few things here. First, MetalLB which provides kubernetes with access to REAL IPs on your network. Second, we’ll configure nginx which will be a proxy for application traffic.

MetalLB

MetalLB is a load balancer for bare (or virtualized?) metal kubernetes clusters. It allows kubernetes services to take a real IP from your clusters physical network. The first step here is to decide what IP or range of IPs we’ll configured MetalLB to use. IPs are not necessarily sensitive information depending who you ask, but we’ll put them in our secure configmap for safe storage anyway.

We’ll also need to chose one IP from this range to assign to our ingress controller.

In my case, I’m taking a range of IPs from my cluster’s local network. My nodes are 192.168.XX.101-112. I will assign the range ~.200-.250 for this use case. I will use the first address of that range for nginx.

Go ahead and decrypt cluster-config.yaml

$ sops -d -i cluster-config.yaml

And add your MetalLB range and the IP you want assigned to NGINX like this:

apiVersion: v1
kind: ConfigMap
metadata:
    name: cluster-config
    namespace: flux-system
data:
    test: Value
    METALLB_RANGE: 192.168.XX.200-192.168.XX.250
    NGINX_IP: 192.168.XX.200
    DOMAIN_0: mydomain.tld
    CLOUDFLARE_TOKEN: token-token-token-token-token
    CLOUDFLARE_EMAIL: email-email-email@domain-domain-domain.tld

Now re-encrypt the file and commit the changes to your repo. It should also go without saying that you should always be sure to re-encrypt your config files before committing.

$ sops -e -i cluster-config.yaml 
$ git add -A && git commit -m 'add metallb range to cluster-config'
$ git push

Now we’re going to create the MetalLB deployment manifests. We’re actually going to create two sets of resources applied by separate kustomizations. The first kustomization will deploy MetalLB itself, and the second will deploy a MetalLB configuration. The configuration can only be applied after MetalLB is deployed, so we’ll use separate kustomizations with the dependsOn configuration to ensure they’re applied in the right order.

Go ahead and create the following new directories and resources:

infrastructure/
├── core
│ ├── metallb
│ │   ├── helm-release.yaml
│ │   ├── kustomization.yaml
│ │   └── namespace.yaml
│ └── metallb-config
│     ├── helm-release.yaml
│     ├── kustomization.yaml
│     └── namespace.yaml
└── production
│ ├── kustomization.yaml
│ ├── metallb-config.yaml
│ └── metallb.yaml
└── repos
  └── metallb.yaml

Where the values of each file are:

repos

metallb.yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
    name: metallb
    namespace: flux-system
spec:
    interval: 10m
    url: https://metallb.github.io/metallb
    timeout: 3m

infrastructure/core/metallb

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
    name: metallb
    namespace: metallb
spec:
    interval: 5m
    chart:
        spec:
            chart: metallb
            sourceRef:
                kind: HelmRepository
                name: metallb
                namespace: flux-system
            interval: 5m
    install:
        createNamespace: true
        remediation:
            retries: 3
    upgrade:
        remediation:
            retries: 3
    values:
      speaker:
        ignoreExcludeLB: true

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: metallb
    labels:
        pod-security.kubernetes.io/enforce: privileged

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - helm-release.yaml

infrastructure/core/metallb-config

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: metallb-config
  namespace: metallb-config
spec:
  interval: 15m
  chart:
    spec:
      chart: metallb-config
      sourceRef:
        kind: HelmRepository
        name: truecharts
        namespace: flux-system
      interval: 15m
  timeout: 20m
  maxHistory: 3
  driftDetection:
    mode: warn
  install:
    createNamespace: true
    remediation:
      retries: 3
  upgrade:
    cleanupOnFail: true
    remediation:
      retries: 3
  uninstall:
    keepHistory: false
  values:
    L2Advertisements:
      - name: main
        addressPools:
          - main
    ipAddressPools:
      - name: main
        autoAssign: false
        avoidBuggyIPs: true
        addresses:
          - ${METALLB_RANGE}

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: metallb-config
    labels:
        pod-security.kubernetes.io/enforce: privileged

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - helm-release.yaml

./infrastructure/production/

metallb.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: metallb
  namespace: flux-system
spec:
  interval: 10m
  path: infrastructure/core/metallb
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

metallb-config.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: metallb-config
  namespace: flux-system
spec:
  dependsOn:
    - name: metallb
  interval: 10m
  path: infrastructure/core/metallb-config
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

And last but not least, edit the infrastructure and repos kustomizations to tie it all together.

./infrastructure/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - metallb.yaml
  - metallb-config.yaml

./repos/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - podinfo.yaml
  - jetstack.yaml
  - truecharts.yaml
  - metallb.yaml

Then commit your changes and watch as flux reconciles the new resources.

git add -A && git commit -m 'added metallb'
git push
flux events --watch

Validating Metallb

To verify MetalLB was deployed and correctly configured, you can use your tools to inspect the different parts of the deployment. Start with inspecting the resources deployed by our infrastructure kustomization:

$ flux tree ks infrastructure
Kustomization/flux-system/infrastructure
├── Kustomization/flux-system/metallb
│   ├── Namespace/metallb
│   └── HelmRelease/metallb/metallb
│       ├── ServiceAccount/metallb/metallb-controller
│       ├── ServiceAccount/metallb/metallb-speaker
│       ├── Secret/metallb/metallb-webhook-cert
│       ├── ConfigMap/metallb/metallb-excludel2
│       ├── ConfigMap/metallb/metallb-frr-startup
│       ├── CustomResourceDefinition/bfdprofiles.metallb.io
│       ├── CustomResourceDefinition/bgpadvertisements.metallb.io
│       ├── CustomResourceDefinition/bgppeers.metallb.io
│       ├── CustomResourceDefinition/communities.metallb.io
│       ├── CustomResourceDefinition/ipaddresspools.metallb.io
│       ├── CustomResourceDefinition/l2advertisements.metallb.io
│       ├── CustomResourceDefinition/servicel2statuses.metallb.io
│       ├── ClusterRole/metallb:controller
│       ├── ClusterRole/metallb:speaker
│       ├── ClusterRoleBinding/metallb:controller
│       ├── ClusterRoleBinding/metallb:speaker
│       ├── Role/metallb/metallb-pod-lister
│       ├── Role/metallb/metallb-controller
│       ├── RoleBinding/metallb/metallb-pod-lister
│       ├── RoleBinding/metallb/metallb-controller
│       ├── Service/metallb/metallb-webhook-service
│       ├── DaemonSet/metallb/metallb-speaker
│       ├── Deployment/metallb/metallb-controller
│       └── ValidatingWebhookConfiguration/metallb-webhook-configuration
└── Kustomization/flux-system/metallb-config
    ├── Namespace/metallb-config
    └── HelmRelease/metallb-config/metallb-config
        ├── IPAddressPool/metallb/main
        └── L2Advertisement/metallb/main

We can see that everything has been deployed. The MetalLB kustomization deployed the Helm chart which brought along all the different parts of MetalLB, and the MetalLB-config kustomization created two custom resource definitions that will configure MetalLB within our cluster. Let’s inspect those resources with kubectl:

$ kubectl get l2advertisements.metallb.io -A
NAMESPACE   NAME   IPADDRESSPOOLS   IPADDRESSPOOL SELECTORS   INTERFACES
metallb     main   ["main"]                                   
$ kubectl get ipaddresspools.metallb.io -A
NAMESPACE   NAME   AUTO ASSIGN   AVOID BUGGY IPS   ADDRESSES
metallb     main   true          true              ["192.168.XX.200-192.168.XX.250"]

So we have proof that MetalLB has been deployed and is configured. These steps of inspecting resources will be helpful to you whenever you are troubleshooting with flux and kubernetes.

Deploy NGINX

With MetalLB configured, we can move on to deploying NGINX.

Create the following directories and manifests:

truek8s
├── infrastructure
│   ├── core
│   │   └── ingress-nginx
│   │      ├── helm-release.yaml
│   │      ├── kustomization.yaml
│   │      └── namespace.yaml
│   └── production
│       └── ingress-nginx.yaml
└── repos
    └── ingress-nginx.yaml

infrastructure/core/ingress-nginx

The configuration of NGINX is specified in the spec.values section of this manifest. Pay particular attention to the loadBalancerIPs and ingressClassResource.

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
spec:
  interval: 15m
  chart:
    spec:
      chart: 'ingress-nginx'
      version: 4.12.2
      sourceRef:
        kind: HelmRepository
        name: 'ingress-nginx'
        namespace: flux-system
      interval: 15m
  timeout: 20m
  maxHistory: 3
  install:
    createNamespace: true
    remediation:
      retries: 3
  upgrade:
    cleanupOnFail: true
    remediation:
      retries: 3
  uninstall:
    keepHistory: false
  values:
    controller:
      replicaCount: 2
      service:
        externalTrafficPolicy: Local
        annotations:
          metallb.io/ip-allocated-from-pool: main
          metallb.io/loadBalancerIPs: ${NGINX_IP}
          metallb.universe.tf/ip-allocated-from-pool: main
      ingressClassByName: true
      ingressClassResource:
        name: internal
        default: true
        controllerValue: k8s.io/internal
      config:
        allow-snippet-annotations: true
        annotations-risk-level: Critical
        client-body-buffer-size: 100M
        client-body-timeout: 120
        client-header-timeout: 120
        enable-brotli: "true"
        enable-ocsp: "true"
        enable-real-ip: "true"
        force-ssl-redirect: "true"
        hide-headers: Server,X-Powered-By
        hsts-max-age: "31449600"
        keep-alive-requests: 10000
        keep-alive: 120
        proxy-body-size: 0
        proxy-buffer-size: 16k
        ssl-protocols: TLSv1.3 TLSv1.2
        use-forwarded-headers: "true"
      metrics:
        enabled: true
      extraArgs:
        default-ssl-certificate: "clusterissuer/certificate-issuer-star-domain-0"
        publish-status-address: ${NGINX_IP}
      terminationGracePeriodSeconds: 120
      publishService:
        enabled: false
      resources:
        requests:
          cpu: 100m
        limits:
          memory: 500Mi
    defaultBackend:
      enabled: false

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - helm-release.yaml

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: ingress-nginx

infrastructure/production

ingress-nginx.yaml

NGINX depends on MetalLB for it’s IP address on the local network, and also needs a certificate, so we’ll configure it to depend on MetalLB-config and cluster-issuer.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: ingress-nginx
  namespace: flux-system
spec:
  dependsOn:
    - name: metallb-config
    - name: cluster-issuer
  interval: 10m
  path: infrastructure/core/ingress-nginx
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

Once you’ve created all of those, reference the new ingress-nginx kustomization in the infrastructure\production and repos ks:

infrastructure/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cert-manager.yaml
  - cluster-issuer.yaml
  - metallb.yaml
  - metallb-config.yaml
  - ingress-nginx.yaml

repos/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - podinfo.yaml
  - jetstack.yaml
  - truecharts.yaml
  - metallb.yaml
  - ingress-nginx.yaml

And push your changes to your repo:

$ git add -A && git commit -m 'add ingress nginx'
$ git push

You can monitor the reconiciliation by waiting for Flux to mark the Helm release as ready.

flux get hr ingress-nginx -n ingress-nginx --watch
NAME            REVISION        SUSPENDED       READY   MESSAGE                                                                                           
ingress-nginx   4.12.2          False           True    Helm install succeeded for release ingress-nginx/ingress-nginx.v1 with chart ingress-nginx@4.12.2

Verify NGINX

Is it “EngineX” or “N-Ginx”? anyway… Once it’s installed and configured properly, it should create an ingress class within your cluster. Other apps will use this ingress class to automatically configure ingress with NGINX. You can use kubectl to inspect:

$ kubectl get ingressclass
NAME         CONTROLLER                                       PARAMETERS   AGE
internal     k8s.io/internal                                  <none>       2d8h

$ kubectl describe ingressclass internal 
Name:         internal
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.12.2
              helm.sh/chart=ingress-nginx-4.12.2
              helm.toolkit.fluxcd.io/name=ingress-nginx
              helm.toolkit.fluxcd.io/namespace=ingress-nginx
Annotations:  ingressclass.kubernetes.io/is-default-class: true
              meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx
Controller:   k8s.io/internal
Events:       <none>

Testing ingress

Having an ingress controller won’t be very helpful unless you also know how to use that ingress controller for a given application. Let me provide an example by creating an ingress resource for our example deployment of podinfo.

apps/general/podinfo

ingress.yaml

Create an ingress.yaml file here, and populate it like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: podinfo-nginx
  namespace: podinfo
spec:
  ingressClassName: internal
  rules:
    - host: podinfo.${DOMAIN_0}
      http:
        paths:
          - backend:
              service:
                name: podinfo
                port:
                  number: 9898
            path: /
            pathType: Prefix

kustomization.yaml

and add a reference to this new file to the kustomization.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - helm-release.yaml
  - ingress.yaml

And commit those changes

$ git add -A && git commit -m 'add podinfo ingress'
$ git push

This will create a new ingress resource from the ‘internal’ ingress class we used earlier. inspect the results with flux and kubectl:

$ flux tree ks podinfo
Kustomization/flux-system/podinfo
├── Namespace/podinfo
├── HelmRelease/podinfo/podinfo
│   ├── Service/podinfo/podinfo
│   └── Deployment/podinfo/podinfo
└── Ingress/podinfo/podinfo-nginx

$ kubectl get ingress -n podinfo podinfo-nginx 
NAME            CLASS      HOSTS             ADDRESS                           PORTS   AGE
podinfo-nginx   internal   podinfo.domain.tld   192.168.XX.111,192.168.XX.112   80      116s
$ kubectl describe  ingress -n podinfo podinfo-nginx 
Name:             podinfo-nginx
Namespace:        podinfo
Address:          192.168.XX.111,192.168.XX.112
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host             Path  Backends
  ----             ----  --------
  podinfo.domain.tld  
                   /   podinfo:9898 (172.XX.2.8:9898)
Annotations:       <none>
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    108s (x2 over 2m43s)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    108s (x2 over 2m43s)  nginx-ingress-controller  Scheduled for sync

You’ll need to configure your local DNS to resolve these names and point at your MetalLB IP. What I do is create a host record pointing at NGINX and then create CNAMES for every service I deploy.

truek8s@debian02:~/Documents/truek8s/config$ nslookup nginx.domain.tld
Server:         192.168.XX.2
Address:        192.168.XX.2#53

Non-authoritative answer:
Name:   nginx.domain.tld
Address: 192.168.XX.200

truek8s@debian02:~/Documents/truek8s/config$ nslookup podinfo.domain.tld
Server:         192.168.XX.2
Address:        192.168.XX.2#53

Non-authoritative answer:
podinfo.domain.tld canonical name = nginx.domain.tld.
Name:   nginx.domain.tld
Address: 192.168.XX.200

When all that is configured, you should be able to access the podinfo web service at the IP\DNS name you configured. Thanks to the work we did in part 4, nginx was able to automatically retrieve a certificate for the podifo service and serve over HTTPS with a valid cert.

sources

TrueK8S Part 06

2025-05-15T12:30:00Z

Cloudflare Ingress

When it comes to hosting services, having them available to you inside your house is nice, but the true value comes from having them externally available. I don’t have the risk appetite to expose services directly from my home network, so I let Cloudflare do it for me. This set up is pretty common among homelabbers, and especially among the TrueCharts community from what I’ve seen. I especially like CloudFlare’s Zero Trust services, which allow me to put an extra layer of security in front of my applications by requiring strong authentication before you can access anything.

One way that I’ve noticed my cluster differs from some of the examples I’ve seen in the TrueCharts community is how I chose to configure my external ingress. The most current recommendation I see from the TrueCharts project right now is to run two instances of NGINX ingrex, one internal, and one external. Then, point the Cloudflare tunnel daemon at your external ingress, either from a deployment in your cluster, or from elsewhere on your network.

Personally, I found another project that I find more convenient: STRRL cloudflare-tunnel-ingress-controller. This ingress controller, available as a helm chart, runs a managed Cloudflare tunnel in your environment and provides all the necessary Kubernetes ingress resources.

If you’re following along and don’t have any interest in configuring Cloudflare ingress, skip ahead to part 7. You won’t miss anything.

Deploying cf-tunnel-ingress

Start by gathering details and generating a unique API token from your Cloudflare account.

Create custom token
Add:

Zone:Zone:Read
Zone:DNS:Edit
Account:Cloudflare Tunnel:Edit

Then finally, grab your account ID

Then add that info to your cluster-config:

apiVersion: v1
kind: ConfigMap
metadata:
    name: cluster-config
    namespace: flux-system
data:
    CF_INGRESS_ACCOUNT_ID: <your account ID>
    CF_INGRESS_API_TOKEN: <your token>
    CF_INGRESS_TUNNEL: <a descriptive name for your tunnel>

Then deploy the helm chart following the structure of our repo:

truek8s-v2
├── infrastructure
│   ├── core
│   │   └── cf-tunnel-ingress
│   │      ├── helm-release.yaml
│   │      ├── kustomization.yaml
│   │      └── namespace.yaml
│   └── production
│       └── cf-tunnel-ingress.yaml
└── repos
    └── strrl.yaml

infrastructure/core/cf-tunnel-ingress

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cloudflare-tunnel-ingress
  namespace: cloudflare-ingress
spec:
  chart:
    spec:
      chart: cloudflare-tunnel-ingress-controller
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: strrl
        namespace: flux-system
  values:
    cloudflare:
      accountId: ${CF_INGRESS_ACCOUNT_ID}
      apiToken: ${CF_INGRESS_API_TOKEN}
      tunnelName: ${CF_INGRESS_TUNNEL}
    ingressClass:
      name: cloudflare
      controllerValue: strrl.dev/cloudflare-tunnel-ingress-controller
      isDefaultClass: false
  interval: 1m0s

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: cloudflare-ingress

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - helm-release.yaml

infrastructure/production

cf-tunnel-ingress.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: cf-tunnel-ingress
  namespace: flux-system
spec:
  interval: 10m
  path: infrastructure/core/cf-tunnel-ingress
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

repos

strrl.yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: strrl
  namespace: flux-system
spec:
  interval: 1m0s
  url: https://helm.strrl.dev

And don’t forget to add the new resources to existing kustomizations:

/infrastructure/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cert-manager.yaml
  - cluster-issuer.yaml
  - metallb.yaml
  - metallb-config.yaml
  - ingress-nginx.yaml
  - cf-tunnel-ingress.yaml

/repos/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - podinfo.yaml
  - jetstack.yaml
  - truecharts.yaml
  - metallb.yaml
  - ingress-nginx.yaml
  - strrl.yaml

Test & Verify

Once the CF-Tunnel helm chart is installed, it will create a new ingress class just like NGINX did. Use kubectl to check for it:
Make sure the ingressclass exists:

$ kubectl get ingressclasses.networking.k8s.io 
NAME         CONTROLLER                                       PARAMETERS   AGE
cloudflare   strrl.dev/cloudflare-tunnel-ingress-controller   <none>       8m12s
internal     k8s.io/internal                                  <none>       5d2h

To test it out, you can append an additional ingress definition on apps/general/podinfo/ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: podinfo-nginx
  namespace: podinfo
spec:
  ingressClassName: internal
  rules:
  - host: podinfo.${DOMAIN_0}
    http:
      paths:
      - backend:
          service:
            name: podinfo
            port:
              number: 9898
        path: /
        pathType: Prefix
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: podinfo-cloudflare
  namespace: podinfo
spec:
  ingressClassName: cloudflare
  rules:
  - host: podinfo.${DOMAIN_0}
    http:
      paths:
      - backend:
          service:
            name: podinfo
            port:
              number: 9898
        path: /
        pathType: Prefix

Commit that and inspect the results:

$ git add -A && git commit -m 'add podinfo cf ingress'
$ git push
$ flux events --watch

inspect any ingress rules create for podinfo:

$ kubectl get ingress -n podinfo 
NAME                 CLASS        HOSTS             ADDRESS                                                 PORTS   AGE
podinfo-cloudflare   cloudflare   podinfo.domain.tld   9a6bc2e4-8c73-40a9-b5a3-5ddd0d937f29.cfargotunnel.com   80      29s
podinfo-nginx        internal     podinfo.domain.tld   192.168.XX.200                                         80      5d1h

If everything looks good, test it out by accessing it remotely. When you’re all done, this would be a good time to delete your test deployment - you don’t need to put podinfo on the internet with your name attached to it. Just delete the following files, and remove the podinfo reference from /apps/production/kustomization.yaml

truek8s
├── apps
   ├── general
   │   └── podinfo *
   │       ├── helm-release.yaml *
   │       ├── ingress.yaml *
   │       ├── kustomization.yaml * 
   │       └── namespace.yaml *
   └── production
       └── podinfo.yaml *

Zero Trust with Cloudflare.

Exposing services through Cloudflare is only a little bit better than exposing them through your own firewall. To really make things interesting, you should configure Cloudflare Zero Trust rules. That’s not within the scope of what I want to do with this guide, but I would like to cover it in more depth on this blog later.

In my case, I’m hosting Authentik, and use it as an OIDC provider for Cloudflare. I have some custom rules set up to pass authorization access to Cloudflare. Whenever I (or anyone) access one of the apps in my cluster, they first have to provide strong authentication through Cloudflare, which significantly reduces the risk of unauthorized access.

Sources

TrueK8S Part 07

2025-05-15T12:30:00Z

Deploying LongHorn to our Kubernetes cluster

I learned a hard lesson about making good choices regarding storage for your kubernetes cluster. In early attempts at building my cluster I tried using an ‘nfs-subdir’ provider that provided pod volume storage backed by a local NFS server. This seemed easy and made good sense to me at the time. In practice, the I/O was so bad that many deployments failed due to enormous storage latency. Not only did it perform poorly, that storage provider also didn’t provide the snapshot mechanism that we’ll need for VolSync (up next). So take it from me, you need a good storage provider that provides features, flexibility, and good performance.

I chose longhorn as the storage provider for my cluster, mainly because there were plenty of examples around longhorn configuration in the TrueCharts community for me to follow. Longhorn allows us to utilize direct attached storage on our worker nodes for high performance, and also provides the volume snapshot features that we’ll need in the next post. You may remember back in part 1 of this guide, I referenced storage for our cluster members and some necessary extensions for TalosOS. If you skipped around, go back to part 1 and make sure that you have the util-linux-tools and iscsi-tools extensions configured.

Deploying LongHorn

You should know the drill by now. Here are the deployment resources. Once again we’ll create a kustomization to deploy longhorn, and then a separate one to configure it. I chose to do it this way because I want to explicitly wait for the longhorn UI to be online before trying to assign ingress to it, which can cause issues in re-deployment scenarios.

truek8s
├── infrastructure
│ ├── core
│ │ ├── longhorn
│ │ │ ├── helm-release.yaml
│ │ │ ├── kustomization.yaml
│ │ │ ├── namespace.yaml
│ │ ├── longhorn-config
│ │   ├── ingress.yaml
│ │   └── kustomization.yaml
│ └── production
│     └── longhorn.yaml
└── repos
    ├── longhorn.yaml
    └── longhorn-config.yaml

./repos

longhorn.yaml

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: longhorn
  namespace: flux-system
spec:
  interval: 2h
  url: https://charts.longhorn.io

kustomization.yaml

Add this new manifest to the repos kustomization.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - longhorn.yaml

./infrastructure/core/longhorn

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: longhorn-system
    labels:
        pod-security.kubernetes.io/enforce: privileged

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
    name: longhorn
    namespace: longhorn-system
spec:
    interval: 5m
    releaseName: longhorn
    chart:
        spec:
            chart: longhorn
            sourceRef:
                kind: HelmRepository
                name: longhorn
                namespace: flux-system
    install:
        createNamespace: true
        crds: CreateReplace
        remediation:
            retries: 3
    upgrade:
        crds: CreateReplace
        remediation:
            retries: 3
    values:
        defaultSettings:
            # Increase to 3 for a multi-node cluster
            defaultReplicaCount: 1
            # Overprovisioning might be needed when using volsync
            storageOverProvisioningPercentage: 100000
            storageReservedPercentageForDefaultDisk: 10
            # v2DataEngine: true
        csi:
            attacherReplicaCount: 1
            provisionerReplicaCount: 1
            resizerReplicaCount: 1
            snapshotterReplicaCount: 1
        persistence:
            # Set to false to pick another CSI as default
            defaultClass: true
            # Increase to 3 for a multi-node cluster
            defaultClassReplicaCount: 1
        longhornUI:
            replicas: 1

./infrastructure/core/longhorn-config

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: longhorn-nginx
  namespace: longhorn-system
spec:
  ingressClassName: internal
  rules:
  - host: longhorn.${DOMAIN_0}
    http:
      paths:
      - backend:
          service:
            name: longhorn-frontend
            port:
              number: 80
        path: /
        pathType: Prefix

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ingress.yaml

./infrastructure/production

The order of operations here is to first install longhorn, then use separate kustomization to configure the ingress once longhorn and nginx are both configured.

/infrastructure/production/longhorn.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: longhorn
  namespace: flux-system
spec:
  interval: 10m
  path: infrastructure/core/longhorn
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

/infrastructure/production/longhorn-config.yaml

Note the dependsOn configuration of this kustomization.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: longhorn-config
  namespace: flux-system
spec:
  dependsOn:
    - name: ingress-nginx
    - name: longhorn
  interval: 10m
  path: infrastructure/core/longhorn-config
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

/infrastructure/production/kustomization.yaml

And add these new manifests to the production kustomization

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - longhorn.yaml
  - longhorn-config.yaml

Finally, commit your changes and watch flux as it reconciles the changes.

git add -A && git commit -m "add longhorn"
git push

flux events --watch

Verify longhorn

Check everything over using flux, helm, and kubectl like we did in previous steps. You should see quite a few resources created by the longhorn helm-release. You can inspect those with kubectl to understand what all it’s doing. You should also have new storage classes created by longhorn.

$ flux get hr longhorn -n longhorn-system
NAME            REVISION        SUSPENDED       READY   MESSAGE                                                                                  
longhorn        1.9.0           False           True    Helm install succeeded for release longhorn-system/longhorn.v1 with chart longhorn@1.9.0

$ flux tree ks longhorn
Kustomization/flux-system/longhorn
├── Namespace/longhorn-system
└── HelmRelease/longhorn-system/longhorn
    ├── PriorityClass/longhorn-critical
    ├── ServiceAccount/longhorn-system/longhorn-service-account
    ├── ServiceAccount/longhorn-system/longhorn-ui-service-account
    ├── ServiceAccount/longhorn-system/longhorn-support-bundle
    ├── ConfigMap/longhorn-system/longhorn-default-resource
    ├── ConfigMap/longhorn-system/longhorn-default-setting
    ├── ConfigMap/longhorn-system/longhorn-storageclass
    ├── CustomResourceDefinition/backingimagedatasources.longhorn.io
    ├── CustomResourceDefinition/backingimagemanagers.longhorn.io
    ├── CustomResourceDefinition/backingimages.longhorn.io
    ├── CustomResourceDefinition/backupbackingimages.longhorn.io
    ├── CustomResourceDefinition/backups.longhorn.io
    ├── CustomResourceDefinition/backuptargets.longhorn.io
    ├── CustomResourceDefinition/backupvolumes.longhorn.io
    ├── CustomResourceDefinition/engineimages.longhorn.io
    ├── CustomResourceDefinition/engines.longhorn.io
    ├── CustomResourceDefinition/instancemanagers.longhorn.io
    ├── CustomResourceDefinition/nodes.longhorn.io
    ├── CustomResourceDefinition/orphans.longhorn.io
    ├── CustomResourceDefinition/recurringjobs.longhorn.io
    ├── CustomResourceDefinition/replicas.longhorn.io
    ├── CustomResourceDefinition/settings.longhorn.io
    ├── CustomResourceDefinition/sharemanagers.longhorn.io
    ├── CustomResourceDefinition/snapshots.longhorn.io
    ├── CustomResourceDefinition/supportbundles.longhorn.io
    ├── CustomResourceDefinition/systembackups.longhorn.io
    ├── CustomResourceDefinition/systemrestores.longhorn.io
    ├── CustomResourceDefinition/volumeattachments.longhorn.io
    ├── CustomResourceDefinition/volumes.longhorn.io
    ├── ClusterRole/longhorn-role
    ├── ClusterRoleBinding/longhorn-bind
    ├── ClusterRoleBinding/longhorn-support-bundle
    ├── Service/longhorn-system/longhorn-backend
    ├── Service/longhorn-system/longhorn-frontend
    ├── Service/longhorn-system/longhorn-conversion-webhook
    ├── Service/longhorn-system/longhorn-admission-webhook
    ├── Service/longhorn-system/longhorn-recovery-backend
    ├── DaemonSet/longhorn-system/longhorn-manager
    ├── Deployment/longhorn-system/longhorn-driver-deployer
    └── Deployment/longhorn-system/longhorn-ui

$ flux tree ks longhorn-config
Kustomization/flux-system/longhorn-config
└── Ingress/longhorn-system/longhorn-nginx

$ kubectl get storageclasses.storage.k8s.io 
NAME                 PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
longhorn (default)   driver.longhorn.io   Delete          Immediate           true                   2d9h
longhorn-static      driver.longhorn.io   Delete          Immediate           true                   2d9h

If everything goes well, you should be able to reach the longhorn webui at longhorn.yourdomain.tld, and it should even be served using it’s own LE certificate.

Sources

TrueK8S Part 08

2025-11-29T12:30:00Z

Configuring VolSync and CNPG for Cluster Backup

One of the things that always made me a little wary of Kubernetes for the homelab was the overall resiliency. In a more traditional scenario running apps on a VM, you can just snapshot/backup the whole virtual machine and restore everything all at once. I would like to have the peace of mind to know that I can restore my apps and data should something ever go wrong. Moreover, I don’t want the management of those backups to constitute a full-time job (like writing this series has).

TrueCharts has a solution for that. All of the apps in the TrueCharts project ship with integration for VolSync and cloudnative PostgreSQL, which are mature solutions to providing resiliency in a Kubernetes environment. The TrueCharts common library provides a consistent mechanism I can use to backup and restore my deployments, which is one of the reasons I’m such a fan of the project.

In this installment, we’ll configure and test our backups before we move on to deploying any workloads or data into our cluster.

Configure Secrets

S3 secrets

Before we can get started, Volsync and CNPG need somewhere to store the backup data. I chose to use an S3 bucket hosted by Backblaze, but any S3 service should work. We’ll start by generating a credential to access our bucket.

In Backblaze, you can generate an application key by going to:

Backblaze > side panel > Application Keys
Add a new application key

This will give you a Key ID and an application key (Secret). In addition to your key, you will create your own encryption key to encrypt data before it ever goes to the S3 bucket. You can put all three of these in your configmap:

apiVersion: v1
kind: ConfigMap
metadata:
    name: cluster-config
    namespace: flux-system
data:
    S3_URL: #The URL for your S3 service, e.g. https://s3.us-east-005.backblazeb2.com
    S3_BUCKET: # This is the base bucket name that will be used by all new buckets
    S3_ACCESS_KEY: #The Key ID
    S3_SECRET_KEY: # The application key (Secret)
    S3_ENCRYPTION_KEY: #User defined.

If I were you, I would make that encryption key 64+ random characters. Consider storing this key safely. If you lose the key you set here, you lose the ability to decrypt the data in your S3 bucket, and will be paying your s3 provider to host waste bits.

Make sure to re-encrypt your config map, then push the updates:

$ sops -e -i cluster-config.yaml
$ git add -A && git commit -m "Add secrets"
$ git push

Installing VolSync and CNPG

For this part, we need to configure 5 new kustomizations and their associated resources. This is a lot of content to copy, so I’ll provide the contents of each file in the usual fashion, but I’d also encourage you to check out the resources directly in my git repo. GitHub - TrueK8s

The kustomizations we’ll be creating are:

cloudnative-pg
kubernetes-reflector
snapshot-controller
volsync
volsync-config

And these are the files we’ll need to create in our repo:

truek8s
├── infrastructure
│   ├── core
│   │   ├── cloudnative-pg
│   │   │   ├── helm-release.yaml
│   │   │   ├── kustomization.yaml
│   │   │   └── namespace.yaml
│   │   ├── kubernetes-reflector
│   │   │   ├── helm-release.yaml
│   │   │   └── kustomization.yaml
│   │   ├── snapshot-controller
│   │   │   ├── helm-release.yaml
│   │   │   ├── kustomization.yaml
│   │   │   └── namespace.yaml
│   │   ├── volsync
│   │   │   ├── helm-release.yaml
│   │   │   ├── kustomization.yaml
│   │   │   └── namespace.yaml
│   │   └── volsync-config
│   │       ├── kustomization.yaml
│   │       └── volumesnapshotclass.yaml
│   └── production
│       ├── cloudnative-pg.yaml
│       ├── kubernetes-reflector.yaml
│       ├── snapshot-controller.yaml
│       ├── volsync-config.yaml
│       └── volsync.yaml
└── repos
    └── cloudnative-pg.yaml

/infrastructure/core/cloudnative-pg

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
    name: cloudnative-pg
    namespace: cloudnative-pg
spec:
    interval: 5m
    chart:
        spec:
            chart: cloudnative-pg
            sourceRef:
                kind: HelmRepository
                name: cloudnative-pg
                namespace: flux-system
            interval: 5m
    install:
        createNamespace: true
        crds: CreateReplace
        remediation:
            retries: 3
    upgrade:
        crds: CreateReplace
        remediation:
            retries: 3
    values:
      crds:
        create: true

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - helm-release.yaml
  - namespace.yaml

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: cloudnative-pg

/infrastructure/core/kubernetes-reflector

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
    name: kubernetes-reflector
    namespace: snapshot-controller
spec:
    interval: 15m
    chart:
        spec:
            chart: kubernetes-reflector
            sourceRef:
                kind: HelmRepository
                name: truecharts
                namespace: flux-system
            interval: 15m
    timeout: 20m
    maxHistory: 3
    driftDetection:
      mode: warn
    install:
        createNamespace: true
        remediation:
            retries: 3
    upgrade:
        cleanupOnFail: true
        remediation:
            retries: 3
    uninstall:
        keepHistory: false
    values:
      podOptions:
        hostUsers: true

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - helm-release.yaml

/infrastructure/core/snapshot-controller

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
    name: snapshot-controller
    namespace: snapshot-controller
spec:
    interval: 5m
    chart:
        spec:
            chart: snapshot-controller
            sourceRef:
                kind: HelmRepository
                name: truecharts
                namespace: flux-system
            interval: 5m
    install:
        createNamespace: true
        remediation:
            retries: 3
    upgrade:
      cleanupOnFail: true
      crds: CreateReplace
      remediation:
        strategy: rollback
        retries: 3
    values:
      metrics:
        main:
          enabled: false
      podOptions:
        hostUsers: true

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - helm-release.yaml
  - namespace.yaml

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: snapshot-controller
    labels:
        pod-security.kubernetes.io/enforce: privileged

/infrastructure/core/volsync

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
    name: volsync
    namespace: volsync
spec:
    interval: 5m
    chart:
        spec:
            chart: volsync
            sourceRef:
                kind: HelmRepository
                name: truecharts
                namespace: flux-system
            interval: 5m
    install:
        createNamespace: true
        remediation:
            retries: 3
    upgrade:
        remediation:
            retries: 3
    values:
      metrics:
        main:
          enabled: false
      podOptions:
        hostUsers: true

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - helm-release.yaml
  - namespace.yaml

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: volsync
    labels:
        pod-security.kubernetes.io/enforce: privileged

/infrastructure/core/volsync-config

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - volumesnapshotclass.yaml

volumesnapshotclass.yaml

kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
metadata:
  name: longhorn-snapshot-vsc
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: 'true'
driver: driver.longhorn.io
deletionPolicy: Delete
parameters:
  type: snap

/infrastructure/production

cloudnative-pg.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: cloudnative-pg
  namespace: flux-system
spec:
  dependsOn:
    - name: longhorn
  interval: 10m
  path: infrastructure/core/cloudnative-pg
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

kubernetes-reflector.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: kubernetes-reflector
  namespace: flux-system
spec:
  dependsOn:
    - name: snapshot-controller
  interval: 10m
  path: infrastructure/core/kubernetes-reflector
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

snapshot-controller.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: snapshot-controller
  namespace: flux-system
spec:
  dependsOn:
    - name: longhorn
  interval: 10m
  path: infrastructure/core/snapshot-controller
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

volsync-config.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: volsync-config
  namespace: flux-system
spec:
  dependsOn:
    - name: volsync
  interval: 10m
  path: infrastructure/core/volsync-config
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

volsync.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: volsync
  namespace: flux-system
spec:
  dependsOn:
    - name: snapshot-controller
  interval: 10m
  path: infrastructure/core/volsync
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

/repos

cloudnative-pg.yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: cloudnative-pg
  namespace: flux-system
spec:
  interval: 2h
  url: https://cloudnative-pg.github.io/charts

Don’t forget to add a reference to each in the production\kustomization file.

/infrastructure/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cert-manager.yaml
  - cluster-issuer.yaml
  - metallb.yaml
  - metallb-config.yaml
  - ingress-nginx.yaml
  - cf-tunnel-ingress.yaml
  - longhorn.yaml
  - longhorn-config.yaml
  - cloudnative-pg.yaml
  - kubernetes-reflector.yaml
  - snapshot-controller.yaml
  - volsync-config.yaml
  - volsync.yaml

/repos/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - podinfo.yaml
  - jetstack.yaml
  - truecharts.yaml
  - metallb.yaml
  - ingress-nginx.yaml
  - strrl.yaml
  - longhorn.yaml
  - cloudnative-pg.yaml

After you’ve created those manifests, go ahead and push them.

$ git add -A && git commit -m "Add VolSync and CNPG"
$ git push

Verify deployment

Deploying VolSync and CNPG along with all their requirement components will create quite a few new resources in your cluster. Here are some commands you can use to verify successful deployment.

Start by checking to make sure the Kustomizations were applied correctly:

$ flux get ks 
NAME                    REVISION                SUSPENDED       READY   MESSAGE                              
apps                    main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
cert-manager            main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
cf-tunnel-ingress       main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
cloudnative-pg          main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
cluster-issuer          main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
config                  main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
flux-system             main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
infrastructure          main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
ingress-nginx           main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
kubernetes-reflector    main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
longhorn                main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
longhorn-config         main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
metallb                 main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
metallb-config          main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
repos                   main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
snapshot-controller     main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
volsync                 main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51
volsync-config          main@sha1:5a540e51      False           True    Applied revision: main@sha1:5a540e51

Next make sure that the custom API resources for CNPG and backube are available in your cluster:

$ kubectl api-resources  | grep cnpg
backups                                                   postgresql.cnpg.io/v1                    true         Backup
clusterimagecatalogs                                      postgresql.cnpg.io/v1                    false        ClusterImageCatalog
clusters                                                  postgresql.cnpg.io/v1                    true         Cluster
databases                                                 postgresql.cnpg.io/v1                    true         Database
failoverquorums                                           postgresql.cnpg.io/v1                    true         FailoverQuorum
imagecatalogs                                             postgresql.cnpg.io/v1                    true         ImageCatalog
poolers                                                   postgresql.cnpg.io/v1                    true         Pooler
publications                                              postgresql.cnpg.io/v1                    true         Publication
scheduledbackups                                          postgresql.cnpg.io/v1                    true         ScheduledBackup
subscriptions                                             postgresql.cnpg.io/v1                    true         Subscription
$ kubectl api-resources  | grep volsync
replicationdestinations                                   volsync.backube/v1alpha1                 true         ReplicationDestination
replicationsources                                        volsync.backube/v1alpha1                 true         ReplicationSource

and also make sure that the custom snapshot class defined in volysnc-config has been created:

$ kubectl get volumesnapshotclass
NAME                    DRIVER               DELETIONPOLICY   AGE
longhorn-snapshot-vsc   driver.longhorn.io   Delete           11m

Deploy Vaultwarden for testing

Now that we’ve configured our backup mechanism, we should test it with a real deployment. I’ll walk you through deploying, destroying, and restoring a Vaultwarden instance. Vaultwarden is a password manager well maintained by the TrueCharts community. It’s ideal for our use case, because it will utilize almost every facet of our cluster we’ve configured so far. NGINX\Cloudflare ingress, certificates issued by cluster-issuer, persistent volumes provided by longhorn and backed up with VolSync, and a cloudnative postgresql cluster for backups.

VW Secrets

To get started, we need to prepare a secret admin token for our Vaultwarden instance. Go ahead and define one in your cluster-config configmap.

apiVersion: v1
kind: ConfigMap
metadata:
    name: cluster-config
    namespace: flux-system
data:
    VW_ADMIN_TOKEN: ###

❗ remember to re-encrypt your config map before proceeding

Vaultwarden Chart

Go ahead and create a vaultwarden directory under apps/general and create the following manifests:

truek8s
└── apps
   ├── general
   │   └── vaultwarden-test
   │       ├── helm-release.yaml
   │       ├── ks.yaml
   │       ├── kustomization.yaml
   │       └── namespace.yaml
   └── production
       └── vaultwarden-test.yaml

./apps/general/vaultwarden

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - helm-release.yaml
  - namespace.yaml

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: vaultwarden-test

helm-release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
    name: vaultwarden-test
    namespace: vaultwarden-test
spec:
    interval: 60m
    chart:
        spec:
            chart: vaultwarden
            sourceRef:
                kind: HelmRepository
                name: truecharts
                namespace: flux-system
            interval: 60m
    install:
        createNamespace: true
        crds: CreateReplace
        timeout: 30m
        remediation:
            retries: 2
    upgrade:
        crds: CreateReplace
        timeout: 30m
        remediation:
            retries: 2
    values:
        global:
            stopAll: false
        credentials:
          backblaze:
            type: s3
            url: "${S3_URL}"
            bucket: "${S3_BUCKET}-vaultwarden-test"
            accessKey: "${S3_ACCESS_KEY}"
            secretKey: "${S3_SECRET_KEY}"
            encrKey: "${S3_ENCRYPTION_KEY}"
        ingress:
          main:
            enabled: true
            ingressClassName: internal
            integrations:
              traefik:
                enabled: false
              certManager:
                enabled: true
                certificateIssuer: letsencrypt-domain-0
            hosts:
              - host: vaultwarden-test.${DOMAIN_0}
                paths:
                  - path: /
                    pathType: Prefix 
          cloudflare:
            enabled: true
            ingressClassName: cloudflare
            integrations:
              traefik:
                enabled: false
            hosts:
            - host: vaultwarden-test.${DOMAIN_0}
              paths:
              - path: /
                pathType: Prefix
        persistence:
            data:
              volsync:
              - name: data
                type: restic
                cleanupTempPVC: false
                cleanupCachePVC: false
                credentials: backblaze
                dest:
                  enabled: true
                src:
                  enabled: true
                  trigger:
                    schedule: 40 1 * * *
        cnpg:
            main:
              cluster:
                singleNode: true
#              mode: recovery
              backups:
                enabled: true
                credentials: backblaze
                scheduledBackups:
                  - name: daily-backup
                    schedule: "0 5 0 * * *"
                    backupOwnerReference: self
                    immediate: true
                    suspend: false
              recovery:
                method: object_store
                credentials: backblaze
        vaultwarden:
            admin:
                enabled: true
                token: ${VW_ADMIN_TOKEN}

The important config items in this manifest are:

values.credentials
- This section provides the info and credentials to reach our S3 bucket. In this case, that is all stored in the cluster-config configmap, and will be the same for all charts except for the bucket name, which we’ll append with a static value for each chart.
values.persistence
- This section performs two functions. It configures the persistent volumes for our deployment, and also configures VolSync for that volume. Vaultwarden needs one persistent volume named ‘data’ which you see defined here. Other persistent volumes can be defined using the same syntax under the persistence section.
- Take note of the dest and src configuration. This tells VolSync whether this volume should be a source of a backup (src: enabled: true) or the destination of a backup (dest: enabled: true). Setting both to true allows the PVC to both restore from an existing ReplicationDestination if one exists and then act as a backup source afterward
- Take note of the backup schedule, which does what it sounds like it does.
values.cnpg
- This section configures CNPG, and can be a bit sticky. Unlike VolSync, CNPG seems to fail if restore is set to true and no backup exists. I run CNPG either in backup mode or in restore mode, but not in both. That’s why the mode: recovery line is commented out, we’ll explore that later in this post.

./apps/production

vaultwarden-test.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: vaultwarden-test
  namespace: flux-system
spec:
  dependsOn:
    - name: ingress-nginx
  interval: 10m
  path: apps/general/vaultwarden-test
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

And don’t forget to add a reference to apps/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - vaultwarden-test.yaml

Commit and Test

Commit your changes:

$ git add -A && git commit -m "add vaultwarden test"
$ git push

$ flux events --watch

As soon as flux reconciles the HR, your cluster should get to work on the deployment. Switch to watching pods in the vaultwarden namespace. Pay particular attention to the VolSync and CNPG pods that appear, do their tasks, then complete. While it works, you can also watch volumes appear in the longhorn UI.

 kubectl get pods -n vaultwarden-test --watch
NAME                                           READY   STATUS              RESTARTS   AGE
vaultwarden-5d5c5bdcdb-djkw2                   0/1     Pending             0          6s
vaultwarden-cnpg-main-1-initdb-xrbt7           0/1     Pending             0          4s
volsync-dst-vaultwarden-data-data-dest-wz44l   0/1     ContainerCreating   0          5s
vaultwarden-cnpg-main-1-initdb-xrbt7           0/1     Pending             0          4s
vaultwarden-cnpg-main-1-initdb-xrbt7           0/1     Init:0/1            0          4s
vaultwarden-cnpg-main-1-initdb-xrbt7           0/1     Init:0/1            0          26s
volsync-dst-vaultwarden-data-data-dest-wz44l   1/1     Running             0          28s
vaultwarden-cnpg-main-1-initdb-xrbt7           0/1     PodInitializing     0          28s
vaultwarden-cnpg-main-1-initdb-xrbt7           1/1     Running             0          30s
volsync-dst-vaultwarden-data-data-dest-wz44l   0/1     Completed           0          32s
vaultwarden-cnpg-main-1-initdb-xrbt7           0/1     Completed           0          32s
volsync-dst-vaultwarden-data-data-dest-wz44l   0/1     Completed           0          33s
vaultwarden-cnpg-main-1-initdb-xrbt7           0/1     Completed           0          32s
vaultwarden-cnpg-main-1-initdb-xrbt7           0/1     Completed           0          33s
volsync-dst-vaultwarden-data-data-dest-wz44l   0/1     Completed           0          34s
vaultwarden-cnpg-main-1                        0/1     Pending             0          0s
vaultwarden-cnpg-main-1                        0/1     Pending             0          0s
vaultwarden-cnpg-main-1                        0/1     Init:0/1            0          0s
volsync-dst-vaultwarden-data-data-dest-wz44l   0/1     Completed           0          37s
volsync-dst-vaultwarden-data-data-dest-wz44l   0/1     Completed           0          37s
vaultwarden-cnpg-main-1                        0/1     Init:0/1            0          15s
vaultwarden-cnpg-main-1                        0/1     PodInitializing     0          16s
vaultwarden-5d5c5bdcdb-djkw2                   0/1     Pending             0          54s
vaultwarden-5d5c5bdcdb-djkw2                   0/1     Init:0/1            0          54s
vaultwarden-cnpg-main-1                        0/1     Running             0          17s
vaultwarden-cnpg-main-1                        0/1     Running             0          20s
vaultwarden-cnpg-main-1                        0/1     Running             0          24s
vaultwarden-cnpg-main-1                        1/1     Running             0          24s
vaultwarden-cnpg-main-2-join-6j6dw             0/1     Pending             0          0s
vaultwarden-cnpg-main-2-join-6j6dw             0/1     Pending             0          0s
vaultwarden-cnpg-main-2-join-6j6dw             0/1     Pending             0          1s
vaultwarden-cnpg-main-2-join-6j6dw             0/1     Pending             0          7s
vaultwarden-cnpg-main-2-join-6j6dw             0/1     Init:0/1            0          7s
vaultwarden-5d5c5bdcdb-djkw2                   0/1     Terminating         0          71s
vaultwarden-5d5c5bdcdb-4v6t6                   0/1     Pending             0          0s
vaultwarden-5d5c5bdcdb-4v6t6                   0/1     Pending             0          0s
vaultwarden-5d5c5bdcdb-4v6t6                   0/1     Init:0/1            0          0s
vaultwarden-5d5c5bdcdb-djkw2                   0/1     Init:ContainerStatusUnknown   0          73s
vaultwarden-5d5c5bdcdb-djkw2                   0/1     Init:ContainerStatusUnknown   0          73s
vaultwarden-5d5c5bdcdb-djkw2                   0/1     Init:ContainerStatusUnknown   0          73s
volsync-src-vaultwarden-data-data-2sgll        0/1     Pending                       0          0s
volsync-src-vaultwarden-data-data-2sgll        0/1     Pending                       0          0s
volsync-src-vaultwarden-data-data-2sgll        0/1     Pending                       0          7s
volsync-src-vaultwarden-data-data-2sgll        0/1     ContainerCreating             0          10s
vaultwarden-cnpg-main-2-join-6j6dw             0/1     Init:0/1                      0          33s
vaultwarden-cnpg-main-2-join-6j6dw             0/1     PodInitializing               0          35s
vaultwarden-cnpg-main-2-join-6j6dw             1/1     Running                       0          40s
vaultwarden-cnpg-main-2-join-6j6dw             0/1     Completed                     0          46s
vaultwarden-cnpg-main-2-join-6j6dw             0/1     Completed                     0          47s
vaultwarden-cnpg-main-2-join-6j6dw             0/1     Completed                     0          48s
vaultwarden-cnpg-main-2                        0/1     Pending                       0          0s
vaultwarden-cnpg-main-2                        0/1     Pending                       0          0s
vaultwarden-cnpg-main-2                        0/1     Init:0/1                      0          2s
vaultwarden-cnpg-main-2                        0/1     Init:0/1                      0          25s
vaultwarden-cnpg-main-2                        0/1     PodInitializing               0          26s
vaultwarden-cnpg-main-2                        0/1     Running                       0          30s
vaultwarden-cnpg-main-2                        0/1     Running                       0          32s
vaultwarden-cnpg-main-2                        0/1     Running                       0          33s
vaultwarden-cnpg-main-2                        1/1     Running                       0          34s
vaultwarden-cnpg-main-1-initdb-xrbt7           0/1     Completed                     0          2m24s
vaultwarden-cnpg-main-2-join-6j6dw             0/1     Completed                     0          84s
vaultwarden-cnpg-main-1-initdb-xrbt7           0/1     Completed                     0          2m24s
vaultwarden-cnpg-main-2-join-6j6dw             0/1     Completed                     0          84s
volsync-src-vaultwarden-data-data-2sgll        1/1     Running                       0          88s
volsync-src-vaultwarden-data-data-2sgll        0/1     Completed                     0          90s
volsync-src-vaultwarden-data-data-2sgll        0/1     Completed                     0          92s
volsync-src-vaultwarden-data-data-2sgll        0/1     Completed                     0          92s
volsync-src-vaultwarden-data-data-2sgll        0/1     Completed                     0          93s
volsync-src-vaultwarden-data-data-2sgll        0/1     Completed                     0          93s
vaultwarden-5d5c5bdcdb-4v6t6                   0/1     Init:0/1                      0          112s
vaultwarden-5d5c5bdcdb-4v6t6                   0/1     PodInitializing               0          113s
vaultwarden-5d5c5bdcdb-4v6t6                   0/1     Running                       0          2m10s
vaultwarden-5d5c5bdcdb-4v6t6                   0/1     Running                       0          2m21s
vaultwarden-5d5c5bdcdb-4v6t6                   1/1     Running                       0          2m33s

If everything went well, you should be able to reach vaultwarden at the ingress url you set up for it. Go ahead and create yourself a test record, You can verify this record later when we restore.

Inspect Your Backups

VolSync and CNPG both offer tools that you can use to inspect, verify, and administer your backups.

VolSync

You can inspect source and destination volume status with Kubectl.

$ kubectl get replicationdestinations.volsync.backube -n vaultwarden-test 
NAME                              LAST SYNC              DURATION       NEXT SYNC
vaultwarden-test-data-data-dest   2025-11-28T01:56:41Z   27.96262596s   
$ kubectl get replicationsources.volsync.backube -n vaultwarden-test 
NAME                         SOURCE                  LAST SYNC              DURATION          NEXT SYNC
vaultwarden-test-data-data   vaultwarden-test-data   2025-11-28T02:02:01Z   5m47.901609977s   2025-11-29T01:40:00Z

If you install the VolSync plugin for kubectl VolSync.readthedocs.io, you can also manually migrate or replicate data. I have not used these commands, but I’m aware they exist, and you should be too.

$ kubectl volsync migration
$ kubectl volsync replication

CNPG

Likewise, you can find your CNPG cluster with kubectl and inspect it with the kubectl plugin clougnative-pg.io

$ kubectl get clusters.postgresql.cnpg.io -n vaultwarden-test vaultwarden-test-cnpg-main 
NAME                         AGE   INSTANCES   READY   STATUS                     PRIMARY
vaultwarden-test-cnpg-main   10h   2           2       Cluster in healthy state   vaultwarden-test-cnpg-main-1

And to check health status.

$ kubectl cnpg status -n vaultwarden-test vaultwarden-test-cnpg-main
Cluster Summary
Name                     vaultwarden-test/vaultwarden-test-cnpg-main
System ID:               7577589366663393300
PostgreSQL Image:        ghcr.io/cloudnative-pg/postgresql:16.11
Primary instance:        vaultwarden-test-cnpg-main-1
Primary promotion time:  2025-11-28 01:57:07 +0000 UTC (10h28m55s)
Status:                  Cluster in healthy state 
Instances:               2
Ready instances:         2
Size:                    176M
Current Write LSN:       0/9000060 (Timeline: 1 - WAL File: 000000010000000000000009)

Continuous Backup status
First Point of Recoverability:  2025-11-26T22:16:29Z
Working WAL archiving:          OK
WALs waiting to be archived:    0
Last Archived WAL:              000000010000000000000008   @   2025-11-28T02:08:20.28951Z
Last Failed WAL:                -

Streaming Replication status
Replication Slots Enabled
Name                          Sent LSN   Write LSN  Flush LSN  Replay LSN  Write Lag  Flush Lag  Replay Lag  State      Sync State  Sync Priority  Replication Slot
----                          --------   ---------  ---------  ----------  ---------  ---------  ----------  -----      ----------  -------------  ----------------
vaultwarden-test-cnpg-main-2  0/9000060  0/9000060  0/9000060  0/9000060   00:00:00   00:00:00   00:00:00    streaming  async       0              active

Instances status
Name                          Current LSN  Replication role  Status  QoS        Manager Version  Node
----                          -----------  ----------------  ------  ---        ---------------  ----
vaultwarden-test-cnpg-main-1  0/9000060    Primary           OK      Burstable  1.27.1           truek8s-w1
vaultwarden-test-cnpg-main-2  0/9000060    Standby (async)   OK      Burstable  1.27.1           truek8s-w2

Restore Test

A backup is only as good as its last verified restore point. Let’s go through the steps of restoring our vaultwarden instance.

Before you proceed, note that CNPG and VolSync take backups immediately upon deployment and then again on the scheduled defined in the helm-release. If you just deployed your chart and committed a record, you should wait for the next backup or trigger one manually using the commands above.

First, suspend the vaultwarden ks, then delete the entire vaultwarden namespace. That will delete all kustomizations, deployments, volumes, snapshots, etc.

$ flux suspend ks vaultwarden-test
► suspending kustomization vaultwarden-test in flux-system namespace
✔ kustomization suspended
#It will hang at this step for a while as longhorn works on deleting volumes and snapshots.
$ kubectl delete namespaces vaultwarden-test 
namespace "vaultwarden-test" deleted

Now, you just need to edit your helm-release and tell CNPG that it needs to attempt to restore from backup. Just uncomment the cnpg.main.cluster.mode line in your helm-release.yaml.

        cnpg:
            main:
              cluster:
                singleNode: true
              mode: recovery
              backups:
                enabled: true
                credentials: backblaze
                scheduledBackups:
                  - name: daily-backup
                    schedule: "0 5 0 * * *"
                    backupOwnerReference: self
                    immediate: true
                    suspend: false
              recovery:
                method: object_store
                credentials: backblaze

Double check that the vaultwarden ks is suspended, and then commit your changes. Watch for the Vaultwarden Kustomization to be reconcile.

$ git add -A && git commit -m 'test vaultwarden restore'
$ flux get ks vaultwarden
NAME            REVISION                SUSPENDED       READY   MESSAGE                              
vaultwarden     main@sha1:af980727      True            True    Applied revision: main@sha1:af980727
$ flux resume ks vaultwarden
► resuming kustomization vaultwarden in flux-system namespace
✔ kustomization resumed
◎ waiting for Kustomization reconciliation
✔ Kustomization vaultwarden reconciliation completed
✔ applied revision main@sha1:9e77a508dbd94f59dd4d1e3c50bf7c2ede29587f

Pay attention to the Longhorn UI and watch pods in the vaultwarden namespace. Pay particularly close attention to the VolSync and CNPG pods that show up.

$ kubectl get pods -n vaultwarden-test --watch
NAME                                                READY   STATUS              RESTARTS   AGE
vaultwarden-test-7996596577-vtfsl                   0/1     Pending             0          6s
vaultwarden-test-cnpg-main-1-full-recovery-gpx84    0/1     Init:0/1            0          5s
volsync-dst-vaultwarden-test-data-data-dest-k5j7t   0/1     ContainerCreating   0          5s
volsync-dst-vaultwarden-test-data-data-dest-k5j7t   1/1     Running             0          15s
vaultwarden-test-cnpg-main-1-full-recovery-gpx84    0/1     PodInitializing     0          15s
vaultwarden-test-cnpg-main-1-full-recovery-gpx84    1/1     Running             0          16s
volsync-dst-vaultwarden-test-data-data-dest-k5j7t   0/1     Completed           0          20s
volsync-dst-vaultwarden-test-data-data-dest-k5j7t   0/1     Completed           0          21s
volsync-dst-vaultwarden-test-data-data-dest-k5j7t   0/1     Completed           0          22s
volsync-dst-vaultwarden-test-data-data-dest-k5j7t   0/1     Completed           0          22s
volsync-dst-vaultwarden-test-data-data-dest-k5j7t   0/1     Completed           0          22s
vaultwarden-test-7996596577-vtfsl                   0/1     Pending             0          32s
vaultwarden-test-7996596577-vtfsl                   0/1     Init:0/1            0          32s
vaultwarden-test-cnpg-main-1-full-recovery-gpx84    0/1     Completed           0          42s
vaultwarden-test-cnpg-main-1-full-recovery-gpx84    0/1     Completed           0          44s
vaultwarden-test-cnpg-main-1-full-recovery-gpx84    0/1     Completed           0          44s
vaultwarden-test-cnpg-main-1                        0/1     Pending             0          0s
vaultwarden-test-cnpg-main-1                        0/1     Pending             0          0s
vaultwarden-test-cnpg-main-1                        0/1     Init:0/1            0          0s
vaultwarden-test-cnpg-main-1                        0/1     PodInitializing     0          15s
vaultwarden-test-cnpg-main-1                        0/1     Running             0          16s
vaultwarden-test-cnpg-main-1                        0/1     Running             0          17s
vaultwarden-test-cnpg-main-1                        0/1     Running             0          24s
vaultwarden-test-cnpg-main-1                        1/1     Running             0          24s
vaultwarden-test-cnpg-main-2-join-kpnrz             0/1     Pending             0          0s
vaultwarden-test-cnpg-main-2-join-kpnrz             0/1     Pending             0          0s
vaultwarden-test-cnpg-main-2-join-kpnrz             0/1     Pending             0          0s
vaultwarden-test-cnpg-main-2-join-kpnrz             0/1     Pending             0          2s
volsync-src-vaultwarden-test-data-data-464ws        0/1     Pending             0          0s
volsync-src-vaultwarden-test-data-data-464ws        0/1     Pending             0          0s
vaultwarden-test-cnpg-main-2-join-kpnrz             0/1     Pending             0          3s
vaultwarden-test-cnpg-main-2-join-kpnrz             0/1     Init:0/1            0          3s
volsync-src-vaultwarden-test-data-data-464ws        0/1     Pending             0          2s
volsync-src-vaultwarden-test-data-data-464ws        0/1     ContainerCreating   0          2s
vaultwarden-test-cnpg-main-2-join-kpnrz             0/1     PodInitializing     0          25s
vaultwarden-test-cnpg-main-2-join-kpnrz             1/1     Running             0          26s
vaultwarden-test-cnpg-main-2-join-kpnrz             0/1     Completed           0          33s
vaultwarden-test-7996596577-vtfsl                   0/1     PodInitializing     0          103s
vaultwarden-test-cnpg-main-2-join-kpnrz             0/1     Completed           0          34s
vaultwarden-test-7996596577-vtfsl                   0/1     Running             0          104s
vaultwarden-test-cnpg-main-2-join-kpnrz             0/1     Completed           0          35s
vaultwarden-test-cnpg-main-2                        0/1     Pending             0          0s
vaultwarden-test-cnpg-main-2                        0/1     Pending             0          0s
vaultwarden-test-cnpg-main-2                        0/1     Init:0/1            0          0s
vaultwarden-test-cnpg-main-2                        0/1     Init:0/1            0          7s
vaultwarden-test-cnpg-main-2                        0/1     PodInitializing     0          8s
vaultwarden-test-cnpg-main-2                        0/1     Running             0          9s
vaultwarden-test-cnpg-main-2                        0/1     Running             0          10s
vaultwarden-test-7996596577-vtfsl                   0/1     Running             0          117s
vaultwarden-test-cnpg-main-2                        0/1     Running             0          15s
vaultwarden-test-cnpg-main-2                        1/1     Running             0          16s
vaultwarden-test-cnpg-main-1-full-recovery-gpx84    0/1     Completed           0          2m
vaultwarden-test-cnpg-main-1-full-recovery-gpx84    0/1     Completed           0          2m
vaultwarden-test-cnpg-main-2-join-kpnrz             0/1     Completed           0          51s
vaultwarden-test-cnpg-main-2-join-kpnrz             0/1     Completed           0          51s
vaultwarden-test-7996596577-vtfsl                   1/1     Running             0          2m9s
volsync-src-vaultwarden-test-data-data-464ws        1/1     Running             0          72s
volsync-src-vaultwarden-test-data-data-464ws        0/1     Completed           0          78s
volsync-src-vaultwarden-test-data-data-464ws        0/1     Completed           0          80s
volsync-src-vaultwarden-test-data-data-464ws        0/1     Completed           0          80s
volsync-src-vaultwarden-test-data-data-464ws        0/1     Completed           0          81s
volsync-src-vaultwarden-test-data-data-464ws        0/1     Completed           0          81s

You’ll know it worked when the main vaultwarden pod reaches ready status. Once it does, you should be able to log in and verify your test record.

If you run into any issues during the restore, use kubectl to inspect pods in the namespace and look for any errors. Most common points of failure are unfulfilled persistent volume claims due to issues with VolSync, or failure of one of the CNPG pods.

$ kubectl logs -n volsync volsync-8547d688c4-hf8ns volsync
$ kubectl logs -n cloudnative-pg cloudnative-pg-777f845d49-qssn5 manager

Advice and Troubleshooting Tips

Of all the features of the TrueCharts\K8S ecosystem, I would say I struggled the most with VolSync and CNPG, mostly because testing and breaking this is costly, and when things went wrong I would have to start all over. Almost the entire point of this whole series was wanting to really work out this backup component and try to share the guide for truecharts backups I wished I had when I set out on this project.

1. Replication is not backup

The title of this post is a bit of a misnomer… CNPG provides true backups through it’s revision feature, but VolSync is exactly what it says. A synchronization tool, not a backup tool. More than once I accidentally overwrote my ‘backups’ while trying to restore. Consider taking snapshots of your S3 buckets for an extra layer of protection.

2. Be safe, waste the space

You may notice that in my deployment cleanupTempPVC and cleanupCachePVC are set to false. I started doing this when I had issues restoring from VolSync backups, and found that VolSync was complaining that the target snapshot wasn’t available. This would cause volumes to be created, but VolSync wouldn’t 'release them, and the storage claim would never be fulfilled. I’m not sure exactly what the bug is, but one of the guys from TrueCharts told me there was a bug in VolSync and I should disable these features. I haven’t had any issues since.

3. Watch for volume not ready errors

While performing restore operations, keep an eye on pods as they spin up. If VolSync restore fails, your first hint will be that the pod is stuck in the pending state. If you check in longhorn, the volume will exist but will be in a not ready for workloads state because VolSync didn’t finish it’s job and release it. (see tip #2.) Inspect pods with kubectl describe pod to check for this.

4. Optionally manage CNPG Revisions manually

In earlier versions of the truecharts common library, CNPG required a revision statement for backup and recovery targets. This is no longer required, but is still supported if you want to manually manage backup revisions. If you do this, make sure you set the correct revision to restore, and also set the backup to an unused revision number, otherwise CNPG will complain about finding existing backups where it’s trying to write new ones.

  cnpg:
      main:
        cluster:
          singleNode: true
        mode: recovery
        backups:
          enabled: true
          revision: "2"
          credentials: backblaze
          scheduledBackups:
            - name: daily-backup
              schedule: "0 5 0 * * *"
              backupOwnerReference: self
              immediate: true
              suspend: false
        recovery:
          method: object_store
          revision: "1"
          credentials: backblaze

Sources

TrueK8S Part 09

2025-11-29T12:30:00Z

Tying It All Together

We’ve finally configured all of the infrastructure components of our cluster. We have Flux, secure secret management, certificate management, ingress, and a backup/replication solution. We’re finally ready to deploy a Helm chart that takes advantage of all of these features in our cluster.

I can’t think of a better example than Nextcloud. Nextcloud was something I had always wanted to host in my homelab to de-consumer-cloud my life, but I never quite got around to it. I never had a solution that felt stable enough to depend on. Through this project, I’ve finally figured out answers to the storage, secure access, and resilience concerns I had about running Nextcloud myself. If you’ve been following along, you should be well equipped to do the same.

Mounting NFS share for Nextcloud

Before we get started, we need to configure an NFS share for Nextcloud. In Part 7 I was complaining about how NFS was a poor solution for PVC storage. That is true, but there are some cases in which NFS makes sense. Nextcloud will have a particularly large dataset, and it doesn’t make sense to stuff that data into a PVC and try to manage it with VolSync. It makes more sense to keep that data external and to do snapshots/backups on the storage side. I’ll be hosting an NFS share on my TrueNAS server.

Configure the Share

Details of configuring an NFS share are beyond the scope of this series, but I will summarize by showing you the dataset I’m sharing. Note the permissions are set to ‘apps:apps’ which is the security context for TrueCharts apps (coming from the default apps user in TrueNAS). Also note below how the path you see in TrueNAS relates to the NFS patch you’ll use in your cluster config.

Save Secrets in cluster-config

You’ll need to save some config options for Nextcloud, so get those in your ConfigMap first. You need to create a username and password, specify your NFS server IP and path, and also a unique credential for the collabora service account. Collabora will complain if there are non-alphanumeric characters in its password, so look out for that.

apiVersion: v1
kind: ConfigMap
metadata:
    name: cluster-config
    namespace: flux-system
data:
    NEXTCLOUDDEMO_PASSWORD: PASSWORD
    NEXTCLOUDDEMO_USER: USERNAME
    NEXTCLOUDDEMOCOLLABORA_PASSWORD: PASSWORD #ALPHANUMERIC ONLY
    NEXTCLOUDDEMO_NFS_PATH: /mnt/Pool1/nextcloud-demo
    NEXTCLOUDDEMO_NFS_IP: 192.168.XX.X

Then go ahead and save your ConfigMap and push it to the repo

$ sops -e -i cluster-config.yaml 
$ git add -A && git commit -m 'added nextcloud secrets to cluster-config'
$ git push

Deploying Nextcloud

Create deployment manifests for Nextcloud. Remember, we’ll define the application itself in the apps directory, and then use a kustomization in production to apply it.

apps
├── general
│ └── nextcloud-demo
│   ├── helm-release.yaml
│   ├── kustomization.yaml
│   └── namespace.yaml
└── production
    └── nextcloud.yaml

./apps/general/nextcloud

helm-release.yaml

There are a couple unique items to pay attention to in this manifest.

I’m using a custom ingress route to route /collabora traffic to a different service
The NFS share is declared as a persistence item
The CNPG recovery configuration is staged but commented out
Cleanup for VolSync is disabled
spec.value.podOptions.hostUsers is set to true - this prevents an odd ‘no space left on device issue’ and I recommend this for all of your charts.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
    name: nextcloud-demo
    namespace: nextcloud-demo
spec:
    interval: 5m
    chart:
        spec:
            chart: nextcloud
            sourceRef:
                kind: HelmRepository
                name: truecharts
                namespace: flux-system
            interval: 5m
    install:
        createNamespace: true
        crds: CreateReplace
        remediation:
            retries: 1
    upgrade:
        crds: CreateReplace
        remediation:
            retries: 1
    timeout: 30m
    values:
      podOptions:
        hostUsers: true
      global:
        stopAll: false
      credentials:
        backblaze:
          type: s3
          url: "${S3_URL}"
          bucket: "${S3_BUCKET}-nextcloud-demo"
          accessKey: "${S3_ACCESS_KEY}"
          secretKey: "${S3_SECRET_KEY}"
          encrKey: "${S3_ENCRYPTION_KEY}"
      ingress:
        main:
          enabled: true
          ingressClassName: internal
          integrations:
            traefik:
              enabled: false
            certManager:
              enabled: true
              certificateIssuer: letsencrypt-domain-0
          hosts:
            - host: nextclouddemo.${DOMAIN_0}
              paths:
                - path: /
                  pathType: Prefix
        cloudflare:
          enabled: true
          ingressClassName: cloudflare
          integrations:
            traefik:
              enabled: false
          hosts:
          - host: nextclouddemo.${DOMAIN_0}
            paths:
            - path: /collabora/
              pathType: Prefix
              overrideService:
                name: collabora
                port: 9980
            - path: /
              pathType: Prefix
      cnpg:
        main:
          cluster:
            singleNode: true
#          mode: recovery
          backups:
            enabled: true
            credentials: backblaze
            scheduledBackups:
              - name: daily-backup
                schedule: "0 0 0 * * *"
                backupOwnerReference: self
                immediate: true
                suspend: false
          recovery:
            method: object_store
            credentials: backblaze
      persistence:
        config:
          accessModes: 
            - ReadWriteMany
          volsync:
          - name: config
            type: restic
            cleanupTempPVC: false
            cleanupCachePVC: false
            credentials: backblaze
            dest:
              enabled: true
            src:
              enabled: true
              trigger:
                schedule: 10 1 * * *
        data:
          enabled: true
          type: nfs
          path: ${NEXTCLOUDDEMO_NFS_PATH}
          server: ${NEXTCLOUDDEMO_NFS_IP}
        html:
          accessModes: 
            - ReadWriteMany
          volsync:
          - name: html
            type: restic
            cleanupTempPVC: false
            cleanupCachePVC: false
            credentials: backblaze
            dest:
              enabled: true
            src:
              enabled: true
              trigger:
                schedule: 20 1 * * *
      nextcloud:
        clamav:
          enabled: true
          infected_action: delete
        collabora:
          enabled: true
          password: "${NEXTCLOUDDEMOCOLLABORA_PASSWORD}"
          dictionaries:
            - en_US
        credentials:
          initialAdminPassword: "${NEXTCLOUDDEMO_PASSWORD}"
          initialAdminUser: "${NEXTCLOUDDEMO_USER}"
        general:
          accessIP: ${NGINX_IP}
          default_phone_region: US

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: nextcloud-demo

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - helm-release.yaml

./apps/production/

nextcloud.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: nextcloud-demo
  namespace: flux-system
spec:
  interval: 10m
  path: apps/general/nextcloud-demo
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

And don’t forget to add nextcloud-demo.yaml to the production kustomization

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - nextcloud-demo.yaml

Finally, commit your changes and watch

git add -A && git commit -m "added nextcloud-demo"
git push

$ flux events --watch | grep nextcloud-demo
0s      Normal  Progressing     Kustomization/apps      Kustomization/flux-system/nextcloud-demo created
0s      Normal  Progressing     Kustomization/nextcloud-demo    Namespace/nextcloud-demo created                 
                                                                HelmRelease/nextcloud-demo/nextcloud-demo created
0s      Normal  ReconciliationSucceeded Kustomization/nextcloud-demo    Reconciliation finished in 104.88413ms, next run in 10m0s
0s      Normal  ChartPullSucceeded      HelmChart/nextcloud-demo-nextcloud-demo pulled 'nextcloud' chart with version '40.4.1'

This will take quite some time to spin up. Be patient and inspect the results as it works. You can check the Longhorn console to view volumes as they’re created.

And you can watch the status of the containers as they spin up to monitor progress:

$ kubectl get pods -n nextcloud-demo --watch
NAME                                                  READY   STATUS              RESTARTS   AGE
nextcloud-demo-7678bd6c84-tnktd                       0/1     Pending             0          10s
nextcloud-demo-clamav-76d59bb979-jfz2r                0/1     Init:0/2            0          10s
nextcloud-demo-cnpg-main-1-initdb-lmfxf               0/1     Init:0/1            0          9s
nextcloud-demo-collabora-845c9c784-qr8rv              0/1     Init:0/2            0          10s
nextcloud-demo-imaginary-77fd75cdcf-bbt6x             0/1     Init:0/2            0          10s
nextcloud-demo-nginx-7b5d55df54-wk6fh                 0/1     Pending             0          10s
nextcloud-demo-notify-654fd67c9c-dwx6b                0/1     Pending             0          10s
nextcloud-demo-redis-0                                0/1     Running             0          10s
volsync-dst-nextcloud-demo-config-config-dest-7fw4f   0/1     ContainerCreating   0          10s
volsync-dst-nextcloud-demo-html-html-dest-827md       0/1     ContainerCreating   0          10s
nextcloud-demo-cnpg-main-1-initdb-lmfxf               0/1     Init:0/1            0          17s
nextcloud-demo-cnpg-main-1-initdb-lmfxf               0/1     PodInitializing     0          18s
nextcloud-demo-cnpg-main-1-initdb-lmfxf               1/1     Running             0          19s
nextcloud-demo-redis-0                                0/1     Running             0          20s
nextcloud-demo-cnpg-main-1-initdb-lmfxf               0/1     Completed           0          20s
nextcloud-demo-cnpg-main-1-initdb-lmfxf               0/1     Completed           0          21s
nextcloud-demo-cnpg-main-1-initdb-lmfxf               0/1     Completed           0          22s
volsync-dst-nextcloud-demo-html-html-dest-827md       1/1     Running             0          23s
nextcloud-demo-cnpg-main-1                            0/1     Pending             0          0s
nextcloud-demo-cnpg-main-1                            0/1     Pending             0          0s
nextcloud-demo-cnpg-main-1                            0/1     Init:0/1            0          0s
volsync-dst-nextcloud-demo-config-config-dest-7fw4f   1/1     Running             0          24s
volsync-dst-nextcloud-demo-html-html-dest-827md       0/1     Completed           0          28s
volsync-dst-nextcloud-demo-config-config-dest-7fw4f   0/1     Completed           0          29s
volsync-dst-nextcloud-demo-html-html-dest-827md       0/1     Completed           0          29s
volsync-dst-nextcloud-demo-html-html-dest-827md       0/1     Completed           0          30s
volsync-dst-nextcloud-demo-config-config-dest-7fw4f   0/1     Completed           0          30s
volsync-dst-nextcloud-demo-html-html-dest-827md       0/1     Completed           0          30s
volsync-dst-nextcloud-demo-html-html-dest-827md       0/1     Completed           0          30s
volsync-dst-nextcloud-demo-config-config-dest-7fw4f   0/1     Completed           0          31s
nextcloud-demo-cnpg-main-1                            0/1     PodInitializing     0          9s
volsync-dst-nextcloud-demo-config-config-dest-7fw4f   0/1     Completed           0          32s
volsync-dst-nextcloud-demo-config-config-dest-7fw4f   0/1     Completed           0          32s
nextcloud-demo-redis-0                                1/1     Running             0          32s
nextcloud-demo-cnpg-main-1                            0/1     Running             0          10s
nextcloud-demo-cnpg-main-1                            0/1     Running             0          11s
nextcloud-demo-cnpg-main-1                            0/1     Running             0          17s
nextcloud-demo-cnpg-main-1                            1/1     Running             0          18s
nextcloud-demo-cnpg-main-2-join-lwbqg                 0/1     Pending             0          0s
nextcloud-demo-cnpg-main-2-join-lwbqg                 0/1     Pending             0          0s
nextcloud-demo-cnpg-main-2-join-lwbqg                 0/1     Pending             0          0s
nextcloud-demo-7678bd6c84-tnktd                       0/1     Pending             0          42s
nextcloud-demo-notify-654fd67c9c-dwx6b                0/1     Pending             0          42s
nextcloud-demo-nginx-7b5d55df54-wk6fh                 0/1     Pending             0          42s
nextcloud-demo-7678bd6c84-tnktd                       0/1     Init:0/3            0          42s
nextcloud-demo-nginx-7b5d55df54-wk6fh                 0/1     Init:0/3            0          42s
nextcloud-demo-notify-654fd67c9c-dwx6b                0/1     Init:0/3            0          42s
nextcloud-demo-clamav-76d59bb979-jfz2r                0/1     Init:1/2            0          43s
nextcloud-demo-collabora-845c9c784-qr8rv              0/1     Init:1/2            0          43s
nextcloud-demo-imaginary-77fd75cdcf-bbt6x             0/1     Init:1/2            0          43s
nextcloud-demo-cnpg-main-2-join-lwbqg                 0/1     Pending             0          3s
nextcloud-demo-cnpg-main-2-join-lwbqg                 0/1     Init:0/1            0          3s
nextcloud-demo-imaginary-77fd75cdcf-bbt6x             0/1     PodInitializing     0          44s
nextcloud-demo-clamav-76d59bb979-jfz2r                0/1     PodInitializing     0          44s
nextcloud-demo-collabora-845c9c784-qr8rv              0/1     PodInitializing     0          44s
nextcloud-demo-clamav-76d59bb979-jfz2r                0/1     Running             0          45s
nextcloud-demo-imaginary-77fd75cdcf-bbt6x             0/1     Running             0          56s
nextcloud-demo-imaginary-77fd75cdcf-bbt6x             0/1     Running             0          70s
nextcloud-demo-clamav-76d59bb979-jfz2r                0/1     Running             0          75s
nextcloud-demo-cnpg-main-2-join-lwbqg                 0/1     Init:0/1            0          54s
nextcloud-demo-imaginary-77fd75cdcf-bbt6x             1/1     Running             0          98s
nextcloud-demo-clamav-76d59bb979-jfz2r                1/1     Running             0          98s
nextcloud-demo-cnpg-main-2-join-lwbqg                 0/1     PodInitializing     0          58s
nextcloud-demo-cnpg-main-2-join-lwbqg                 1/1     Running             0          60s
nextcloud-demo-collabora-845c9c784-qr8rv              0/1     Running             0          102s
nextcloud-demo-cnpg-main-2-join-lwbqg                 0/1     Completed           0          74s
nextcloud-demo-collabora-845c9c784-qr8rv              0/1     Running             0          115s
nextcloud-demo-cnpg-main-2-join-lwbqg                 0/1     Completed           0          75s
volsync-src-nextcloud-demo-config-config-4g24x        0/1     Pending             0          1s
nextcloud-demo-nextcloud-cron-29405805-bz2bs          0/1     Pending             0          1s
volsync-src-nextcloud-demo-html-html-94mkf            0/1     Pending             0          1s
volsync-src-nextcloud-demo-config-config-4g24x        0/1     Pending             0          1s
nextcloud-demo-nextcloud-cron-29405805-bz2bs          0/1     Pending             0          1s
volsync-src-nextcloud-demo-html-html-94mkf            0/1     Pending             0          1s
nextcloud-demo-cnpg-main-2-join-lwbqg                 0/1     Completed           0          79s
nextcloud-demo-nextcloud-cron-29405805-bz2bs          0/1     ContainerCreating   0          1s
nextcloud-demo-cnpg-main-2                            0/1     Pending             0          0s
nextcloud-demo-cnpg-main-2                            0/1     Pending             0          0s
nextcloud-demo-cnpg-main-2                            0/1     Init:0/1            0          0s
volsync-src-nextcloud-demo-config-config-4g24x        0/1     Pending             0          4s
volsync-src-nextcloud-demo-html-html-94mkf            0/1     Pending             0          4s
volsync-src-nextcloud-demo-config-config-4g24x        0/1     ContainerCreating   0          4s
volsync-src-nextcloud-demo-html-html-94mkf            0/1     ContainerCreating   0          4s
nextcloud-demo-collabora-845c9c784-qr8rv              1/1     Running             0          2m7s
nextcloud-demo-cnpg-main-2                            0/1     Init:0/1            0          9s
nextcloud-demo-cnpg-main-2                            0/1     PodInitializing     0          9s
nextcloud-demo-cnpg-main-2                            0/1     Running             0          10s
nextcloud-demo-cnpg-main-2                            0/1     Running             0          12s
nextcloud-demo-cnpg-main-2                            0/1     Running             0          17s
nextcloud-demo-cnpg-main-2                            1/1     Running             0          18s
nextcloud-demo-cnpg-main-1-initdb-lmfxf               0/1     Completed           0          2m18s
nextcloud-demo-cnpg-main-1-initdb-lmfxf               0/1     Completed           0          2m18s
nextcloud-demo-cnpg-main-2-join-lwbqg                 0/1     Completed           0          98s
nextcloud-demo-cnpg-main-2-join-lwbqg                 0/1     Completed           0          98s
nextcloud-demo-7678bd6c84-tnktd                       0/1     Init:1/3            0          2m43s
nextcloud-demo-nginx-7b5d55df54-wk6fh                 0/1     Init:0/3            0          2m44s
nextcloud-demo-nginx-7b5d55df54-wk6fh                 0/1     Init:1/3            0          2m45s
nextcloud-demo-notify-654fd67c9c-dwx6b                0/1     Init:1/3            0          2m46s
nextcloud-demo-nextcloud-cron-29405805-bz2bs          0/1     Completed           0          47s
nextcloud-demo-nextcloud-cron-29405805-bz2bs          0/1     Completed           0          47s
nextcloud-demo-7678bd6c84-tnktd                       0/1     Init:2/3            0          2m46s
nextcloud-demo-nginx-7b5d55df54-wk6fh                 0/1     Init:1/3            0          2m47s
nextcloud-demo-nextcloud-cron-29405805-bz2bs          0/1     Completed           0          49s
nextcloud-demo-nginx-7b5d55df54-wk6fh                 0/1     Init:2/3            0          2m48s
nextcloud-demo-notify-654fd67c9c-dwx6b                0/1     Init:2/3            0          2m48s
nextcloud-demo-7678bd6c84-tnktd                       0/1     PodInitializing     0          2m49s
nextcloud-demo-nginx-7b5d55df54-wk6fh                 0/1     Init:2/3            0          2m51s
nextcloud-demo-notify-654fd67c9c-dwx6b                0/1     Init:2/3            0          2m51s
nextcloud-demo-7678bd6c84-tnktd                       0/1     Running             0          2m53s
volsync-src-nextcloud-demo-html-html-94mkf            1/1     Running             0          77s
volsync-src-nextcloud-demo-config-config-4g24x        1/1     Running             0          77s
volsync-src-nextcloud-demo-html-html-94mkf            0/1     Completed           0          79s
volsync-src-nextcloud-demo-config-config-4g24x        0/1     Completed           0          79s
volsync-src-nextcloud-demo-html-html-94mkf            0/1     Completed           0          80s
volsync-src-nextcloud-demo-html-html-94mkf            0/1     Completed           0          80s
volsync-src-nextcloud-demo-config-config-4g24x        0/1     Completed           0          81s
volsync-src-nextcloud-demo-html-html-94mkf            0/1     Completed           0          82s
volsync-src-nextcloud-demo-html-html-94mkf            0/1     Completed           0          82s
volsync-src-nextcloud-demo-config-config-4g24x        0/1     Completed           0          82s
volsync-src-nextcloud-demo-config-config-4g24x        0/1     Completed           0          84s
volsync-src-nextcloud-demo-config-config-4g24x        0/1     Completed           0          84s
nextcloud-demo-nextcloud-cron-29405805-bz2bs          0/1     Completed           0          2m49s
nextcloud-demo-nextcloud-cron-29405805-bz2bs          0/1     Completed           0          2m49s
nextcloud-demo-7678bd6c84-tnktd                       0/1     Error               0          6m40s
nextcloud-demo-7678bd6c84-tnktd                       0/1     Running             1 (1s ago)   6m41s
nextcloud-demo-nextcloud-cron-29405810-nts66          0/1     Pending             0            0s
nextcloud-demo-nextcloud-cron-29405810-nts66          0/1     Pending             0            0s
nextcloud-demo-nextcloud-cron-29405810-nts66          0/1     ContainerCreating   0            0s
nextcloud-demo-nextcloud-cron-29405810-nts66          1/1     Running             0            1s
nextcloud-demo-nextcloud-cron-29405810-nts66          0/1     Completed           0            2s
nextcloud-demo-nextcloud-cron-29405810-nts66          0/1     Completed           0            3s
nextcloud-demo-nextcloud-cron-29405810-nts66          0/1     Completed           0            4s
nextcloud-demo-nextcloud-cron-29405810-nts66          0/1     Completed           0            2m4s
nextcloud-demo-nextcloud-cron-29405810-nts66          0/1     Completed           0            2m4s
nextcloud-demo-7678bd6c84-tnktd                       0/1     Running             2 (1s ago)   9m8s
nextcloud-demo-nextcloud-cron-29405815-t6k7m          0/1     Pending             0            0s
nextcloud-demo-nextcloud-cron-29405815-t6k7m          0/1     Pending             0            0s
nextcloud-demo-nextcloud-cron-29405815-t6k7m          0/1     ContainerCreating   0            0s
nextcloud-demo-nextcloud-cron-29405815-t6k7m          0/1     Completed           0            1s
nextcloud-demo-nextcloud-cron-29405815-t6k7m          0/1     Completed           0            2s
nextcloud-demo-nextcloud-cron-29405815-t6k7m          0/1     Completed           0            3s
nextcloud-demo-nextcloud-cron-29405815-t6k7m          0/1     Completed           0            2m3s
nextcloud-demo-nextcloud-cron-29405815-t6k7m          0/1     Completed           0            2m3s
nextcloud-demo-7678bd6c84-tnktd                       0/1     Running             2 (5m5s ago)   14m
nextcloud-demo-7678bd6c84-tnktd                       1/1     Running             2 (5m17s ago)   14m
nextcloud-demo-nginx-7b5d55df54-wk6fh                 0/1     PodInitializing     0               14m
nextcloud-demo-notify-654fd67c9c-dwx6b                0/1     PodInitializing     0               14m
nextcloud-demo-nginx-7b5d55df54-wk6fh                 0/1     Running             0               14m
nextcloud-demo-notify-654fd67c9c-dwx6b                0/1     Running             0               14m
nextcloud-demo-nginx-7b5d55df54-wk6fh                 0/1     Running             0               14m
nextcloud-demo-nginx-7b5d55df54-wk6fh                 1/1     Running             0               14m
nextcloud-demo-notify-654fd67c9c-dwx6b                0/1     Running             0               14m
nextcloud-demo-notify-654fd67c9c-dwx6b                1/1     Running             0               15m
nextcloud-demo-nextcloud-cron-29405820-kl57w          0/1     Pending             0               0s
nextcloud-demo-preview-cron-29405820-bgffl            0/1     Pending             0               0s
nextcloud-demo-nextcloud-cron-29405820-kl57w          0/1     Pending             0               0s
nextcloud-demo-preview-cron-29405820-bgffl            0/1     Pending             0               0s
nextcloud-demo-nextcloud-cron-29405820-kl57w          0/1     ContainerCreating   0               0s
nextcloud-demo-preview-cron-29405820-bgffl            0/1     ContainerCreating   0               0s
nextcloud-demo-preview-cron-29405820-bgffl            1/1     Running             0               0s
nextcloud-demo-nextcloud-cron-29405820-kl57w          1/1     Running             0               1s
nextcloud-demo-preview-cron-29405820-bgffl            0/1     Completed           0               2s
nextcloud-demo-preview-cron-29405820-bgffl            0/1     Completed           0               4s
nextcloud-demo-preview-cron-29405820-bgffl            0/1     Completed           0               5s

Once the main Nextcloud container spins up, you can check the logs to see what it’s doing. You’ll know it’s done when it starts logging HTTP requests.

$ kubectl logs -n nextcloud-demo nextcloud-demo-6695869969-2rdt5 
Initializing nextcloud 32.0.1 ...
[...]
--------------------------------------------------
Starting Nextcloud PHP-FPM
[28-Nov-2025 16:57:23] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[28-Nov-2025 16:57:23] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[28-Nov-2025 16:57:23] NOTICE: fpm is running, pid 1
[28-Nov-2025 16:57:23] NOTICE: ready to handle connections
127.0.0.1 -  28/Nov/2025:16:57:27 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:57:37 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:57:39 +0000 "GET status.php" 200
172.19.0.105 -  28/Nov/2025:16:57:39 +0000 "GET status.php" 200
172.19.0.103 -  28/Nov/2025:16:57:40 +0000 "GET status.php" 200
172.19.0.105 -  28/Nov/2025:16:57:42 +0000 "GET status.php" 200
172.19.0.103 -  28/Nov/2025:16:57:43 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:57:51 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:57:52 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:58:03 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:58:07 +0000 "GET status.php" 200
172.19.0.103 -  28/Nov/2025:16:58:10 +0000 "GET /status.php" 200
172.19.0.103 -  28/Nov/2025:16:58:10 +0000 "GET /index.php" 200
127.0.0.1 -  28/Nov/2025:16:58:15 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:58:22 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:58:27 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:58:37 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:58:39 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:58:51 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:58:52 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:59:03 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:59:07 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:59:15 +0000 "GET status.php" 200
127.0.0.1 -  28/Nov/2025:16:59:22 +0000 "GET status.php" 200

When all pods show running, you should have a new Nextcloud instance at the ingress URL you configured.

You can inspect the final results with the kubectl commands we’ve been using throughout the series.

Inspect all resources created in your cluster by the nextcloud-demo kustomization.

$ flux tree ks nextcloud-demo 
Kustomization/flux-system/nextcloud-demo
├── Namespace/nextcloud-demo
└── HelmRelease/nextcloud-demo/nextcloud-demo
    ├── Secret/nextcloud-demo/nextcloud-demo-cnpg-main-provider-backup-s3-creds
    ├── Secret/nextcloud-demo/nextcloud-demo-cnpg-main-urls
    ├── Secret/nextcloud-demo/nextcloud-demo-cnpg-main-user
    ├── Secret/nextcloud-demo/nextcloud-demo-rediscreds
    ├── Secret/nextcloud-demo/nextcloud-demo-config-volsync-config
    ├── Secret/nextcloud-demo/nextcloud-demo-html-volsync-html
    ├── ConfigMap/nextcloud-demo/nextcloud-demo-redis-health
    ├── ConfigMap/nextcloud-demo/nextcloud-demo-clamav-config
    ├── ConfigMap/nextcloud-demo/nextcloud-demo-collabora-config
    ├── ConfigMap/nextcloud-demo/nextcloud-demo-hpb-config
    ├── ConfigMap/nextcloud-demo/nextcloud-demo-nextcloud-config
    ├── ConfigMap/nextcloud-demo/nextcloud-demo-nginx-config
    ├── ConfigMap/nextcloud-demo/nextcloud-demo-opcache
    ├── ConfigMap/nextcloud-demo/nextcloud-demo-php-tune
    ├── ConfigMap/nextcloud-demo/nextcloud-demo-redis-session
    ├── PersistentVolumeClaim/nextcloud-demo/nextcloud-demo-config
    ├── PersistentVolumeClaim/nextcloud-demo/nextcloud-demo-html
    ├── Service/nextcloud-demo/nextcloud-demo-redis
    ├── Service/nextcloud-demo/nextcloud-demo-clamav
    ├── Service/nextcloud-demo/nextcloud-demo-collabora
    ├── Service/nextcloud-demo/nextcloud-demo-imaginary
    ├── Service/nextcloud-demo/nextcloud-demo
    ├── Service/nextcloud-demo/nextcloud-demo-nextcloud
    ├── Service/nextcloud-demo/nextcloud-demo-notify
    ├── Deployment/nextcloud-demo/nextcloud-demo-clamav
    ├── Deployment/nextcloud-demo/nextcloud-demo-collabora
    ├── Deployment/nextcloud-demo/nextcloud-demo-imaginary
    ├── Deployment/nextcloud-demo/nextcloud-demo
    ├── Deployment/nextcloud-demo/nextcloud-demo-nginx
    ├── Deployment/nextcloud-demo/nextcloud-demo-notify
    ├── StatefulSet/nextcloud-demo/nextcloud-demo-redis
    ├── CronJob/nextcloud-demo/nextcloud-demo-nextcloud-cron
    ├── CronJob/nextcloud-demo/nextcloud-demo-preview-cron
    ├── Ingress/nextcloud-demo/nextcloud-demo-cloudflare
    ├── Ingress/nextcloud-demo/nextcloud-demo
    ├── Cluster/nextcloud-demo/nextcloud-demo-cnpg-main
    ├── ReplicationDestination/nextcloud-demo/nextcloud-demo-config-config-dest
    ├── ReplicationDestination/nextcloud-demo/nextcloud-demo-html-html-dest
    ├── ReplicationSource/nextcloud-demo/nextcloud-demo-config-config
    ├── ReplicationSource/nextcloud-demo/nextcloud-demo-html-html
    └── ScheduledBackup/nextcloud-demo/nextcloud-demo-cnpg-main-sched-backup-daily-backup

Verify volsync source and destination configuration:

$ kubectl get replicationdestinations.volsync.backube -n nextcloud-demo 
NAME                                LAST SYNC              DURATION        NEXT SYNC
nextcloud-demo-config-config-dest   2025-11-28T16:43:47Z   32.604100966s   
nextcloud-demo-html-html-dest       2025-11-28T16:43:45Z   30.501917968s   
$ kubectl get replicationsources.volsync.backube -n nextcloud-demo 
NAME                           SOURCE                  LAST SYNC              DURATION          NEXT SYNC
nextcloud-demo-config-config   nextcloud-demo-config   2025-11-28T16:46:37Z   3m22.289395955s   2025-11-29T01:10:00Z
nextcloud-demo-html-html       nextcloud-demo-html     2025-11-28T16:46:35Z   3m20.125654664s   2025-11-29T01:20:00Z

Inspect the CNPG cluster created for Nextcloud:

$  kubectl get cluster -n nextcloud-demo
NAME                       AGE   INSTANCES   READY   STATUS                     PRIMARY
nextcloud-demo-cnpg-main   18m   2           2       Cluster in healthy state   nextcloud-demo-cnpg-main-1

$ kubectl cnpg status -n nextcloud-demo nextcloud-demo-cnpg-main
Cluster Summary
Name                     nextcloud-demo/nextcloud-demo-cnpg-main
System ID:               7577817879212458003
PostgreSQL Image:        ghcr.io/cloudnative-pg/postgresql:16.11
Primary instance:        nextcloud-demo-cnpg-main-1
Primary promotion time:  2025-11-28 16:43:48 +0000 UTC (18m52s)
Status:                  Cluster in healthy state 
Instances:               2
Ready instances:         2
Size:                    183M
Current Write LSN:       0/90D3EB8 (Timeline: 1 - WAL File: 000000010000000000000009)

Continuous Backup status
First Point of Recoverability:  2025-11-28T16:44:59Z
Working WAL archiving:          OK
WALs waiting to be archived:    0
Last Archived WAL:              000000010000000000000008   @   2025-11-28T17:00:03.47943Z
Last Failed WAL:                -

Streaming Replication status
Replication Slots Enabled
Name                        Sent LSN   Write LSN  Flush LSN  Replay LSN  Write Lag  Flush Lag  Replay Lag  State      Sync State  Sync Priority  Replication Slot
----                        --------   ---------  ---------  ----------  ---------  ---------  ----------  -----      ----------  -------------  ----------------
nextcloud-demo-cnpg-main-2  0/90D3EB8  0/90D3EB8  0/90D3EB8  0/90D3EB8   00:00:00   00:00:00   00:00:00    streaming  async       0              active

Instances status
Name                        Current LSN  Replication role  Status  QoS        Manager Version  Node
----                        -----------  ----------------  ------  ---        ---------------  ----
nextcloud-demo-cnpg-main-1  0/90D3EB8    Primary           OK      Burstable  1.27.1           truek8s-w2
nextcloud-demo-cnpg-main-2  0/90D3EB8    Standby (async)   OK      Burstable  1.27.1           truek8s-w1

Troubleshooting a failed deployment

If you have any issues during deployment (which is common - I did even while writing this guide) here are some steps to help:

Determine the cause of failure: Use flux get hr to see if the chart was deployed successfully or not. It may fail if there’s an error in your helm-release.yaml. Example:

$ flux get hr -n nextcloud-demo 
NAME            REVISION        SUSPENDED       READY   MESSAGE                                                                                                                                                                                                   
nextcloud-demo  40.4.1          False           False   Helm install failed for release nextcloud-demo/nextcloud-demo with chart nextcloud@40.4.1: execution error at (nextcloud/templates/common.yaml:16:4): Nextcloud - Collabora [Password] cannot contain [$]

Use kubectl describe pods on pods in the nextcloud-demo namespace. Look for unfulfilled persistent volume claims, or any pods in error state.
Restart and try again
– Suspend the KS with flux suspend ks nextcloud-demo.
– Delete the entire namespace with kubectl delete namespaces nextcloud-demo
– Delete any files written to your S3 bucket
– (unique to Nextcloud) delete any files written to the NFS shared by the failed deployment
– Make and commit your changes, then try again.

Sources

TrueK8S Part 10

2025-11-29T12:30:00Z

Destroying and Rebuilding our Talos Cluster

For the final post of this series, we’ll put our cluster through a disaster recovery scenario. Because our cluster’s configuration is defined declaratively and all of our volumes and databases are replicated to cloud storage, we can re-create our cluster with little more than our git repo and our SOPS key. For this exercise, I’ll be simulating a complete disaster recovery. Let’s assume that my current cluster and administration machine have been destroyed and work through the process of rebuilding the Talos cluster, rebuilding our administration workstation, and then bootstrapping our new cluster with our config.

Before getting started (and especially before destroying any of your work) you’ll need to make sure you have the following things lined up:

Your Talos patch files (optional, but convenient)
Access to your repo with the same or a new personal access token
Your SOPS key
Your Flux bootsrap command (optional, but convenient)

Destroy the Cluster

Go ahead and give your existing cluster a graceful shutdown. If you’re feeling brave you can delete them, otherwise power them off and keep them as a backup.

Rebuilding the Cluster

I’ll start by creating three new VMs, just like we did in part 1.

Don’t forget to install Talosctl if you haven’t already.

curl -sL https://talos.dev/install | sh

Save your IPs and cluster names as variables, then generate your initial talos config file.

export CONTROL_IP=192.168.XX.101
export WORKER1_IP=192.168.XX.111
export WORKER2_IP=192.168.XX.112
export CLUSTERNAME=truek8s

$ talosctl gen config $CLUSTERNAME https://$CONTROL_IP:6443 --install-image factory.talos.dev/installer/82866c01b2842b490c27a6f1a4996aae05f096c83db40ecda166b03da9deae46/:v1.9.2
generating PKI and tokens
Created /home/bmel/Documents/truek8s-talos/controlplane.yaml
Created /home/bmel/Documents/truek8s-talos/worker.yaml
Created /home/bmel/Documents/truek8s-talos/talosconfig

💡 Remember, you can specify any version using the same schematic ID. This guide was written with v1.9.2 but I’ve tested v1.10.8 and the process is the same.

Then go ahead and recreate or copy your patch files to your working directory. Check part 1 for details on recreating them. Remember, the only thing different between these three is the hostname field.

truek8s-c1.patch
truek8s-w1.patch
truek8s-w2.patch

And then patch your new machine config files:

$ talosctl machineconfig patch controlplane.yaml --patch @truek8s-c1.patch --output truek8s-c1.yaml
$ talosctl machineconfig patch worker.yaml --patch @truek8s-w1.patch --output truek8s-w1.yaml
$ talosctl machineconfig patch worker.yaml --patch @truek8s-w2.patch --output truek8s-w2.yaml

Double-check the machine config files you just generated, and then apply them to your cluster. Start with the control node, wait for it to come up, and then do your workers.

$ talosctl apply-config --insecure -n $CONTROL_IP --file truek8s-c1.yaml

$ talosctl apply-config --insecure -n $WORKER1_IP --file truek8s-w1.yaml
$ talosctl apply-config --insecure -n $WORKER2_IP --file truek8s-w2.yaml

When all three are back up and show healthy, finalize the install by bootstrapping the cluster:

$ talosctl bootstrap --nodes $CONTROL_IP --endpoints $CONTROL_IP --talosconfig talosconfig

Finally, generate and save your talos and kube config files.

copy and edit the talos config file

$ cp talosconfig ~/.talos/config
$ nano ~/.talos/config 
$ cat ~/.talos/config 
context: truek8s
contexts:
    truek8s:
        endpoints: [192.168.XX.101]
############################
############################
############################

Generate kubeconfig

$ talosctl kubeconfig -n $CONTROL_IP

If you don’t already have kubectl installed, install it:

$ apt install kubectl

And finally, check to make sure talosctl and kubectl are working.

$ talosctl dashboard -n $CONTROL_IP 

$ kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
truek8s-c1   Ready    control-plane   62s   v1.33.2
truek8s-w1   Ready    <none>          55s   v1.33.2
truek8s-w2   Ready    <none>          66s   v1.33.2

Clone your repo

Once the cluster is up, clone your flux repo to your admin workstation so you can make changes. You will need a private access token, but it doesn’t have to be the same one. Check part 2 for details on how to make a new one if you need it.

#You don't need quotes here
export GITHUB_TOKEN=<your-PAT>
export GITHUB_USER=<your-username>
export REPO=truek8s

git clone https://$GITHUB_TOKEN@github.com/$GITHUB_USER/$REPO
Cloning into 'truek8s'...

Configure SOPS

Before we can move on to bootstrapping flux, we need to configure SOPS. We’ll need to provide the key to flux as a kube secret later, but also need to have a copy of it on our admin workstation so we can decrypt our config map.

If you haven’t already, install SOPS and AGE.

# Download the binary
curl -LO https://github.com/getsops/sops/releases/download/v3.10.2/sops-v3.10.2.linux.amd64

# Move the binary in to your PATH
mv sops-v3.10.2.linux.amd64 /usr/local/bin/sops

# Make the binary executable
chmod +x /usr/local/bin/sops

#Install age
$ apt install age

💡 This command installs version 3.10.2. Check SOPS documentation to see if a newer version is available.

Copy your sops-key.text file to ~/.config/sops/age/keys.txt. You can test that SOPS is configured and that you have the right key by decrypting and re-encrypting your configmap.

$ sops -d -i cluster-config.yaml
$ sops -e -i cluster-config.yaml

Make sure you keep that key file handy. You’ll need to provide it to flux in a later step.

Volsync and CNPG Configuration

Before we bootstrap flux and commit our config to the cluster, we need to make sure that any charts using VolSync or CNPG are configured to restore. In the examples we’ve worked on through this guide, that simply means uncommenting the mode: recovery statement in any helm charts.

Using the configuration we made for nextcloud as an example:

      cnpg:
        main:
          cluster:
            singleNode: true
          pgVersion: 15
# Uncomment this line          
          mode: recovery
          backups:
            enabled: true
            credentials: backblaze
            scheduledBackups:
              - name: daily-backup
                schedule: "0 0 0 * * *"
                backupOwnerReference: self
                immediate: true
                suspend: false
          recovery:
            method: object_store
            credentials: backblaze

Make sure you do this for any CNPG enabled charts in your repo. When you’re all done, commit any changes to your repo:

⚠️ make sure you config map is encrypted before committing.

git add -A && git commit -m 'set recovery'
git push

Flux Bootstrap

Now all you need to do is bootstrap flux with your repo, hand it the sops key, and it will take care of everything from there. For the github PAT, you can create a new one or re-use the same one from earlier.

if you don’t already have it installed, go ahead and install flux:

curl -s https://fluxcd.io/install.sh | sudo bash

And then re-run the flux bootstrap command:

$ export GITHUB_TOKEN=<PAT>
$ export GITHUB_USER=<Github Username>
$ export REPO=truek8s
$ flux bootstrap github --token-auth --owner=$GITHUB_USER   --repository=$REPO --branch=main   --path=./clusters/$REPO   --personal
► connecting to github.com
► cloning branch "main" from Git repository "https://github.com/bmel343/truek8s.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed sync manifests to "main" ("6b49c159dff1f604c5df9e542731c41e0d4412f1")
► pushing component manifests to "https://github.com/bmel343/truek8s.git"
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ sync manifests are up to date
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy

As soon as flux is done bootstrapping, create the sops-age secret in the flux-system namespace. Don’t worry, thanks to the dependency configuration, Flux will wait for the secret to exist to reconcile the config ks, and all other KS depend on that.

$ kubectl -n flux-system create secret generic sops-age --from-file=age.agekey=sops-key.txt

secret/sops-age created

Sit Back and Watch

Flux will take it from here, working diligently to apply all the resources defined in our repo to the cluster. In my experience, it can take up 20-30 minutes for actual apps to start working again. I like to watch flux events, and check on specific kustomizations while I wait. Flux will be pretty noisy, and you may even see a few errors in there, but flux can often reconcile that by waiting and trying again. In this example it’s complaining about not finding the sops-age secret, but later finds it and continues along.

$ flux events --watch
LAST SEEN       TYPE    REASON                  OBJECT                  MESSAGE                                                   
11s             Normal  DependencyNotReady      Kustomization/apps      Dependencies do not meet ready condition, retrying in 30s
10s     Normal  NewArtifact     HelmRepository/cloudnative-pg   stored fetched index of size 77.48kB from 'https://cloudnative-pg.github.io/charts'
11s     Warning BuildFailed     Kustomization/config    secrets "sops-age" not found
18s     Normal  NewArtifact     GitRepository/flux-system       stored artifact for commit 'set recovery'
12s     Normal  Progressing     Kustomization/flux-system       CustomResourceDefinition/alerts.notification.toolkit.fluxcd.io configured     
                                                                CustomResourceDefinition/buckets.source.toolkit.fluxcd.io configured          
                                                                CustomResourceDefinition/externalartifacts.source.toolkit.fluxcd.io configured
                                                                CustomResourceDefinition/gitrepositories.source.toolkit.fluxcd.io configured  
                                                                CustomResourceDefinition/helmcharts.source.toolkit.fluxcd.io configured       
                                                                CustomResourceDefinition/helmreleases.helm.toolkit.fluxcd.io configured       
                                                                CustomResourceDefinition/helmrepositories.source.toolkit.fluxcd.io configured 
                                                                CustomResourceDefinition/kustomizations.kustomize.toolkit.fluxcd.io configured
                                                                CustomResourceDefinition/ocirepositories.source.toolkit.fluxcd.io configured  
                                                                CustomResourceDefinition/providers.notification.toolkit.fluxcd.io configured  
                                                                CustomResourceDefinition/receivers.notification.toolkit.fluxcd.io configured  
                                                                Namespace/flux-system configured                                              
                                                                ClusterRole/crd-controller-flux-system configured                             
                                                                ClusterRole/flux-edit-flux-system configured                                  
                                                                ClusterRole/flux-view-flux-system configured                                  
                                                                ClusterRoleBinding/cluster-reconciler-flux-system configured                  
                                                                ClusterRoleBinding/crd-controller-flux-system configured                      
                                                                ResourceQuota/flux-system/critical-pods-flux-system configured                
                                                                ServiceAccount/flux-system/helm-controller configured                         
                                                                ServiceAccount/flux-system/kustomize-controller configured                    
                                                                ServiceAccount/flux-system/notification-controller configured                 
                                                                ServiceAccount/flux-system/source-controller configured                       
                                                                Service/flux-system/notification-controller configured                        
                                                                Service/flux-system/source-controller configured                              
                                                                Service/flux-system/webhook-receiver configured                               
                                                                Deployment/flux-system/helm-controller configured                             
                                                                Deployment/flux-system/kustomize-controller configured                        
                                                                Deployment/flux-system/notification-controller configured                     
                                                                Deployment/flux-system/source-controller configured                           
                                                                Kustomization/flux-system/apps created                                        
                                                                Kustomization/flux-system/config created                                      
                                                                Kustomization/flux-system/flux-system configured                              
                                                                Kustomization/flux-system/infrastructure created                              
                                                                Kustomization/flux-system/repos created                                       
                                                                NetworkPolicy/flux-system/allow-egress configured                             
                                                                NetworkPolicy/flux-system/allow-scraping configured                           
                                                                NetworkPolicy/flux-system/allow-webhooks configured                           
                                                                GitRepository/flux-system/flux-system configured                              
12s     Normal  ReconciliationSucceeded Kustomization/flux-system       Reconciliation finished in 6.02411383s, next run in 10m0s
11s     Normal  DependencyNotReady      Kustomization/infrastructure    Dependencies do not meet ready condition, retrying in 30s
10s     Normal  NewArtifact     HelmRepository/ingress-nginx    stored fetched index of size 224.7kB from 'https://kubernetes.github.io/ingress-nginx'
10s     Normal  NewArtifact     HelmRepository/jetstack stored fetched index of size 492.1kB from 'https://charts.jetstack.io/'
10s     Normal  NewArtifact     HelmRepository/longhorn stored fetched index of size 70.95kB from 'https://charts.longhorn.io'
9s      Normal  NewArtifact     HelmRepository/metallb  stored fetched index of size 29.23kB from 'https://metallb.github.io/metallb'
11s     Normal  Progressing     Kustomization/repos     HelmRepository/flux-system/cloudnative-pg created
                                                        HelmRepository/flux-system/ingress-nginx created 
                                                        HelmRepository/flux-system/jetstack created      
                                                        HelmRepository/flux-system/longhorn created      
                                                        HelmRepository/flux-system/metallb created       
                                                        HelmRepository/flux-system/podinfo created       
                                                        HelmRepository/flux-system/strrl created         
                                                        HelmRepository/flux-system/truecharts created    
6s      Normal  Progressing     Kustomization/repos     Health check passed in 5.044890457s

Pay particular attention to the order in which kustomizations are applied. Think back to part 2 when we considered how best to structure our repository, and how to configure dependencies. For example, the apps kustomization watis for the infrastructure kustomization to be deployed first. An application like Vualtwarden or Nextcloud depends on VolSync, CNPG, clusterissuer, metallb, longhorn, etc. The way we’ve configured things, Flux knows to wait until all of those resources are available to start deploying any apps, which prevents all sorts of potential deployment errors.

$ flux get ks
NAME                    REVISION                SUSPENDED       READY   MESSAGE                                              
apps                                            False           False   dependency 'flux-system/infrastructure' is not ready
cert-manager            main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3                
cf-tunnel-ingress       main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3                
cloudnative-pg          main@sha1:0cf3b4e3      False           False   dependency 'flux-system/longhorn' is not ready      
cluster-issuer          main@sha1:0cf3b4e3      False           False   dependency 'flux-system/cert-manager' is not ready  
config                  main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3                
flux-system             main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3                
infrastructure                                  False           Unknown Reconciliation in progress                          
ingress-nginx           main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3                
kubernetes-reflector    main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3                
longhorn                main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3                
longhorn-config                                 False           False   dependency 'flux-system/longhorn' is not ready      
metallb                 main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3                
metallb-config          main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3                
repos                   main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3                
snapshot-controller     main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3                
volsync                 main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3                
volsync-config          main@sha1:0cf3b4e3      False           True    Applied revision: main@sha1:0cf3b4e3

Your first sign that things are working is when the infrastructure ks reconciles. Once it does, infrastructure components like longhorn, metallb, nginx should come up. One of my first checks is the longhorn UI, which will show volumes created for any restore tasks or new deployments.

Once flux starts deploying your applications, you can keep an eye on any pods created in that applicaiton namespace. It can be anxiety inducing to wait for a large deployment like nextcloud to reach ready status. Just know that if the CNPG and Volsync containers reach complete or running status, you’re probably fine.

$ kubectl get pods -n nextcloud-demo 
NAME                                             READY   STATUS              RESTARTS   AGE
nextcloud-demo-5c4777bf8d-f8dhp                  0/1     Init:0/3            0          8m11s
nextcloud-demo-clamav-76666fbd67-tsrbd           0/1     Init:0/2            0          8m11s
nextcloud-demo-cnpg-main-1-full-recovery-pws75   1/1     Running             0          8m10s
nextcloud-demo-collabora-644c55596f-rrglv        0/1     Init:0/2            0          8m11s
nextcloud-demo-imaginary-5d55d79fcf-2jbzd        0/1     Init:0/2            0          8m11s
nextcloud-demo-nextcloud-cron-29407260-7c2pq     0/1     ContainerCreating   0          7m8s
nextcloud-demo-nginx-675b945674-8n5d4            0/1     Init:0/3            0          8m11s
nextcloud-demo-notify-66bc88d785-zrfkj           0/1     Init:0/3            0          8m12s
nextcloud-demo-preview-cron-29407260-748xr       0/1     ContainerCreating   0          7m8s
nextcloud-demo-redis-0                           1/1     Running             0          8m11s

And when it’s all done, your applications should be back online. Here I am checking for my test record in vaultwarden, and logging back into nextcloud.

Conclusion

I could go on writing about this little project of mine for another 10 posts, but I need to draw a line in the sand so I can close this out and return to some of my other projects. All in all, it’s been over a year since I set out to migrate all of my TrueCharts apps into the Talos\Kubernetes ecosystem, and it’s been 6 months since I started writing this series. I think I had everything ‘working’ a few months ago, but challenging myself along the way to write this series of posts forced me to do my best to do things the right way. I’m sure I will have other posts related to this, but let’s call this the ‘core’ guide for my TrueK8S project. Thanks for reading, and I hope you found something useful here.