~jonashietala.se

Modern Kubernetes homelab

SOPS + Age and Sealed Secrets

Published: in f60b1c6
This is part 4 of the Modern Kubernetes homelab series.

When I’ve read other series about Kubernetes and reach the secrets section my eyes glaze over. I can’t help myself; I want to read about the fun stuff. Secrets are necessary to be sure, but it’s a little boring…

But if I want to do proper GitOps I need to manage secrets (and to document the process). The sooner I set it up the better.

In and outside cluster

Kubernetes has different solutions for secrets management. Of particular note is Sealed Secrets which creates files that are safe to commit to git and Kubernetes decrypts them in-cluster.

This is pretty great but has one big drawback: it can only manage secrets inside Kubernetes. It cannot be used to encrypt things like the talosconfig or the Proxmox password Terraform uses.

That’s why I’ll also use SOPS + Age, which allows us to encrypt whatever file we want. The idea is to use SOPS + Age to manage the bootstrapping secrets and let Sealed Secrets take over when ArgoCD is up. This way there’s only one private key I need to manage and the rest is available from the git repo.

SOPS + Age

First, we need to install sops and age locally (I found them in my package manager). Then we can generate our private key:

mkdir -p ~/.config/sops/age

age-keygen -o ~/.config/sops/age/keys.txt

You don’t want to lose this key, store it somewhere safe. I stored it in Bitwarden (although I’m migrating to Vaultwarden hosted in-cluster, which is a bit weird as keeping the key there risks a lock-out).

You then need a .sops.yaml that describes the files to encrypt/decrypt. For example, this is an entry for talosconfig.yaml:

creation_rules:

  - path_regex: infrastructure/talosconfig(\.encrypted)?.yaml$

    age: age1rrkgd5yza053qk9m8lp0ww39apdarz7w0rjyq85493g8l9gufgnq9cehzx

    encrypted_regex: '^(ca|crt|key)$'

(age contains the public key, safe to share.)

With encrypted_regex you can limit encryption to certain fields; if you leave it out you’ll encrypt the entire file.

Then to encrypt and decrypt talosconfig.yaml we generated in a previous post:

sops --encrypt talosconfig.yaml > talosconfig.encrypted.yaml

sops --decrypt talosconfig.encrypted.yaml > talosconfig.yaml

talosconfig.encrypted.yaml is safe to commit to git but the cleartext file talosconfig.yaml should be added to .gitignore.

Just commands

It won’t take long for me to forget these commands so I’ll add them to Just. These recipes will take care of the secrets we’ve handled so far and the cleartext files (talosconfig, kubeconfig, secrets.auto.tfvars, terraform.tfstate) should allow us to regain cluster control, or to bootstrap the cluster anew from the git repo and the sops key.

[doc("Decrypt required files committed to git")]

decrypt_required:

    just secrets decrypt_cluster_config

    just secrets decrypt_terraform_secrets

    just secrets decrypt_terraform_state



[doc("Encrypt talosconfig and kubeconfig")]

[working-directory('../infrastructure')]

encrypt_cluster_config:

    sops --encrypt kubeconfig.yaml > kubeconfig.encrypted.yaml

    sops --encrypt talosconfig.yaml > talosconfig.encrypted.yaml



[doc("Decrypt talosconfig and kubeconfig")]

[working-directory('../infrastructure')]

decrypt_cluster_config:

    sops --decrypt talosconfig.encrypted.yaml > talosconfig.yaml

    sops --decrypt kubeconfig.encrypted.yaml > kubeconfig.yaml

    chmod 600 talosconfig.yaml kubeconfig.yaml



[doc("Encrypt secrets.auto.tfvars")]

[working-directory('../infrastructure')]

encrypt_terraform_secrets:

    sops --encrypt secrets.auto.tfvars > secrets.auto.encrypted.tfvars



[doc("Decrypt secrets.auto.tfvars")]

[working-directory('../infrastructure')]

decrypt_terraform_secrets:

    sops --decrypt secrets.auto.encrypted.tfvars > secrets.auto.tfvars

    chmod 600 secrets.auto.tfvars



[doc("Encrypt terraform.tfstate")]

[working-directory('../infrastructure')]

encrypt_terraform_state:

    sops --encrypt --input-type json --output-type json terraform.tfstate > terraform.encrypted.tfstate



[doc("Decrypt terraform.tfstate")]

[working-directory('../infrastructure')]

decrypt_terraform_state:

    # If the encrypted state doesn't exist yet (fresh repo), skip silently.

    test -f terraform.encrypted.tfstate || exit 0

    sops --decrypt --input-type json --output-type json terraform.encrypted.tfstate > terraform.tfstate

    chmod 600 terraform.tfstate

I also added encryption to create_cluster_config bootstrap command, to make it harder for me to forget to add them to the repo:

[doc("Create talosconfig and kubeconfig")]

[working-directory('../infrastructure')]

create_cluster_config:

    terraform output -raw talosconfig > talosconfig.yaml

    terraform output -raw kubeconfig > kubeconfig.yaml

    just secrets::encrypt_cluster_config

And for the terraform state too (run with just tf::apply instead of a plain terraform apply):

[doc("terraform apply, then re-encrypt state (encrypts even on failure)")]

[working-directory('../infrastructure')]

apply *args:

    #!/usr/bin/env bash

    set -e

    trap 'just secrets::encrypt_terraform_state' EXIT

    terraform apply {{args}}



[doc("terraform destroy, then re-encrypt state (encrypts even on failure)")]

[working-directory('../infrastructure')]

destroy *args:

    #!/usr/bin/env bash

    set -e

    trap 'just secrets::encrypt_terraform_state' EXIT

    terraform destroy {{args}}

Sealed Secrets

Let’s move on to sealed secrets. There are more setup steps than with SOPS + Age but it’s not that bad.

Installation

I’ll install the sealed secrets controller using helm:

helm install sealed-secrets \

    --repo https://bitnami-labs.github.io/sealed-secrets \

    sealed-secrets \

    --version 2.16.2 \

    --namespace sealed-secrets \

    --create-namespace \

    --set fullnameOverride=sealed-secrets-controller

# Wait for it to deploy

kubectl rollout status deployment/sealed-secrets-controller -n sealed-secrets

To create secrets on the client we also need the kubeseal command. It wasn’t available on the Void Linux package manager, so let’s do the hard way:

set -x KUBESEAL_VERSION '0.36.1'

curl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v$KUBESEAL_VERSION/kubeseal-$KUBESEAL_VERSION-linux-amd64.tar.gz"

tar -xvzf kubeseal-$KUBESEAL_VERSION-linux-amd64.tar.gz kubeseal

sudo install -m 755 kubeseal /usr/local/bin/kubeseal

rm kubeseal-$KUBESEAL_VERSION-linux-amd64.tar.gz

Create a secret

To create a secret we can either use the cluster (needs an active connection) or offline via a certificate. I prefer the certificate simply because you need to pass fewer arguments (--cert vs --controller-name and --controller-namespace). Here’s how to fetch the certificate:

kubeseal --fetch-cert \

         --controller-name=sealed-secrets-controller \

         --controller-namespace=sealed-secrets \

    > infrastructure/sealed-secrets-cert.pem

(It’s a public key, safe to commit to git.)

And this is how we can use it to generate a secret my-secret with the two fields username and password:

kubectl create secret generic my-secret \

  --namespace some-namespace \

  --from-literal=username="user" \

  --from-literal=password="password1" \

  --dry-run=client -o yaml \

  | kubeseal --cert infrastructure/sealed-secrets-cert.pem \

             --format yaml \

  > gitops/apps/myapp/my-secret.yaml

(There might be other ways to do this. You can generate json files for example, but this works and I don’t care to do research.)

It will be stored in gitops/apps/myapp/my-secret.yaml that we can apply:

kubectl apply -f gitops/apps/myapp/my-secret.yaml

In the future when we get our GitOps setup up the process is the same except we don’t apply the secret; just create, commit, and push and it’ll get applied automatically. It may feel like a lot of effort, but it’s quite nice to work with in day-to-day operations.

The process to update a secret is exactly the same; update the file with new contents and reapply.

View a secret

Check that the secret has been applied:

kubectl get secret my-secret -n some-namespace -o yaml

The data fields username and password shown above will be base64 encoded. Here’s how to print out the password in cleartext:

kubectl get secret my-secret -n some-namespace -o jsonpath='{.data.password}' | base64 -d

Or you can view the secrets in a dashboard such as Headlamp, which is arguably easier.

Surviving a cluster reset

There’s one gotcha to sealed secrets: when the controller is installed it will generate a new public/private key pair so all existing sealed secrets are invalidated. We’d have to reseal all secrets after we reset the cluster, which is highly annoying.

We can circumvent this by exporting the private key, encrypt it with SOPS + Age, and store it in git. Then during the bootstrap process we can import the private key to the controller, allowing it to reuse all existing sealed secrets.

First export the key and encrypt it so we can keep it in git (gitignore sealed-secrets-key.yaml):

kubectl get secret -n sealed-secrets \

  -l sealedsecrets.bitnami.com/sealed-secrets-key=active -o yaml \

     > sealed-secrets-key.yaml

sops --encrypt sealed-secrets-key.yaml > sealed-secrets-key.encrypted.yaml

This needs a .sops.yaml rule:

creation_rules:

  - path_regex: infrastructure/sealed-secrets-key(\.encrypted)?.yaml$

    age: age1rrkgd5yza053qk9m8lp0ww39apdarz7w0rjyq85493g8l9gufgnq9cehzx

Then to import it we simply apply it:

kubectl apply -f sealed-secrets-key.yaml

Just commands

We’ve added a few steps to the bootstrap process to setup the sealed secrets controller:

[doc("Bootstrap everything from zero")]

full:

    just bootstrap::cluster

    just bootstrap::cilium

    just secrets::restore_sealed_secrets_private_key

    just bootstrap::sealed_secrets

[doc("Restore sealed-secrets-key.yaml")]

[working-directory('../infrastructure')]

restore_sealed_secrets_private_key:

    # If the encrypted private key doesn't exist, skip the whole recipe.

    test -f sealed-secrets-key.encrypted.yaml || exit 0

    just secrets::decrypt_sealed_secrets_private_key

    # Don't exit if namespace already exists.

    kubectl create namespace sealed-secrets || true

    # Restore the private key.

    kubectl apply -f sealed-secrets-key.yaml

    rm sealed-secrets-key.yaml

    # We may need to restart the controller, but it may not exist, which is fine.

    kubectl rollout restart deployment/sealed-secrets-controller -n sealed-secrets || true

I’ve tried to safeguard the recipe to not crash if we haven’t created a key or bootstrapped the controller yet.

[doc("Install sealed secrets controller")]

sealed_secrets:

    helm install sealed-secrets \

        --repo https://bitnami-labs.github.io/sealed-secrets \

        sealed-secrets \

        --version 2.16.2 \

        --namespace sealed-secrets \

        --create-namespace \

        --set fullnameOverride=sealed-secrets-controller

    # Wait for it to deploy

    kubectl rollout status deployment/sealed-secrets-controller -n sealed-secrets

And some extra management recipes:

[doc("Fetch sealed-secrets-cert.pem, necessary to encrypt secrets offline")]

[working-directory('../infrastructure')]

fetch_sealed_secrets_cert:

    kubeseal --fetch-cert \

             --controller-name=sealed-secrets-controller \

             --controller-namespace=sealed-secrets \

        > sealed-secrets-cert.pem



[doc("Fetch sealed-secrets-key.yaml")]

[working-directory('../infrastructure')]

fetch_sealed_secrets_private_key:

    kubectl get secret -n sealed-secrets -l sealedsecrets.bitnami.com/sealed-secrets-key=active -o yaml > sealed-secrets-key.yaml

    just secrets::encrypt_sealed_secrets_private_key

    rm sealed-secrets-key.yaml



[doc("Encrypt sealed-secrets-key.yaml")]

[working-directory('../infrastructure')]

encrypt_sealed_secrets_private_key:

    sops --encrypt sealed-secrets-key.yaml > sealed-secrets-key.encrypted.yaml



[doc("Decrypt sealed-secrets-key.yaml")]

[working-directory('../infrastructure')]

decrypt_sealed_secrets_private_key:

    sops --decrypt sealed-secrets-key.encrypted.yaml > sealed-secrets-key.yaml

With this we’re prepared to setup GitOps with ArgoCD in the next post.

This is part 4 of the Modern Kubernetes homelab series.
This series is ongoing.
View all series
Published: in f60b1c6