GitOps with ArgoCD ★

Fixed IP

You can use the port forward method above to access ArgoCD but having a fixed IP is nicer, especially if you want to use the argocd CLI (I only used it when trying to get it all working).

I use a separate service for the IP, which allows me to assign an IP outside of the Cilium load balance pool we setup previously (10.1.4.101–10.1.4.255). It looks like this:

apiVersion: v1
kind: Service
metadata:
  name: argocd-server-ip
  namespace: argocd
spec:
  type: ClusterIP
  externalIPs:
    - 10.1.4.51
  selector:
    app.kubernetes.io/name: argocd-server
  ports:
    - name: http
      port: 80
      targetPort: 8080

It simply routes 10.1.4.51:80 to the IP of argocd-server and port 8080. We need to apply it:

kubectl apply -f gitops/bootstrap/argocd.yaml

And you should see it assigned and able to visit it on the web:

$ kubectl get svc -n argocd argocd-server-ip
NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
argocd-server-ip   ClusterIP   10.105.209.116   10.1.4.51     80/TCP    3d18h

Repository

I use a self-hosted Forgejo instance (hosted in a Proxmox LXC) but you can use whatever you want. You need an API token with read permissions.

ArgoCD documents a declarative setup that we’ll use. For repositories we need to create a secret:

kubectl create secret generic home-ops-repo \
  --namespace argocd \
  --from-literal=type=git \
  --from-literal=url=<repo-url> \
  --from-literal=username=<username> \
  --from-literal=password=<token> \
  --dry-run=client -o yaml \
  | kubeseal --cert infrastructure/sealed-secrets-cert.pem -o yaml \
  > gitops/bootstrap/argocd-repo-secret.yaml

We also need to add the repository label to the generated file so it looks something like this:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: home-ops-repo
  namespace: argocd
spec:
  encryptedData:
    password: ...
    type: ...
    url: ...
    username: ...
  template:
    metadata:
      name: home-ops-repo
      namespace: argocd
      labels:
        argocd.argoproj.io/secret-type: repository

Apply it:

kubectl apply -f gitops/bootstrap/argocd-repo-secret.yaml

And the repo should show up in the UI or argocd repo list.

Initial admin password

Instead of letting ArgoCD generate a random password we can set the initial password explicitly. It’s possible to do it either via the Helm chart or the declarative way, which I’ll continue with.

After generating it with kubeseal the secret should look like this:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: argocd-secret
  namespace: argocd
spec:
  encryptedData:
    admin.password: ...
    admin.passwordMtime: ...
    server.secretkey: ...
  template:
    metadata:
      labels:
        app.kubernetes.io/name: argocd-secret
        app.kubernetes.io/part-of: argocd
      name: argocd-secret
      namespace: argocd

With three keys:

• password is a bcrypt hash:

  argocd account bcrypt --password 'supersecret'
  

• passwordMtime is the plaintext modification time:

  date -u +"%Y-%m-%dT%H:%M:%SZ"
  

• secretkey is a random value:

  openssl rand -base64 48
  

  ArgoCD uses server.secretkey to sign session and API tokens, and it will regenerate the key if restarted. I kept getting logged out when I was rebuilding things all over the place and it was annoying so I added this to make it stable.

App-of-apps

root.yaml will be a master “app-of-apps” as it will be an Application that spawns other applications. It looks like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: all-apps
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-100"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://git.hietala.xyz/tree/home-ops.git
    targetRevision: master
    path: gitops/bootstrap
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true

(You’re going to see lots of yaml files like this, better get used to it.)

The important bits are

• The name all-apps in the argocd namespace

• It should be synced before everyone else (sync-wave: -100)

• Should load everything under gitops/bootstrap from my home-ops repo

• I’ll be using pruning and self-healing to tell ArgoCD to keep everything in sync.

This is the entrypoint and it’s the last manual apply command we’ll use, as it’ll load everything else:

kubectl apply -f gitops/root.yaml

Application sets

Inside the bootstrap/ folder lives, among other things, bootstrap/appset.yaml which is responsible for turning subfolders under apps/ into applications and loading them.

This is what I use and it’s an adaption of the code from the aforementioned blog post:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: appset
  namespace: argocd
spec:
  # Use Go's engine so we can use basename/path
  goTemplate: true
  # The engine silently fails on missing keys by default
  goTemplateOptions: ["missingkey=error"]
  # Generate from all subfolders from my repo under apps/
  generators:
    - git:
        repoURL: https://git.hietala.xyz/tree/home-ops.git
        revision: master
        directories:
          - path: gitops/apps/*
  template:
    # The Application name will be folder + "-app"
    # Creative, I know.
    metadata:
      name: "{{.path.basename}}-app"
    spec:
      project: default
      # Describes the manifests of the application,
      # which is everything under apps/<app-folder>/
      source:
        repoURL: https://git.hietala.xyz/tree/home-ops.git
        targetRevision: master
        path: "{{.path.path}}"
      # Give the application its own namespace
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{.path.basename}}"
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
          # Needed for larger manifests
          - ServerSideApply=true
          # ArgoCD does a dry run to check for resource kinds
          # it doesn't yet recognize. This breaks some things
          # so turn it off.
          - SkipDryRunOnMissingResource=true
        automated:
          prune: true
          selfHeal: true

I hope the comments make it clear how the above works. The end result is that everything under apps/ will be loaded and organized under its own application.

Manifests under bootstrap/

Even though we manually apply some manifests during the bootstrap sequence, they can still be managed by ArgoCD after the fact. This is solved by simply placing the relevant files in bootstrap/, and they’ll be managed going forward.

These are what we’ve collected so far:

1. Cilium

   We installed Cilium using a helm install command. Here’s a (hopefully) corresponding example of a manifest that ArgoCD can manage:

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: cilium
     namespace: argocd
     annotations:
       argocd.argoproj.io/sync-wave: "-1"
     finalizers:
       - resources-finalizer.argocd.argoproj.io
   spec:
     project: default
     source:
       repoURL: https://helm.cilium.io
       chart: cilium
       targetRevision: 1.19.2
       helm:
         values: |
           kubeProxyReplacement: true
           k8sServiceHost: 10.1.4.10
           k8sServicePort: 6443
           l2announcements:
             enabled: true
           externalIPs:
             enabled: true
           gatewayAPI:
             enabled: true
           ipam:
             mode: kubernetes
           operator:
             replicas: 1
           securityContext:
             privileged: true
     destination:
       server: https://kubernetes.default.svc
       namespace: kube-system
     syncPolicy:
       syncOptions:
         - CreateNamespace=false
         - ServerSideApply=true
       automated:
         prune: true
         selfHeal: true
   

   The duplication here is unfortunate but I couldn’t find a way around it.

   At least we can reuse cilium_config.yaml with the load balancer IP specifications that we created during cluster setup. Just plop it under bootstrap/ and we can move on.

2. Gateway API

   Before installing Cilium we also applied the Gateway CRDs using this command:

   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml
   

   This is the corresponding YAML manifest (smaller as we don’t pass dozens of options):

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: gateway-api
     namespace: argocd
     annotations:
       argocd.argoproj.io/sync-wave: "-2"
     finalizers:
       - resources-finalizer.argocd.argoproj.io
   spec:
     project: default
     source:
       repoURL: https://github.com/kubernetes-sigs/gateway-api.git
       targetRevision: v1.2.1
       path: config/crd/standard
     destination:
       server: https://kubernetes.default.svc
       namespace: kube-system
     syncPolicy:
       automated:
         prune: true
         selfHeal: true
   

   Note that I specify sync-wave -2 for the Gateway and -1 for Cilium because the Gateway should sync before Cilium.

3. Sealed Secrets

   We continue in the same manner of converting Helm installations to manifests:

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: sealed-secrets
     namespace: argocd
     annotations:
       argocd.argoproj.io/sync-wave: "-1"
     finalizers:
       - resources-finalizer.argocd.argoproj.io
   spec:
     project: default
     source:
       repoURL: https://bitnami-labs.github.io/sealed-secrets
       chart: sealed-secrets
       targetRevision: 2.16.2
       helm:
         values: |
           fullnameOverride: sealed-secrets-controller
     destination:
       server: https://kubernetes.default.svc
       namespace: sealed-secrets
     syncPolicy:
       syncOptions:
         - CreateNamespace=true
       automated:
         prune: true
         selfHeal: true
   

4. ArgoCD

   In addition to the secrets used for the declarative setup (repository, admin password) ArgoCD can also manage itself:

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: argocd
     namespace: argocd
     finalizers:
       - resources-finalizer.argocd.argoproj.io
   spec:
     project: default
     source:
       repoURL: https://argoproj.github.io/argo-helm
       chart: argo-cd
       targetRevision: 10.1.4
       helm:
         values: |
           configs:
             secret:
               createSecret: false
     destination:
       server: https://kubernetes.default.svc
       namespace: argocd
     syncPolicy:
       syncOptions:
         - CreateNamespace=false
       automated:
         prune: true
         selfHeal: true
   ---
   # Static IP as shown previously, `---` separates different kinds
   
   

Updated bootstrap sequence

This is the updated bootstrap recipe that I manage with Just:

[doc("Bootstrap everything from zero")]
full:
    just bootstrap::cluster
    just bootstrap::cilium
    just secrets::restore_sealed_secrets_private_key
    just bootstrap::sealed_secrets
    just bootstrap::argocd

[doc("Bootstrap ArgoCD")]
[working-directory('../gitops')]
argocd:
    helm install argocd \
        --repo https://argoproj.github.io/argo-helm \
        argo-cd \
        --namespace argocd \
        --create-namespace \
        --version 10.1.4 \
        --set configs.secret.createSecret=false
    # We don't let ArgoCD generate the secret, we supply one:
    kubectl apply -f bootstrap/argocd-secret.yaml
    # Needed to be able to sync from repo.
    kubectl apply -f bootstrap/argocd-repo-secret.yaml
    # Wait for ArgoCD to deploy.
    kubectl rollout status deployment/argocd-server -n argocd --timeout=300s
    # Technically not needed as it'll get loaded on sync, this just speeds it up a little.
    kubectl apply -f bootstrap/argocd.yaml --force-conflicts --server-side
    # Sync everything else from the repo.
    kubectl apply -f root.yaml

The secret

Renovate will pick up RENOVATE_TOKEN in secrets, generated like so:

kubectl create secret generic renovate-token -n renovate \
  --from-literal=RENOVATE_TOKEN='<token>' \
  --dry-run=client -o yaml \
| kubeseal --cert infrastructure/sealed-secrets-cert.pem -o yaml \
  > gitops/apps/renovate/renovate-secret.yaml

We’ll later add the name renovate-token to the main manifest.

The configuration file

Renovate has a bunch of configurations you can add. I’m good with the defaults except to modify the ArgoCD settings to support the custom directory structure I’m using:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "argocd": {
    "managerFilePatterns": ["/gitops/(apps|bootstrap)/.+\\.yaml$/"]
  },
  "kubernetes": {
    "managerFilePatterns": ["/gitops/(apps|bootstrap)/.+\\.yaml$/"]
  }
}

This activates the argocd and kubernetes managers, where argocd watches ArgoCD specific manifests (Application/ApplicationSet) and kubernetes watches plain manifests (such as container images in Deployment).

The manifest

The Application manifest is similar to the ones we’ve seen before:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: renovate
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://docs.renovatebot.com/helm-charts
    chart: renovate
    targetRevision: 46.236.3
    helm:
      values: |
        existingSecret: renovate-token
        cronjob:
          schedule: "0 1 * * *"
        renovate:
          config: |
            {
              "platform": "forgejo",
              "endpoint": "https://git.hietala.xyz",
              "repositories": ["tree/home-ops"],
              "gitAuthor": "Renovate Bot <renovate@git.hietala.xyz>",
              "automerge": false
            }
  destination:
    server: https://kubernetes.default.svc
    namespace: renovate
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true

Note that we configure the secret (renovate-token), how often it should run, and the repository information.

Commit and push and it should be ready to go. ArgoCD has a really good UI where you can see what’s happening, if you made a mistake somewhere, and also trigger the cron job manually.

Or you can use kubectl, for example:

# See the health status of all applications
kubectl get applications -A
# Trigger the cron job
kubectl create job -n renovate --from=cronjob/renovate renovate-manual
# Watch the logs
kubectl logs -n renovate -l job-name=renovate-manual -f

With this we’ve got our first example of managing a Kubernetes service via ArgoCD. We’ll continue to see more examples of this as the series continues.