Diffs & Manifests in Github PRs with ArgoCD

ArgoCD is a lot of things: a way to manage Kubernetes manifests with a GitOps philosophy, an observability portal, a tool in the larger Argo* ecosystem. But for our developers? The features they love the most are diffs and manifests. Diffs help developers have confidence in their changes. Manifests allow us (the Platform Engineering team) run linters and checks as early in the development lifecycle together. Together, diffs and manifests help encourage developers to build and release faster.

In contrast, the actual deployment (especially with autosync enabled) is magically boring!

We're going to see how ArgoCD and off-the-shelf tools makes adding diffs and manifest checks to your Github PRs using Github Actions easy and performant.

So what do we want to do?

Developer pushes a change to the config repo. A config repo that contains code (Helm, jsonnet, kustomize, raw manifests) that ArgoCD uses to deploy one-to-many Applications.
- This repository may - or may not! - contain the actual application code.
Developer opens a PR and triggers Github Actions.
Github Actions uses the argocd app manifests (docs) command to generate manifests.
Github Actions runs validation on the manifests. Some tools we could use are:
- yannh/kubeconform to check for errors in the manifests. (This has eliminated an entire class of k8s bugs!)
- stackrox/kube-linter to check for best practices
- Run rego checks so manifests are blocked at the PR stage and not on apply.
Github Actions generates diffs using the argocd app diff (doc) command to generate diffs and post them to the PR comment.
Developer reviews the diffs, merges changes, and ArgoCD does ArgoCD things!

Authenticating to ArgoCD

We will use an user with an API key and minimal permissions to authenticate to ArgoCD from inside Github Actions. This can be done through the standard Helm chart configuration:

server:
  config:
    ...
    accounts.GithubActions: apiKey
    ...

  rbacConfig:
    policy.csv: |
      ...
      g, GithubActions, role:readonly
      ...

Tooling Docker Image

A Job in a Github Actions Workflow can only run in a single container. We want to minimize the number of jobs per Application so we can parallelize the process. To this this, we want to make a relatively simple tooling container that contains all the tools we want to use.

FROM golang:1.21 as go_builder

RUN go install github.com/yannh/kubeconform/cmd/[email protected]

FROM ubuntu:24.04

# Update ca-certs
RUN apt-get update && apt-get install --no-install-recommends ca-certificates -y \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

# Create working folder
RUN mkdir /argocd_diff
WORKDIR /argocd_diff

# Install tools
COPY --from=quay.io/argoproj/argocd:v2.9.5  /usr/local/bin/argocd /usr/local/bin
COPY --from=cloudposse/github-commenter:0.19.0 /usr/bin/github-commenter /usr/local/bin
COPY --from=hairyhenderson/gomplate:stable /gomplate /bin/gomplate
COPY --from=go_builder /go/bin/kubeconform /usr/local/bin

COPY *.md.tmpl .

Example

Figuring out what's changed

Manifests and manifest-adjacent configuration (e.g. Helm) may not be the only thing in your repos. Additionally, if you manage multiple different applications in your config repository (e.g. in a infrastructure repository), you want to minimize the load on your ArgoCD repo server by only generating diffs/manifests when the code is modified.

First, we create a workflow that only runs when manifests change.

name: k8s-manifests

on:
  pull_request:
    paths:
    - kubernetes/applications/**

Next, we use a job to find the changed files (using git diff) and extracting the application name from the file path. (This is made easier by using ApplicationSets and the git file generator to standardize the Application name.)

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      changed-folders: ${{steps.output-step.outputs.folders}}
    steps:
      - uses: actions/checkout@v4
      - name: Setup main branch locally without switching current branch
        run: git fetch origin main:main
      - name: 'Setup jq'
        uses: dcarbone/install-jq-action@v2
      # Credit to https://www.kenmuse.com/blog/dynamic-build-matrices-in-github-actions/
      - name: 'Setup ripgrep'
        run: sudo apt-get install -y ripgrep
      - id: output-step
        name: Generate list of changed folders
        # Use ripgrep to extract application name from path aka kubernetes/applications/[group]/[application]/...
        run: |
          TARGETS=$(git diff --name-only main -- kubernetes/applications | rg 'kubernetes/applications/[\w]*/([\w]*)/.*' -or '$1' | uniq | jq -cRn '[inputs]')
          echo "folders=$(jq -cn --argjson environments "$TARGETS" '{folder: $environments}')" >> "$GITHUB_OUTPUT"

This creates an output that looks something like:

{
  folder: [
    'application-name-1',
    'application-name-2'
  ]
}

We can use this output in a following job to run the diff steps in parallel.

  run-matrix:
    needs: changes
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    container:
      image: cluelesshamster86/argocd_tools:main
    strategy:
      matrix: ${{ fromJson(needs.changes.outputs.changed-folders) }}
    ... manifest generation stuff ...

Actually doing the thing

Now with the preparations out of the way, we can get down to business.

argocd app manifests --grpc-web --server $ARGOCD_URL --auth-token $ARGOCD_TOKEN $(basename ${{matrix.folder}}) --revision $GITHUB_SHA >> manifests.yaml

Breaking down the arguments:

--grpc-web instructs the CLI to use the grpc/web protocol.
--server $ARGOCD_URL --auth-token $ARGOCD_TOKEN authenticates to the ArgoCD server using JWT.
--revision $GITHUB_SHA instructs ArgoCD to generate manifests using the git revision.

We can then run the manifests through kubeonform. Remember to -ignore-missing-schemas so custom resources won't cause failures!

kubeconform -ignore-missing-schemas manifests.yaml

Generating the diff is slightly more complicated because the ArgoCD CLI returns an exit code of 1 if diffs are found and 2 for general errors (e.g. error generating manifests). We can capture the exit code and use it to decide whether it's OK (diffs found, post to PR) or if we should fail the workflow.

set +e
argocd app diff --grpc-web --server $ARGOCD_URL --auth-token $ARGOCD_TOKEN $(basename ${{matrix.folder}}) --refresh --revision $GITHUB_SHA >> k8s.diff
exitcode="$?"
echo "exitcode=$exitcode" >> $GITHUB_OUTPUT
exit "$exitcode"

We pass the additional --refresh argument as well to ensure ArgoCD has the latest-and-greatest configuration from the Kubernetes repository.

We can use hairyhenderson/gomplate to jam the diff into a template simple Markdown template.

gomplate -f /argocd_diff/diff.md.tmpl -o diff.md -d diff=k8s.diff?type=text/plain

**ArgoCD Diff Report - {{ .Env.APPLICAITON_PATH }}**

*Application*: {{ filepath.Base .Env.APPLICAITON_PATH }}

```diff
{{ds "diff"}}
```

Finally, we can use cloudposse/github-commenter to post the comment. By specifying a -edit-comment-regex contained in the Markdown template, we update the same comment whenever additional changes are made.

cat diff.md | github-commenter \
  -token $GITHUB_TOKEN \
  -owner $GITHUB_REPOSITORY_OWNER \
  -repo $(basename $GITHUB_REPOSITORY) \
  -type pr \
  -number ${{github.event.pull_request.number}} \
  -edit-comment-regex "ArgoCD Diff Report - ${{matrix.folder}}"