CI/CD & Release Engineering

This page walks through the full journey of a code change — from the moment a developer opens a pull request, all the way to production. There are three main systems involved:

• GitHub Actions validates code on every pull request.
• Concourse builds release images, validates deploy manifests, and deploys environments.
• Cepler controls which environments get updated and in what order.

The design philosophy is simple: every step must pass before the next one starts, and there are human checkpoints at the most important moments. Nothing reaches production by accident.

High-Level Overview

Now let's walk through each step in detail.

---

Step 1: Pull Request Checks (GitHub Actions)

When a developer opens a pull request against main, GitHub Actions kicks off a suite of checks that all run in parallel. Every single one must pass before the PR can be merged — there are no exceptions.

What gets checked

Workflow	What it does	Why it matters
nextest	Runs all Rust unit and integration tests via nix run .#nextest	Catches logic bugs and regressions in the backend
bats	Spins up the full application stack and runs BATS end-to-end tests against it	Verifies that the whole system works together, not just individual pieces
cypress	Runs Cypress browser tests against the admin panel	Makes sure the UI actually works; also generates screenshots for regulatory manuals
check-code-apps	Lints, type-checks, and builds the frontend apps	Catches TypeScript errors, lint violations, and broken builds in the frontend
flake-check	Runs nix flake check to validate the Nix flake	Ensures the build system itself is healthy
kustomize	Renders the Kubernetes deploy tree and validates it with kubeconform	Catches invalid Kubernetes manifests before release
pnpm-audit	Audits npm dependencies for known vulnerabilities	Blocks PRs that introduce dependencies with high-severity CVEs
validate-report-columns	Parses the dbt project with dbt parse (dbt-duckdb, using scratch DuckDB locations) and validates report definitions and generated event models against the manifest	Validates that the Rust/dbt reporting pipeline still works with schema changes
cocogitto	Checks that commit messages follow the conventional commits format	Needed because the version number and changelog are generated automatically from commit messages
spelling	Runs the typos tool to catch common misspellings	Simple but catches embarrassing typos in code and docs
lana-bank-docs	Builds the full documentation site (API docs, versioned docs, screenshot validation)	Catches broken doc builds, missing API descriptions, and invalid doc site configuration

How Nix caching makes this fast

Compiling the Rust codebase from scratch takes a long time. To avoid doing that on every single PR, all the GitHub Actions workflows pull pre-built binaries from a shared Cachix binary cache called galoymoney.

Here's the pattern you'll see in every workflow file:

- uses: DeterminateSystems/nix-installer-action@v16
- uses: cachix/cachix-action@v15
  with:
    name: galoymoney
    authToken: ${{ secrets.CACHIX_AUTH_TOKEN }}
    skipPush: true

The skipPush: true part is key — GitHub Actions only reads from the cache, it never writes to it.

Most workflows also reclaim 10-20 GB of disk space at the start by removing pre-installed software (Docker images, Android SDK, etc.) that GitHub runners ship with. The large Rust compilations need that breathing room.

What happens when a check fails

If any check fails, the PR is blocked from merging. The developer fixes the issue, pushes again, and the checks re-run. There's no way to bypass a failing check.

---

Step 2: Building a Release (Concourse, lana-bank repo)

Once a PR is merged into main, the Concourse release pipeline takes over. This pipeline lives in the ci/release/ directory of the lana-bank repo and is written using YTT templates.

The pipeline has a clear dependency chain:

2a. Re-running the tests on main

You might wonder: we already ran tests in GitHub Actions on the PR, so why run them again? Because the PR was tested against a potentially stale version of main. Between the time the PR was opened and the time it was merged, other PRs may have landed. Running tests again on the actual merged commit catches integration issues that only appear when multiple changes combine.

Three jobs run in parallel:

• test-integration runs cargo nextest — the same Rust test suite from the PR checks.
• test-bats runs the BATS end-to-end tests (with up to 2 attempts, since E2E tests can be flaky).
• flake-check validates the Nix flake.

All three must pass before anything gets built.

2b. Building the Release Candidate (build-rc)

Once tests pass, the pipeline builds a release candidate (RC). The idea behind RCs is that you can build and test multiple candidates before committing to a final release. Here's what happens:

1. Figure out the version number. A Nix script called next-version uses cocogitto to scan the conventional commit messages since the last release and determine the next semantic version. For example, if the last release was 0.41.0 and there's been a feat: commit, the next version becomes 0.42.0-rc.1. If another RC was already built for this version, it increments to rc.2, rc.3, and so on.

2. Set the release version. The pipeline exports RELEASE_BUILD_VERSION=0.42.0-rc.1 before building the Nix image. The Rust binary embeds this version in its build metadata and exposes it through the admin GraphQL buildInfo query.

3. Compile the Rust binary and admin panel assets. nix build --impure .#lana-bank-image builds the statically-linked lana-cli binary and the Vite admin panel assets, then packages both into the lana-bank image.

4. Build one Docker image and push it to Google Artifact Registry (gcr.io/galoyorg):

   • lana-bank — the server image built by Nix, containing the statically-linked binary, the dbt-duckdb runtime, the report project, and packaged admin panel assets

5. Tag the image. The image gets both an edge tag (meaning "latest RC") and a version-specific tag like 0.42.0-rc.1.

The lana-bank image built here contains a fast dev-profile binary so the staging rollout stays quick. In parallel, a sidecar job build-rc-release builds .#lana-bank-image-release — the same image but with the true release-profile (opt-level 3) binary — and pushes it to gcr.io/galoyorg/lana-bank-release. It's a separate job precisely so the slower optimized compile never delays the staging rollout; nothing deploys from that repository yet.

2c. Publishing an immutable deploy bundle

Both bump-image-in-deploy-rc and bump-image-in-deploy take their source commit, image digest, and version from the corresponding successful image build. They:

1. Check out the committed image source, including deploy/ and tf/.
2. Pin the Lana image digest in deployment Kustomizations. Local-kind newTag entries and other images retain their source values.
3. Write deploy/VERSION, deploy/is-release, and deploy/METADATA.
4. Create one chore(deploy): bundle lana-bank <version> commit on that source and publish only the Git tag deploy/<version> (for example, deploy/0.62.0-rc.1220 or deploy/0.62.0). Main is not changed.

METADATA.commit_ref is the full image-source SHA. The tagged bundle commit is a different SHA: it includes the generated pins and metadata, and is the ref that galoy-deployments vendors. All existing metadata fields are retained. deploy/METADATA on source main is no longer a release-discovery mechanism; read the tagged bundle or the metadata vendored into galoy-deployments.

The bundle commit has deterministic content, parent, identity, and dates. Re-publishing identical inputs therefore pushes the same tag to the same SHA, including when a successful push loses its acknowledgement. A different image or source for an existing version is rejected by the non-forcing tag push. Publication retries transport failures three times. It never rebases and needs no comparison with current main: an older final-release source keeps its own matching templates even after main advances. Template and tf/ edits still trigger normal image builds; tag publication needs no [ci skip] marker.

Successful publication registers the exact bundle SHA in Concourse's new deploy-bundle resource. Automatic checks are disabled (check_every: never); this resource is fed by publication puts, rather than branch/path or tag-date polling. This also handles a final release whose source timestamp predates a newer RC. Do not manually check this resource to discover arbitrary tags; re-run the appropriate publication job to register an existing bundle.

2d. Testflight: a throwaway deployment

The RC deploy bundle is validated with a testflight on the staging GKE cluster. The testflight uses the same Crossplane compositions and workload Kustomize paths as GitOps environments, but with temporary instance names.

Here's what happens during a testflight:

1. Prepare a testflight bundle. The pipeline copies deploy/, generates LanaInstanceDeps claims, and renders plain ArgoCD Application resources for the temporary instances with deploy/bin/generate-lana-applications.py.
2. Crossplane creates prerequisites. The LanaInstanceDeps claim creates runtime Kubernetes resources, runtime ConfigMaps, storage/IAM/KMS resources, DNS records, Cloud SQL/Postgres resources, and Keycloak realms. The claim's bundled OpenTofu workspace creates Keycloak realms and clients through provider-opentofu.
3. ArgoCD applies workloads. The pipeline applies the generated ArgoCD Application resources and waits for the lana-bank-server Deployment to roll out.
4. Smoketests run against the deployed services.
5. Cleanup runs regardless of result. The pipeline deletes the ArgoCD Application resources and Crossplane claims, which tears down the temporary resources.

If the smoketest fails, the pipeline stops and someone needs to investigate what went wrong.

2e. Opening the Promote-RC PR (open-promote-rc-pr)

After the RC image is built and testflight has a candidate to validate, the pipeline automatically opens a pull request back in the lana-bank repo. This PR does a few things:

• It generates a CHANGELOG entry using git-cliff, which reads the conventional commit messages and groups them into categories (features, bug fixes, etc.).
• It regenerates API documentation and event schemas, and creates a versioned snapshot of the docs site.
• It pushes everything to a branch called bot-promote-rc and opens a draft PR labeled promote-rc.

This PR is the human gate in the pipeline. An engineer reviews the changelog to make sure it looks right, checks that the RC looks good in any ad-hoc testing, and then merges the PR when they're ready to cut a release. Nothing happens automatically from here — the release only proceeds when a human says "go."

There's also a safety check: the promote-rc-file-check GitHub Action verifies that this PR only touches CHANGELOG.md and docs-site/** files. If the bot accidentally included other changes, the check fails and blocks the merge.

2f. Cutting the Final Release (release)

When someone merges the promote-rc PR, the release job triggers. It does four things:

1. Builds the final Docker image. This is the same image as the RC, but now tagged with the clean version number (e.g., 0.42.0) and also latest. The lana-bank image is built by Nix (dockerTools.buildImage), producing a deterministic image.
2. Creates a GitHub Release. This includes the lana-cli binary as a downloadable artifact and the changelog as the release notes. The release is tagged with the version number.
3. Updates the version counter. The pipeline stores the current version in a dedicated git branch called version (just a text file with the version number in it). This gets bumped so the next RC starts from the right base.
4. Publishes the final deploy bundle. bump-image-in-deploy tags a bundle from the final image's exact source, with its digest and release=true metadata. Newer source templates do not block publication.

How version numbers work

Versions follow Semantic Versioning and are derived automatically from conventional commit messages using cocogitto:

• feat: commits produce a minor bump (e.g., 0.41.0 -> 0.42.0)
• fix: commits produce a patch bump (e.g., 0.42.0 -> 0.42.1)
• feat!: or BREAKING CHANGE produce a major bump (e.g., 0.42.0 -> 1.0.0)

This is why the cocogitto GitHub Action enforces conventional commit format on every PR — if commit messages don't follow the convention, the version can't be computed automatically.

The current version is stored in a git branch called version as a plain text file. It's managed by the Concourse semver resource.

---

Step 3: Vendoring the Deploy Bundle into galoy-deployments

bump-in-deployments considers every successfully published bundle for galoy-deployments. Publication does not by itself change an environment. The selection policy compares the candidate with vendor/lana-bank/deploy/METADATA on the current deployments branch:

• RCs and final releases are eligible, preserving the existing staging flow.
• Selection advances by semantic version: rc.10 follows rc.2, a final follows its own RCs, and an older final does not replace a newer minor's RC.
• A version advance must also preserve source ancestry: the candidate's METADATA.commit_ref must contain the currently selected source. A final rebuilt from an older RC must not replace newer source just because its version sorts higher. It could be missing migrations already applied there.
• An RC that does not contain the selected source is skipped. A final in that situation fails and uses the existing failure notification, because skipping it would also prevent that release from reaching the downstream release gates.
• Older and already-selected bundles are successful skips, with no deployment push or Zenduty failure alert. Conflicting metadata for an already-selected version is an error.

Selection, vendoring, and the non-rebasing deployment push run together. Each of up to five attempts fetches the current deployment branch and re-evaluates the policy, even if Concourse supplied a cached checkout. A concurrent upgrade therefore causes an older candidate to skip, or a higher-version final with older source to fail, on retry. Missing source history is fetched into a temporary bundle checkout; shallow history is completed before comparing ancestry. Legacy short source SHAs must resolve unambiguously. Fetch errors, unknown/ambiguous sources, and comparison errors fail closed. The cached input and the candidate bundle's HEAD are preserved. A lost acknowledgement resolves to a successful skip. Vendoring pins the bundle commit SHA, not the tag name or METADATA.commit_ref. Frozen image overlays remain pinned. The existing vendir credential authenticates this guarded push from a temporary checkout; no key file or checkout is emitted as a task output.

galoy-deployments then uses Vendir to copy the exact deployment inputs it needs from that lana-bank commit:

• deploy/ — Kustomize base, components, overlays, and deploy metadata
• tf/modules/keycloak-realms — OpenTofu module used by the LanaInstanceDeps Crossplane composition for Keycloak realms and clients
• tf/honeycomb — observability configuration used by the monitoring deployment

Local-only OpenTofu modules, such as tf/local/modules/postgres, stay in lana-bank and are not vendored into deployment environments.

The config looks like this:

# vendir.yml (simplified)
directories:
  - path: modules/lana-bank/vendor
    contents:
      - path: lana-bank/deploy
        git:
          url: git@github.com:GaloyMoney/lana-bank.git
          ref: <lana-bank-git-ref>
        includePaths:
          - deploy/*
          - deploy/**/*
        newRootPath: deploy
      - path: keycloak-realms
        git:
          url: git@github.com:GaloyMoney/lana-bank.git
          ref: <lana-bank-git-ref>
        includePaths:
          - tf/modules/keycloak-realms/**/*
        newRootPath: tf/modules/keycloak-realms

This keeps galoy-deployments self-contained at deploy time while preserving a direct trace back to the lana-bank source commit.

Bundle verification and recovery

1. Record the currently vendored Lana ref, version, image digest, and source SHA from galoy-deployments.
2. Inspect deploy/<version>: its parent must equal the full METADATA.commit_ref, its image digest must equal the image build output, and its deploy/ and vendored tf/ must come from that source (except the generated pins and metadata). Confirm the successful put registered that bundle SHA in the deployment resource.
3. Compare its semantic version and source ancestry with the recorded deployment. Resolve missing or malformed existing metadata; the selector fails closed. Verify the selection's vendored SHA, metadata, and image pins.
4. Verify the deployments pipeline and environment health separately. Pairing fixes image/template mismatch; it does not make database migrations reversible or remove the need for an explicit QA data reset when required.

For transient publication failures, Concourse retries automatically; if exhausted, re-run the same publication job with its original inputs. An existing tag with different content requires investigation and a new version; never force-move or delete a published deployment tag. Normal Git push protection enforces this for CI; a GitHub tag ruleset preventing updates/deletions to deploy/** is also recommended before rollout (this PR does not configure repository rules).

A final rejected by the ancestry guard remains published but is not selected for deployment. Release a new version from a newer RC containing the selected source. Rerunning the same build retains the old inputs and cannot repair this; never replace the existing immutable tag. This guard prevents the rollback but does not automatically coordinate RC and final releases: a new release can lose the race again. No job pause/unpause automation is installed.

Selection retries refresh the remote branch before deciding. Persistent transport, vendir, malformed-metadata, ancestry, or immutable-version errors fail visibly and retain the existing failure notification. To stop automatic selection, pause bump-in-deployments; bundle publication can continue. An intentional rollback is a separate reviewed deployments change, not replaying an old publication.

---

Step 4: Environment Deployment (galoy-deployments + Cepler)

The galoy-deployments repository is where the rubber meets the road. It contains OpenTofu configurations for every environment (staging, QA, production), vendored lana-bank deploy inputs, environment-specific Kustomize overlays, and the Cepler configuration that controls environment progression.

What is Cepler and why do we need it?

Cepler is a deployment promotion tool. The problem it solves is straightforward: when you have multiple environments (staging, QA, production), you don't want a change to reach production until it's been validated in the earlier environments first.

Cepler tracks which files have changed and which environments have successfully deployed those changes. It enforces rules like "QA can only deploy changes that have already succeeded in staging." This prevents the classic mistake of accidentally deploying untested code to production.

Cepler has a few core concepts:

• Deployment: A named unit of work (e.g., lana-bank). Each deployment has its own configuration file and its own set of environments.
• Environment: A deploy target like gcp-galoy-staging or gcp-volcano-qa. Each environment defines which file patterns it watches for changes.
• latest: A list of glob patterns. When files matching these patterns change, Cepler considers this environment "out of date" and triggers a deployment.
• passed: The name of another environment that must have successfully deployed the same changes first. This is how you create a promotion chain (staging -> QA -> production).
• propagated: Files that should be inherited from the passed environment rather than tracked independently. This is how shared module code flows from staging to QA without QA needing to independently track those files.
• State files: Cepler keeps state files in the .cepler/ directory that record exactly which commit and file versions have been deployed to each environment.

Cepler configuration in practice

Here's a simplified version of the lana-bank Cepler config:

# cepler/lana-bank.yml
deployment: lana-bank
environments:
  gcp-galoy-staging:
    latest:
      - modules/lana-bank-gcp-pg/**
      - modules/lana-bank-postgres-mcp/**
      - modules/lana-bank/config/**
      - modules/lana-bank/vendor/**
      - modules/infra/vendor/tf/postgresql/**
      - gcp/galoy-staging/lana-bank/*

  gcp-volcano-qa:
    passed: gcp-galoy-staging
    propagated:
      - modules/lana-bank-gcp-pg/**
      - modules/lana-bank-postgres-mcp/**
      - modules/lana-bank/config/**
      - modules/lana-bank/vendor/**
      - modules/infra/vendor/tf/postgresql/**
    latest:
      - gcp/volcano-qa/lana-bank/*

Reading this from top to bottom:

• Staging (gcp-galoy-staging) watches the vendored lana-bank deploy inputs, the GCP PostgreSQL module, the Postgres MCP module, and its own environment-specific config. Whenever any of those files change, staging gets a new deployment.
• QA (gcp-volcano-qa) has passed: gcp-galoy-staging, which means it will only deploy changes that have already been successfully deployed to staging. The propagated section lists the shared modules — Cepler inherits the staging-tested versions of these files rather than tracking them independently. QA also watches its own environment-specific config in latest, so changes to QA-only settings deploy immediately without waiting for staging.

How Cepler works with Concourse

The galoy-deployments Concourse pipeline uses two custom resource types to integrate with Cepler:

1. cepler-in is a Concourse resource that periodically checks the cepler-gates git branch. When it detects that there are pending changes for a given environment (based on the rules in the cepler config), it triggers a deployment job.
2. The deployment job runs OpenTofu first. OpenTofu provisions prerequisites such as databases, database roles, Keycloak realms, storage resources, Kubernetes namespaces, and Kubernetes Secrets.
3. The deployment job then applies each environment overlay with kubectl apply -k. One-shot bootstrap/simulation Jobs are deleted before apply so they can run again when the overlay includes them.
4. The job waits for Kustomize-managed bootstrap and simulation Jobs when those components are enabled.
5. cepler-out is called after a successful deployment. It updates the cepler state file, recording that this environment is now at the new version. This is what unblocks downstream environments — when staging's state is updated, Cepler knows that QA can now proceed.

If a deployment fails, the state is not updated, and downstream environments remain blocked. This is the safety net that prevents broken code from cascading through environments.

Repository structure

galoy-deployments/
├── modules/
│   ├── lana-bank/
│   │   ├── config/                  # Shared bootstrap/module config JSON
│   │   └── vendor/                  # Vendored deploy + deployment-facing tf modules from lana-bank
│   │       ├── lana-bank/deploy/    # Kustomize base/components/metadata
│   │       └── keycloak-realms/     # OpenTofu Keycloak realm module
│   ├── lana-bank-gcp-pg/            # Provisions shared PostgreSQL instances on GCP
│   └── lana-bank-postgres-mcp/      # Optional Postgres MCP Deployment
├── gcp/
│   ├── galoy-staging/
│   │   ├── shared/                  # GCP project config shared by all modules
│   │   └── lana-bank/
│   │       ├── main.tf              # Staging prerequisite wiring
│   │       └── deploy/overlays/     # Staging Kustomize overlays
│   └── volcano-qa/
│       └── lana-bank/
│           ├── main.tf              # QA prerequisite wiring
│           └── deploy/overlays/     # QA Kustomize overlays
├── cepler/
│   ├── lana-bank.yml                # Environment progression rules
│   └── .cepler/lana-bank/           # State files (one per environment)
└── vendir.yml                       # Vendir config for lana-bank deploy inputs

How environment overrides work

The vendored deploy/ tree provides common Kubernetes resources:

• deploy/base contains always-on resources such as the lana-bank-server Deployment and Services.
• deploy/components contains optional features such as ingress, workload identity, bootstrap Jobs, and simulation Jobs.
• make check-kustomize generates a temporary fixture that renders every component for schema validation.

Real environment overlays live in galoy-deployments. They compose the shared base and components, then patch environment-specific values such as hosts, Keycloak realm names, storage settings, tracing service names, and image digests.

OpenTofu and Kustomize intentionally split responsibilities:

• OpenTofu owns long-lived prerequisites and secrets.
• Kustomize owns workload manifests and one-shot operational Jobs.

---

Step 5: Production Promotion

Production deployments follow the same Cepler-driven pattern as staging and QA, but with an extra layer of human oversight.

1. Changes must pass staging first. The cepler config's passed: field ensures that any change headed for production has already been successfully deployed to staging (and potentially QA). If staging is broken, production won't even be attempted.
2. The cepler-gates branch adds a manual gate. Even after staging succeeds, production doesn't deploy automatically. The galoy-deployments repo has a special git branch called cepler-gates that contains promotion controls. Cepler checks this branch to determine whether a production deployment is "allowed."
3. A human approves the promotion. To release to production, an engineer updates the cepler-gates branch to indicate that the current staging version is approved for production. This is an explicit, auditable action.
4. Cepler detects the gate is open and the Concourse pipeline deploys to production using OpenTofu and Kustomize, just like it does for staging. After a successful deployment, the cepler state is updated.

This design means that you always know exactly what's running in production, and you can always trace it back through QA, staging, the vendored lana-bank deploy metadata, the Docker image digest, and the GitHub Release.

---

Putting It All Together

Here's the complete journey one more time, but now you should understand what's happening at each step and why:

1. Developer opens a PR. GitHub Actions runs 10+ parallel checks (tests, lint, security scans), using the shared Cachix binary cache when derivations are available.
2. PR is merged to main. Concourse re-runs the tests against the actual merged commit to catch integration issues. On success, it builds a release candidate image tagged with an RC version.
3. The deploy bundle is updated and testflight runs. The pipeline pins deploy/ to the RC image digest, provisions throwaway prerequisites, applies generated LanaInstanceDeps claims and ArgoCD Application resources, runs smoketests, and cleans up.
4. The pipeline opens a promote-rc PR. This PR contains the generated CHANGELOG and updated docs. An engineer reviews it and merges when they're ready to release. This is the first human checkpoint.
5. The release job runs. It builds the final Docker image, creates a GitHub Release, and pins deploy/ to the final image digest.
6. galoy-deployments vendors the lana-bank deploy inputs. Vendir copies deploy/ and the OpenTofu modules from the exact lana-bank commit.
7. Cepler picks up the change in galoy-deployments. It deploys to staging first. Only after staging succeeds does QA become eligible.
8. An engineer approves the production gate. They update the cepler-gates branch, Cepler detects the change, and Concourse deploys to production. This is the second human checkpoint.

At any point, you can trace what's running in an environment all the way back to the source commit — through the cepler state, the vendir config, deploy/METADATA, the Docker image digest, and the GitHub Release.

---

Quick Reference

Tool	What it does	Where it's configured
GitHub Actions	Runs PR validation checks	.github/workflows/ in lana-bank
Concourse	Builds releases, runs testflight, deploys to environments	ci/ in lana-bank and galoy-deployments
Cachix	Stores pre-built Nix binaries (galoymoney cache)	GitHub Actions workflows and CI jobs configured with Cachix
YTT	Templates Concourse pipeline YAML	ci/release/ and other Concourse pipeline configs in lana-bank
Cocogitto	Computes the next version from conventional commits	cog.toml in lana-bank
git-cliff	Generates the CHANGELOG from conventional commits	ci/config/git-cliff.toml in lana-bank
Vendir	Vendors lana-bank deploy inputs into galoy-deployments	vendir.yml in galoy-deployments
Kustomize	Composes and patches Kubernetes workload manifests	deploy/ in lana-bank and environment overlays in galoy-deployments
Cepler	Controls environment promotion (staging -> QA -> production)	cepler/*.yml in galoy-deployments
OpenTofu	Provisions databases, secrets, identity, storage, and other prerequisites	tf/ in lana-bank and modules/ in galoy-deployments