Caching GDAL Build Artifacts in GitHub Actions

Cache the compiled dist/ directory of GDAL, PROJ, and GEOS with actions/cache keyed on gdal-${{ matrix.arch }}-${{ hashFiles('layers/gdal/versions.lock') }}, then gate the source-compilation step on steps.cache.outputs.cache-hit != 'true'. A warm cache restores the ~90 MB stripped dist in under 10 seconds and skips the 12–15 minute from-source rebuild, and the aws lambda publish-layer-version step runs only on a cache miss or a version bump — so unchanged dependency locks never mint a duplicate layer version.

Context

The CI/CD pipeline sync for geo-dependencies workflow depends on reproducible, deterministic builds of the native geospatial stack. But reproducibility does not require recompiling on every push. Building GDAL, PROJ, GEOS, and their transitive dependencies from source inside a runtime-matched container takes 12–15 minutes on a standard ubuntu-latest runner — a cost that dominates pipeline wall-clock time when the actual change is a one-line handler edit that never touches the C toolchain.

Cache hit or miss decides whether GDAL is recompiledA single decision on the cache key, which is the versions.lock hash plus the target architecture. On a hit the compiled dist is restored in under ten seconds and both the compile and publish steps are skipped. On a miss GEOS, PROJ and GDAL are compiled from source in twelve to fifteen minutes and a new layer version is published.One key lookup decides whether the C toolchain runs at allIs gdal-dist-<arch>-<hash ofversions.lock> already in the Actionscache?hit — lock unchangedRestore and skip~90 MB stripped dist restored in under 10 scompile skipped, publish skippedjob finishes in well under a minutemiss — version bumpedCompile and publishGEOS then PROJ then GDAL, 12–15 minstrip, zip, publish-layer-versionone new layer version per lock change
The compile step and the publish step are gated on the same condition, so a run either rebuilds and publishes together or does neither — a lock that has not changed can never mint a duplicate layer version.

The native library compilation for serverless process is expensive precisely because it is thorough: OS-aligned build container, static linking of PROJ and GEOS, symbol stripping, and ldd validation. The output of that entire process is a single deterministic artifact — the stripped dist/ directory — whose contents change only when a pinned version changes. That property makes it an ideal caching target: hash the inputs, and reuse the output whenever the inputs are identical.

Caching also shortens the feedback loop that keeps the compiled libgdal/libproj in step with the application wheel. When you pin GDAL and PROJ versions across build and runtime, the same lock file that guarantees CRS correctness becomes the cache key, so a version bump invalidates the cache and republishes the layer in one coherent step.

Prerequisites

Before wiring the cache into your workflow, confirm the following:

  • Runner: ubuntu-latest with Docker available (default on GitHub-hosted runners), or a self-hosted arm64 runner for Graviton layers
  • Build container: public.ecr.aws/lambda/python:3.11 for AWS Lambda targets, matching Amazon Linux 2023 glibc 2.34
  • Version-lock file: a single layers/gdal/versions.lock pinning every native component (this file is the only cache-key input that matters)
  • Build script: layers/gdal/build.sh that reads the lock file, compiles the stack into build/dist, strips symbols, and zips the layer
  • Environment variables baked into the eventual layer configuration:
    code 
    GDAL_DATA=/opt/share/gdal
PROJ_LIB=/opt/share/proj
LD_LIBRARY_PATH=/opt/lib
  • IAM (OIDC role assumed by the workflow): lambda:PublishLayerVersion on the target layer ARN, plus s3:PutObject on the staging bucket when the zipped layer exceeds the 50 MB direct-upload limit

Keep the lock file minimal and exact so its hash is stable:

code 
# layers/gdal/versions.lock — the single cache-key input
GDAL=3.9.0
PROJ=9.4.0
GEOS=3.12.1
SQLITE3=3.45.3

Implementation

The workflow below restores the compiled dist keyed on the lock-file hash and the target architecture. The compile step runs only on a cache miss; the publish step runs only when the same condition holds, so an unchanged lock never produces a redundant layer version.

yaml 
# .github/workflows/gdal-layer-cache.yml
name: Build & Cache GDAL Lambda Layer

on:
  push:
    paths:
      - 'layers/gdal/**'
      - '.github/workflows/gdal-layer-cache.yml'
  workflow_dispatch:

permissions:
  id-token: write   # for AWS OIDC role assumption
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        arch: [x86_64, arm64]
    steps:
      - uses: actions/checkout@v4

      # Restore the stripped dist keyed ONLY on the version lock + arch.
      # No volatile inputs (github.sha, run_id) in the key — those force
      # a permanent miss and a full 15-minute rebuild every run.
      - name: Restore compiled GDAL dist
        id: cache
        uses: actions/cache@v4
        with:
          path: build/dist
          key: gdal-dist-${{ matrix.arch }}-${{ hashFiles('layers/gdal/versions.lock') }}

      # Compile from source ONLY on a miss. On a hit this whole step is
      # skipped, turning a ~15-minute job into a ~10-second cache restore.
      - name: Compile GDAL/PROJ/GEOS (cache miss only)
        if: steps.cache.outputs.cache-hit != 'true'
        run: |
          docker run --rm \
            --platform linux/${{ matrix.arch == 'arm64' && 'arm64' || 'amd64' }} \
            -v "${{ github.workspace }}":/workspace \
            -w /workspace \
            public.ecr.aws/lambda/python:3.11 \
            bash layers/gdal/build.sh   # reads versions.lock, writes build/dist

      # Always assemble the zip — cheap, and needed on hit and miss alike.
      - name: Package layer zip
        run: |
          cd build/dist && zip -r9 "../gdal-layer-${{ matrix.arch }}.zip" . && cd -
          echo "ZIP_SIZE=$(du -h build/gdal-layer-${{ matrix.arch }}.zip | cut -f1)" >> "$GITHUB_ENV"

      - name: Configure AWS credentials
        if: steps.cache.outputs.cache-hit != 'true'
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gh-actions-layer-publisher
          aws-region: us-east-1

      # Publish ONLY on a miss (i.e. the version lock changed and the dist
      # was rebuilt). A cache hit means the layer already exists — skip.
      - name: Publish Lambda layer (version bump only)
        if: steps.cache.outputs.cache-hit != 'true'
        run: |
          aws lambda publish-layer-version \
            --layer-name "gdal-py311-${{ matrix.arch }}" \
            --description "GDAL $(grep '^GDAL=' layers/gdal/versions.lock | cut -d= -f2) / PROJ $(grep '^PROJ=' layers/gdal/versions.lock | cut -d= -f2)" \
            --zip-file "fileb://build/gdal-layer-${{ matrix.arch }}.zip" \
            --compatible-runtimes python3.11 \
            --compatible-architectures ${{ matrix.arch }}

The key design decision is that build/dist — the stripped output of the compile, not the raw build tree — is what gets cached. The raw build tree contains gigabytes of object files and CMake scratch; the stripped dist is roughly 90 MB and restores in seconds. Caching the small deterministic artifact keeps you well inside GitHub’s 10 GB per-repository cache budget even across two architectures and several historical version locks.

A volatile cache key compared with a lock-file cache keyTwo panels. A key containing the commit SHA changes on every push, misses every time, pays the full twelve to fifteen minute rebuild and publishes a redundant layer version each run. A key made of the versions.lock hash and the architecture changes only on a version bump, restores in seconds and publishes once per real change.What belongs in the cache key, and what silently disables itKey includes ${{ github.sha }}Key changes on every commit, including handler-only editsMiss on every run — the cache never restoresFull 12–15 minute from-source rebuild each pushA redundant layer version published every runNothing in the log says caching is brokenKey = versions.lock hash + archKey changes only when a pinned version changesWarm restore of the ~90 MB dist in under 10 secondsCompile and publish both skipped on a hitOne layer version per real dependency changeArchitecture segment stops an x86_64 dist landing onarm64Cache the stripped dist, not the raw build tree — roughly 90 MB per architecture.
Both workflows look identical in the YAML and both report a cache step. Only the second one ever restores anything — a volatile key fails quietly by always missing.

Verification

On the first run (cold cache) the compile step executes and the log records a cache save. On the second run with an unchanged versions.lock, the compile and publish steps are skipped entirely. Confirm both paths from the run summary:

bash 
# Inspect existing cache entries for the repo (requires gh CLI + actions:read)
gh cache list --key gdal-dist- --json key,sizeInBytes,createdAt

Expected output after two architectures have built once:

code 
KEY                                              SIZE     CREATED
gdal-dist-x86_64-4f2a9c...b1  93.4 MB  2026-07-11T09:14:03Z
gdal-dist-arm64-4f2a9c...b1   88.1 MB  2026-07-11T09:15:22Z

In the warm-run job log you should see the restore and the skips:

code 
Cache restored from key: gdal-dist-x86_64-4f2a9c...b1
Compile GDAL/PROJ/GEOS (cache miss only)   ⏭  skipped
Publish Lambda layer (version bump only)   ⏭  skipped

A warm run should complete the build job in well under a minute versus 12–15 minutes cold. To force a rebuild for testing, bump any version in versions.lock and push — the hash changes, the cache misses, the compile runs, and a new layer version is published.

Build job duration on a cold cache compared with a warm cacheThree measured durations for the same build job: a cold run compiling GEOS, PROJ and GDAL from source at twelve to fifteen minutes, a warm run end to end in under one minute, and the cache restore step itself in under ten seconds.Wall-clock time for the build job, cold against warmCold run — compile from source12–15 minWarm run — whole build jobunder 1 minWarm run — cache restore step onlyunder 10 s015 minMeasured on a GitHub-hosted ubuntu-latest runner for one architecture; the arm64 matrix leg restores its own separately keyed entry inparallel.
A warm run is faster than a cold one by more than an order of magnitude, and almost all of what is left is zipping and uploading — the restore itself is under ten seconds.

Gotchas and Edge Cases

  • Volatile cache keys silently disable the cache. Including ${{ github.sha }}, ${{ github.run_id }}, or a broad hashFiles('**') in the key guarantees a miss on every run, so you pay the full 15-minute rebuild while believing caching is active. Restrict the key to hashFiles('layers/gdal/versions.lock') and the architecture, nothing else.

  • Architecture must be part of the key. An x86_64 dist restored onto an arm64 build produces ELFCLASS and wrong architecture errors at Lambda invocation. The ${{ matrix.arch }} segment prevents cross-architecture cache poisoning — the same discipline that keeps Alpine-based minimal GDAL images architecture-correct.

  • Branch-scoped cache isolation. GitHub Actions caches are scoped to the branch that created them, with read-through from the default branch. A feature branch that first builds a new version lock cannot read another feature branch’s cache. Merge dependency bumps to the default branch promptly so the warm cache is shared repo-wide.

  • Cache hit does not re-publish, by design. If someone manually deletes a layer version in the console, a warm run will not recreate it because the publish step is gated on a miss. Recover by bumping the lock (or clearing the specific cache entry) to force the rebuild-and-publish path, rather than relying on the workflow to detect the missing AWS-side artifact.

Frequently Asked Questions

Why does my GDAL cache never hit even when versions are unchanged?

The cache key almost certainly includes a volatile input such as ${{ github.sha }} or a repo-wide hashFiles('**'). Any file that changes every commit forces a permanent miss and a full from-source rebuild. Restrict the key to hashFiles('layers/gdal/versions.lock') plus the runner architecture so the key is stable across commits that do not touch the native stack.

How large can a GitHub Actions cache entry be?

GitHub allocates 10 GB of total cache per repository with least-recently-used eviction once you exceed it. A stripped GDAL/PROJ/GEOS dist is roughly 90 MB, so dozens of architecture and version permutations fit comfortably. Cache the stripped dist/ rather than the raw build tree — the latter can be several gigabytes and would evict everything else.

Should I cache the compiled dist or the published layer?

Cache the compiled dist directory. The published layer is an immutable AWS artifact addressed by ARN and version; re-publishing an identical dist wastes a layer version and confuses IaC drift detection. Restore the dist on a hit, and only call publish-layer-version when the version lock actually changed — which is exactly when the cache misses.

Back to CI/CD Pipeline Sync for Geo Dependencies