Caching GDAL Build Artifacts in GitHub Actions

Q: Why does my GDAL cache never hit even when versions are unchanged?

The cache key almost always includes a volatile input such as ${{ github.sha }} or hashFiles over the whole repo. Restrict the key to hashFiles('layers/gdal/versions.lock') plus the runner architecture. Any file that changes every commit will force a permanent miss and a full rebuild on every run.

Q: How large can a GitHub Actions cache entry be?

GitHub gives each repository 10 GB of total cache with least-recently-used eviction. A stripped GDAL/PROJ/GEOS dist is roughly 90 MB, so dozens of architecture and version permutations fit comfortably. Cache the stripped dist, not the raw build tree, to stay small and restore fast.

Cache the compiled dist/ directory of GDAL, PROJ, and GEOS with actions/cache keyed on gdal-${{ matrix.arch }}-${{ hashFiles('layers/gdal/versions.lock') }}, then gate the source-compilation step on steps.cache.outputs.cache-hit != 'true'. A warm cache restores the ~90 MB stripped dist in under 10 seconds and skips the 12–15 minute from-source rebuild, and the aws lambda publish-layer-version step runs only on a cache miss or a version bump — so unchanged dependency locks never mint a duplicate layer version.

Context

The CI/CD pipeline sync for geo-dependencies workflow depends on reproducible, deterministic builds of the native geospatial stack. But reproducibility does not require recompiling on every push. Building GDAL, PROJ, GEOS, and their transitive dependencies from source inside a runtime-matched container takes 12–15 minutes on a standard ubuntu-latest runner — a cost that dominates pipeline wall-clock time when the actual change is a one-line handler edit that never touches the C toolchain.

The native library compilation for serverless process is expensive precisely because it is thorough: OS-aligned build container, static linking of PROJ and GEOS, symbol stripping, and ldd validation. The output of that entire process is a single deterministic artifact — the stripped dist/ directory — whose contents change only when a pinned version changes. That property makes it an ideal caching target: hash the inputs, and reuse the output whenever the inputs are identical.

Caching also shortens the feedback loop that keeps the compiled libgdal/libproj in step with the application wheel. When you pin GDAL and PROJ versions across build and runtime, the same lock file that guarantees CRS correctness becomes the cache key, so a version bump invalidates the cache and republishes the layer in one coherent step.

Prerequisites

Before wiring the cache into your workflow, confirm the following:

Runner: ubuntu-latest with Docker available (default on GitHub-hosted runners), or a self-hosted arm64 runner for Graviton layers
Build container: public.ecr.aws/lambda/python:3.11 for AWS Lambda targets, matching Amazon Linux 2023 glibc 2.34
Version-lock file: a single layers/gdal/versions.lock pinning every native component (this file is the only cache-key input that matters)
Build script: layers/gdal/build.sh that reads the lock file, compiles the stack into build/dist, strips symbols, and zips the layer

Environment variables baked into the eventual layer configuration:

code

GDAL_DATA=/opt/share/gdal
PROJ_LIB=/opt/share/proj
LD_LIBRARY_PATH=/opt/lib

IAM (OIDC role assumed by the workflow): lambda:PublishLayerVersion on the target layer ARN, plus s3:PutObject on the staging bucket when the zipped layer exceeds the 50 MB direct-upload limit

Keep the lock file minimal and exact so its hash is stable:

code

# layers/gdal/versions.lock — the single cache-key input
GDAL=3.9.0
PROJ=9.4.0
GEOS=3.12.1
SQLITE3=3.45.3

Implementation

The workflow below restores the compiled dist keyed on the lock-file hash and the target architecture. The compile step runs only on a cache miss; the publish step runs only when the same condition holds, so an unchanged lock never produces a redundant layer version.

yaml

# .github/workflows/gdal-layer-cache.yml
name: Build & Cache GDAL Lambda Layer

on:
  push:
    paths:
      - 'layers/gdal/**'
      - '.github/workflows/gdal-layer-cache.yml'
  workflow_dispatch:

permissions:
  id-token: write   # for AWS OIDC role assumption
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        arch: [x86_64, arm64]
    steps:
      - uses: actions/checkout@v4

      # Restore the stripped dist keyed ONLY on the version lock + arch.
      # No volatile inputs (github.sha, run_id) in the key — those force
      # a permanent miss and a full 15-minute rebuild every run.
      - name: Restore compiled GDAL dist
        id: cache
        uses: actions/cache@v4
        with:
          path: build/dist
          key: gdal-dist-${{ matrix.arch }}-${{ hashFiles('layers/gdal/versions.lock') }}

      # Compile from source ONLY on a miss. On a hit this whole step is
      # skipped, turning a ~15-minute job into a ~10-second cache restore.
      - name: Compile GDAL/PROJ/GEOS (cache miss only)
        if: steps.cache.outputs.cache-hit != 'true'
        run: |
          docker run --rm \
            --platform linux/${{ matrix.arch == 'arm64' && 'arm64' || 'amd64' }} \
            -v "${{ github.workspace }}":/workspace \
            -w /workspace \
            public.ecr.aws/lambda/python:3.11 \
            bash layers/gdal/build.sh   # reads versions.lock, writes build/dist

      # Always assemble the zip — cheap, and needed on hit and miss alike.
      - name: Package layer zip
        run: |
          cd build/dist && zip -r9 "../gdal-layer-${{ matrix.arch }}.zip" . && cd -
          echo "ZIP_SIZE=$(du -h build/gdal-layer-${{ matrix.arch }}.zip | cut -f1)" >> "$GITHUB_ENV"

      - name: Configure AWS credentials
        if: steps.cache.outputs.cache-hit != 'true'
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gh-actions-layer-publisher
          aws-region: us-east-1

      # Publish ONLY on a miss (i.e. the version lock changed and the dist
      # was rebuilt). A cache hit means the layer already exists — skip.
      - name: Publish Lambda layer (version bump only)
        if: steps.cache.outputs.cache-hit != 'true'
        run: |
          aws lambda publish-layer-version \
            --layer-name "gdal-py311-${{ matrix.arch }}" \
            --description "GDAL $(grep '^GDAL=' layers/gdal/versions.lock | cut -d= -f2) / PROJ $(grep '^PROJ=' layers/gdal/versions.lock | cut -d= -f2)" \
            --zip-file "fileb://build/gdal-layer-${{ matrix.arch }}.zip" \
            --compatible-runtimes python3.11 \
            --compatible-architectures ${{ matrix.arch }}

The key design decision is that build/dist — the stripped output of the compile, not the raw build tree — is what gets cached. The raw build tree contains gigabytes of object files and CMake scratch; the stripped dist is roughly 90 MB and restores in seconds. Caching the small deterministic artifact keeps you well inside GitHub’s 10 GB per-repository cache budget even across two architectures and several historical version locks.

Verification

On the first run (cold cache) the compile step executes and the log records a cache save. On the second run with an unchanged versions.lock, the compile and publish steps are skipped entirely. Confirm both paths from the run summary:

bash

# Inspect existing cache entries for the repo (requires gh CLI + actions:read)
gh cache list --key gdal-dist- --json key,sizeInBytes,createdAt

Expected output after two architectures have built once:

code

KEY                                              SIZE     CREATED
gdal-dist-x86_64-4f2a9c...b1  93.4 MB  2026-07-11T09:14:03Z
gdal-dist-arm64-4f2a9c...b1   88.1 MB  2026-07-11T09:15:22Z

In the warm-run job log you should see the restore and the skips:

code

Cache restored from key: gdal-dist-x86_64-4f2a9c...b1
Compile GDAL/PROJ/GEOS (cache miss only)   ⏭  skipped
Publish Lambda layer (version bump only)   ⏭  skipped

A warm run should complete the build job in well under a minute versus 12–15 minutes cold. To force a rebuild for testing, bump any version in versions.lock and push — the hash changes, the cache misses, the compile runs, and a new layer version is published.

Gotchas and Edge Cases

Volatile cache keys silently disable the cache. Including ${{ github.sha }}, ${{ github.run_id }}, or a broad hashFiles('**') in the key guarantees a miss on every run, so you pay the full 15-minute rebuild while believing caching is active. Restrict the key to hashFiles('layers/gdal/versions.lock') and the architecture, nothing else.
Architecture must be part of the key. An x86_64 dist restored onto an arm64 build produces ELFCLASS and wrong architecture errors at Lambda invocation. The ${{ matrix.arch }} segment prevents cross-architecture cache poisoning — the same discipline that keeps Alpine-based minimal GDAL images architecture-correct.
Branch-scoped cache isolation. GitHub Actions caches are scoped to the branch that created them, with read-through from the default branch. A feature branch that first builds a new version lock cannot read another feature branch’s cache. Merge dependency bumps to the default branch promptly so the warm cache is shared repo-wide.
Cache hit does not re-publish, by design. If someone manually deletes a layer version in the console, a warm run will not recreate it because the publish step is gated on a miss. Recover by bumping the lock (or clearing the specific cache entry) to force the rebuild-and-publish path, rather than relying on the workflow to detect the missing AWS-side artifact.

Frequently Asked Questions

Why does my GDAL cache never hit even when versions are unchanged?

The cache key almost certainly includes a volatile input such as ${{ github.sha }} or a repo-wide hashFiles('**'). Any file that changes every commit forces a permanent miss and a full from-source rebuild. Restrict the key to hashFiles('layers/gdal/versions.lock') plus the runner architecture so the key is stable across commits that do not touch the native stack.

How large can a GitHub Actions cache entry be?

GitHub allocates 10 GB of total cache per repository with least-recently-used eviction once you exceed it. A stripped GDAL/PROJ/GEOS dist is roughly 90 MB, so dozens of architecture and version permutations fit comfortably. Cache the stripped dist/ rather than the raw build tree — the latter can be several gigabytes and would evict everything else.

Should I cache the compiled dist or the published layer?

Cache the compiled dist directory. The published layer is an immutable AWS artifact addressed by ARN and version; re-publishing an identical dist wastes a layer version and confuses IaC drift detection. Restore the dist on a hit, and only call publish-layer-version when the version lock actually changed — which is exactly when the cache misses.

CI/CD Pipeline Sync for Geo Dependencies — the full reproducible-build discipline this caching step accelerates
Pinning GDAL and PROJ Versions Across Build and Runtime — the version-lock file that doubles as the cache key
Native Library Compilation for Serverless — the expensive build whose deterministic output you are caching
Stripping Unnecessary Python Packages from AWS Lambda Layers — keep the cached dist small before it is packaged
Cold Start Mapping for Python GDAL — why the stripped, cached artifact loads faster at invocation

Back to CI/CD Pipeline Sync for Geo Dependencies