CI/CD Pipeline Sync for Geo Dependencies

A reproducible CI/CD sync pipeline for geospatial dependencies locks every transitive package to a specific hash, compiles native extensions inside a container that matches the target serverless OS, strips debug symbols to stay under the 250 MB uncompressed layer limit on AWS Lambda, and blocks deployment until a synthetic import test confirms binary compatibility. Without this discipline, rasterio, GDAL, and PROJ drift silently between developer machines and cloud runtimes, producing ImportError crashes that are invisible until the first production invocation.

Why Dependency Drift Destroys Geospatial Serverless Workloads

Native C/C++ libraries like GDAL, PROJ, and GEOS are not pure-Python wheels. They embed compiled .so files that link dynamically against the host OS’s libc, libstdc++, and sometimes libssl. The native library compilation for serverless constraints make this concrete: a wheel compiled on Ubuntu 22.04 against glibc 2.35 will fail on an Amazon Linux 2023 runtime that ships glibc 2.34 if the symbol versions diverge.

Three failure patterns emerge from uncontrolled dependency state:

• Silent version skew. A developer upgrades rasterio locally; the CI runner pulls a cached wheel from a previous job; production receives a third version from a weekly scheduled run. All three import without error locally but produce different CRS handling or driver availability in production.
• Binary incompatibility at import time. Functions cold-start cleanly in staging but crash with libgdal.so.34: cannot open shared object file in production because staging and production runtimes differ by one minor OS version.
• Size limit breaches during deployment. The Docker container optimization for GIS stack routinely produces uncompressed layers of 350-500 MB when debug symbols and unused GDAL drivers are left in place. AWS Lambda rejects the upload entirely; GCP Cloud Functions fails the build step silently.

A synchronized pipeline eliminates all three by treating the layer artifact as an immutable, versioned build output rather than a re-resolved set of packages.

Platform-by-Platform Constraints for Geospatial Layers

The exact quotas determine which compression and splitting strategies are mandatory, not optional.

The ephemeral storage limits in AWS Lambda matter here too: even if the layer deploys successfully, a GeoTIFF extraction step that writes intermediate files to /tmp can exhaust the configurable 10 GB ceiling and crash the function mid-pipeline.

Pipeline Architecture

The diagram below shows how the six pipeline stages compose from lock file through validated deployment. Each stage is a discrete job with an explicit pass/fail gate.

Step-by-Step Implementation

Step 1 — Deterministic Dependency Locking

Generate a fully pinned requirements file that includes cryptographic hashes for every wheel. This prevents supply-chain tampering and guarantees that rasterio, shapely, and fiona resolve to the identical binary distribution across every pipeline run.

# Install pip-tools in your local venv
pip install pip-tools

# Compile a locked, hash-pinned requirements file from a minimal input spec
pip-compile requirements.in \
  --output-file requirements.txt \
  --generate-hashes \
  --no-header

Commit requirements.txt to the repository. The CI pipeline must treat this file as the single source of truth; it must never re-resolve packages from requirements.in during the build job.

Set the following environment variables explicitly in every job that touches geospatial packages — do not rely on auto-discovery:

export GDAL_DATA=/opt/python/share/gdal
export PROJ_LIB=/opt/python/share/proj
export LD_LIBRARY_PATH=/opt/python/lib:$LD_LIBRARY_PATH

Step 2 — Runtime-Matched Containerized Build

The build container must match the exact glibc version and Python minor version of the target serverless runtime. A mismatch of even one glibc minor version produces unresolvable symbol errors at cold start — the cold start mapping for Python GDAL sequence begins with shared-library resolution, so a bad .so fails before any application code runs.

# Dockerfile — multi-stage build for AWS Lambda Python 3.12 on Amazon Linux 2023
FROM public.ecr.aws/sam/build-python3.12:latest AS builder

# Install system-level GDAL build dependencies
RUN dnf install -y \
    gdal-devel \
    proj-devel \
    geos-devel \
    gcc \
    gcc-c++ \
    make \
    zip \
  && dnf clean all

WORKDIR /build

COPY requirements.txt .

# Install wheels into a clean prefix that mirrors the Lambda layer mount path
RUN pip install \
    --no-cache-dir \
    --require-hashes \
    --only-binary=:none: \
    -r requirements.txt \
    --target /opt/python

# --- extraction stage ---
FROM scratch AS layer-export
COPY --from=builder /opt/python /opt/python

Use Docker BuildKit’s --cache-from type=gha to reuse the compiled layer across runs. Cache invalidation should be keyed on the hash of requirements.txt, not the wall-clock date.

For GCP Cloud Functions 2nd gen targeting Debian Bookworm, replace the base image with python:3.12-bookworm and substitute apt-get for dnf. For Azure Functions on Debian Bullseye, use python:3.12-bullseye.

Step 3 — Binary Stripping and Data Deduplication

After pip installs into /opt/python, strip debug symbols from every .so file and remove documentation, locale data, and test directories. Then relocate the GDAL and PROJ data files so they appear once rather than duplicated across sub-packages.

#!/usr/bin/env python3
"""strip_layer.py — run inside the builder container after pip install."""
import os
import shutil
import subprocess
from pathlib import Path

LAYER_ROOT = Path("/opt/python")

# 1. Strip debug symbols from all shared objects
so_files = list(LAYER_ROOT.rglob("*.so")) + list(LAYER_ROOT.rglob("*.so.*"))
for so in so_files:
    subprocess.run(
        ["strip", "--strip-unneeded", str(so)],
        check=False,  # non-zero exit on already-stripped files is harmless
        capture_output=True,
    )

# 2. Remove directories that add size without runtime value
PRUNE = ["share/man", "share/locale", "share/doc", "*.dist-info/tests"]
for pattern in PRUNE:
    for path in LAYER_ROOT.glob(f"**/{pattern}"):
        if path.is_dir():
            shutil.rmtree(path, ignore_errors=True)

# 3. Canonicalise GDAL and PROJ data to a single shared location
GDAL_DATA_DEST = LAYER_ROOT / "share/gdal"
PROJ_DATA_DEST = LAYER_ROOT / "share/proj"
GDAL_DATA_DEST.mkdir(parents=True, exist_ok=True)
PROJ_DATA_DEST.mkdir(parents=True, exist_ok=True)

for duplicate in LAYER_ROOT.rglob("gdal-data"):
    if duplicate != GDAL_DATA_DEST and duplicate.is_dir():
        for f in duplicate.iterdir():
            dest = GDAL_DATA_DEST / f.name
            if not dest.exists():
                shutil.copy2(f, dest)
        shutil.rmtree(duplicate)

print(f"Layer size after strip: {sum(f.stat().st_size for f in LAYER_ROOT.rglob('*') if f.is_file()) / 1e6:.1f} MB")

This step typically reduces uncompressed layer size from 400-500 MB to 180-220 MB. The Python layer management and size reduction page covers additional techniques such as splitting numpy/scipy into a separate math layer when the GDAL stack alone is at the limit.

Step 4 — Layer Packaging and Artifact Versioning

Package the stripped output into the provider-specific archive format and upload it with a version tag that embeds the git commit SHA.

#!/usr/bin/env bash
# package_layer.sh — runs in CI after the strip step completes

set -euo pipefail

COMMIT_SHA=$(git rev-parse --short HEAD)
LAYER_NAME="geo-dependencies"
ARCHIVE="geo-layer-${COMMIT_SHA}.zip"

# Produce the layer archive (python/ prefix required by AWS Lambda)
cd /opt
zip -r9 "/workspace/${ARCHIVE}" python/

echo "Uncompressed size: $(du -sh python/ | cut -f1)"
echo "Compressed archive: $(du -h /workspace/${ARCHIVE} | cut -f1)"

# Publish to AWS Lambda and capture the version ARN
LAYER_ARN=$(aws lambda publish-layer-version \
  --layer-name "${LAYER_NAME}" \
  --zip-file "fileb:///workspace/${ARCHIVE}" \
  --compatible-runtimes python3.11 python3.12 \
  --compatible-architectures x86_64 arm64 \
  --description "geo-deps ${COMMIT_SHA}" \
  --query 'LayerVersionArn' \
  --output text)

# Write the version manifest for downstream IaC (Terraform/CDK) to consume
cat > /workspace/layer-manifest.json <<EOF
{
  "commit": "${COMMIT_SHA}",
  "layer_arn": "${LAYER_ARN}",
  "built_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
}
EOF

echo "Published: ${LAYER_ARN}"

Commit layer-manifest.json back to the repository so that Terraform or CDK stacks always reference the exact ARN built from the current lock file. This connects IAM security boundaries for cloud GIS to the deployment: the CI role needs only lambda:PublishLayerVersion and s3:PutObject — scope it to the specific layer name and artifact bucket prefix.

For GCP Cloud Functions, substitute the aws lambda publish-layer-version call with uploading a tarball to a GCS bucket and referencing it in cloudfunctions.googleapis.com/Function resource definitions. Azure Functions uses extension bundles or a custom Docker container image published to Azure Container Registry.

Step 5 — Automated Binary Validation Gate

Never publish a layer without a synthetic test that confirms the compiled binaries load correctly and the CRS database is reachable. Run this inside a container that reproduces the Lambda execution environment, not the build environment.

#!/usr/bin/env python3
"""validate_layer.py — import smoke test, exit 1 on any failure."""
import sys
import os

# Mirror the environment variables that Lambda sets at runtime
os.environ["GDAL_DATA"] = "/opt/python/share/gdal"
os.environ["PROJ_LIB"] = "/opt/python/share/proj"
os.environ["LD_LIBRARY_PATH"] = "/opt/python/lib"

results = []

try:
    from osgeo import gdal, osr
    gdal.UseExceptions()
    version = gdal.VersionInfo("VERSION_NUM")
    assert int(version) >= 3060000, f"GDAL too old: {version}"
    results.append(f"GDAL {gdal.__version__} OK")
except Exception as e:
    results.append(f"GDAL FAILED: {e}")

try:
    import rasterio
    from rasterio.crs import CRS
    crs = CRS.from_epsg(4326)
    assert crs.is_geographic
    results.append(f"rasterio {rasterio.__version__} OK")
except Exception as e:
    results.append(f"rasterio FAILED: {e}")

try:
    from shapely.geometry import Point
    area = Point(0, 0).buffer(1).area
    assert area > 3.0, f"Unexpected area: {area}"
    results.append("shapely OK")
except Exception as e:
    results.append(f"shapely FAILED: {e}")

try:
    import pyproj
    transformer = pyproj.Transformer.from_crs("EPSG:4326", "EPSG:3857", always_xy=True)
    x, y = transformer.transform(-0.1276, 51.5074)  # London
    assert abs(x - -14209.4) < 1.0
    results.append(f"pyproj {pyproj.__version__} OK")
except Exception as e:
    results.append(f"pyproj FAILED: {e}")

failures = [r for r in results if "FAILED" in r]
for r in results:
    print(r)

if failures:
    print(f"\nValidation FAILED: {len(failures)} error(s)", file=sys.stderr)
    sys.exit(1)

print("\nAll geospatial binaries validated successfully.")

Expected output on a healthy layer:

GDAL 3.9.0 OK
rasterio 1.3.10 OK
shapely OK
pyproj 3.6.1 OK

All geospatial binaries validated successfully.

The CI pipeline must gate the publish step on this script returning exit code 0. Any failure blocks the layer-manifest.json update and prevents the layer ARN from reaching downstream infrastructure.

Step 6 — Full GitHub Actions Workflow

# .github/workflows/geo-dep-sync.yml
name: Geo Dependency Sync

on:
  push:
    paths:
      - "requirements.txt"
      - "Dockerfile"
      - ".github/workflows/geo-dep-sync.yml"
  schedule:
    - cron: "0 03 * * 1"   # Weekly rebuild every Monday at 03:00 UTC

env:
  PYTHON_VERSION: "3.12"
  LAYER_NAME: "geo-dependencies"

jobs:
  sync:
    runs-on: ubuntu-24.04
    permissions:
      contents: write          # commit layer-manifest.json back
      id-token: write          # OIDC auth to AWS — no long-lived credentials

    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials (OIDC)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_CI_ROLE_ARN }}
          aws-region: eu-west-1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build layer (cached)
        uses: docker/build-push-action@v5
        with:
          context: .
          target: layer-export
          outputs: type=local,dest=./layer-output
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Strip and measure
        run: |
          docker run --rm \
            -v ${{ github.workspace }}/layer-output:/opt \
            public.ecr.aws/sam/build-python${{ env.PYTHON_VERSION }}:latest \
            python3 /opt/strip_layer.py

      - name: Validate binaries
        run: |
          docker run --rm \
            -v ${{ github.workspace }}/layer-output/python:/opt/python \
            public.ecr.aws/lambda/python:${{ env.PYTHON_VERSION }} \
            python3 /opt/python/validate_layer.py

      - name: Package and publish layer
        if: success()
        run: |
          bash layer-output/package_layer.sh

      - name: Commit layer manifest
        if: success()
        run: |
          git config user.name "geo-dep-sync[bot]"
          git config user.email "[email protected]"
          git add layer-manifest.json
          git diff --cached --quiet || git commit -m "chore: sync geo layer $(git rev-parse --short HEAD)"
          git push

Measurement and Verification

After the pipeline publishes a new layer, confirm the optimization worked before rolling it out to all functions.

#!/usr/bin/env python3
"""measure_cold_start.py — invoke a test function and report InitDuration."""
import json
import boto3
import statistics

client = boto3.client("lambda", region_name="eu-west-1")
FUNCTION = "geo-dep-smoke-test"

durations = []
for _ in range(10):
    response = client.invoke(
        FunctionName=FUNCTION,
        InvocationType="RequestResponse",
        LogType="Tail",
    )
    # Parse InitDuration from the X-Ray / CloudWatch Logs Insights payload
    import base64
    log = base64.b64decode(response["LogResult"]).decode()
    for line in log.splitlines():
        if "Init Duration" in line:
            ms = float(line.split("Init Duration:")[1].split("ms")[0].strip())
            durations.append(ms)
            break

if durations:
    print(f"Cold starts measured: {len(durations)}")
    print(f"  Mean   InitDuration: {statistics.mean(durations):.0f} ms")
    print(f"  Median InitDuration: {statistics.median(durations):.0f} ms")
    print(f"  P95    InitDuration: {sorted(durations)[int(len(durations)*0.95)]:.0f} ms")
    # A stripped geo-deps layer on Python 3.12 should cold-start in < 2500 ms
    assert statistics.mean(durations) < 2500, "Cold start budget exceeded"

Target benchmarks for a stripped GDAL + rasterio + shapely + pyproj stack:

Monitor aws cloudwatch get-metric-statistics for InitDuration using the aws/lambda namespace. Alert on P95 exceeding the red-flag threshold, not the mean — a single bloated cold start in a geo-processing batch can cascade into downstream SQS and Pub/Sub queue routing strategy timeouts.

Failure Modes and Debugging

ImportError: libgdal.so.34: cannot open shared object file The shared library is missing from the layer or LD_LIBRARY_PATH is not set. Run ldd /opt/python/rasterio/_shim.cpython-312-x86_64-linux-gnu.so inside a container matching the Lambda runtime and trace which .so files resolve to not found. Add the missing library to the build step or extend LD_LIBRARY_PATH.

CRSError: PROJ: proj_create_from_database: Cannot find proj.db The PROJ_LIB environment variable is absent or points to a directory that does not contain proj.db. Verify with:

import os, pathlib
proj_lib = os.environ.get("PROJ_LIB", "")
db_path = pathlib.Path(proj_lib) / "proj.db"
print(f"PROJ_LIB={proj_lib!r}  exists={db_path.exists()}")

If proj.db is missing from the layer, the stripping step deleted it. Add proj.db to a preserve list in strip_layer.py.

Layer publish fails: UnzipOperationException The uncompressed layer exceeds the 250 MB AWS limit. Check the measurement output from strip_layer.py. If the stripped size is still above 220 MB, split the layer: put GDAL + rasterio + PROJ in one layer and shapely + pyproj + fiona in a second. Review stripping unnecessary Python packages from AWS Lambda layers for driver pruning options.

Validation passes in CI but fails in production The Lambda execution role lacks the environment variables set in CI. Add a Lambda configuration block to your IaC that explicitly sets GDAL_DATA, PROJ_LIB, and LD_LIBRARY_PATH — never rely on the layer to inject them automatically. The IAM security boundaries for cloud GIS page shows how to scope the execution role without granting over-broad lambda:* permissions.

Weekly rebuild produces a different layer ARN for identical inputs Non-deterministic builds stem from uncontrolled wheel resolution. Verify that pip install in the Dockerfile uses --require-hashes and requirements.txt (not requirements.in). If upstream packages publish a new patch release between runs, the hashes will not match and the install will fail loudly rather than silently upgrading.

Cost and Scaling Considerations

Each layer version persists in the provider’s artifact store and counts toward storage costs. AWS charges $0.023 per GB-month for layer storage in S3; a 50 MB compressed layer accumulates to roughly $0.014/month, negligible in isolation but significant if the pipeline publishes dozens of versions per week without pruning old ones.

Implement a retention policy: keep the three most recent validated layer ARNs and delete older versions with aws lambda delete-layer-version. Automate this as a post-deploy step in the same CI workflow.

For multi-region deployments, the layer must be published to each region independently — Lambda layers are not global. The package_layer.sh script should iterate over target regions and capture all ARNs in layer-manifest.json. Alternatively, publish the compressed archive to an S3 bucket with replication enabled and trigger region-specific publish jobs from an S3 event, connecting this pipeline to the S3 and GCS event triggers for shapefiles pattern.

When the geospatial function fleet scales beyond 500 concurrent executions, provisioned concurrency becomes cheaper than absorbing repeated cold starts. The reducing Python GDAL cold starts with provisioned concurrency page provides the cost-per-invocation crossover calculation. The synchronization pipeline described here is a prerequisite: provisioned concurrency only eliminates cold starts reliably when the layer is deterministic and validated — otherwise provisioned instances themselves may crash on import.

Frequently Asked Questions

Why does hash pinning matter more for geospatial stacks than pure-Python stacks?

Pure-Python packages degrade gracefully across minor versions — API changes are the main risk. Geospatial wheels embed compiled C extensions that link against specific shared library versions. A hash mismatch can install a wheel compiled against a different libgdal symbol table, producing crashes that are indistinguishable from environment misconfiguration.

Can I skip containerized builds if I develop on Linux?

Only if your workstation runs the identical Linux distribution, kernel, and glibc version as the serverless target. In practice this is rare. Even Ubuntu 22.04 (glibc 2.35) vs Amazon Linux 2023 (glibc 2.34) can produce symbol mismatches in GDAL’s HDF5 driver. Use the provider’s official build image.

How often should the weekly rebuild run?

Weekly is a sensible default for most teams. Increase to daily if your threat model includes rapid CVE response for geospatial libraries, or if you use >= constraints in requirements.in and want to track upstream releases. Decrease to monthly if the layer is stable and the rebuild takes more than 20 minutes.

Related

• Docker Container Optimization for GIS — multi-stage build patterns and layer cache strategies
• Python Layer Management and Size Reduction — splitting and pruning strategies for layers that exceed the 250 MB ceiling
• Stripping Unnecessary Python Packages from AWS Lambda Layers — targeted removal of test suites, locale files, and unused GDAL drivers
• Native Library Compilation for Serverless — compiling GDAL and PROJ from source for maximum driver control
• Cold Start Mapping for Python GDAL — measuring and reducing shared-library load time at function startup

Back to Packaging & Dependency Management for Serverless GIS.