Setting GDAL_DATA and PROJ_LIB in Lambda

Set five variables in the function configuration, not in the handler: GDAL_DATA=/opt/share/gdal, PROJ_DATA=/opt/share/proj, PROJ_LIB=/opt/share/proj, LD_LIBRARY_PATH=/opt/lib:/var/task/lib:/var/lang/lib:/lib64:/usr/lib64, and PYTHONPATH=/opt/python/lib/python3.12/site-packages. The /opt prefix is not a convention you can choose — it is where AWS Lambda extracts every attached layer, so the value of each variable is the layer zip’s internal path with /opt glued to the front. Get the prefix right and EPSG:2193 resolves in about 12 ms; get it wrong and the first transformation raises PROJ: proj_create_from_database: Cannot find proj.db with no fallback.

Context

A Lambda layer is a zip archive whose contents are unpacked into /opt before the runtime starts. Up to five layers can be attached to one function, they are merged into a single tree in attachment order, and together with the function package they must stay under 250 MB unzipped. Nothing in that arrangement tells GDAL where its data went — GDAL looks in GDAL_DATA, then in the prefix that was compiled into libgdal.so on the build machine, which on Amazon Linux 2023 is /usr/share/gdal and is empty inside a Lambda.

That is the entire failure. The library is present, the Python import succeeds, pixel reads work, and then the first ImportFromEPSG or rasterio.warp.reproject call fails. Because the failure is deferred until a CRS lookup, it routinely passes a smoke test that only opens a raster and reads a window. The PROJ and GDAL runtime configuration overview covers the resolution order in detail; this page is the AWS-specific recipe with the values written out.

Two facts about layers drive everything below. First, the merge is silent: two layers that each carry share/proj overwrite one another, so the proj.db your function actually reads may not be the one you think it is. Keep the data directories in exactly one layer. Second, the zip’s internal structure is preserved verbatim, so a build script that copies proj.db into python/share/proj produces the runtime path /opt/python/share/proj — and the environment must say so.

Prerequisites

Confirm all of these before deploying:

• A layer built on Amazon Linux 2023 — public.ecr.aws/lambda/python:3.12, per building rasterio Lambda layers on Amazon Linux 2023
• The data directories at a known path in the zip — this guide assumes share/gdal/ and share/proj/ at the zip root, with share/proj/proj.db present
• Shared objects at lib/ in the same zip, or in one other layer whose path you also record
• Under 250 MB unzipped across the function package and all attached layers, with at most five layers
• Python 3.12 on x86_64 — change the PYTHONPATH version segment if you are on another runtime
• The layer ARN pinned in IaC, including its version number — an unpinned :latest is how the path and the environment drift apart
• lambda:UpdateFunctionConfiguration on the deployer role, since the variables are function configuration rather than code

Which PROJ Variable to Set

PROJ 9.1 renamed the data-directory variable from PROJ_LIB to PROJ_DATA. Both names are alive in the field, and a single layer frequently contains two PROJ builds — one vendored by pyproj, another vendored by rasterio — which can sit on opposite sides of that release.

Take the third branch unconditionally. Assigning the same path to both variables is correct for every PROJ release from 6.0 onwards, adds one line to the environment map, and means a pyproj version bump in a future build cannot break coordinate resolution. The only visible cost is a deprecation line in CloudWatch on a PROJ 9.1+ build, which is a fair trade for removing a version dependency from the deployment.

Implementation

The Terraform below defines the environment as a single reusable local so the same map can be attached to every function in the pipeline. Note that LD_LIBRARY_PATH is prepended rather than replaced — overwriting it with a bare /opt/lib drops the runtime’s own library directories and breaks native components inside the AWS SDK.

# lambda_gdal.tf
locals {
  # The layer zip root holds share/, lib/ and python/, so /opt gains those names.
  gdal_prefix = "/opt"

  gdal_env = {
    # --- Data directories: the four variables the stack cannot run without ---
    GDAL_DATA = "${local.gdal_prefix}/share/gdal"
    PROJ_DATA = "${local.gdal_prefix}/share/proj" # PROJ >= 9.1
    PROJ_LIB  = "${local.gdal_prefix}/share/proj" # PROJ  < 9.1, harmless above

    # Prepend. Replacing this list breaks the runtime's own native modules.
    LD_LIBRARY_PATH = join(":", [
      "${local.gdal_prefix}/lib",
      "/var/task/lib",
      "/var/lang/lib",
      "/lib64",
      "/usr/lib64",
    ])

    # Implicit on the managed runtime, explicit here so subprocesses inherit it.
    PYTHONPATH = "${local.gdal_prefix}/python/lib/python3.12/site-packages"

    # --- /var/task is read-only: keep every GDAL write inside /tmp ---
    GDAL_PAM_ENABLED = "NO"
    CPL_TMPDIR       = "/tmp"

    # --- Grid cache, if PROJ_NETWORK is used; HOME is not writable ---
    PROJ_NETWORK                 = "ON"
    PROJ_USER_WRITABLE_DIRECTORY = "/tmp/proj_cache"
  }
}

resource "aws_lambda_function" "reprojector" {
  function_name = "geo-reprojector"
  role          = aws_iam_role.reprojector.arn
  handler       = "handler.main"
  runtime       = "python3.12"
  architectures = ["x86_64"]

  filename         = data.archive_file.src.output_path
  source_code_hash = data.archive_file.src.output_base64sha256

  # Pin the version. ":latest" lets the paths move without the config moving.
  layers = ["arn:aws:lambda:us-east-1:123456789012:layer:gdal-3-9-py312-x86_64:14"]

  memory_size = 1769 # one full vCPU
  timeout     = 300  # well inside the 15 min ceiling

  ephemeral_storage {
    size = 1024 # /tmp: 512 MB by default, up to 10,240 MB
  }

  environment {
    variables = local.gdal_env
  }
}

The handler needs one thing IaC cannot express: PROJ_USER_WRITABLE_DIRECTORY names a directory that must exist. Create it at module scope, above the imports, so it is created once per execution environment and before GDAL’s driver registration caches anything.

# handler.py
import os
from pathlib import Path

# Must precede the rasterio import: driver registration resolves and caches
# GDAL_DATA and the PROJ search path on the very first import in the process.
Path(os.environ.get("PROJ_USER_WRITABLE_DIRECTORY", "/tmp/proj_cache")).mkdir(
    parents=True, exist_ok=True
)

import rasterio                                   # noqa: E402
from rasterio.crs import CRS                      # noqa: E402
from rasterio.warp import calculate_default_transform, reproject, Resampling  # noqa: E402


def main(event, context):
    dst_crs = CRS.from_epsg(event.get("dst_epsg", 2193))
    with rasterio.open(event["src"]) as src:
        transform, width, height = calculate_default_transform(
            src.crs, dst_crs, src.width, src.height, *src.bounds
        )
        profile = src.profile | {
            "crs": dst_crs, "transform": transform,
            "width": width, "height": height, "driver": "GTiff",
        }
        with rasterio.open("/tmp/out.tif", "w", **profile) as dst:
            for band in range(1, src.count + 1):
                reproject(
                    source=rasterio.band(src, band),
                    destination=rasterio.band(dst, band),
                    src_transform=src.transform, src_crs=src.crs,
                    dst_transform=transform, dst_crs=dst_crs,
                    resampling=Resampling.bilinear,
                )
    return {"crs": dst_crs.to_string(), "width": width, "height": height}

Every Value, Per Platform

The same five variable names apply everywhere — they belong to GDAL and PROJ, not to any cloud. Only the values move, and they move because the artifact lands in a different place.

The AWS column is the only one using /opt, and it uses it only because a layer is attached. The same function shipped as a container image resolves its data wherever the Dockerfile put it, which usually means the Cloud Run column’s shape — one more reason the value belongs in version control beside the artifact that produced it, as pinning GDAL and PROJ versions across build and runtime argues.

Verification

Deploy a throwaway function that shares the same layer and environment, invoke it once, and assert on what comes back. This is the only check that tests the real runtime rather than a container that resembles it.

# verify.py
import os

import pyproj
from osgeo import gdal, osr


def main(event, context):
    gdal.UseExceptions()
    gdal.AllRegister()
    srs = osr.SpatialReference()
    srs.ImportFromEPSG(2193)
    proj_dir = pyproj.datadir.get_data_dir()
    return {
        "gdal_data": gdal.GetConfigOption("GDAL_DATA"),
        "gdal_data_exists": os.path.isdir(gdal.GetConfigOption("GDAL_DATA") or ""),
        "proj_data_env": os.environ.get("PROJ_DATA"),
        "proj_lib_env": os.environ.get("PROJ_LIB"),
        "pyproj_data_dir": proj_dir,
        "proj_db_bytes": os.path.getsize(os.path.join(proj_dir, "proj.db")),
        "epsg_2193": srs.GetAttrValue("PROJCS"),
        "driver_count": gdal.GetDriverCount(),
        "proj_version": pyproj.__proj_version__,
    }

aws lambda invoke --function-name geo-verify --payload '{}' /dev/stdout | head -1

Expected output:

{"gdal_data": "/opt/share/gdal", "gdal_data_exists": true,
 "proj_data_env": "/opt/share/proj", "proj_lib_env": "/opt/share/proj",
 "pyproj_data_dir": "/opt/share/proj", "proj_db_bytes": 9445376,
 "epsg_2193": "NZGD2000 / New Zealand Transverse Mercator 2000",
 "driver_count": 247, "proj_version": "9.4.1"}

Three of those fields carry the whole result. pyproj_data_dir must equal proj_data_env — when it does not, pyproj is reading its own vendored proj.db while GDAL reads yours, and the two can disagree about which transformation pipeline exists. epsg_2193 must be the full projected-CRS name rather than None, which proves proj.db was found and is complete. And driver_count in the low hundreds rather than the low tens proves GDAL_DATA points at the real directory instead of one that merely exists.

Gotchas and Edge Cases

• Setting the variables in the handler is usually too late. GDAL resolves and caches GDAL_DATA and the PROJ search path during driver registration, which fires on the first import rasterio or from osgeo import gdal in the process. A module-scope import followed by an os.environ assignment inside the function body has already lost, and the wrong value persists for every warm invocation in that execution environment. If code truly must switch at runtime, call gdal.SetConfigOption("PROJ_LIB", path) and pyproj.datadir.set_data_dir(path) together before the first transform.
• Replacing LD_LIBRARY_PATH breaks more than it fixes. Setting it to bare /opt/lib removes /var/lang/lib and /lib64 from the search path, which can surface as an unrelated ImportError from a boto3 native dependency long after the GDAL problem is solved. Always prepend.
• /var/task is read-only, and GDAL wants to write. Opening a GeoTIFF makes GDAL try to create a .aux.xml sidecar next to it. When the raster was shipped in the package that write fails with EACCES. Set GDAL_PAM_ENABLED=NO and CPL_TMPDIR=/tmp in the same environment map, and remember that /tmp is 512 MB by default and provisionable to 10,240 MB — the sizing question covered in managing /tmp storage limits for GeoTIFF extraction.
• Correct paths still do not guarantee correct coordinates. proj.db holds transformation definitions, not the datum grids some of them require. A missing grid produces no error — PROJ falls back to a ballpark transformation and returns coordinates that are metres out. Check with TransformerGroup(...).unavailable_operations and size the grids using shipping PROJ transformation grids without blowing the package limit.

Frequently Asked Questions

Can I set GDAL_DATA inside the Lambda handler instead of in IaC?

Only if the assignment runs before the first import rasterio or from osgeo import gdal, because GDAL caches the resolved data directory during driver registration. Since most handlers import at module scope, an assignment in the function body runs too late and the wrong value sticks for the life of the execution environment. Setting the variables in the function configuration guarantees they exist before the interpreter starts.

Why does my layer resolve to /opt/python/share/proj instead of /opt/share/proj?

Lambda extracts the layer zip at /opt with its internal structure preserved. If the build copied the data into the python/ prefix, the runtime path gains that directory. Either move the data to the zip root during the build or point PROJ_DATA and PROJ_LIB at the deeper path — but pick one, record it beside the layer ARN, and do not let the two drift.

Do I still need PROJ_LIB if the layer ships PROJ 9?

PROJ 9.1 and later read PROJ_DATA and treat PROJ_LIB as a deprecated alias that still works and logs a warning. A layer usually contains more than one PROJ build — one vendored by pyproj, one by rasterio — so set both variables to the same path. One extra line removes the version dependency entirely.

Related

• PROJ and GDAL Runtime Configuration — the full resolution order, the datum-grid problem, and the read-performance options
• Shipping PROJ Transformation Grids Without Blowing the Package Limit — what proj.db deliberately does not contain, and three ways to supply it
• GDAL Config Options That Actually Change COG Read Performance — what to add to this environment map once the paths are correct
• Building Rasterio Lambda Layers on Amazon Linux 2023 — how the zip whose paths these variables name gets built
• Cold Start Mapping for Python GDAL — where driver registration sits in the initialisation budget

Back to PROJ and GDAL Runtime Configuration