Building Rasterio Lambda Layers on Amazon Linux 2023

Build the layer inside public.ecr.aws/lambda/python:3.11 — the Amazon Linux 2023 image AWS actually runs — and install with pip install --only-binary :all: --platform manylinux2014_x86_64 --target python/lib/python3.11/site-packages rasterio==1.4.3 so pip fetches the prebuilt GDAL wheel instead of compiling from source. Then copy the bundled gdal_data and proj trees into the layer, run strip --strip-unneeded on every .so, zip the python/ prefix, and register it with aws lambda publish-layer-version. A rasterio-only layer built this way unzips to roughly 90-110 MB after stripping — well inside the 250 MB unzipped ceiling.

Rasterio Lambda layer build on Amazon Linux 2023 Five stages: AL2023 container, manylinux-constrained pip install into python/ prefix, bundle GDAL_DATA and PROJ_LIB, strip .so symbols, then zip and publish-layer-version to the /opt mount. AL2023 Container lambda/python:3.11 glibc 2.34 exact runtime pip install --only-binary manylinux2014 into python/ Bundle Data GDAL_DATA PROJ_LIB proj.db + csv Strip .so rasterio.libs --strip-unneeded shrink payload Publish zip python/ layer-version mount at /opt Rasterio layer build — container to /opt Lambda mount, every stage inside AL2023

Context

The native library compilation discipline draws a sharp line between two ways to get GDAL into a Lambda: compile the C toolchain from source, or lean on the manylinux wheels that the rasterio maintainers already publish. For the overwhelming majority of raster workloads the second path is correct — the prebuilt wheel bundles a tested GDAL, PROJ, GEOS, and their data files, and it installs in seconds. The only hard requirement is that you assemble it inside an environment that matches the Lambda runtime’s C ABI, which on modern functions is Amazon Linux 2023 with glibc 2.34.

The trap people fall into is running pip install rasterio on a macOS or Ubuntu laptop and zipping the result. That produces a layer whose .so files link against the wrong libc, and the function fails at cold start with GLIBC_2.38 not found or an ImportError on libgdal.so before your handler runs. Building inside public.ecr.aws/lambda/python:3.11 removes that entire class of failure because it is the same image AWS mounts. This page is the wheel-based counterpart to the from-source recipe in the parent guide, and it pairs naturally with stripping unnecessary Python packages once the layer exists.

Prerequisites

Confirm each of these before you start the build:

  • Runtime: Python 3.11 on x86_64 (add --platform linux/arm64 and a Graviton wheel tag for arm64)
  • Build image: public.ecr.aws/lambda/python:3.11 pulled locally — this is Amazon Linux 2023
  • Pinned versions: rasterio==1.4.3, which vendors GDAL 3.9.x and PROJ 9.x in its manylinux wheel
  • System tools: strip (GNU binutils), zip, find, docker, and the AWS CLI v2
  • Layer layout: everything under python/lib/python3.11/site-packages/ so Lambda finds it when the layer mounts at /opt
  • Runtime environment variables (set on the function, not the build):
    code 
    GDAL_DATA=/opt/python/lib/python3.11/site-packages/rasterio/gdal_data
PROJ_LIB=/opt/python/lib/python3.11/site-packages/pyproj/proj_dir/share/proj
LD_LIBRARY_PATH=/opt/python/lib/python3.11/site-packages/rasterio.libs
  • IAM for the upload step: lambda:PublishLayerVersion on the target layer ARN, plus s3:PutObject on a staging bucket because the zip exceeds the 50 MB direct-upload threshold

Implementation

The script below runs entirely inside the AL2023 container. It pins the platform tag so pip resolves the prebuilt manylinux wheel rather than triggering a source build, copies the data directories to predictable paths, strips debug symbols, and lays out the python/ prefix.

bash 
#!/usr/bin/env bash
# build_rasterio_layer.sh — run on the host; it drives the AL2023 container.
set -euo pipefail

RASTERIO_VERSION="1.4.3"
PY="python3.11"
LAYER_DIR="$(pwd)/layer"
SITE="python/lib/${PY}/site-packages"

rm -rf "${LAYER_DIR}" && mkdir -p "${LAYER_DIR}"

docker run --rm \
  --platform linux/amd64 \
  -v "${LAYER_DIR}:/layer" \
  public.ecr.aws/lambda/python:3.11 \
  bash -c "
    set -euo pipefail
    dnf install -y binutils >/dev/null   # provides 'strip' on the minimal image

    # --only-binary :all: forbids any source build; --platform pins the ABI tag
    # so pip downloads the manylinux2014_x86_64 wheel that vendors GDAL + PROJ.
    pip install \
      --only-binary :all: \
      --platform manylinux2014_x86_64 \
      --python-version 3.11 \
      --implementation cp \
      --abi cp311 \
      --target /layer/${SITE} \
      rasterio==${RASTERIO_VERSION}

    # rasterio ships gdal_data (EPSG lookups, driver metadata) and pyproj ships
    # its proj database. Confirm both landed so GDAL_DATA / PROJ_LIB resolve.
    test -f /layer/${SITE}/rasterio/gdal_data/gdalvrt.xsd
    test -f /layer/${SITE}/pyproj/proj_dir/share/proj/proj.db

    # Strip debug symbols from every bundled shared object. rasterio.libs holds
    # libgdal, libproj, libgeos and friends; --strip-unneeded keeps the dynamic
    # symbol table intact so dlopen still resolves them at import.
    find /layer/${SITE} -name '*.so*' -exec strip --strip-unneeded {} + 2>/dev/null || true

    # Drop bytecode and test trees the wheel carries — pure size, zero runtime value.
    find /layer/${SITE} -type d -name '__pycache__' -prune -exec rm -rf {} +
    find /layer/${SITE} -type d -name 'tests' -prune -exec rm -rf {} +
  "

# Zip with the python/ prefix at the archive root — Lambda mounts this at /opt.
cd "${LAYER_DIR}" && zip -r9 ../rasterio-layer.zip python/ >/dev/null && cd ..
echo "Unzipped: $(du -sh "${LAYER_DIR}/python" | cut -f1)   Zipped: $(du -sh rasterio-layer.zip | cut -f1)"

# Stage to S3 (the zip is > 50 MB, so direct --zip-file upload is rejected).
aws s3 cp rasterio-layer.zip s3://my-lambda-artifacts/layers/rasterio-1-4-3.zip

# Register the new layer version and capture its ARN for downstream IaC.
aws lambda publish-layer-version \
  --layer-name "rasterio-1-4-3-py311-x86_64" \
  --description "rasterio ${RASTERIO_VERSION} on Amazon Linux 2023, stripped" \
  --content "S3Bucket=my-lambda-artifacts,S3Key=layers/rasterio-1-4-3.zip" \
  --compatible-runtimes python3.11 \
  --compatible-architectures x86_64

Verification

Mount the built layer directory as /opt in the same AL2023 image and confirm every import resolves and the data paths are live. This mirrors exactly what the function sees at runtime.

bash 
docker run --rm \
  -v "$(pwd)/layer:/opt:ro" \
  -e GDAL_DATA=/opt/python/lib/python3.11/site-packages/rasterio/gdal_data \
  -e PROJ_LIB=/opt/python/lib/python3.11/site-packages/pyproj/proj_dir/share/proj \
  -e LD_LIBRARY_PATH=/opt/python/lib/python3.11/site-packages/rasterio.libs \
  -e PYTHONPATH=/opt/python/lib/python3.11/site-packages \
  public.ecr.aws/lambda/python:3.11 \
  python3 -c "
import rasterio
from rasterio.crs import CRS
from osgeo import gdal

# Driver registration + a CRS lookup proves gdal_data and proj.db are reachable.
gdal.AllRegister()
crs = CRS.from_epsg(3857)
print(f'rasterio {rasterio.__version__} OK')
print(f'GDAL     {gdal.__version__} drivers={gdal.GetDriverCount()}')
print(f'CRS      {crs.to_authority()} -> {crs.linear_units}')
print('LAYER VERIFIED')
"

Expected output:

code 
rasterio 1.4.3 OK
GDAL     3.9.0 drivers=247
CRS      ('EPSG', '3857') -> metre
LAYER VERIFIED

A non-zero driver count and a resolved linear unit confirm both GDAL_DATA and PROJ_LIB point at real data. If GetDriverCount() returns a suspiciously low number or the CRS lookup throws, the data directories did not get bundled at the paths your environment variables name.

Gotchas and Edge Cases

  • --platform forces --only-binary, and that is the point. When you pass --platform to pip you must also pass --only-binary :all: or pip errors out. Embrace it: the whole reason to build a wheel-based layer is to avoid a source compile. If pip reports “no matching distribution,” the version you pinned has no manylinux2014 wheel for cp311 — bump to a release that does rather than dropping the platform flag.
  • The zip almost always exceeds the 50 MB direct-upload limit. aws lambda publish-layer-version --zip-file fileb://... caps at 50 MB. A rasterio layer zips to 55-75 MB, so stage it to S3 and use --content S3Bucket=... as shown. The separate 250 MB limit applies to the unzipped size across all attached layers.
  • arm64 needs its own wheel and its own build. A layer built for x86_64 will not load on a Graviton function. Add --platform manylinux2014_aarch64, --platform linux/arm64 on the container, and publish with --compatible-architectures arm64. Keep the two layers as distinct versions.
  • Adding shapely or pyproj means deduplicating numpy. rasterio vendors its own numpy; stacking more geospatial wheels on top multiplies it. Pin one numpy in a base layer and install the rest with --no-deps as described in deduplicating numpy across geospatial Lambda layers, or the combined unzipped size creeps toward the 250 MB wall.

Frequently Asked Questions

Why use the public.ecr.aws/lambda/python image instead of amazonlinux:2023?

The public.ecr.aws/lambda/python:3.11 image is the exact runtime AWS mounts for your function — the same glibc build, the same dynamic linker path, and a preinstalled Python that matches byte-for-byte. A bare amazonlinux:2023 image can carry slightly different minor library versions, and that gap is enough to surface a GLIBC or missing-symbol error at invocation that never appeared during your local test.

Do I need to set GDAL_DATA and PROJ_LIB if rasterio bundles its own data?

Yes. The manylinux wheel physically includes gdal_data and pyproj’s proj database, but the runtime only finds them when GDAL_DATA and PROJ_LIB point at the extracted /opt paths. Without those variables set on the function, GDAL raises DataDirNotFoundError or silently falls back to an incomplete datum grid, which corrupts reprojection results without an obvious error.

How large is a rasterio-only Lambda layer after stripping?

A rasterio 1.4.x wheel with its bundled GDAL and PROJ data unzips to roughly 120-160 MB. After strip --strip-unneeded runs across the .so files in rasterio.libs, the unzipped footprint drops to around 90-110 MB — comfortably under the 250 MB per-layer ceiling with headroom for shapely or pyproj alongside it, provided you avoid duplicating numpy.

Back to Native Library Compilation for Serverless