Building Rasterio Lambda Layers on Amazon Linux 2023

Build the layer inside public.ecr.aws/lambda/python:3.11 — the Amazon Linux 2023 image AWS actually runs — and install with pip install --only-binary :all: --platform manylinux2014_x86_64 --target python/lib/python3.11/site-packages rasterio==1.4.3 so pip fetches the prebuilt GDAL wheel instead of compiling from source. Then copy the bundled gdal_data and proj trees into the layer, run strip --strip-unneeded on every .so, zip the python/ prefix, and register it with aws lambda publish-layer-version. A rasterio-only layer built this way unzips to roughly 90-110 MB after stripping — well inside the 250 MB unzipped ceiling.

Context

The native library compilation discipline draws a sharp line between two ways to get GDAL into a Lambda: compile the C toolchain from source, or lean on the manylinux wheels that the rasterio maintainers already publish. For the overwhelming majority of raster workloads the second path is correct — the prebuilt wheel bundles a tested GDAL, PROJ, GEOS, and their data files, and it installs in seconds. The only hard requirement is that you assemble it inside an environment that matches the Lambda runtime’s C ABI, which on modern functions is Amazon Linux 2023 with glibc 2.34.

The trap people fall into is running pip install rasterio on a macOS or Ubuntu laptop and zipping the result. That produces a layer whose .so files link against the wrong libc, and the function fails at cold start with GLIBC_2.38 not found or an ImportError on libgdal.so before your handler runs. Building inside public.ecr.aws/lambda/python:3.11 removes that entire class of failure because it is the same image AWS mounts. This page is the wheel-based counterpart to the from-source recipe in the parent guide, and it pairs naturally with stripping unnecessary Python packages once the layer exists.

Building the layer on a laptop against building it in the Lambda imageTwo panels comparing a layer built on a macOS or Ubuntu laptop with one built inside public.ecr.aws/lambda/python:3.11. The laptop panel lists the wrong libc, a build that appears to succeed, and a GLIBC or ImportError failure at the first cold start. The container panel lists the exact glibc 2.34 runtime, a matching preinstalled Python, the pinned manylinux2014 platform tag, and an import that resolves on the first invocation.The same pip command, two incompatible C ABIsBuilt on a laptop (macOS or Ubuntu)pip resolves a wheel for the host libc, not the runtime onemacOS links libSystem; a current Ubuntu links a newer glibcthan 2.34The build succeeds and the zip looks correct — nothingwarns youThe first cold start raises GLIBC_X.XX not found orImportError on libgdal.soBuilt in public.ecr.aws/lambda/python:3.11The exact image AWS mounts: Amazon Linux 2023 with glibc2.34Same dynamic linker path and a preinstalled Python thatmatches the runtime--platform manylinux2014_x86_64 pins the wheel ABIexplicitlyThe import resolves on the first cold start, and every oneafter itBuilding inside the runtime image does not reduce the chance of an ABI failure — it removes the entire class, because thebuild environment and the execution environment are the same image.
Nothing in the left-hand column fails at build time. The zip is well formed, the file listing looks right, and the error arrives on the first invocation in production.

Prerequisites

Confirm each of these before you start the build:

  • Runtime: Python 3.11 on x86_64 (add --platform linux/arm64 and a Graviton wheel tag for arm64)
  • Build image: public.ecr.aws/lambda/python:3.11 pulled locally — this is Amazon Linux 2023
  • Pinned versions: rasterio==1.4.3, which vendors GDAL 3.9.x and PROJ 9.x in its manylinux wheel
  • System tools: strip (GNU binutils), zip, find, docker, and the AWS CLI v2
  • Layer layout: everything under python/lib/python3.11/site-packages/ so Lambda finds it when the layer mounts at /opt
  • Runtime environment variables (set on the function, not the build):
    code 
    GDAL_DATA=/opt/python/lib/python3.11/site-packages/rasterio/gdal_data
PROJ_LIB=/opt/python/lib/python3.11/site-packages/pyproj/proj_dir/share/proj
LD_LIBRARY_PATH=/opt/python/lib/python3.11/site-packages/rasterio.libs
  • IAM for the upload step: lambda:PublishLayerVersion on the target layer ARN, plus s3:PutObject on a staging bucket because the zip exceeds the 50 MB direct-upload threshold

Implementation

The script below runs entirely inside the AL2023 container. It pins the platform tag so pip resolves the prebuilt manylinux wheel rather than triggering a source build, copies the data directories to predictable paths, strips debug symbols, and lays out the python/ prefix.

Rasterio Lambda layer build sequence on Amazon Linux 2023Five build steps in order: launch the Amazon Linux 2023 Lambda container, install rasterio with the manylinux2014 platform tag and only-binary so no source build occurs, assert that the bundled GDAL and PROJ data directories landed, strip debug symbols from every shared object in rasterio.libs, then zip the python prefix and publish the layer version from S3.Five steps, all of them inside the Amazon Linux 2023 image1Launch the AL2023 containerdocker run --platform linux/amd64 against the image AWS actually mountslambda/python:3.112Install under manylinux constraints--only-binary :all: with --platform, --python-version, --implementation and --abi pinnedrasterio==1.4.33Assert the data trees landedgdalvrt.xsd and proj.db must exist, or GDAL_DATA and PROJ_LIB will point at nothinggdal_data + proj.db4Strip debug symbolsacross rasterio.libs — 120-160 MB unzipped drops to 90-110 MB--strip-unneeded5Zip the python/ prefix and publishstage to S3 first: the archive is past the 50 MB direct-upload thresholdpublish-layer-version
Step 3 is an assertion, not a copy: the wheel already carries gdal_data and proj.db, and the build fails loudly if they are not where GDAL_DATA and PROJ_LIB will point.
bash 
#!/usr/bin/env bash
# build_rasterio_layer.sh — run on the host; it drives the AL2023 container.
set -euo pipefail

RASTERIO_VERSION="1.4.3"
PY="python3.11"
LAYER_DIR="$(pwd)/layer"
SITE="python/lib/${PY}/site-packages"

rm -rf "${LAYER_DIR}" && mkdir -p "${LAYER_DIR}"

docker run --rm \
  --platform linux/amd64 \
  -v "${LAYER_DIR}:/layer" \
  public.ecr.aws/lambda/python:3.11 \
  bash -c "
    set -euo pipefail
    dnf install -y binutils >/dev/null   # provides 'strip' on the minimal image

    # --only-binary :all: forbids any source build; --platform pins the ABI tag
    # so pip downloads the manylinux2014_x86_64 wheel that vendors GDAL + PROJ.
    pip install \
      --only-binary :all: \
      --platform manylinux2014_x86_64 \
      --python-version 3.11 \
      --implementation cp \
      --abi cp311 \
      --target /layer/${SITE} \
      rasterio==${RASTERIO_VERSION}

    # rasterio ships gdal_data (EPSG lookups, driver metadata) and pyproj ships
    # its proj database. Confirm both landed so GDAL_DATA / PROJ_LIB resolve.
    test -f /layer/${SITE}/rasterio/gdal_data/gdalvrt.xsd
    test -f /layer/${SITE}/pyproj/proj_dir/share/proj/proj.db

    # Strip debug symbols from every bundled shared object. rasterio.libs holds
    # libgdal, libproj, libgeos and friends; --strip-unneeded keeps the dynamic
    # symbol table intact so dlopen still resolves them at import.
    find /layer/${SITE} -name '*.so*' -exec strip --strip-unneeded {} + 2>/dev/null || true

    # Drop bytecode and test trees the wheel carries — pure size, zero runtime value.
    find /layer/${SITE} -type d -name '__pycache__' -prune -exec rm -rf {} +
    find /layer/${SITE} -type d -name 'tests' -prune -exec rm -rf {} +
  "

# Zip with the python/ prefix at the archive root — Lambda mounts this at /opt.
cd "${LAYER_DIR}" && zip -r9 ../rasterio-layer.zip python/ >/dev/null && cd ..
echo "Unzipped: $(du -sh "${LAYER_DIR}/python" | cut -f1)   Zipped: $(du -sh rasterio-layer.zip | cut -f1)"

# Stage to S3 (the zip is > 50 MB, so direct --zip-file upload is rejected).
aws s3 cp rasterio-layer.zip s3://my-lambda-artifacts/layers/rasterio-1-4-3.zip

# Register the new layer version and capture its ARN for downstream IaC.
aws lambda publish-layer-version \
  --layer-name "rasterio-1-4-3-py311-x86_64" \
  --description "rasterio ${RASTERIO_VERSION} on Amazon Linux 2023, stripped" \
  --content "S3Bucket=my-lambda-artifacts,S3Key=layers/rasterio-1-4-3.zip" \
  --compatible-runtimes python3.11 \
  --compatible-architectures x86_64

Verification

Mount the built layer directory as /opt in the same AL2023 image and confirm every import resolves and the data paths are live. This mirrors exactly what the function sees at runtime.

bash 
docker run --rm \
  -v "$(pwd)/layer:/opt:ro" \
  -e GDAL_DATA=/opt/python/lib/python3.11/site-packages/rasterio/gdal_data \
  -e PROJ_LIB=/opt/python/lib/python3.11/site-packages/pyproj/proj_dir/share/proj \
  -e LD_LIBRARY_PATH=/opt/python/lib/python3.11/site-packages/rasterio.libs \
  -e PYTHONPATH=/opt/python/lib/python3.11/site-packages \
  public.ecr.aws/lambda/python:3.11 \
  python3 -c "
import rasterio
from rasterio.crs import CRS
from osgeo import gdal

# Driver registration + a CRS lookup proves gdal_data and proj.db are reachable.
gdal.AllRegister()
crs = CRS.from_epsg(3857)
print(f'rasterio {rasterio.__version__} OK')
print(f'GDAL     {gdal.__version__} drivers={gdal.GetDriverCount()}')
print(f'CRS      {crs.to_authority()} -> {crs.linear_units}')
print('LAYER VERIFIED')
"

Expected output:

code 
rasterio 1.4.3 OK
GDAL     3.9.0 drivers=247
CRS      ('EPSG', '3857') -> metre
LAYER VERIFIED

A non-zero driver count and a resolved linear unit confirm both GDAL_DATA and PROJ_LIB point at real data. If GetDriverCount() returns a suspiciously low number or the CRS lookup throws, the data directories did not get bundled at the paths your environment variables name.

Gotchas and Edge Cases

  • --platform forces --only-binary, and that is the point. When you pass --platform to pip you must also pass --only-binary :all: or pip errors out. Embrace it: the whole reason to build a wheel-based layer is to avoid a source compile. If pip reports “no matching distribution,” the version you pinned has no manylinux2014 wheel for cp311 — bump to a release that does rather than dropping the platform flag.
  • The zip almost always exceeds the 50 MB direct-upload limit. aws lambda publish-layer-version --zip-file fileb://... caps at 50 MB. A rasterio layer zips to 55-75 MB, so stage it to S3 and use --content S3Bucket=... as shown. The separate 250 MB limit applies to the unzipped size across all attached layers.
  • arm64 needs its own wheel and its own build. A layer built for x86_64 will not load on a Graviton function. Add --platform manylinux2014_aarch64, --platform linux/arm64 on the container, and publish with --compatible-architectures arm64. Keep the two layers as distinct versions.
  • Adding shapely or pyproj means deduplicating numpy. rasterio vendors its own numpy; stacking more geospatial wheels on top multiplies it. Pin one numpy in a base layer and install the rest with --no-deps as described in deduplicating numpy across geospatial Lambda layers, or the combined unzipped size creeps toward the 250 MB wall.
Rasterio layer size against the 250 MB unzipped and 50 MB upload limitsThree meters against their limits: the unzipped layer before stripping at 120 to 160 megabytes of the 250 megabyte per-layer ceiling, the unzipped layer after stripping at 90 to 110 megabytes of the same ceiling, and the zip archive at 55 to 75 megabytes against the 50 megabyte direct-upload cap, which it exceeds.Where the built layer sits against Lambda's two packaging limitsUnzipped, before strippingceiling 250 MB per layer120-160 MBUnzipped, after strippingceiling 250 MB per layer90-110 MBZip archive--zip-file caps at 50 MB55-75 MBThe zip exceeding 50 MB is not a failure — it is why publish-layer-version reads the archive from S3 rather than the request body. Strippingis what buys the headroom for shapely or pyproj beside rasterio.
Two different limits with two different remedies: stripping keeps the unzipped total under 250 MB, and S3 staging is the only way past the 50 MB direct-upload cap.

Frequently Asked Questions

Why use the public.ecr.aws/lambda/python image instead of amazonlinux:2023?

The public.ecr.aws/lambda/python:3.11 image is the exact runtime AWS mounts for your function — the same glibc build, the same dynamic linker path, and a preinstalled Python that matches byte-for-byte. A bare amazonlinux:2023 image can carry slightly different minor library versions, and that gap is enough to surface a GLIBC or missing-symbol error at invocation that never appeared during your local test.

Do I need to set GDAL_DATA and PROJ_LIB if rasterio bundles its own data?

Yes. The manylinux wheel physically includes gdal_data and pyproj’s proj database, but the runtime only finds them when GDAL_DATA and PROJ_LIB point at the extracted /opt paths. Without those variables set on the function, GDAL raises DataDirNotFoundError or silently falls back to an incomplete datum grid, which corrupts reprojection results without an obvious error.

How large is a rasterio-only Lambda layer after stripping?

A rasterio 1.4.x wheel with its bundled GDAL and PROJ data unzips to roughly 120-160 MB. After strip --strip-unneeded runs across the .so files in rasterio.libs, the unzipped footprint drops to around 90-110 MB — comfortably under the 250 MB per-layer ceiling with headroom for shapely or pyproj alongside it, provided you avoid duplicating numpy.

Back to Native Library Compilation for Serverless