Deduplicating NumPy Across Geospatial Lambda Layers

Rasterio, shapely, and scipy each declare NumPy as a dependency, so a naive multi-layer install lands three separate ~40 MB NumPy copies. Pin one NumPy build in a shared base layer, then install every other package with pip install --no-deps against it — reclaiming 70-90 MB unzipped. The single NumPy must satisfy every package’s compiled C-API expectation, so pin it to a version at or above the highest each package was built against, and verify with an import smoke test in the Amazon Linux 2023 container before publishing.

Duplicate NumPy versus shared base-layer NumPy Left: three stacked layers for rasterio, shapely, and scipy each containing its own NumPy copy. Right: a shared base layer holding one NumPy that all three packages install against with --no-deps, saving the duplicated megabytes. Naive: NumPy per package Deduplicated: one base layer rasterio numpy 40MB shapely numpy 40MB scipy numpy 40MB 3 x numpy = ~120 MB duplicated --no-deps base layer: numpy 40MB (shared) geo layer (--no-deps) rasterio shapely scipy 1 x numpy = ~80 MB reclaimed One shared NumPy under the combined 250 MB unzipped budget

Context

Every serious raster or vector stack sits on NumPy. rasterio returns bands as NumPy arrays, shapely 2.x vectorizes geometry operations through NumPy, and scipy — pulled in for interpolation, filtering, or spatial KD-trees — depends on it too. Because each of these declares NumPy in its own metadata, the default layer build workflow resolves and installs NumPy separately for each package when they are split across layers or installed in isolated passes. NumPy is not small: a single install unzips to 35-45 MB once you count its bundled OpenBLAS. Carry three copies and you have burned over 100 MB of your 250 MB budget on identical bytes.

The fix is a shared base layer holding exactly one NumPy, with every geospatial package installed on top using pip --no-deps so pip’s resolver never pulls a second copy. This is the same discipline as stripping unnecessary Python packages, applied to duplication rather than dead weight, and it composes cleanly with a wheel-based rasterio layer or a slim GDAL build. The single subtlety is the NumPy C ABI: compiled extensions embed the NumPy version they built against, so the one shared copy must be new enough for all of them.

Prerequisites

  • Runtime: Python 3.11 on x86_64, Amazon Linux 2023 target
  • Build image: public.ecr.aws/lambda/python:3.11 so compiled extensions match the Lambda ABI
  • Pinned versions: numpy==2.0.2, rasterio==1.4.3, shapely==2.0.6, scipy==1.13.1 — pin NumPy at or above the highest version each package was compiled against
  • Tools: pip, du, docker, and the AWS CLI v2
  • Layer layout: base and package installs both under python/lib/python3.11/site-packages/, resolved via /opt when both layers attach
  • Runtime environment variables on the function:
    code 
    PYTHONPATH=/opt/python/lib/python3.11/site-packages
GDAL_DATA=/opt/python/lib/python3.11/site-packages/rasterio/gdal_data
PROJ_LIB=/opt/python/lib/python3.11/site-packages/pyproj/proj_dir/share/proj
LD_LIBRARY_PATH=/opt/python/lib/python3.11/site-packages/rasterio.libs
  • IAM for upload: lambda:PublishLayerVersion on both layer ARNs and s3:PutObject on the staging bucket

Implementation

Build two layers. The base layer contains only NumPy. The geospatial layer installs everything else with --no-deps, meaning you take on responsibility for every transitive dependency — so the script installs the real non-NumPy dependencies explicitly and then adds the top-level packages without the resolver.

bash 
#!/usr/bin/env bash
# build_dedup_layers.sh — one NumPy base layer + a --no-deps geospatial layer.
set -euo pipefail

SITE="python/lib/python3.11/site-packages"
BASE="$(pwd)/base-layer"
GEO="$(pwd)/geo-layer"
rm -rf "${BASE}" "${GEO}" && mkdir -p "${BASE}" "${GEO}"

docker run --rm --platform linux/amd64 \
  -v "${BASE}:/base" -v "${GEO}:/geo" \
  public.ecr.aws/lambda/python:3.11 \
  bash -c "
    set -euo pipefail

    # 1) The single shared NumPy. Nothing else goes in the base layer.
    pip install --quiet --target /base/${SITE} numpy==2.0.2

    # 2) Install the geospatial packages' *real* non-numpy dependencies first,
    #    with --no-deps so numpy is never pulled a second time. pyproj, affine,
    #    click, certifi, attrs are the transitive set rasterio/shapely need.
    pip install --quiet --no-deps --target /geo/${SITE} \
      pyproj==3.7.0 affine==2.4.0 click==8.1.7 \
      cligj==0.7.2 click-plugins==1.1.1 attrs==24.2.0 certifi==2024.8.30

    # 3) Install the top-level packages themselves, still --no-deps. They compile
    #    against the numpy already staged in the base layer at import time.
    PYTHONPATH=/base/${SITE} pip install --quiet --no-deps --target /geo/${SITE} \
      rasterio==1.4.3 shapely==2.0.6 scipy==1.13.1

    # 4) Guarantee numpy did NOT sneak into the geo layer.
    if [ -d /geo/${SITE}/numpy ]; then
      echo 'ERROR: numpy leaked into the geo layer' >&2; exit 1
    fi
    echo 'geo layer is numpy-free — single copy lives in the base layer'
  "

# Zip and publish both layers; attach BOTH to the function (base first).
for name in base geo; do
  dir="$(pwd)/${name}-layer"
  (cd "${dir}" && zip -qr9 "../${name}-layer.zip" python/)
  aws s3 cp "${name}-layer.zip" "s3://my-lambda-artifacts/layers/${name}.zip"
  aws lambda publish-layer-version \
    --layer-name "geo-${name}-py311-x86_64" \
    --content "S3Bucket=my-lambda-artifacts,S3Key=layers/${name}.zip" \
    --compatible-runtimes python3.11 \
    --compatible-architectures x86_64
done

Verification

First audit the savings with du, then prove the single shared NumPy satisfies every compiled extension’s ABI. The ABI check is the one that catches a NumPy pinned too low.

bash 
# Size audit: base holds numpy, geo holds none of it.
du -sh base-layer/python | sed 's/$/  (base: numpy only)/'
du -sh geo-layer/python  | sed 's/$/  (geo: no numpy)/'
du -sh geo-layer/python/lib/python3.11/site-packages/numpy 2>/dev/null \
  || echo "0  (confirmed: no numpy in geo layer)"

# ABI + import smoke test with both layers stacked as /opt.
docker run --rm \
  -v "$(pwd)/base-layer:/opt/base:ro" \
  -v "$(pwd)/geo-layer:/opt/geo:ro" \
  -e PYTHONPATH="/opt/geo/python/lib/python3.11/site-packages:/opt/base/python/lib/python3.11/site-packages" \
  -e GDAL_DATA=/opt/geo/python/lib/python3.11/site-packages/rasterio/gdal_data \
  -e PROJ_LIB=/opt/geo/python/lib/python3.11/site-packages/pyproj/proj_dir/share/proj \
  -e LD_LIBRARY_PATH=/opt/geo/python/lib/python3.11/site-packages/rasterio.libs \
  public.ecr.aws/lambda/python:3.11 \
  python3 -c "
import numpy, rasterio, shapely, scipy
from shapely.geometry import Point

# If any extension was built against a newer numpy than the base pin, THIS line
# raises: 'module compiled against API version ... but this version of numpy is ...'
arr = rasterio.__version__ and numpy.arange(6).reshape(2, 3)
buf = Point(0, 0).buffer(1.0)            # exercises shapely's numpy vectorization
from scipy.spatial import cKDTree        # exercises scipy's numpy C-API binding

print(f'numpy    {numpy.__version__} (single shared copy)')
print(f'rasterio {rasterio.__version__} OK')
print(f'shapely  {shapely.__version__} OK  area={buf.area:.3f}')
print(f'scipy    {scipy.__version__} OK')
print('ABI COMPATIBLE — one numpy satisfies all extensions')
"

Expected output:

code 
0  (confirmed: no numpy in geo layer)
numpy    2.0.2 (single shared copy)
rasterio 1.4.3 OK
shapely  2.0.6 OK  area=3.137
scipy    1.13.1 OK
ABI COMPATIBLE — one numpy satisfies all extensions

Gotchas and Edge Cases

  • Pin NumPy too low and the ABI check fails at import. A compiled extension built against NumPy 2.0 raises module compiled against API version 0x10 but this version of numpy is 0x... when loaded against an older NumPy. Always pin the shared NumPy at or above the newest version any package in the stack was built against; when in doubt, match the version the fat wheel would have installed.
  • --no-deps makes you the resolver. Because pip no longer walks the dependency graph, a forgotten transitive package — cligj, attrs, affine — surfaces as an ImportError only at runtime. List every real dependency explicitly as the script does, and let the import smoke test catch omissions before you publish.
  • Both layers must attach, and PYTHONPATH must reach both. With NumPy in a separate layer, the function needs both layer ARNs attached and PYTHONPATH covering both site-packages roots. Miss the base layer and every import dies with ModuleNotFoundError: No module named 'numpy'. This is the recurring failure when someone redeploys with only the geospatial layer.
  • Five-layer limit still applies. Splitting NumPy into its own layer consumes one of the five layer slots AWS allows per function. If you already run a GDAL layer and a monitoring extension, budget the slots deliberately; consolidating into a container image is the escape hatch when you run out.

Frequently Asked Questions

Why does each geospatial package ship its own NumPy?

They do not vendor NumPy’s source, but rasterio, shapely, and scipy each declare NumPy as a dependency, so a naive pip install resolves and installs it once per target directory when the packages land in separate layers or install passes. Because they are compiled against NumPy’s C API, their .so files also embed a NumPy ABI expectation — which is why the single shared copy must be new enough to satisfy every package.

How much space does deduplicating NumPy actually save?

A single NumPy install unzips to roughly 35-45 MB including its bundled OpenBLAS. Three separate copies across separate layers cost 105-135 MB; collapsing to one shared copy reclaims 70-90 MB unzipped. On a stack already near the 250 MB per-layer ceiling that is frequently the difference between a layer that deploys and one that is rejected.

Is pip --no-deps safe for rasterio and shapely?

It is safe provided you have already installed every real dependency yourself. --no-deps disables pip’s resolver, so you own providing NumPy and each transitive requirement at compatible versions. Verify with an import smoke test in the Amazon Linux 2023 container — a missing or ABI-incompatible NumPy surfaces immediately as an ImportError at load rather than silently later.

Back to Python Layer Management and Size Reduction