Deduplicating NumPy Across Geospatial Lambda Layers

Rasterio, shapely, and scipy each declare NumPy as a dependency, so a naive multi-layer install lands three separate ~40 MB NumPy copies. Pin one NumPy build in a shared base layer, then install every other package with pip install --no-deps against it — reclaiming 70-90 MB unzipped. The single NumPy must satisfy every package’s compiled C-API expectation, so pin it to a version at or above the highest each package was built against, and verify with an import smoke test in the Amazon Linux 2023 container before publishing.

Context

Every serious raster or vector stack sits on NumPy. rasterio returns bands as NumPy arrays, shapely 2.x vectorizes geometry operations through NumPy, and scipy — pulled in for interpolation, filtering, or spatial KD-trees — depends on it too. Because each of these declares NumPy in its own metadata, the default layer build workflow resolves and installs NumPy separately for each package when they are split across layers or installed in isolated passes. NumPy is not small: a single install unzips to 35-45 MB once you count its bundled OpenBLAS. Carry three copies and you have burned over 100 MB of your 250 MB budget on identical bytes.

The fix is a shared base layer holding exactly one NumPy, with every geospatial package installed on top using pip --no-deps so pip’s resolver never pulls a second copy. This is the same discipline as stripping unnecessary Python packages, applied to duplication rather than dead weight, and it composes cleanly with a wheel-based rasterio layer or a slim GDAL build. The single subtlety is the NumPy C ABI: compiled extensions embed the NumPy version they built against, so the one shared copy must be new enough for all of them.

Naive per-layer NumPy compared with a shared base-layer NumPyTwo panels. The naive build gives rasterio, shapely and scipy a NumPy copy each, 40 MB apiece for 120 MB of identical bytes and three ABI versions that can drift. The deduplicated build pins numpy 2.0.2 in a base layer and installs the three packages with pip --no-deps, reclaiming 80 MB and leaving one ABI to satisfy.One NumPy per layer, or one NumPy for the whole functionNaive: NumPy per packagerasterio 1.4.3 resolves and installs its own NumPy — 40 MBshapely 2.0.6 lands a second copy — 40 MBscipy 1.13.1 lands a third — 40 MB120 MB of the 250 MB budget is identical bytesThree NumPy versions that can silently divergeBase layer + pip --no-depsOne pinned numpy==2.0.2 in a base layer — 40 MBrasterio, shapely and scipy installed with --no-depsBuild fails if numpy appears in the geo layer80 MB reclaimed unzippedOne C-ABI version every extension must matchSame four packages, 80 MB less of the 250 MB unzipped budget — at the cost of owning every transitive dependencyyourself.
The saving is not compression — it is the same 40 MB of NumPy counted once instead of three times against the 250 MB unzipped limit.

Prerequisites

  • Runtime: Python 3.11 on x86_64, Amazon Linux 2023 target
  • Build image: public.ecr.aws/lambda/python:3.11 so compiled extensions match the Lambda ABI
  • Pinned versions: numpy==2.0.2, rasterio==1.4.3, shapely==2.0.6, scipy==1.13.1 — pin NumPy at or above the highest version each package was compiled against
  • Tools: pip, du, docker, and the AWS CLI v2
  • Layer layout: base and package installs both under python/lib/python3.11/site-packages/, resolved via /opt when both layers attach
  • Runtime environment variables on the function:
    code 
    PYTHONPATH=/opt/python/lib/python3.11/site-packages
GDAL_DATA=/opt/python/lib/python3.11/site-packages/rasterio/gdal_data
PROJ_LIB=/opt/python/lib/python3.11/site-packages/pyproj/proj_dir/share/proj
LD_LIBRARY_PATH=/opt/python/lib/python3.11/site-packages/rasterio.libs
  • IAM for upload: lambda:PublishLayerVersion on both layer ARNs and s3:PutObject on the staging bucket

Implementation

Build two layers. The base layer contains only NumPy. The geospatial layer installs everything else with --no-deps, meaning you take on responsibility for every transitive dependency — so the script installs the real non-NumPy dependencies explicitly and then adds the top-level packages without the resolver.

Unzipped composition of the deduplicated base and geospatial layersA proportional stack of the deduplicated function: a 40 MB base layer holding numpy 2.0.2 with OpenBLAS, then the geospatial layer's rasterio and GDAL at 96 MB, pyproj with proj.db at 34 MB, scipy at 34 MB and shapely with libgeos at 14 MB, a 4 MB handler, and 28 MB of headroom under the 250 MB ceiling.What the two layers put inside the 250 MB budgetBase layer — numpy 2.0.2the single shared copy, plus OpenBLAS40 MBGeo layer — rasterio 1.4.3bundled GDAL and rasterio.libs96 MBGeo layer — pyproj 3.7.0proj.db and transformation grids34 MBGeo layer — scipy 1.13.1installed with --no-deps34 MBGeo layer — shapely 2.0.6libgeos, plus affine, cligj, attrs, certifi14 MBHandler codeyour function, no vendored dependencies4 MBHeadroomwhat is left for the next dependency bump28 MBWeights are unzipped megabytes. The 250 MB AWS Lambda limit counts function code and every attached layer together, so both ARNs aremeasured as one total.
The headroom is 28 MB. Carrying the two extra NumPy copies would have cost 80 MB, so the naive build does not overflow the budget by a little — it never fits at all.
bash 
#!/usr/bin/env bash
# build_dedup_layers.sh — one NumPy base layer + a --no-deps geospatial layer.
set -euo pipefail

SITE="python/lib/python3.11/site-packages"
BASE="$(pwd)/base-layer"
GEO="$(pwd)/geo-layer"
rm -rf "${BASE}" "${GEO}" && mkdir -p "${BASE}" "${GEO}"

docker run --rm --platform linux/amd64 \
  -v "${BASE}:/base" -v "${GEO}:/geo" \
  public.ecr.aws/lambda/python:3.11 \
  bash -c "
    set -euo pipefail

    # 1) The single shared NumPy. Nothing else goes in the base layer.
    pip install --quiet --target /base/${SITE} numpy==2.0.2

    # 2) Install the geospatial packages' *real* non-numpy dependencies first,
    #    with --no-deps so numpy is never pulled a second time. pyproj, affine,
    #    click, certifi, attrs are the transitive set rasterio/shapely need.
    pip install --quiet --no-deps --target /geo/${SITE} \
      pyproj==3.7.0 affine==2.4.0 click==8.1.7 \
      cligj==0.7.2 click-plugins==1.1.1 attrs==24.2.0 certifi==2024.8.30

    # 3) Install the top-level packages themselves, still --no-deps. They compile
    #    against the numpy already staged in the base layer at import time.
    PYTHONPATH=/base/${SITE} pip install --quiet --no-deps --target /geo/${SITE} \
      rasterio==1.4.3 shapely==2.0.6 scipy==1.13.1

    # 4) Guarantee numpy did NOT sneak into the geo layer.
    if [ -d /geo/${SITE}/numpy ]; then
      echo 'ERROR: numpy leaked into the geo layer' >&2; exit 1
    fi
    echo 'geo layer is numpy-free — single copy lives in the base layer'
  "

# Zip and publish both layers; attach BOTH to the function (base first).
for name in base geo; do
  dir="$(pwd)/${name}-layer"
  (cd "${dir}" && zip -qr9 "../${name}-layer.zip" python/)
  aws s3 cp "${name}-layer.zip" "s3://my-lambda-artifacts/layers/${name}.zip"
  aws lambda publish-layer-version \
    --layer-name "geo-${name}-py311-x86_64" \
    --content "S3Bucket=my-lambda-artifacts,S3Key=layers/${name}.zip" \
    --compatible-runtimes python3.11 \
    --compatible-architectures x86_64
done

Verification

First audit the savings with du, then prove the single shared NumPy satisfies every compiled extension’s ABI. The ABI check is the one that catches a NumPy pinned too low.

bash 
# Size audit: base holds numpy, geo holds none of it.
du -sh base-layer/python | sed 's/$/  (base: numpy only)/'
du -sh geo-layer/python  | sed 's/$/  (geo: no numpy)/'
du -sh geo-layer/python/lib/python3.11/site-packages/numpy 2>/dev/null \
  || echo "0  (confirmed: no numpy in geo layer)"

# ABI + import smoke test with both layers stacked as /opt.
docker run --rm \
  -v "$(pwd)/base-layer:/opt/base:ro" \
  -v "$(pwd)/geo-layer:/opt/geo:ro" \
  -e PYTHONPATH="/opt/geo/python/lib/python3.11/site-packages:/opt/base/python/lib/python3.11/site-packages" \
  -e GDAL_DATA=/opt/geo/python/lib/python3.11/site-packages/rasterio/gdal_data \
  -e PROJ_LIB=/opt/geo/python/lib/python3.11/site-packages/pyproj/proj_dir/share/proj \
  -e LD_LIBRARY_PATH=/opt/geo/python/lib/python3.11/site-packages/rasterio.libs \
  public.ecr.aws/lambda/python:3.11 \
  python3 -c "
import numpy, rasterio, shapely, scipy
from shapely.geometry import Point

# If any extension was built against a newer numpy than the base pin, THIS line
# raises: 'module compiled against API version ... but this version of numpy is ...'
arr = rasterio.__version__ and numpy.arange(6).reshape(2, 3)
buf = Point(0, 0).buffer(1.0)            # exercises shapely's numpy vectorization
from scipy.spatial import cKDTree        # exercises scipy's numpy C-API binding

print(f'numpy    {numpy.__version__} (single shared copy)')
print(f'rasterio {rasterio.__version__} OK')
print(f'shapely  {shapely.__version__} OK  area={buf.area:.3f}')
print(f'scipy    {scipy.__version__} OK')
print('ABI COMPATIBLE — one numpy satisfies all extensions')
"

Expected output:

code 
0  (confirmed: no numpy in geo layer)
numpy    2.0.2 (single shared copy)
rasterio 1.4.3 OK
shapely  2.0.6 OK  area=3.137
scipy    1.13.1 OK
ABI COMPATIBLE — one numpy satisfies all extensions
Deduplicated layer set measured against the AWS Lambda quotasThree meters: the duplicated build at 302 MB against the 250 MB unzipped limit, over the ceiling and rejected; the deduplicated build at 222 MB, inside it; and two of the five layer slots AWS permits per function consumed by splitting NumPy into a base layer.The same stack, before and after the duplicate NumPy comes outThree NumPy copiescode plus all layers, unzipped302 / 250 MBOne shared NumPysame four packages, same handler222 / 250 MBLayer slots consumedbase and geo, of the five per function2 of 5AWS rejects the function update when code plus every attached layer exceeds 250 MB unzipped — the duplicated build is 52 MB over beforeanything else is added.
Deduplication buys 80 MB and spends one layer slot. With a GDAL layer and a monitoring extension already attached, that slot is the resource to watch — not the megabytes.

Gotchas and Edge Cases

  • Pin NumPy too low and the ABI check fails at import. A compiled extension built against NumPy 2.0 raises module compiled against API version 0x10 but this version of numpy is 0x... when loaded against an older NumPy. Always pin the shared NumPy at or above the newest version any package in the stack was built against; when in doubt, match the version the fat wheel would have installed.
  • --no-deps makes you the resolver. Because pip no longer walks the dependency graph, a forgotten transitive package — cligj, attrs, affine — surfaces as an ImportError only at runtime. List every real dependency explicitly as the script does, and let the import smoke test catch omissions before you publish.
  • Both layers must attach, and PYTHONPATH must reach both. With NumPy in a separate layer, the function needs both layer ARNs attached and PYTHONPATH covering both site-packages roots. Miss the base layer and every import dies with ModuleNotFoundError: No module named 'numpy'. This is the recurring failure when someone redeploys with only the geospatial layer.
  • Five-layer limit still applies. Splitting NumPy into its own layer consumes one of the five layer slots AWS allows per function. If you already run a GDAL layer and a monitoring extension, budget the slots deliberately; consolidating into a container image is the escape hatch when you run out.

Frequently Asked Questions

Why does each geospatial package ship its own NumPy?

They do not vendor NumPy’s source, but rasterio, shapely, and scipy each declare NumPy as a dependency, so a naive pip install resolves and installs it once per target directory when the packages land in separate layers or install passes. Because they are compiled against NumPy’s C API, their .so files also embed a NumPy ABI expectation — which is why the single shared copy must be new enough to satisfy every package.

How much space does deduplicating NumPy actually save?

A single NumPy install unzips to roughly 35-45 MB including its bundled OpenBLAS. Three separate copies across separate layers cost 105-135 MB; collapsing to one shared copy reclaims 70-90 MB unzipped. On a stack already near the 250 MB per-layer ceiling that is frequently the difference between a layer that deploys and one that is rejected.

Is pip --no-deps safe for rasterio and shapely?

It is safe provided you have already installed every real dependency yourself. --no-deps disables pip’s resolver, so you own providing NumPy and each transitive requirement at compatible versions. Verify with an import smoke test in the Amazon Linux 2023 container — a missing or ABI-incompatible NumPy surfaces immediately as an ImportError at load rather than silently later.

Back to Python Layer Management and Size Reduction