Python Layer Management and Size Reduction for Serverless GIS

AWS Lambda enforces a hard 250 MB unzipped limit on the combined function code and attached layers. A bare pip install rasterio shapely pyproj fiona routinely produces 320–380 MB before any business logic is added. Closing that gap requires a disciplined four-stage pipeline: deterministic dependency resolution, architecture-matched wheel selection, aggressive binary pruning, and container-validated packaging. Teams that treat this pipeline as a one-time setup rather than a repeatable build step end up with environment drift, silent ABI mismatches, and deployment failures at the worst possible moment.

Why Layer Size Matters for Geospatial Workloads

Geospatial Python packages are not pure Python. rasterio, fiona, and pyproj each wrap compiled C/C++ extensions (libgdal, libgeos, libproj) and bundle supporting data: coordinate reference system grids, PROJ databases, and GDAL format drivers. These binary payloads inflate layer footprints far beyond what pip show suggests because pip show reports only the installed size on disk after native linking — it does not show what the linker resolved from the OS or what debug symbols remain embedded in each .so file.

The problem compounds with transitive dependencies. rasterio pulls numpy (typically 60–80 MB unzipped), certifi, click, attrs, and sometimes scipy depending on optional feature groups. Without explicit dependency isolation, a single pip install can introduce dozens of packages that are never imported by your function.

Beyond raw size, cold-start latency for Python GDAL stacks is directly proportional to the number of .so files the dynamic linker must resolve at initialization. A bloated layer with 800+ shared objects can add 4–8 seconds to every cold start. Stripping and pruning improve both package size and initialization time.

Ephemeral storage limits in AWS Lambda create a second pressure: large layers are extracted to /opt at function startup, which counts against the 512 MB default /tmp budget (expandable to 10 GB). If your function also writes intermediate GeoTIFFs to /tmp, a bloated layer can exhaust scratch space before processing begins.

Platform-by-Platform Limits

All three providers bill on memory allocation, not disk footprint — but a larger layer forces higher minimum memory to avoid OOM kills during GDAL driver registration. AWS Lambda automatically allocates CPU proportional to memory, so over-sized layers that require 2048 MB to avoid extraction failures also double the per-GB-second billing rate relative to a properly trimmed 1024 MB allocation.

Step-by-Step Implementation

1. Dependency Pinning and Isolation

Isolate geospatial requirements into a dedicated file, separate from application-level dependencies. Mixing boto3, requests, and rasterio in a single requirements.txt makes it impossible to reason about what belongs in the layer versus the function package.

# requirements-geo.txt — layer dependencies only
rasterio==1.4.3
shapely==2.0.6
pyproj==3.7.0
fiona==1.10.1
numpy==1.26.4

Compile to a lockfile using uv (preferred for speed) or pip-compile:

uv pip compile requirements-geo.txt \
  --python-version 3.11 \
  --output-file requirements-geo.lock

The lockfile pins every transitive dependency, including hash digests. Every CI/CD run that builds the layer uses this file, not the loose requirements-geo.txt. This is the foundation of reproducibility in the broader Packaging & Dependency Management for Serverless GIS workflow.

2. Architecture-Matched Wheel Resolution

Serverless runtimes execute on specific Linux ABI targets. AWS Lambda Python 3.11 on x86_64 runs on Amazon Linux 2023 (glibc 2.34, manylinux_2_28 compatible). GCP Cloud Functions 2nd gen uses Debian 11 (glibc 2.31). Azure Functions Consumption uses Ubuntu 20.04 (glibc 2.31).

Use pip with explicit platform and ABI tags to download only the binary wheels required for the target runtime — never resolve against your local machine’s architecture:

mkdir -p layer/python

pip install \
  --only-binary=:all: \
  --platform manylinux_2_28_x86_64 \
  --python-version 3.11 \
  --implementation cp \
  --abi cp311 \
  --target layer/python \
  --no-deps \
  -r requirements-geo.lock

The --no-deps flag is intentional here because the lockfile already captured the full dependency closure. Using --no-deps against a verified lockfile prevents pip from re-resolving and potentially pulling a newer transitive dependency that conflicts with the pinned versions.

For arm64 targets (Lambda Graviton3, GCP arm64 preview, Azure Premium EP3), swap manylinux_2_28_x86_64 for manylinux_2_28_aarch64. Graviton3 instances typically show 15–20% lower cold-start latency and ~20% better cost efficiency for compute-bound raster operations.

If a package requires custom C extensions not available as precompiled wheels — for example, a GDAL build with proprietary format drivers — consult the Native Library Compilation for Serverless patterns for sysroot alignment and static linking to avoid runtime symbol failures.

3. Binary Pruning and Symbol Stripping

After wheel download, the layer/python directory typically holds 280–350 MB of content. Pruning reduces this by 30–50%.

Remove non-runtime artifacts first:

cd layer/python

# Remove Python bytecache, dist metadata, and test suites
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
find . -type d -name "*.dist-info" -exec rm -rf {} + 2>/dev/null || true
find . -type d -name "*.egg-info" -exec rm -rf {} + 2>/dev/null || true
find . -type d -name "tests" -exec rm -rf {} + 2>/dev/null || true
find . -type d -name "test" -exec rm -rf {} + 2>/dev/null || true
find . -type f -name "*.pyc" -delete
find . -type f -name "*.c" -delete
find . -type f -name "*.h" -delete

# Remove duplicate PROJ databases (keep only the one pyproj expects)
# pyproj bundles proj.db; fiona may ship a second copy — confirm paths first
find . -path "*/fiona/proj_data/proj.db" -delete 2>/dev/null || true

Strip debug symbols from every compiled extension:

find . -name "*.so" -exec strip --strip-unneeded {} + 2>/dev/null || true

strip --strip-unneeded removes debugging symbols and local symbol table entries while preserving the symbol table entries needed for dynamic linking. On a typical geospatial stack, this step alone reduces .so file sizes by 40–60%, often saving 60–90 MB.

Validate that no symbols went missing:

find . -name "*.so" -exec ldd {} + 2>/dev/null | grep "not found"

A clean output (no lines printed) confirms all .so dependencies are either self-contained or available in the Lambda runtime OS. Libraries like libz, libcurl, libsqlite3, and libssl are present in Amazon Linux 2023 and do not need to be bundled.

For geospatial projection data, set explicit environment variables rather than bundling duplicate data files. The Docker Container Optimization for GIS multi-stage build patterns show how to isolate these data directories cleanly; the same approach applies to layer builds:

# In your Lambda handler — set before any geospatial import
import os
os.environ.setdefault("PROJ_LIB", "/opt/python/pyproj/proj_data")
os.environ.setdefault("GDAL_DATA", "/opt/python/rasterio/gdal_data")
os.environ.setdefault("GDAL_DRIVER_PATH", "")  # disable scanning for unused drivers
os.environ.setdefault("CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE", "NO")

Detailed methodologies for safely identifying and removing unused binaries without breaking import chains are in Stripping Unnecessary Python Packages from AWS Lambda Layers.

4. Layer Assembly and Local Validation

Package the pruned directory:

cd layer
zip -r9 ../geospatial-layer.zip python/
ls -lh ../geospatial-layer.zip   # should be under 100 MB compressed
unzip -l ../geospatial-layer.zip | tail -1  # confirm unzipped total < 250 MB

Validate imports inside a container that mirrors the production runtime before any cloud upload:

docker run --rm \
  -v "$(pwd)/geospatial-layer.zip:/opt/geospatial-layer.zip:ro" \
  public.ecr.aws/sam/build-python3.11:latest \
  bash -c "
    unzip -q /opt/geospatial-layer.zip -d /opt
    export PROJ_LIB=/opt/python/pyproj/proj_data
    export GDAL_DATA=/opt/python/rasterio/gdal_data
    python3 -c '
import rasterio
import shapely.geometry
import fiona
import pyproj
# Verify a live coordinate transformation — not just import
t = pyproj.Transformer.from_crs(\"EPSG:4326\", \"EPSG:3857\", always_xy=True)
x, y = t.transform(-74.006, 40.713)
assert abs(x - -8238310) < 1000, f\"Unexpected x: {x}\"
print(\"All imports and projection transform: OK\")
'
  "

This validation step catches ABI mismatches, missing .so files, incorrect PROJ_LIB paths, and GDAL driver registration failures before any bytes leave your machine.

5. Deployment and Runtime Routing

AWS Lambda:

LAYER_ARN=$(aws lambda publish-layer-version \
  --layer-name geospatial-core-py311 \
  --zip-file fileb://geospatial-layer.zip \
  --compatible-runtimes python3.11 \
  --compatible-architectures x86_64 \
  --query 'LayerVersionArn' \
  --output text)

echo "Published: $LAYER_ARN"

# Attach to function
aws lambda update-function-configuration \
  --function-name my-geo-function \
  --layers "$LAYER_ARN" \
  --memory-size 1024 \
  --environment "Variables={PROJ_LIB=/opt/python/pyproj/proj_data,GDAL_DATA=/opt/python/rasterio/gdal_data}"

GCP Cloud Functions 2nd gen:

gcloud functions deploy geo-processor \
  --gen2 \
  --runtime python311 \
  --region us-central1 \
  --source . \
  --entry-point handler \
  --memory 1024MB \
  --set-env-vars PROJ_LIB=/usr/local/lib/python3.11/site-packages/pyproj/proj_data,GDAL_DATA=/usr/local/lib/python3.11/site-packages/rasterio/gdal_data

Azure Functions (zip deploy):

# Azure expects site-packages in the zip root, not under python/
mkdir -p azure-package
cp -r layer/python/. azure-package/

cd azure-package && zip -r9 ../azure-geo-package.zip . && cd ..

az functionapp deployment source config-zip \
  --resource-group geo-rg \
  --name my-geo-func \
  --src azure-geo-package.zip

az functionapp config appsettings set \
  --resource-group geo-rg \
  --name my-geo-func \
  --settings \
    PROJ_LIB="/home/site/wwwroot/pyproj/proj_data" \
    GDAL_DATA="/home/site/wwwroot/rasterio/gdal_data"

Measurement and Verification

Confirm the optimization worked by measuring initialization duration and import time, not just package size.

AWS Lambda — measure cold-start init time:

import time, os, json

def handler(event, context):
    start = time.perf_counter()
    import rasterio
    import shapely
    import pyproj
    elapsed = time.perf_counter() - start
    return {
        "statusCode": 200,
        "body": json.dumps({
            "import_ms": round(elapsed * 1000, 1),
            "mem_mb": int(context.memory_limit_in_mb),
            "remaining_ms": context.get_remaining_time_in_millis()
        })
    }

Expected output after a trimmed layer: import_ms below 800 on a cold start at 1024 MB. A value above 3000 ms signals that the layer is still too large or that GDAL_DATA is causing a filesystem scan.

CloudWatch metric filter for initialization duration:

fields @timestamp, @initDuration
| filter @type = "REPORT"
| stats avg(@initDuration), max(@initDuration) by bin(5m)

A successful reduction should drop avg(@initDuration) from the 3000–5000 ms range to below 1200 ms on a 1024 MB function. If init duration remains high, enable GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR and CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif,.tiff to suppress expensive filesystem probes.

Benchmark layer size across iterations:

python3 - <<'EOF'
import subprocess, json

result = subprocess.run(
    ["aws", "lambda", "get-layer-version",
     "--layer-name", "geospatial-core-py311",
     "--version-number", "1",
     "--output", "json"],
    capture_output=True, text=True
)
data = json.loads(result.stdout)
content = data.get("Content", {})
zipped_mb = content.get("CodeSize", 0) / 1_048_576
print(f"Zipped layer size: {zipped_mb:.1f} MB")
# Unzipped is roughly 2.5-3x compressed for .so-heavy packages
print(f"Estimated unzipped: {zipped_mb * 2.7:.0f} MB")
EOF

Failure Modes and Debugging

ImportError: libgdal.so.X.Y: cannot open shared object file

The wheel was resolved for a different glibc version than the runtime. Reproduce with: find layer/python -name "*.so" | head -5 | xargs file — if output shows ELF 64-bit LSB shared object, x86-64 with a GLIBC reference newer than the runtime’s glibc, you must re-resolve wheels against the correct manylinux tag (manylinux_2_17, _2_28, etc.).

proj_create: Cannot find proj.db

PROJ_LIB is unset or points to the wrong path. After layer extraction on Lambda, pyproj’s data lands at /opt/python/pyproj/proj_data/proj.db. Set PROJ_LIB=/opt/python/pyproj/proj_data in the function environment variables, not just in handler code — the Lambda init phase resolves this before your handler runs.

OSError: [Errno 28] No space left on device during extraction

The combined unzipped size of your layers plus function code exceeds the /tmp ephemeral budget (512 MB default). Either trim the layer further, raise the ephemeral storage limit (up to 10 GB on Lambda), or switch to a container image. Managing /tmp storage limits for GeoTIFF extraction covers /tmp partitioning strategies when both layer extraction and raster I/O compete for the same scratch space.

Cold-start init duration spike after a dependency update

A new transitive dependency introduced additional .so files. Run pip show <package> | grep Requires against the updated lockfile to identify what changed, then re-run the prune + strip pipeline against the new install. Track @initDuration in CloudWatch as a deployment quality gate.

ModuleNotFoundError for a package you know is in the layer

The layer directory structure is wrong for the runtime. Lambda expects dependencies under python/ or python/lib/python3.11/site-packages/ when using --target. If you used a virtual environment layout (lib/python3.11/site-packages/), Lambda will not add it to sys.path automatically. Verify with aws lambda invoke --log-type Tail and decode the base64 log to see the actual sys.path at runtime.

strip: Unable to recognise the format of the input file

This is a harmless warning for pure-Python .pyc files or archive files that find located before the .so files. Redirect stderr with 2>/dev/null and confirm the exit code was 0 for genuine .so files. The warning does not indicate a failed strip on binary extensions.

Cost and Scaling Considerations

Cost-per-invocation math (AWS Lambda example):

The trimmed layer scenario costs roughly 88% less per cold start than the unoptimized baseline. At scale — 10 M invocations per month with a 5% cold-start ratio — that difference is ~$725/month before accounting for the memory reduction’s effect on warm-invocation cost.

When to prefer container images over layers:

• Unzipped stack exceeds 200 MB after aggressive stripping
• Custom GDAL builds with HDF5, NetCDF, or proprietary format drivers
• Multi-language runtimes (Python + Node.js side processes)
• Teams that need reproducible base OS environments across dev/staging/prod

When to prefer EFS-backed dependency shares:

If you have 20+ Lambda functions that all import the same geospatial stack, mounting a shared Amazon EFS access point at /mnt/geo-packages eliminates redundant layer extraction per function. EFS adds ~5–10 ms of network latency per cold start but removes the 250 MB cap entirely and allows hot-swapping packages without redeployment. This is most effective for event-driven pipelines where SQS and Pub/Sub queue routing strategies fan work out to many identical consumer functions.

Frequently Asked Questions

Why does a plain pip install of rasterio exceed the AWS Lambda 250 MB limit?

A standard rasterio install pulls compiled GDAL binaries, PROJ data grids, numpy, certifi, click, and their transitive C extensions. Unzipped, this typically reaches 320–380 MB. The fix is to target manylinux wheels, strip debug symbols from every .so file, and remove dist-info, test directories, and duplicate PROJ databases.

Can I reuse a layer built on macOS for AWS Lambda?

No. macOS produces Mach-O binaries with Darwin system paths. AWS Lambda runs Amazon Linux 2023 (glibc 2.34). Wheels must be downloaded targeting manylinux_2_28_x86_64 or manylinux_2_28_aarch64 to avoid ImportError or OSError at runtime.

Should I use a Lambda layer or a container image for a 400 MB geospatial stack?

Container images (up to 10 GB) remove the 250 MB layer cap and give you full OS control, but add 1–3 seconds of cold-start overhead and require ECR management. For stacks that compress below 200 MB after stripping, layers deploy faster and cost nothing extra. For heavier stacks or custom GDAL driver builds, container images are the better choice. The building minimal Docker images with Alpine and GDAL guide covers the container path in detail.

Related

• Stripping Unnecessary Python Packages from AWS Lambda Layers
• Docker Container Optimization for GIS
• Native Library Compilation for Serverless
• Cold Start Mapping for Python GDAL
• Ephemeral Storage Limits in AWS Lambda

Back to Packaging & Dependency Management for Serverless GIS