GDAL Wheel Size Reduction with pip --no-binary

Compile the GDAL bindings against a lean system GDAL with pip install --no-binary gdal gdal==3.9.0 instead of accepting the fat manylinux wheel, which bundles a full driver set and its own copies of PROJ, GEOS, and dozens of format libraries. Building against a system libgdal you configured with only the drivers you use — GTiff, COG, MEM, VRT — cuts the shared-library footprint by 60-110 MB unzipped, and setting GDAL_SKIP at runtime trims driver-registration time on top of that. The size win is real only when you scope --no-binary to gdal and control the system build; a blanket --no-binary :all: recompiles the whole tree for no benefit.

Context

The prebuilt GDAL wheel is a marvel of convenience and a liability for size. To work on any machine without system dependencies, the manylinux wheel statically vendors a maximal GDAL: raster and vector drivers for formats most pipelines never touch — NetCDF, HDF5, PostgreSQL, Oracle, GRIB, and dozens more — plus its own PROJ and GEOS. That is exactly what you want on a laptop and exactly what you do not want inside a Lambda layer fighting for room under the 250 MB unzipped ceiling described in native library compilation.

pip --no-binary inverts the default. Instead of downloading the fat wheel, pip fetches the source distribution and compiles the bindings against whatever libgdal your build environment exposes through gdal-config. If you have installed a slim system GDAL configured with only the drivers your handler reads, the resulting install links against that lean library and drops the bloat. This is the middle path between accepting the wheel wholesale (covered in building rasterio Lambda layers on Amazon Linux 2023) and compiling the entire GDAL toolchain from source. It pairs well with the stripping unnecessary Python packages pass for a final layer that is as small as your driver requirements allow.

Choosing between the prebuilt wheel, a --no-binary build and a full source buildOne decision with three outcomes. If the prebuilt manylinux wheel fits under the 250 megabyte unzipped ceiling, keep the wheel. If it does not, compile the bindings with pip --no-binary gdal against a driver-trimmed system libgdal, saving 60 to 110 megabytes. If the wheel omits a driver you need, compile the whole GDAL toolchain from source.Three routes to a GDAL layer, one question apartDoes the prebuilt manylinux GDAL wheelfit the layer budget?yes, under 250 MBKeep the wheelFastest to build and hardest to breakThe rasterio layer recipe covers itno, over the ceilingpip --no-binary gdalLink against a driver-trimmed systemlibgdalSaves 60-110 MB unzippedwheel omits a driverCompile GDAL from sourceFull toolchain build; you own every flagAnd every rebuild after it
The middle branch is this page. It is only the right answer when the wheel genuinely does not fit — a source-built binding is a build you now own and have to keep in step with the system libgdal.

The economics are straightforward once you know your workload. Most serverless raster pipelines read a single input family — Cloud-Optimized GeoTIFF from S3, occasionally JPEG2000 or a VRT mosaic — and write GeoTIFF or COG. That is four or five drivers out of the 140-plus a maximal GDAL registers. Every driver you do not need is dead weight in three ways: bytes on disk that eat your unzipped budget, symbols the dynamic linker resolves at load, and driver-registration work GDAL performs during AllRegister() on the initialization path. Trimming the driver set therefore attacks size and cold-start latency at once. The catch is operational: you now own the build, so a new format requirement means a rebuild rather than a config change — a fair trade for a function whose format inputs are stable and known in advance.

Registered GDAL drivers against the drivers a serverless raster pipeline actually callsA grid of 140 tiles, one per driver registered by a maximal GDAL build. Four tiles are highlighted as the drivers a typical serverless raster pipeline reads — GTiff, COG, MEM and VRT. The remaining tiles are registered anyway, costing disk space, symbols for the dynamic linker, and registration work on every cold start.Drivers a maximal GDAL registers, against the four this pipeline reads140+ drivers in amaximal GDALThe trimmed system buildregisters 38. The handlerreads four. The rest costdisk, symbols andregistration time.Read by this pipeline: GTiff, COG, MEM, VRTRegistered but never calledGDAL_SKIP removes drivers from the registration pass but not from the shared library — it buys back initialization time, never disk space.
The trimmed system build in this recipe registers 38 drivers rather than 140-plus, and the handler calls four of them. Every other tile is paid for three times: in bytes, in symbols, and in AllRegister() time.

Prerequisites

  • Runtime: Python 3.11 on x86_64, targeting Amazon Linux 2023 (glibc 2.34)
  • Build image: public.ecr.aws/lambda/python:3.11 — build here so the compiled bindings match the Lambda ABI
  • System GDAL: a libgdal installed in the container with gdal-config on PATH; the leaner its driver set, the smaller the result
  • Toolchain: gcc, gcc-c++, make, and the Python gdal/pygdal source that matches your libgdal major version exactly
  • Pinned version: the GDAL Python binding version must equal the system libgdal version — e.g. gdal==3.9.0 against system GDAL 3.9.x
  • Runtime environment variables on the function:
    code 
    GDAL_DATA=/opt/share/gdal
PROJ_LIB=/opt/share/proj
LD_LIBRARY_PATH=/opt/lib
GDAL_SKIP=NetCDF HDF5 GRIB PDF PostgreSQL   # unregister formats you never read
  • IAM for upload: lambda:PublishLayerVersion and s3:PutObject on the staging bucket

Implementation

The build script installs a slim system GDAL, then compiles only the bindings with --no-binary gdal. The binding version is pinned to the system libgdal version so the compiled extension matches the ABI it links against.

bash 
#!/usr/bin/env bash
# build_slim_gdal.sh — compile GDAL bindings against a driver-trimmed system GDAL.
set -euo pipefail

GDAL_VERSION="3.9.0"
LAYER_DIR="$(pwd)/layer"
rm -rf "${LAYER_DIR}" && mkdir -p "${LAYER_DIR}"

docker run --rm \
  --platform linux/amd64 \
  -v "${LAYER_DIR}:/layer" \
  public.ecr.aws/lambda/python:3.11 \
  bash -c "
    set -euo pipefail

    # Install a lean system GDAL + build toolchain. On AL2023 the packaged
    # libgdal already omits many heavyweight optional drivers; installing only
    # gdal-devel (not gdal-plugins) keeps the driver set minimal.
    dnf install -y gcc gcc-c++ make binutils \
                   gdal gdal-devel proj proj-data >/dev/null

    # Confirm the bindings will link against THIS libgdal, not a hidden copy.
    gdal-config --version
    export GDAL_CONFIG=/usr/bin/gdal-config
    export CFLAGS=\"-O2 \$(gdal-config --cflags)\"

    # --no-binary gdal forces a source build of ONLY the bindings; numpy and any
    # other dependency still install as normal wheels. The version must equal the
    # system libgdal major.minor so the compiled extension is ABI-compatible.
    pip install \
      --no-binary gdal \
      --target /layer/python/lib/python3.11/site-packages \
      gdal==${GDAL_VERSION}

    # Bundle the runtime data and the slim libgdal + PROJ into the layer tree.
    mkdir -p /layer/lib /layer/share/gdal /layer/share/proj
    cp -a /usr/lib64/libgdal.so.* /layer/lib/
    cp -a /usr/lib64/libproj.so.* /layer/lib/
    cp -a /usr/share/gdal/. /layer/share/gdal/
    cp -a /usr/share/proj/. /layer/share/proj/

    # Strip debug symbols from the bundled shared objects.
    find /layer/lib -name '*.so*' -exec strip --strip-unneeded {} + 2>/dev/null || true
  "

# Report the delta against a fat-wheel baseline (see Verification below).
echo "Slim unzipped: $(du -sh "${LAYER_DIR}" | cut -f1)"

cd "${LAYER_DIR}" && zip -r9 ../gdal-slim-layer.zip . >/dev/null && cd ..
aws s3 cp gdal-slim-layer.zip s3://my-lambda-artifacts/layers/gdal-slim-3-9-0.zip
aws lambda publish-layer-version \
  --layer-name "gdal-slim-3-9-0-py311-x86_64" \
  --description "Driver-trimmed GDAL ${GDAL_VERSION}, --no-binary build" \
  --content "S3Bucket=my-lambda-artifacts,S3Key=layers/gdal-slim-3-9-0.zip" \
  --compatible-runtimes python3.11 \
  --compatible-architectures x86_64

Verification

Measure the size delta directly by building both ways in the same container and comparing du. This is the number that justifies the extra build complexity.

bash 
docker run --rm public.ecr.aws/lambda/python:3.11 bash -c '
  set -e
  dnf install -y gcc gcc-c++ make gdal gdal-devel >/dev/null

  # A) Fat prebuilt wheel with its fully-loaded, self-contained libgdal.
  pip install --quiet --only-binary gdal --target /tmp/fat gdal==3.9.0
  FAT=$(du -sm /tmp/fat | cut -f1)

  # B) --no-binary build linking the driver-trimmed system libgdal.
  pip install --quiet --no-binary gdal --target /tmp/slim gdal==3.9.0
  SLIM=$(du -sm /tmp/slim | cut -f1)

  echo "fat wheel   : ${FAT} MB"
  echo "no-binary   : ${SLIM} MB"
  echo "delta       : $((FAT - SLIM)) MB saved"
'

Expected output (exact numbers vary with the system driver set):

code 
fat wheel   : 168 MB
no-binary   : 71 MB
delta       : 97 MB saved
Prebuilt manylinux wheel against a --no-binary build for gdal 3.9.0Two measured sizes for the same package and version installed in the same Amazon Linux 2023 container: the prebuilt manylinux wheel with its full driver set and bundled PROJ and GEOS at 168 megabytes, and the --no-binary build linked against a driver-trimmed system libgdal at 71 megabytes — a 97 megabyte saving against a 250 megabyte unzipped layer ceiling.Unzipped footprint of gdal==3.9.0, both ways, in the same containermanylinux wheel (all drivers)168 MB--no-binary, slim system GDAL71 MB0MB unzipped — track ends at the 250 MB layer ceilingThe wheel is not wasteful by accident: it vendors every driver, PROJ and GEOS so it installs on any machine without system dependencies. Alayer needs the opposite trade, and --no-binary is how you take it.
Same package, same version, same container. The 97 MB difference is entirely the drivers the wheel bundles so that it can install anywhere — which is precisely what a Lambda layer does not need.

Then confirm the slim build actually reads your target format and that GDAL_SKIP is unregistering the drivers you dropped:

bash 
GDAL_SKIP="NetCDF HDF5 GRIB" python3 -c "
from osgeo import gdal
gdal.UseExceptions()
gdal.AllRegister()
print('GTiff driver:', gdal.GetDriverByName('GTiff') is not None)
print('NetCDF driver:', gdal.GetDriverByName('NetCDF'))   # None after GDAL_SKIP
print('total drivers:', gdal.GetDriverCount())
"
# Expected:
# GTiff driver: True
# NetCDF driver: None
# total drivers: 38

Gotchas and Edge Cases

  • The binding version must match the system libgdal exactly. Compiling gdal==3.9.0 bindings against a system GDAL 3.8 or 3.10 produces undefined symbol errors at import. Pin both to the same major.minor, and verify with gdal-config --version inside the container before pip runs.
  • GDAL_SKIP trims time, not bytes. It unregisters drivers at initialization so cold-start driver-registration is faster, which matters for cold-start-sensitive GDAL functions. It does not shrink libgdal.so — for that you must compile GDAL with fewer optional drivers or use a leaner system package.
  • --no-binary :all: is a trap. Scoping to --no-binary gdal compiles only the bindings; the blanket :all: form forces a source build of numpy and every transitive dependency, ballooning build time from seconds to many minutes with no size payoff for the non-GDAL packages.
  • You now own the libraries the wheel used to bundle. The fat wheel carried PROJ, GEOS, and libtiff internally. Once you go --no-binary, you must bundle the system libproj, proj.db, and set LD_LIBRARY_PATH, PROJ_LIB, and GDAL_DATA yourself — otherwise reprojection fails at runtime even though import succeeds.

Frequently Asked Questions

Does GDAL_SKIP make the shared library smaller?

No. GDAL_SKIP is a runtime setting that unregisters named drivers during initialization; it removes zero bytes from libgdal.so. The disk-size win comes from compiling or installing a GDAL that physically excludes optional format drivers. GDAL_SKIP complements that by shaving cold-start driver-registration time, which is a latency optimization rather than a footprint one.

When is --no-binary worth it over the prebuilt wheel?

Reach for --no-binary when the manylinux wheel’s bundled GDAL pushes your combined layer over the 250 MB unzipped limit, or when you need a driver the wheel omits. For a straightforward rasterio-only function the prebuilt wheel is faster to build and less error-prone. --no-binary earns its place when you control the system GDAL and want a minimal, driver-trimmed libgdal sized to exactly what your pipeline reads.

Will pip --no-binary gdal accidentally recompile numpy too?

Only if you write --no-binary :all:. Scope it to the package you mean — pip install --no-binary gdal gdal==3.9.0 compiles just the GDAL bindings from source and installs numpy and everything else as ordinary wheels. The blanket form forces a source build of the entire dependency tree and lengthens the build dramatically for no benefit.

Back to Native Library Compilation for Serverless