GDAL Wheel Size Reduction with pip --no-binary

Q: Does GDAL_SKIP make the shared library smaller?

No. GDAL_SKIP is a runtime setting that unregisters drivers during initialization; it does not remove any bytes from libgdal.so. The size win comes from compiling GDAL itself with fewer optional format drivers so the shared library is physically smaller. GDAL_SKIP complements that by shaving cold-start driver-registration time, not disk footprint.

Q: When is --no-binary worth it over the prebuilt wheel?

Reach for --no-binary when the prebuilt manylinux wheel's bundled GDAL pushes your combined layer over the 250 MB unzipped limit, or when you need a driver build the wheel omits. For the common rasterio-only case the wheel is faster and safer; --no-binary earns its keep when you control the system GDAL and want a minimal, driver-trimmed libgdal.

Q: Will pip --no-binary gdal accidentally recompile numpy too?

Only if you pass --no-binary :all:. Scope it to the package you mean: pip install --no-binary gdal gdal==3.9.0 compiles just the GDAL bindings from source and takes numpy and any other dependency as a normal wheel. A blanket --no-binary :all: forces a source build of the entire tree and dramatically lengthens the build.

Compile the GDAL bindings against a lean system GDAL with pip install --no-binary gdal gdal==3.9.0 instead of accepting the fat manylinux wheel, which bundles a full driver set and its own copies of PROJ, GEOS, and dozens of format libraries. Building against a system libgdal you configured with only the drivers you use — GTiff, COG, MEM, VRT — cuts the shared-library footprint by 60-110 MB unzipped, and setting GDAL_SKIP at runtime trims driver-registration time on top of that. The size win is real only when you scope --no-binary to gdal and control the system build; a blanket --no-binary :all: recompiles the whole tree for no benefit.

Context

The prebuilt GDAL wheel is a marvel of convenience and a liability for size. To work on any machine without system dependencies, the manylinux wheel statically vendors a maximal GDAL: raster and vector drivers for formats most pipelines never touch — NetCDF, HDF5, PostgreSQL, Oracle, GRIB, and dozens more — plus its own PROJ and GEOS. That is exactly what you want on a laptop and exactly what you do not want inside a Lambda layer fighting for room under the 250 MB unzipped ceiling described in native library compilation.

pip --no-binary inverts the default. Instead of downloading the fat wheel, pip fetches the source distribution and compiles the bindings against whatever libgdal your build environment exposes through gdal-config. If you have installed a slim system GDAL configured with only the drivers your handler reads, the resulting install links against that lean library and drops the bloat. This is the middle path between accepting the wheel wholesale (covered in building rasterio Lambda layers on Amazon Linux 2023) and compiling the entire GDAL toolchain from source. It pairs well with the stripping unnecessary Python packages pass for a final layer that is as small as your driver requirements allow.

The economics are straightforward once you know your workload. Most serverless raster pipelines read a single input family — Cloud-Optimized GeoTIFF from S3, occasionally JPEG2000 or a VRT mosaic — and write GeoTIFF or COG. That is four or five drivers out of the 140-plus a maximal GDAL registers. Every driver you do not need is dead weight in three ways: bytes on disk that eat your unzipped budget, symbols the dynamic linker resolves at load, and driver-registration work GDAL performs during AllRegister() on the initialization path. Trimming the driver set therefore attacks size and cold-start latency at once. The catch is operational: you now own the build, so a new format requirement means a rebuild rather than a config change — a fair trade for a function whose format inputs are stable and known in advance.

Prerequisites

Runtime: Python 3.11 on x86_64, targeting Amazon Linux 2023 (glibc 2.34)
Build image: public.ecr.aws/lambda/python:3.11 — build here so the compiled bindings match the Lambda ABI
System GDAL: a libgdal installed in the container with gdal-config on PATH; the leaner its driver set, the smaller the result
Toolchain: gcc, gcc-c++, make, and the Python gdal/pygdal source that matches your libgdal major version exactly
Pinned version: the GDAL Python binding version must equal the system libgdal version — e.g. gdal==3.9.0 against system GDAL 3.9.x

Runtime environment variables on the function:

code

GDAL_DATA=/opt/share/gdal
PROJ_LIB=/opt/share/proj
LD_LIBRARY_PATH=/opt/lib
GDAL_SKIP=NetCDF HDF5 GRIB PDF PostgreSQL   # unregister formats you never read

IAM for upload: lambda:PublishLayerVersion and s3:PutObject on the staging bucket

Implementation

The build script installs a slim system GDAL, then compiles only the bindings with --no-binary gdal. The binding version is pinned to the system libgdal version so the compiled extension matches the ABI it links against.

bash

#!/usr/bin/env bash
# build_slim_gdal.sh — compile GDAL bindings against a driver-trimmed system GDAL.
set -euo pipefail

GDAL_VERSION="3.9.0"
LAYER_DIR="$(pwd)/layer"
rm -rf "${LAYER_DIR}" && mkdir -p "${LAYER_DIR}"

docker run --rm \
  --platform linux/amd64 \
  -v "${LAYER_DIR}:/layer" \
  public.ecr.aws/lambda/python:3.11 \
  bash -c "
    set -euo pipefail

    # Install a lean system GDAL + build toolchain. On AL2023 the packaged
    # libgdal already omits many heavyweight optional drivers; installing only
    # gdal-devel (not gdal-plugins) keeps the driver set minimal.
    dnf install -y gcc gcc-c++ make binutils \
                   gdal gdal-devel proj proj-data >/dev/null

    # Confirm the bindings will link against THIS libgdal, not a hidden copy.
    gdal-config --version
    export GDAL_CONFIG=/usr/bin/gdal-config
    export CFLAGS=\"-O2 \$(gdal-config --cflags)\"

    # --no-binary gdal forces a source build of ONLY the bindings; numpy and any
    # other dependency still install as normal wheels. The version must equal the
    # system libgdal major.minor so the compiled extension is ABI-compatible.
    pip install \
      --no-binary gdal \
      --target /layer/python/lib/python3.11/site-packages \
      gdal==${GDAL_VERSION}

    # Bundle the runtime data and the slim libgdal + PROJ into the layer tree.
    mkdir -p /layer/lib /layer/share/gdal /layer/share/proj
    cp -a /usr/lib64/libgdal.so.* /layer/lib/
    cp -a /usr/lib64/libproj.so.* /layer/lib/
    cp -a /usr/share/gdal/. /layer/share/gdal/
    cp -a /usr/share/proj/. /layer/share/proj/

    # Strip debug symbols from the bundled shared objects.
    find /layer/lib -name '*.so*' -exec strip --strip-unneeded {} + 2>/dev/null || true
  "

# Report the delta against a fat-wheel baseline (see Verification below).
echo "Slim unzipped: $(du -sh "${LAYER_DIR}" | cut -f1)"

cd "${LAYER_DIR}" && zip -r9 ../gdal-slim-layer.zip . >/dev/null && cd ..
aws s3 cp gdal-slim-layer.zip s3://my-lambda-artifacts/layers/gdal-slim-3-9-0.zip
aws lambda publish-layer-version \
  --layer-name "gdal-slim-3-9-0-py311-x86_64" \
  --description "Driver-trimmed GDAL ${GDAL_VERSION}, --no-binary build" \
  --content "S3Bucket=my-lambda-artifacts,S3Key=layers/gdal-slim-3-9-0.zip" \
  --compatible-runtimes python3.11 \
  --compatible-architectures x86_64

Verification

Measure the size delta directly by building both ways in the same container and comparing du. This is the number that justifies the extra build complexity.

bash

docker run --rm public.ecr.aws/lambda/python:3.11 bash -c '
  set -e
  dnf install -y gcc gcc-c++ make gdal gdal-devel >/dev/null

  # A) Fat prebuilt wheel with its fully-loaded, self-contained libgdal.
  pip install --quiet --only-binary gdal --target /tmp/fat gdal==3.9.0
  FAT=$(du -sm /tmp/fat | cut -f1)

  # B) --no-binary build linking the driver-trimmed system libgdal.
  pip install --quiet --no-binary gdal --target /tmp/slim gdal==3.9.0
  SLIM=$(du -sm /tmp/slim | cut -f1)

  echo "fat wheel   : ${FAT} MB"
  echo "no-binary   : ${SLIM} MB"
  echo "delta       : $((FAT - SLIM)) MB saved"
'

Expected output (exact numbers vary with the system driver set):

code

fat wheel   : 168 MB
no-binary   : 71 MB
delta       : 97 MB saved

Then confirm the slim build actually reads your target format and that GDAL_SKIP is unregistering the drivers you dropped:

bash

GDAL_SKIP="NetCDF HDF5 GRIB" python3 -c "
from osgeo import gdal
gdal.UseExceptions()
gdal.AllRegister()
print('GTiff driver:', gdal.GetDriverByName('GTiff') is not None)
print('NetCDF driver:', gdal.GetDriverByName('NetCDF'))   # None after GDAL_SKIP
print('total drivers:', gdal.GetDriverCount())
"
# Expected:
# GTiff driver: True
# NetCDF driver: None
# total drivers: 38

Gotchas and Edge Cases

The binding version must match the system libgdal exactly. Compiling gdal==3.9.0 bindings against a system GDAL 3.8 or 3.10 produces undefined symbol errors at import. Pin both to the same major.minor, and verify with gdal-config --version inside the container before pip runs.
GDAL_SKIP trims time, not bytes. It unregisters drivers at initialization so cold-start driver-registration is faster, which matters for cold-start-sensitive GDAL functions. It does not shrink libgdal.so — for that you must compile GDAL with fewer optional drivers or use a leaner system package.
--no-binary :all: is a trap. Scoping to --no-binary gdal compiles only the bindings; the blanket :all: form forces a source build of numpy and every transitive dependency, ballooning build time from seconds to many minutes with no size payoff for the non-GDAL packages.
You now own the libraries the wheel used to bundle. The fat wheel carried PROJ, GEOS, and libtiff internally. Once you go --no-binary, you must bundle the system libproj, proj.db, and set LD_LIBRARY_PATH, PROJ_LIB, and GDAL_DATA yourself — otherwise reprojection fails at runtime even though import succeeds.

Frequently Asked Questions

Does `GDAL_SKIP` make the shared library smaller?

No. GDAL_SKIP is a runtime setting that unregisters named drivers during initialization; it removes zero bytes from libgdal.so. The disk-size win comes from compiling or installing a GDAL that physically excludes optional format drivers. GDAL_SKIP complements that by shaving cold-start driver-registration time, which is a latency optimization rather than a footprint one.

When is `--no-binary` worth it over the prebuilt wheel?

Reach for --no-binary when the manylinux wheel’s bundled GDAL pushes your combined layer over the 250 MB unzipped limit, or when you need a driver the wheel omits. For a straightforward rasterio-only function the prebuilt wheel is faster to build and less error-prone. --no-binary earns its place when you control the system GDAL and want a minimal, driver-trimmed libgdal sized to exactly what your pipeline reads.

Will `pip --no-binary gdal` accidentally recompile numpy too?

Only if you write --no-binary :all:. Scope it to the package you mean — pip install --no-binary gdal gdal==3.9.0 compiles just the GDAL bindings from source and installs numpy and everything else as ordinary wheels. The blanket form forces a source build of the entire dependency tree and lengthens the build dramatically for no benefit.

Native Library Compilation for Serverless — full from-source toolchain build when even a slim system GDAL is not enough
Building Rasterio Lambda Layers on Amazon Linux 2023 — the wheel-based path that trades size for build simplicity
Stripping Unnecessary Python Packages from AWS Lambda Layers — reclaim further headroom after the GDAL build
Deduplicating NumPy Across Geospatial Lambda Layers — avoid stacking duplicate numpy copies on top of a slim GDAL
Cold Start Mapping for Python GDAL — how driver count and binary size shape initialization latency

Back to Native Library Compilation for Serverless