Reproducible GDAL Builds with pip Constraints and Hashes

Compile one lock per target platform with pip-compile --generate-hashes, install it with pip install --require-hashes --only-binary :all:, and gate the result on the wheel tags and GLIBC symbol versions the shared objects actually require. --require-hashes refuses to install anything not pinned with == and accompanied by a sha256 digest, so a silently republished wheel fails the build instead of shipping. What it will not do is tell you whether the vendored libgdal inside that wheel matches the runtime — that needs the ABI gate at the end of this page.

Context

The CI/CD pipeline sync for geo-dependencies discipline exists because a geospatial layer that rebuilds differently on Tuesday than it did on Monday is not a build artifact, it is a lottery ticket. Locking the direct dependencies is the easy half. The hard half is the transitive tree: rasterio pulls affine, attrs, certifi, click, cligj, click-plugins, numpy and pyparsing, and a version bump in any of them changes the bytes of your layer without changing a single line you wrote.

Hash pinning closes that gap in a way version pinning cannot. A version pin trusts the index to keep serving the same artifact for that version; a sha256 pin verifies it. That distinction matters more for compiled geospatial wheels than for pure Python, because the failure is not an API change you can read in a diff — it is a different libgdal inside rasterio.libs, linked against different symbol versions, presenting as a cold-start ImportError that looks identical to the environment misconfiguration described in the native library compilation guide.

From three direct pins to a verified geospatial layerFive stages left to right. A requirements.in file holding three direct pins for rasterio, pyproj and shapely. A pip-compile pass with generate-hashes run once per target container. A merged constraints.txt carrying every transitive pin with its sha256 digests. A pip install with require-hashes and only-binary. Finally an ABI gate that reads wheel tags and objdump symbol versions before the layer is published.requirements.in3 direct pinsnothing derivedpip-compile--generate-hashesonce per targetconstraints.txt11 pins, 22 digestscommittedpip install--require-hashes--only-binary :all:ABI gatewheel tag + objdumpthen publish
The digest check and the ABI check answer different questions — did I get the bytes I locked, and can those bytes run on Amazon Linux 2023. A build needs both, and neither implies the other.

What --require-hashes refuses

Hash-checking mode is deliberately unforgiving, and the refusals are the feature. Turning it on means every requirement pip is asked to install must be pinned with == and must carry at least one --hash=sha256:..., including every transitive dependency and every build requirement. Editable installs are rejected outright. A URL requirement with no digest is rejected. And a requirement pip resolves but cannot find a digest for stops the install with Hashes are required in --require-hashes mode, but they are missing from some requirements.

What pip does with each requirement under --require-hashesThree states with transitions. Pip reads the constraints file and moves on only when every requirement is pinned with a double equals. It then verifies each downloaded file against its sha256 digest, repeating that check for every wheel file in the requirement. On a match it installs into the target directory. On a mismatch or a missing digest it abandons the install entirely and returns to the lock step, so nothing partially installed reaches the layer.Hash-checking mode fails closed, and the failure is the featureRead constraintsevery pin must be ==Verify sha256per downloaded fileInstall to targetpython/lib/.../site-packagesall pinnedper wheel filedigest matchesmismatch or missing hash — abort and re-lockThe two messages to recognise: "Hashes are required in --require-hashes mode" and "THESE PACKAGES DO NOT MATCH THE HASHES FROMTHE REQUIREMENTS FILE".
There is no partial success path. A republished wheel or an unhashed transitive dependency stops the build at the verify state rather than producing a layer that only fails at a cold start.

If the digest is present but wrong, the message is different and much louder: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Either way the build stops before a byte reaches the layer, which is the entire point — the alternative is discovering the substitution at a cold start in production.

Prerequisites

  • Runtime: Python 3.11 on AWS Lambda, targeting both x86_64 and arm64
  • Tooling: pip 24.0 or newer, pip-tools 7.4.1 (pip-compile), binutils for objdump, Docker with the aarch64 binfmt handler registered if the host is x86_64
  • Build containers: public.ecr.aws/lambda/python:3.11 pulled for both linux/amd64 and linux/arm64, so each lock is compiled inside the environment it will be installed in
  • Direct dependencies, and only these, in requirements.in: rasterio==1.4.3, pyproj==3.7.0, shapely==2.0.6
  • Committed artifacts: requirements.in, locks/amd64.txt, locks/arm64.txt, and the merged constraints.txt — CI must never re-resolve from requirements.in outside the lock step
  • IAM for the publish step: lambda:PublishLayerVersion and s3:PutObject on the staging bucket
  • Runtime environment variables for the resulting layer, set on the function:
    code 
    GDAL_DATA=/opt/python/lib/python3.11/site-packages/rasterio/gdal_data
PROJ_LIB=/opt/python/lib/python3.11/site-packages/pyproj/proj_dir/share/proj
LD_LIBRARY_PATH=/opt/python/lib/python3.11/site-packages/rasterio.libs

Implementation

One script produces the locks and installs from them. The critical detail is that pip-compile resolves for the interpreter and platform it is running on, so it runs twice — once inside each target container — and the results are merged.

bash 
#!/usr/bin/env bash
# lock_geo_stack.sh — one hashed lock per target, merged into one constraints file.
set -euo pipefail

SITE="python/lib/python3.11/site-packages"
mkdir -p locks layer

cat > requirements.in <<'EOF'
# Only what the handler imports directly. Everything else is derived.
rasterio==1.4.3
pyproj==3.7.0
shapely==2.0.6
EOF

# pip-compile records a sha256 for every *file* it would download, and it only
# sees the files that match the running interpreter. Compiling on the host would
# produce a lock with x86_64 digests that cannot install on Graviton, so each
# pass runs inside the container it is locking for.
for arch in amd64 arm64; do
  docker run --rm --platform "linux/${arch}" -v "$(pwd):/w" -w /w \
    public.ecr.aws/lambda/python:3.11 bash -c "
      pip install --quiet --disable-pip-version-check pip-tools==7.4.1
      pip-compile requirements.in \
        --generate-hashes \
        --no-header \
        --output-file locks/${arch}.txt
    "
done

# A requirement line may carry as many --hash entries as it likes, so the union
# of both passes is a single file that installs correctly on either target.
python3 - <<'PY' > constraints.txt
import re, pathlib
from collections import OrderedDict

pins = OrderedDict()
for f in ("locks/amd64.txt", "locks/arm64.txt"):
    text = pathlib.Path(f).read_text().replace("\\\n", " ")
    for line in text.splitlines():
        if "==" not in line or line.lstrip().startswith("#"):
            continue
        name = re.split(r"==", line, 1)[0].strip()
        version = re.search(r"==([^\s;]+)", line).group(1)
        hashes = set(re.findall(r"--hash=(sha256:[0-9a-f]{64})", line))
        key = f"{name}=={version}"
        pins.setdefault(key, set()).update(hashes)

for key, hashes in pins.items():
    print(key + " \\")
    print(" \\\n".join(f"    --hash={h}" for h in sorted(hashes)))
PY

# --require-hashes needs every requirement it installs to carry a digest, so the
# merged file is passed as -r here. --only-binary :all: is mandatory alongside
# it: without it pip may fall back to an sdist whose hash matches but whose
# *build* is not reproducible.
docker run --rm --platform linux/amd64 -v "$(pwd):/w" -w /w \
  public.ecr.aws/lambda/python:3.11 \
  pip install \
    --require-hashes \
    --only-binary :all: \
    --no-cache-dir \
    --target "layer/${SITE}" \
    -r constraints.txt

# A second layer installed with --no-deps passes the SAME file as -c. A
# constraints file installs nothing; it only caps what the resolver may pick,
# which is exactly how two layers stay on one NumPy without duplicating it.
docker run --rm --platform linux/amd64 -v "$(pwd):/w" -w /w \
  public.ecr.aws/lambda/python:3.11 \
  pip install --no-deps -c constraints.txt --target "layer-extra/${SITE}" fiona==1.10.1

The two targets differ in exactly one thing that matters, and it is worth seeing laid out, because a lock that is missing one column fails at install time rather than at review time.

What each target requires from the lock fileA comparison of three deployment targets — AWS Lambda on x86_64, AWS Lambda on arm64 Graviton, and GCP Cloud Run — across five rows. All three use the cp311 interpreter and ABI tags. The platform tags differ: manylinux_2_28_x86_64 for the two x86_64 targets and manylinux_2_28_aarch64 for Graviton. The glibc floor is 2.34 on Amazon Linux 2023 and 2.36 on Debian 12. Each target needs its own compile pass, and a lock missing that target's digests fails the install with a missing-hash error rather than a warning.One merged constraints file, three deployment targetsAWS Lambda x86_64AWS Lambda arm64GCP Cloud RunInterpreter / ABI tagcp311 / cp311cp311 / cp311cp311 / cp311Platform tag in the lockmanylinux_2_28_x86_64manylinux_2_28_aarch64manylinux_2_28_x86_64glibc floor at runtime2.34Amazon Linux 20232.34Amazon Linux 20232.36Debian 12Where the pass must runAL2023 amd64containerAL2023 arm64containerpython:3.11-bookwormIf the lock omits this targetmissing-hashinstall refusesmissing-hashinstall refusesmissing-hashinstall refusesA requirement line accepts any number of --hash entries, so merging the passes yields one file that satisfies all three columns.
The only row that changes between the first two columns is the platform tag — and that one row is the whole reason a single compile pass on a developer laptop cannot lock a Graviton layer.

Verification

A passing pip install proves the digests matched. It does not prove the wheels can run on the target. The gate below reads what each wheel claims — its platform tag — and what each bundled shared object actually demands — the highest GLIBC_x.y symbol version referenced in its dynamic symbol table — and refuses anything above the runtime’s floor.

python 
# verify_wheel_abi.py — run inside the target container against layer/.
import pathlib
import re
import subprocess
import sys

TARGET_GLIBC = (2, 34)          # Amazon Linux 2023
ALLOWED_TAGS = ("manylinux_2_28_x86_64", "manylinux2014_x86_64", "any")
SITE = pathlib.Path(sys.argv[1])

failures, so_count, highest = [], 0, (2, 0)

for wheel_meta in sorted(SITE.glob("*.dist-info/WHEEL")):
    dist = wheel_meta.parent.name.rsplit("-", 1)[0]
    # The WHEEL file records the tag pip actually chose, which is the only
    # durable record of which manylinux policy this build depends on.
    tags = re.findall(r"^Tag: (.+)$", wheel_meta.read_text(), re.M)
    ok = any(t.rsplit("-", 1)[-1] in ALLOWED_TAGS for t in tags)
    print(f"{dist:<28} tag={tags[0]:<38} {'ok' if ok else 'REJECTED'}")
    if not ok:
        failures.append(f"{dist}: no acceptable platform tag in {tags}")

for so in sorted(SITE.rglob("*.so*")):
    so_count += 1
    # objdump -T lists versioned symbol references; the highest GLIBC_x.y here
    # is the real floor this object imposes, regardless of the wheel's tag.
    out = subprocess.run(["objdump", "-T", str(so)],
                         capture_output=True, text=True).stdout
    versions = [tuple(int(p) for p in m.split("."))
                for m in re.findall(r"GLIBC_(\d+\.\d+)", out)]
    if not versions:
        continue
    highest = max(highest, max(versions))
    if max(versions) > TARGET_GLIBC:
        failures.append(
            f"{so.relative_to(SITE)}: needs GLIBC_"
            f"{'.'.join(str(p) for p in max(versions))} > target"
        )

if failures:
    print("\nABI GATE FAILED")
    for f in failures:
        print(f"  {f}")
    sys.exit(1)
print(f"\nABI GATE PASSED — {so_count} shared objects, highest required symbol "
      f"GLIBC_{'.'.join(str(p) for p in highest)}, "
      f"target GLIBC_{'.'.join(str(p) for p in TARGET_GLIBC)}")

Expected output:

code 
pyproj-3.7.0                 tag=cp311-cp311-manylinux_2_28_x86_64      ok
rasterio-1.4.3               tag=cp311-cp311-manylinux_2_28_x86_64      ok
shapely-2.0.6                tag=cp311-cp311-manylinux_2_28_x86_64      ok

ABI GATE PASSED — 31 shared objects, highest required symbol GLIBC_2.28, target GLIBC_2.34

The interesting failure is a wheel whose tag is acceptable but whose objects are not — a manylinux_2_28 wheel is allowed to require up to GLIBC_2.28, and a mis-repaired one can slip past auditwheel requiring more. Reading objdump -T catches that in CI rather than at the first cold start, and it is the check that distinguishes a reproducible build from a merely repeatable one.

Gotchas and Edge Cases

  • --require-hashes fails on transitive dependencies you never named. The moment it is enabled, certifi, attrs, cligj and every other derived package must appear pinned and hashed. That is why the input to pip install is the compiled lock, never requirements.in. If the error names a package you have not heard of, the fix is to recompile, not to add a one-off --hash by hand.
  • A matching hash does not make an sdist build reproducible. If no wheel matches the target, pip will happily download an sdist whose digest is correct and then compile it against whatever headers happen to be in the container. Always pair --require-hashes with --only-binary :all: so the fallback is a hard failure rather than a silent source build — the same reasoning behind the platform-tag discipline in building rasterio Lambda layers on Amazon Linux 2023.
  • The lock is per platform, and Graviton is a different platform. Digests recorded on x86_64 do not include the manylinux_2_28_aarch64 files, so an arm64 build fails with a missing-hash error even though nothing changed. Compile inside each target container as the script does, which is the same reason cross-compiling GEOS for arm64 Graviton Lambda treats arm64 as a second build chain rather than a flag.
  • Hashes pin bytes, not compatibility. Two wheels can both be exactly what their digests promise and still disagree about GEOS: shapely vendors one libgeos_c and a source-built GDAL links another. Hash pinning guarantees the inputs; keeping the linked versions in agreement is the job of pinning GDAL and PROJ versions across build and runtime, and the two checks are complements rather than substitutes.

Frequently Asked Questions

What is the difference between a requirements file and a constraints file?

A requirements file is a list of things to install. A constraints file installs nothing — it only caps which version the resolver may choose if something else pulls that package in. That difference is load-bearing for a split layer: the base layer installs the compiled lock with -r, and the geo layer installs with --no-deps against the same file passed as -c, so both agree on NumPy and certifi without the base layer’s packages being written twice. Because hash-checking mode requires a digest for everything it installs, the compiled file is passed as -r where it installs and as -c where it must merely agree.

Does --require-hashes protect me from a GDAL ABI change?

No, and treating it as if it does is the most common way this setup gives false confidence. A digest pins the bytes of a wheel, so you get the same rasterio with the same vendored libgdal. It says nothing about whether that libgdal agrees with a system libgdal elsewhere in the layer, or whether the wheel’s manylinux policy is satisfied by the runtime’s glibc. Hashes make the build reproducible; the wheel-tag and objdump -T gate makes it correct.

Why does pip-compile produce a lock that fails on Graviton?

Because it resolves for the interpreter and platform it runs on. A compile pass on an x86_64 host records --hash lines for manylinux_2_28_x86_64 files, and the aarch64 wheel pip wants inside an arm64 container has no digest in that file, so the install refuses. Compile once inside each target container and merge — a requirement line accepts as many --hash entries as you give it, so one file can legitimately cover both architectures.

Back to CI/CD Pipeline Sync for Geo-Dependencies