Shipping PROJ Transformation Grids Without Blowing the Package Limit

The full proj-data release is roughly 655 MB and the AWS Lambda ceiling is 250 MB unzipped across the function and all five layers, so the whole grid set cannot ship — and it does not need to. A single country’s datum-shift grids run 4–34 MB, which fits comfortably; find them with projinfo -s EPSG:4267 -t EPSG:4326 --spatial-test intersects -o PROJ, download only those files into share/proj/, and assert at deploy time that TransformerGroup(...).unavailable_operations is empty. When the CRS pairs are not enumerable, set PROJ_NETWORK=ON with PROJ_USER_WRITABLE_DIRECTORY=/tmp/proj_cache and PROJ range-requests the grid cells it needs from a CDN for 80–400 ms on the first transform in each execution environment.

Context

proj.db is about 9.4 MB and holds every CRS, datum and transformation definition PROJ knows. It holds no grid data at all. A datum shift such as NAD27 → NAD83, NZGD49 → NZGD2000, or an orthometric-height conversion through GEOID18, is defined by a raster of per-cell offsets stored in a separate file, and PROJ looks for those files in the same directory PROJ_DATA names — the directory whose value is set per the runtime configuration overview and spelled out for AWS in setting GDAL_DATA and PROJ_LIB in Lambda.

What makes this dangerous rather than merely inconvenient is the failure mode. A missing grid raises nothing. PROJ ranks the candidate operations for a CRS pair by accuracy, skips the ones whose grids are absent, and quietly uses the next one down — typically a Helmert transformation or a null shift accurate to a few metres instead of a few centimetres. The pipeline runs green, the STAC item claims survey-grade georeferencing, and the error surfaces months later when someone overlays the output on a cadastral layer.

Everything below therefore treats the grid list as a build input, discovered from the CRS pairs, verified at deploy time, and regenerated whenever the PROJ version in the layer moves.

Prerequisites

• The CRS pairs the pipeline actually transforms between, enumerated. If they cannot be enumerated, skip to the PROJ_NETWORK section.
• PROJ 7 or later in the layer — grids moved to a cloud-optimized GeoTIFF format in PROJ 7, which is what makes range requests possible. The older .gsb/.gtx files still work but are not range-readable.
• projinfo available in the build container, which the proj-bin (Debian) or proj (Amazon Linux) package provides.
• share/proj/ writable during the build, at the same path the runtime environment names — /opt/share/proj for a Lambda layer.
• Package budget known: 250 MB unzipped across the function and all layers on AWS Lambda, at most 5 layers, mounted at /opt. Cloud Run has no zip ceiling; Azure Functions on the Consumption plan is bounded by its 1,536 MB memory and 10-minute timeout rather than by package size.
• /tmp sizing decided if using the network strategy — 512 MB by default, provisionable to 10,240 MB.
• Outbound HTTPS from the function if using the network strategy. A function inside a VPC needs a NAT gateway or a VPC endpoint, or the fetch hangs until the timeout.

Finding the Grids a Transformation Needs

projinfo prints every candidate operation between two CRS, ranked by accuracy, with each pipeline’s +grids= clause visible. The --spatial-test intersects flag stops it from discarding operations whose area of use only partly overlaps.

projinfo -s EPSG:4267 -t EPSG:4326 --spatial-test intersects -o PROJ

Candidate operations found: 6
-------------------------------------
Operation No. 1:
NADCON5, NAD27 to NAD83(1986) (CONUS), 0.15 m, United States (USA) - CONUS
PROJ string:
+proj=pipeline +step +proj=axisswap +order=2,1 +step +proj=unitconvert
  +xy_in=deg +xy_out=rad +step +proj=hgridshift
  +grids=us_noaa_nadcon5_nad27_nad83_1986_conus.tif ...

The grid file names in +grids= are what you need. Do not copy them by hand into a build script — derive them, so the list regenerates when PROJ changes:

#!/usr/bin/env python3
"""grid_manifest.py — emit the exact grid files a set of CRS pairs requires."""
import json
import sqlite3
import sys
from pathlib import Path

import pyproj
from pyproj.transformer import TransformerGroup

# The pipeline's real CRS pairs. Keep this list in version control next to the IaC.
PAIRS = [
    ("EPSG:4267", "EPSG:4326"),   # NAD27  -> WGS84
    ("EPSG:4269", "EPSG:6318"),   # NAD83  -> NAD83(2011)
]


def needed_grids(pairs):
    grids = set()
    for src, dst in pairs:
        tg = TransformerGroup(src, dst, always_xy=True)
        for op in list(tg.transformers) + list(tg.unavailable_operations):
            for g in op.grids:
                grids.add(g.short_name)
    return sorted(grids)


def resolve_urls(grid_names):
    """proj.db carries the CDN URL for every grid it references."""
    db = Path(pyproj.datadir.get_data_dir()) / "proj.db"
    con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
    out = {}
    for name in grid_names:
        row = con.execute(
            "SELECT full_name, url FROM grid_alternatives WHERE proj_grid_name = ?",
            (name,),
        ).fetchone()
        if row:
            out[row[0]] = row[1]
    con.close()
    return out


if __name__ == "__main__":
    names = needed_grids(PAIRS)
    urls = resolve_urls(names)
    json.dump(urls, sys.stdout, indent=2)
    print(f"\n{len(urls)} grid file(s) required", file=sys.stderr)

Feed the manifest straight into the layer build, so the grid set and the layer are produced by one command and cannot drift apart:

#!/usr/bin/env bash
# fetch_grids.sh — run inside the build container, after normalising share/proj.
set -euo pipefail
LAYER=${1:?usage: fetch_grids.sh ./layer}
DEST="${LAYER}/share/proj"

python3 grid_manifest.py > /tmp/grids.json
python3 - "$DEST" <<'PY'
import json, sys, urllib.request
from pathlib import Path
dest = Path(sys.argv[1]); dest.mkdir(parents=True, exist_ok=True)
for name, url in json.load(open("/tmp/grids.json")).items():
    target = dest / name
    if target.exists():
        continue
    print(f"fetching {name} <- {url}")
    urllib.request.urlretrieve(url, target)
PY

echo "grid payload: $(du -sh "${DEST}" | cut -f1)"

What the Grids Cost

Concrete numbers, because the decision is entirely about arithmetic against a hard ceiling. One New Zealand grid, nz_linz_nzgd2kgrid0005.tif, is 4.6 MB. The CONUS NADCON5 set for NAD27 → NAD83 is about 34 MB. Adding GEOID18 for orthometric heights costs another 27 MB. The Australian GDA94 → GDA2020 conformal-plus-distortion grid is about 51 MB. And the full proj-data release, which is what pip install pyproj[network] documentation implicitly assumes you have, is roughly 655 MB — two and a half times the entire AWS Lambda ceiling.

On a layer that already carries 118 MB of stripped rasterio shared objects and 41 MB of numpy and friends, one country’s grid set fits with 44 MB to spare. Two countries do not. That is the whole decision boundary, and it is why the strategies below exist. The pruning techniques in stripping unnecessary Python packages from AWS Lambda Layers buy another 20–30 MB of headroom, which is enough for one more grid and not enough to change the strategy.

Three Strategies

PROJ_NETWORK with a CDN is the strategy to reach for the moment the CRS pairs stop being enumerable. PROJ fetches grid data over HTTPS using range requests against the cloud-optimized GeoTIFF form of each grid, so a 27 MB continental grid usually costs a few hundred kilobytes of transfer for one scene’s bounding box. The environment for it:

# Add to the environment map from the runtime configuration guide.
PROJ_NETWORK = "ON"
PROJ_NETWORK_ENDPOINT = "https://cdn.proj.org"        # or your own mirror
PROJ_USER_WRITABLE_DIRECTORY = "/tmp/proj_cache"      # HOME is read-only

Mirroring cdn.proj.org into your own bucket is worth doing for anything production-facing: it removes a third-party dependency from the request path, keeps egress inside your account, and lets a VPC endpoint serve the traffic. Sync it once and point PROJ_NETWORK_ENDPOINT at the bucket’s HTTPS URL.

EFS or GCS FUSE staging holds the full 655 MB release outside the package entirely. On AWS, create an EFS access point, mount it at /mnt/proj, and set PROJ_DATA=/mnt/proj — everything else in the environment is unchanged. The cost is a VPC configuration, an ENI attachment that adds to cold-start latency, and a NAT path for anything else the function needs from the internet. Reach for it only when both the grid set and the CRS pairs are genuinely open-ended, for example a public reprojection API. On Cloud Run the equivalent is a GCS FUSE volume mount, and its 60-minute request timeout and 32 GiB ceiling make the mount latency far easier to amortise than on a 15-minute Lambda.

Verification

The check that matters runs inside the deployed function, against the CRS pairs the pipeline uses, and fails the deployment rather than warning.

# verify_grids.py — deploy alongside the pipeline, invoke after every layer bump.
from pyproj.transformer import TransformerGroup

PAIRS = [("EPSG:4267", "EPSG:4326"), ("EPSG:4269", "EPSG:6318")]


def main(event, context):
    report = {}
    for src, dst in PAIRS:
        tg = TransformerGroup(src, dst, always_xy=True)
        report[f"{src}->{dst}"] = {
            "available": len(tg.transformers),
            "missing_grids": [
                g.short_name
                for op in tg.unavailable_operations
                for g in op.grids
                if not g.available
            ],
            "best": tg.transformers[0].description if tg.transformers else None,
            "best_accuracy_m": tg.transformers[0].accuracy if tg.transformers else None,
        }
    return report

Expected output when every required grid is present:

{
  "EPSG:4267->EPSG:4326": {
    "available": 6,
    "missing_grids": [],
    "best": "NADCON5, NAD27 to NAD83(1986) (CONUS)",
    "best_accuracy_m": 0.15
  },
  "EPSG:4269->EPSG:6318": {
    "available": 4,
    "missing_grids": [],
    "best": "NAD83 to NAD83(2011) (NADCON5)",
    "best_accuracy_m": 0.05
  }
}

The two fields to assert on are missing_grids, which must be empty, and best_accuracy_m, which must be sub-metre. An available count that is non-zero with a best_accuracy_m of 2.0 or null means PROJ found a ballpark operation and is using it — technically successful, materially wrong.

Gotchas and Edge Cases

• PROJ_NETWORK=ON without PROJ_USER_WRITABLE_DIRECTORY fails on the write, not the fetch. PROJ defaults its cache to $HOME/.local/share/proj, and HOME in the Lambda execution environment is /home/sbx_user1051, which is read-only. The symptom is OSError: [Errno 30] Read-only file system. Set the variable to /tmp/proj_cache and create the directory at module scope, before the first import rasterio.
• A function inside a VPC cannot reach cdn.proj.org without a route. PROJ_NETWORK=ON inside a private subnet with no NAT gateway does not error immediately — it stalls on the connection until the socket times out, which reads as a slow transform rather than a network fault. Either mirror the grids into an S3 bucket behind a gateway endpoint, or bundle them.
• The /tmp cache is per execution environment, not per function. Every cold start pays the fetch again. On a bursty pipeline with poor warm-instance reuse this dominates; on a steady tiling fan-out it amortises to nothing. Measure with the same instrumentation used for cold start mapping for Python GDAL before assuming which case you are in.
• Bundled grids and proj.db must come from the same PROJ release. proj.db references grids by name and records their expected format; a .tif grid from PROJ 9 paired with a proj.db from PROJ 6 will not be found, and a .gtx grid from PROJ 6 in a PROJ 9 layer will be found but is not range-readable. Regenerate the grid manifest as part of the same build that produces the layer, exactly as pinning GDAL and PROJ versions across build and runtime describes.

Frequently Asked Questions

How do I find out which grid a transformation needs?

Run projinfo -s <source CRS> -t <target CRS> --spatial-test intersects -o PROJ. Every candidate operation prints its pipeline, and any operation requiring a grid names it inline in a +grids= clause. In Python the same information comes from pyproj’s TransformerGroup, whose unavailable_operations list holds precisely the operations whose grids are not installed.

Does PROJ_NETWORK download the whole grid file?

No. PROJ issues HTTP range requests against the cloud-optimized GeoTIFF form of each grid and fetches only the blocks covering the coordinates being transformed. A continental grid that is 27 MB on disk typically costs a few hundred kilobytes of transfer for a single scene’s bounding box, which is why the network strategy is cheap in bandwidth and expensive only in first-transform latency.

Where does PROJ cache network-fetched grids on AWS Lambda?

In the directory named by PROJ_USER_WRITABLE_DIRECTORY, which must be under /tmp because HOME is read-only in the execution environment. Set PROJ_USER_WRITABLE_DIRECTORY=/tmp/proj_cache and create the directory at module scope. /tmp is 512 MB by default and provisionable to 10,240 MB — far more than any realistic grid cache needs.

Related

• PROJ and GDAL Runtime Configuration — the environment these grids are read from, and what else has to be set
• Setting GDAL_DATA and PROJ_LIB in Lambda — the exact /opt path this guide writes grids into
• GDAL Config Options That Actually Change COG Read Performance — the range-request tuning that also governs grid fetches over /vsicurl
• Stripping Unnecessary Python Packages from AWS Lambda Layers — where the headroom for a grid set comes from
• Managing /tmp Storage Limits for GeoTIFF Extraction — budgeting the same /tmp the grid cache lives in

Back to PROJ and GDAL Runtime Configuration