Comparing Layers, Container Images, and EFS for GDAL

A zip layer gives you 250 MB unzipped across the function and all five layers combined, mounted at /opt; a container image gives you 10 GB but pays for it in deploy time and a chunk-faulting first invocation; an EFS access point gives you one shared copy at /mnt/geo and drags the function into a VPC. For a stripped GDAL stack under about 200 MB the layer wins on every axis. Past 250 MB the choice is between an image and a share, and it is decided by how many functions read the same GDAL — one or two, take the image; twenty, take the share.

Context

The Python layer management and size reduction workflow exists to keep a geospatial stack inside a single hard number: 250 MB unzipped, counted across the function’s own code and every layer attached to it, with at most five layers per function and everything mounted at /opt. Two techniques buy headroom under that number — stripping unnecessary packages and deduplicating NumPy across layers — and both stop working at the same point. When the stripped, deduplicated stack still does not fit, no further pruning helps, because the remaining bytes are the ones the handler actually calls.

That is the moment this page addresses. The failure it prevents is a team spending a week shaving megabytes off a package that was never going to fit, when the correct move was to change the delivery mechanism. AWS offers three, and they are not variations on a theme: they differ in where the bytes live, when they are paid for, and what else changes when you adopt them.

The stack that forces the choice

Take a concrete example. A hydrology pipeline needs rasterio, fiona, pyproj, shapely, scipy, and a GDAL built with the NetCDF, HDF5, and GRIB drivers so it can read forecast grids directly. After strip --strip-unneeded, after deleting test trees, and after collapsing NumPy into a single shared copy, the tree unzips to 412 MB. There is nothing left to remove — the NetCDF driver alone is the reason the pipeline exists.

Where a 412 MB GDAL stack sits against each delivery mechanism's ceilingFour meters for the same stripped 412 MB stack. As zip layers it needs 412 of the 250 MB unzipped allowance, which is over the ceiling. It would occupy 3 of the 5 available layer slots. As a container image it is 1,240 of the 10,240 MB image allowance. Delivered from an EFS access point it consumes 4.1 of the 10 second initialisation window. Only the zip meter is past its limit.One 412 MB hydrology stack measured against the binding constraint of each mechanismZip layers — unzipped at /optafter stripping, test-tree deletion andNumPy dedup412 / 250 MBZip layers — layer slotsGDAL, the science stack, the sharedNumPy base3 / 5 layersContainer image — uncompressedAL2023 base plus the same 412 MB ofdependencies1,240 / 10,240 MBEFS route — initialisation windowENI attach, NFS mount, then libgdal readover the wire4.1 / 10.0 sMeasured on a 1,769 MB x86_64 function in eu-west-1 running one handler against all three delivery paths.
The stack does not fail on memory, timeout or /tmp — it fails on exactly one number. Once that number is out of reach, further pruning buys nothing and the mechanism has to change.

The zip route does not fail gracefully. UpdateFunctionConfiguration rejects the layer attachment outright, with no partial mode and no way to page bytes in on demand. The image route and the share route both have room; they simply move the cost somewhere else, and the rest of this page is about where.

The three mechanisms compared

Every figure below was measured on a 1,769 MB x86_64 function in eu-west-1 running the same handler against the same 412 MB stack, so the columns are comparable to each other rather than to a vendor benchmark.

Zip layers Container image EFS access point
Size ceiling 250 MB unzipped across function + 5 layers 10 GB image No practical ceiling; the share is the limit
Where it mounts /opt The image’s own filesystem /mnt/geo (you choose the path)
Cold start, first environment 1.9 s 2.3 s 4.1 s
Cold start, once warm on the host 1.9 s 1.4 s 2.8 s
Deploy time for a dependency bump ~40 s (zip, S3, publish-layer-version) 3–6 min (build, push ~1.2 GB to ECR) ~90 s to sync; no function deploy at all
Operational cost S3 storage of the zip only ECR at $0.10 per GB-month EFS at $0.30 per GB-month, plus VPC and NAT
Extra infrastructure None An ECR repository per image VPC, subnets, security groups, mount targets
Blast radius of a bad build One layer version; roll back the ARN One image tag; roll back the tag Every function on the share, immediately

The last row is the one teams under-weight. A layer version and an image tag are both immutable artifacts you roll back by pointing at the previous one. An EFS share is mutable shared state: overwrite libgdal.so.35 on the share and every function reading it changes on its next cold start, with no version to revert to. That is the same property that makes the share attractive for twenty functions and dangerous for two.

Outside AWS this comparison mostly collapses. Neither Cloud Run nor Cloud Functions 2nd gen has a layer mechanism, so the container image is the only route there — with a far more generous envelope of 60 minutes, up to 32 GiB and 8 vCPU, as the multi-stage Cloud Run recipe sets out. Azure Functions on the Consumption plan (10 min, 1,536 MB) is tight enough that a full GDAL stack rarely belongs there at all.

Choosing between them

Choosing between zip layers, a container image, and an EFS access pointA decision with three outcomes. If the stripped stack fits within 250 MB unzipped, use zip layers: they mount at /opt, need no extra infrastructure and deploy in about forty seconds. If the stack exceeds 250 MB and only a handful of functions need it, use a container image: a 10 GB ceiling, a three to six minute build and push, and rollback by image tag. If twenty or more functions share one stack, use an EFS access point: one stored copy, but the functions move into a VPC and the share becomes mutable shared state.Size only decides whether the layer is still available; fan-out decides the restDoes the stripped stack fit in 250 MB,and how many functions read the sameGDAL?fits in 250 MBZip layersMounts at /opt, no extra infrastructure~40 s deploy, roll back by layer ARNover 250 MB, few consumersContainer image10 GB ceiling, ~1.2 GB push to ECR3-6 min deploy, roll back by image tag20+ functions, one stackEFS access pointStored once, updated without redeployVPC required; the share has no version
An image copies the whole GDAL stack into every function that needs it; a share stores it once. At two functions that duplication is free and the VPC is not, and at twenty the arithmetic reverses.

The decision is not really about size. Size only tells you whether the layer is still available. Past that point the question is how many functions read the same bytes: an image duplicates the whole GDAL stack into every function that needs it, and a share stores it once. At two functions the duplication is free and the VPC is not; at twenty the arithmetic reverses.

Prerequisites

  • Runtime: Python 3.11 on x86_64, function memory 1,769 MB, timeout 300 s (the ceiling is 15 min)
  • Zip route: a published layer ARN, built as in the Amazon Linux 2023 layer recipe; lambda:PublishLayerVersion and s3:PutObject on the staging bucket
  • Image route: an ECR repository in the same region as the function; ecr:BatchGetImage, ecr:GetDownloadUrlForLayer on the repository for the Lambda service principal, and ecr:PutImage for the deployer
  • EFS route: a VPC with at least two private subnets, an EFS file system with a mount target per subnet, an access point pinning uid/gid 1000, and elasticfilesystem:ClientMount plus ClientWrite on the access point; the function role also needs ec2:CreateNetworkInterface, DescribeNetworkInterfaces, and DeleteNetworkInterface
  • Gateway VPC endpoint for S3 if the function reads imagery from S3, because a VPC-attached function has no default egress
  • Pinned versions: rasterio==1.4.3, fiona==1.10.1, pyproj==3.7.0, shapely==2.0.6, scipy==1.13.1, GDAL 3.9.0
  • Runtime environment variables, which differ per mechanism and are the single most common migration bug:
    code 
    # zip layers
GDAL_DATA=/opt/share/gdal
PROJ_LIB=/opt/share/proj
LD_LIBRARY_PATH=/opt/lib

# container image
GDAL_DATA=/usr/local/share/gdal
PROJ_LIB=/usr/local/share/proj
LD_LIBRARY_PATH=/usr/local/lib

# EFS access point mounted at /mnt/geo
GDAL_DATA=/mnt/geo/share/gdal
PROJ_LIB=/mnt/geo/share/proj
LD_LIBRARY_PATH=/mnt/geo/lib

Implementation

One SAM template defines all three so the differences are visible side by side rather than spread across three repositories. Deploy it once and you can invoke the same handler through each path and compare Init Duration directly in CloudWatch.

yaml 
# template.yaml — the same GDAL handler delivered three ways.
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Parameters:
  GdalLayerArn:
    Type: String   # arn:aws:lambda:eu-west-1:123456789012:layer:gdal-3-9-py311:7
  EfsAccessPointArn:
    Type: String
  EfsFileSystemId:
    Type: String   # fs-0a1b2c3d4e5f67890
  PrivateSubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
  LambdaSecurityGroupId:
    Type: AWS::EC2::SecurityGroup::Id

Globals:
  Function:
    Runtime: python3.11
    MemorySize: 1769
    Timeout: 300
    Architectures: [x86_64]
    EphemeralStorage:
      Size: 4096          # /tmp: 512 MB by default, 10,240 MB maximum

Resources:

  # --- 1. Zip package + layers. Ceiling is 250 MB unzipped across the
  #        function code and every attached layer, five layers maximum.
  ZipLayerFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: handler/
      Handler: app.lambda_handler
      Layers: [!Ref GdalLayerArn]
      Environment:
        Variables:
          GDAL_DATA: /opt/share/gdal        # layers always mount at /opt
          PROJ_LIB: /opt/share/proj
          LD_LIBRARY_PATH: /opt/lib
          GDAL_PAM_ENABLED: "NO"            # /opt and /var/task are read-only

  # --- 2. Container image. Ceiling is a 10 GB image. Note the absence of a
  #        Layers key: an Image function cannot attach one, and /opt is empty.
  ImageFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      ImageUri: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/gdal-lambda:3.9.0
      ImageConfig:
        Command: ["app.lambda_handler"]
      Environment:
        Variables:
          GDAL_DATA: /usr/local/share/gdal  # wherever the image installed it
          PROJ_LIB: /usr/local/share/proj
          LD_LIBRARY_PATH: /usr/local/lib
          GDAL_PAM_ENABLED: "NO"

  # --- 3. EFS access point. No size ceiling, but the function now lives in a
  #        VPC and every shared object is read over NFS at first touch.
  EfsFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: handler/                     # handler only — no dependencies
      Handler: app.lambda_handler
      VpcConfig:
        SubnetIds: !Ref PrivateSubnetIds
        SecurityGroupIds: [!Ref LambdaSecurityGroupId]
      FileSystemConfigs:
        - Arn: !Ref EfsAccessPointArn
          LocalMountPath: /mnt/geo          # must start with /mnt
      Environment:
        Variables:
          GDAL_DATA: /mnt/geo/share/gdal
          PROJ_LIB: /mnt/geo/share/proj
          LD_LIBRARY_PATH: /mnt/geo/lib
          PYTHONPATH: /mnt/geo/python/lib/python3.11/site-packages
          GDAL_PAM_ENABLED: "NO"
      Policies:
        - EFSWriteAccessPolicy:
            FileSystem: !Ref EfsFileSystemId
            AccessPoint: !Ref EfsAccessPointArn

The EFS route is the only one that changes the shape of a cold start rather than just its length. Before a single byte of GDAL is read, Lambda has to attach a network interface into your VPC and complete an NFS mount, and only then does the dynamic linker start resolving libgdal.so.35 over the network.

The initialisation sequence of an EFS-backed GDAL LambdaA sequence between four participants: Lambda init, the VPC network interface, the EFS access point, and the GDAL tree on the share. Lambda init attaches a hyperplane network interface, which mounts the file system over NFS version 4.1 at /mnt/geo with the access point enforcing uid and gid 1000. Lambda init then reads libgdal.so.35, roughly 22 MB, across the wire, opens proj.db as a SQLite file on the share, and resolves the rasterio import against LD_LIBRARY_PATH pointing at /mnt/geo/lib. The share finally reports drivers registered after about 4.1 seconds.What the EFS route adds before the first line of GDAL is executedLambda initVPC interfaceEFS access pointGDAL on the shareAttach hyperplane ENIonce per execution environmentMount NFS v4.1 at /mnt/geoaccess point pins uid/gid 1000Resolve LD_LIBRARY_PATH/mnt/geo/lib, not /opt/libRead libgdal.so.3522 MB pulled over the wire, not off local diskOpen proj.dbSQLite on NFS — every datum lookup is a network readDrivers registeredcold init total 4.1 s
Two of these steps have no counterpart in the layer or image path. They are paid once per execution environment, not once per invocation — which is why the EFS route suits steady traffic and punishes bursty fan-out.

Verification

Whichever mechanism you pick, verify where libgdal actually came from rather than that import rasterio succeeded. The same handler works against all three, and the resolved path is what tells them apart.

python 
# app.py — prints the mechanism-independent facts that actually differ.
import os
import time
from ctypes.util import find_library


def lambda_handler(event, context):
    t0 = time.perf_counter()

    import rasterio
    from rasterio.crs import CRS
    from osgeo import gdal

    gdal.UseExceptions()
    # Resolve the real on-disk location of the loaded libgdal: /opt/lib for a
    # layer, /usr/local/lib inside an image, /mnt/geo/lib over EFS.
    libgdal = find_library("gdal") or "bundled in the wheel"
    proj_db = os.path.join(os.environ["PROJ_LIB"], "proj.db")

    return {
        "package_type": os.environ.get("AWS_LAMBDA_FUNCTION_NAME", "?"),
        "libgdal_path": libgdal,
        "ld_library_path": os.environ.get("LD_LIBRARY_PATH", "<unset>"),
        "proj_db_present": os.path.exists(proj_db),
        "gdal_version": gdal.VersionInfo("RELEASE_NAME"),
        "driver_count": gdal.GetDriverCount(),
        "epsg_3857_units": CRS.from_epsg(3857).linear_units,
        "import_ms": round((time.perf_counter() - t0) * 1000, 1),
    }

Expected output from the EFS-backed function on a genuinely cold environment:

json 
{
  "package_type": "hydro-efs",
  "libgdal_path": "/mnt/geo/lib/libgdal.so.35",
  "ld_library_path": "/mnt/geo/lib",
  "proj_db_present": true,
  "gdal_version": "3.9.0",
  "driver_count": 251,
  "epsg_3857_units": "metre",
  "import_ms": 2180.4
}

An import_ms near 2,200 on EFS against roughly 700 on a layer is the shared-object read cost, not a bug. proj_db_present: false with a healthy driver count is the classic half-migrated state: LD_LIBRARY_PATH was moved to the new mechanism and PROJ_LIB was not, so reprojection silently falls back to a null datum shift.

Gotchas and Edge Cases

  • An image function cannot attach a layer, and /opt is empty. Setting PackageType: Image silently drops any layer you were relying on — CloudFormation will reject the Layers key outright, but a hand-migrated function simply starts failing at import. Move GDAL_DATA, PROJ_LIB, and LD_LIBRARY_PATH off /opt in the same change that switches the packaging type, never in a follow-up.
  • EFS costs you default internet egress. Attaching the function to a VPC removes its route to the public internet. Reads from S3 need a gateway VPC endpoint, a public STAC catalogue needs a NAT gateway at roughly $32 per month per availability zone before data charges, and neither failure appears until the handler makes its first outbound call — long after the mount succeeded.
  • The share is mutable, so treat writes to it as a deploy. Sync a new GDAL onto /mnt/geo and every function reading it picks up the change on its next cold start, with warm environments still holding the old one. That mixed state is the worst possible way to discover an ABI break. Write versioned directories (/mnt/geo/3.9.0/lib) and move the environment variables, so a rollback is an environment change rather than a second sync.
  • /tmp is unchanged by all three. Every mechanism still gives the function 512 MB of /tmp by default and up to 10,240 MB when configured. Delivering GDAL from a 10 GB image does not give the handler more scratch space for intermediate GeoTIFFs, which is a separate budget covered in ephemeral storage limits in AWS Lambda.

Frequently Asked Questions

Does the 250 MB limit apply to Lambda container images?

No. The 250 MB unzipped ceiling governs zip-packaged functions and counts the function code plus every attached layer together, with a maximum of five layers. A container-image function is bounded instead by a 10 GB image, which is why a GDAL build carrying NetCDF, HDF5, and GRIB drivers usually ends up as an image. Nothing else about the execution envelope changes: the same 15 minute timeout, the same 10,240 MB memory ceiling, and the same 10,240 MB of /tmp apply to both packaging types.

Does mounting EFS put a Lambda function in a VPC?

Yes, and that is the real price of the share. A mount requires the function to sit in subnets with a security group that can reach a mount target, and a VPC-attached function loses default internet egress. S3 then needs a gateway endpoint and anything public needs NAT. Design the VPC before the storage, because the networking work usually exceeds the storage work by an order of magnitude.

Can I mix a zip layer with a container image?

No. Layers apply only to zip-packaged functions. A function with PackageType: Image cannot attach one and /opt stays empty unless the image itself creates it. Everything a layer provided has to be baked into the image or mounted from EFS — which is exactly why the migration is a packaging change and a configuration change at the same time.

Back to Python Layer Management and Size Reduction