How to Configure 10 GB Memory for AWS Lambda Raster Processing

Set MemorySize to 10240 and EphemeralStorage.Size to 10240 in your Lambda function configuration. At this ceiling AWS allocates approximately 5.79 vCPUs — enabling parallel tile decoding, vectorized NumPy band math, and concurrent GDAL reprojection threads. Pair maximum memory with windowed I/O and explicit GDAL cache limits to prevent out-of-memory crashes that occur even at 10 GB when full rasters are loaded naively.

Context

The core tension in serverless raster processing is that Memory and CPU Allocation for Raster Workloads on AWS Lambda couples vCPU count directly to RAM tier: 1,769 MB per full vCPU. Below the 3,008 MB mark your function operates on a fraction of a single core, making CPU-bound operations like reprojection and band algebra painfully slow regardless of dataset size. At 10,240 MB you reach the current hard ceiling — roughly 6 vCPUs and network/disk I/O bandwidth that scales proportionally.

The failure mode that drives engineers to this configuration is predictable: a multi-band Sentinel-2 mosaic or a DEM hydrological conditioning job runs fine on a 3 GB memory tier for small tiles but OOMs or times out on production-scale inputs. Allocating 10 GB removes the compute bottleneck while Ephemeral Storage Limits in AWS Lambda — kept at the default 512 MB — silently starves GDAL’s disk cache and intermediate VRT staging. Both parameters must be raised together.

The diagram below shows how memory tier maps to vCPU allocation and where the 10 GB configuration sits relative to common raster workload sizes:

vCPU delivered at four Lambda memory tiers and the workload each one carriesFour meters measured against the 5.79 vCPU ceiling: 1,769 MB delivers one vCPU and suits a single-band rescale, 3,538 MB delivers two for a multi-band tile, 7,077 MB delivers four for a Sentinel-2 mosaic, and 10,240 MB delivers 5.79 — the hard ceiling — for DEM hydrological conditioning.Every tier as a fraction of the 5.79 vCPU ceiling1,769 MBsingle-band GeoTIFF rescale1.00 vCPU3,538 MBmulti-band tile, parallel band math2.00 vCPU7,077 MBSentinel-2 mosaic, 4 bands float324.00 vCPU10,240 MB — the hard ceilingDEM hydrological conditioning5.79 vCPUAWS grants one vCPU per 1,769 MB of configured memory, so 10,240 MB is worth approximately 5.79 vCPU — the most CPU a single Lambdainvocation can hold.
There is no tier above the last bar. Once a job needs more than 5.79 vCPU the answer is a smaller unit of work or a container runtime, not a larger function.

Prerequisites

Before raising the memory ceiling, confirm the following are in place:

  • Runtime: Python 3.11 or 3.12. Earlier runtimes lack the improved gc behaviour needed for reliable C-extension deallocation.
  • IAM permission: lambda:UpdateFunctionConfiguration on the target function ARN. Without this the AWS CLI and Terraform will silently fall back to current settings.
  • Environment variables set explicitly: GDAL_DATA, PROJ_LIB, and LD_LIBRARY_PATH must point to your Lambda layer paths. Never rely on defaults inside the execution environment.
  • Dependency versions: rasterio>=1.3, numpy>=1.24, GDAL>=3.6. Earlier versions have known memory-leak paths in windowed write operations.
  • Service quota: Verify your account’s Lambda function-and-layer-storage and per-function memory quota via the Service Quotas console. New accounts are sometimes capped below 10 GB until a quota increase is approved.
  • EphemeralStorage configured alongside MemorySize: As explained under Ephemeral Storage Limits in AWS Lambda, the /tmp partition does not expand automatically when you raise RAM — it must be provisioned separately.

Implementation

The following snippets show the three most common configuration paths. After configuration, the Python handler demonstrates the windowed-I/O pattern that stabilises peak memory at 10 GB without OOM.

The five configuration steps that make the 10 GB tier usableFive ordered steps with the exact parameter for each: confirm the per-region service quota, set MemorySize to 10240, set EphemeralStorage to 10240 separately because /tmp does not follow memory, cap the GDAL cache at 2048 MB, and confirm Max Memory Used in the REPORT line stays under 8,000 MB.Five settings, or the 10 GB tier buys you nothing1Confirm the regional quotaPer-function memory quota is per region, not per accountService Quotas2Raise MemorySize to the ceilingUnlocks approximately 5.79 vCPU for parallel tile decode102403Raise ephemeral storage separately/tmp does not follow MemorySize; the default stays 512 MB102404Cap the GDAL cache2 GB of cache leaves headroom for numpy buffers and window arraysGDAL_CACHEMAX=20485Read it back from the REPORT lineMax Memory Used must stay below 8,000 MB on a representative tileMax Memory Used
Steps two and three are the pair that gets missed. Raising memory alone leaves /tmp at its 512 MB default, so a job that now has six cores still fails staging a single intermediate VRT.

AWS CLI

bash 
aws lambda update-function-configuration \
  --function-name raster-processor \
  --memory-size 10240 \
  --timeout 900 \
  --ephemeral-storage '{"Size": 10240}' \
  --environment 'Variables={
    GDAL_CACHEMAX=2048,
    GDAL_NUM_THREADS=4,
    GDAL_DATA=/opt/share/gdal,
    PROJ_LIB=/opt/share/proj,
    LD_LIBRARY_PATH=/opt/lib,
    GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
  }'

Terraform

hcl 
resource "aws_lambda_function" "raster_processor" {
  function_name = "raster-processor"
  runtime       = "python3.11"
  handler       = "handler.process_tile"
  memory_size   = 10240
  timeout       = 900

  ephemeral_storage {
    size = 10240
  }

  environment {
    variables = {
      # Hard limits: GDAL cache capped at 2 GB to leave headroom for numpy buffers
      GDAL_CACHEMAX                  = "2048"
      GDAL_NUM_THREADS               = "4"
      # Explicit paths — never rely on container defaults
      GDAL_DATA                      = "/opt/share/gdal"
      PROJ_LIB                       = "/opt/share/proj"
      LD_LIBRARY_PATH                = "/opt/lib"
      GDAL_DISABLE_READDIR_ON_OPEN   = "EMPTY_DIR"
    }
  }
}

AWS SAM

yaml 
Resources:
  RasterProcessor:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.11
      Handler: handler.process_tile
      MemorySize: 10240
      Timeout: 900
      EphemeralStorage:
        Size: 10240
      Environment:
        Variables:
          GDAL_CACHEMAX: "2048"
          GDAL_NUM_THREADS: "4"
          GDAL_DATA: "/opt/share/gdal"
          PROJ_LIB: "/opt/share/proj"
          LD_LIBRARY_PATH: "/opt/lib"
          GDAL_DISABLE_READDIR_ON_OPEN: "EMPTY_DIR"

Production handler with windowed I/O

python 
import os
import gc
import uuid
import logging
import rasterio
from rasterio.windows import Window
import numpy as np

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)


def _check_tmp_headroom(required_mb: int = 2048) -> None:
    """Abort early if /tmp does not have enough free space."""
    stat = os.statvfs("/tmp")
    free_mb = (stat.f_frsize * stat.f_bavail) / (1024 * 1024)
    if free_mb < required_mb:
        raise RuntimeError(
            f"/tmp headroom too low: {free_mb:.0f} MB free, {required_mb} MB required"
        )


def process_tile(event: dict, context) -> dict:
    """
    Process a raster tile from S3 using windowed reads.
    Requires MemorySize=10240, EphemeralStorage=10240, and explicit GDAL env vars.
    """
    # Convert s3:// URI to the GDAL virtual filesystem path so rasterio
    # streams byte ranges directly without a full download to /tmp.
    src_path = event["s3_uri"].replace("s3://", "/vsis3/")

    # Isolate each invocation under a unique subdirectory so warm-start
    # container reuse does not corrupt previous outputs.
    invocation_dir = f"/tmp/{uuid.uuid4().hex}"
    os.makedirs(invocation_dir, exist_ok=True)
    dst_path = os.path.join(invocation_dir, os.path.basename(src_path))

    # Guard against /tmp exhaustion before writing any data.
    _check_tmp_headroom(required_mb=2048)

    window_size = 1024  # pixels per tile dimension; align with COG block size

    with rasterio.open(src_path) as src:
        profile = src.profile.copy()
        profile.update(
            dtype=rasterio.uint16,
            count=1,
            compress="deflate",
            tiled=True,
            blockxsize=512,
            blockysize=512,
            # Ensure GDAL writes the COG-compatible layout
            interleave="band",
        )

        with rasterio.open(dst_path, "w", **profile) as dst:
            for row_off in range(0, src.height, window_size):
                for col_off in range(0, src.width, window_size):
                    width  = min(window_size, src.width  - col_off)
                    height = min(window_size, src.height - row_off)
                    window = Window(col_off, row_off, width, height)

                    # Read only the current 1024×1024 window — not the full raster.
                    # At 10240 MB, ~6 vCPUs decode tiles in parallel across bands.
                    data = src.read(window=window)

                    # Band normalisation: scale uint16 to 0-100 integer range
                    processed = np.clip(
                        (data.astype(np.float32) / 65535.0 * 100),
                        0, 100
                    ).astype(np.uint16)

                    dst.write(processed, window=window)

                    # Explicit deallocation releases C-extension buffers before
                    # gc runs automatically — critical for back-to-back windows.
                    del data, processed
                    gc.collect()

    logger.info("Processed %s → %s", src_path, dst_path)
    return {"statusCode": 200, "output_path": dst_path}

Verification

Run the following after deployment to confirm the memory tier is active and peak usage is within bounds:

bash 
# Invoke the function with a representative test event and capture the log output
aws lambda invoke \
  --function-name raster-processor \
  --cli-binary-format raw-in-base64-out \
  --payload '{"s3_uri":"s3://your-bucket/test-tile.tif"}' \
  --log-type Tail \
  response.json \
  --query 'LogResult' --output text | base64 --decode

Expected output excerpt:

code 
REPORT RequestId: … Duration: 4321.00 ms  Billed Duration: 4400 ms
Memory Size: 10240 MB  Max Memory Used: 3847 MB  Init Duration: 2100.00 ms

Max Memory Used should stay below 8,000 MB under normal raster workloads. If it regularly exceeds 9,000 MB, reduce the window_size from 1024 to 512 pixels per dimension and verify GDAL_CACHEMAX is honoured:

python 
import os
from osgeo import gdal

# Confirm GDAL sees the cache cap at runtime
cache_mb = gdal.GetCacheMax() / (1024 * 1024)
print(f"GDAL cache cap: {cache_mb:.0f} MB")  # Expected: 2048 MB
assert cache_mb == 2048, f"GDAL_CACHEMAX not applied — got {cache_mb} MB"
Full-raster read versus 1024 by 1024 windowed reads at the 10 GB tierTwo panels. A full-raster read of a 10,000 by 10,000 four-band float32 scene is 1.6 GB of raw pixels but peaks at 4 to 6 GB once the GDAL cache, numpy copies and collector lag are counted, and an out-of-memory kill at the ceiling has nowhere left to go. Windowed reads hold one 1024 by 1024 window at a time, peak at the measured 3,847 MB of 10,240 MB, and can be halved to 512 pixels if peak usage passes 9,000 MB.The tier is not the thing that stops the OOMFull-raster readsrc.read() on a 10,000 x 10,000 four-band float32 scene1.6 GB of raw pixels, 4–6 GB peak with cache and copiesOne collector lag away from a status 137 killAlready at the ceiling, so there is no larger tier to move to1024 x 1024 windowed readsOne window resident, written before the next is readMax Memory Used 3,847 MB of 10,240 MB in the REPORTlineGDAL_CACHEMAX pinned at 2,048 MB and asserted atruntimeHalve the window to 512 px if peak passes 9,000 MBThe tier buys vCPU; the windowed loop is what holds peak usage at 3,847 MB.
Both configurations are the same 10,240 MB function. The difference is entirely in the read pattern, which is why the OOM survives a memory increase but not a windowed loop.

To query CloudWatch directly:

bash 
aws cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name MaxMemoryUsed \
  --dimensions Name=FunctionName,Value=raster-processor \
  --start-time "$(date -u -d '-1 hour' '+%Y-%m-%dT%H:%M:%SZ')" \
  --end-time "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" \
  --period 300 \
  --statistics Maximum \
  --output table

Gotchas and Edge Cases

  • GDAL_NUM_THREADS=ALL_CPUS causes thread contention. Lambda’s execution environment shares host-level CPU resources. Pinning to an explicit integer (4 for 10 GB, 2 for tiers below 7 GB) prevents the thread pool from spawning more workers than vCPUs actually available to your container slice.

  • Warm-start /tmp accumulation. Lambda reuses containers across invocations. If your handler writes to a fixed /tmp/output.tif path, a second invocation on the same container can inherit stale data. Use the uuid.uuid4().hex subdirectory pattern shown above and delete the directory at the end of the handler.

  • InitDuration spikes at 10 GB. Cold-start overhead increases slightly at higher memory tiers because AWS provisions a larger execution environment. The Cold Start Mapping for Python GDAL sequence shows that GDAL shared-library resolution accounts for 60–80% of InitDuration. Use Lambda Layers with pre-built GDAL binaries and enable Provisioned Concurrency for latency-sensitive pipelines.

  • Service quota is per-region. A quota increase in us-east-1 does not propagate to eu-west-1. If you deploy to multiple regions, raise the quota in each independently via the AWS Service Quotas console before applying Terraform.

Frequently Asked Questions

Does raising MemorySize increase my per-invocation cost?

Yes. Lambda billing is Duration × MemorySize in GB-seconds. At 10 GB, a 5-second invocation costs 50 GB-seconds versus 25 GB-seconds at 5 GB. The cost-per-tile crossover depends on how much the extra vCPUs reduce duration. For CPU-bound reprojection jobs, measured speedup typically reaches 2–3× at the 10 GB tier, making the net cost flat or slightly lower despite the higher per-ms rate. For I/O-bound workloads (streaming COGs from S3), memory above 5–6 GB rarely improves duration enough to justify the cost.

Can I use multiprocessing instead of windowed reads to saturate the 6 vCPUs?

Avoid multiprocessing.Pool in Lambda. Forking inside the execution environment is unreliable — child processes inherit file descriptors, GDAL drivers, and PROJ context that were initialised for the parent, leading to silent corruption. Use concurrent.futures.ThreadPoolExecutor instead; rasterio releases the GIL during file I/O, allowing multiple threads to download and decode windows concurrently across the available vCPUs.

Back to Serverless Geospatial Architecture & Platform Limits