Process a 10 GB GeoTIFF with AWS Step Functions and Lambda

A 10 GB GeoTIFF cannot be opened whole inside AWS Lambda — the runtime caps memory at 10,240 MB, /tmp at 10 GB, and wall-clock at 15 minutes per invocation. The working pattern is to tile the scene and let the platform parallelise: an S3 upload triggers a Step Functions state machine, a metadata Lambda writes a manifest of ~400 tile windows, a distributed Map state fans out one Lambda per window (each reading its window through /vsis3 range requests against the Cloud Optimized GeoTIFF), and a merge Lambda reassembles the outputs into a COG registered in a STAC catalog. Peak memory per worker stays near 512 MB regardless of scene size, and the whole 10 GB job completes in two to four minutes of wall-clock at a few hundred-way concurrency. This recipe is the flagship worked example of the serverless geospatial pipeline skeleton.

Why This Recipe Matters for Geospatial Workloads

Large single-file rasters are the default deliverable of aerial survey, drone mosaicking, and hydrological modelling. They arrive as one object and, naively, invite one big processing job — exactly the shape serverless refuses to run. Loading a 40000×40000 pixel, four-band Float32 scene into memory would need roughly 24 GB, more than double the Lambda ceiling, and reading it sequentially would blow the 15-minute timeout ceiling long before the transform finished.

Tiling converts one impossible invocation into hundreds of trivial ones. Because a Cloud Optimized GeoTIFF stores its pixels in independent internal blocks with an offset index, a worker can fetch just the byte ranges covering its window using chunked I/O — no full download, no full decode. Step Functions then supplies the orchestration that raw Lambda lacks: durable state, automatic per-tile retry, bounded concurrency, and a clean fan-in point. The result respects every constraint documented in Serverless Geospatial Architecture & Platform Limits while still turning a 10 GB scene around in minutes.

Platform-by-Platform Limits

Although this recipe is written for AWS, the same tiling maths must be checked against whichever provider you deploy on, because the tile size that fits one runtime overruns another. The figures below are the ceilings each stage is sized against.

The Azure Consumption row is the binding constraint: at 1.5 GB memory a single 2048×2048 four-band Float32 tile (~256 MB decoded, plus GDAL working buffers) is close to the ceiling, so on Azure the manifest must use smaller 1024×1024 tiles. On AWS, the 10 GB memory and disk-backed 10 GB /tmp give comfortable headroom — which is why the flagship recipe targets Lambda. The per-tile packaging of GDAL that each worker depends on follows Packaging & Dependency Management for Serverless GIS.

Step-by-Step Implementation

Step 1: Trigger the State Machine on S3 Upload

Configure an S3 ObjectCreated notification on the ingest prefix (for example s3://geo-ingest/incoming/) to start the state machine. Use EventBridge rather than a direct Lambda notification so the payload starts Step Functions and the input carries the object key.

{
  "detail-type": ["Object Created"],
  "source": ["aws.s3"],
  "detail": {
    "bucket": {"name": ["geo-ingest"]},
    "object": {"key": [{"prefix": "incoming/"}]}
  }
}

The EventBridge rule targets the state machine with an input transformer that passes {"bucket": "geo-ingest", "key": "<object key>"}. Deriving a deterministic run ID from that key keeps the pipeline idempotent, matching the idempotency guidance for S3 events.

Step 2: Extract Metadata and Emit a Tile Manifest

The metadata Lambda opens only the COG header — never pixel data — to read dimensions, block size, CRS, and band count, then writes a manifest of tile windows to S3. The detailed windowing algorithm lives in partitioning a GeoTIFF into Step Functions Map-state tiles; the handler below is the recipe’s entry point.

import json
import os
import boto3
import rasterio
from rasterio.windows import Window

os.environ.setdefault("GDAL_DISABLE_READDIR_ON_OPEN", "EMPTY_DIR")
os.environ.setdefault("CPL_VSIL_CURL_ALLOWED_EXTENSIONS", ".tif,.tiff")

s3 = boto3.client("s3")
TILE = int(os.environ.get("TILE_PIXELS", "2048"))
OUTPUT_BUCKET = os.environ["OUTPUT_BUCKET"]


def handler(event, context):
    bucket, key = event["bucket"], event["key"]
    run_id = key.rsplit("/", 1)[-1].rsplit(".", 1)[0]
    src_uri = f"/vsis3/{bucket}/{key}"

    with rasterio.open(src_uri) as src:
        width, height, count = src.width, src.height, src.count
        profile = {"crs": str(src.crs), "dtype": src.dtypes[0],
                   "transform": list(src.transform)[:6]}

    # Build the manifest of bounded work items — one per tile window.
    tiles = []
    for row_off in range(0, height, TILE):
        for col_off in range(0, width, TILE):
            w = min(TILE, width - col_off)
            h = min(TILE, height - row_off)
            index = f"{row_off:06d}_{col_off:06d}"
            tiles.append({
                "index": index,
                "src_uri": src_uri,
                "col_off": col_off, "row_off": row_off,
                "width": w, "height": h,
                "out_key": f"runs/{run_id}/tiles/{index}.tif",
            })

    manifest_key = f"runs/{run_id}/manifest.json"
    # Newline-delimited JSON so the distributed Map ItemReader can stream it.
    body = "\n".join(json.dumps(t) for t in tiles)
    s3.put_object(Bucket=OUTPUT_BUCKET, Key=manifest_key, Body=body.encode())

    return {
        "run_id": run_id,
        "manifest_bucket": OUTPUT_BUCKET,
        "manifest_key": manifest_key,
        "tile_count": len(tiles),
        "profile": profile,
        "out_prefix": f"runs/{run_id}/tiles/",
        "output_bucket": OUTPUT_BUCKET,
    }

The Lambda returns only a manifest pointer and small summary — never the tile list itself — so the state payload stays far below the 256 KB transition limit.

Step 3: Fan Out Per-Tile Lambdas with a Distributed Map State

The Amazon States Language definition below reads the manifest with ItemReader and runs one tile-worker execution per line, capped at 500-way concurrency so the fan-out cannot exhaust regional quota. A reserved concurrency allocation on the worker function protects it from starvation.

{
  "Comment": "Process a 10 GB GeoTIFF by tiling across Lambda",
  "StartAt": "ExtractMetadata",
  "States": {
    "ExtractMetadata": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${MetadataFunctionArn}",
        "Payload.$": "$"
      },
      "ResultSelector": {"meta.$": "$.Payload"},
      "ResultPath": "$.plan",
      "Next": "ProcessTiles"
    },
    "ProcessTiles": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {"Mode": "DISTRIBUTED", "ExecutionType": "STANDARD"},
        "StartAt": "ProcessTile",
        "States": {
          "ProcessTile": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": {
              "FunctionName": "${TileWorkerFunctionArn}",
              "Payload.$": "$"
            },
            "Retry": [
              {"ErrorEquals": ["States.TaskFailed", "Lambda.TooManyRequestsException"],
               "IntervalSeconds": 2, "MaxAttempts": 4, "BackoffRate": 2.0}
            ],
            "Catch": [
              {"ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
               "Next": "TileToDlq"}
            ],
            "End": true
          },
          "TileToDlq": {
            "Type": "Task",
            "Resource": "arn:aws:states:::sqs:sendMessage",
            "Parameters": {
              "QueueUrl": "${TileDlqUrl}",
              "MessageBody.$": "$"
            },
            "End": true
          }
        }
      },
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:getObject",
        "ReaderConfig": {"InputType": "JSONL"},
        "Parameters": {
          "Bucket.$": "$.plan.meta.manifest_bucket",
          "Key.$": "$.plan.meta.manifest_key"
        }
      },
      "MaxConcurrency": 500,
      "ResultPath": "$.mapResult",
      "Next": "MergeTiles"
    },
    "MergeTiles": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${MergeFunctionArn}",
        "Payload.$": "$.plan.meta"
      },
      "ResultPath": "$.merge",
      "Next": "Catalog"
    },
    "Catalog": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${CatalogFunctionArn}",
        "Payload.$": "$.merge.Payload"
      },
      "End": true
    }
  }
}

Each tile-worker Lambda reads only its window via /vsis3 range requests, applies the transform (here a reprojection and rescale), and writes an independent GeoTIFF to a deterministic key:

import os
import boto3
import numpy as np
import rasterio
from rasterio.windows import Window

# COG-friendly GDAL config, applied before the first rasterio open.
os.environ.setdefault("GDAL_DISABLE_READDIR_ON_OPEN", "EMPTY_DIR")
os.environ.setdefault("GDAL_HTTP_MULTIRANGE", "YES")
os.environ.setdefault("VSI_CACHE", "TRUE")
os.environ.setdefault("GDAL_DATA", "/opt/share/gdal")
os.environ.setdefault("PROJ_LIB", "/opt/share/proj")

s3 = boto3.client("s3")
LOCAL = "/tmp/tile.tif"


def handler(tile, context):
    """Process one manifest window; peak memory is proportional to a single tile."""
    with rasterio.open(tile["src_uri"]) as src:
        window = Window(tile["col_off"], tile["row_off"],
                        tile["width"], tile["height"])
        data = src.read(window=window)                 # only this window's bytes
        transform = src.window_transform(window)
        profile = src.profile.copy()

    # Example per-tile transform: clamp to valid range and cast to float32.
    data = np.clip(data, 0, 10000).astype("float32")

    profile.update(width=tile["width"], height=tile["height"],
                   transform=transform, dtype="float32",
                   driver="GTiff", tiled=True, blockxsize=512, blockysize=512,
                   compress="deflate")

    with rasterio.open(LOCAL, "w", **profile) as dst:
        dst.write(data)

    out_bucket = tile["out_key"].split("/")[0] if False else os.environ["OUTPUT_BUCKET"]
    s3.upload_file(LOCAL, out_bucket, tile["out_key"])
    os.remove(LOCAL)  # release /tmp immediately for the next warm invocation

    return {"index": tile["index"], "out_key": tile["out_key"],
            "bytes": os.path.getsize(LOCAL) if os.path.exists(LOCAL) else None}

Because workers write to deterministic keys under runs/<run_id>/tiles/, a retried tile overwrites rather than duplicates, and the merge stage can enumerate outputs by prefix.

Step 4: Merge Tiled Outputs into a COG

The merge Lambda builds a VRT over every per-tile output and translates it to a single Cloud Optimized GeoTIFF with overviews, watching /tmp as it goes. The full merge implementation, including the /tmp guard and rio-cogeo validation, is documented in merging tiled Lambda outputs into a COG. In outline:

import os
import boto3
from osgeo import gdal

s3 = boto3.client("s3")
gdal.UseExceptions()


def handler(meta, context):
    bucket, prefix = meta["output_bucket"], meta["out_prefix"]
    # Enumerate tile outputs and reference them in-place via /vsis3 (no download).
    keys = [o["Key"] for page in s3.get_paginator("list_objects_v2")
            .paginate(Bucket=bucket, Prefix=prefix)
            for o in page.get("Contents", [])]
    vsis3 = [f"/vsis3/{bucket}/{k}" for k in keys]

    vrt_path = "/tmp/mosaic.vrt"
    gdal.BuildVRT(vrt_path, vsis3)

    cog_local = "/tmp/product.tif"
    gdal.Translate(
        cog_local, vrt_path,
        format="COG",
        creationOptions=["COMPRESS=DEFLATE", "OVERVIEWS=AUTO", "BLOCKSIZE=512"],
    )

    out_key = f"runs/{meta['run_id']}/product.tif"
    s3.upload_file(cog_local, bucket, out_key)
    return {"run_id": meta["run_id"], "product_bucket": bucket,
            "product_key": out_key, "crs": meta["profile"]["crs"]}

Step 5: Register a STAC Item

The catalog Lambda writes a STAC item describing the product’s bounds, CRS, and asset href to the catalog prefix, making the result queryable without scanning the bucket.

import json
import boto3

s3 = boto3.client("s3")


def handler(product, context):
    item = {
        "type": "Feature",
        "stac_version": "1.0.0",
        "id": product["run_id"],
        "properties": {"proj:epsg": product["crs"]},
        "assets": {
            "data": {
                "href": f"s3://{product['product_bucket']}/{product['product_key']}",
                "type": "image/tiff; application=geotiff; profile=cloud-optimized",
                "roles": ["data"],
            }
        },
    }
    s3.put_object(
        Bucket=product["product_bucket"],
        Key=f"catalog/{product['run_id']}.json",
        Body=json.dumps(item).encode(),
        ContentType="application/json",
    )
    return {"stac_id": product["run_id"], "status": "cataloged"}

Measurement and Verification

Confirm the pipeline stayed within limits by inspecting the Step Functions execution and the per-tile metrics. A healthy run shows every tile worker peaking well below its memory allocation and finishing in seconds:

# Inspect the most recent execution and its Map run distribution
aws stepfunctions describe-execution \
  --execution-arn "$EXECUTION_ARN" \
  --query '{status:status, started:startDate, stopped:stopDate}'

# Tile-worker memory headroom from CloudWatch Lambda Insights
aws cloudwatch get-metric-statistics \
  --namespace "LambdaInsights" --metric-name "memory_utilization" \
  --dimensions Name=function_name,Value=tile-worker \
  --start-time "$(date -u -d '-1 hour' +%FT%TZ)" \
  --end-time "$(date -u +%FT%TZ)" \
  --period 300 --statistics Maximum

Expected shape of a successful 10 GB run (40000×40000, 4-band, 2048 px tiles):

status        : SUCCEEDED
tile_count    : 400
map_max_conc  : 500
worker p95 mem: 1180 MB / 2048 MB allocated
worker p95 dur: 24 s
end_to_end    : ~3 min
product       : s3://geo-ingest/runs/<id>/product.tif  (valid COG)

Validate the merged product is a genuine COG with rio cogeo validate s3://.../product.tif — a non-zero exit means the overviews or internal tiling were not written and downstream range reads will be slow.

Failure Modes and Debugging

States.DataLimitExceeded on the Map transition: The manifest was inlined into the state payload instead of read via ItemReader, breaching the 256 KB limit. Confirm the metadata Lambda returns only a manifest pointer and that ItemReader points at the S3 object — the fix is detailed in partitioning a GeoTIFF into Map-state tiles.

Tile worker Task timed out after 900.00 seconds: A window is too large or the source is not a real COG, so each read pulls far more than its bytes. Verify the source with rio cogeo validate; if it is a striped (non-tiled) GeoTIFF, add a one-off translation step to a COG before tiling, or shrink TILE_PIXELS.

Runtime.OutOfMemory in the worker: The decoded tile plus GDAL buffers exceeded the allocation. Either raise the worker memory tier (which also adds vCPU per the memory and CPU allocation model) or reduce tile pixels; a four-band Float32 2048×2048 tile decodes to ~256 MB before working copies.

/tmp fills during merge — [Errno 28] No space left on device: The merge Lambda downloaded tiles instead of referencing them via /vsis3, or the output COG plus VRT exceeded 10 GB /tmp. Reference tiles in place and stream the translate; the ephemeral storage limits page covers provisioning and guarding /tmp.

Silent throttling storm: The Map concurrency exceeded the worker’s reserved concurrency and tiles retried in a loop. Cap MaxConcurrency below the reserved allocation and watch map_concurrency against the regional quota.

Cost and Scaling Considerations

The bill for this recipe is dominated by GB-seconds across the fan-out, and it scales almost linearly with pixel count rather than with the number of tiles — smaller tiles simply mean more, shorter invocations for roughly the same total compute. A 10 GB scene at 400 tiles × 24 s × 2 GB is about 19,000 GB-seconds, a few cents at current Lambda pricing, plus a small Step Functions state-transition charge. The larger cost lever is data transfer: keep source, tiles, and product in one region behind a VPC endpoint so range reads and writes never traverse NAT.

Scaling up follows the 15-minute ceiling chunking rule: for a 100 GB scene, hold tile pixels constant and let the tile count grow, raising MaxConcurrency toward the distributed Map’s 10,000-child limit while keeping worker reserved concurrency comfortably above it. If a single tile ever cannot be subdivided below the memory ceiling — a pathological block layout — route just that tile to a Fargate task via the circuit-breaker pattern, preserving the serverless profile for the rest of the run. Provisioning provisioned concurrency on the worker is worthwhile only for latency-critical pipelines, since the fan-out already hides cold starts behind parallelism.

Frequently Asked Questions

How can Lambda process a 10 GB GeoTIFF within a 10 GB memory limit?

It never loads the whole file. Each tile-worker Lambda reads only its window through /vsis3 range requests against the Cloud Optimized GeoTIFF, so peak memory is proportional to one tile — typically 256 to 512 MB decoded — not the 10 GB scene. The distributed Map state runs hundreds of these workers in parallel.

How many tiles should a 10 GB GeoTIFF be split into?

Size tiles so one worker finishes well under the 15-minute timeout and peaks under its memory allocation. For a 40000×40000 pixel scene, 2048×2048 tiles yield about 400 windows, each processed in 10–40 seconds at a 2 GB tier — a safe margin against both ceilings.

Why use a distributed Map state instead of a standard Map?

A standard inline Map holds all iterations in the 256 KB state payload and caps at 40 concurrent iterations. A distributed Map reads items from an S3 manifest with ItemReader, supports up to 10,000 concurrent child executions, and keeps the payload tiny — essential when a scene tiles into hundreds or thousands of windows.

What happens when one tile fails?

The Map state retries that tile with exponential backoff, then routes it to a dead-letter queue while sibling tiles complete. Because each tile writes to a deterministic S3 key, re-running reprocesses only the failed window.

Related

• Partitioning a GeoTIFF into Step Functions Map-State Tiles — how the metadata Lambda computes the tile manifest and keeps payloads under 256 KB
• Merging Tiled Lambda Outputs into a Cloud Optimized GeoTIFF — the VRT-to-COG merge step with overviews and /tmp guarding
• Cold Start Mapping for Python GDAL — the initialization cost each tile worker pays across the fan-out
• Ephemeral Storage Limits in AWS Lambda — provisioning and guarding /tmp in the worker and merge stages
• Chunked I/O for Large Satellite Imagery — the window and range-read mechanics behind /vsis3 tile reads

Back to Serverless Geospatial Pipeline Recipes