Partitioning a GeoTIFF into Step Functions Map-State Tiles

Partition the raster in the metadata Lambda by reading only the COG header — width, height, and internal block size — then emit one window record per tile as newline-delimited JSON to S3, snapping tile dimensions to whole multiples of the block size (usually 512 px). Return only the manifest’s S3 URI from the Lambda and point the distributed Map state’s ItemReader at it, so the state payload carries a ~100-byte pointer instead of thousands of window objects and never approaches the 256 KB Step Functions transition limit.

GeoTIFF partitioning into a Map manifest A metadata Lambda reads the COG header for width, height, and block size, computes block-aligned tile windows, writes them as newline-delimited JSON to S3, and returns only the manifest URI. The distributed Map state's ItemReader streams the manifest so the state payload stays tiny. COG header 40000×40000 block 512 Metadata Lambda block-aligned window loop S3 manifest manifest.jsonl ~400 lines Map ItemReader streams JSONL 1 tile / child state payload = manifest URI only (~100 bytes) · never the tile list · 256 KB limit safe Header to manifest to ItemReader — tile count scales freely; the transition payload stays constant.

Context

The 10 GB GeoTIFF recipe depends entirely on the manifest produced here: it is the contract between the planner and the fan-out. Get the windowing right and every downstream tile worker reads cleanly aligned blocks and finishes in seconds; get it wrong and you either overflow the state payload or force each worker into wasteful partial-block reads. Two hard limits shape the design. The first is the Step Functions 256 KB payload cap on data passed between states — inline a few thousand tile objects and the execution fails with States.DataLimitExceeded before a single worker runs. The second is the COG’s internal block size: windows that do not align to block boundaries make GDAL fetch and discard partial blocks over HTTP, multiplying the range-request count that the chunked I/O for large satellite imagery patterns work to minimise.

The distributed Map state’s ItemReader resolves the first limit by streaming the manifest from S3 rather than carrying it in the payload, so the manifest can describe ten tiles or ten thousand without changing the transition size. Block alignment resolves the second. This page covers only the partitioning stage; the merge that reassembles the results is handled in merging tiled Lambda outputs into a COG.

The tile size is the single lever that balances three competing pressures. Larger tiles mean fewer invocations and less orchestration overhead, but each worker holds more decoded pixels in memory and runs closer to the 15-minute timeout ceiling. Smaller tiles keep every worker comfortably within its memory tier but multiply the invocation count, the cold-start surface, and the manifest length. For a four-band Float32 scene, a 2048-pixel tile decodes to roughly 256 MB before GDAL’s working buffers — a safe fit at a 2 GB worker tier — while a 4096-pixel tile quadruples that to ~1 GB and starts to crowd the allocation. The metadata Lambda encodes this decision once, in the TILE_PIXELS environment variable, so the same partitioning code serves any provider simply by lowering the value to fit a tighter memory ceiling such as Azure’s 1.5 GB.

Prerequisites

  • Runtime: Python 3.11 on AWS Lambda with a rasterio layer (GDAL 3.9+). See Packaging & Dependency Management for Serverless GIS for the layer build.
  • IAM: the metadata Lambda role needs s3:GetObject on the source prefix and s3:PutObject on the manifest prefix — nothing more, per IAM security boundaries for Cloud GIS.
  • Source must be a real COG: internally tiled with an offset index. Validate with rio cogeo validate; a striped GeoTIFF makes block-aligned windowing meaningless because every window read pulls whole scanlines.
  • Environment variables (set on the function, applied before the first rasterio.open):
    code 
    GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif,.tiff
GDAL_DATA=/opt/share/gdal
PROJ_LIB=/opt/share/proj
  • Manifest bucket distinct from, or prefixed separately within, the ingest bucket so lifecycle rules can expire manifests independently of source data.

Implementation

The function below reads the header, snaps the requested tile size down to a whole multiple of the block size, iterates windows in row-major order, and streams the manifest to S3 as newline-delimited JSON. It returns only pointers and a summary — never the window list.

python 
#!/usr/bin/env python3
"""metadata_partition.py — Lambda: partition a COG into a Map-state manifest.

Reads only the COG header, emits block-aligned tile windows as JSONL to S3,
and returns the manifest URI so the state payload stays under 256 KB.
"""
import io
import json
import os

import boto3
import rasterio

# COG-friendly config so opening the header issues minimal range requests.
os.environ.setdefault("GDAL_DISABLE_READDIR_ON_OPEN", "EMPTY_DIR")
os.environ.setdefault("CPL_VSIL_CURL_ALLOWED_EXTENSIONS", ".tif,.tiff")

s3 = boto3.client("s3")

TARGET_TILE = int(os.environ.get("TILE_PIXELS", "2048"))  # requested tile edge
OUTPUT_BUCKET = os.environ["OUTPUT_BUCKET"]


def _align_down(value: int, block: int) -> int:
    """Snap a tile edge down to a whole multiple of the block size (min one block)."""
    return max(block, (value // block) * block)


def handler(event, context):
    bucket, key = event["bucket"], event["key"]
    run_id = key.rsplit("/", 1)[-1].rsplit(".", 1)[0]
    src_uri = f"/vsis3/{bucket}/{key}"

    with rasterio.open(src_uri) as src:
        width, height = src.width, src.height
        # Internal block size — the atomic COG read unit (often 512x512).
        bx, by = src.block_shapes[0][1], src.block_shapes[0][0]
        crs, dtype, count = str(src.crs), src.dtypes[0], src.count
        transform6 = list(src.transform)[:6]

    tile_w = _align_down(TARGET_TILE, bx)
    tile_h = _align_down(TARGET_TILE, by)

    # Stream the manifest straight to S3 as newline-delimited JSON.
    buf = io.StringIO()
    tile_count = 0
    for row_off in range(0, height, tile_h):
        for col_off in range(0, width, tile_w):
            w = min(tile_w, width - col_off)
            h = min(tile_h, height - row_off)
            index = f"{row_off:06d}_{col_off:06d}"
            buf.write(json.dumps({
                "index": index,
                "src_uri": src_uri,
                "col_off": col_off, "row_off": row_off,
                "width": w, "height": h,
                "out_key": f"runs/{run_id}/tiles/{index}.tif",
            }))
            buf.write("\n")
            tile_count += 1

    manifest_key = f"runs/{run_id}/manifest.jsonl"
    s3.put_object(Bucket=OUTPUT_BUCKET, Key=manifest_key,
                  Body=buf.getvalue().encode(), ContentType="application/x-ndjson")

    # Return pointers + summary ONLY — the state payload must not carry the tiles.
    return {
        "run_id": run_id,
        "manifest_bucket": OUTPUT_BUCKET,
        "manifest_key": manifest_key,
        "tile_count": tile_count,
        "tile_w": tile_w, "tile_h": tile_h,
        "block": [by, bx],
        "profile": {"crs": crs, "dtype": dtype, "count": count, "transform": transform6},
        "out_prefix": f"runs/{run_id}/tiles/",
        "output_bucket": OUTPUT_BUCKET,
    }

The corresponding ItemReader in the state machine consumes this manifest without ever inlining it. Note that only manifest_bucket and manifest_key cross the state boundary:

json 
"ItemReader": {
  "Resource": "arn:aws:states:::s3:getObject",
  "ReaderConfig": {"InputType": "JSONL"},
  "Parameters": {
    "Bucket.$": "$.plan.meta.manifest_bucket",
    "Key.$": "$.plan.meta.manifest_key"
  }
}

Verification

Confirm the manifest is well-formed, block-aligned, and small enough that its pointer — not its content — is what flows through the state machine.

bash 
# Pull the manifest and inspect the first record + line count
aws s3 cp "s3://$OUTPUT_BUCKET/runs/$RUN_ID/manifest.jsonl" - | head -n 1 | python -m json.tool
LINES=$(aws s3 cp "s3://$OUTPUT_BUCKET/runs/$RUN_ID/manifest.jsonl" - | wc -l)
echo "tiles: $LINES"

# Assert every tile edge is a multiple of the block size (512)
aws s3 cp "s3://$OUTPUT_BUCKET/runs/$RUN_ID/manifest.jsonl" - \
  | python -c "import sys,json; \
bad=[l for l in sys.stdin if (json.loads(l)['col_off']%512) or (json.loads(l)['row_off']%512)]; \
print('misaligned:', len(bad))"

Expected output for a 40000×40000 COG with 512-px blocks and 2048-px tiles:

code 
{
    "index": "000000_000000",
    "src_uri": "/vsis3/geo-ingest/incoming/scene.tif",
    "col_off": 0, "row_off": 0, "width": 2048, "height": 2048,
    "out_key": "runs/scene/tiles/000000_000000.tif"
}
tiles: 400
misaligned: 0

The metadata Lambda’s own returned payload should be well under 1 KB — verify in the Step Functions execution event history that the ExtractMetadata result contains a manifest_key, not a tiles array.

Gotchas and Edge Cases

  • Edge tiles are smaller than the tile size. The last column and row use min(tile, remaining), so they are not block-aligned in extent — that is fine. Alignment matters for the offset (where the read starts), which stays on a block boundary; a short trailing tile still reads whole blocks up to the raster edge.

  • block_shapes reports the first band only. For a pixel-interleaved multi-band COG all bands share a block layout, but band-interleaved files can differ. Read src.block_shapes per band and take the common divisor if they diverge, or the alignment guarantee breaks for some bands.

  • A manifest under ~1,200 tiles could be inlined, but do not. It is tempting to skip S3 for small scenes, but keeping one code path — always ItemReader from S3 — means a scene that grows past the threshold never silently trips States.DataLimitExceeded in production.

  • Very high tile counts hit the distributed Map child ceiling, not the payload. Above ~10,000 tiles you approach the distributed Map concurrent-child limit; increase tile size or process in batched sub-manifests rather than shrinking tiles further.

  • Overviews are not tiles. The manifest partitions only the full-resolution image. If the source COG carries internal overviews, do not add manifest entries for them — the merge stage regenerates overviews from the reassembled full-resolution mosaic. Emitting overview windows here would double-process pixels and corrupt the output extent.

  • CRS lives in the summary, not per tile. Every tile inherits the source CRS and transform, so record them once in the returned profile rather than repeating them on each manifest line. Repeating the transform on 400 lines inflates the manifest and tempts a worker to trust a stale copy instead of reading window_transform from the source.

Frequently Asked Questions

Why does the tile manifest have to live in S3 rather than in the state payload?

Step Functions caps the data passed between states at 256 KB. A single JSON tile record is roughly 200 bytes, so a scene of more than about 1,200 tiles overflows the limit if inlined. Writing the manifest to S3 and reading it with the distributed Map ItemReader keeps the payload to a small URI regardless of tile count.

Should tile size be aligned to the COG block size?

Yes. Align tile width and height to whole multiples of the internal block size — typically 512 pixels — so each windowed read fetches complete blocks. Misaligned windows force GDAL to fetch and discard partial blocks, inflating range-request count and cost.

What format should the manifest use for ItemReader?

Newline-delimited JSON (JSONL) or CSV. The distributed Map ItemReader streams JSONL line by line, so the manifest can be arbitrarily large without ever being loaded whole into memory or into the state payload.

Back to Process a 10 GB GeoTIFF with AWS Step Functions and Lambda