Partitioning a GeoTIFF into Step Functions Map-State Tiles

Partition the raster in the metadata Lambda by reading only the COG header — width, height, and internal block size — then emit one window record per tile as newline-delimited JSON to S3, snapping tile dimensions to whole multiples of the block size (usually 512 px). Return only the manifest’s S3 URI from the Lambda and point the distributed Map state’s ItemReader at it, so the state payload carries a ~100-byte pointer instead of thousands of window objects and never approaches the 256 KB Step Functions transition limit.

Context

The 10 GB GeoTIFF recipe depends entirely on the manifest produced here: it is the contract between the planner and the fan-out. Get the windowing right and every downstream tile worker reads cleanly aligned blocks and finishes in seconds; get it wrong and you either overflow the state payload or force each worker into wasteful partial-block reads. Two hard limits shape the design. The first is the Step Functions 256 KB payload cap on data passed between states — inline a few thousand tile objects and the execution fails with States.DataLimitExceeded before a single worker runs. The second is the COG’s internal block size: windows that do not align to block boundaries make GDAL fetch and discard partial blocks over HTTP, multiplying the range-request count that the chunked I/O for large satellite imagery patterns work to minimise.

Block-aligned tile windows over a COG's internal 512 pixel blocksAn eight by eight arrangement of 512 pixel internal COG blocks, coloured into four quadrants. Each quadrant of sixteen blocks is one 2048 pixel tile window from the manifest, so every window offset falls on a whole multiple of 512 and every read fetches complete blocks.One 2048 px window is exactly 4 x 4 COG blocks4 x 4 blocks of 512 px = one2048 px window_align_down() snaps the requested2,048 px edge to a whole multiple of the512 px block size. An offset of 1,800px instead would straddle two blockcolumns, forcing GDAL to fetch eachboundary block twice and discard mostof it.row 0, col 0row 0, col 2048row 2048, col 0row 2048, col 2048A 40,000 x 40,000 scene tiles into 400 such windows: 19 full columns and rows of 2,048 px plus a 1,088 px remainder at each edge.
Each coloured quadrant is one manifest entry. Because every offset is a multiple of 512, a worker's windowed read fetches whole blocks and never pays for a partial one.

The distributed Map state’s ItemReader resolves the first limit by streaming the manifest from S3 rather than carrying it in the payload, so the manifest can describe ten tiles or ten thousand without changing the transition size. Block alignment resolves the second. This page covers only the partitioning stage; the merge that reassembles the results is handled in merging tiled Lambda outputs into a COG.

The tile size is the single lever that balances three competing pressures. Larger tiles mean fewer invocations and less orchestration overhead, but each worker holds more decoded pixels in memory and runs closer to the 15-minute timeout ceiling. Smaller tiles keep every worker comfortably within its memory tier but multiply the invocation count, the cold-start surface, and the manifest length. For a four-band Float32 scene, a 2048-pixel tile decodes to roughly 256 MB before GDAL’s working buffers — a safe fit at a 2 GB worker tier — while a 4096-pixel tile quadruples that to ~1 GB and starts to crowd the allocation. The metadata Lambda encodes this decision once, in the TILE_PIXELS environment variable, so the same partitioning code serves any provider simply by lowering the value to fit a tighter memory ceiling such as Azure’s 1.5 GB.

Prerequisites

  • Runtime: Python 3.11 on AWS Lambda with a rasterio layer (GDAL 3.9+). See Packaging & Dependency Management for Serverless GIS for the layer build.
  • IAM: the metadata Lambda role needs s3:GetObject on the source prefix and s3:PutObject on the manifest prefix — nothing more, per IAM security boundaries for Cloud GIS.
  • Source must be a real COG: internally tiled with an offset index. Validate with rio cogeo validate; a striped GeoTIFF makes block-aligned windowing meaningless because every window read pulls whole scanlines.
  • Environment variables (set on the function, applied before the first rasterio.open):
    code 
    GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif,.tiff
GDAL_DATA=/opt/share/gdal
PROJ_LIB=/opt/share/proj
  • Manifest bucket distinct from, or prefixed separately within, the ingest bucket so lifecycle rules can expire manifests independently of source data.

Implementation

The function below reads the header, snaps the requested tile size down to a whole multiple of the block size, iterates windows in row-major order, and streams the manifest to S3 as newline-delimited JSON. It returns only pointers and a summary — never the window list.

The five steps the metadata Lambda performs to build a tile manifestFive sequential steps inside the metadata Lambda: open the COG header for width, height and block shape; snap the requested tile edge down to a whole multiple of the block size; iterate windows in row-major order clamping the trailing edges; stream one JSON object per line to S3; and return only pointers and a summary so the state payload stays tiny.Header to manifest, without reading a pixel1Open the COG headerwidth, height, block_shapes, CRS, dtype - no pixel data is fetchedrasterio.open2Snap the tile edge to a block multiple2,048 px requested against 512 px blocks stays 2,048 px; a 1,500 px request becomes 1,024 px_align_down()3Iterate windows row-majormin(tile, remaining) clamps the right and bottom edges to 1,088 px400 windows4Stream one JSON object per linenewline-delimited so ItemReader can read it lazily, line by linemanifest.jsonl5Return pointers, never the listmanifest_bucket, manifest_key, tile_count, profile< 1 KB
Only step one touches the source file, and it reads the header alone. Everything after that is arithmetic on four integers.
python 
#!/usr/bin/env python3
"""metadata_partition.py — Lambda: partition a COG into a Map-state manifest.

Reads only the COG header, emits block-aligned tile windows as JSONL to S3,
and returns the manifest URI so the state payload stays under 256 KB.
"""
import io
import json
import os

import boto3
import rasterio

# COG-friendly config so opening the header issues minimal range requests.
os.environ.setdefault("GDAL_DISABLE_READDIR_ON_OPEN", "EMPTY_DIR")
os.environ.setdefault("CPL_VSIL_CURL_ALLOWED_EXTENSIONS", ".tif,.tiff")

s3 = boto3.client("s3")

TARGET_TILE = int(os.environ.get("TILE_PIXELS", "2048"))  # requested tile edge
OUTPUT_BUCKET = os.environ["OUTPUT_BUCKET"]


def _align_down(value: int, block: int) -> int:
    """Snap a tile edge down to a whole multiple of the block size (min one block)."""
    return max(block, (value // block) * block)


def handler(event, context):
    bucket, key = event["bucket"], event["key"]
    run_id = key.rsplit("/", 1)[-1].rsplit(".", 1)[0]
    src_uri = f"/vsis3/{bucket}/{key}"

    with rasterio.open(src_uri) as src:
        width, height = src.width, src.height
        # Internal block size — the atomic COG read unit (often 512x512).
        bx, by = src.block_shapes[0][1], src.block_shapes[0][0]
        crs, dtype, count = str(src.crs), src.dtypes[0], src.count
        transform6 = list(src.transform)[:6]

    tile_w = _align_down(TARGET_TILE, bx)
    tile_h = _align_down(TARGET_TILE, by)

    # Stream the manifest straight to S3 as newline-delimited JSON.
    buf = io.StringIO()
    tile_count = 0
    for row_off in range(0, height, tile_h):
        for col_off in range(0, width, tile_w):
            w = min(tile_w, width - col_off)
            h = min(tile_h, height - row_off)
            index = f"{row_off:06d}_{col_off:06d}"
            buf.write(json.dumps({
                "index": index,
                "src_uri": src_uri,
                "col_off": col_off, "row_off": row_off,
                "width": w, "height": h,
                "out_key": f"runs/{run_id}/tiles/{index}.tif",
            }))
            buf.write("\n")
            tile_count += 1

    manifest_key = f"runs/{run_id}/manifest.jsonl"
    s3.put_object(Bucket=OUTPUT_BUCKET, Key=manifest_key,
                  Body=buf.getvalue().encode(), ContentType="application/x-ndjson")

    # Return pointers + summary ONLY — the state payload must not carry the tiles.
    return {
        "run_id": run_id,
        "manifest_bucket": OUTPUT_BUCKET,
        "manifest_key": manifest_key,
        "tile_count": tile_count,
        "tile_w": tile_w, "tile_h": tile_h,
        "block": [by, bx],
        "profile": {"crs": crs, "dtype": dtype, "count": count, "transform": transform6},
        "out_prefix": f"runs/{run_id}/tiles/",
        "output_bucket": OUTPUT_BUCKET,
    }

The corresponding ItemReader in the state machine consumes this manifest without ever inlining it. Note that only manifest_bucket and manifest_key cross the state boundary:

json 
"ItemReader": {
  "Resource": "arn:aws:states:::s3:getObject",
  "ReaderConfig": {"InputType": "JSONL"},
  "Parameters": {
    "Bucket.$": "$.plan.meta.manifest_bucket",
    "Key.$": "$.plan.meta.manifest_key"
  }
}

Verification

Confirm the manifest is well-formed, block-aligned, and small enough that its pointer — not its content — is what flows through the state machine.

bash 
# Pull the manifest and inspect the first record + line count
aws s3 cp "s3://$OUTPUT_BUCKET/runs/$RUN_ID/manifest.jsonl" - | head -n 1 | python -m json.tool
LINES=$(aws s3 cp "s3://$OUTPUT_BUCKET/runs/$RUN_ID/manifest.jsonl" - | wc -l)
echo "tiles: $LINES"

# Assert every tile edge is a multiple of the block size (512)
aws s3 cp "s3://$OUTPUT_BUCKET/runs/$RUN_ID/manifest.jsonl" - \
  | python -c "import sys,json; \
bad=[l for l in sys.stdin if (json.loads(l)['col_off']%512) or (json.loads(l)['row_off']%512)]; \
print('misaligned:', len(bad))"

Expected output for a 40000×40000 COG with 512-px blocks and 2048-px tiles:

code 
{
    "index": "000000_000000",
    "src_uri": "/vsis3/geo-ingest/incoming/scene.tif",
    "col_off": 0, "row_off": 0, "width": 2048, "height": 2048,
    "out_key": "runs/scene/tiles/000000_000000.tif"
}
tiles: 400
misaligned: 0

The metadata Lambda’s own returned payload should be well under 1 KB — verify in the Step Functions execution event history that the ExtractMetadata result contains a manifest_key, not a tiles array.

Inlined tile manifests against the 256 KB Step Functions payload limitFour meters against the 256 KB Step Functions state-transition limit: a 400-tile manifest inlined uses about 80 KB, a 1,200-tile manifest about 240 KB and sits at the cap, a 10,000-tile manifest at about 2,000 KB is eight times over it, and a manifest URI passed to ItemReader is about 100 bytes regardless of tile count.Manifest bytes against the 256 KB transition cap400-tile manifest, inlined~200 bytes per window record80 KB1,200-tile manifest, inlinedthe practical cap for one transition240 KB10,000-tile manifest, inlinedStates.DataLimitExceeded before any worker2,000 KBManifest URI via ItemReaderbucket + key, whatever the tile count~100 BThe bar is clipped at the limit; the printed value is the real size. A 40,000 px scene inlines safely and a 70,000 px scene does not, which iswhy there is only one code path.
The first three meters scale with the scene; the fourth does not. That is the whole argument for ItemReader — the payload becomes a constant.

Gotchas and Edge Cases

  • Edge tiles are smaller than the tile size. The last column and row use min(tile, remaining), so they are not block-aligned in extent — that is fine. Alignment matters for the offset (where the read starts), which stays on a block boundary; a short trailing tile still reads whole blocks up to the raster edge.

  • block_shapes reports the first band only. For a pixel-interleaved multi-band COG all bands share a block layout, but band-interleaved files can differ. Read src.block_shapes per band and take the common divisor if they diverge, or the alignment guarantee breaks for some bands.

  • A manifest under ~1,200 tiles could be inlined, but do not. It is tempting to skip S3 for small scenes, but keeping one code path — always ItemReader from S3 — means a scene that grows past the threshold never silently trips States.DataLimitExceeded in production.

  • Very high tile counts hit the distributed Map child ceiling, not the payload. Above ~10,000 tiles you approach the distributed Map concurrent-child limit; increase tile size or process in batched sub-manifests rather than shrinking tiles further.

  • Overviews are not tiles. The manifest partitions only the full-resolution image. If the source COG carries internal overviews, do not add manifest entries for them — the merge stage regenerates overviews from the reassembled full-resolution mosaic. Emitting overview windows here would double-process pixels and corrupt the output extent.

  • CRS lives in the summary, not per tile. Every tile inherits the source CRS and transform, so record them once in the returned profile rather than repeating them on each manifest line. Repeating the transform on 400 lines inflates the manifest and tempts a worker to trust a stale copy instead of reading window_transform from the source.

Frequently Asked Questions

Why does the tile manifest have to live in S3 rather than in the state payload?

Step Functions caps the data passed between states at 256 KB. A single JSON tile record is roughly 200 bytes, so a scene of more than about 1,200 tiles overflows the limit if inlined. Writing the manifest to S3 and reading it with the distributed Map ItemReader keeps the payload to a small URI regardless of tile count.

Should tile size be aligned to the COG block size?

Yes. Align tile width and height to whole multiples of the internal block size — typically 512 pixels — so each windowed read fetches complete blocks. Misaligned windows force GDAL to fetch and discard partial blocks, inflating range-request count and cost.

What format should the manifest use for ItemReader?

Newline-delimited JSON (JSONL) or CSV. The distributed Map ItemReader streams JSONL line by line, so the manifest can be arbitrarily large without ever being loaded whole into memory or into the state payload.

Back to Process a 10 GB GeoTIFF with AWS Step Functions and Lambda