Chunking Raster Jobs to Fit the 15-Minute AWS Lambda Ceiling

Split a large raster operation into fixed-size tile windows whose worst-case runtime stays under ~630 seconds — 70% of Lambda’s 900-second wall — and record each finished tile in a durable checkpoint so a killed invocation resumes without redoing or skipping work. The two rules that make this safe are a deterministic tile id derived from the source URI and window offsets, and a remaining-time check that exits cleanly before the platform force-terminates the function.

Tile windows, checkpoint store, and resume path A raster grid of nine tiles. Six are marked done and recorded in a checkpoint store; three are outstanding. A resumed Lambda invocation reads the checkpoint set and processes only the outstanding tiles, each within the 630-second budget, exiting before the 900-second ceiling. Raster split into tile windows done done done done done done todo todo todo Checkpoint store DynamoDB / S3 markers completed tile ids Resumed invocation skip done, process todo each tile < 630 s budget exit before 900 s wall Deterministic tile ids make every retry idempotent — no tile redone, none skipped
Completed tiles are durable; a resumed run reads the checkpoint set and processes only the remainder.

Context

This page implements the AWS-specific decomposition that the timeout ceiling comparison for long-running geospatial jobs recommends whenever a worst-case input exceeds ~70% of the 900-second Lambda wall. Global reprojection, DEM mosaicking, and per-pixel ML inference all blow past 15 minutes on a full scene, so the operation has to be expressed as many small units of work rather than one long one. Tiling is the natural unit for raster: a window of the source array is self-contained, so it can be processed, checkpointed, and retried independently.

The subtlety that makes chunking production-grade rather than a naive loop is durability. Lambda gives no warning before the 900-second kill; if you are 80% through a scene when the wall hits, you must not lose that 80% or reprocess it on the next run. A deterministic tile id plus a checkpoint store turns the job into a resumable, idempotent set-difference: outstanding tiles equal all tiles minus completed tiles. This is the same idempotency discipline used across the broader architecture and platform-limits guidance, applied to the timeout dimension specifically. When you fan these tiles out concurrently rather than looping, orchestrate them the way the 10 GB GeoTIFF Step Functions recipe does.

Two loop topologies are possible, and the handler below deliberately uses the simpler one. In the sequential-resume model shown here, a single invocation walks the tile grid, checkpointing as it goes, and returns the outstanding list when it runs low on time; a driver re-invokes it until complete is true. In the fan-out model, a planner emits every outstanding tile id and a Step Functions Map state processes them in parallel, one invocation per tile. The sequential model is cheaper to build and needs no orchestrator, and it is the right starting point for jobs that finish in a handful of resume cycles. The fan-out model wins when wall-clock latency matters — 200 tiles at 40-way concurrency finish in the time five serial tiles would take. Both share the exact same checkpoint contract, so you can start sequential and graduate to fan-out later without changing the tile-id or completion logic. The timeout ceiling comparison covers when that graduation is worth the added orchestration.

Prerequisites

  • Runtime: Python 3.11 Lambda, memory_size=10240 (10 GB — max CPU for per-tile speed, per the memory and CPU allocation model), timeout=900
  • IAM: dynamodb:GetItem and dynamodb:PutItem on the checkpoint table; s3:GetObject on the source bucket; s3:PutObject on the tile-output prefix
  • Layer: GDAL/rasterio layer with data dirs under /opt/share, built per the native library compilation discipline
  • Environment variables:
    code 
    GDAL_DATA=/opt/share/gdal
PROJ_LIB=/opt/share/proj
LD_LIBRARY_PATH=/opt/lib
CHECKPOINT_TABLE=raster-tile-progress
GDAL_PAM_ENABLED=NO      # avoid .aux.xml writes to read-only /var/task
  • Dependency versions: rasterio==1.3.10, boto3>=1.34
  • Checkpoint table: DynamoDB table raster-tile-progress with partition key job_id and sort key tile_id

Implementation

The handler reads the checkpoint set, computes the outstanding tiles, and processes them until the remaining-time budget runs low — then returns the still-outstanding list so the orchestrator (or a self-reinvoke) can continue. Each tile is sized to stay well under the 900-second ceiling.

python 
# handler.py — resumable tiled raster processing under the 900 s Lambda wall
import os, hashlib
import boto3
import rasterio
from rasterio.windows import Window

os.environ.setdefault("GDAL_DATA", "/opt/share/gdal")
os.environ.setdefault("PROJ_LIB", "/opt/share/proj")

ddb = boto3.resource("dynamodb").Table(os.environ["CHECKPOINT_TABLE"])
TILE = 2048                     # window edge; profile to keep worst-case < 630 s
SAFETY_MS = 90_000              # stop with 90 s to spare before the 900 s kill


def tile_id(src_uri: str, col: int, row: int) -> str:
    """Deterministic id so retries are idempotent — same tile, same id, always."""
    return hashlib.sha1(f"{src_uri}:{col}:{row}".encode()).hexdigest()[:16]


def completed_tiles(job_id: str) -> set[str]:
    """Read the durable checkpoint set for this job."""
    resp = ddb.query(
        KeyConditionExpression=boto3.dynamodb.conditions.Key("job_id").eq(job_id),
        ProjectionExpression="tile_id",
    )
    return {item["tile_id"] for item in resp.get("Items", [])}


def process_tile(src, col: int, row: int) -> float:
    """Per-pixel work on one window. Replace mean() with the real transform."""
    window = Window(col, row, TILE, TILE)
    data = src.read(1, window=window, boundless=True, fill_value=0)
    return float(data.mean())


def handler(event, context):
    job_id = event["job_id"]
    src_uri = event["cog_url"]
    done = completed_tiles(job_id)

    outstanding: list[dict] = []
    processed = 0

    with rasterio.open(f"/vsicurl/{src_uri}") as src:
        cols = range(0, src.width, TILE)
        rows = range(0, src.height, TILE)
        for row in rows:
            for col in cols:
                tid = tile_id(src_uri, col, row)
                if tid in done:
                    continue                      # already durable — skip
                # Stop cleanly before the platform force-kills the invocation
                if context.get_remaining_time_in_millis() < SAFETY_MS:
                    outstanding.append({"col": col, "row": row})
                    continue

                process_tile(src, col, row)        # do the work
                ddb.put_item(Item={"job_id": job_id, "tile_id": tid,
                                   "col": col, "row": row})  # checkpoint it
                processed += 1

    return {
        "job_id": job_id,
        "processed": processed,
        "remaining": len(outstanding),
        "complete": len(outstanding) == 0,
    }

Verification

Invoke the function twice and confirm the second run skips everything already checkpointed. A dry first run that is interrupted (simulate by lowering SAFETY_MS) should leave complete: false; the follow-up run finishes the remainder and reports complete: true.

bash 
# First pass — processes a batch, may exit with remaining > 0
aws lambda invoke --function-name reproject-tile \
  --payload '{"job_id":"dem-2025-03","cog_url":"data.example.com/dem.tif"}' \
  out1.json && cat out1.json

# Second pass — same payload; completed tiles are skipped via the checkpoint
aws lambda invoke --function-name reproject-tile \
  --payload '{"job_id":"dem-2025-03","cog_url":"data.example.com/dem.tif"}' \
  out2.json && cat out2.json

Expected output — the first run does real work and the resumed run processes only the leftovers, never re-touching a completed tile:

json 
{"job_id": "dem-2025-03", "processed": 96, "remaining": 40, "complete": false}
{"job_id": "dem-2025-03", "processed": 40, "remaining": 0,  "complete": true}

Gotchas and Edge Cases

  • get_remaining_time_in_millis() measures wall clock, not CPU. A tile that stalls on a slow /vsicurl range read can blow the budget even though CPU was idle. Keep SAFETY_MS generous (90 s here) so a single slow tile cannot push the invocation into the hard kill.

  • Checkpoint the tile after the output is durable, never before. If you write the DynamoDB marker first and the output write then fails, the tile is falsely recorded as done and its data is lost. Order it: produce output, confirm the write, then checkpoint.

  • Edge tiles are partial — use boundless=True. The final column and row rarely divide evenly into TILE. Reading with boundless=True, fill_value=0 avoids a window that runs off the raster edge and raises, which would otherwise strand the last tiles permanently.

  • A single tile that itself exceeds 630 s cannot be saved by chunking alone. If one window still runs too long at 10 GB, the per-pixel operation is too heavy for Lambda — shrink TILE further, or offload that job to a container runtime as the timeout comparison describes.

  • Overlapping tiles need halo handling, or seams appear. Operations with a spatial kernel — focal statistics, resampling, edge-aware filters — read a few pixels beyond the tile boundary. If you read exactly TILE×TILE with no overlap, the output shows discontinuities at every seam. Read each window with a halo margin (Window(col-h, row-h, TILE+2h, TILE+2h) with boundless=True), process, then write back only the inner TILE×TILE core. The halo widens each tile’s runtime slightly, so factor it into the 630 s budget rather than discovering it after the seams show up in production.

  • DynamoDB query pagination can under-report completed tiles. A query returns at most 1 MB per page; a job with thousands of checkpointed tiles needs LastEvaluatedKey paging or the resumed run will re-process tiles beyond the first page. Page the checkpoint read to completion before computing the outstanding set — a silent partial read is worse than a crash because it wastes compute without any error signal.

Frequently Asked Questions

What tile size keeps a raster Lambda under the 15-minute ceiling?

Size the tile so its worst-case processing time is under about 630 seconds — 70% of the 900-second limit — leaving headroom for cold start, network variance, and retries. For most per-pixel operations a 2048×2048 or 4096×4096 window at 10 GB memory lands well inside that budget, but the only reliable method is to profile your specific operation on the densest tile.

How does the checkpoint/resume pattern survive a timeout mid-job?

Each completed tile is recorded in a durable store (DynamoDB or an S3 marker object) keyed by a deterministic tile id. If an invocation is killed at the ceiling, the next run reads the set of completed tile ids and processes only the remainder, so no tile is redone and none is skipped.

Why use context.get_remaining_time_in_millis() inside the loop?

It lets the function stop cleanly before the platform force-kills it. By checking remaining time before starting each tile and exiting with the outstanding list when the budget is low, you avoid a hard timeout that would leave a tile in an unknown, half-written state.

Back to Timeout Ceiling Comparison for Long-Running Geospatial Jobs