Chunking Raster Jobs to Fit the 15-Minute AWS Lambda Ceiling

Split a large raster operation into fixed-size tile windows whose worst-case runtime stays under ~630 seconds — 70% of Lambda’s 900-second wall — and record each finished tile in a durable checkpoint so a killed invocation resumes without redoing or skipping work. The two rules that make this safe are a deterministic tile id derived from the source URI and window offsets, and a remaining-time check that exits cleanly before the platform force-terminates the function.

Context

This page implements the AWS-specific decomposition that the timeout ceiling comparison for long-running geospatial jobs recommends whenever a worst-case input exceeds ~70% of the 900-second Lambda wall. Global reprojection, DEM mosaicking, and per-pixel ML inference all blow past 15 minutes on a full scene, so the operation has to be expressed as many small units of work rather than one long one. Tiling is the natural unit for raster: a window of the source array is self-contained, so it can be processed, checkpointed, and retried independently.

The subtlety that makes chunking production-grade rather than a naive loop is durability. Lambda gives no warning before the 900-second kill; if you are 80% through a scene when the wall hits, you must not lose that 80% or reprocess it on the next run. A deterministic tile id plus a checkpoint store turns the job into a resumable, idempotent set-difference: outstanding tiles equal all tiles minus completed tiles. This is the same idempotency discipline used across the broader architecture and platform-limits guidance, applied to the timeout dimension specifically. When you fan these tiles out concurrently rather than looping, orchestrate them the way the 10 GB GeoTIFF Step Functions recipe does.

The lifecycle of a single tile under the checkpoint contractFour states for one tile window: outstanding when its id is not in the checkpoint set, processing once the window is read, output written when the tile is durable in object storage, and checkpointed once the marker is recorded in DynamoDB. A return path shows that an invocation killed at 900 seconds leaves the tile unmarked, so the resumed run redoes only that tile.One tile, four states, and the transition that must not be reorderedOutstandingid absent from the setProcessingwindow read, transformrunningOutput writtentile durable in S3Checkpointedjob_id and tile_id storednot in the setwrite confirmedmarker putkilled at 900 s — unmarked, so only this tile is redoneWrite the marker before the output and the arrow above becomes a lie: the tile is recorded as done while its data was never written, and no laterrun will ever revisit it.
The tile is only ever done or not done — there is no half-done state a resumed run could inherit, because the marker is written after the output and never before it.

Two loop topologies are possible, and the handler below deliberately uses the simpler one. In the sequential-resume model shown here, a single invocation walks the tile grid, checkpointing as it goes, and returns the outstanding list when it runs low on time; a driver re-invokes it until complete is true. In the fan-out model, a planner emits every outstanding tile id and a Step Functions Map state processes them in parallel, one invocation per tile. The sequential model is cheaper to build and needs no orchestrator, and it is the right starting point for jobs that finish in a handful of resume cycles. The fan-out model wins when wall-clock latency matters — 200 tiles at 40-way concurrency finish in the time five serial tiles would take. Both share the exact same checkpoint contract, so you can start sequential and graduate to fan-out later without changing the tile-id or completion logic. The timeout ceiling comparison covers when that graduation is worth the added orchestration.

Prerequisites

  • Runtime: Python 3.11 Lambda, memory_size=10240 (10 GB — max CPU for per-tile speed, per the memory and CPU allocation model), timeout=900
  • IAM: dynamodb:GetItem and dynamodb:PutItem on the checkpoint table; s3:GetObject on the source bucket; s3:PutObject on the tile-output prefix
  • Layer: GDAL/rasterio layer with data dirs under /opt/share, built per the native library compilation discipline
  • Environment variables:
    code 
    GDAL_DATA=/opt/share/gdal
PROJ_LIB=/opt/share/proj
LD_LIBRARY_PATH=/opt/lib
CHECKPOINT_TABLE=raster-tile-progress
GDAL_PAM_ENABLED=NO      # avoid .aux.xml writes to read-only /var/task
  • Dependency versions: rasterio==1.3.10, boto3>=1.34
  • Checkpoint table: DynamoDB table raster-tile-progress with partition key job_id and sort key tile_id

Implementation

The handler reads the checkpoint set, computes the outstanding tiles, and processes them until the remaining-time budget runs low — then returns the still-outstanding list so the orchestrator (or a self-reinvoke) can continue. Each tile is sized to stay well under the 900-second ceiling.

python 
# handler.py — resumable tiled raster processing under the 900 s Lambda wall
import os, hashlib
import boto3
import rasterio
from rasterio.windows import Window

os.environ.setdefault("GDAL_DATA", "/opt/share/gdal")
os.environ.setdefault("PROJ_LIB", "/opt/share/proj")

ddb = boto3.resource("dynamodb").Table(os.environ["CHECKPOINT_TABLE"])
TILE = 2048                     # window edge; profile to keep worst-case < 630 s
SAFETY_MS = 90_000              # stop with 90 s to spare before the 900 s kill


def tile_id(src_uri: str, col: int, row: int) -> str:
    """Deterministic id so retries are idempotent — same tile, same id, always."""
    return hashlib.sha1(f"{src_uri}:{col}:{row}".encode()).hexdigest()[:16]


def completed_tiles(job_id: str) -> set[str]:
    """Read the durable checkpoint set for this job."""
    resp = ddb.query(
        KeyConditionExpression=boto3.dynamodb.conditions.Key("job_id").eq(job_id),
        ProjectionExpression="tile_id",
    )
    return {item["tile_id"] for item in resp.get("Items", [])}


def process_tile(src, col: int, row: int) -> float:
    """Per-pixel work on one window. Replace mean() with the real transform."""
    window = Window(col, row, TILE, TILE)
    data = src.read(1, window=window, boundless=True, fill_value=0)
    return float(data.mean())


def handler(event, context):
    job_id = event["job_id"]
    src_uri = event["cog_url"]
    done = completed_tiles(job_id)

    outstanding: list[dict] = []
    processed = 0

    with rasterio.open(f"/vsicurl/{src_uri}") as src:
        cols = range(0, src.width, TILE)
        rows = range(0, src.height, TILE)
        for row in rows:
            for col in cols:
                tid = tile_id(src_uri, col, row)
                if tid in done:
                    continue                      # already durable — skip
                # Stop cleanly before the platform force-kills the invocation
                if context.get_remaining_time_in_millis() < SAFETY_MS:
                    outstanding.append({"col": col, "row": row})
                    continue

                process_tile(src, col, row)        # do the work
                ddb.put_item(Item={"job_id": job_id, "tile_id": tid,
                                   "col": col, "row": row})  # checkpoint it
                processed += 1

    return {
        "job_id": job_id,
        "processed": processed,
        "remaining": len(outstanding),
        "complete": len(outstanding) == 0,
    }
The checkpoint protocol between driver, handler, DynamoDB and the source COGA sequence across four participants. The driver invokes the handler with a job id and COG URL; the handler queries DynamoDB for completed tile ids and receives 96 of them; it reads a 2,048 pixel window from the COG over a range request and writes the tile output; it records the tile id in DynamoDB only after the write is confirmed; it checks that more than 90 seconds of budget remain before starting each tile; and it returns complete false with 40 remaining, which the driver re-invokes with the identical payload.What one resumable invocation actually says to whomDriverLambda handlerDynamoDB checkpointCOG on S3invoke(job_id, cog_url)query completed tile idspage to LastEvaluatedKey orthe resumed run redoes work96 tile idsread 2,048 px windowrange request, boundless=True for edge tileswrite the tile outputdurable before anything is markedput_item(job_id, tile_id)only after the write isconfirmedremaining time under 90,000 ms?stop cleanly rather than be killed at 900 scomplete: false, remaining:40driver re-invokes with the samepayload
Every message is idempotent. The same payload replayed after a kill produces the same tile ids, finds 96 of them already recorded, and picks up at the 97th.

Verification

Invoke the function twice and confirm the second run skips everything already checkpointed. A dry first run that is interrupted (simulate by lowering SAFETY_MS) should leave complete: false; the follow-up run finishes the remainder and reports complete: true.

bash 
# First pass — processes a batch, may exit with remaining > 0
aws lambda invoke --function-name reproject-tile \
  --payload '{"job_id":"dem-2025-03","cog_url":"data.example.com/dem.tif"}' \
  out1.json && cat out1.json

# Second pass — same payload; completed tiles are skipped via the checkpoint
aws lambda invoke --function-name reproject-tile \
  --payload '{"job_id":"dem-2025-03","cog_url":"data.example.com/dem.tif"}' \
  out2.json && cat out2.json

Expected output — the first run does real work and the resumed run processes only the leftovers, never re-touching a completed tile:

json 
{"job_id": "dem-2025-03", "processed": 96, "remaining": 40, "complete": false}
{"job_id": "dem-2025-03", "processed": 40, "remaining": 0,  "complete": true}
The tile grid of a 136-window job with 96 windows checkpointedA grid of 136 tile windows, 17 across and 8 down, each 2,048 by 2,048 pixels. The first 96 are filled to show they are recorded in the checkpoint table; the remaining 40 are empty, matching the first invocation's report of 96 processed and 40 remaining.The job after the first pass: 96 recorded, 40 to go136 windows of 2,048by 2,048Outstanding tiles are all tilesminus the checkpointed set.Nothing else is stored, sothe resumed run needs nomemory of how far the killedone got.checkpointed — 96 tilesoutstanding — 40 tilesThe tile id is a hash of the source URI and the window offsets, so the same window always produces the same id and a retry can neverdouble-write a tile.
This is the state the first invocation's JSON describes. The resumed run reads the filled set, subtracts it, and starts at the 97th window — the same result whether the first run was killed at the wall or exited cleanly.

Gotchas and Edge Cases

  • get_remaining_time_in_millis() measures wall clock, not CPU. A tile that stalls on a slow /vsicurl range read can blow the budget even though CPU was idle. Keep SAFETY_MS generous (90 s here) so a single slow tile cannot push the invocation into the hard kill.

  • Checkpoint the tile after the output is durable, never before. If you write the DynamoDB marker first and the output write then fails, the tile is falsely recorded as done and its data is lost. Order it: produce output, confirm the write, then checkpoint.

  • Edge tiles are partial — use boundless=True. The final column and row rarely divide evenly into TILE. Reading with boundless=True, fill_value=0 avoids a window that runs off the raster edge and raises, which would otherwise strand the last tiles permanently.

  • A single tile that itself exceeds 630 s cannot be saved by chunking alone. If one window still runs too long at 10 GB, the per-pixel operation is too heavy for Lambda — shrink TILE further, or offload that job to a container runtime as the timeout comparison describes.

  • Overlapping tiles need halo handling, or seams appear. Operations with a spatial kernel — focal statistics, resampling, edge-aware filters — read a few pixels beyond the tile boundary. If you read exactly TILE×TILE with no overlap, the output shows discontinuities at every seam. Read each window with a halo margin (Window(col-h, row-h, TILE+2h, TILE+2h) with boundless=True), process, then write back only the inner TILE×TILE core. The halo widens each tile’s runtime slightly, so factor it into the 630 s budget rather than discovering it after the seams show up in production.

  • DynamoDB query pagination can under-report completed tiles. A query returns at most 1 MB per page; a job with thousands of checkpointed tiles needs LastEvaluatedKey paging or the resumed run will re-process tiles beyond the first page. Page the checkpoint read to completion before computing the outstanding set — a silent partial read is worse than a crash because it wastes compute without any error signal.

Frequently Asked Questions

What tile size keeps a raster Lambda under the 15-minute ceiling?

Size the tile so its worst-case processing time is under about 630 seconds — 70% of the 900-second limit — leaving headroom for cold start, network variance, and retries. For most per-pixel operations a 2048×2048 or 4096×4096 window at 10 GB memory lands well inside that budget, but the only reliable method is to profile your specific operation on the densest tile.

How does the checkpoint/resume pattern survive a timeout mid-job?

Each completed tile is recorded in a durable store (DynamoDB or an S3 marker object) keyed by a deterministic tile id. If an invocation is killed at the ceiling, the next run reads the set of completed tile ids and processes only the remainder, so no tile is redone and none is skipped.

Why use context.get_remaining_time_in_millis() inside the loop?

It lets the function stop cleanly before the platform force-kills it. By checking remaining time before starting each tile and exiting with the outstanding list when the budget is low, you avoid a hard timeout that would leave a tile in an unknown, half-written state.

Back to Timeout Ceiling Comparison for Long-Running Geospatial Jobs