Handling Throttling Errors in Step Functions Map State
Give Lambda.TooManyRequestsException its own Retry block with IntervalSeconds: 2, MaxAttempts: 6, BackoffRate: 2.0 and JitterStrategy: FULL, then set MaxConcurrency to about 95% of the worker’s reserved concurrency so the retries have somewhere to land. Add ToleratedFailurePercentage: 2 so a handful of poison tiles cannot discard a scene that is 99.9% complete. A Distributed Map will run up to 10,000 parallel child executions if you let it — ten times the default Lambda concurrency in the same region — so the cap is not a tuning preference, it is the thing that stops the orchestrator from manufacturing its own throttles.
What the Map State Is Actually Doing
A Distributed Map reads an item source, starts a child execution per item (or per batch), and tracks their outcomes. Each child invokes the tile worker. When the worker’s concurrency is exhausted, the invocation is refused before an execution environment exists — the child records a LambdaFunctionScheduleFailed event carrying Lambda.TooManyRequestsException, and unless a Retry block matches that error name, the child fails immediately.
This produces a distinctive and easily misread failure. The tile worker’s own log group is empty for the affected tiles, because no handler ever ran. The Errors metric for the function is zero. What is non-zero is Throttles, and what is visible in the Step Functions console is a Map state with, say, 4,100 succeeded and 2,192 failed children, all failing within the first thirty seconds. It looks like a code fault affecting a third of the data. It is a capacity fault affecting whichever tiles happened to arrive after the ceiling was reached — the concurrency and throttling for tile fan-out overview covers why those tiles are indistinguishable from each other.
The retry loop is the recovery path, but only for errors that are genuinely transient. Retrying a throttle is correct: the capacity will exist a few seconds later. Retrying a CPLE_OpenFailed because PROJ_LIB is unset is not: it will fail identically six times and consume six invocations’ worth of concurrency doing so, making the throttling worse. Separating the two error classes into separate Retry blocks is the single most useful change to make to a Map state that is misbehaving.
Prerequisites
- Reserved concurrency already set on the tile worker. The retry configuration below assumes a known ceiling; without one the backoff is chasing a number that other functions move. See reserved concurrency for geospatial Lambda fan-out.
- A Distributed Map, not an inline Map. Inline Map is capped at 40 concurrent iterations and keeps all state in the execution’s 256 KB payload, which a tile manifest exceeds quickly.
ProcessorConfig.Modemust beDISTRIBUTED. - An S3 location for the results manifests.
ResultWriterneeds a bucket and prefix, and the state machine role needss3:PutObjectthere pluss3:GetObjecton the item source. - State machine role permissions:
states:StartExecution,states:DescribeExecutionandstates:StopExecutionon itself — a Distributed Map starts its own child executions — pluslambda:InvokeFunctionon the worker. - Idempotent workers. Retries mean a tile may be processed more than once. Derive the output key deterministically from the input URI and tile index so a repeat write is harmless, in the same way deduplicating S3 event notifications for idempotent ingestion does for the trigger side.
Implementation
One Map state, two retry classes, a bounded concurrency, a failure tolerance, and a results manifest. This is the whole configuration; everything else is the worker’s business.
{
"Comment": "Tile a scene with a bounded Distributed Map",
"StartAt": "TileFanout",
"States": {
"TileFanout": {
"Type": "Map",
"MaxConcurrency": 380,
"ToleratedFailurePercentage": 2,
"Label": "TileFanout",
"ItemReader": {
"Resource": "arn:aws:states:::s3:getObject",
"ReaderConfig": { "InputType": "JSON" },
"Parameters": {
"Bucket.$": "$.manifest_bucket",
"Key.$": "$.manifest_key"
}
},
"ItemBatcher": {
"MaxItemsPerBatch": 8,
"BatchInput": {
"source_uri.$": "$.source_uri",
"dst_crs.$": "$.dst_crs"
}
},
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "DISTRIBUTED",
"ExecutionType": "STANDARD"
},
"StartAt": "ProcessTileBatch",
"States": {
"ProcessTileBatch": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "tile-worker",
"Payload.$": "$"
},
"Retry": [
{
"Comment": "Capacity errors. Patient, jittered, many attempts.",
"ErrorEquals": [
"Lambda.TooManyRequestsException",
"Lambda.ServiceException",
"Lambda.SdkClientException",
"States.TaskFailed"
],
"IntervalSeconds": 2,
"MaxAttempts": 6,
"BackoffRate": 2.0,
"MaxDelaySeconds": 60,
"JitterStrategy": "FULL"
},
{
"Comment": "Everything else. One retry, then give up and record it.",
"ErrorEquals": ["States.ALL"],
"IntervalSeconds": 1,
"MaxAttempts": 1,
"BackoffRate": 1.0
}
],
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"ResultPath": "$.error",
"Next": "RecordFailedTile"
}
],
"End": true
},
"RecordFailedTile": {
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage",
"Parameters": {
"QueueUrl": "https://sqs.eu-west-1.amazonaws.com/123456789012/tile-dlq",
"MessageBody.$": "$"
},
"End": true
}
}
},
"ResultWriter": {
"Resource": "arn:aws:states:::s3:putObject",
"Parameters": {
"Bucket": "geo-pipeline-artifacts",
"Prefix": "map-results"
}
},
"End": true
}
}
}
Four decisions in that definition are worth naming.
JitterStrategy: FULL rather than the default. Without jitter, every throttled tile waits exactly 2, 4, 8, 16, 32 and 64 seconds, which means every throttled tile retries at the same instant. The retry wave is as wide as the original wave and throttles identically — a synchronised retry storm. Full jitter randomises each wait uniformly between zero and the computed interval, spreading the retries across the window and letting capacity be reclaimed incrementally.
MaxDelaySeconds: 60. Six attempts at BackoffRate: 2.0 from a 2-second base would otherwise reach 64 seconds on the last attempt, and a wide fan-out compounds that into a long tail where the state machine is mostly waiting. Capping the delay keeps the worst-case retry latency bounded at roughly 2+4+8+16+32+60 seconds.
ItemBatcher with MaxItemsPerBatch: 8. This is the lever for the 10,000 child-execution ceiling. A 24,037-tile fan-out cannot run as 24,037 simultaneous children, but batching eight tiles per invocation reduces the child count to 3,005 — and reduces the invocation count by the same factor, which cuts cold starts proportionally. Batch size should be set so the batch still fits inside the worker’s timeout: eight 512-px tiles at roughly 900 ms each is 7.2 seconds against a 5-minute limit, comfortable even with a cold start.
A Catch that writes to a queue instead of failing. Combined with ToleratedFailurePercentage, this converts an unrecoverable tile into a record rather than an outage. The dead-letter queue pattern is the same one described in implementing dead-letter queues for failed vector jobs.
Failure Tolerance and the Child-Execution Ceiling
ToleratedFailurePercentage defaults to zero, which means a single failed child fails the Map state and, with it, the scene. For a 6,292-tile fan-out that is a poor trade: one corrupt band or one tile that straddles a nodata boundary discards 6,291 successful outputs. Setting the tolerance to 2 lets the Map complete and record the failures, so a repair pass can process 126 tiles rather than the whole scene.
Choosing the number is a judgement about what a partial scene is worth. For a mosaic that will be published as a single COG, any missing tile is a hole in the product and the tolerance should be near zero — you want the Map to fail loudly and the merge step never to run, which is the assumption merging tiled Lambda outputs into a COG is built on. For an independently addressable tile pyramid, where each tile is served on its own, a missing tile is a gap a repair pass can fill later and a tolerance of 1–2% is the difference between a five-minute repair and a forty-minute re-run.
Use ToleratedFailureCount instead when the fan-out width varies a lot between scenes — a percentage of a small manifest can round down to zero tolerance, which is rarely what was intended. The two are mutually exclusive.
The 10,000 parallel child-execution ceiling is a separate constraint and is often misread as a limit on total items. It is not: a Distributed Map can iterate millions of items from an S3 inventory or a CSV manifest. What it cannot do is have more than 10,000 child executions in flight simultaneously. In practice MaxConcurrency should sit far below that anyway — bounded by the Lambda quota, not by the Map’s own ceiling — so the 10,000 figure only becomes binding if you have raised the account concurrency quota into the same range.
Verification
After a run, the results manifest is the authoritative record of what happened. Parse it rather than reading the console.
import boto3
import json
from collections import Counter
s3 = boto3.client("s3")
sfn = boto3.client("stepfunctions")
BUCKET = "geo-pipeline-artifacts"
def audit_map_run(execution_arn: str) -> dict:
desc = sfn.describe_execution(executionArn=execution_arn)
output = json.loads(desc.get("output") or "{}")
prefix = output.get("ResultWriterDetails", {}).get("Key", "")
manifest = json.loads(
s3.get_object(Bucket=BUCKET, Key=prefix)["Body"].read()
)
reasons = Counter()
failed_tiles = []
for entry in manifest.get("ResultFiles", {}).get("FAILED", []):
body = s3.get_object(Bucket=BUCKET, Key=entry["Key"])["Body"].read()
for line in body.decode().splitlines()[1:]: # skip CSV header
_, _, _, cause = line.split(",", 3)
reasons[cause.strip('"').split(":")[0]] += 1
failed_tiles.append(line)
succeeded = len(manifest.get("ResultFiles", {}).get("SUCCEEDED", []))
report = {
"status": desc["status"],
"succeeded_result_files": succeeded,
"failed_children": len(failed_tiles),
"failure_reasons": dict(reasons),
}
throttle_share = reasons.get("Lambda.TooManyRequestsException", 0)
assert throttle_share == 0, (
f"{throttle_share} children exhausted their retries on throttles — "
"MaxConcurrency is above what the worker's reservation can serve."
)
return report
Expected output from a correctly bounded run:
{'status': 'SUCCEEDED',
'succeeded_result_files': 7,
'failed_children': 4,
'failure_reasons': {'CPLE_AppDefined': 4}}
Four failures with a GDAL cause and none with a throttle cause is the target state: the capacity problem is solved, and what remains is genuine data trouble in four tiles. The inverse — zero GDAL failures and a hundred throttle failures — means the retry block is working but the ceiling is still wrong, and no amount of retry tuning will fix it.
The manifest also carries a PENDING section, which is the one worth checking when a run looks complete but the outputs are short. Items land there when the Map stopped before dispatching them — because the failure tolerance was breached mid-run, or because someone stopped the execution. A non-empty PENDING list is the input to a resumed run: feed it back as the item source rather than re-reading the original manifest, and the repair pass processes only what was never attempted.
Cross-check against AWS/States metrics for the state machine: ExecutionThrottled should be zero, and ExecutionsFailed should be zero when the tolerance absorbed the residue. A non-zero ExecutionThrottled points at state-transition throttling rather than Lambda throttling, which is a different fix — reduce the number of states inside the ItemProcessor, or switch the inner workflow to EXPRESS.
Gotchas
-
States.TaskFailedin a capacity retry block is a blunt instrument. It matches any task failure, including a worker that raised a Python exception, so putting it alongsideLambda.TooManyRequestsExceptionmeans real bugs get six patient retries too. It is included above because a Lambda-invoke task wraps some transient service errors that way, but if your worker raises typed errors, list those explicitly in the second block and removeStates.TaskFailedfrom the first. -
ToleratedFailurePercentagedoes not stop the failures from being retried. Tolerance is evaluated after retries are exhausted. A 2% tolerance on a fan-out where 40% of tiles are throttling still burns every retry attempt on every one of those tiles before the Map decides it has failed. Tolerance is a completion policy, not a circuit breaker; the circuit breaker isMaxConcurrency. -
The results manifest is written even when the Map fails, but not when the execution is stopped. Manually stopping a runaway Map execution loses the record of which tiles completed, forcing a full re-run. Prefer setting the worker’s reserved concurrency to zero to stall the fan-out — the Map keeps its state, retries stall harmlessly, and restoring the reservation resumes it.
-
ItemBatcherchanges the shape of the worker’s event. With batching enabled the payload is{"BatchInput": {...}, "Items": [...]}rather than a bare item. A worker written for unbatched Map will receive an event it does not recognise and fail every child at once, which looks alarmingly like a total outage on the first deployment after enabling batching.
Frequently Asked Questions
Which error name does Step Functions use for a Lambda throttle?
Lambda.TooManyRequestsException. It appears in the execution history as a LambdaFunctionScheduleFailed event whose cause contains Rate Exceeded. It is distinct from States.TaskFailed, which wraps an error your handler actually raised, and from States.ThrottledEvent, which is Step Functions throttling its own state transitions.
What does ToleratedFailurePercentage do?
It sets the share of child executions that may fail before the Map state itself is marked failed. At the default of zero, one bad tile out of 6,292 fails the entire scene. A tolerance of 2 lets the Map complete, records the failures in the results manifest, and leaves the successful outputs in place for a targeted repair pass.
How many child executions can a Distributed Map run in parallel?
Up to 10,000. This is a parallelism ceiling, not a limit on total items — a Distributed Map can iterate millions of items. Since the default Lambda concurrency in the same region is 1,000, MaxConcurrency should normally sit an order of magnitude below the Map’s own ceiling.
Should I retry throttles in the worker instead of in the Map state?
No. A throttled invocation never reaches your code, so there is nothing there to retry from. In-worker retries are the right place for downstream throttles — a DynamoDB ProvisionedThroughputExceededException or an external tile API 429 — because those happen inside a running invocation. Retrying an invocation throttle is necessarily the orchestrator’s job.
Related
- Concurrency and Throttling for Tile Fan-Out — where the ceilings sit and how a fan-out reaches them
- Reserved Concurrency for Geospatial Lambda Fan-Out — the reservation that
MaxConcurrencyis set against - Comparing Concurrency Quotas Across AWS, GCP, and Azure — the orchestrator equivalents on Cloud Workflows and Durable Functions
- Partitioning a GeoTIFF into Step Functions Map Tiles — building the manifest the Map state reads
- Implementing Dead-Letter Queues for Failed Vector Jobs — where the tiles the tolerance absorbed should land