Handling Partial Failures in a Step Functions Map Run

Set ToleratedFailurePercentage: 2 on the Distributed Map so 40 bad windows out of 4,000 do not discard 3,960 finished tiles, add a ResultWriter pointing at an S3 prefix so the run writes manifest.json plus FAILED_*.json arrays, and rebuild a retry manifest from those failed items rather than re-running the whole scene. Then put a Choice state between the Map and the merge that compares tiles written against the planned tile count, because a Map run can end SUCCEEDED with 40 missing tiles and the merge step will happily assemble the other 3,960 into a mosaic with 40 nodata holes and register a STAC item for it.

Context

The 10 GB GeoTIFF recipe fans a scene out across thousands of child executions, each processing one window from the manifest built in partitioning a GeoTIFF into Step Functions Map-state tiles. At that width, failure stops being an exception and becomes a statistic. A 4,000-tile run touches object storage 4,000 times and starts 4,000 Lambda invocations; a single throttled minute, one slow availability zone, or one corrupt block in the source file will take some of them down. The question is not whether tiles fail but what the state machine does about it.

The default behaviour is the strictest possible: a Distributed Map tolerates zero failures, so the first child execution that exhausts its retries fails the entire Map state and the run ends with 3,999 wasted invocations. Raising the tolerance fixes that waste and introduces a subtler hazard — the run now reports success while the output is incomplete.

How a Distributed Map run with 40 failed tiles reaches the publish gateFour states in a row. The Map run dispatches 4,000 children 500 at a time and each child self-retries four times with a backoff rate of two. Forty children still fail, so the run evaluates them against the tolerance budget, ends SUCCEEDED with 3,960 tiles written, and arrives at the publish gate which blocks the merge and routes the failed windows back into a second execution.The dangerous outcome is the one labelled SUCCEEDEDMap RUNNING4,000 children,MaxConcurrency 500Tolerance check40 failed = 1.0% of 4,000Map SUCCEEDED3,960 tiles on the prefixPublish gatewritten 3,960 vs planned4,000Retry x4, backoff 2.0child exhausts retriesunder 2%before mergesecond execution over retry.jsonlRaising the tolerance above zero is what turns the second box green instead of red — and is also what makes the original execution ineligiblefor redrive.
Nothing in the state machine is broken at the third box — the run really did succeed. Only the gate knows the mosaic is 40 tiles short.

Three mechanisms have to work together. ToleratedFailurePercentage (or ToleratedFailureCount) decides whether the run keeps going. ResultWriter decides whether you can find out afterwards which windows died. A publish gate decides whether a mosaic with known holes is ever allowed to become a catalog entry. Get the first without the other two and the pipeline becomes a machine for producing plausible-looking wrong data.

Per-tile retry sits below all of this. The Retry and Catch clauses on the inner ProcessTile task — four attempts with a backoff rate of 2.0, then a dead-letter route — absorb transient errors before they ever count as a Map failure, exactly as dead-letter queues for failed vector jobs do for queue workers. The 40 failures discussed here are the ones that survived all four attempts.

Prerequisites

  • A Standard workflow. Redrive and long-running Distributed Map runs both require Standard, not Express. The Map run itself can dispatch up to 10,000 parallel child executions; the parent recipe caps MaxConcurrency at 500 to stay well under the 1,000 default regional Lambda concurrency, as covered in concurrency and throttling for tile fan-out.
  • Deterministic output keys. Every window must write to runs/<run_id>/tiles/<row>_<col>.tif, so re-running a window overwrites rather than duplicates — the property established in deterministic job IDs from object URI and ETag.
  • IAM on the state machine role: s3:PutObject on the results prefix for ResultWriter, s3:GetObject and s3:ListBucket on the manifest and results prefixes, and states:RedriveExecution for whoever operates the pipeline.
  • A staging prefix separate from the published prefix. Tiles and the merged mosaic land under staging/; nothing is copied to published/ or catalogued until the gate passes.
  • Environment values used below:
    code 
    RESULTS_BUCKET=geo-pipeline-results
RESULTS_PREFIX=maprun/
TILE_COUNT_SOURCE=$.plan.meta.tile_count
TOLERATED_FAILURE_PERCENTAGE=2

Choosing the tolerance

Tolerance is a judgement about which failures are worth continuing through. Transient failures are sparse and uncorrelated: a handful of Lambda.TooManyRequestsException responses during a fan-out ramp, an S3 503 SlowDown on a hot prefix, one worker that hit the 15-minute ceiling on an unusually dense window. Systemic failures are dense and correlated: a wrong CRS in the manifest, an expired credential, a source object that was replaced mid-run. The tolerance you want is a line drawn between those two populations.

Failed windows in three tiling runs measured against a 2 percent toleranceThree usage meters against a budget of 80 failed tiles, which is 2 percent of a 4,000-tile run. A throttling burst produced 12 failures, a corrupt source block produced 40 failures, and an expired credential produced 612 failures which is more than seven times the budget and aborts the run.Three real runs against an 80-tile failure budgetThrottling burst on fan-out rampLambda.TooManyRequestsException, scattered12 / 80 tilesCorrupt block in the source COGRasterioIOError on 40 adjacent windows40 / 80 tilesCredential expired mid-runAccessDenied on every window after 00:41612 / 80 tilesThe bar is clipped at the budget; the printed value is the real count. 2 percent of a 30-window NDVI grid rounds to zero, which is whyToleratedFailureCount is set alongside the percentage.
The first two runs finish and leave a retry manifest behind. The third stops after roughly 700 children instead of burning all 4,000 — which is the entire point of setting the budget below the systemic-failure line.

Two percent of 4,000 tiles is 80 windows. Any transient burst realistically stays well under that, so the run survives it; anything systemic blows past 80 within the first few hundred children and stops the run before it has burned an hour of compute. ToleratedFailureCount is the better control for small manifests — 2 percent of a 30-tile NDVI grid is zero tiles, which silently restores the strict default. Set both and Step Functions applies whichever is breached first.

Implementation

The Map state below carries the tolerance, the ResultWriter, and a Choice gate that refuses to merge an incomplete tile set. Only the states that differ from the parent recipe’s definition are shown.

json 
{
  "ProcessTiles": {
    "Type": "Map",
    "ItemProcessor": {
      "ProcessorConfig": {"Mode": "DISTRIBUTED", "ExecutionType": "STANDARD"},
      "StartAt": "ProcessTile",
      "States": {
        "ProcessTile": {
          "Type": "Task",
          "Resource": "arn:aws:states:::lambda:invoke",
          "Parameters": {"FunctionName": "${TileWorkerFunctionArn}", "Payload.$": "$"},
          "Retry": [
            {"ErrorEquals": ["Lambda.TooManyRequestsException", "Lambda.ServiceException",
                             "S3.SlowDown"],
             "IntervalSeconds": 2, "MaxAttempts": 4, "BackoffRate": 2.0}
          ],
          "End": true
        }
      }
    },
    "ItemReader": {
      "Resource": "arn:aws:states:::s3:getObject",
      "ReaderConfig": {"InputType": "JSONL"},
      "Parameters": {"Bucket.$": "$.plan.meta.manifest_bucket",
                     "Key.$": "$.plan.meta.manifest_key"}
    },
    "ResultWriter": {
      "Resource": "arn:aws:states:::s3:putObject",
      "Parameters": {"Bucket": "${ResultsBucket}", "Prefix": "maprun"}
    },
    "MaxConcurrency": 500,
    "ToleratedFailurePercentage": 2,
    "ToleratedFailureCount": 80,
    "ResultPath": "$.mapResult",
    "Next": "CountTiles"
  },

  "CountTiles": {
    "Comment": "Lists the tile prefix and returns written vs planned.",
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Parameters": {"FunctionName": "${TileCountFunctionArn}",
                   "Payload": {"run.$": "$.plan.meta", "mapResult.$": "$.mapResult"}},
    "ResultSelector": {"tally.$": "$.Payload"},
    "ResultPath": "$.gate",
    "Next": "PublishGate"
  },

  "PublishGate": {
    "Type": "Choice",
    "Choices": [
      {"Variable": "$.gate.tally.complete", "BooleanEquals": true, "Next": "MergeTiles"}
    ],
    "Default": "RetryFailedWindows"
  },

  "RetryFailedWindows": {
    "Comment": "Builds retry.jsonl from the FAILED_*.json result files.",
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Parameters": {"FunctionName": "${RetryManifestFunctionArn}",
                   "Payload.$": "$.mapResult"},
    "ResultSelector": {"retry.$": "$.Payload"},
    "ResultPath": "$.retryPlan",
    "End": true
  }
}

Note what the gate does not do: it does not route to MergeTiles with a warning, and it does not attempt an inline second Map. It ends the execution having written a retry manifest, so an operator or a scheduled sweeper starts a second, ordinary execution against retry.jsonl. Keeping the retry as a separate execution means the retry run gets its own Map run ID, its own result files, and its own gate.

The RetryManifestFunction reconstructs window records straight from the failure arrays. A failed child execution’s stored Input is the original manifest line, so the retry manifest is a filtered copy of the original — no re-partitioning, no header read.

python 
# retry_manifest.py — rebuild a JSONL manifest from a Map run's FAILED results.
import json
import os

import boto3

s3 = boto3.client("s3")
RESULTS_BUCKET = os.environ["RESULTS_BUCKET"]


def _load_json(bucket: str, key: str):
    return json.loads(s3.get_object(Bucket=bucket, Key=key)["Body"].read())


def handler(map_result, context):
    # ResultWriter reports where it put the run's index file.
    manifest_key = map_result["ResultWriterDetails"]["Key"]
    index = _load_json(RESULTS_BUCKET, manifest_key)

    failed_windows, causes = [], {}
    # index["ResultFiles"] groups the per-batch arrays by outcome.
    for entry in index["ResultFiles"].get("FAILED", []):
        for child in _load_json(RESULTS_BUCKET, entry["Key"]):
            # Input is the exact manifest line this child was started with,
            # so the retry manifest needs no re-partitioning of the source.
            window = json.loads(child["Input"])
            failed_windows.append(window)
            # Group by error name to spot a systemic cause before retrying.
            cause = json.loads(child.get("Output") or "{}").get("Error", "Unknown")
            causes[cause] = causes.get(cause, 0) + 1

    if not failed_windows:
        return {"retry_key": None, "failed": 0, "causes": {}}

    run_id = failed_windows[0]["out_key"].split("/")[1]
    retry_key = f"runs/{run_id}/retry.jsonl"
    body = "\n".join(json.dumps(w) for w in failed_windows) + "\n"
    s3.put_object(Bucket=RESULTS_BUCKET, Key=retry_key,
                  Body=body.encode(), ContentType="application/x-ndjson")

    # A single dominant cause means retrying will fail the same way; surface it.
    dominant = max(causes, key=causes.get)
    return {
        "retry_key": retry_key,
        "retry_bucket": RESULTS_BUCKET,
        "failed": len(failed_windows),
        "causes": causes,
        "systemic": causes[dominant] / len(failed_windows) > 0.9,
    }

The systemic flag matters more than it looks. If 39 of 40 failures carry the same error name, re-running them produces 40 more failures and the pipeline enters a retry loop that costs money and fixes nothing. Route on that flag: retry when the causes are mixed, alarm when they are not.

The publish gate

The publish gate decision after a Distributed Map run finishesA decision with three outcomes. Listing the tile prefix gives the number of tiles actually written against the 4,000 planned. An exact match promotes the mosaic to the published prefix and writes the STAC item. A shortfall of forty tiles with mixed error causes writes a retry manifest and starts a second execution. A shortfall with one dominant error cause quarantines the staging prefix and raises an alarm instead of retrying.One question, asked after every Map runTiles written on the staging prefix vs the4,000 the partitioner planned?4,000 of 4,000Merge and publishbuild the mosaic, copy to published/, writethe STAC item3,960 — mixed causesWrite retry.jsonl40 window records lifted fromFAILED_0.jsonsecond execution, its own gate3,388 — one dominant causeQuarantine and alarmsystemic flag set: retrying reproducesthe same 612 failures
The gate reads the tile prefix, not the Map run's own tally, because a timed-out child can leave a valid tile behind and a succeeded child can write to the wrong key.

The counter behind the gate is deliberately dumb — it counts objects under the tile prefix and compares that number with the tile_count the partitioner recorded. It does not trust the Map run’s own tally, because the two can legitimately disagree: a child execution that timed out after writing its object counts as failed while its tile exists, and a child that succeeded but wrote to the wrong prefix counts as succeeded while its tile does not.

python 
# tile_count.py — the gate's only job: does the tile prefix hold every window?
import boto3

s3 = boto3.client("s3")


def handler(event, context):
    meta = event["run"]
    paginator = s3.get_paginator("list_objects_v2")
    written = 0
    for page in paginator.paginate(Bucket=meta["output_bucket"],
                                   Prefix=meta["out_prefix"]):
        written += sum(1 for o in page.get("Contents", []) if o["Key"].endswith(".tif"))

    planned = int(meta["tile_count"])
    return {"planned": planned, "written": written,
            "missing": planned - written, "complete": written == planned}

Only when complete is true does the run reach merging tiled Lambda outputs into a COG and, after it, the catalog write.

Verification

Force a partial failure by injecting a window whose src_uri points at a deleted key, then confirm the tolerance, the failure manifest, and the gate all behave.

bash 
# 1) What the Map run itself recorded
aws stepfunctions describe-map-run --map-run-arn "$MAP_RUN_ARN" \
  --query '{status:status, tolPct:toleratedFailurePercentage, items:itemCounts}'

# 2) The failure arrays ResultWriter left behind
aws s3 cp "s3://$RESULTS_BUCKET/maprun/$EXEC_NAME/$MAP_RUN_ID/manifest.json" - \
  | python -m json.tool | grep -A3 FAILED

# 3) Did the gate hold?
aws stepfunctions describe-execution --execution-arn "$EXEC_ARN" \
  --query 'status' --output text
aws s3 ls "s3://$PUBLISHED_BUCKET/published/$RUN_ID/" | wc -l

Expected output for a run where 40 of 4,000 windows failed:

code 
{
    "status": "SUCCEEDED",
    "tolPct": 2,
    "items": {
        "pending": 0, "running": 0, "succeeded": 3960,
        "failed": 40, "timedOut": 0, "aborted": 0,
        "resultsWritten": 4000, "total": 4000
    }
}
        "FAILED": [
            {"Key": "maprun/tile-run-8813/.../FAILED_0.json", "Size": 71204}
SUCCEEDED
0

The three lines that matter: the Map run status is SUCCEEDED with failed: 40, which is precisely the dangerous state; a FAILED_0.json exists to rebuild from; and the published prefix is still empty because the gate routed to RetryFailedWindows instead of MergeTiles. If that last count is not zero, the gate is not wired between the Map and the merge.

Gotchas and Edge Cases

  • A tolerated failure forfeits redrive. Redrive re-runs only the failed and unstarted children of a Map run, which is exactly what you want — but it applies only to an execution that ended FAILED, TIMED_OUT or ABORTED, and only within 14 days of it stopping. Raising ToleratedFailurePercentage above zero means successful-with-holes runs end SUCCEEDED and can never be redriven. That is the whole reason the retry manifest exists; do not assume both tools are available on the same run.
  • ToleratedFailurePercentage rounds against you on small manifests. Two percent of a 30-window job is 0.6 windows, and Step Functions will not tolerate a fraction — the effective budget is zero. Always pair the percentage with an explicit ToleratedFailureCount so small runs get the same protection as large ones.
  • ResultWriter output is not free and not small. Every child execution’s input and output is copied into the result arrays. For 4,000 windows carrying ~200-byte inputs plus worker return payloads, expect a few megabytes per run. Give the results prefix a lifecycle rule — 30 days is usually plenty, and it must outlive the 14-day redrive window.
  • The Map run can fail without a single tile failing. States.DataLimitExceeded on a child’s output, an ItemReader that cannot parse a manifest line, or exhausting the account’s open execution quota all fail the Map run with failed: 0. Read itemCounts before assuming the failure arrays hold anything at all.
  • Never let the merge “fill” a missing tile. It is tempting to have the merge write nodata for absent windows so the run always completes. That converts a loud failure into a silent one and is the exact anti-pattern the gate exists to prevent — a mosaic with fabricated nodata is indistinguishable from a legitimate one once it is in the catalog.

Frequently Asked Questions

What is a sensible ToleratedFailurePercentage for a tiling job?

Set it just high enough that a realistic burst of transient errors does not throw away thousands of completed tiles, and low enough that a systemic failure still stops the run early. For a 4,000-tile mosaic, 2 percent — 80 tiles — absorbs a throttling burst or a slow availability zone, while a credential or CRS error fails every window and trips the budget within the first few hundred children. Pair it with ToleratedFailureCount so the same setting behaves sensibly on a 30-tile job.

Can I redrive a Map run that succeeded with tolerated failures?

No. Redrive applies only to a Standard execution that ended FAILED, TIMED_OUT or ABORTED, and only for 14 days after it stopped. An execution that finished SUCCEEDED because its failures were inside the tolerance cannot be redriven at all, so raising the tolerance is a deliberate trade: you keep 3,960 finished tiles and give up automatic redrive in exchange for an explicit retry manifest.

Where does a Distributed Map write the list of failed items?

Only where you tell it to, via ResultWriter. Given a bucket and prefix, Step Functions writes manifest.json under <prefix>/<execution-name>/<map-run-id>/ alongside SUCCEEDED_n.json, FAILED_n.json and PENDING_n.json arrays, each holding the child execution ARN, status, input and output. Without ResultWriter the only record is the execution history, which is far more expensive to page through for 4,000 children.

Why must a partially written mosaic never be published?

Because a merge step cannot distinguish a missing tile from a tile that legitimately contains nodata. Forty absent windows become forty nodata rectangles inside a structurally valid Cloud Optimized GeoTIFF, with a valid STAC item and no error anywhere in the pipeline. That is worse than a failed run: the system has published a confidently wrong product. Gate the merge on tiles written equalling tiles planned, and keep everything else in a staging prefix.

Back to Process a 10 GB GeoTIFF with AWS Step Functions and Lambda