Idempotency and Exactly-Once Spatial Processing

Every event source across AWS, GCP and Azure is at-least-once, so a duplicate delivery is a certainty at scale, not a possibility. A pipeline that fans one 10 GB scene out to 400 tile jobs issues 400 deliveries per scene; at a duplicate rate of one in ten thousand — well inside what S3 event notifications and Pub/Sub consider normal — that is one duplicated tile roughly every twenty-five scenes, and a mosaic manifest that lists 412 objects where the grid says 400. The fix is not a better queue. It is a deterministic job key plus a conditional write, applied at three separate points in the pipeline so that re-executing the job body has no additional effect on the output.

This page sets out where duplicates come from, what they do to raster and vector outputs specifically, and how to enforce idempotency at the trigger, the worker and the write. It sits alongside the broader Event-Driven Geospatial Processing Patterns reference, and it generalises the object-storage-specific technique described in deduplicating S3 event notifications for idempotent ingestion into a pipeline-wide discipline.

Why At-Least-Once Delivery Corrupts Spatial Outputs

Duplicate delivery is a property of the transport, not a bug in it. S3 sends a notification, waits for the destination to accept it, and re-sends when the acknowledgement is lost or a partition fails over. Pub/Sub redelivers when the ack deadline elapses before the subscriber acks. SQS standard queues redeliver when the visibility timeout expires — which is exactly what happens to a raster job that runs longer than the timeout you sized for it, a hazard covered in detail under sizing SQS visibility timeout for long-running raster jobs. Lambda itself retries an asynchronous invocation twice by default. Stack four such layers and the compound probability of at least one redelivery per scene stops being negligible.

What makes this worse in geospatial work than in, say, a thumbnail service is that spatial outputs are usually aggregates, and aggregates do not tolerate double counting.

Duplicate COGs. Two workers handed the same window both run rasterio.open(..., "w") against the same output key. Neither write is atomic from the reader’s point of view on S3 — the second PutObject replaces the object wholesale, but a consumer that opened a /vsis3/ handle against the first version and is part-way through a range read now sees an object whose ETag changed underneath it. GDAL surfaces this as a truncated read or an IReadBlock failed at X offset error rather than a clean conflict.

Double-counted features. A vector job that appends to PostGIS with a plain INSERT turns one duplicate delivery into duplicated geometry. A subsequent ST_Area roll-up by admin unit is then wrong by exactly the area of the duplicated parcels, and nothing in the pipeline errors. This is the most expensive failure class here: it is silent, it is discovered downstream weeks later by an analyst who notices a total that does not reconcile, and by then the provenance is gone.

Corrupted mosaics. The assembly step described in merging tiled Lambda outputs into a COG usually lists the scratch prefix and mosaics whatever it finds. If a duplicate worker wrote its tile under a key containing a UUID or a timestamp instead of the deterministic window index, the listing returns 412 objects for a 400-window grid, and gdalbuildvrt happily produces a VRT with overlapping sources whose resolution of the overlap depends on source order.

The sequence above is the canonical shape of the bug. Note where the damage happens: not at the duplicate delivery, which is harmless, but at the unconditional write. That is the observation the rest of this page is built on.

Delivery Semantics Across AWS, GCP and Azure

Before choosing a defence, be precise about what each layer actually promises. The table below is the guarantee each service publishes, not the behaviour you usually observe.

Two rows deserve comment. SQS FIFO’s five-minute deduplication interval is shorter than a raster job. A 12-minute reprojection that trips its visibility timeout is redelivered well outside the interval, and FIFO’s dedup does nothing. FIFO earns its place for ordering — see guaranteeing order with SQS FIFO for sequential tile jobs — but never for correctness over the timescales a geospatial backfill operates on.

Pub/Sub exactly-once delivery is a pull-subscription feature. If your tiler is a push-triggered Cloud Function, you are on at-least-once regardless of the subscription setting. Even on pull, the guarantee covers delivery to a subscriber that acks successfully; it does not cover a subscriber that writes an output and then crashes before acking. Azure’s Service Bus duplicate detection is the most generous of the three at a seven-day window, but it deduplicates on MessageId, which means it is only as good as the determinism of the ID you set — the subject of deterministic job IDs from object URI and ETag.

The Three Places to Enforce Idempotency

Idempotency is not one mechanism. It is three, at increasing cost and increasing strength, and a production pipeline uses all three because each catches a class the others miss.

At the trigger, you narrow what can even become a job. A suffix and prefix filter on the S3 notification, an Event Grid subject filter, or a Pub/Sub filter expression removes the events you never wanted, which removes an entire population of would-be duplicates for free. This is filtering, not deduplication: it costs nothing and it catches nothing that is a genuine repeat. Do it anyway, because every event you never dispatch is an event that cannot be duplicated downstream.

At the worker, you claim the job before doing the work. A conditional write against a key-value store — DynamoDB PutItem with ConditionExpression="attribute_not_exists(job_id)", Firestore create(), Table Storage insert — is a compare-and-set that the storage engine serialises on the partition key. Exactly one of two simultaneous claims wins. This is the layer that saves you money, because the loser returns before it downloads a 10 GB scene or spends 12 minutes of 10,240 MB Lambda time.

At the write, you make the output itself refuse a second copy. A conditional PutObject on the tile key means that even if both workers somehow got through the claim — because the claim row’s TTL expired, because a redrive stole a stale lease, because someone re-ran the backfill — only one object lands. This is the layer that saves your data, and it is the only one that is still correct when every other assumption has failed.

The ordering matters. The trigger gate is cheap and weak, the claim gate is cheap and strong-but-leasable, the write gate is the one that cannot be bypassed. Treat the first two as optimisations and the third as the invariant.

Step-by-Step Implementation

Step 1 — Derive a Deterministic Job Key

The job key must be a pure function of everything that determines the output, and of nothing else. Concretely: source bucket, URL-decoded key, the object’s version token, and a canonical serialisation of the processing parameters.

"""job_key.py — deterministic identity for one unit of spatial work."""
import hashlib
import json
import urllib.parse


def job_key(bucket: str, raw_key: str, version_token: str, params: dict) -> str:
    """Stable SHA-256 over source identity plus processing parameters.

    version_token is the S3 ETag (or versionId), the GCS generation, or the
    Azure blob ETag — whatever the provider gives you that changes when the
    bytes change. params must be canonicalised: sorted keys, no floats whose
    repr varies, and an explicit pipeline version so a code change produces a
    genuinely new job rather than silently reusing an old output.
    """
    key = urllib.parse.unquote_plus(raw_key)
    canonical = json.dumps(params, sort_keys=True, separators=(",", ":"))
    material = "\n".join((bucket, key, version_token, canonical))
    return hashlib.sha256(material.encode("utf-8")).hexdigest()


PARAMS = {
    "pipeline": "ndvi-tiler",
    "version": "3.2.0",       # bump this and every job legitimately re-runs
    "dst_crs": "EPSG:3857",
    "resampling": "bilinear",
    "tile_px": 512,
    "expression": "(B08 - B04) / (B08 + B04)",
}

print(job_key("sentinel-scenes", "L2A/T31UDQ%2FB08.tif", "9b2cf5a1-3", PARAMS))

Why the pipeline version belongs in the key is the part most implementations miss. Without it, deploying a resampling fix and re-running the backfill produces job keys identical to the ones already claimed, every claim loses, and the pipeline reports a clean run that changed nothing. The derivation rules, including what to do when the ETag is not a content hash, are worked through in deterministic job IDs from object URI and ETag.

Step 2 — Claim the Job with a Leased Conditional Write

A bare attribute_not_exists claim has one weakness: a worker that dies after claiming leaves the row wedged until its TTL fires, and DynamoDB TTL deletion runs within roughly 48 hours of the timestamp, not at it. A lease fixes that. The claim succeeds if the row does not exist or if the existing row’s lease has expired.

"""claim.py — leased idempotency claim on DynamoDB."""
import os
import time

import boto3
from botocore.exceptions import ClientError

_ddb = boto3.client("dynamodb")
TABLE = os.environ["IDEMPOTENCY_TABLE"]
LEASE_SECONDS = int(os.environ.get("LEASE_SECONDS", "1200"))   # 20 min > 15 min Lambda ceiling
RETENTION_SECONDS = int(os.environ.get("RETENTION_SECONDS", "604800"))  # 7 days


def claim(job_id: str, worker_id: str) -> bool:
    """Win the right to execute job_id. True means this worker owns it."""
    now = int(time.time())
    try:
        _ddb.put_item(
            TableName=TABLE,
            Item={
                "job_id": {"S": job_id},
                "state": {"S": "CLAIMED"},
                "owner": {"S": worker_id},
                "lease_until": {"N": str(now + LEASE_SECONDS)},
                "expires_at": {"N": str(now + RETENTION_SECONDS)},
            },
            # Free row, or a lease that has lapsed. A COMMITTED row never
            # matches, so a finished job is never re-executed.
            ConditionExpression=(
                "attribute_not_exists(job_id) OR "
                "(#s = :claimed AND lease_until < :now)"
            ),
            ExpressionAttributeNames={"#s": "state"},
            ExpressionAttributeValues={
                ":claimed": {"S": "CLAIMED"},
                ":now": {"N": str(now)},
            },
        )
        return True
    except ClientError as exc:
        if exc.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False    # someone else owns it, or it is already COMMITTED
        raise


def commit(job_id: str, worker_id: str, output_uri: str) -> None:
    """Seal the job. Fails loudly if the lease was stolen mid-flight."""
    _ddb.update_item(
        TableName=TABLE,
        Key={"job_id": {"S": job_id}},
        UpdateExpression="SET #s = :done, output_uri = :o, finished_at = :t",
        ConditionExpression="#o = :me",
        ExpressionAttributeNames={"#s": "state", "#o": "owner"},
        ExpressionAttributeValues={
            ":done": {"S": "COMMITTED"},
            ":me": {"S": worker_id},
            ":o": {"S": output_uri},
            ":t": {"N": str(int(time.time()))},
        },
    )

Set LEASE_SECONDS above your compute ceiling, never below it. Twenty minutes clears the 15-minute Lambda ceiling with margin; on Cloud Functions 2nd gen you need at least 65 minutes to clear the 60-minute ceiling; on Azure Functions Consumption, 15 minutes clears the 10-minute one.

The ConditionExpression="#o = :me" on commit is not decoration. If the lease lapsed and a second worker stole the claim, the first worker’s commit must fail rather than mark a job done that the second worker is still executing. A ConditionalCheckFailedException here is a genuine alarm and should be logged at ERROR, unlike the same exception on claim, which is the normal path.

Step 3 — Make the Write Itself Conditional

The claim is a lease. The write is the invariant. Every output object gets a key derived from the job key, and every PutObject carries a precondition.

"""commit_tile.py — the last line of defence."""
import boto3
from botocore.exceptions import ClientError

_s3 = boto3.client("s3")


def put_tile_once(bucket: str, key: str, body: bytes) -> str:
    """Write a tile at most once. Returns 'written' or 'already-present'."""
    try:
        _s3.put_object(
            Bucket=bucket,
            Key=key,
            Body=body,
            IfNoneMatch="*",            # succeeds only if the key is absent
            ContentType="image/tiff; application=geotiff; profile=cloud-optimized",
        )
        return "written"
    except ClientError as exc:
        if exc.response["Error"]["Code"] in ("PreconditionFailed", "ConditionalRequestConflict"):
            return "already-present"    # a duplicate got here first; this is success
        raise

PreconditionFailed is not an error condition in this design — it is the mechanism working. Log it at INFO with the job key, emit it as a metric, and return normally so the message is acked and does not redrive. Treating it as a failure is the single most common way teams turn a working idempotency layer into an infinite retry loop. Provider-specific forms of the same precondition — GCS ifGenerationMatch=0, Azure If-None-Match: *, DynamoDB attribute_not_exists — are compared in conditional writes for idempotent tile outputs.

Step 4 — Provision the Table and the Queue

resource "aws_dynamodb_table" "idempotency" {
  name         = "spatial-job-idempotency"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "job_id"

  attribute {
    name = "job_id"
    type = "S"
  }

  ttl {
    attribute_name = "expires_at"
    enabled        = true
  }

  point_in_time_recovery { enabled = true }
}

resource "aws_sqs_queue" "tile_jobs" {
  name                       = "tile-jobs"
  visibility_timeout_seconds = 960   # 16 min: above the 15 min Lambda ceiling
  message_retention_seconds  = 345600

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.tile_jobs_dlq.arn
    maxReceiveCount     = 3
  })
}

resource "aws_sqs_queue" "tile_jobs_dlq" {
  name                      = "tile-jobs-dlq"
  message_retention_seconds = 1209600
}

Keep the visibility timeout strictly above the handler ceiling. A visibility timeout below the timeout guarantees redelivery of every slow job, which converts a rare duplicate into a systematic one; where those messages end up once they stop succeeding is covered in implementing dead-letter queues for failed vector jobs.

Reprocessing and Backfills That Are Safe to Re-Run

A backfill is a deliberate flood of duplicates. Nine thousand scenes re-enqueued after a resampling fix will, on any pipeline that has ever been retried, contain jobs whose outputs already exist. The design above handles that correctly — provided you decide, explicitly, what “already exists” means for this run.

There are exactly three sane policies, and you must choose one per backfill:

• Skip. Same job key, output present, do nothing. Correct when you are filling gaps after a partial failure. Cheapest; a 9,000-scene gap-fill that finds 8,700 outputs present costs 8,700 PutObject calls that return 412 and nothing else.
• New key. Bump the pipeline version in the parameter set. Every job key changes, every claim wins, every output lands beside the old one under a version-qualified prefix. Correct when the algorithm changed and you need both results to compare. Doubles storage for the duration.
• Overwrite under a fresh generation. Drop the If-None-Match and instead pass the expected current version — S3 If-Match with the known ETag, GCS ifGenerationMatch=<n>, Azure If-Match: <etag>. Correct when you are repairing a known-bad output and want the write to fail if anything else touched it in the meantime.

What is never acceptable is an unconditional overwrite during a backfill, because a backfill runs concurrently with live ingest by definition and the two will collide on at least one key.

The state column makes each policy a one-line change to the claim’s ConditionExpression, which is why the state belongs in the row rather than being inferred from the output’s existence. Inferring it means a HeadObject per job — one extra request, one extra race, and a listing cost that scales with the backfill.

Idempotent Is Not Exactly-Once

These are routinely conflated, and the distinction is the whole design.

Exactly-once delivery means the transport hands the message to the consumer precisely one time. It is unachievable across an unreliable network, for a reason that has nothing to do with cloud vendors: a sender that does not receive an acknowledgement cannot distinguish “the message was lost” from “the acknowledgement was lost”. Resending risks a duplicate; not resending risks a loss. Every system chooses one, and every system worth using chooses the duplicate.

Exactly-once processing, as SQS FIFO uses the phrase, means the broker suppresses a repeat within a bounded window — five minutes for FIFO. It is a real guarantee inside its window and nothing at all outside it.

Idempotency is a property of your handler: executing it n times has the same effect on the world as executing it once. It makes no claim about how many times the handler runs. This is what you can actually build, and it is strictly more useful, because it survives every layer below it being wrong.

The practical consequence: stop trying to prevent the second execution and start making the second execution harmless. A tiler that writes to tiles/{job_key}/{row}_{col}.tif with If-None-Match: * is idempotent no matter how many times it runs, on any transport, with no coordination. A tiler that writes to tiles/{uuid4()}.tif cannot be made idempotent by any queue configuration in any cloud.

Measurement and Verification

Idempotency that has never rejected anything is untested. Instrument the rejection paths and alert on their absence as much as their volume.

The direct test is to replay a delivery and assert the output did not change:

"""verify_idempotent.py — prove re-execution is a no-op."""
import boto3
from handler import handler   # your tiler's entry point

s3 = boto3.client("s3")
BUCKET, KEY = "tiles-out", "tiles/ab12cd34/0_0.tif"

event = {"Records": [{"s3": {
    "bucket": {"name": "sentinel-scenes"},
    "object": {"key": "L2A/T31UDQ/B08.tif", "eTag": '"9b2cf5a1c0d3e4f5"'},
}}]}

handler(event, None)
first = s3.head_object(Bucket=BUCKET, Key=KEY)

handler(event, None)          # identical delivery, second time
second = s3.head_object(Bucket=BUCKET, Key=KEY)

assert first["ETag"] == second["ETag"], "output changed on replay — not idempotent"
assert first["LastModified"] == second["LastModified"], "object was rewritten"
print("idempotent:", first["ETag"])

Expected output is a single ETag line and no assertion failure. Run it in CI against a scratch bucket on every deploy; it is the assertion that catches a refactor which quietly reintroduced a UUID into an output key.

Failure Modes and Debugging

1. ConditionalCheckFailedException counted as an error rate. The claim’s rejection path is the normal path for a duplicate. Teams wire it into their generic error handler, the invocation fails, Lambda retries, the retry is also rejected, and the message eventually lands in the DLQ. Symptom: a DLQ full of messages whose objects processed correctly. Fix: catch the code explicitly and return success.

2. PreconditionFailed (HTTP 412) on PutObject with If-None-Match: *. Same shape at the write layer. In a healthy pipeline this is a low-single-digit percentage of writes. If it is a majority of writes, your claim gate is not running at all and every duplicate is reaching the expensive step before being stopped — check that the claim table name is actually populated in the environment.

3. The specified bucket does not have versioning enabled when using If-Match for repair writes. S3 accepts If-Match on PutObject only in the sense of matching the current object’s ETag; a repair workflow that assumes it can pin a versionId needs versioning turned on first. Enable versioning on the output bucket before writing any repair tooling that depends on it.

4. 412 Precondition Failed: At least one of the pre-conditions you specified did not hold on GCS. The GCS form of case 2, raised by ifGenerationMatch=0. The trap specific to GCS is that a resumable upload’s precondition is evaluated when the session is created, not when it is finalised — two workers can both create sessions against an absent object and both finalise. Set the precondition on the finalising request, or use a single-request upload for objects small enough to allow it.

5. ConditionNotMet / BlobAlreadyExists (409) on Azure Blob. Azure returns 409 BlobAlreadyExists for If-None-Match: * on a create, and 412 ConditionNotMet for a failed If-Match. Handling only 412 leaves the create path unguarded, and the resulting 409 propagates as an unhandled exception into the Functions host, which retries up to five times. Catch both codes.

6. LeaseStolen: two workers both hold a claim. Root cause is a lease shorter than the real job duration — typically a lease sized for the average tile and a scene whose edge windows take four times as long. The commit’s owner check surfaces it. Fix by sizing the lease against the compute ceiling (15 min on Lambda, 60 min on Cloud Functions 2nd gen, 10 min on Azure Consumption) rather than against observed runtimes, and by heartbeating the lease from long jobs.

Cost and Scaling

The idempotency layer is cheap in absolute terms and its cost is dominated by the claim table, not the conditional writes.

Against that, one duplicated 12-minute run of a 10,240 MB Lambda costs about $0.12 on its own, so the layer pays for itself at a duplicate rate above roughly two in a million. Real rates are orders of magnitude higher than that.

The scaling constraint is the claim table’s partition throughput. A SHA-256 job key distributes uniformly across DynamoDB partitions by construction, so there is no hot-partition problem — which is a second reason to hash the key rather than use a readable composite like bucket/key/etag, whose prefix clusters. At a 400-window fan-out against the default 1,000 regional Lambda concurrency, peak claim rate is 1,000 writes per second, comfortably inside on-demand’s auto-scaling response. Above 3,000 concurrent workers, pre-warm the table with provisioned capacity or accept a brief burst of ProvisionedThroughputExceededException retries at the start of each backfill.

One deliberate trade-off worth naming: the claim gate adds one round trip — 3 to 6 ms to DynamoDB — to every job, including the 99%+ that are not duplicates. For a tiling job measured in seconds that is invisible. For a per-feature stream processor handling 50,000 messages per second it is not, and the right answer there is to move the idempotency boundary up to the batch rather than the message, claiming once per micro-batch and making the batch’s write conditional. The same reasoning that governs batch versus stream geospatial processing applies to where the idempotency gate belongs.

Frequently Asked Questions

Is exactly-once processing possible in a serverless geospatial pipeline?

Not in the general case. Exactly-once delivery is unachievable across an unreliable network, because a sender cannot distinguish a lost message from a lost acknowledgement. What you can buy is exactly-once effect: the job body may execute several times, but every execution converges on one output because the write is conditional on a deterministic key. SQS FIFO’s 5-minute deduplication interval and Pub/Sub’s exactly-once delivery option on pull subscriptions narrow the window considerably, but neither removes the need for an idempotent write, and neither survives a redrive hours later.

Why is check-then-write not enough to make a tile write idempotent?

Because it is two requests with a gap between them. Worker A issues HeadObject, gets a 404, and begins encoding a 40 MB tile. Worker B issues HeadObject in that same window, also gets a 404, and also begins encoding. Both then PutObject, and the second silently replaces the first. Atomicity requires the check and the write to be a single request the storage engine evaluates under its own lock: If-None-Match: * on S3 and Azure Blob, ifGenerationMatch=0 on GCS, attribute_not_exists on DynamoDB.

What belongs in a deterministic job key, and what does not?

In: the source bucket, the URL-decoded object key, a version token that changes when the bytes change, and a canonical serialisation of every processing parameter including the pipeline’s own version. Out: timestamps, UUIDs, request IDs, the worker’s hostname, and anything else that varies between two deliveries of the same event. The test is simple — if two duplicate deliveries would produce different keys, the key is wrong; if two genuinely different jobs would produce the same key, the key is also wrong.

Does SQS FIFO give me exactly-once processing for tile jobs?

Within five minutes, yes. A tiling job that takes twelve minutes, trips its visibility timeout, and is redelivered has already left the deduplication interval, and FIFO’s dedup contributes nothing. FIFO is the right tool for ordering sequential work; for correctness across the hours-to-days window that a geospatial retry or backfill spans, use a durable conditional claim and a conditional write.

Where should the idempotency gate live if my jobs are tiny and very high volume?

Move it up a level. Claiming once per message costs one key-value round trip per message, which dominates when the message itself is a single AIS position. Instead, claim once per micro-batch — a window, a Kinesis shard checkpoint, a Pub/Sub pull batch — and make the batch’s aggregate write conditional on the batch key. The gate then costs one round trip per thousand messages instead of one per message, at the price of coarser retry granularity.

Do I still need this if my pipeline only ever writes to PostGIS?

Yes, and the mechanism is the same shape. Replace the conditional PutObject with an INSERT ... ON CONFLICT (job_id) DO NOTHING against a unique constraint on the job key, inside the same transaction as the feature insert. The unique index is the atomic gate; without it, a duplicated delivery duplicates geometry, and no amount of application-level checking closes the window between the SELECT and the INSERT.

Guides in this topic

• Conditional Writes for Idempotent Tile Outputs — Make a tile write refuse its own duplicate: S3 If-None-Match and If-Match preconditions, DynamoDB…
• Deterministic Job IDs from Object URI and ETag — Build a job ID that is a pure function of the source object and the processing parameters: canonical URI, the…

Related

• Deterministic Job IDs from Object URI and ETag — how to build the key this page depends on, and what to do when the ETag is not a content hash
• Conditional Writes for Idempotent Tile Outputs — the precondition APIs on S3, GCS, Azure Blob and DynamoDB, side by side
• Deduplicating S3 Event Notifications for Idempotent Ingestion — the ingest-side application of the same claim pattern to multi-file shapefile uploads
• SQS and Pub/Sub Queue Routing Strategies — the transport layer whose at-least-once semantics make all of this necessary
• Sizing SQS Visibility Timeout for Long-Running Raster Jobs — the single most common source of systematic, self-inflicted duplicate delivery

Back to Event-Driven Geospatial Processing Patterns