Conditional Writes for Idempotent Tile Outputs

A tile write is idempotent when the storage engine itself refuses the second copy, in the same request that performs the first. That means PutObject with If-None-Match: * on S3 and Azure Blob, an upload with ifGenerationMatch=0 on Cloud Storage, and PutItem with ConditionExpression="attribute_not_exists(pk)" on DynamoDB — all of which return a 412 or 409 that your handler must treat as success, not failure. Get that last part wrong and a correct pipeline still fills its dead-letter queue, because every rejected duplicate is retried, rejected again, and eventually exhausts maxReceiveCount.

Context

The idempotency and exactly-once spatial processing overview names three enforcement points and calls the write the invariant — the only one still correct after a lease has lapsed or a backfill has been re-run beside live ingest. This page is that layer in detail.

The reason it has to be the storage engine and not your code is that HeadObject followed by PutObject is two requests. Between them sits a window measured in tens of milliseconds during which a second worker can perform the same check and reach the same conclusion. At a 400-window fan-out running against the default 1,000 regional Lambda concurrency, that window is entered thousands of times per scene. Making it small does not make it safe; only collapsing the check and the write into one request the engine evaluates under its own lock does.

Why HeadObject followed by PutObject is not atomicA sequence between four participants. Worker A asks the output bucket whether the tile key exists and is told 404. Worker B, a duplicate delivery, asks the same question in the same window and is also told 404. Both encode the tile and both write it unconditionally, so the second overwrites the first while a mosaic reader is part-way through a range read against the earlier version.Two requests leave a window; one request does notWorker AWorker B (duplicate)Output bucketMosaic readerHeadObject 0007_0012.tif404 — key is absentHeadObject, same keyalso 404, same windowPutObject 4 MBunconditional, ETag 6d1f0a9crange read on /vsis3/opened against 6d1f0a9cPutObject 4 MB againunconditional, ETag changesIReadBlock failed atoffsetETag moved mid-read
The gap between the check and the write is tens of milliseconds, and a 400-window fan-out enters it thousands of times per scene. Shrinking the gap never closes it.

There is a second, subtler reason specific to raster work. A COG is not a small object — a 512×512 float32 tile with overviews runs to a few megabytes, and a scene-level output can run to gigabytes. An unconditional overwrite replaces the object atomically from S3’s point of view, but a consumer already part-way through a /vsis3/ range read against the previous version sees its ETag change mid-read and GDAL reports a truncated block. The precondition prevents the overwrite from happening at all, which is a stronger property than making it atomic.

Prerequisites

  • A deterministic job ID. Every output key must be a pure function of it — see deterministic job IDs from object URI and ETag. A precondition on a key containing a UUID protects nothing, because the duplicate writes to a different key.
  • SDK versions that support the preconditions. boto3 1.34.x or later for IfNoneMatch on put_object; google-cloud-storage 2.x for if_generation_match; azure-storage-blob 12.x for match_condition.
  • IAM: s3:PutObject on the output prefix only, dynamodb:PutItem on the manifest table ARN. A precondition is not an authorisation control — scope the role as tightly as you would without it, following the boundaries in IAM security boundaries for cloud GIS.
  • Retry policy that distinguishes outcomes. Your SQS maxReceiveCount and Lambda retry settings must not be reached by precondition failures. If they can be, the handler is misclassifying them.
  • Tiles under 5 GB. Single-request PutObject is what makes the precondition cheap; multipart moves the check to completion time, after every byte has already been transferred.

Implementation

One module, four providers, one contract: commit_tile returns "written" or "already-present" and raises only on genuine faults.

python 
"""conditional_commit.py — write a tile at most once, on any of four backends.

The contract: 'written' means this call created the object; 'already-present'
means a duplicate created it first and the output is correct either way.
Anything else raises so the platform can retry.
"""
from __future__ import annotations

import boto3
from botocore.exceptions import ClientError

_s3 = boto3.client("s3")
_ddb = boto3.client("dynamodb")

COG_TYPE = "image/tiff; application=geotiff; profile=cloud-optimized"

# S3 raises PreconditionFailed for a lost race and ConditionalRequestConflict
# when two conditional writes to the same key overlap in flight. Both mean
# "someone else owns this key" and both are success for our purposes.
_S3_ALREADY = {"PreconditionFailed", "ConditionalRequestConflict"}


def commit_tile_s3(bucket: str, key: str, body: bytes) -> str:
    """S3: create-if-absent via If-None-Match: *."""
    try:
        _s3.put_object(Bucket=bucket, Key=key, Body=body,
                       IfNoneMatch="*", ContentType=COG_TYPE)
        return "written"
    except ClientError as exc:
        if exc.response["Error"]["Code"] in _S3_ALREADY:
            return "already-present"
        raise


def repair_tile_s3(bucket: str, key: str, body: bytes, expect_etag: str) -> str:
    """S3: replace-only-this-version via If-Match.

    Used during a repair backfill: the write lands only if the object is
    still exactly the one you inspected. If anything else rewrote it in the
    meantime the write fails rather than clobbering a newer result.
    """
    try:
        _s3.put_object(Bucket=bucket, Key=key, Body=body,
                       IfMatch=expect_etag, ContentType=COG_TYPE)
        return "written"
    except ClientError as exc:
        if exc.response["Error"]["Code"] in _S3_ALREADY:
            return "superseded"
        raise


def commit_tile_gcs(bucket_name: str, blob_name: str, body: bytes) -> str:
    """GCS: create-if-absent via generation precondition zero."""
    from google.api_core.exceptions import PreconditionFailed
    from google.cloud import storage

    blob = storage.Client().bucket(bucket_name).blob(blob_name)
    try:
        # if_generation_match=0 means 'only if no live generation exists'.
        # Passing this to a single-request upload evaluates it before any
        # bytes move; a resumable session evaluates it at finalise instead.
        blob.upload_from_string(body, content_type=COG_TYPE,
                                if_generation_match=0)
        return "written"
    except PreconditionFailed:
        return "already-present"


def commit_tile_azure(container_url: str, blob_name: str, body: bytes) -> str:
    """Azure Blob: create-if-absent via If-None-Match: *.

    Azure answers 409 BlobAlreadyExists on a failed create and 412
    ConditionNotMet on a failed If-Match. Catching only one of the two
    leaves the other to escape into the Functions host, which retries the
    invocation up to five times before giving up.
    """
    from azure.core import MatchConditions
    from azure.core.exceptions import ResourceExistsError, ResourceModifiedError
    from azure.storage.blob import ContainerClient

    client = ContainerClient.from_container_url(container_url)
    try:
        client.upload_blob(name=blob_name, data=body, overwrite=False,
                           etag="*", match_condition=MatchConditions.IfMissing)
        return "written"
    except (ResourceExistsError, ResourceModifiedError):
        return "already-present"


def record_manifest_entry(table: str, job_id: str, window: str,
                          output_uri: str, bytes_written: int) -> str:
    """DynamoDB: one manifest row per window, created once.

    The assembly step counts rows here rather than listing the output prefix,
    so a duplicate object that somehow landed cannot inflate the tile count.
    """
    try:
        _ddb.put_item(
            TableName=table,
            Item={
                "job_id": {"S": job_id},
                "window": {"S": window},
                "output_uri": {"S": output_uri},
                "bytes": {"N": str(bytes_written)},
            },
            # Composite key (job_id, window): both parts must be absent.
            ConditionExpression="attribute_not_exists(job_id) AND attribute_not_exists(#w)",
            ExpressionAttributeNames={"#w": "window"},
        )
        return "written"
    except ClientError as exc:
        if exc.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return "already-present"
        raise
Create-if-absent preconditions across S3, GCS, Azure Blob and DynamoDBComparison grid across AWS S3, GCP Cloud Storage, Azure Blob Storage and DynamoDB of five properties: the create-if-absent precondition, the replace-only-this-version precondition, the status code returned when the precondition fails, the SDK exception raised, and when the precondition is evaluated relative to the data transfer.The same idea, four spellingsAWS S3GCSAzure BlobDynamoDBCreate if absentIf-None-Match: *ifGenerationMatch=0If-None-Match: *attribute_not_exists(pk)Replace this versionIf-Match: etagifGenerationMatch=nIf-Match: etagattribute_exists+ #v = :vCode on rejection412PreconditionFailed412preconditionFailed409 / 412two codes, catch both400conditional checkSDK exceptionClientErrorPreconditionFailedResourceExistsErrorConditionalCheckFailedChecked before bytes moveYessingle PutObjectOnly if notresumablesession checks atcreateYesupload_blob createYesitem is the payloadMultipart on S3 moves the check to CompleteMultipartUpload — correct, but both duplicates have already paid for the transfer.
Only the last row differs in substance: a GCS resumable session checks its precondition at session creation, so two workers can both pass it. Everything above is the same mechanism spelled four ways.

The manifest row is not redundant with the object. The object proves a tile exists; the manifest row proves this pipeline wrote it, with a byte count the assembly step can check. Counting manifest rows rather than listing the prefix is what makes the gap check in merging tiled Lambda outputs into a COG immune to stray objects left by an earlier, unconditional version of the pipeline.

Classifying the rejection correctly

Every function above returns a string, not an exception, when the precondition rejects the write. That choice is the whole difference between a pipeline that absorbs duplicates silently and one that alarms on them. A rejected conditional write means the desired end state was reached by another path, so the message must be acknowledged; re-raising instead sends the message back to the queue, where the retry issues the identical write, receives the identical rejection, and burns one of the three maxReceiveCount attempts each time.

Treating a precondition failure as an error versus as successTwo panels contrasting the outcomes of misclassifying a 412 precondition failure and handling it correctly. The left panel shows the handler raising, the invocation failing, SQS redelivering, the retry also receiving 412, and the message reaching maxReceiveCount and landing in the dead-letter queue with the output already correct. The right panel shows the handler returning already-present, emitting a counter, acknowledging the message, and leaving the dead-letter queue empty.The same 412, classified two ways412 raised as a failureHandler re-raises PreconditionFailedInvocation marked failed, message returns to the queueRetry issues the identical write, gets 412 againmaxReceiveCount 3 exhausted in under a minuteDLQ fills with messages whose tiles are all correctAlarm fires on ApproximateNumberOfMessagesVisible412 returned as already-presentHandler catches PreconditionFailed and returnsCounter ConditionalWritePreconditionFailed incrementedMessage acknowledged and deleted from the queueManifest row already recorded by the winning workerDLQ stays empty; duplicate rate stays observableA sustained zero on the counter means the gate is untestedA precondition failure is the mechanism succeeding. Log it at INFO, count it, and acknowledge the message.
Both panels end with the correct tile in the bucket. Only one of them ends with an empty dead-letter queue and an on-call engineer who is asleep.

Emit the rejection as a counter rather than swallowing it. A sustained zero on ConditionalWritePreconditionFailed does not mean the pipeline has no duplicates — it usually means the metric is not wired up, and an assertion that has never rejected anything proves nothing about the assertion.

Verification

Write the same tile twice and assert nothing changed. LastModified is the sharper of the two assertions — an identical body produces an identical ETag even on a genuine rewrite, so ETag alone would pass a broken implementation.

python 
"""test_conditional_commit.py — prove the second write did not land."""
import boto3
from conditional_commit import commit_tile_s3

s3 = boto3.client("s3")
BUCKET = "tiles-scratch"
KEY = "tiles/4f1d9c7ab3e05628/0007_0012.tif"
BODY = open("fixtures/window_0007_0012.tif", "rb").read()

print("first :", commit_tile_s3(BUCKET, KEY, BODY))
head1 = s3.head_object(Bucket=BUCKET, Key=KEY)

print("second:", commit_tile_s3(BUCKET, KEY, BODY))
head2 = s3.head_object(Bucket=BUCKET, Key=KEY)

assert head1["LastModified"] == head2["LastModified"], "object was rewritten"
assert head1["ETag"] == head2["ETag"]
print("stable:", head1["ETag"], head1["ContentLength"], "bytes")

Expected output — the second call reports the duplicate and the object’s modification time is untouched:

code 
first : written
second: already-present
stable: "6d1f0a9c4b7e2d38" 4194304 bytes

Confirm the same on the manifest side, where the count is what the assembly step trusts:

bash 
aws dynamodb query \
  --table-name tile-manifest \
  --key-condition-expression 'job_id = :j' \
  --expression-attribute-values '{":j":{"S":"4f1d9c7ab3e05628"}}' \
  --select COUNT
# Expected: {"Count": 400, "ScannedCount": 400}  — never 412

Gotchas and Edge Cases

  • If-None-Match on a multipart upload is checked at completion, not at creation. Two duplicate workers can both run CreateMultipartUpload, both upload every part of a 6 GB scene-level output, and only then does one of them lose at CompleteMultipartUpload. The result is correct and the bill is double. Keep individual tile outputs under the 5 GB single-PUT limit so the precondition is evaluated before any bytes move — which is another argument for the window sizing discussed in chunked I/O for large satellite imagery. Where a large single output is unavoidable, put the claim gate in front of it so the loser never starts the upload.

  • A GCS resumable session evaluates its precondition when the session is created. blob.upload_from_string switches to a resumable upload above roughly 8 MiB by default. Two workers can then both create sessions against an absent object, both see ifGenerationMatch=0 satisfied at session creation, and both finalise — the second overwriting the first. Force a single-request upload for small tiles by raising the chunk-size threshold, or re-assert the precondition on the finalising request. This is the one place where the four providers genuinely differ in strength rather than in spelling.

  • Azure returns two different codes for the same idea. A failed create-if-absent is 409 BlobAlreadyExists (ResourceExistsError); a failed If-Match is 412 ConditionNotMet (ResourceModifiedError). Handlers written against S3 first tend to catch only the 412 shape, and the uncaught 409 escapes into the Azure Functions host, which retries the invocation up to five times. Catch both, and remember the Consumption plan’s 10-minute ceiling and 1,536 MB memory limit mean those five retries are five full re-runs of a job that was already correct.

  • Never mix a conditional write with a non-deterministic key. tiles/{job_id}/{row}_{col}.tif is protected; tiles/{job_id}/{uuid4()}.tif is not, because the duplicate writes somewhere else entirely and the precondition trivially succeeds. The precondition is only as strong as the determinism of the key it guards, which is why the job ID derivation and this page are two halves of one mechanism.

Frequently Asked Questions

Why is a 412 PreconditionFailed not an error?

Because it is the precondition working. On a create-if-absent write, a 412 means a duplicate reached the key first and the correct output is already in place — the desired end state, arrived at by a different path. Log it at INFO with the job ID, emit it as a counter, and return success so the message is acknowledged. Classifying it as a failure produces the characteristic symptom of a dead-letter queue full of messages whose outputs are all present and correct.

Do multipart uploads honour If-None-Match?

They do, but at CompleteMultipartUpload rather than at CreateMultipartUpload. That is still correct — exactly one completion wins — but both duplicates have already paid for the full transfer by the time the loser finds out. Prefer a single PutObject for anything under the 5 GB limit, and put the claim gate in front of any output large enough to require multipart.

What is the difference between If-None-Match and If-Match here?

If-None-Match: * is create-only: succeed only if nothing exists at the key. If-Match: "<etag>" is replace-only-this-version: succeed only if the current object is exactly the one you read. Normal tile commits use the first. Repair backfills use the second, so that a corrective write fails rather than clobbering a result that something else produced while you were computing.

Does a conditional write protect me if two workers use different output keys?

No, and this is the most common way the mechanism is defeated. A precondition is a statement about one key; two workers writing to two keys never contend. Everything therefore rests on the output key being a pure function of the job ID and the window index, with no UUID, timestamp, hostname or attempt counter anywhere in the path.

How much does the precondition cost?

Nothing extra on the successful path — it is a header on a request you were already making. On the rejected path you pay one PUT request, about $5 per million on S3, and transfer no body. At a realistic duplicate rate of 0.1% that is roughly $0.005 per million tiles, against the several cents a single duplicated 12-minute run of a 10,240 MB Lambda costs on its own.

Back to Idempotency and Exactly-Once Spatial Processing