Deterministic Job IDs from Object URI and ETag

A job ID is correct when two duplicate deliveries of the same event produce the identical string, and any genuinely different job produces a different one. That means hashing exactly four things — the canonical object URI, a version token that changes when the bytes change, a canonical serialisation of the processing parameters, and the pipeline’s own version — into a 64-character SHA-256 digest. Everything else that appears in an event payload (request ID, event time, sequencer, the notification’s own delivery timestamp) is deliberately excluded, because it varies between duplicates and would defeat the entire mechanism.

Context

The idempotency and exactly-once spatial processing overview establishes that duplicates are inevitable and that the defence is a deterministic key plus a conditional write. This page is about the key. It is the smaller half of the problem and the half that quietly goes wrong, because a broken key does not throw — it just makes the claim table useless while every dashboard stays green.

There are two failure directions and they are not symmetrical. Under-determinism — the same job producing different IDs — is the loud one: duplicate outputs appear, the mosaic manifest disagrees with the grid, and someone notices. Over-determinism — two different jobs collapsing onto one ID — is the silent one: a legitimate re-upload of a corrected scene finds its ID already COMMITTED, the claim loses, the pipeline reports success, and the corrected data never enters the catalogue. The second is far more dangerous, and it is what you get from the obvious-looking key sha256(bucket + key).

The ingest-side treatment of this pattern in deduplicating S3 event notifications hashes bucket, key and ETag for a shapefile bundle. This page generalises that to the three providers and adds the parameter digest, which is what a processing job needs and a pure ingest job does not.

Prerequisites

• Runtime: Python 3.11 on AWS Lambda (Amazon Linux 2023), Cloud Functions 2nd gen, or Azure Functions. The derivation is pure computation — no SDK calls, no network, so it runs identically in all three and in your unit tests.
• Event payloads available: the S3 record’s s3.object.eTag and optional s3.object.versionId; the GCS Pub/Sub message attribute objectGeneration; the Event Grid data.eTag for Azure Blob. Confirm your notification format actually carries these before building on them — Event Grid’s Microsoft.Storage.BlobCreated schema includes eTag, the older Storage Queue schema does not.
• A frozen parameter contract. The set of processing parameters must be a fixed, versioned structure. Adding a parameter later changes every ID, so decide up front which knobs are part of the identity and record the decision.
• A PIPELINE_VERSION constant wired to the build, not to a mutable environment variable someone can edit in the console.
• Python dependencies: none beyond the standard library — hashlib, json, urllib.parse. Deliberately so: the job ID must never depend on a library whose serialisation could change under a patch bump.

Implementation

The module below is the whole thing. It is small on purpose: every branch in a job-ID function is a chance for two duplicates to take different paths.

"""job_identity.py — deterministic job IDs for spatial processing.

Pure standard library, no network, no clock, no randomness. Anything that
varies between two deliveries of the same event is excluded by construction.
"""
from __future__ import annotations

import hashlib
import json
import urllib.parse
from typing import Any, Mapping

# Bumped by the build. A change here re-runs every job under a new identity,
# which is exactly what you want after an algorithm fix.
PIPELINE_VERSION = "3.2.0"

# US ASCII unit separator: cannot occur in an object key, a bucket name, an
# ETag, or JSON output, so no field can impersonate a boundary.
SEP = "\x1f"


def canonical_uri(scheme: str, container: str, raw_key: str) -> str:
    """Provider-neutral object URI, decoded exactly once.

    Notification payloads percent-encode keys and use '+' for spaces. Hashing
    the raw form makes the ID disagree with the one derived from a GetObject
    path for every key containing a space or a slash-encoded segment.
    """
    key = urllib.parse.unquote_plus(raw_key)
    key = "/".join(seg for seg in key.split("/") if seg != "")   # collapse //
    return f"{scheme}://{container.strip().lower()}/{key}"


def canonical_params(params: Mapping[str, Any]) -> str:
    """Sorted, separator-free, float-free JSON.

    Floats are rendered by repr and repr is not stable across every runtime,
    so numeric parameters are formatted to a fixed precision as strings.
    """
    def norm(v: Any) -> Any:
        if isinstance(v, float):
            return f"{v:.6f}"
        if isinstance(v, (list, tuple)):
            return [norm(i) for i in v]
        if isinstance(v, dict):
            return {k: norm(v[k]) for k in sorted(v)}
        return v

    body = {k: norm(params[k]) for k in sorted(params)}
    body["_pipeline_version"] = PIPELINE_VERSION
    return json.dumps(body, sort_keys=True, separators=(",", ":"), ensure_ascii=True)


def job_id(scheme: str, container: str, raw_key: str,
           version_token: str, params: Mapping[str, Any]) -> str:
    """SHA-256 over URI, version token and parameter digest."""
    if not version_token:
        raise ValueError(
            "empty version token — the notification did not carry an ETag or "
            "generation; refuse rather than silently key on the URI alone"
        )
    material = SEP.join((
        canonical_uri(scheme, container, raw_key),
        version_token.strip('"'),          # S3 quotes its ETags in payloads
        canonical_params(params),
    ))
    return hashlib.sha256(material.encode("utf-8")).hexdigest()


# ---- provider adapters: one place where the payload shapes differ ----------

def from_s3_record(record: Mapping[str, Any], params: Mapping[str, Any]) -> str:
    obj = record["s3"]["object"]
    # versionId is stronger than ETag when the bucket is versioned: it is
    # unique per write even if two writes happen to produce identical bytes.
    token = obj.get("versionId") or obj.get("eTag", "")
    return job_id("s3", record["s3"]["bucket"]["name"], obj["key"], token, params)


def from_gcs_attributes(attrs: Mapping[str, str], params: Mapping[str, Any]) -> str:
    # GCS generation is a monotonic int64 assigned per write — the cleanest
    # version token of the three providers. metageneration is NOT usable:
    # it changes on a metadata-only edit that leaves the bytes untouched.
    return job_id("gs", attrs["bucketId"], attrs["objectId"],
                  attrs["objectGeneration"], params)


def from_event_grid(data: Mapping[str, Any], params: Mapping[str, Any]) -> str:
    # url is like https://acct.blob.core.windows.net/container/path/to.tif
    parsed = urllib.parse.urlparse(data["url"])
    container, _, blob = parsed.path.lstrip("/").partition("/")
    return job_id("azblob", container, blob, data["eTag"], params)

Note what job_id does when the version token is missing: it raises. The tempting alternative — fall back to hashing the URI alone — is precisely the over-determinism failure described above, and it fails silently. Refusing turns a configuration mistake into a visible error at deploy time instead of a data loss six months later.

Verification

Two properties must hold, and both are cheap to assert. Determinism: duplicate deliveries agree. Sensitivity: every field that affects the output changes the ID.

"""test_job_identity.py — the two properties that matter."""
from job_identity import from_s3_record, job_id

PARAMS = {"dst_crs": "EPSG:3857", "resampling": "bilinear", "tile_px": 512}

rec = {"s3": {
    "bucket": {"name": "sentinel-scenes"},
    "object": {"key": "L2A/T31UDQ%2FB08.tif", "eTag": '"9b2cf5a1c0d3e4f5"'},
}}
dup = {"s3": {                       # same event, different delivery envelope
    "bucket": {"name": "SENTINEL-SCENES"},
    "object": {"key": "L2A//T31UDQ%2FB08.tif", "eTag": "9b2cf5a1c0d3e4f5"},
}}

a = from_s3_record(rec, PARAMS)
b = from_s3_record(dup, PARAMS)
print("determinism  :", a == b, a[:16])

# Sensitivity: each of these must differ from `a`.
changed_crs = job_id("s3", "sentinel-scenes", "L2A/T31UDQ/B08.tif",
                     "9b2cf5a1c0d3e4f5", {**PARAMS, "dst_crs": "EPSG:4326"})
changed_etag = job_id("s3", "sentinel-scenes", "L2A/T31UDQ/B08.tif",
                      "aa11bb22cc33dd44", PARAMS)
print("crs-sensitive:", changed_crs != a)
print("etag-sensitive:", changed_etag != a)

Expected output:

determinism  : True 4f1d9c7ab3e05628
crs-sensitive: True
etag-sensitive: True

The first line is the one that catches real bugs: it proves case-folded buckets, doubled separators, quoted ETags and percent-encoded keys all collapse to one identity. Wire it into CI alongside the replay assertion from the parent overview’s verification section, so a refactor that reintroduces a timestamp into the material string fails the build rather than the mosaic.

Gotchas and Edge Cases

• A multipart ETag is not an MD5, and that is fine. For a single-part PutObject the ETag is the hex MD5 of the body. For a multipart upload it is the MD5 of the concatenated part MD5s, suffixed with -N for the part count — so d41d8cd98f00b204e9800998ecf8427e-6 is a perfectly good version token and a useless checksum. Satellite scenes almost always arrive multipart because they exceed the 5 GB single-PUT limit, so this is the common case, not the exotic one. The same applies to SSE-KMS objects, whose ETag is not an MD5 under any upload mode. Use the ETag for identity; use x-amz-checksum-sha256 or your own sidecar digest for integrity.

• metageneration is not generation. On GCS, generation increments on every content write; metageneration increments when metadata changes with the bytes untouched. Keying on metageneration, or on a concatenation of both, means that setting a cache-control header on 9,000 archived scenes enqueues 9,000 unnecessary reprocessing jobs. Key on generation alone. If you also want metadata edits to trigger reprocessing, that is a separate pipeline with a separate parameter set, not a wider job ID.

• A versioned bucket makes versionId strictly better than the ETag. Two writes of byte-identical content produce the same ETag but different versionIds. If your workflow can legitimately re-upload identical bytes and expects a re-run — a common pattern when the parameters are what changed but you re-trigger by re-uploading — the ETag collapses those into one job and the second run is silently skipped. Prefer versionId when the bucket has versioning enabled, and fall back to the ETag when it does not, exactly as from_s3_record does.

• Never put the S3 sequencer in the key. The sequencer field orders events for a single object key and is explicitly documented as varying between notifications. It is the correct tool for deciding which of two events is newer; it is the wrong tool for deciding whether they are the same event. Including it guarantees that every duplicate gets a fresh job ID — the exact failure the whole mechanism exists to prevent. The same rule rules out eventTime, the Lambda request ID, and the Pub/Sub message_id.

• Hash length beats readability at the storage layer. A composite key like s3://bucket/very/long/scene/path.tif|etag|params is easier to read in a table scan, but object keys can run to 1,024 bytes and DynamoDB caps a partition key at 2,048 bytes — a limit a deeply nested Sentinel-2 path plus a parameter blob can reach. The fixed 64-character digest never can. It also spreads writes uniformly across partitions, which matters at the 1,000-concurrency fan-out described under chunked I/O for large satellite imagery. Store the human-readable fields as separate non-key attributes when you need them for debugging.

Frequently Asked Questions

Is an S3 ETag safe to use as a version token?

Yes for identity, no for integrity. A single-part upload’s ETag is the MD5 of the body; a multipart upload’s is a hash-of-hashes with a -N suffix; an SSE-KMS object’s is neither. In all three cases it changes whenever the object is rewritten, which is the only property a job ID needs. Problems only arise when a team later reuses the same field to verify content and finds it does not match a locally computed MD5.

Should the processing parameters really be part of the job ID?

Yes, because the ID identifies the output, not the input. Reprojecting one scene to EPSG:3857 and to EPSG:4326 are two different jobs producing two different tile sets; giving them one ID means the second loses its claim and never runs. Include the pipeline version for the same reason in reverse — after an algorithm fix, you want every ID to change so the backfill actually reprocesses rather than reporting a clean no-op run.

Why hash rather than use a readable composite key?

Fixed length, uniform distribution, and no separator ambiguity. Sixty-four hex characters always fit DynamoDB’s partition-key limit where a 1,024-byte object key plus a parameter blob may not; a digest spreads uniformly across partitions where a bucket-prefixed composite clusters into one; and no field can contain the separator, which a raw composite cannot promise about object keys.

What if the notification does not carry a version token at all?

Refuse the event rather than degrade the key. Some legacy notification configurations, and the older Azure Storage Queue event schema, omit the ETag. The correct response is a HeadObject/get_blob_properties call to fetch it before deriving the ID — one extra request, and it produces the same value every time for a given object version, so determinism is preserved. Falling back to hashing the URI alone silently converts every legitimate re-upload into a skipped job.

Does the job ID change if I add a new processing parameter?

Yes, for every job, immediately. That is the correct behaviour but it is disruptive, so treat the parameter set as a versioned contract. Add the parameter and bump PIPELINE_VERSION in the same commit, plan for the backfill, and never add a parameter with a default value under the assumption that existing IDs will be unaffected — the canonical serialisation includes defaults, so they are not.

Related

• Idempotency and Exactly-Once Spatial Processing — the parent pattern this key feeds: the trigger, worker and write gates
• Conditional Writes for Idempotent Tile Outputs — where the derived ID is spent, as an output key and a write precondition
• Deduplicating S3 Event Notifications for Idempotent Ingestion — the ingest-side version of the same key, applied to multi-file shapefile bundles
• Triggering GCP Cloud Functions on New Shapefile Uploads — where the GCS generation attribute reaches your handler
• Guaranteeing Order with SQS FIFO for Sequential Tile Jobs — the job ID doubles as the MessageDeduplicationId when the queue is FIFO

Back to Idempotency and Exactly-Once Spatial Processing