Guaranteeing Order with SQS FIFO for Sequential Tile Jobs

Route sequential tile jobs through an SQS FIFO queue and set a MessageGroupId per ordering unit — a tile column, a mosaic strip, or a scene id — so dependent tiles process in exact publish order while independent tiles still run in parallel. A standard FIFO queue sustains 300 messages per second, or 3,000 with 10-message batches; enabling high-throughput mode lifts that ceiling to roughly 9,000 messages per second per queue. On GCP the same guarantee comes from Pub/Sub ordering keys, which pin same-key messages to publish order.

FIFO Ordering of Sequential Tile Jobs Diagram showing tile jobs entering an SQS FIFO queue tagged with a MessageGroupId per column. Messages sharing a group id are delivered strictly in publish order to the consumer, which writes mosaic seams and an incremental catalog while independent columns process in parallel. Tile Producer GroupId = col-X SQS FIFO col-3 · z0 → z1 → z2 col-4 · z0 → z1 → z2 col-5 · z0 → z1 → z2 one per group Tile Consumer rasterio mosaic Mosaic seam COG write Catalog STAC append 300 msg/s · 3,000 batched · 9,000 high-throughput order preserved within each MessageGroupId

Context

Most tile jobs are embarrassingly parallel — but not all of them. A handful of geospatial write patterns break if two tiles land out of sequence, and those are exactly the cases where the strict ordering of an SQS FIFO queue earns its throughput penalty. The SQS and Pub/Sub Queue Routing Strategies layer decides which jobs deserve ordering guarantees and which can flow through a cheaper standard queue.

Three ordering-sensitive cases recur in raster pipelines:

  • Mosaic seam blending. When adjacent tiles feather their overlap into a seamless mosaic, tile N+1 reads the already-written edge pixels of tile N. If N+1 runs first, the blend samples empty or stale nodata and leaves a visible seam. Column-major ordering keeps each strip internally consistent.
  • Z-order pyramid builds. Overview levels in a Cloud-Optimized GeoTIFF are derived from the level below. A parent tile at zoom z must not be assembled before its four children at z+1 are written, so the reduce step needs deterministic ordering per pyramid column.
  • Incremental catalog writes. Appending STAC items or updating a spatial index with last-writer-wins semantics requires that revisions to the same scene arrive in order; otherwise an older footprint can clobber a newer one.

FIFO ordering complements — it does not replace — the failure isolation in dead letter queues for failed vector jobs. Ordering decides when a tile runs; the DLQ decides where it goes when it cannot run at all.

Prerequisites

  • Runtime: Python 3.11+ on AWS Lambda, or Python 3.12 on GCP Cloud Functions 2nd gen for the ordering-key path.
  • Dependencies: boto3>=1.34.0 (AWS) or google-cloud-pubsub>=2.21.0 (GCP); rasterio>=1.3.9 for the tile write itself.
  • Queue: A FIFO queue whose name ends in .fifo. Decide up front between content-based deduplication and explicit MessageDeduplicationId values — you cannot rely on both simultaneously without surprises.
  • IAM (AWS): sqs:SendMessage, sqs:ReceiveMessage, sqs:DeleteMessage, and sqs:GetQueueAttributes on the FIFO queue ARN.
  • IAM (GCP): pubsub.topics.publish on the ordered topic and pubsub.subscriptions.consume on a subscription with enableMessageOrdering set.
  • Environment variables: TILE_QUEUE_URL (AWS) or PROJECT_ID and TILE_TOPIC_ID (GCP). Never hardcode the queue URL in the handler.

Ordering Mechanisms Across Providers

Every major serverless queue offers an ordering primitive, but the granularity and throughput trade-offs differ sharply. The contract is the same: pin related messages to one ordering unit, accept lower per-unit throughput, and process that unit serially.

Platform Ordering primitive Ordering scope Base throughput Batched / boosted
AWS SQS FIFO MessageGroupId Per group 300 msg/s 3,000 batched; ~9,000 high-throughput mode
GCP Pub/Sub Ordering key Per key, per region 1 MB/s per ordering key Unlimited keys in parallel
Azure Service Bus Session id (SessionId) Per session Varies by tier Sessions processed by one locked consumer

On AWS, high-throughput mode trades a subtle guarantee: ordering holds within each MessageGroupId but no longer across the queue as a whole, which is exactly what you want when each tile column is an independent strip. On GCP, ordering keys have no fixed message-per-second cap but do impose a 1 MB/s per-key sustained limit — spread scenes across many keys and you scale horizontally. Azure sessions lock the whole session to a single receiver, which serializes a scene at the cost of receiver-side parallelism.

Implementation

The producer below fans a partitioned raster into ordered tile jobs. It derives the MessageGroupId from the tile column so that seam blending within a column stays sequential, while distinct columns receive distinct groups and process concurrently. The deduplication id is deterministic, so a retried publish never enqueues the same tile twice. This pairs directly with partitioning a GeoTIFF into Step Functions map tiles upstream.

python 
import hashlib
import json
import os

import boto3

sqs = boto3.client("sqs")
TILE_QUEUE_URL = os.environ["TILE_QUEUE_URL"]  # must end in .fifo


def _dedup_id(scene_id: str, col: int, row: int, revision: int) -> str:
    """Deterministic id so replays of the same tile revision are deduplicated."""
    raw = f"{scene_id}:{col}:{row}:{revision}"
    return hashlib.sha256(raw.encode()).hexdigest()


def enqueue_tile_column(scene_id: str, col: int, rows: list[int], revision: int) -> None:
    """Publish one column of tiles as an ordered group.

    All tiles in a column share a MessageGroupId, so SQS delivers them in the
    exact order sent — required for mosaic seam blending down the column.
    Different columns get different group ids and run in parallel.
    """
    group_id = f"{scene_id}:col-{col}"

    for row in rows:  # rows must already be sorted in dependency order
        body = {
            "scene_id": scene_id,
            "col": col,
            "row": row,
            "revision": revision,
        }
        sqs.send_message(
            QueueUrl=TILE_QUEUE_URL,
            MessageBody=json.dumps(body),
            MessageGroupId=group_id,
            # Explicit dedup id — do NOT enable ContentBasedDeduplication too,
            # or an unchanged body across revisions would be silently dropped.
            MessageDeduplicationId=_dedup_id(scene_id, col, row, revision),
        )

The consumer processes exactly one in-flight message per group. SQS FIFO enforces this automatically: it will not deliver a second message from a group while an earlier one is still in flight, so a slow tile write cannot let the next tile in its column jump ahead.

python 
import json
import os

import boto3
import rasterio  # used by write_tile / update_catalog below

sqs = boto3.client("sqs")
TILE_QUEUE_URL = os.environ["TILE_QUEUE_URL"]


def handler(event, context):
    # Configure the Lambda event source mapping with BatchSize up to 10 and
    # order is still preserved per MessageGroupId within the batch.
    for record in event.get("Records", []):
        receipt_handle = record["receiptHandle"]
        group_id = record["attributes"]["MessageGroupId"]
        body = json.loads(record["body"])

        # 1. Write the tile — seam blending reads the previous tile in this column
        write_tile(body["scene_id"], body["col"], body["row"], body["revision"])

        # 2. Append to the catalog only after the pixel write succeeds
        update_catalog(body["scene_id"], body["col"], body["row"], body["revision"])

        # 3. Delete last — an unhandled error above leaves the message in flight
        #    so the whole column stalls rather than skipping a seam.
        sqs.delete_message(QueueUrl=TILE_QUEUE_URL, ReceiptHandle=receipt_handle)

The delete-last ordering matters: if write_tile raises, the message stays in flight, its visibility timeout eventually expires, and the tile is redelivered — the column pauses instead of proceeding with a missing seam. Sizing that visibility window correctly for a slow raster write is its own decision, covered in sizing SQS visibility timeout for long-running raster jobs.

Verification

Replay a known tile sequence and assert the consumer observed strictly ascending rows within each group. Because FIFO holds one message per group in flight, a correct consumer will always log rows in the order they were sent.

bash 
# Send three tiles in column 3 in dependency order
for row in 0 1 2; do
  aws sqs send-message \
    --queue-url "$TILE_QUEUE_URL" \
    --message-body "{\"scene_id\":\"S2A_2025\",\"col\":3,\"row\":$row,\"revision\":1}" \
    --message-group-id "S2A_2025:col-3" \
    --message-deduplication-id "verify-col3-$row"
done

# Inspect delivery order in the consumer log group
aws logs filter-log-events \
  --log-group-name /aws/lambda/tile-consumer \
  --filter-pattern '"col=3"' \
  --query 'events[].message' --output text

Expected output — rows appear in ascending order, never interleaved:

code 
Processed S2A_2025 col=3 row=0 revision=1
Processed S2A_2025 col=3 row=1 revision=1
Processed S2A_2025 col=3 row=2 revision=1

If you see row=2 before row=1, the queue is almost certainly a standard (non-FIFO) queue — confirm the queue name ends in .fifo and that MessageGroupId was set on every send.

Gotchas and Edge Cases

  • A single hot MessageGroupId serializes everything. If you key every tile in a scene to one group id, FIFO processes them one at a time and you lose all parallelism. Key by the finest unit that still preserves the real dependency — column for seams, pyramid column for overviews — so unrelated tiles fan out.
  • Do not combine explicit MessageDeduplicationId with ContentBasedDeduplication. The 5-minute deduplication window silently drops any message whose id (or content hash) it has seen. Two revisions with an identical body but no revision bump in the id will collapse into one — always fold the revision into the dedup id.
  • Ordering does not survive a poorly sized retry budget. When a tile exhausts maxReceiveCount and lands in a FIFO dead-letter queue, the rest of its group keeps flowing — the seam it should have written is simply missing. Treat a DLQ arrival for an ordered group as a stop-the-column signal, not a background cleanup task.
  • High-throughput mode changes the guarantee you get. It preserves order per group but not across the queue, and it is not available in every region. If a downstream reduce step assumes global queue order, enabling high-throughput mode will silently break it; confirm your only ordering assumption is per-group before switching it on.

Frequently Asked Questions

How many messages per second can an SQS FIFO queue handle for tile jobs?

A standard FIFO queue supports up to 300 send, receive, or delete operations per second, or 3,000 messages per second with batches of 10. Enabling high-throughput mode raises this to roughly 9,000 messages per second per queue in supported regions, at the cost of guaranteeing order only within each MessageGroupId rather than across the whole queue.

What should I use as the MessageGroupId for tile processing?

Use the smallest unit that carries a true ordering dependency: a tile column index for column-major mosaics, a mosaic strip id for seam blending, or a scene id for incremental catalog writes. Independent tiles should get distinct group ids so they process in parallel.

What is the GCP Pub/Sub equivalent of an SQS FIFO queue?

Pub/Sub offers ordered delivery via ordering keys. You publish messages with the same ordering key, enable message ordering on the subscription, and Pub/Sub delivers same-key messages in publish order. That is the direct analog of an SQS MessageGroupId, with no fixed message-per-second cap but a 1 MB/s sustained limit per key.

Back to SQS and Pub/Sub Queue Routing Strategies