Comparing SQS, Pub/Sub and Service Bus for Spatial Jobs

For a 900-second tiling job, SQS is the only broker whose redelivery window already exceeds the compute ceiling — 12 hours against a 15-minute function — while Pub/Sub caps the ack deadline at 600 seconds and Azure Service Bus caps the message lock at 5 minutes, so both need a renewal loop. On payload size the ranking inverts: SQS and Service Bus Standard stop at 256 KB, Pub/Sub carries 10 MB and Service Bus Premium 100 MB, which is the difference between publishing a dissolved coastline geometry directly and having to front it with a claim-check. Pick on whichever of the two constraints your workload actually hits, because no broker wins both.

Context

The queue routing layer is usually chosen for its filtering model — whether routing logic lives in the broker or in your own dispatcher. That is the right lens for a fan-out of small envelopes. It is the wrong lens for the two workloads that dominate a spatial pipeline: a long-running raster job, and a vector job whose payload is the geometry itself.

Both of those press on limits the routing discussion never reaches. A tiling job that reprojects and resamples a Sentinel-2 window runs for minutes, so the question is whether the broker will keep the message invisible for the whole run without help. A dissolve or a spatial join over administrative boundaries produces a geometry that is the interesting part of the message, so the question is whether the broker will carry it at all. These two constraints eliminate options faster than filtering syntax does, and they point in opposite directions.

A 15-minute tiling job measured against each broker's redelivery windowFour meters showing a 900-second tiling job against the limits it must fit inside: the SQS visibility timeout maximum of 43,200 seconds, the Pub/Sub ack deadline maximum of 600 seconds which the job exceeds, the Azure Service Bus message lock maximum of 300 seconds which the job exceeds threefold, and the Azure Functions Consumption execution ceiling of 600 seconds which the job also exceeds.One 900-second tiling job against four hard windowsSQS visibility timeoutno renewal needed900 s of 12 hPub/Sub ack deadlineneeds modify_ack_deadline extension900 s over 600 sService Bus message lockneeds three lock renewals900 s over 300 sAzure Functions Consumptionthe job cannot complete here900 s over 600 sA short redelivery window is survivable on its own; it turns fatal when it pairs with a short execution ceiling, which is the Azure Consumptionrow.
Only SQS holds the message for longer than the compute can possibly run. The other two need a renewal loop, and on Azure Functions Consumption the job cannot finish at all.

Read the meters as three different amounts of extra machinery. On SQS, a 900-second job sits at 2 % of the 12-hour ceiling and the only sizing question is the multiple of the function timeout, which sizing SQS visibility timeout for long-running raster jobs answers. On Pub/Sub the job outlives the 600-second maximum ack deadline outright, so the subscriber client must extend the lease while it works. On Service Bus the 5-minute lock is exceeded three times over, so a renewal thread runs for the whole job — and on the Consumption plan the Azure Function would be killed at 10 minutes anyway, which means the workload does not belong there at all and needs Premium, Container Apps, or a container job.

That last point is the one worth generalising. A short redelivery window is not fatal by itself; it becomes fatal when it pairs with a short execution ceiling. GCP is the clearest case of the mismatch running the other way: Cloud Functions 2nd gen allows 60 minutes of execution but Pub/Sub’s ack deadline maxes out at 600 seconds, so the platform will happily run a 45-minute job whose message it will not hold.

Prerequisites

  • Runtime: Python 3.11+ on AWS Lambda, Python 3.12 on Cloud Functions 2nd gen or Cloud Run, and .NET or Python on Azure Container Apps — not Azure Functions Consumption, whose 10-minute ceiling and 1,536 MB memory rule out the workload discussed here.
  • Dependencies: boto3>=1.34.0, google-cloud-pubsub>=2.21.0, azure-servicebus>=7.11, and shapely>=2.0.0 for the geometry serialisation in the publisher.
  • A measured p99 job duration and a measured p99 payload size. Both decisions below are made from the tail, not the mean; a broker chosen from an average payload will fail on the one national boundary set in the corpus.
  • Object storage in the same region as the broker. The claim-check adds a PUT and a GET to every job, and a cross-region round trip on each of those is a far worse tax than the message size ever was.
  • IAM: queue send and receive on the broker of choice, plus s3:PutObject, s3:GetObject and s3:DeleteObject scoped to the payload prefix only. The payload prefix should be separate from the data prefix so a lifecycle rule can expire one without touching the other.

The Size Ceiling and Where Geometry Crosses It

Spatial payloads are unusual among queue messages in that the payload is sometimes the point. A job envelope carrying a source_uri, a CRS and a window is under two kilobytes and no broker cares. A job envelope carrying the clip geometry is a different object entirely.

Maximum message size across SQS, Service Bus and Pub/SubFour inline message ceilings in megabytes: AWS SQS at 0.25 MB, Azure Service Bus Standard at 0.25 MB, GCP Pub/Sub at 10 MB, and Azure Service Bus Premium at 100 MB. A dissolved national coastline of 240,000 vertices serialises to about 5.8 MB of GeoJSON, which clears the first two ceilings by more than twenty times.Inline message ceilings against a 5.8 MB coastline geometryAWS SQS256 KBAzure Service Bus Standard256 KBGCP Pub/Sub10 MBAzure Service Bus Premium100 MB0MB per messageA 240,000-vertex coastline is about 5.8 MB as GeoJSON and roughly half that as WKB — still twelve times the SQS ceiling.
The 256 KB brokers are barely visible at this scale, which is the point: a geometry payload does not creep past their ceiling, it clears it by an order of magnitude.

A dissolved national coastline with 240,000 vertices serialises to roughly 5.8 MB of GeoJSON — comfortably inside Pub/Sub’s 10 MB and Service Bus Premium’s 100 MB, and more than twenty times over the 256 KB that SQS and Service Bus Standard allow. Encoding it as WKB rather than GeoJSON roughly halves it and still does not fit. This is where the claim-check pattern earns its place: write the payload to object storage, publish a pointer, and let the consumer resolve it.

The pattern is well known; what is usually left out is the threshold. Below the ceiling, a claim-check costs more than the message it replaces. SQS bills per 64 KB request unit, so a 250 KB message is four units — about $1.60 per million jobs at $0.40 per million requests. Replacing it with a 1.1 KB pointer is one unit, $0.40, but adds an S3 PUT at $5.00 per million and a GET at $0.40 per million: $5.80 against $1.60. The claim-check is a correctness mechanism for payloads that do not fit, not an optimisation for payloads that do.

Implementation

One publisher, three brokers, and a single threshold that decides whether the payload travels inline or by reference:

python 
import json
import os
import uuid

import boto3
from shapely import to_wkb
from shapely.geometry.base import BaseGeometry

s3 = boto3.client("s3")
sqs = boto3.client("sqs")

PAYLOAD_BUCKET = os.environ["PAYLOAD_BUCKET"]
PAYLOAD_PREFIX = "queue-payloads/"          # separate prefix so a lifecycle rule
                                            # can expire payloads and nothing else

# Per-broker inline ceilings. Publish below these and the message is self-contained;
# above them the broker rejects it and the claim-check is the only option.
INLINE_LIMIT = {
    "sqs": 256 * 1024,          # also Azure Service Bus Standard
    "pubsub": 10 * 1024 * 1024,
    "servicebus_premium": 100 * 1024 * 1024,
}


def build_envelope(job_id: str, geom: BaseGeometry, broker: str) -> dict:
    """Return an envelope that fits the broker, inlining the geometry if it can."""
    body = to_wkb(geom, hex=True)           # roughly half the size of GeoJSON
    envelope = {"job_id": job_id, "crs": "EPSG:4326", "processing_mode": "vector"}

    # Compare the serialised envelope, not the geometry alone — attributes,
    # message metadata and base64 expansion all count against the ceiling.
    candidate = {**envelope, "geometry_wkb": body}
    if len(json.dumps(candidate).encode()) < INLINE_LIMIT[broker]:
        return candidate

    key = f"{PAYLOAD_PREFIX}{job_id}/{uuid.uuid4().hex}.wkb"
    s3.put_object(Bucket=PAYLOAD_BUCKET, Key=key, Body=bytes.fromhex(body))
    return {
        **envelope,
        # The pointer, plus enough metadata that a router can make decisions
        # without paying the GET. Bounds and vertex count are the two a spatial
        # dispatcher actually needs.
        "payload_uri": f"s3://{PAYLOAD_BUCKET}/{key}",
        "payload_bytes": len(body) // 2,
        "bounds": list(geom.bounds),
        "vertex_count": sum(len(g.exterior.coords) for g in geom.geoms),
    }


def publish_sqs(queue_url: str, envelope: dict) -> None:
    """SQS: 256 KB ceiling, but a 12-hour visibility window needing no renewal."""
    sqs.send_message(QueueUrl=queue_url, MessageBody=json.dumps(envelope))

The two non-AWS consumers need the lease machinery the gauge implied. On Pub/Sub the streaming pull client extends the deadline for you, but only up to a bound you set — leave it at the default and a 900-second job loses its lease partway:

python 
from google.cloud import pubsub_v1

# max_lease_duration is the ceiling on automatic modify_ack_deadline extension.
# The ack deadline itself may never exceed 600 s, so a 900-second tiling job is
# only safe because the client keeps re-extending it underneath.
flow = pubsub_v1.types.FlowControl(max_messages=4, max_lease_duration=1800)
future = pubsub_v1.SubscriberClient().subscribe(SUBSCRIPTION, callback, flow_control=flow)
python 
from azure.servicebus import AutoLockRenewer

# Service Bus caps the lock at 5 minutes, so a 900-second job needs three
# renewals. AutoLockRenewer runs them on a background thread, and
# max_lock_renewal_duration must exceed the p99 job duration — otherwise
# renewal stops mid-job and the message is redelivered to a second consumer.
renewer = AutoLockRenewer(max_lock_renewal_duration=1800)
renewer.register(receiver, message, max_lock_renewal_duration=1800)

Both renewal mechanisms have the property that makes them worth flagging in review: they fail by stopping, not by raising. A job whose lease lapses keeps running to completion and writes its output, while a second consumer starts the same work — the duplicate-processing failure that a mosaic write cannot survive.

Claim-check publish and consume path for an oversized geometry payloadFive steps for moving a geometry payload that exceeds the broker's message ceiling: serialise the geometry to WKB, write it to a dedicated payload prefix in object storage, publish a 1.1 kilobyte pointer envelope carrying bounds and vertex count, resolve the pointer in the consumer with a single get, and expire the payload with a lifecycle rule that outlives the retry budget.The claim-check, and the step everyone omits1Serialise to WKBAbout half the bytes of GeoJSON, and no base64 expansionshapely.to_wkb2Write to the payload prefixA separate prefix so a lifecycle rule can expire it alones3:PutObject3Publish the pointer envelopeBounds and vertex count travel inline so the router never pays a get1.1 KB4Resolve in the consumerOne get, streamed straight into the geometry parsers3:GetObject5Expire the payloadMust outlive the last redelivery, so 21 days for a 14-day queueExpiration Days=21
Steps one to four are the pattern as usually described. Step five is what keeps it from accumulating one orphaned object per job forever.

The lifecycle rule in the last step is not optional bookkeeping. A claim-check without expiry accumulates one object per job forever, and the objects are invisible in the sense that nothing references them once the message is deleted. Set the expiry to comfortably exceed the retry budget — a queue retaining messages for 14 days with a maxReceiveCount of 4 needs the payload to outlive the last redelivery, so 21 days rather than 7.

Verification

Confirm the envelope crosses over at the right size and that the pointer resolves. Both halves matter — a claim-check that silently inlines a 400 KB payload fails at publish time, and one that always externalises costs money for nothing:

python 
from shapely.geometry import MultiPolygon, Polygon

small = MultiPolygon([Polygon([(0, 0), (1, 0), (1, 1), (0, 0)])])
large = load_coastline("data/coastline_240k_vertices.gpkg")   # ~240,000 vertices

for name, geom in (("small", small), ("large", large)):
    env = build_envelope(f"job-{name}", geom, broker="sqs")
    print(name, "->", "inline" if "geometry_wkb" in env else "claim-check",
          "|", len(json.dumps(env).encode()), "bytes on the wire")

Expected output — the small geometry stays inline, the large one becomes a pointer envelope of about a kilobyte, and neither is anywhere near the 256 KB ceiling on the wire:

code 
small -> inline | 214 bytes on the wire
large -> claim-check | 1104 bytes on the wire

Then confirm the lease behaviour under load rather than trusting the configuration. On SQS, watch ApproximateReceiveCount on a completed job: anything above 1 for a message you sent once means the window lapsed. On Pub/Sub, alert on subscription/oldest_unacked_message_age exceeding the p99 job duration. On Service Bus, a MessageLockLostError in the consumer log is the same signal arriving as an exception instead of a metric.

Gotchas and Edge Cases

  • Base64 expansion eats a quarter of the ceiling. Pub/Sub and Service Bus transport bytes, but any JSON envelope that embeds binary geometry has to encode it, and base64 adds about 33 %. A 200 KB WKB geometry is a 267 KB message on SQS — over the limit while the geometry itself was under it. Measure the serialised envelope, as the publisher above does, not the geometry.
  • A claim-check breaks broker-side filtering. Pub/Sub subscription filters and Service Bus SQL rules evaluate message attributes, not the object the pointer names. Any field the router needs — CRS, priority, bounds — has to be lifted out of the payload and into the envelope, or the routing decision now requires a GET before it can be made.
  • Service Bus Premium’s 100 MB is per-namespace pricing, not per-message pricing. Premium is a fixed hourly charge per messaging unit regardless of volume, so choosing it purely to avoid a claim-check on a low-volume queue is an expensive way to skip fifteen lines of code. It earns its keep when you are also using sessions or long locks.
  • Deduplication windows and long jobs interact badly. SQS FIFO deduplicates on a 5-minute interval, which is shorter than a single 900-second tiling job. A retry issued after the job fails at minute 12 is outside the window and will be accepted as a new message — correct behaviour, but not what “deduplicated queue” leads people to expect. The ordering and dedup mechanics are covered in guaranteeing order with SQS FIFO for sequential tile jobs.
  • Dead-lettering a claim-check message strands the payload. The message moves to the dead-letter queue and the pointer still resolves, right up until the lifecycle rule expires the object — after which the dead-lettered job cannot be replayed at all. Either lengthen the expiry to cover triage or copy the payload when the message is dead-lettered, as the dead-letter queue setup triage workflow allows for.

Frequently Asked Questions

Which broker handles a 15-minute tiling job with the least work?

SQS, because a 12-hour maximum visibility timeout is the only redelivery window that already exceeds the compute ceiling in front of it. A 900-second job needs no lease extension there at all. Pub/Sub caps the ack deadline at 600 seconds and Service Bus caps the lock at 5 minutes, so both require a background renewal that runs for the whole job — one more component whose failure mode is silence.

At what payload size does the claim-check pattern start paying for itself?

At the broker’s ceiling, and not before. On SQS a 250 KB message bills as four 64 KB request units, about $1.60 per million jobs; replacing it with a pointer is one unit at $0.40 plus an S3 PUT at $5.00 and a GET at $0.40 per million, so $5.80 against $1.60. Use the claim-check because the payload does not fit, not because you expect it to be cheaper.

Can a spatial job payload really exceed 256 KB?

Routinely. A dissolved national coastline with 240,000 vertices is roughly 5.8 MB of GeoJSON and still over a megabyte as WKB. Tiling manifests get there too: a per-scene job list for a 22 × 22 window grid crosses 256 KB as soon as each window’s bounds and band indices are inlined rather than derived from the scene id.

Does Pub/Sub’s 10 MB limit make the claim-check unnecessary on GCP?

It moves the threshold rather than removing it, and it introduces a cost the other brokers do not have. Pub/Sub bills per gigabyte of message data in both directions, so a 6 MB geometry published once and delivered to three subscriptions moves 24 GB per thousand jobs. At that size a pointer plus a Cloud Storage read in the same region is both cheaper and faster, even though the message would have fitted.

Back to SQS and Pub/Sub Queue Routing Strategies