Vector Tile Pipeline with Cloud Run and Pub/Sub

A production vector tile pipeline on GCP watches a GCS bucket for new GeoJSON or GeoParquet, delivers the object-finalize event through Pub/Sub to a Cloud Run service, runs tippecanoe to build a Mapbox Vector Tile pyramid, and uploads the .pbf tiles to a CDN-backed bucket — all inside a single 60-minute request budget. The design decision that makes this work is running the tiler on Cloud Run rather than Cloud Functions: tippecanoe is a compiled C++ binary that needs a container you control, a long timeout, and generous CPU, and Cloud Run gives all three where a source-deployed function fights you at every step. This recipe is one of the serverless geospatial pipeline recipes and leans on the packaging and queue-routing patterns documented elsewhere on this site.

Why a Cloud Run Tile Pipeline Matters for Geospatial Workloads

Web maps do not render raw GeoJSON. A 400 MB nationwide parcels file will lock a browser solid; the same data served as pre-generated Mapbox Vector Tiles streams only the ~30 KB of geometry visible in the current viewport at the current zoom. Generating that tile pyramid is tippecanoe’s entire job — it clips, simplifies, drops, and coalesces features per zoom level so each .pbf tile stays small while the map stays faithful. The catch is that tippecanoe is a native binary with no serverless-native runtime, and a continent-scale build can run for tens of minutes and consume several gigabytes of scratch disk.

That profile rules out most of the obvious serverless targets. A source-deployed function assumes your code is Python, Node, or Go with a requirements.txt; a standalone C++ tiler does not fit that mold without contortions. Cloud Run inverts the contract — you hand it a container image and it runs your HTTP server inside it, so bundling tippecanoe, its libsqlite3 dependency, and a thin request handler is a normal Dockerfile. The multi-stage Dockerfile for GDAL on Cloud Run pattern applies almost verbatim here: compile the tiler in a builder stage, copy only the binary and its shared objects into a slim runtime stage.

The event backbone is Pub/Sub queue routing. A GCS object-finalize notification publishes to a topic; a push subscription authenticates to the Cloud Run URL and delivers the event as an HTTP POST. This decouples upload rate from tiling throughput and gives you retries, dead-letter routing, and back-pressure for free — the same reasons you would reach for S3 and GCS event triggers on any ingestion path. The heavy lifting — the exact tippecanoe invocation, flag selection, and upload loop — is covered in depth in generating MVT tiles with tippecanoe in Cloud Run.

Platform-by-Platform Limits

The pipeline is GCP-native, but the same shape can be rebuilt on AWS or Azure with different compute and queue primitives. What changes is the timeout ceiling and how you ship the tippecanoe binary. The table gives the exact quotas that decide whether a given feature set fits.

The practical reading: only GCP Cloud Run and AWS Fargate comfortably host a multi-continent tile build. AWS Lambda works when your feature set tiles in under 15 minutes — beyond that you must chunk the input or fall back to Fargate, exactly as the timeout ceiling comparison for geospatial jobs lays out. Azure Functions Consumption is the wrong tool for anything but small extracts; Container Apps is the Azure equivalent of Cloud Run.

Step-by-Step Implementation

Step 1: Create the buckets and wire GCS notifications to Pub/Sub

Two buckets: one private source bucket for uploads, one public tiles bucket fronted by Cloud CDN. GCS publishes an OBJECT_FINALIZE notification to a Pub/Sub topic whenever a new object lands under the watched prefix.

# Source and tiles buckets
gsutil mb -l us-central1 gs://geo-vt-source
gsutil mb -l us-central1 gs://geo-vt-tiles

# Topic that receives object-finalize events
gcloud pubsub topics create vt-source-uploads

# Notify only on new GeoJSON/GeoParquet under the incoming/ prefix
gsutil notification create \
  -t vt-source-uploads \
  -f json \
  -e OBJECT_FINALIZE \
  -p incoming/ \
  gs://geo-vt-source

Restricting the notification to the incoming/ prefix keeps the tiler from re-triggering on any bookkeeping objects you write elsewhere in the bucket — a cheap first line of idempotency that complements the deduplication of event notifications you should still enforce in the handler.

Step 2: Deploy the Cloud Run tiler service

Build and deploy the container that holds tippecanoe and the request handler. The full Dockerfile is in the tippecanoe in Cloud Run guide; here is the deploy step that matters for the pipeline.

gcloud run deploy vt-tiler \
  --image us-central1-docker.pkg.dev/PROJECT/geo/vt-tiler:1.4.0 \
  --region us-central1 \
  --no-allow-unauthenticated \
  --memory 8Gi \
  --cpu 4 \
  --timeout 3600 \
  --concurrency 1 \
  --max-instances 20 \
  --set-env-vars TILES_BUCKET=geo-vt-tiles,TMP_DIR=/tmp

The load-bearing flags: --timeout 3600 claims the full 60-minute Cloud Run budget; --concurrency 1 guarantees one tile build per instance so a memory-hungry tippecanoe run never shares an instance with another; and --cpu 4 gives the tiler real parallelism, since tippecanoe scales across cores during feature reading.

Step 3: Create the Pub/Sub push subscription with a matched ack deadline

The push subscription authenticates to Cloud Run with a service-account OIDC token and must give the run enough time to finish before redelivering.

# Service account Pub/Sub uses to invoke Cloud Run
gcloud iam service-accounts create vt-pubsub-invoker

gcloud run services add-iam-policy-binding vt-tiler \
  --region us-central1 \
  --member serviceAccount:[email protected] \
  --role roles/run.invoker

# Push subscription; ack deadline is capped at 600s so we rely on the
# Cloud Run timeout plus a dead-letter topic for anything longer.
gcloud pubsub subscriptions create vt-tiler-push \
  --topic vt-source-uploads \
  --push-endpoint https://vt-tiler-xxxxx-uc.a.run.app/ \
  --push-auth-service-account [email protected] \
  --ack-deadline 600 \
  --dead-letter-topic vt-tiler-dead-letter \
  --max-delivery-attempts 5

Pub/Sub caps the acknowledgement deadline at 600 seconds, which is shorter than a worst-case tile build. The way to reconcile this is the modAckDeadline behavior — Pub/Sub keeps extending the lease while the push request is still open — combined with a dead-letter topic so a genuinely stuck message lands somewhere visible rather than looping forever. Scoping the invoker role tightly follows the same least-privilege discipline as IAM security boundaries for cloud GIS.

Step 4: Parse the push envelope and run the tiler

The handler decodes the Pub/Sub push envelope, pulls the GCS object into scratch, runs tippecanoe, and uploads the pyramid. This is the pipeline glue; the tiler internals live in the child guide.

import base64
import json
import os
import subprocess
from flask import Flask, request
from google.cloud import storage

app = Flask(__name__)
storage_client = storage.Client()
TILES_BUCKET = os.environ["TILES_BUCKET"]
TMP = os.environ.get("TMP_DIR", "/tmp")

@app.post("/")
def handle_push():
    envelope = request.get_json(force=True, silent=True)
    if not envelope or "message" not in envelope:
        # Malformed envelope: 400 so Pub/Sub does not retry a poison message
        return ("bad pub/sub envelope", 400)

    # GCS notification payload is base64-encoded JSON in message.data
    payload = json.loads(base64.b64decode(envelope["message"]["data"]))
    bucket_name, object_name = payload["bucket"], payload["name"]
    generation = payload.get("generation", "0")

    # Idempotency key derived from object + generation; skip if tiles exist
    layer = os.path.splitext(os.path.basename(object_name))[0]
    marker = storage_client.bucket(TILES_BUCKET).blob(f"{layer}/.done-{generation}")
    if marker.exists():
        return ("", 204)  # already tiled this exact object version

    src_path = os.path.join(TMP, os.path.basename(object_name))
    storage_client.bucket(bucket_name).blob(object_name).download_to_filename(src_path)

    # generate_tiles() wraps the tippecanoe invocation + upload loop
    generate_tiles(src_path, layer)

    marker.upload_from_string(b"")  # write completion marker last
    return ("", 204)  # ack only after tiles are durable

Returning 204 only after the completion marker is written is the whole contract: if the process is killed mid-build, no marker exists, Pub/Sub redelivers, and the run restarts cleanly. Returning 400 on a malformed envelope short-circuits poison messages so they exhaust max-delivery-attempts and drain to the dead-letter topic instead of hammering the service.

Step 5: Serve tiles from the CDN-backed bucket

tippecanoe output uploaded as an unpacked z/x/y.pbf directory needs two headers on every tile: Content-Type: application/x-protobuf and Content-Encoding: gzip (tippecanoe gzips each tile by default). Set them at upload time and enable Cloud CDN on the bucket’s backend.

gsutil -m -h "Content-Type:application/x-protobuf" \
  -h "Content-Encoding:gzip" \
  -h "Cache-Control:public,max-age=86400" \
  cp -r "${TMP}/tiles/parcels" gs://geo-vt-tiles/parcels

gsutil iam ch allUsers:objectViewer gs://geo-vt-tiles

A missing Content-Encoding: gzip header is the single most common reason a MapLibre or Mapbox GL client renders a blank map from otherwise-correct tiles — the browser hands tippecanoe’s gzipped bytes to the vector-tile parser without inflating them.

Measurement and Verification

Confirm the pipeline end to end by uploading a source file and watching a tile appear on the CDN.

# Trigger the pipeline
gsutil cp parcels.geojson gs://geo-vt-source/incoming/parcels.geojson

# Tail the Cloud Run logs for the tippecanoe run
gcloud run services logs read vt-tiler --region us-central1 --limit 40

# Fetch a mid-zoom tile once the run finishes; -I shows headers
curl -sI "https://cdn.example.com/parcels/10/301/389.pbf"

Expected header output for a correctly served tile:

HTTP/2 200
content-type: application/x-protobuf
content-encoding: gzip
cache-control: public,max-age=86400

Decode a tile locally to confirm it holds real geometry, not an empty layer:

# vt2geojson round-trips a .pbf tile back to GeoJSON features
curl -s "https://cdn.example.com/parcels/10/301/389.pbf" \
  | vt2geojson --z 10 --x 301 --y 389 - \
  | python3 -c "import sys,json; print(len(json.load(sys.stdin)['features']), 'features')"
# Expected: a non-zero feature count, e.g. 1423 features

Failure Modes and Debugging

Pub/Sub redelivers and a second tippecanoe starts before the first finishes: the acknowledgement deadline expired. Verify the push subscription relies on lease extension and that your handler is not buffering the entire request before starting work. The completion-marker idempotency check in Step 4 makes the duplicate run a no-op, but you still want --concurrency 1 so the duplicate does not double memory pressure on one instance.

tippecanoe: Killed with exit code 137: the container hit its memory ceiling. Cloud Run tmpfs counts against total memory, so a 6 GB intermediate .mbtiles on an 8 GB instance leaves little headroom for the tiler’s own buffers. Raise --memory, write intermediate output with -o /tmp/out.mbtiles and stream it out, or drop features earlier with --drop-densest-as-needed. The ephemeral storage comparison across serverless platforms explains why GCP scratch behaves differently from Lambda’s disk-backed /tmp here.

Blank map despite HTTP 200 tiles: missing Content-Encoding: gzip header, or a source/style layer-name mismatch. tippecanoe -l parcels forces the layer name; the client style must reference the same source-layer. Re-check with the vt2geojson decode above.

403 on the push endpoint: the invoker service account lacks roles/run.invoker, or the audience in the OIDC token does not match the Cloud Run URL. Re-run the add-iam-policy-binding from Step 3 and confirm the subscription’s --push-auth-service-account.

Messages pile up in the dead-letter topic: a poison source file — malformed GeoJSON, wrong CRS, or a zero-feature upload — fails every attempt. Route the dead-letter topic to a second Cloud Run service or a notification channel, following the dead-letter queue pattern for failed vector jobs.

Cost and Scaling Considerations

Cloud Run bills per vCPU-second and GiB-second only while a request is in flight, so an idle pipeline costs nothing beyond bucket storage and CDN egress. A --cpu 4 --memory 8Gi instance running a 12-minute continental parcels build costs on the order of a few cents; the tiles it produces then serve from Cloud CDN at cache-hit rates above 95% for any popular basemap, which is where the real savings sit compared with an always-on tile server.

Two levers dominate the cost/latency trade. First, --concurrency 1 means one instance per build — correct for large jobs, but wasteful for a flood of tiny extracts, where raising concurrency and lowering per-instance memory packs more work per instance. Second, --max-instances bounds fan-out so a burst of uploads cannot spin up hundreds of 4-CPU instances at once; pair it with the Pub/Sub subscription’s flow control so back-pressure builds in the queue rather than in your billing. For inputs that reliably exceed the 60-minute ceiling, stop scaling Cloud Run and pre-split the source, applying the same reasoning as chunking raster jobs to fit the 15-minute Lambda ceiling — partition by region, tile each partition, and merge the pyramids.

Frequently Asked Questions

Why use Cloud Run instead of Cloud Functions for tippecanoe?

tippecanoe is a compiled C++ binary, not a Python or Node package, so it has to ship inside a container image you build and control. Cloud Run takes a container directly, offers the full 60-minute request timeout for large feature sets, and lets you dial CPU up to 4 vCPU or more. Cloud Functions 2nd gen runs on the same Cloud Run substrate but is oriented around source-based deploys with a language runtime, which makes bundling a standalone binary and its shared libraries awkward and fragile.

What is the maximum request timeout for Cloud Run tile generation?

Cloud Run allows a request timeout up to 60 minutes (3,600 seconds), set with --timeout 3600. The Pub/Sub push subscription must lease the message for at least as long, which it does by extending the acknowledgement deadline while the push request stays open. Anything that genuinely needs more than 60 minutes should be split into per-region partitions before tiling.

How would I run the same vector tile pipeline on AWS or Azure?

On AWS, ship tippecanoe in a Lambda container image for feature sets that tile inside the 15-minute ceiling, or trigger a Fargate task from an S3 event through SQS for longer builds. On Azure, use Container Apps with a KEDA scaler bound to a Storage Queue, because Azure Functions on the Consumption plan is capped at 10 minutes and 1.5 GB of memory — too tight for continental tiling.

Do I need MBTiles or can tippecanoe write loose tiles directly?

tippecanoe writes a single .mbtiles SQLite file by default, which is convenient to move as one object. To serve straight from GCS you unpack it to a z/x/y.pbf directory with tile-join or mb-util. For CDN-fronted static serving, the unpacked directory is simpler; for a dynamic tile server, ship the .mbtiles and read from it.

Related

• Generating MVT Tiles with Tippecanoe in Cloud Run — the Dockerfile, exact tippecanoe flags, and upload loop behind Step 2 and Step 4
• Multi-Stage Dockerfile for GDAL on Cloud Run — the builder/runtime split that keeps the tiler image slim
• SQS and Pub/Sub Queue Routing Strategies — push subscriptions, back-pressure, and dead-letter routing for the event backbone
• Native Library Compilation for Serverless — building tippecanoe and its dependencies against the right OS ABI
• Timeout Ceiling Comparison for Geospatial Jobs — why Cloud Run’s 60 minutes matters versus Lambda’s 15 and Azure’s 10

Back to Serverless Geospatial Pipeline Recipes