Moving a 60-Minute Mosaic to Cloud Run Jobs

Deploy the mosaic as a Cloud Run job with --tasks 48 --parallelism 12 --task-timeout 3600s --max-retries 3 --memory 8Gi --cpu 4, and replace the Lambda event payload with CLOUD_RUN_TASK_INDEX — each task derives its own tile stripe from its index rather than receiving one. A 60-minute continental DEM mosaic that AWS Lambda cannot run at all, because 15 minutes is a hard ceiling, becomes 48 tasks of roughly 6 minutes each finishing in about 25 wall-clock minutes at parallelism 12. The port is mostly deletion: no orchestrator, no chunk-size arithmetic, no state machine.

Why the Job Cannot Stay on Lambda

The Timeout Ceiling Comparison for Long-Running Geospatial Jobs lays out the three ceilings: AWS Lambda stops at 15 minutes, GCP Cloud Functions 2nd gen and Cloud Run stop at 60, and Azure Functions on the Consumption plan stops at 10. A mosaic that builds a seamless hillshade over a 2,400 × 1,800 km footprint from 1,140 source tiles is a 55-minute job on 4 vCPU — it does not fit two of those three, and the usual answer is to chunk it until it does.

Chunking is the right answer often enough that Chunking Raster Jobs to Fit the 15-Minute Lambda Ceiling is worth reading before this page. But it stops working when the job has a stage that cannot be split. A mosaic’s overview pyramid is the classic case: you can tile the pixel-copy phase across a hundred workers, and then one worker must read the assembled result end-to-end to build the overviews, and that single pass is 22 minutes. No amount of fan-out shortens it. At that point the honest move is to a runtime whose ceiling is above the irreducible stage, not to a more elaborate orchestration of one that is not.

Continental hillshade mosaic runtime compared with platform timeout ceilingsHorizontal bars comparing the serial runtime of a continental hillshade mosaic at 55 minutes with the Azure Functions Consumption ceiling of 10 minutes, the AWS Lambda ceiling of 15 minutes, the Cloud Run job task ceiling of 60 minutes, the wall clock of 25 minutes once split across 48 tasks at parallelism 12, and the longest individual task at 6 minutes.A 55-minute mosaic against the three timeout ceilingsCloud Run job task ceiling60 minMosaic serial runtime, 4 vCPU55 minWall clock, 48 tasks at parallelism 1225 minAWS Lambda ceiling15 minAzure Consumption ceiling10 minLongest single task6 min060 minLambda also caps at 10,240 MB and roughly 6 vCPU; a Cloud Run job task can take 32 GiB and 8 vCPU, which is the second reason theirreducible overview pass fits there.
The job clears only one ceiling unsplit. Splitting it 48 ways puts every task an order of magnitude inside the limit and finishes in less than half the serial time.

Cloud Run jobs are the closest thing to Lambda that clears the bar. A job task gets up to 60 minutes, up to 32 GiB of memory and up to 8 vCPU — against Lambda’s 15 minutes and 10,240 MB (roughly 6 vCPU at the ceiling). It runs the same container image you would have used for a Lambda container deployment, so the packaging work from Multi-Stage Dockerfile for GDAL on Cloud Run carries over unchanged. And it has built-in task parallelism, which removes the Step Functions Map state that the Lambda version needed.

Prerequisites

  • Runtime: a container image on Artifact Registry built FROM ghcr.io/osgeo/gdal:ubuntu-small-3.8.4 or equivalent, with Python 3.11 and rasterio 1.3.9+. Cloud Run jobs do not use buildpacks for this — build and push the image yourself.
  • gcloud 465.0.0 or later, which is where --task-timeout accepts a bare seconds value with the s suffix.
  • IAM: the job’s runtime service account needs roles/storage.objectViewer on the source bucket, roles/storage.objectCreator on the output bucket, and roles/logging.logWriter. The principal executing the job needs roles/run.invoker and roles/run.developer. Grant them on the bucket, not the project — the reasoning is in IAM Security Boundaries for Cloud GIS.
  • Environment variables set on the job, not baked into the image:
    • GDAL_DATA=/usr/share/gdal and PROJ_LIB=/usr/share/proj — the paths inside the OSGeo base image, which differ from the /opt/share/... layout a Lambda layer uses.
    • LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu
    • GDAL_CACHEMAX=2048 — with 8 GiB allocated and 4 vCPU, a 2 GiB block cache is the right quarter-share.
    • GDAL_NUM_THREADS=4 — match the --cpu value exactly; ALL_CPUS sees the host’s core count, not your allocation.
    • GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR, CPL_VSIL_CURL_CACHE_SIZE=134217728, VSI_CACHE=TRUE
    • MOSAIC_MANIFEST=gs://dem-work/manifests/eu-hillshade-2026.json — the tile list, written once by a planning step.
  • A manifest, not a payload. Tasks cannot be given arguments individually. Write the full ordered tile list to object storage first and let every task read the same manifest.

What Changes in the Code

Three things, and only three.

The trigger becomes an index. A Lambda handler receives event["tile_id"]. A Cloud Run job task receives nothing; it reads CLOUD_RUN_TASK_INDEX (zero-based) and CLOUD_RUN_TASK_COUNT and slices the manifest itself. This is a better arrangement than it first looks — the assignment is deterministic, so a retried task provably gets the same work.

The entry point becomes __main__. There is no handler signature and no return value. The task’s exit code is its result: 0 is success, anything else is a failure that counts against --max-retries.

Retry stops being someone else’s problem. Lambda’s retries are configured on the event source, and failures land in a dead-letter queue as described in Implementing Dead-Letter Queues for Failed Vector Jobs. A Cloud Run job re-runs the failed task in place with the same index and an incremented CLOUD_RUN_TASK_ATTEMPT, with no backoff and no dead-letter destination. If the task is not idempotent on its own index, a retry corrupts the output.

Cloud Run job task grid mid-executionA grid of 48 cells in twelve columns and four rows, one per job task. The twelve cells of the top row are complete, the twelve of the second row are in flight, one of which is on its second attempt after a non-zero exit, and the twenty-four cells of the bottom two rows are still queued.48 tasks, 12 in flight, one on its second attempt17--tasks 48--parallelism 1212 concurrent tasks at 4vCPU is 48 vCPU and capsthe read rate against thesource bucket.stripe committedin flightattempt 2, same indexqueuedTask 17 exited non-zero and was re-dispatched with the same index; already_done() lets it skip a stripe it had already committed.
Each cell is one CLOUD_RUN_TASK_INDEX. No payload is delivered — a task derives its contiguous stripe of the manifest from its own index and the task count.

Implementation

python 
#!/usr/bin/env python3
"""Cloud Run job task: build one stripe of a continental hillshade mosaic.

Invoked with no arguments. The task's identity comes entirely from
CLOUD_RUN_TASK_INDEX, so the same image and the same command produce
different work in each of the 48 tasks.
"""
import json
import logging
import os
import sys

import rasterio
from rasterio.merge import merge
from rasterio.session import GSSession
from google.cloud import storage

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
LOG = logging.getLogger("mosaic-task")

# --- Task identity, supplied by the Cloud Run jobs runtime ------------------
TASK_INDEX = int(os.environ["CLOUD_RUN_TASK_INDEX"])      # 0 .. count-1
TASK_COUNT = int(os.environ["CLOUD_RUN_TASK_COUNT"])      # --tasks
TASK_ATTEMPT = int(os.environ.get("CLOUD_RUN_TASK_ATTEMPT", "0"))

MANIFEST = os.environ["MOSAIC_MANIFEST"]
OUT_PREFIX = os.environ["MOSAIC_OUT_PREFIX"]

# GDAL config that must match the --cpu and --memory the job was deployed with.
GDAL_ENV = dict(
    GDAL_CACHEMAX="2048",              # MB — a quarter of the 8 GiB allocation
    GDAL_NUM_THREADS="4",              # exactly --cpu; ALL_CPUS sees the host
    GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR",
    CPL_VSIL_CURL_CACHE_SIZE="134217728",
    VSI_CACHE="TRUE",
    GDAL_TIFF_INTERNAL_MASK="YES",
)

_gcs = storage.Client()


def read_manifest(uri: str) -> list[dict]:
    bucket, _, blob = uri.removeprefix("gs://").partition("/")
    payload = _gcs.bucket(bucket).blob(blob).download_as_bytes()
    return json.loads(payload)["tiles"]


def my_stripe(tiles: list[dict]) -> list[dict]:
    """Deterministic contiguous slice for this task index.

    Contiguous rather than strided: neighbouring source tiles overlap, so a
    contiguous stripe reuses the /vsicurl/ chunk cache across merges. A strided
    assignment (tiles[i::count]) balances better but throws the cache away.
    """
    per = -(-len(tiles) // TASK_COUNT)          # ceiling division
    return tiles[TASK_INDEX * per:(TASK_INDEX + 1) * per]


def output_uri() -> str:
    # Named by index, never by attempt — a retry overwrites its own output
    # rather than creating a second partial object for the merge step to find.
    return f"{OUT_PREFIX}/stripe-{TASK_INDEX:04d}.tif"


def already_done(uri: str) -> bool:
    """Idempotency check. Cloud Run retries in place with no dead-letter queue,
    so the task itself has to recognise work it already committed."""
    bucket, _, blob = uri.removeprefix("gs://").partition("/")
    obj = _gcs.bucket(bucket).get_blob(blob)
    return obj is not None and obj.metadata is not None \
        and obj.metadata.get("mosaic-complete") == "true"


def main() -> int:
    LOG.info("task %d/%d attempt %d", TASK_INDEX, TASK_COUNT, TASK_ATTEMPT)

    out = output_uri()
    if TASK_ATTEMPT > 0 and already_done(out):
        LOG.info("stripe already committed on an earlier attempt — exiting 0")
        return 0

    tiles = my_stripe(read_manifest(MANIFEST))
    if not tiles:
        # More tasks than tiles is not an error; it is an over-provisioned job.
        LOG.info("no tiles for this index — exiting 0")
        return 0

    with rasterio.Env(session=GSSession(), **GDAL_ENV):
        srcs = [rasterio.open(t["uri"]) for t in tiles]
        try:
            data, transform = merge(srcs, resampling=rasterio.enums.Resampling.bilinear)
            profile = srcs[0].profile | {
                "height": data.shape[1], "width": data.shape[2],
                "transform": transform, "driver": "GTiff",
                "tiled": True, "blockxsize": 512, "blockysize": 512,
                "compress": "deflate", "predictor": 2, "BIGTIFF": "IF_SAFER",
            }
            # Write straight to GCS over /vsigs/ — the task has no persistent
            # disk worth using and the object is the unit of completion.
            with rasterio.open(out.replace("gs://", "/vsigs/", 1), "w", **profile) as dst:
                dst.write(data)
        finally:
            for s in srcs:
                s.close()

    # Mark completion in object metadata, after the write has closed. This is
    # what already_done() reads on a retry.
    bucket, _, blob = out.removeprefix("gs://").partition("/")
    obj = _gcs.bucket(bucket).blob(blob)
    obj.metadata = {"mosaic-complete": "true", "tiles": str(len(tiles))}
    obj.patch()

    LOG.info("stripe %d complete: %d tiles -> %s", TASK_INDEX, len(tiles), out)
    return 0


if __name__ == "__main__":
    # The exit code IS the result. There is no return value and no callback.
    sys.exit(main())

Deploy it with the parallelism and retry policy explicit:

bash 
gcloud run jobs deploy eu-hillshade-mosaic \
  --image europe-west1-docker.pkg.dev/$PROJECT/geo/mosaic:2026-08-07 \
  --region europe-west1 \
  --tasks 48 \
  --parallelism 12 \
  --task-timeout 3600s \
  --max-retries 3 \
  --memory 8Gi \
  --cpu 4 \
  --service-account mosaic-runner@$PROJECT.iam.gserviceaccount.com \
  --set-env-vars "GDAL_DATA=/usr/share/gdal,PROJ_LIB=/usr/share/proj,LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu,GDAL_CACHEMAX=2048,GDAL_NUM_THREADS=4,GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR,MOSAIC_MANIFEST=gs://dem-work/manifests/eu-hillshade-2026.json,MOSAIC_OUT_PREFIX=gs://dem-out/hillshade/2026"

gcloud run jobs execute eu-hillshade-mosaic --region europe-west1 --wait

--parallelism 12 against --tasks 48 is deliberate: twelve concurrent tasks at 4 vCPU each is 48 vCPU, which is a quota-friendly footprint, and it caps the read rate against the source bucket. Setting parallelism equal to task count finishes sooner but hammers the source with 192 concurrent range-request streams.

State machine for a single Cloud Run job taskFour states in a row: queued, running, stripe committed, and task succeeded. Forward arrows move from dispatch through merge completion to the metadata patch, and a dashed return path from stripe committed back to running carries the non-zero exit case.One task's lifecycle, and where the retry re-entersQueuedindex assignedRunning3,600 s, 8 GiB, 4 vCPUStripe committed/vsigs/ write closedTask succeededexit code 0dispatchmerge completemetadata patchednon-zero exitA failure anywhere before the metadata patch returns the task to Running. Because the object is named by index and never by attempt, the retryoverwrites its own partial write rather than leaving a second object behind.
The retry re-enters at Running with the same index and an incremented CLOUD_RUN_TASK_ATTEMPT — there is no backoff and no dead-letter destination.

Verification

Execute the job and confirm the tasks were distinct, complete, and inside the ceiling:

bash 
EXEC=$(gcloud run jobs executions list --job eu-hillshade-mosaic \
        --region europe-west1 --limit 1 --format='value(name)')

gcloud run jobs executions describe "$EXEC" --region europe-west1 \
  --format='table(status.taskCount, status.succeededCount, status.failedCount, status.retriedCount)'

# Longest task, from the structured logs — this is the number that must stay
# under the 3600 s task timeout.
gcloud logging read \
  "resource.type=cloud_run_job AND labels.\"run.googleapis.com/execution_name\"=$EXEC
   AND jsonPayload.message=~\"stripe .* complete\"" \
  --format='value(timestamp)' --limit 100 | sort | sed -n '1p;$p'

gsutil ls -l gs://dem-out/hillshade/2026/ | tail -3

Expected output for a healthy run — 48 tasks, no failures, one retry absorbed, and 48 distinct objects:

code 
TASK_COUNT  SUCCEEDED  FAILED  RETRIED
48          48         0       1

2026-08-07T09:14:22Z
2026-08-07T09:39:51Z

  1207431680  2026-08-07T09:39:44Z  gs://dem-out/hillshade/2026/stripe-0047.tif
TOTAL: 48 objects, 54983271424 bytes

Twenty-five wall-clock minutes for a job whose serial runtime is 55, with one task having failed and been retried transparently. If RETRIED is high, the tasks are being evicted rather than failing — check the memory ceiling before assuming a data problem. If succeededCount is 48 but you have 47 objects, two tasks computed the same index, which means the manifest changed between task starts.

Gotchas

  • GDAL_NUM_THREADS=ALL_CPUS reads the host, not your allocation. On a Cloud Run node the container sees the machine’s full core count, so ALL_CPUS on a --cpu 4 task can spawn 32 GDAL worker threads competing for 4 vCPU. Set the number literally and keep it equal to --cpu.

  • There is no dead-letter queue. After --max-retries attempts the task is simply marked failed and the execution reports a non-zero failedCount. Nothing captures the input for later inspection. If you need one, have the task write a failure record to Pub/Sub in an except block before returning non-zero.

  • A task that exceeds --task-timeout is killed with SIGKILL after a SIGTERM grace period. Any /vsigs/ write in flight leaves a partial object with no completion metadata — which is why already_done() checks the metadata flag rather than the object’s existence.

  • Cloud Run jobs have no request-driven scale-to-zero cost model. You are billed for vCPU and memory for the full duration of every running task, including tasks that are waiting on I/O. A task that spends 70% of its 6 minutes on range requests is 70% paid idle — which is the argument for higher --parallelism and fewer, chunkier tasks rather than the opposite.

Frequently Asked Questions

What is the maximum runtime of a Cloud Run job task?

60 minutes, the same ceiling as a Cloud Run request and a Cloud Functions 2nd gen invocation. A task can be allocated up to 32 GiB of memory and 8 vCPU, against AWS Lambda’s 15 minutes and 10,240 MB.

How does a Cloud Run job task know which tile to process?

From CLOUD_RUN_TASK_INDEX, a zero-based integer, together with CLOUD_RUN_TASK_COUNT. The task slices a shared manifest with those two numbers instead of receiving a payload — the single largest code change when porting from an event-driven Lambda.

How do Cloud Run job retries differ from Lambda retries?

A failed task is re-run in place up to --max-retries times with the same index and an incremented CLOUD_RUN_TASK_ATTEMPT. There is no automatic backoff and no dead-letter queue, so the task must be idempotent on its own index and must recognise work it already committed.

Should I use Cloud Run jobs or Cloud Functions 2nd gen for a long raster job?

Both stop at 60 minutes, but a job is the better fit for batch work: it has native task parallelism, it exits with a status rather than returning a response, and it is not sized around an HTTP request. Reach for Cloud Functions 2nd gen when the work is genuinely request-shaped and for a job when it is a fan-out over a fixed list.

Back to Timeout Ceiling Comparison for Long-Running Geospatial Jobs