Streaming COGs Without Touching /tmp

Open the raster as /vsis3/bucket/key.tif and set GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR, CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif,.tiff,.ovr, GDAL_CACHEMAX=512 and CPL_VSIL_CURL_CACHE_SIZE=67108864 — GDAL then satisfies every window from HTTP range requests held in process memory, and the function’s ephemeral disk stays at zero bytes used. A 4 GB Sentinel-2 scene read this way costs about 1.4 MB of transferred bytes for a 512 × 512 window, against a 4 GB download that no /tmp allocation on Azure Consumption could hold in the first place. The technique removes ephemeral storage from the sizing conversation entirely, which is the cheapest way to stop worrying about it.

The Failure This Removes

Ephemeral Storage Comparison Across Serverless Platforms sets out the ceilings you are working against: AWS Lambda gives 512 MB of /tmp by default and 10,240 MB if you pay for it, GCP Cloud Functions 2nd gen gives roughly 8 GB of tmpfs that is deducted from the same 32,768 MB the process is running in, and Azure Functions on the Consumption plan gives roughly 1.5 GB of local storage shared across every function in the app. Only one of those numbers is generous, and it is generous because it is really memory.

The classic download-then-open handler collides with all three. It calls s3.download_file() into /tmp/scene.tif, opens the local copy, reads one window, and returns. Three things go wrong. The obvious one is that a 4 GB scene does not fit anywhere except a maximum-size Lambda. The second is that the download is serial and uncached — the function pays full transfer time before the first pixel is decoded, which on a cold container stacks directly on top of the initialisation cost mapped in Cold Start Mapping for Python GDAL. The third is the quiet one: warm containers are reused, so a handler that writes /tmp/scene.tif and does not delete it will fill the disk after four or five invocations and start failing with OSError: [Errno 28] No space left on device on a container that worked fine ten minutes earlier.

Streaming deletes the whole class of problem. A Cloud Optimized GeoTIFF places its IFD headers and tile offset tables at the front of the file, so GDAL can learn the full layout in two or three range requests and then fetch exactly the tiles that intersect the requested window. Nothing is materialised, so nothing needs cleaning up, and the same code runs unchanged on a 512 MB Lambda and a 1,536 MB Azure Consumption function.

Range-request conversation between a handler, GDAL /vsis3/ and S3A sequence diagram with four participants: the handler, the GDAL block cache, the vsicurl chunk cache, and the S3 object. The handler opens the raster, GDAL fetches the first 16 kilobytes of header and then the tile offset table, then requests two coalesced tile ranges, and finally returns decoded pixels from the block cache without any filesystem access.One 512 x 512 window, four HTTP requests, zero disk writesHandlerGDAL block cache/vsicurl chunk cacheS3 objectrasterio.open()/vsis3/eo-scenes/scene.tifGET bytes=0-16383IFD header, block shape 512x512GET tile offset tableoverview levels 2, 4, 8, 16read(window)col 4096, row 4096, 512 x512need 16 tilesconsecutive ranges mergedGET 2 merged ranges1.4 MB transferredraw chunksheld in the 64 MB curl cacheuint16 array524,288 bytes, no file created
The header reads happen once per container; the tile fetches happen once per window. Nothing in this exchange has a filename, which is why /tmp stays empty.

Prerequisites

  • Runtime: Python 3.11 or 3.12, with rasterio 1.3.9+ (GDAL 3.6+) or GDAL 3.6+ Python bindings. GDAL below 3.4 lacks the /vsis3/ multi-range coalescing that makes this pattern efficient.
  • The source must be a real COG. Run rio cogeo validate scene.tif or gdalinfo and confirm the output reports internal tiling (Block=512x512, not Block=10980x1) and an overview pyramid. A striped GeoTIFF streams correctly but pathologically — see the decision figure below.
  • IAM: the execution role needs s3:GetObject on the object prefix only. s3:ListBucket is not required once GDAL_DISABLE_READDIR_ON_OPEN is set, and leaving it off is the point of the scoping described in IAM Security Boundaries for Cloud GIS.
  • Environment variables, set in the function configuration rather than in code:
    • GDAL_DATA=/opt/share/gdal and PROJ_LIB=/opt/share/proj — resolve the driver metadata and datum grids from the layer, not from a network fetch.
    • LD_LIBRARY_PATH=/opt/lib — where libgdal.so, libproj.so and libcurl.so live in a layer-based deployment.
    • GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR — stops GDAL issuing a ListObjectsV2 per open to hunt for .ovr and .aux.xml sidecars. EMPTY_DIR rather than YES keeps explicitly named sidecars working.
    • GDAL_CACHEMAX=512 — decoded block cache, in megabytes when the value is a bare integer.
    • CPL_VSIL_CURL_CACHE_SIZE=67108864 — 64 MB, in bytes, for the raw fetched-chunk cache. The two caches are separate and the units differ.
    • CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif,.tiff,.ovr — refuses to probe anything else over HTTP.
    • VSI_CACHE=TRUE and VSI_CACHE_SIZE=26214400 — 25 MB per-handle read-ahead cache, again in bytes.
    • PROJ_NETWORK=OFF — otherwise PROJ may try to download a datum grid at transform time and cache it under /tmp, which is exactly the write you are trying to avoid.
  • Networking: if the function runs in a VPC, an S3 gateway endpoint must exist or every range request leaves through a NAT gateway at per-GB cost.

Sizing the Two Caches

The mistake that makes streaming look slow is leaving GDAL_CACHEMAX at its default. GDAL’s default is 5% of physical RAM, and on a 1,024 MB function that is about 51 MB — smaller than a single 512 × 512 × 4-band Float32 block set, so every window read evicts the blocks the next window needs. Raising it is free until you take memory the rest of the handler needs.

Memory budget for a streaming COG reader on a 2,048 MB functionA proportional stack of a 2,048 megabyte function allocation, split into the Python and GDAL runtime at 190 megabytes, the GDAL block cache at 512 megabytes, the vsicurl chunk cache at 64 megabytes, the per-handle read-ahead cache at 25 megabytes, and roughly 1,257 megabytes left free for the handler's own NumPy arrays.Carving a 2,048 MB allocation into caches and working setPython + GDAL/PROJ/GEOS runtimeResident before the first line of handler code~190 MBGDAL block cacheGDAL_CACHEMAX=512 — decoded tiles, megabyte units512 MB/vsicurl chunk cacheCPL_VSIL_CURL_CACHE_SIZE=67108864 — raw bytes, byte units64 MBPer-handle read-aheadVSI_CACHE_SIZE=2621440025 MBFree for handler arraysA 3-band Float32 4096 x 4096 mosaic is 201 MB of this~1,257 MBMultiply the two cache layers by worker count if the handler fans out with a ProcessPoolExecutor — each process gets its own.
Budget downwards from the allocation. If the handler's arrays need more than the free band, cut GDAL_CACHEMAX before raising the memory tier.

Budget from the allocation downwards, not from the cache upwards. On a 2,048 MB function: about 190 MB is Python plus the linked GDAL/PROJ/GEOS stack, 512 MB goes to GDAL_CACHEMAX, 64 MB to the /vsicurl/ chunk cache, 25 MB to the per-handle read-ahead, and the remaining ~1,250 MB belongs to your NumPy arrays and whatever the output encoder allocates. If your handler builds a 3-band Float32 mosaic of 4,096 × 4,096 pixels — 201 MB — that fits with room to spare. If it builds ten of them, cut GDAL_CACHEMAX before you raise the allocation. The same arithmetic drives the tiers in Memory and CPU Allocation for Raster Workloads.

GDAL_CACHEMAX also accepts a percentage string. GDAL_CACHEMAX=25% on a Lambda that is later moved from 2,048 MB to 4,096 MB doubles the cache without a config change, which is the safer setting for functions whose allocation is tuned by a cost review.

Implementation

python 
"""Stream a window out of a COG over /vsis3/ with zero ephemeral disk use."""
import json
import logging
import os
import shutil

import numpy as np
import rasterio
from rasterio.session import AWSSession
from rasterio.windows import from_bounds

logger = logging.getLogger()
logger.setLevel(logging.INFO)

# ---------------------------------------------------------------------------
# Module scope: build the GDAL environment once per container. rasterio.Env is
# a context manager over GDAL's config option stack, so entering it per request
# would re-parse and re-push every option on every invocation.
# ---------------------------------------------------------------------------
GDAL_ENV = dict(
    # Never list the prefix looking for .ovr/.aux.xml sidecars. EMPTY_DIR keeps
    # an explicitly named sidecar usable; YES would break it.
    GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR",
    # Decoded-block cache, megabytes. Sized against the 2,048 MB allocation.
    GDAL_CACHEMAX="512",
    # Raw fetched-chunk cache, BYTES (64 MB). Different unit from CACHEMAX.
    CPL_VSIL_CURL_CACHE_SIZE="67108864",
    # Per-file-handle read-ahead, BYTES (25 MB).
    VSI_CACHE="TRUE",
    VSI_CACHE_SIZE="26214400",
    # Refuse to issue HTTP requests for anything that is not a raster.
    CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".tif,.tiff,.ovr",
    # Merge range requests for tiles that are within 64 KB of each other into
    # one HTTP call. This is the single largest win on a tiled COG.
    GDAL_HTTP_MERGE_CONSECUTIVE_RANGES="YES",
    GDAL_HTTP_MULTIPLEX="YES",
    GDAL_HTTP_VERSION="2",
    # A stalled socket must fail fast enough to retry inside the invocation.
    GDAL_HTTP_TIMEOUT="20",
    GDAL_HTTP_MAX_RETRY="3",
    GDAL_HTTP_RETRY_DELAY="1",
    # PROJ must not fetch grids at transform time — that write lands in /tmp.
    PROJ_NETWORK="OFF",
)

AWS_SESSION = AWSSession(aws_unsigned=False)


def _tmp_bytes_used() -> int:
    """Ephemeral disk consumed so far. Used as an assertion, not a metric."""
    usage = shutil.disk_usage("/tmp")
    return usage.used


def handler(event, context):
    """event: {"uri": "s3://bucket/key.tif", "bbox": [w, s, e, n]}"""
    uri = event["uri"]
    west, south, east, north = event["bbox"]

    # s3://  ->  /vsis3/ . rasterio accepts either, but the explicit VSI path
    # makes it obvious in logs which virtual file system is in play.
    vsi_path = uri.replace("s3://", "/vsis3/", 1)

    tmp_before = _tmp_bytes_used()

    with rasterio.Env(session=AWS_SESSION, **GDAL_ENV):
        with rasterio.open(vsi_path) as src:
            if src.block_shapes[0][1] == src.width:
                # A striped file: every window read pulls whole scanline runs.
                # Fail loudly rather than silently issuing 400 range requests.
                raise ValueError(
                    f"{uri} is striped (block {src.block_shapes[0]}), not tiled. "
                    "Re-encode as a COG before streaming it."
                )

            window = from_bounds(west, south, east, north, transform=src.transform)
            # boundless=False: a bbox partly outside the raster raises here
            # rather than allocating a padded array we did not budget for.
            data = src.read(window=window, boundless=False, masked=True)

            # Real work goes here. Kept trivial so the I/O path stays the point.
            band_means = [float(np.ma.mean(band)) for band in data]

            profile = {
                "driver": src.driver,
                "blocks": list(src.block_shapes[0]),
                "overviews": src.overviews(1),
                "window": [int(v) for v in (window.col_off, window.row_off,
                                            window.width, window.height)],
            }

    tmp_after = _tmp_bytes_used()
    # The whole point of the technique, asserted rather than assumed.
    if tmp_after > tmp_before:
        logger.error("streaming read wrote %d bytes to /tmp", tmp_after - tmp_before)

    return {
        "statusCode": 200,
        "body": json.dumps({
            "band_means": band_means,
            "profile": profile,
            "tmp_bytes_written": tmp_after - tmp_before,
        }),
    }

Two details carry most of the weight. GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES lets GDAL coalesce the tile fetches for a window into far fewer HTTP calls — on a 512 × 512 window crossing sixteen internal tiles it typically turns sixteen requests into two or three. And the striped-file guard fails the invocation immediately instead of letting it run for four minutes issuing range requests, which is the difference between a clear error and a timeout you have to profile. The same range-request economics are worked through in Tuning HTTP Range Requests for COG Reads on S3.

Verification

Turn on GDAL’s cURL tracing and count the requests. Run this locally against the same object with the same environment before you deploy:

bash 
CPL_CURL_VERBOSE=YES \
GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR \
GDAL_CACHEMAX=512 \
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES \
python -c "
import rasterio
from rasterio.windows import Window
with rasterio.open('/vsis3/eo-scenes/S2B_33UUP_20260714.tif') as s:
    print('blocks', s.block_shapes[0], 'overviews', s.overviews(1))
    a = s.read(1, window=Window(4096, 4096, 512, 512))
    print('window', a.shape, a.dtype, 'bytes', a.nbytes)
" 2>&1 | grep -c '^> GET'

Expected output on a correctly tiled COG — the read reports its shape, and the request count stays in single digits:

code 
blocks (512, 512) overviews [2, 4, 8, 16]
window (512, 512) uint16 bytes 524288
4

Four GETs is two header reads plus two coalesced tile fetches. If that count comes back as 40 or 400, the file is striped, GDAL_HTTP_MERGE_CONSECUTIVE_RANGES is off, or GDAL_DISABLE_READDIR_ON_OPEN is unset and GDAL is probing for sidecars. In the deployed function, assert the same property directly: shutil.disk_usage("/tmp").used must be identical before and after the read, and the handler above already logs an error when it is not.

When Streaming Loses

Streaming is not free, and there are three shapes of work where downloading the object into /tmp is genuinely faster.

Decision between streaming a COG and downloading it to /tmpA decision tree asking how much of the raster the job will read. Three outcomes: reading a spatial subset of a tiled COG streams over /vsis3/, reading more than about 60 percent of the pixels downloads once to /tmp, and a striped or non-tiled source is re-encoded on ingest before either path is worth taking.Streaming is the default, not the answer to everythingHow much of the raster will this invocationactually read?a spatial subsetStream over /vsis3/4 range requests per window/tmp usage stays at 0 bytesRuns unchanged on Azure's 1.5 GB poolmost of the pixelsDownload once to /tmpOne GetObject at line rateNeeds the file to fit the quota512 MB default on Lambdasource is stripedRe-encode on ingestBlock 10980x1 has no cheap subsetConvert once, stream forever
The break-even sits near 60% of pixels touched on files above roughly 200 MB — measure it on your own objects before adopting the number.

The first is whole-file reads. If you will touch more than roughly 60% of the pixels — a full reprojection, a global statistics pass, an overview build — you pay HTTP framing on every tile for no benefit. One GetObject at line rate beats several hundred range requests, and here the /tmp sizing guidance in Managing /tmp Storage Limits for GeoTIFF Extraction is the right reference instead of this page.

The second is non-COG inputs. A striped GeoTIFF, a JPEG2000 without a tiled codestream, or an ESRI Grid has no layout that lets GDAL fetch a spatial subset cheaply. Convert it once on ingest and stream from the converted copy afterwards.

The third is many small scattered reads from a small file. Below about 200 MB, latency dominates: fifty windows at three range requests each is 150 round trips, where the whole file would have arrived in one. The break-even shifts with object size, so measure it on your own data rather than adopting a number.

Gotchas

  • GDAL_CACHEMAX and CPL_VSIL_CURL_CACHE_SIZE use different units. A bare integer in GDAL_CACHEMAX is megabytes; CPL_VSIL_CURL_CACHE_SIZE is always bytes. Setting CPL_VSIL_CURL_CACHE_SIZE=64 gives you a 64-byte chunk cache and a read pattern that looks like a network fault.

  • The chunk cache is per process, not per container. If you fan out with a ProcessPoolExecutor, each worker gets its own GDAL_CACHEMAX allocation. Four workers at GDAL_CACHEMAX=512 is 2 GB of cache, and on a 2,048 MB function that is an OOM kill with no Python traceback.

  • PROJ can still write to disk even when GDAL does not. A transform that needs a datum grid not present in PROJ_LIB will, with PROJ_NETWORK=ON, fetch it from the CDN and cache it in $PROJ_USER_WRITABLE_DIRECTORY — which on Lambda defaults under /tmp. Set PROJ_NETWORK=OFF and bake the grids you need into the layer.

  • Unsigned reads need to be explicit. Public buckets such as the Sentinel-2 open data archive require AWS_NO_SIGN_REQUEST=YES (or AWSSession(aws_unsigned=True)); with a signed session against a bucket your role cannot access, GDAL reports a bare Access Denied with no indication that the credentials, rather than the path, were the problem.

Frequently Asked Questions

Does /vsis3/ write anything to /tmp?

No. /vsis3/ and /vsicurl/ hold fetched byte ranges in a process-memory chunk cache sized by CPL_VSIL_CURL_CACHE_SIZE, and decoded raster blocks in the GDAL block cache sized by GDAL_CACHEMAX. Neither touches the filesystem. The only way a streaming read reaches /tmp is if PROJ downloads a datum grid, which PROJ_NETWORK=OFF prevents.

What should GDAL_CACHEMAX be on a 2,048 MB function?

Around 512 MB. Budget roughly 190 MB for the Python and GDAL runtime, 64 MB for the /vsicurl/ chunk cache, and leave at least 40% of the allocation free for the NumPy arrays your own code builds. Setting GDAL_CACHEMAX=25% makes the value track the allocation automatically if the function is later resized.

When is downloading the file faster than streaming it?

When you will read most of the pixels anyway, when the file is not internally tiled, or when you need many small scattered reads from a file below roughly 200 MB. A striped GeoTIFF forces GDAL to fetch whole scanline runs per window, so a single full download beats hundreds of range requests.

Can I stream from GCS and Azure Blob the same way?

Yes — /vsigs/bucket/key.tif and /vsiaz/container/blob.tif use the same block cache, the same CPL_VSIL_CURL_* options and the same coalescing logic. Credentials differ: /vsigs/ picks up the Cloud Functions metadata-server identity automatically, and /vsiaz/ needs AZURE_STORAGE_ACCOUNT plus either AZURE_STORAGE_ACCESS_KEY or AZURE_STORAGE_SAS_TOKEN.

Back to Ephemeral Storage Comparison Across Serverless Platforms