Tuning HTTP Range Requests for COG Reads on S3

Q: How many HTTP range requests does a single COG window read cost?

At minimum one GET for the header (IFD plus tile offset/byte-count arrays) and one GET per COG tile the window overlaps, minus any adjacent tiles GDAL merges into a single range. A 512×512 window aligned to a 512-tiled COG at native resolution touches one tile — two GETs total. A misaligned window straddling four tiles costs up to five.

Q: Does /vsis3 cache range reads across Lambda invocations?

No. The CPL_VSIL_CURL block cache and GDAL_CACHEMAX live in process memory and vanish when the container is recycled. They help within a warm invocation that re-reads overlapping windows, but a cold start starts empty. For cross-invocation reuse, stage the header or use a warm container pool via provisioned concurrency.

Reading a window from a Cloud Optimized GeoTIFF on S3 costs one HTTP GET for the header plus one GET per tile the window overlaps — so the goal is to shrink both the count and the latency of those range requests. Set GDAL_INGESTED_BYTES_AT_OPEN=32768 to pull the full tile index in the opening GET, CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif to stop GDAL probing for nonexistent sidecars, GDAL_HTTP_MULTIPLEX=YES plus GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES to reuse the connection and coalesce adjacent tiles, and grow GDAL_CACHEMAX/CPL_VSIL_CURL_CACHE_SIZE so re-read tiles are served from memory. On a warm Lambda, a well-aligned 512×512 read then resolves in as few as two GETs.

Context

A Cloud Optimized GeoTIFF is laid out so a reader can fetch just the bytes it needs: a header block holds the image file directory (IFD) with per-tile byte offsets and lengths, and the pixel data follows as internally tiled, typically 512×512, blocks. GDAL’s /vsis3 virtual file system turns each read into an HTTP Range: bytes= GET against the S3 object. That is exactly what makes windowed reads cheap — but a carelessly configured GDAL turns a two-request read into seven: it probes for .ovr, .aux.xml, and .msk sidecars that do not exist, fetches the header and the tile index in separate round trips, and issues one un-merged GET per straddled tile. Each round trip to S3 adds tens of milliseconds, and in a Lambda with a 15-minute ceiling those milliseconds multiply across thousands of windows.

This tuning is the transport-layer complement to the chunked I/O patterns for large satellite imagery covered here, where the guide on multi-band Sentinel-2 chunking aligns read windows to the COG block grid. Aligning windows decides how many tiles you touch; the range-request configuration here decides how many GETs and how much latency each of those tiles costs. Both matter under the memory and CPU budget of a raster Lambda, and both feed into whether a job stays inside the 15-minute Lambda timeout ceiling.

Prerequisites

Runtime: Python 3.11 on AWS Lambda with GDAL ≥ 3.6 (via a rasterio layer). AWS Lambda gives 15 minutes and up to 10,240 MB; allocate enough memory that GDAL_CACHEMAX has room to grow without contending with your NumPy arrays.
COG on S3: the source object must be a valid COG — internally tiled with the IFD at the front. Validate with rio cogeo validate s3://bucket/scene.tif before tuning; a non-COG (striped, or IFD at the end) defeats every setting here because the reader cannot seek to tiles.
IAM — execution role: s3:GetObject on the source object/prefix. No ListBucket is needed for a direct read by key.

Environment variables (set on the function; these are the tuning surface):

code

GDAL_INGESTED_BYTES_AT_OPEN=32768        # pull IFD + tile index in the open GET
CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif    # never probe for .ovr/.aux.xml/.msk sidecars
GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR   # do not LIST the prefix on open
GDAL_HTTP_MULTIPLEX=YES                  # reuse one HTTP/2 connection
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES   # coalesce adjacent tile ranges
GDAL_HTTP_VERSION=2                       # require HTTP/2 for multiplexing
VSI_CACHE=TRUE
VSI_CACHE_SIZE=5000000                    # per-file VSI cache bytes
CPL_VSIL_CURL_CACHE_SIZE=200000000        # global curl block cache (~200 MB)
GDAL_CACHEMAX=512                          # block cache in MB
AWS_REGION=us-east-1                        # same region as the bucket

Dependencies: rasterio==1.4.* (bundles GDAL and links libcurl with HTTP/2).

Implementation

Set the GDAL config in the environment so it applies process-wide, or scope it per-read with rasterio.Env. The example below reads a single window and, on a warm container, re-reads an overlapping window to demonstrate the VSI cache serving the second read without new GETs.

python

"""cog_window_read.py — minimize range GETs for a COG window on S3.

The GDAL config is the whole optimization. rasterio.Env applies it for
the duration of the block so it does not leak into other handlers.
"""
import os
import rasterio
from rasterio.windows import Window, from_bounds

# These map 1:1 onto the CPL/GDAL knobs documented for the /vsis3 driver.
GDAL_CONFIG = {
    # Pull the IFD and tile offset/byte-count arrays in the opening GET so
    # the first pixel read does not trigger a second round trip for offsets.
    "GDAL_INGESTED_BYTES_AT_OPEN": 32768,
    # Restrict the reader to .tif: it will not issue HEAD/GET probes for
    # companion .ovr, .aux.xml, or .msk files that a plain COG never has.
    "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif",
    # Skip the directory listing GDAL otherwise performs on open.
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
    # Reuse a single HTTP/2 connection and coalesce neighbouring tile ranges
    # into one Range request instead of one GET per tile.
    "GDAL_HTTP_MULTIPLEX": "YES",
    "GDAL_HTTP_MERGE_CONSECUTIVE_RANGES": "YES",
    "GDAL_HTTP_VERSION": "2",
    # Keep fetched blocks in memory so overlapping re-reads hit the cache.
    "VSI_CACHE": True,
    "VSI_CACHE_SIZE": 5_000_000,
    "CPL_VSIL_CURL_CACHE_SIZE": 200_000_000,
    "GDAL_CACHEMAX": 512,  # MB
}

S3_COG = "s3://sentinel-cogs/tiles/32/T/NM/scene_B04.tif"


def read_window(bounds: tuple[float, float, float, float]):
    """Read a georeferenced window from a COG with minimal range GETs.

    bounds is (west, south, east, north) in the COG's CRS. Aligning the
    request to the COG's internal block grid keeps the tile count minimal;
    see the multi-band Sentinel-2 chunking guide for window alignment.
    """
    with rasterio.Env(**GDAL_CONFIG):
        with rasterio.open(S3_COG) as ds:
            # Snap the geographic window to the dataset's transform.
            win = from_bounds(*bounds, transform=ds.transform)
            # Round to whole pixels so we do not straddle an extra tile row.
            win = win.round_offsets().round_lengths()
            data = ds.read(1, window=win)

            # Warm re-read of an overlapping window: served from the VSI
            # block cache with zero additional GETs to S3.
            overlap = Window(win.col_off, win.row_off,
                             win.width // 2, win.height // 2)
            _ = ds.read(1, window=overlap)

            return data, ds.block_shapes[0]


if __name__ == "__main__":
    arr, block_shape = read_window((499980, 5090220, 509980, 5100220))
    print(f"read {arr.shape} from COG with internal blocks {block_shape}")

Verification

Count the actual range GETs by enabling GDAL’s curl verbosity and grepping the log for Range: headers. A window aligned to a 512-tiled COG should show one header GET and one (merged) tile GET.

bash

# Run the read with curl tracing on; count the Range requests it issues.
CPL_CURL_VERBOSE=YES GDAL_HTTP_VERSION=2 \
  python cog_window_read.py 2> curl_trace.log

grep -c 'Range: bytes=' curl_trace.log
# Expected for an aligned single-tile window: 2  (header + merged tile)

# Inspect which byte ranges were fetched.
grep 'Range: bytes=' curl_trace.log

Expected trace for a well-aligned read — a header range near byte 0 and one contiguous tile range:

code

> Range: bytes=0-32767
> Range: bytes=6815744-7078911
read (512, 512) from COG with internal blocks (512, 512)

Compare against a run with the tuning removed (unset the config) and you should see extra Range: lines for the separate index fetch and per-tile GETs, plus HEAD requests probing for .ovr and .aux.xml sidecars.

Gotchas and Edge Cases

The COG must actually be a COG. If the IFD sits at the end of the file or the raster is striped rather than tiled, GDAL cannot seek to a tile and falls back to reading large contiguous spans — sometimes the whole file. No /vsis3 setting rescues a non-COG. Run rio cogeo validate first, and rewrite offenders following the multi-band chunked I/O patterns.
GDAL_INGESTED_BYTES_AT_OPEN set too high wastes bandwidth. For COGs with many overviews the tile index can exceed 32 KB, so 65536 helps; for a small single-resolution COG, 16384 is plenty and a larger value just fetches unused bytes on every open. Measure the header size with tiffinfo or gdalinfo -json and size it to just cover the IFD plus the largest tile-offset array.
The VSI and block caches do not survive a cold start. CPL_VSIL_CURL_CACHE_SIZE and GDAL_CACHEMAX live in process memory and are wiped when Lambda recycles the container. They pay off within a warm invocation that re-reads overlapping windows, not across cold invocations. If cross-invocation locality matters, keep containers warm with provisioned concurrency.
Cross-region reads dominate the latency budget. A /vsis3 GET from a Lambda in us-east-1 to a bucket in eu-central-1 adds ~90 ms per round trip on top of everything above; no GDAL knob offsets the speed of light. Co-locate the function with the bucket, and if you cannot, front the object with a same-region cache. Watch for CPL_VSIL_CURL retries in the trace, which signal throttling or transient 5xx from S3.

Frequently Asked Questions

How many HTTP range requests does a single COG window read cost?

At minimum one GET for the header (the IFD plus the tile offset and byte-count arrays) and one GET per COG tile the window overlaps, minus any adjacent tiles GDAL merges into a single range. A 512×512 window aligned to a 512-tiled COG at native resolution touches one tile — two GETs total. A window misaligned to the grid straddles up to four tiles and costs up to five GETs. Alignment and merging are what keep the count near the floor.

What does `GDAL_INGESTED_BYTES_AT_OPEN` actually change?

It sets how many bytes GDAL reads from the start of the file on open. For a COG whose header and tile index occupy the first tens of kilobytes, raising it to 16384 or 32768 pulls the entire IFD in the opening GET, so the first pixel read does not incur a second round trip to fetch tile offsets. Set it too high and you waste bandwidth on files that do not need it; measure the header size and size the value to cover it.

Does `/vsis3` cache range reads across Lambda invocations?

No. The CPL_VSIL_CURL block cache and GDAL_CACHEMAX live in process memory and vanish when the container is recycled. They help within a warm invocation that re-reads overlapping windows, but a cold start begins empty. For cross-invocation reuse, keep a warm container pool via provisioned concurrency, or stage the header/tile index in a fast side store keyed by object ETag.

Should I use `/vsis3` or `/vsicurl` for S3 COGs?

Use /vsis3 when GDAL should sign requests with the Lambda role’s credentials — it handles SigV4 and regional endpoints natively. Reserve /vsicurl for public or pre-signed HTTPS URLs where no signing is required. Both honor the same range-request, multiplexing, and cache knobs described here; the difference is authentication, not transport tuning.

Chunked I/O for Large Satellite Imagery — parent overview on streaming windowed reads and writes for large rasters
Optimizing Chunked I/O for Multi-Band Sentinel-2 Processing — align read windows to the COG block grid so each window touches the fewest tiles
Memory and CPU Allocation for Raster Workloads — size GDAL_CACHEMAX against the function’s memory so caching does not starve NumPy
Chunking Raster Jobs to Fit the 15-Minute Lambda Ceiling — keep per-window GET latency small enough that a tiling job finishes inside the timeout
Reducing Python GDAL Cold Starts with Provisioned Concurrency — keep containers warm so the VSI block cache survives across reads

Back to Chunked I/O for Large Satellite Imagery