Tuning HTTP Range Requests for COG Reads on S3

Reading a window from a Cloud Optimized GeoTIFF on S3 costs one HTTP GET for the header plus one GET per tile the window overlaps — so the goal is to shrink both the count and the latency of those range requests. Set GDAL_INGESTED_BYTES_AT_OPEN=32768 to pull the full tile index in the opening GET, CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif to stop GDAL probing for nonexistent sidecars, GDAL_HTTP_MULTIPLEX=YES plus GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES to reuse the connection and coalesce adjacent tiles, and grow GDAL_CACHEMAX/CPL_VSIL_CURL_CACHE_SIZE so re-read tiles are served from memory. On a warm Lambda, a well-aligned 512×512 read then resolves in as few as two GETs.

Context

A Cloud Optimized GeoTIFF is laid out so a reader can fetch just the bytes it needs: a header block holds the image file directory (IFD) with per-tile byte offsets and lengths, and the pixel data follows as internally tiled, typically 512×512, blocks. GDAL’s /vsis3 virtual file system turns each read into an HTTP Range: bytes= GET against the S3 object. That is exactly what makes windowed reads cheap — but a carelessly configured GDAL turns a two-request read into seven: it probes for .ovr, .aux.xml, and .msk sidecars that do not exist, fetches the header and the tile index in separate round trips, and issues one un-merged GET per straddled tile. Each round trip to S3 adds tens of milliseconds, and in a Lambda with a 15-minute ceiling those milliseconds multiply across thousands of windows.

This tuning is the transport-layer complement to the chunked I/O patterns for large satellite imagery covered here, where the guide on multi-band Sentinel-2 chunking aligns read windows to the COG block grid. Aligning windows decides how many tiles you touch; the range-request configuration here decides how many GETs and how much latency each of those tiles costs. Both matter under the memory and CPU budget of a raster Lambda, and both feed into whether a job stays inside the 15-minute Lambda timeout ceiling.

A 512-pixel window on a COG internal block grid, aligned and offsetA grid of internal 512-pixel COG blocks. One window sits exactly on a block boundary and covers a single block, costing one tile GET. A second window of the same size, offset by 64 pixels, overlaps four adjacent blocks, each of which must be fetched and decompressed in full.How many blocks one window touches1 GETpartialpartialpartialpartialThe codec works per blockGDAL cannot fetch part of a tile — the smallest addressable unitis one whole compressed block. A window offset by 64 pixelsreturns the same 512 x 512 pixels after fetching anddecompressing 1024 x 1024.Aligned window: 1 block, 1 tile GETOffset window: 4 blocks, up to 4 tile GETsBlock never fetchedMERGE_CONSECUTIVE_RANGES coalesces the two horizontally adjacent blocks in each row into one Range header, so the offset case costs 4blocks but often only 2 GETs.
Alignment decides the tile count; the range-request settings then decide how many GETs those tiles cost. Get the first wrong and no amount of the second recovers it.

The reason alignment dominates is that the compression codec operates per block: GDAL cannot ask S3 for half a tile, decompress it, and stop. The smallest addressable unit is one whole compressed block, so a window offset by 64 pixels fetches and decompresses 1024 × 1024 pixels to return 512 × 512.

Prerequisites

  • Runtime: Python 3.11 on AWS Lambda with GDAL ≥ 3.6 (via a rasterio layer). AWS Lambda gives 15 minutes and up to 10,240 MB; allocate enough memory that GDAL_CACHEMAX has room to grow without contending with your NumPy arrays.
  • COG on S3: the source object must be a valid COG — internally tiled with the IFD at the front. Validate with rio cogeo validate s3://bucket/scene.tif before tuning; a non-COG (striped, or IFD at the end) defeats every setting here because the reader cannot seek to tiles.
  • IAM — execution role: s3:GetObject on the source object/prefix. No ListBucket is needed for a direct read by key.
  • Environment variables (set on the function; these are the tuning surface):
    code 
    GDAL_INGESTED_BYTES_AT_OPEN=32768        # pull IFD + tile index in the open GET
CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif    # never probe for .ovr/.aux.xml/.msk sidecars
GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR   # do not LIST the prefix on open
GDAL_HTTP_MULTIPLEX=YES                  # reuse one HTTP/2 connection
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES   # coalesce adjacent tile ranges
GDAL_HTTP_VERSION=2                       # require HTTP/2 for multiplexing
VSI_CACHE=TRUE
VSI_CACHE_SIZE=5000000                    # per-file VSI cache bytes
CPL_VSIL_CURL_CACHE_SIZE=200000000        # global curl block cache (~200 MB)
GDAL_CACHEMAX=512                          # block cache in MB
AWS_REGION=us-east-1                        # same region as the bucket
  • Dependencies: rasterio==1.4.* (bundles GDAL and links libcurl with HTTP/2).

Implementation

Set the GDAL config in the environment so it applies process-wide, or scope it per-read with rasterio.Env. The five settings that matter divide cleanly into two groups: two of them delete round trips outright, and three of them only pay off on a warm container.

The GDAL /vsis3 settings that reduce range-request costFive configuration steps with their exact values: suppress sidecar and directory probes, pull the image file directory in the opening GET, reuse one HTTP/2 connection, coalesce adjacent tile ranges, and size the in-process block caches so warm re-reads never leave the process.Five settings, in the order they pay off1Stop the probesNo HEAD or GET for .ovr, .aux.xml or .msk, and no prefix listing on openALLOWED_EXTENSIONS=.tif2Fold the index into the openIFD plus TileOffsets and TileByteCounts arrive in the first GETINGESTED_BYTES_AT_OPEN=327683Reuse the connectionOne HTTP/2 connection carries every subsequent range requestGDAL_HTTP_MULTIPLEX=YES4Coalesce adjacent rangesNeighbouring tiles become one Range header instead of one GET eachMERGE_CONSECUTIVE_RANGES=YES5Cache the fetched blocksOverlapping re-reads inside a warm invocation never reach S3 againGDAL_CACHEMAX=512
The first two settings remove round trips outright; the last two only pay off on a warm container that re-reads overlapping windows.

The example below reads a single window and, on a warm container, re-reads an overlapping window to demonstrate the VSI cache serving the second read without new GETs.

python 
"""cog_window_read.py — minimize range GETs for a COG window on S3.

The GDAL config is the whole optimization. rasterio.Env applies it for
the duration of the block so it does not leak into other handlers.
"""
import os
import rasterio
from rasterio.windows import Window, from_bounds

# These map 1:1 onto the CPL/GDAL knobs documented for the /vsis3 driver.
GDAL_CONFIG = {
    # Pull the IFD and tile offset/byte-count arrays in the opening GET so
    # the first pixel read does not trigger a second round trip for offsets.
    "GDAL_INGESTED_BYTES_AT_OPEN": 32768,
    # Restrict the reader to .tif: it will not issue HEAD/GET probes for
    # companion .ovr, .aux.xml, or .msk files that a plain COG never has.
    "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif",
    # Skip the directory listing GDAL otherwise performs on open.
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
    # Reuse a single HTTP/2 connection and coalesce neighbouring tile ranges
    # into one Range request instead of one GET per tile.
    "GDAL_HTTP_MULTIPLEX": "YES",
    "GDAL_HTTP_MERGE_CONSECUTIVE_RANGES": "YES",
    "GDAL_HTTP_VERSION": "2",
    # Keep fetched blocks in memory so overlapping re-reads hit the cache.
    "VSI_CACHE": True,
    "VSI_CACHE_SIZE": 5_000_000,
    "CPL_VSIL_CURL_CACHE_SIZE": 200_000_000,
    "GDAL_CACHEMAX": 512,  # MB
}

S3_COG = "s3://sentinel-cogs/tiles/32/T/NM/scene_B04.tif"


def read_window(bounds: tuple[float, float, float, float]):
    """Read a georeferenced window from a COG with minimal range GETs.

    bounds is (west, south, east, north) in the COG's CRS. Aligning the
    request to the COG's internal block grid keeps the tile count minimal;
    see the multi-band Sentinel-2 chunking guide for window alignment.
    """
    with rasterio.Env(**GDAL_CONFIG):
        with rasterio.open(S3_COG) as ds:
            # Snap the geographic window to the dataset's transform.
            win = from_bounds(*bounds, transform=ds.transform)
            # Round to whole pixels so we do not straddle an extra tile row.
            win = win.round_offsets().round_lengths()
            data = ds.read(1, window=win)

            # Warm re-read of an overlapping window: served from the VSI
            # block cache with zero additional GETs to S3.
            overlap = Window(win.col_off, win.row_off,
                             win.width // 2, win.height // 2)
            _ = ds.read(1, window=overlap)

            return data, ds.block_shapes[0]


if __name__ == "__main__":
    arr, block_shape = read_window((499980, 5090220, 509980, 5100220))
    print(f"read {arr.shape} from COG with internal blocks {block_shape}")

Verification

Count the actual range GETs by enabling GDAL’s curl verbosity and grepping the log for Range: headers. A window aligned to a 512-tiled COG should show one header GET and one (merged) tile GET. Applying the settings one at a time and re-counting is the only way to know which of them your workload actually needed — each removes a specific, identifiable round trip.

HTTP round trips per window as each GDAL setting is appliedFour measured round-trip counts for the same window read: seven with default settings, five after restricting allowed extensions to .tif, four after raising GDAL_INGESTED_BYTES_AT_OPEN to 32768, and two once consecutive ranges are merged and the window is aligned to the block grid.Range GETs for one 512 x 512 window readDefaults, misaligned window7+CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif5+GDAL_INGESTED_BYTES_AT_OPEN=327684+ MERGE_CONSECUTIVE_RANGES, aligned20HTTP round tripsThe floor is two: one GET for the header and tile index, one for the single block the aligned window covers. Nothing below that is reachable ona cold container.
Each setting removes a specific round trip, and the order matters — sidecar probes and the separate index fetch come off before alignment has anything left to save.

Two is the floor on a cold container, and it is not beatable: one GET for the header and tile index, one for the block the window covers.

bash 
# Run the read with curl tracing on; count the Range requests it issues.
CPL_CURL_VERBOSE=YES GDAL_HTTP_VERSION=2 \
  python cog_window_read.py 2> curl_trace.log

grep -c 'Range: bytes=' curl_trace.log
# Expected for an aligned single-tile window: 2  (header + merged tile)

# Inspect which byte ranges were fetched.
grep 'Range: bytes=' curl_trace.log

Expected trace for a well-aligned read — a header range near byte 0 and one contiguous tile range:

code 
> Range: bytes=0-32767
> Range: bytes=6815744-7078911
read (512, 512) from COG with internal blocks (512, 512)

Compare against a run with the tuning removed (unset the config) and you should see extra Range: lines for the separate index fetch and per-tile GETs, plus HEAD requests probing for .ovr and .aux.xml sidecars.

Gotchas and Edge Cases

  • The COG must actually be a COG. If the IFD sits at the end of the file or the raster is striped rather than tiled, GDAL cannot seek to a tile and falls back to reading large contiguous spans — sometimes the whole file. No /vsis3 setting rescues a non-COG. Run rio cogeo validate first, and rewrite offenders following the multi-band chunked I/O patterns.

  • GDAL_INGESTED_BYTES_AT_OPEN set too high wastes bandwidth. For COGs with many overviews the tile index can exceed 32 KB, so 65536 helps; for a small single-resolution COG, 16384 is plenty and a larger value just fetches unused bytes on every open. Measure the header size with tiffinfo or gdalinfo -json and size it to just cover the IFD plus the largest tile-offset array.

  • The VSI and block caches do not survive a cold start. CPL_VSIL_CURL_CACHE_SIZE and GDAL_CACHEMAX live in process memory and are wiped when Lambda recycles the container. They pay off within a warm invocation that re-reads overlapping windows, not across cold invocations. If cross-invocation locality matters, keep containers warm with provisioned concurrency.

  • Cross-region reads dominate the latency budget. A /vsis3 GET from a Lambda in us-east-1 to a bucket in eu-central-1 adds ~90 ms per round trip on top of everything above; no GDAL knob offsets the speed of light. Co-locate the function with the bucket, and if you cannot, front the object with a same-region cache. Watch for CPL_VSIL_CURL retries in the trace, which signal throttling or transient 5xx from S3.

Frequently Asked Questions

How many HTTP range requests does a single COG window read cost?

At minimum one GET for the header (the IFD plus the tile offset and byte-count arrays) and one GET per COG tile the window overlaps, minus any adjacent tiles GDAL merges into a single range. A 512×512 window aligned to a 512-tiled COG at native resolution touches one tile — two GETs total. A window misaligned to the grid straddles up to four tiles and costs up to five GETs. Alignment and merging are what keep the count near the floor.

What does GDAL_INGESTED_BYTES_AT_OPEN actually change?

It sets how many bytes GDAL reads from the start of the file on open. For a COG whose header and tile index occupy the first tens of kilobytes, raising it to 16384 or 32768 pulls the entire IFD in the opening GET, so the first pixel read does not incur a second round trip to fetch tile offsets. Set it too high and you waste bandwidth on files that do not need it; measure the header size and size the value to cover it.

Does /vsis3 cache range reads across Lambda invocations?

No. The CPL_VSIL_CURL block cache and GDAL_CACHEMAX live in process memory and vanish when the container is recycled. They help within a warm invocation that re-reads overlapping windows, but a cold start begins empty. For cross-invocation reuse, keep a warm container pool via provisioned concurrency, or stage the header/tile index in a fast side store keyed by object ETag.

Should I use /vsis3 or /vsicurl for S3 COGs?

Use /vsis3 when GDAL should sign requests with the Lambda role’s credentials — it handles SigV4 and regional endpoints natively. Reserve /vsicurl for public or pre-signed HTTPS URLs where no signing is required. Both honor the same range-request, multiplexing, and cache knobs described here; the difference is authentication, not transport tuning.

Back to Chunked I/O for Large Satellite Imagery