GDAL Config Options That Actually Change COG Read Performance

Four settings take a cold 512×512 window read of a Sentinel-2 COG on S3 from 2,410 ms to 520 ms on a 1,769 MB Lambda, and the first of them accounts for more than half the gain: GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR, CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif,.tiff,.vrt,.ovr, VSI_CACHE=TRUE with VSI_CACHE_SIZE=536870912, and GDAL_CACHEMAX=512. Everything else on the usual tuning list — GDAL_HTTP_MULTIPLEX, GDAL_NUM_THREADS, GDAL_HTTP_MAX_RETRY — is worth single-digit percentages or nothing at all, and two of them are actively harmful below 1,769 MB. The numbers below are medians of 40 cold reads with each option added one at a time.

Context

A COG is designed so that a reader can fetch its header once and then request exactly the byte ranges holding the tiles it wants. GDAL implements that well. What GDAL also does, by default, is behave as though the object store were a filesystem: on open it lists the containing directory so it can discover sidecar files, and it speculatively requests .aux.xml, .msk and .ovr companions that a properly-built COG does not have. On a local disk those are microseconds. On S3 with a prefix holding thousands of scenes, the LIST alone is 860 ms.

That is the entire performance story, and it is why this page is a configuration page rather than a code page. The reads are already efficient; the discovery around them is not. The variables belong in the same function environment map as the data-directory settings from PROJ and GDAL runtime configuration and setting GDAL_DATA and PROJ_LIB in Lambda, because the same rule applies: set them before the interpreter starts, not inside the handler.

S3 request sequence for one COG open, before and after tuningA sequence between the Lambda handler, the GDAL VSI curl layer and the S3 bucket showing the default behaviour: a LIST of the containing prefix, speculative GET requests for the aux.xml, msk and ovr sidecars that all return 404, then the header range request and finally the tile range request. Only the last two remain once readdir is disabled and allowed extensions are restricted.The seven requests defaults make, and the two that surviveHandlerGDAL /vsis3S3 bucketrasterio.openone scene, 10 m bandLIST prefixremoved byREADDIR_ON_OPEN=EMPTY_DIRGET .aux.xml404 — removed by ALLOWED_EXTENSIONSGET .msk404 — removed by ALLOWED_EXTENSIONSGET .ovr404 — removed by ALLOWED_EXTENSIONSGET header rangebytes=0-16383, always neededGET tile rangeone 512×512 block, always needednumpy arrayfirst pixel available
Five of the seven round trips exist only so GDAL can discover sidecar files that a COG does not have. Two environment variables delete all five.

Prerequisites

  • A genuine COG, internally tiled with overviews. Validate with python -m rio_cogeo validate <url>; a striped GeoTIFF cannot benefit from any of this because a window read has to fetch whole scanlines regardless.
  • Function and bucket in the same region. Cross-region reads add 60–90 ms per round trip and swamp the differences being measured here.
  • 1,769 MB of Lambda memory for the numbers to reproduce — that is where one full vCPU is allocated. The AWS ceiling is 10,240 MB and 15 minutes; Cloud Run allows 60 minutes and up to 32 GiB with 8 vCPU; Azure Functions on the Consumption plan caps at 1,536 MB and 10 minutes.
  • GDAL 3.6 or later, for reliable /vsis3 multiplexing behaviour.
  • A cold-read harness. Warm reads measure the cache, not the configuration — see the measurement block below.
  • CloudWatch or equivalent request-count metrics, so the S3 request count can be recorded next to the wall clock. The count is the honest signal; the wall clock moves with network weather.

What Each Option Costs and Saves

Measured COG window read time as each GDAL config option is addedFive measured wall-clock times for the same 512 by 512 window read from a Sentinel-2 COG on S3 by a 1,769 megabyte Lambda: 2,410 milliseconds with defaults, 1,180 after disabling readdir on open, 940 after restricting allowed extensions, 610 after enabling the VSI cache at 512 megabytes, and 520 after adding GDAL_CACHEMAX and HTTP multiplexing.One 512×512 window read, options added one at a timeDefaults, nothing set2,410 ms+GDAL_DISABLE_READDIR_ON_OPEN1,180 ms+CPL_VSIL_CURL_ALLOWED_EXTENSIONS940 ms+ VSI_CACHE, VSI_CACHE_SIZE610 ms+ GDAL_CACHEMAX,HTTP_MULTIPLEX520 ms02,500 msMedian of 40 cold reads of one 10 m Sentinel-2 band on S3 from a 1,769 MB x86_64 Lambda in the same region as the bucket; each rowkeeps every option from the rows above it.
The first change is worth more than the other three combined. Everything after VSI_CACHE is single-digit percentage tuning, so stop measuring once the LIST and the sidecar probes are gone.

GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR — 2,410 ms to 1,180 ms. The largest single win available. EMPTY_DIR tells GDAL to behave as though the containing prefix were empty, so it issues no LIST and looks for no siblings. Use EMPTY_DIR rather than YES: YES disables the directory read but still permits some sibling probing, while EMPTY_DIR removes both. The one thing it breaks is a workflow that relies on an external .ovr overview file or an ESRI .aux.xml — if the pipeline genuinely reads those, name their extensions in the next variable rather than turning this one off.

CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif,.tiff,.vrt,.ovr — 1,180 ms to 940 ms. GDAL will not attempt to open any object whose extension is not on this list, which suppresses the remaining speculative sidecar GETs. Each of those is a full round trip that returns 404, so the saving is roughly 80 ms per suppressed probe and is independent of how large the prefix is. Include every extension the pipeline actually opens; omitting .vrt while using VRTs produces a confusing not recognized as a supported file format error.

VSI_CACHE=TRUE and VSI_CACHE_SIZE=536870912 — 940 ms to 610 ms. The VSI range cache is off by default, and its default size when enabled is 25 MB, which is far too small for multi-band work. It caches the compressed bytes returned by range requests, per file handle, so re-reading an overlapping range within one dataset does not repeat the request. The 330 ms saving here comes almost entirely from the header and directory-block ranges being re-read during band iteration.

GDAL_CACHEMAX=512 and GDAL_HTTP_MULTIPLEX=YES — 610 ms to 520 ms. GDAL_CACHEMAX is the decompressed block cache, shared across open datasets. GDAL_HTTP_MULTIPLEX=YES lets concurrent range requests share one HTTP/2 connection instead of opening several. Together they are worth 90 ms on a single-window read and considerably more on a multi-band read where the same overview blocks are touched repeatedly, which is the pattern in optimizing chunked I/O for multi-band Sentinel-2 processing.

GDAL_NUM_THREADS=ALL_CPUS — no measurable change at 1,769 MB. It parallelises block decompression, and at 1,769 MB there is exactly one vCPU to parallelise across. Above 3,538 MB a second vCPU appears and it starts to pay; on Cloud Run with 8 vCPU it is one of the larger remaining wins. Set it, because it costs nothing and the function may be resized, but do not expect it to show up in a single-window benchmark.

Breakdown of an untuned 2,410 millisecond COG window readA 2,410 millisecond read split into five segments: listing the containing prefix at 860 milliseconds, three speculative sidecar GET requests at 240 milliseconds, the header range request at 190 milliseconds, tile range requests at 830 milliseconds, and block decompression into numpy at 290 milliseconds.Where the untuned 2,410 ms actually goesLIST containing prefixSidecarprobesTile range requestsDecompressto numpy860 ms240 ms830 ms290 msHeader range — 190 msSame 40-read sample as the bar chart. Tuning removes the first two segments outright and shortens the fourth through connection reuse andthe VSI range cache.
Only the last three segments are work. The prefix LIST and the sidecar probes are 1,100 ms of pure discovery overhead on a file format designed so that discovery is unnecessary.

Two caches, two budgets, one memory allocation — this is the detail that catches people. VSI_CACHE_SIZE holds compressed bytes; GDAL_CACHEMAX holds decompressed blocks; both are carved out of the function’s memory before numpy gets any. At 512 MB each on a 1,769 MB Lambda, roughly 700 MB remains for the arrays and the Python runtime — comfortable for 512×512 tiles and not comfortable for a whole-scene read. Size them against the model in memory and CPU allocation for raster workloads. Note too that GDAL_CACHEMAX interprets values under 100,000 as megabytes and larger values as bytes, so 512 and 536870912 both mean 512 MB — mixing the conventions in one codebase is how someone ends up with a 512-byte cache.

Implementation

python 
# cog_read_env.py — the environment block, applied identically in CDK, SAM or Terraform.
COG_READ_ENV = {
    # --- Discovery suppression: the two settings that matter most ---
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
    "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.tiff,.vrt,.ovr",

    # --- Range cache: compressed bytes, per file handle ---
    "VSI_CACHE": "TRUE",
    "VSI_CACHE_SIZE": "536870912",        # 512 MB, in bytes

    # --- Block cache: decompressed blocks, shared across datasets ---
    "GDAL_CACHEMAX": "512",               # < 100000 means megabytes

    # --- Connection behaviour ---
    "GDAL_HTTP_MULTIPLEX": "YES",
    "GDAL_HTTP_VERSION": "2",
    "GDAL_HTTP_MAX_RETRY": "3",
    "GDAL_HTTP_RETRY_DELAY": "1",
    "GDAL_NUM_THREADS": "ALL_CPUS",       # pays above 3,538 MB on Lambda

    # --- Never write into the read-only package ---
    "GDAL_PAM_ENABLED": "NO",
    "CPL_TMPDIR": "/tmp",

    # --- Credentials: use the execution role, skip the EC2 metadata probe ---
    "AWS_REGION": "us-west-2",
    "GDAL_HTTP_UNSAFESSL": "NO",
}

The reading code needs no special handling once the environment is right — which is the point. A rasterio.Env block is only necessary when a single function reads from two stores with different requirements:

python 
# handler.py
import rasterio
from rasterio.windows import Window


def main(event, context):
    url = event["url"]                      # s3://bucket/key or /vsis3/bucket/key
    col, row, size = event["col"], event["row"], event.get("size", 512)
    with rasterio.open(url) as src:
        arr = src.read(1, window=Window(col, row, size, size))
    return {"shape": list(arr.shape), "mean": float(arr.mean())}

Verification

Measure cold, and record the S3 request count alongside the wall clock. A warm read measures the cache; a request count that did not fall means the configuration did not apply.

python 
# bench.py — deploy as its own function, invoke with a fresh URL each time.
import time

import rasterio
from osgeo import gdal
from rasterio.windows import Window


def main(event, context):
    gdal.UseExceptions()
    t0 = time.perf_counter()
    with rasterio.open(event["url"]) as src:
        t_open = time.perf_counter()
        arr = src.read(1, window=Window(0, 0, 512, 512))
        t_read = time.perf_counter()
    return {
        "open_ms": round((t_open - t0) * 1000, 1),
        "read_ms": round((t_read - t_open) * 1000, 1),
        "total_ms": round((t_read - t0) * 1000, 1),
        "readdir": gdal.GetConfigOption("GDAL_DISABLE_READDIR_ON_OPEN"),
        "allowed_ext": gdal.GetConfigOption("CPL_VSIL_CURL_ALLOWED_EXTENSIONS"),
        "vsi_cache": gdal.GetConfigOption("VSI_CACHE"),
        "cachemax_bytes": gdal.GetCacheMax(),
        "mean": float(arr.mean()),
    }

Expected output on a tuned 1,769 MB function reading a same-region Sentinel-2 COG:

json 
{
  "open_ms": 214.6,
  "read_ms": 305.9,
  "total_ms": 520.5,
  "readdir": "EMPTY_DIR",
  "allowed_ext": ".tif,.tiff,.vrt,.ovr",
  "vsi_cache": "TRUE",
  "cachemax_bytes": 536870912,
  "mean": 1843.7
}

An open_ms above 800 means the discovery traffic is still happening — check that readdir really reads back as EMPTY_DIR rather than null, which is what a variable set in the handler after import looks like. Confirm independently against the bucket:

bash 
aws s3api select-object-content --help >/dev/null   # ensure CLI v2
aws cloudwatch get-metric-statistics \
  --namespace AWS/S3 --metric-name AllRequests \
  --dimensions Name=BucketName,Value=my-imagery Name=FilterId,Value=entire-bucket \
  --start-time "$(date -u -d '10 minutes ago' +%FT%TZ)" \
  --end-time "$(date -u +%FT%TZ)" --period 60 --statistics Sum

Seven requests per read is the untuned signature; two is the tuned one. That ratio, not the millisecond figure, is what survives a noisy network.

Gotchas and Edge Cases

  • EMPTY_DIR hides real sidecar files. If the pipeline depends on an external .ovr pyramid or an ESRI .aux.xml statistics file, GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR makes them invisible and GDAL silently computes statistics from the data instead — slower, and different numbers. Rebuild the imagery with internal overviews rather than turning the option off; the COG format exists precisely so that sidecars are unnecessary.
  • Setting these in the handler is too late for the ones GDAL reads at registration. GDAL_CACHEMAX is consulted when the cache is first allocated, and the VSI options are read per file handle. In practice mixing the two mechanisms produces a function where some options apply and others do not, which is far harder to debug than none applying. Put all of them in the function configuration.
  • The two caches compete with the raster you are reading. VSI_CACHE_SIZE plus GDAL_CACHEMAX plus the working numpy arrays must fit inside the memory allocation, and Lambda kills the invocation with Runtime exited with error: signal: killed rather than raising MemoryError. Halve both caches before halving the tile size when memory is tight — 128 MB of block cache is enough for a 512×512 pipeline.
  • GDAL_HTTP_MULTIPLEX needs HTTP/2 on both ends. S3 and GCS support it; some corporate proxies and VPC endpoint configurations negotiate down to HTTP/1.1, in which case the setting is inert rather than harmful. Pair it with GDAL_HTTP_VERSION=2 and confirm with CPL_CURL_VERBOSE=YES in a scratch invocation if the expected gain does not appear. The interaction with range-request sizing is covered in tuning HTTP range requests for COG reads on S3.

Frequently Asked Questions

Which single GDAL option gives the biggest COG read speedup?

GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR. On the measured 512×512 window read it took the total from 2,410 ms to 1,180 ms by itself, because it removes a LIST of the containing prefix that costs 860 ms when that prefix holds thousands of scenes. Every other option combined accounted for the remaining 660 ms.

What is the difference between VSI_CACHE_SIZE and GDAL_CACHEMAX?

VSI_CACHE_SIZE caches compressed bytes fetched over HTTP, per file handle, so a re-read of the same byte range does not repeat the request. GDAL_CACHEMAX caches decompressed raster blocks across all open datasets. They are separate allocations out of the same function memory, so setting both to 512 MB on a 1,769 MB Lambda leaves roughly 700 MB for numpy arrays and the Python runtime.

Does GDAL_NUM_THREADS help on AWS Lambda?

Only above 1,769 MB, which is where a second vCPU appears. Below that, ALL_CPUS resolves to one core and the setting adds thread coordination overhead for nothing. On Cloud Run, with up to 8 vCPU and a 60-minute request timeout, it is one of the larger remaining wins; on Azure Functions Consumption at 1,536 MB it is not worth setting.

Back to PROJ and GDAL Runtime Configuration