Sizing /tmp for Shapefile Unzip on Azure Functions

Azure Functions on the Consumption plan gives roughly 1.5 GB of local storage shared by every function in the app, and a shapefile archive expands to 2.5–4× its zipped size — so a 300 MB .zip needs about 1.2 GB of free disk and will not survive alongside a deployment package plus a second function’s temp files. Measure the real expansion ratio with zipfile.ZipInfo.file_size before extracting anything, extract to tempfile.mkdtemp() under the path in %TMP%, and delete each member as soon as it has been consumed. When the arithmetic does not close, convert to FlatGeobuf at ingest and stop extracting at all.

Why Azure Is the Tight One

Ephemeral Storage Comparison Across Serverless Platforms puts the three providers side by side, and Azure Consumption is the outlier in a way that is easy to misread. AWS Lambda’s /tmp is 512 MB by default and configurable to 10,240 MB, and it belongs to one execution environment. GCP Cloud Functions 2nd gen gives roughly 8 GB of tmpfs — but that tmpfs is carved out of the same 32,768 MB memory allocation the process runs in, so a gigabyte written to disk is a gigabyte the process cannot allocate. Azure Consumption gives roughly 1.5 GB, it is a real disk rather than a tmpfs, and it is shared across the function app, not per function and not per invocation.

That sharing is the part that produces surprising failures. A function app containing an HTTP trigger, a blob trigger and a timer trigger has one storage pool between them on each instance. The deployment package itself lives there — under Run-From-Package deployments the mounted archive still consumes space — as do the extracted site content, any Python wheel caches, and the temp files of whichever function ran last. In practice the usable headroom for a shapefile unzip on a Consumption instance is 1.0–1.2 GB, not 1.5. Raising the memory tier does not help, because on Consumption memory tops out at 1,536 MB and the storage pool is a separate quota that does not scale with it.

Local storage claimed on one Azure Consumption instanceFour meters against the roughly 1,536 megabyte shared local storage pool of an Azure Functions Consumption instance: the deployment package and extracted site content at 180 megabytes, a neighbouring function's leaked temp directory at 240 megabytes, the shapefile archive itself at 302 megabytes, and the expanded shapefile members at 929 megabytes which exceeds what remains.What is left of the 1.5 GB pool once the app has taken its shareDeployment package + site contentPresent before any invocation runs180 MBNeighbouring function's temp dirLeaked by a timer trigger on the sameinstance240 MBparcels_2026.zip on diskZipFile needs random access, so it mustland302 MBExpanded .shp + .shx + .dbf + .prjNeeds 929 MB; only 814 MB remain929 MBConsumption tops out at 1,536 MB of memory and ~1.5 GB of local storage; raising one does not raise the other.
The pool is per instance and shared by every function in the app, so the last meter is measured against whatever the first three left behind — not against 1.5 GB.

The other half of the problem is that shapefiles are not one file. A single logical dataset is a .shp, a .shx, a .dbf, a .prj, usually a .cpg, and often a .sbn/.sbx pair and an .xml metadata sidecar — the same multi-part awkwardness that Aggregating Multipart Shapefile Uploads Before Processing deals with at the event layer. GDAL’s OGR driver needs the .shp, .shx and .dbf present simultaneously in the same directory, so the “extract one member at a time” trick only works if you understand which three are mandatory.

What the Expansion Ratio Really Is

Zip’s DEFLATE compresses the three parts of a shapefile very differently, and the ratio is what determines whether the job fits.

Compression ratio by shapefile member typeHorizontal bars showing the expansion ratio of each shapefile member when unzipped: the DBF attribute table expands 6.2 times, the SBN spatial index 4.4 times, the SHX offset index 4.1 times, the overall archive 3.3 times, and the SHP geometry only 1.3 times.DEFLATE treats the three shapefile members completely differently.dbf — attribute table6.2x.sbn — ArcGIS spatial index4.4x.shx — record offset index4.1xwhole archive3.32x.shp — packed coordinate pairs1.3x07x expansionMeasured on a 301.9 MB parcel archive that expands to 1,003.5 MB; dropping the unused .sbn takes the extracted footprint to 929.4 MB.
An attribute-heavy dataset has the worst overall ratio, because the .dbf that costs almost nothing in the archive is the largest file on disk.

The .shp holds packed double-precision coordinate pairs. Consecutive vertices in a polygon differ in their low-order bits, so DEFLATE finds very little to work with — expect a ratio between 1.1× and 1.4×. The .dbf is the opposite: a fixed-width, space-padded, largely repetitive attribute table that routinely compresses 6:1 or better. The .shx is a dense array of 8-byte offset/length records with a strong arithmetic pattern, and compresses around 4:1. A parcel dataset that is 40% geometry and 55% attributes by uncompressed size lands at roughly 3.2× overall, which is why the working rule of 2.5–4× is a range rather than a number.

Do not guess it. A zip central directory records the uncompressed size of every member, and reading it costs one range request rather than a full download.

Prerequisites

  • Runtime: Azure Functions Python worker on the v4 runtime, Python 3.11. azure-functions 1.18+, azure-storage-blob 12.19+, and fiona 1.9+ or GDAL 3.6+ Python bindings.
  • Plan awareness: this page assumes the Consumption plan — 10 minute default timeout (configurable to 10 minutes maximum on Consumption), 1,536 MB memory, ~1.5 GB shared local storage. On Elastic Premium the storage picture changes entirely and most of this becomes unnecessary.
  • Identity and permissions: a system-assigned managed identity on the function app with the Storage Blob Data Reader role on the source container and Storage Blob Data Contributor on the output container. Scope the assignment to the container, not the storage account, per Least-Privilege IAM Policies for Azure Blob Geospatial Access.
  • Application settings (Configuration → Application settings, not hard-coded):
    • GDAL_DATA=/home/site/wwwroot/.python_packages/lib/site-packages/fiona/gdal_data
    • PROJ_LIB=/home/site/wwwroot/.python_packages/lib/site-packages/fiona/proj_data
    • LD_LIBRARY_PATH=/home/site/wwwroot/.python_packages/lib/site-packages/fiona.libs
    • AZURE_STORAGE_ACCOUNT=<account> — used by /vsiaz/ when you take the streaming fallback
    • SHP_MAX_EXPANDED_BYTES=1073741824 — the 1 GiB budget the guard below enforces
    • GDAL_CACHEMAX=128 — keep GDAL’s block cache small; on Consumption the memory ceiling of 1,536 MB is the tighter constraint
    • SHAPE_RESTORE_SHX=YES — lets OGR rebuild a missing .shx rather than refusing to open the layer
  • Do not rely on /tmp literally. On a Windows Consumption instance the temp path is D:\local\Temp. Always resolve it with tempfile.gettempdir(), which reads %TMP%.

Implementation

python 
"""Blob-triggered shapefile unzip that refuses to start a job it cannot finish."""
import json
import logging
import os
import shutil
import tempfile
import zipfile

import azure.functions as func
import fiona

LOG = logging.getLogger("shp-unzip")

# The three members OGR must see in one directory to open a shapefile layer.
REQUIRED = (".shp", ".shx", ".dbf")
# Everything else is useful but individually droppable if space is tight.
OPTIONAL = (".prj", ".cpg")

# Budget, not the quota. The Consumption pool is ~1.5 GB shared across the app;
# 1 GiB is what one function may claim without starving its neighbours.
BUDGET = int(os.environ.get("SHP_MAX_EXPANDED_BYTES", 1_073_741_824))


def plan_extraction(zf: zipfile.ZipFile) -> dict:
    """Read the central directory only — no member is decompressed here.

    ZipInfo.file_size is the recorded uncompressed size, so the whole
    extraction can be costed before a single byte is written to disk.
    """
    members = {}
    for info in zf.infolist():
        ext = os.path.splitext(info.filename)[1].lower()
        members.setdefault(ext, []).append(info)

    required_bytes = sum(
        i.file_size for ext in REQUIRED for i in members.get(ext, [])
    )
    optional_bytes = sum(
        i.file_size for ext in OPTIONAL for i in members.get(ext, [])
    )
    total_bytes = sum(i.file_size for i in zf.infolist())
    compressed = sum(i.compress_size for i in zf.infolist())

    return {
        "members": members,
        "required_bytes": required_bytes,
        "optional_bytes": optional_bytes,
        "total_bytes": total_bytes,
        "compressed_bytes": compressed,
        "ratio": round(total_bytes / max(compressed, 1), 2),
    }


def free_temp_bytes() -> int:
    """Free space in the shared Consumption pool, wherever it is mounted."""
    return shutil.disk_usage(tempfile.gettempdir()).free


def main(blob: func.InputStream, out: func.Out[str]) -> None:
    workdir = tempfile.mkdtemp(prefix="shp-", dir=tempfile.gettempdir())
    try:
        # The archive itself has to land on disk: ZipFile needs random access,
        # and an InputStream is forward-only.
        archive = os.path.join(workdir, "input.zip")
        with open(archive, "wb") as fh:
            shutil.copyfileobj(blob, fh, length=4 * 1024 * 1024)

        with zipfile.ZipFile(archive) as zf:
            plan = plan_extraction(zf)

            missing = [e for e in REQUIRED if not plan["members"].get(e)]
            if missing:
                raise ValueError(f"archive is not a shapefile: missing {missing}")

            # Cost the extraction against BOTH the policy budget and the disk
            # that is actually free right now — the pool is shared, so the
            # second number moves under you between invocations.
            needed = plan["required_bytes"] + plan["optional_bytes"]
            free = free_temp_bytes()
            LOG.info(
                "plan: archive=%d expanded=%d required=%d ratio=%.2f free=%d",
                plan["compressed_bytes"], plan["total_bytes"],
                needed, plan["ratio"], free,
            )

            if needed > BUDGET:
                raise MemoryError(
                    f"expanded shapefile needs {needed} bytes, budget is {BUDGET}. "
                    "Convert to FlatGeobuf at ingest and read it over /vsiaz/."
                )
            # 128 MB of slack for the output writer and the GDAL block cache.
            if needed + 134_217_728 > free:
                raise MemoryError(
                    f"needs {needed} bytes, only {free} free in the shared pool"
                )

            # Extract only what OGR requires. .sbn/.sbx/.xml sidecars can be
            # a third of the archive and OGR never reads them.
            for ext in REQUIRED + OPTIONAL:
                for info in plan["members"].get(ext, []):
                    zf.extract(info, path=workdir)

        # The archive is dead weight from here on — reclaim it before opening
        # the layer, which is when GDAL starts allocating.
        os.remove(archive)

        shp = next(
            os.path.join(dp, f)
            for dp, _, fns in os.walk(workdir)
            for f in fns if f.lower().endswith(".shp")
        )
        with fiona.open(shp) as src:
            summary = {
                "features": len(src),
                "crs": src.crs_wkt[:60] if src.crs_wkt else None,
                "geometry": src.schema["geometry"],
                "fields": len(src.schema["properties"]),
                "expansion_ratio": plan["ratio"],
                "peak_temp_bytes": needed,
            }
        out.set(json.dumps(summary))

    finally:
        # Warm Consumption instances are reused. A directory left behind here
        # is subtracted from the next invocation's headroom, on this function
        # and on every other function in the app.
        shutil.rmtree(workdir, ignore_errors=True)

The guard is the important part, not the extraction. Costing the job from the central directory turns “the function died at 78% with no traceback” into a MemoryError carrying both numbers, thrown before any disk was consumed. Checking shutil.disk_usage(...).free as well as the static budget catches the shared-pool case, where another function in the same app has already taken 700 MB and your perfectly-sized job no longer fits.

Selective extraction is the cheap win. .sbn, .sbx and .shp.xml are ArcGIS artefacts that OGR does not read, and they are frequently 25–35% of the archive by uncompressed size.

Verification

Run the planner against a real archive before deploying, so you know the expansion ratio of your own data rather than the generic one:

bash 
python -c "
import zipfile, os, collections
zf = zipfile.ZipFile('parcels_2026.zip')
by = collections.Counter()
for i in zf.infolist():
    by[os.path.splitext(i.filename)[1].lower()] += i.file_size
comp = sum(i.compress_size for i in zf.infolist())
tot  = sum(by.values())
for ext, n in by.most_common():
    print(f'{ext:10s} {n/1e6:9.1f} MB')
print(f'{\"archive\":10s} {comp/1e6:9.1f} MB')
print(f'{\"expanded\":10s} {tot/1e6:9.1f} MB   ratio {tot/comp:.2f}x')
"

Expected output for a mid-size parcel dataset — note that the .dbf dominates the expanded size while contributing little to the archive:

code 
.dbf          612.4 MB
.shp          298.7 MB
.sbn           74.1 MB
.shx           18.3 MB
.prj            0.0 MB
archive       301.9 MB
expanded     1003.5 MB   ratio  3.32x

At 3.32× a 301.9 MB archive expands to 1,003.5 MB — and dropping .sbn takes the extracted footprint to 929.4 MB, which is what actually has to fit. Against a 1.0–1.2 GB practical headroom that is a job with almost no margin, and the right call is the fallback rather than a redeploy that gets lucky. In the deployed function, watch the FileSystemUsage metric on the storage account backing the app and alert at 1.2 GB.

When It Will Not Fit

Fallback options for a shapefile archive too large for Consumption storageThree side-by-side panels comparing fallbacks: converting to FlatGeobuf at ingest and reading it over the Azure virtual file system with zero disk use, extracting only the three required members and streaming features to the destination in batches, and moving the workload to Elastic Premium or a container where the disk is not shared.Three ways out when 929 MB will not fit in 814 MBConvert at ingest to FlatGeobufRead over/vsiaz/container/parcels.fgbPacked Hilbert R-tree — bbox queryfetches byte ranges onlyPeak local storage: 0 bytesCosts one conversion job with realdiskExtract required members only.shp, .shx, .dbf — drop .sbn, .sbx,.shp.xmlStream features out in batches of50,000Peak local storage: 929 MBOnly helps when sidecars are whattipped it overLeave the Consumption planElastic Premium or Azure ContainerAppsDisk is not shared across the appAlso lifts the 10 min and 1,536 MBceilingsBilled per instance-hour, not perexecutionIf the same dataset arrives every week, convert it once at ingest — the other two paths re-pay the cost on every invocation.
Convert-at-ingest is the only option that removes the constraint rather than deferring it; the other two buy headroom that the next larger dataset will consume.

The strongest option is to stop unzipping. Convert the dataset to FlatGeobuf or GeoParquet on ingest — once, in a job that has room — and read the converted file over /vsiaz/container/parcels.fgb. FlatGeobuf carries a packed Hilbert R-tree, so a bbox query fetches only the relevant byte ranges and the function’s disk usage stays at zero, exactly as in Streaming COGs Without Touching /tmp but for vector data.

The middle option keeps the shapefile but never holds it whole: extract .shp, .shx and .dbf, open the layer, write features to the destination in batches of 50,000, and let the destination — a PostGIS table, a partitioned GeoParquet dataset — own the result. Peak disk is still the three required members, so this only helps when it is the optional sidecars pushing you over.

The last option is to leave Consumption. Elastic Premium gives a much larger and less contended local disk, and a container on Azure Container Apps gives you whatever you provision. That is the same escape hatch the Timeout Ceiling Comparison recommends when 10 minutes is not enough, and the two constraints tend to bite the same jobs.

Gotchas

  • The pool is shared, so your function can be starved by a function you did not change. A neighbouring timer trigger that caches 400 MB of wheels leaves you 400 MB short with no code change of your own. This is why the handler checks disk_usage().free at runtime instead of trusting a number computed at design time.

  • tempfile.gettempdir() is not /tmp on Windows Consumption instances. It resolves to D:\local\Temp. Hard-coding /tmp produces a FileNotFoundError that only appears on the Windows-hosted plan, so it passes every local test.

  • shutil.rmtree in a finally block is not optional on a reused instance. Consumption instances stay warm for several minutes and serve many invocations. A leaked working directory is permanent for the life of that instance and is invisible in the invocation logs — it shows up only as a later invocation failing at a smaller input size than one that succeeded.

  • A .dbf above 2 GB uncompressed cannot be opened at all. The DBF format stores its record count in a 32-bit field and OGR enforces the limit; the archive may extract successfully and then fail at fiona.open(). Check plan["members"][".dbf"] sizes in the planner and reject early with a message that names the format limit rather than the disk.

Frequently Asked Questions

How much local storage does an Azure Functions Consumption app get?

Roughly 1.5 GB, shared by every function in the app on that instance rather than allocated per invocation. The deployment package, extracted site content and temp files all draw on the same pool, so usable headroom for an unzip is typically 1.0–1.2 GB.

How large does a shapefile get when unzipped?

Between 2.5× and 4× the archive size for typical polygon data. The .shp geometry compresses poorly at 1.1–1.4×, while the .dbf attribute table often compresses 6:1 and the .shx index around 4:1 — so an attribute-heavy dataset has the highest overall ratio.

What do I do when the archive will not fit in 1.5 GB?

Convert to FlatGeobuf or GeoParquet at ingest and read it over /vsiaz/ without extracting; or extract only .shp, .shx and .dbf and stream features to the destination in batches; or move to Elastic Premium or a container. Raising the Consumption memory tier does not raise the storage pool — they are separate quotas.

Does the 1.5 GB limit apply to Azure Functions Premium too?

No. The ~1.5 GB figure is specific to the Consumption plan. Elastic Premium instances get a substantially larger local disk sized by the instance tier, and they also lift the 10 minute timeout and the 1,536 MB memory ceiling, which is usually the real reason a geospatial workload moves there.

Back to Ephemeral Storage Comparison Across Serverless Platforms