Spatial IaC Terraform & Pulumi for GIS

Provisioning Lambda for On-the-Fly Raster Reprojection

You need a raster tile reprojected from EPSG:4326 to EPSG:3857 the instant a map client requests it, without pre-warping an entire pyramid or paying for an always-on server, and every attempt so far either times out on large extents, fills /tmp, or returns a tile whose coordinate reference system is still the source CRS. This guide provisions a Lambda function that reprojects raster tiles on demand — packaged as a container image carrying GDAL and rasterio, wired behind a Lambda function URL or API Gateway, sized for the memory and timeout a warp actually needs, and backed by an S3 result cache so a repeat request never re-warps. It is the concrete build behind the Serverless Geospatial Processing Provisioning cluster and sits within the broader Geospatial Resource Provisioning domain.

Symptom Identification and Triage

On-the-fly reprojection fails in a few distinct ways, and each points at a different provisioning fix. Classify the failure before changing the function, because raising memory will not fix a wrong-CRS output and raising timeout will not fix a /tmp overflow:

  • Task timed out after N seconds on large extents: the function log ends abruptly and API Gateway returns a 502. Duration metrics hug the configured timeout only for big tiles. This is under-sizing — timeout, memory, or both.
  • OSError: [Errno 28] No space left on device: GDAL spilled intermediates to the 512 MB default /tmp and ran out. It correlates with input raster size, not request rate. This is an ephemeral_storage sizing problem.
  • 200 OK but the tile is in the wrong CRS: gdalinfo on the output still reports the source SRS. The warp never ran, or ran with a source-equals-target no-op. This is a handler logic or environment-variable defect, not a capacity one.
  • Multi-second stalls on the first request after idle: a bimodal latency distribution where warm invocations are fast and the first is slow. This is cold start on a heavy GDAL image, addressed with provisioned concurrency.
  • AccessDenied reading the source or writing the cache: the execution role is missing a prefix-scoped grant. Route this to the identity conventions in IAM Role Mapping for GIS.
Reprojection Lambda Failure Triage A top-down decision tree. The failing reprojection symptom leads to a central question: which signal do you observe? Five branches follow. A task-timed-out message routes to raising timeout and memory size. An errno 28 no-space-left error routes to raising ephemeral storage and streaming windowed reads. A 200 response whose output CRS is still the source routes to fixing the handler warp logic and TARGET_EPSG. A slow first request after idle routes to provisioned concurrency. An AccessDenied routes to widening the prefix-scoped execution role. Reprojection fails or returns wrong tile Which signal? check logs + metrics Timed out raise timeout + memory_size tile the extent Errno 28 raise ephemeral storage windowed reads Wrong CRS fix handler warp + TARGET_EPSG verify with gdalinfo Cold start provisioned concurrency slim base image AccessDenied widen role prefix scope source + cache Match the signal to the provisioning lever before redeploying.

Prerequisites and Environment Assumptions

This guide assumes an AWS target with source rasters — ideally cloud-optimized GeoTIFFs — in an S3 bucket, and a map client that will request reprojected tiles by extent and target CRS. To provision the function you will need:

  • Terraform >= 1.5 with the AWS provider pinned. An unpinned provider can change a Lambda default and force-replace the function on an unrelated apply, so pin it explicitly:

    terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = { source = "hashicorp/aws", version = ">= 5.30, < 6.0" }
  }
}

  • A container image built on a GDAL base (for example ghcr.io/osgeo/gdal:ubuntu-small-3.9.2) with rasterio installed and the handler copied in, pushed to Amazon ECR and referenced by immutable digest. The zip-plus-layer route cannot hold a full GDAL build under the 250 MB unzipped ceiling, so container packaging is assumed throughout.

  • A locking-capable state backend (S3 plus DynamoDB) so a concurrent apply cannot corrupt the function’s image-digest or alias pointer, the same discipline enforced in Managing Terraform State Locks for Spatial Data.

  • IAM permissions for the deploying principal: lambda:* on the function, iam:CreateRole/iam:PutRolePolicy for the execution role, ecr:* on the image repository, and read/write to the state backend and its lock table. The function’s own execution role needs only s3:GetObject on the source prefix and s3:PutObject/s3:GetObject on the cache prefix.

  • Network reachability to S3 through a gateway VPC endpoint if the function runs in a VPC, so raster reads stay on the cloud backbone and off metered NAT egress, per VPC Routing for Tile Servers.

Step-by-Step Remediation

Build the handler, size the function, wire the trigger and cache, then verify the output CRS. Size deliberately — the defaults are set for lightweight functions and will fail a raster warp.

  1. Write a handler that checks the cache, warps, and writes back. The function derives a deterministic cache key from the request parameters, returns the cached tile on a hit, and only warps on a miss. Warping to TARGET_EPSG is the step that matters spatially: without it the endpoint returns source-CRS pixels that the map client will misplace.

    import os
import hashlib
import boto3
import rasterio
from rasterio.warp import calculate_default_transform, reproject, Resampling

s3 = boto3.client("s3")
SRC_BUCKET = os.environ["SOURCE_BUCKET"]
CACHE_BUCKET = os.environ["RESULT_BUCKET"]
TARGET_EPSG = int(os.environ.get("TARGET_EPSG", "3857"))

def handler(event, context):
    q = event.get("queryStringParameters") or {}
    scene = q["scene"]                       # e.g. scenes/2026/aoi.tif
    key = "derived/" + hashlib.sha256(
        f"{scene}:{TARGET_EPSG}".encode()).hexdigest() + ".tif"

    # Cache check: a repeat request never re-warps.
    try:
        s3.head_object(Bucket=CACHE_BUCKET, Key=key)
        return _redirect(CACHE_BUCKET, key)
    except s3.exceptions.ClientError:
        pass

    src_path = f"/vsis3/{SRC_BUCKET}/{scene}"   # GDAL reads S3 directly
    out_path = "/tmp/out.tif"                    # sized by ephemeral_storage
    dst_crs = f"EPSG:{TARGET_EPSG}"

    with rasterio.open(src_path) as src:
        transform, w, h = calculate_default_transform(
            src.crs, dst_crs, src.width, src.height, *src.bounds)
        meta = src.meta.copy()
        meta.update(crs=dst_crs, transform=transform, width=w, height=h)
        with rasterio.open(out_path, "w", **meta) as dst:
            for i in range(1, src.count + 1):
                reproject(
                    source=rasterio.band(src, i),
                    destination=rasterio.band(dst, i),
                    src_crs=src.crs, dst_crs=dst_crs,
                    resampling=Resampling.bilinear)

    s3.upload_file(out_path, CACHE_BUCKET, key)
    os.remove(out_path)   # free /tmp on a reused warm container
    return _redirect(CACHE_BUCKET, key)

def _redirect(bucket, key):
    url = s3.generate_presigned_url(
        "get_object", Params={"Bucket": bucket, "Key": key}, ExpiresIn=300)
    return {"statusCode": 302, "headers": {"Location": url}}

  2. Provision the function with raster-appropriate sizing and a function URL. Memory drives CPU, so a warp runs faster with more of it; timeout must cover a worst-case extent; and ephemeral_storage must hold the output plus GDAL’s spilled intermediates.

    resource "aws_lambda_function" "reproject" {
  function_name = "raster-reproject"
  role          = aws_iam_role.reproject.arn
  package_type  = "Image"
  image_uri     = var.image_uri            # ECR digest, not :latest
  memory_size   = 3008                      # MB — more memory = more CPU
  timeout       = 90                         # s  — covers a large warp
  architectures = ["x86_64"]                # match GDAL's compiled arch

  ephemeral_storage { size = 5120 }         # MB /tmp for output + spill

  environment {
    variables = {
      SOURCE_BUCKET = var.source_bucket
      RESULT_BUCKET = var.result_bucket
      TARGET_EPSG   = "3857"
      GDAL_CACHEMAX = "512"                  # MB — keep under memory_size
      CPL_TMPDIR    = "/tmp"                 # GDAL scratch to sized disk
    }
  }
}

# Public function URL; front with API Gateway if you need auth or WAF.
resource "aws_lambda_function_url" "reproject" {
  function_name      = aws_lambda_function.reproject.function_name
  authorization_type = "AWS_IAM"            # not NONE — signed requests
}

    Note: memory_size and GDAL_CACHEMAX are Lambda-level integers in megabytes, unrelated to RDS tuning. If this endpoint later reads tile metadata from PostGIS, that instance’s parameter-group memory values (such as shared_buffers) must be set in AWS 8 kB block units rather than percentage strings — a separate concern handled in PostGIS Cluster Provisioning.

  3. Scope the execution role to exactly the two prefixes. Reads from the source scenes, writes to the derived cache, and nothing else — a leaked credential cannot then walk the whole bucket.

    resource "aws_iam_role_policy" "reproject" {
  name = "raster-reproject-policy"
  role = aws_iam_role.reproject.id
  policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [
      { Effect = "Allow", Action = ["s3:GetObject"],
        Resource = "${var.source_bucket_arn}/scenes/*" },
      { Effect = "Allow", Action = ["s3:GetObject", "s3:PutObject"],
        Resource = "${var.result_bucket_arn}/derived/*" }
    ]
  })
}

  4. Apply after a clean plan. Run terraform apply only when terraform plan shows the function created or updated in place and the image digest resolving as expected. A plan that proposes replacing the function on an unrelated change is the unpinned-provider hazard the version constraint prevents.

Verification

Confirm the fix end to end, at both the endpoint and the pixel level, before closing the work.

  1. Fetch a reprojected tile. Invoke the function URL for a known scene and follow the redirect to the cached object:

    curl -sSL --aws-sigv4 "aws:amz:us-east-1:lambda" \
  "https://<url-id>.lambda-url.us-east-1.on.aws/?scene=scenes/2026/aoi.tif" \
  -o /tmp/reprojected.tif

  2. Assert the output CRS is the target, not the source. This is the check the wrong-CRS symptom is about — a 200 alone is not success:

    gdalinfo /tmp/reprojected.tif | grep -E "ID\\[\"EPSG\",3857\\]|3857"

    The projection block must report EPSG:3857. If it still shows 4326, the warp did not run — revisit the handler and TARGET_EPSG.

  3. Confirm the cache is populated. A second request for the same scene should return far faster and the object should exist under the derived prefix:

    aws s3 ls s3://<result-bucket>/derived/ | head

  4. Check duration and errors in CloudWatch. Verify the Duration metric sits well under the configured timeout for typical extents and that Errors and Throttles are zero after a short load test.

Preventing Recurrence

Encode the sizing and scope so the same failures cannot return:

  • Policy-as-code gate. Add a rule that fails any aws_lambda_function reprojecting rasters whose timeout is under 60 s or ephemeral_storage is left at the 512 MB default, and any aws_lambda_function_url with authorization_type = "NONE". This blocks the under-sizing and open-endpoint regressions before merge.
  • Pin the image by digest. Reference the ECR image by @sha256: digest in a tracked variable, never :latest, so a rebuilt tag cannot silently swap the GDAL build under a redeploy.
  • Provisioned concurrency on the interactive path. Keep a small warm pool so the first request after idle does not pay the heavy-image cold start, and pair it with the demand-driven scaling patterns in Auto-Scaling EC2 Instances for WMS Endpoints when sustained load outgrows the serverless cost curve.
  • Scheduled drift check. Run terraform plan -detailed-exitcode nightly; a non-zero exit flags an out-of-band console edit to memory, timeout, or the role before it surfaces as a production failure.

Frequently Asked Questions

Why package the function as a container image instead of a zip and a layer?

A full GDAL build with its driver set, PROJ transformation grids, and rasterio exceeds the 250 MB unzipped ceiling that applies to a zipped function plus its layers. A container image raises that ceiling to 10 GB, so it is the only reliable way to ship the complete spatial runtime without pruning drivers the workload may need.

The endpoint returns 200 but the tile is still in EPSG:4326 — what happened?

The warp step never ran, or ran with the source equal to the target. Confirm TARGET_EPSG is set and differs from the source SRS, verify the handler calls reproject into a new dataset rather than copying the source through, and re-check the output with gdalinfo, which must report EPSG:3857.

How do I size memory and timeout for reprojection?

Start from the largest extent you must serve, warp it locally, and note peak memory and wall-clock time. Set memory_size above the peak array plus GDAL_CACHEMAX (more memory also buys more CPU, shortening the warp), and set timeout above the measured time with headroom. Raise ephemeral_storage to hold the output plus GDAL’s spilled intermediates.

Why does the first request take several seconds while the rest are instant?

That is a cold start: the first invocation after idle initializes the container image and the GDAL runtime before your handler runs. Enable provisioned concurrency to keep a pool warm, use a slimmer base image, and move imports to module scope so the runtime initializes once during init rather than on every request.