Serverless Geospatial Processing Provisioning

Serverless geospatial processing is the practice of declaring short-lived, event-triggered functions — AWS Lambda, Google Cloud Functions, or Fargate tasks — that perform a single spatial operation on demand and then vanish, with the entire runtime, IAM boundary, and trigger wiring expressed as version-controlled infrastructure code. The operational problem it solves is expensive idle capacity: a full GeoServer Deployment Patterns stack or an always-on render fleet burns money around the clock even when nobody is requesting a reprojected tile, a simplified vector layer, or a freshly warped raster. This work belongs to the broader Geospatial Resource Provisioning domain and sits directly alongside two sibling resource types: the demand-driven virtual machines described in Compute Node Orchestration and the durable asset tier defined in Object Storage for Raster/Vector Workloads. Provisioning serverless spatial compute well means accepting a hard set of platform constraints — a size-limited function package, a capped /tmp scratch area, a wall-clock timeout, and a cold-start penalty — and encoding each of them as an explicit, tested decision rather than discovering them in production when a 4 GB mosaic evicts the ephemeral disk mid-warp.

Architectural Context

The reason serverless is attractive for spatial work is the shape of the demand curve. Reprojecting a tile from EPSG:4326 to EPSG:3857, simplifying a vector layer for a low zoom level, or warping a scene into an analysis-ready grid are all bursty, embarrassingly parallel, per-request operations. A tile viewer might fire hundreds of these in a second when a user pans across a fresh area, then nothing for an hour. Paying for a running GeoServer JVM and its backing instance through that idle hour is pure waste, whereas a function bills only for the milliseconds it executes and scales each invocation independently. The same logic applies to ingestion: when a new scene lands in a bucket, one event should fan out to a function that builds overviews, rather than a cron job polling a queue on an always-warm box.

What makes the serverless model genuinely hard for geospatial is that the spatial toolchain is heavy and the platform is deliberately lean. A generic web function ships a few megabytes of interpreted code; a spatial function must carry GDAL, PROJ’s transformation grids, GEOS, and often rasterio or Shapely bindings — hundreds of megabytes of native libraries that blow straight past the zipped-layer size ceiling. The workloads themselves are memory- and disk-hungry in ways a JSON transform never is: a single warp can allocate a multi-hundred-megabyte in-memory array and spill intermediate tiles to scratch disk. Every one of the platform’s four hard limits — package size, /tmp capacity, memory ceiling, and execution timeout — is the exact resource a raster operation is most likely to exhaust. Provisioning is therefore an exercise in sizing against those limits deliberately, and in choosing the packaging format (container image versus zipped layer) that lifts the ones you cannot live under.

Environment Parity and Configuration Drift Mitigation

Parity for a serverless spatial function is anchored in the deployment artifact, exactly as it is for a baked machine image in Compute Node Orchestration. The function’s GDAL build, driver matrix, and PROJ grid set must be identical across development, staging, and production, or a reprojection that succeeds in staging will silently pick the wrong transformation pipeline in production. Two packaging strategies deliver that parity, and the choice drives everything downstream:

• Container image. The function is a Docker image built from a base that already carries the geospatial runtime, pushed to a registry, and referenced by an immutable digest. This lifts the deployment ceiling to 10 GB, which is the only realistic option once GDAL plus its drivers are in scope. The digest — not a latest tag — is the pin, so every environment deploys the byte-identical layer stack.
• Zipped package with a shared layer. The handler is a small zip that attaches a published Lambda layer holding a stripped GDAL. This keeps cold starts marginally faster but fights a hard 250 MB unzipped size limit across function plus layers, which a full driver set exceeds; it only works with an aggressively pruned build.

Drift creeps in through the same channels regardless of packaging. Reference the image by digest, not by a mutable tag, so a rebuilt latest cannot silently change the runtime under a redeploy. Keep memory, timeout, and reserved-concurrency values in tracked .tfvars rather than hand-edited in the console — a console change to a function’s timeout is invisible to the next terraform plan until it force-reverts, and a raster function that quietly lost 60 seconds of timeout will start failing only on the largest inputs. Resolve source and destination bucket names, cache prefixes, and the target CRS through variables and environment configuration, never hardcoded, so one module serves every environment.

CI/CD Validation and Operational Guardrails

A serverless spatial deploy moves through the pipeline as a state-managed change with ordered, fail-closed gates: syntax validation, terraform plan, policy-as-code, and drift detection against live functions. The spatial-specific checks are what separate this from a generic function deploy. Pre-apply validation should confirm that the referenced container digest actually exists in the registry and that the image was built with the expected GDAL version — a smoke test that runs gdalinfo --version and asserts a driver is present inside the built image, gating the push. Policy rules enforce mandatory cost-allocation tags, forbid a function_url with authorization_type = "NONE" unless the function is explicitly public, and require reserved concurrency on any function that reads a shared database so a burst cannot exhaust a connection pool.

The concurrency guardrail is acute for spatial workloads. An S3 event trigger on a bucket receiving a batch upload of a hundred scenes will attempt a hundred concurrent invocations by default; if each one opens a PostGIS connection to look up metadata, that burst can saturate the pool provisioned in PostGIS Cluster Provisioning and take the interactive query path down with it. Reserved and provisioned concurrency belong in code, tuned against the downstream tier’s real capacity. Rollback is a version pointer: publish each deploy as a numbered function version behind an alias, and a regression reverts the alias to the previous version rather than rebuilding, so a bad GDAL bump is undone in seconds. Wiring these gates into pull-request workflows gives serverless changes the same scrutiny the render fleet gets, and the engine trade-offs behind that workflow are covered in Terraform vs Pulumi for GIS.

Resource Architecture and Service Integration

A serverless spatial function is the transient center of a small constellation of persistent services, and the provisioning code is mostly about wiring those edges with least privilege. Upstream sit the triggers: an S3 event notification for ingestion-style work, or API Gateway / a Lambda function URL for request-driven work like tile reprojection. Sideways, the function reads source rasters and writes results to object storage, and — for cataloged workloads — reads metadata from the spatial database. Downstream, it emits structured logs and custom metrics.

The asset edge is the most consequential and the one most often provisioned wrong. Heavy raster reads, cloud-optimized GeoTIFF range requests, and result writes all traverse the path to Object Storage for Raster/Vector Workloads; routing that traffic through a NAT gateway rather than an S3 gateway VPC endpoint turns every warp into a metered egress charge and inflates the bill past the always-on GeoServer the serverless design was meant to undercut. When the function participates in a larger ingestion flow — building overviews, retiling, or generating cloud-optimized outputs from a fresh scene — it is one stage of the sequence described in Raster Data Pipeline Provisioning, invoked as a step rather than a standalone service. The identity edge is governed by the least-privilege trust and prefix-scoping conventions in IAM Role Mapping for GIS: the function role should grant s3:GetObject on the source prefix and s3:PutObject on the result prefix only, never a bucket-wide wildcard. The single most common concrete realization of this whole pattern is a request-time reprojection endpoint, detailed in Provisioning Lambda for On-the-Fly Raster Reprojection.

Runnable Configuration

The following Terraform provisions a container-image Lambda carrying the GDAL runtime, an S3 event trigger, a least-privilege execution role scoped to distinct source and result prefixes, and sized memory and timeout tuned for raster work. Provider versions are pinned so an upstream default change cannot force-replace the function.

terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = { source = "hashicorp/aws", version = ">= 5.30, < 6.0" }
  }

  # Locked, versioned state: a concurrent apply against a live function
  # can corrupt the image-digest / alias pointer and boot a bad runtime.
  backend "s3" {
    bucket         = "spatial-iac-state-prod"
    key            = "serverless-geo/production/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "spatial-iac-state-lock"
  }
}

# Least-privilege role: read source rasters, write results, log — nothing more.
resource "aws_iam_role" "geo_fn" {
  name = "geo-processing-fn-role"
  assume_role_policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "geo_fn" {
  name = "geo-processing-fn-policy"
  role = aws_iam_role.geo_fn.id
  policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [
      {
        Sid      = "ReadSourceRasters"
        Effect   = "Allow"
        Action   = ["s3:GetObject"]
        Resource = "${var.source_bucket_arn}/scenes/*"
      },
      {
        Sid      = "WriteResults"
        Effect   = "Allow"
        Action   = ["s3:PutObject"]
        Resource = "${var.result_bucket_arn}/derived/*"
      },
      {
        Sid      = "Logs"
        Effect   = "Allow"
        Action   = ["logs:CreateLogStream", "logs:PutLogEvents"]
        Resource = "${aws_cloudwatch_log_group.geo_fn.arn}:*"
      }
    ]
  })
}

resource "aws_cloudwatch_log_group" "geo_fn" {
  name              = "/aws/lambda/geo-processing-fn"
  retention_in_days = 14
}

# Container image carrying GDAL/PROJ/GEOS, pinned by immutable digest.
# memory_size is sized for raster arrays; the /tmp scratch budget rises
# with ephemeral_storage. timeout covers a worst-case large warp.
resource "aws_lambda_function" "geo_fn" {
  function_name = "geo-processing-fn"
  role          = aws_iam_role.geo_fn.arn
  package_type  = "Image"
  image_uri     = var.image_uri # e.g. <acct>.dkr.ecr...@sha256:<digest>
  memory_size   = 3008          # MB — CPU scales with memory; warps are CPU-bound
  timeout       = 120           # s  — a big reprojection can run for a minute+
  architectures = ["x86_64"]    # match the arch GDAL was compiled for

  ephemeral_storage { size = 4096 } # MB of /tmp for spilled raster tiles

  environment {
    variables = {
      SOURCE_BUCKET = var.source_bucket
      RESULT_BUCKET = var.result_bucket
      TARGET_EPSG   = "3857"
      GDAL_CACHEMAX = "512" # MB block cache — bound it under memory_size
    }
  }

  # Reserved concurrency caps the burst so an S3 batch upload cannot
  # exhaust the downstream PostGIS connection pool.
  reserved_concurrent_executions = var.max_concurrency
}

# S3 event trigger: a new scene invokes the function once per object.
resource "aws_lambda_permission" "from_s3" {
  statement_id  = "AllowS3Invoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.geo_fn.function_name
  principal     = "s3.amazonaws.com"
  source_arn    = var.source_bucket_arn
}

resource "aws_s3_bucket_notification" "scene_landed" {
  bucket = var.source_bucket
  lambda_function {
    lambda_function_arn = aws_lambda_function.geo_fn.arn
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "scenes/"
    filter_suffix       = ".tif"
  }
  depends_on = [aws_lambda_permission.from_s3]
}

Note: GDAL_CACHEMAX and memory_size are function-level tuning values, not RDS parameters — they take plain megabyte integers here. When this function reads catalog metadata from a PostGIS instance, any RDS parameter-group memory value (for example shared_buffers) must be expressed in AWS 8 kB block units, never a percentage string, and the instance is provisioned separately in PostGIS Cluster Provisioning.

The ephemeral_storage block is the spatial-specific lever: the default 512 MB /tmp is exhausted by a single large mosaic, so raster functions raise it toward the 10 GB ceiling and the code writes intermediates there, cleaning up before returning so a warm container’s reused disk does not accumulate across invocations.

Guardrails Embedded in the Configuration

The configuration encodes four guardrails that survive operator turnover because they live in code. State locking through the DynamoDB table stops two concurrent applies from interleaving the image-digest and alias pointers, which could otherwise leave the function serving from a half-written version. Least-privilege identity scopes the role to scenes/* for reads and derived/* for writes, so a bug — or a compromised dependency — cannot exfiltrate the whole bucket; this mirrors the prefix-scoping posture in IAM Role Mapping for GIS.

Concurrency capping through reserved_concurrent_executions is the guardrail that protects everything downstream: it converts an unbounded fan-out from an S3 batch into a throttled queue, so the shared database and any rate-limited upstream API see a ceiling instead of a spike. Resource sizing ties memory_size, timeout, ephemeral_storage, and GDAL_CACHEMAX together as one versioned profile. These must stay coherent: setting GDAL_CACHEMAX above memory_size invites an out-of-memory kill, and raising memory without raising timeout leaves the function fast but still liable to be cut off mid-warp on the largest inputs. Because the whole runtime profile versions together with the image digest, a rollback restores a known-good combination of all four at once.

Troubleshooting and Failure Modes

/tmp exhaustion on large rasters. The function warps a multi-gigabyte mosaic, GDAL spills intermediate tiles to /tmp, and the write fails with No space left on device once the ephemeral disk fills — often only on the largest scenes, so it passes every small-input test. The signature is an errno 28 in the logs correlated with input size. Resolve it by raising ephemeral_storage, streaming the operation with windowed reads instead of materializing the whole array, and deleting intermediates before return so a reused warm container does not carry stale files forward.

GDAL layer size limit exceeded. A zip-plus-layer deployment fails at terraform apply with Layers consume more than the available size or Unzipped size must be smaller than 262144000 bytes, because the full GDAL driver set plus PROJ grids blows past the 250 MB unzipped ceiling. Fix it by switching to a container-image package, which raises the ceiling to 10 GB, or by pruning the build to only the drivers the workload actually opens.

Timeout on big reprojections. A request-driven reprojection of a large extent runs past the configured timeout and Lambda kills it with Task timed out after N seconds, returning a 502 through API Gateway while the caller sees an intermittent failure that tracks input extent. Detect it from duration metrics hugging the timeout ceiling. Address it by raising timeout toward the 15-minute maximum, increasing memory_size so more CPU shortens the warp, tiling the work into smaller sub-requests, and caching results so a repeat request is a cache hit rather than a re-warp.

Cold-start latency spikes on the render path. The first request after idle pays a multi-second cold start while the container image and GDAL runtime initialize, so a user panning to a fresh area sees a stall while warm invocations are instant. The symptom is a bimodal latency distribution. Mitigate it with provisioned concurrency on the interactive function to keep a pool warm, a slimmer base image, and by moving lazy imports to module scope so initialization happens once during init rather than per request.

Concurrency throttling under burst. A batch upload triggers more concurrent invocations than the account limit or the function’s reserved concurrency allows, and excess invocations are throttled with 429 TooManyRequestsException — silently dropping S3 events if the dead-letter path is not configured. Catch it from the Throttles metric. Resolve it by setting reserved concurrency deliberately against downstream capacity, adding an SQS queue with a dead-letter target between the trigger and the function to absorb the burst, and requesting a concurrency quota increase where sustained throughput demands it.

Related

• Provisioning Lambda for On-the-Fly Raster Reprojection — the request-time reprojection endpoint this pattern most often becomes.
• Compute Node Orchestration — the always-on fleet alternative, for sustained rather than bursty spatial compute.
• Object Storage for Raster/Vector Workloads — the asset tier functions read and write over a VPC endpoint.
• Raster Data Pipeline Provisioning — the multi-stage ingestion flow that invokes these functions as steps.
• IAM Role Mapping for GIS — least-privilege execution roles and prefix scoping for the function’s asset access.
• Geospatial Resource Provisioning — the provisioning domain this serverless layer belongs to.