Spatial IaC Terraform & Pulumi for GIS

Provisioning Batch Compute for Raster Mosaicking

Mosaicking is the raster operation that breaks function-based pipelines. Stitching four hundred overlapping scenes into a seamless national coverage is not a bigger version of converting one scene — it reads tens of gigabytes, needs the intermediate results on local disk, runs for hours, and is embarrassingly parallel only if you partition it by output tile rather than by input scene. A serverless function will time out; a single large instance will work and cost a fortune while idle. Managed batch compute is the shape that fits, and provisioning it well is mostly a matter of getting four things right: the partition, the storage, the instance selection, and the retry policy. This guide extends Raster Pipeline Provisioning within Geospatial Resource Provisioning.

Partition by output, not by input

The instinct is to fan out over input scenes, because that is how the data arrives. It produces a job graph where every worker needs to know about every other worker’s overlap, and a final serial merge that dominates the runtime.

Partition by output tile instead. Divide the target coverage into a grid, and give each worker one output cell plus the list of input scenes intersecting it. Workers then never interact: each reads only the scenes it needs, produces one finished output, and writes it. The merge disappears because there is nothing to merge, and the job scales linearly with the number of cells rather than quadratically with overlaps.

The input-scene list per cell comes from a spatial index over the scene footprints — in practice a PostGIS query against a footprints table, which is exactly the kind of read the cluster described in PostGIS Cluster Provisioning is good at. Computing the partition once, up front, and passing each worker an explicit scene list also makes the job reproducible: a re-run of cell 47 reads the same inputs it read the first time.

Output-cell partitioning compared with input-scene partitioning On the left, partitioning by input scene: each worker takes one scene, workers must coordinate over overlapping areas, and the job ends with a serial merge step that dominates total runtime. On the right, partitioning by output cell: a spatial query over the scene footprint table assigns each grid cell the explicit list of scenes intersecting it, each worker reads only those scenes and writes one finished output cell, no worker interacts with another, and there is no merge step at all. The output partitioning also makes a re-run of a single cell reproducible, because its input list is fixed. By input scene scene A scene B scene C workers must coordinate over overlaps serial merge dominates By output cell cell 1 + list cell 2 + list cell 3 + list no worker touches another finished outputs, no merge The scene list per cell comes from one spatial query over the footprints table, computed once up front. Fixing that list is what makes a re-run of a single failed cell read exactly what it read the first time.

Prerequisites and environment assumptions

Terraform 1.6 or later with hashicorp/aws pinned at ~> 5.60. A container image containing GDAL, pinned by digest — a mosaicking result depends on the GDAL and PROJ versions that produced it, so a floating tag means two runs of the same job can disagree at the sub-pixel level. A footprints table in PostGIS with a GiST index on the geometry column. A bucket layout separating source scenes from mosaic outputs, per Object Storage for Raster and Vector Data.

Size the local storage before anything else. Mosaicking writes intermediate files, and a container with the default ephemeral storage will fail partway through on a large cell with a disk-full error that reads like a permissions problem. Estimate the intermediate footprint as roughly three times the uncompressed size of the inputs for one cell and provision accordingly.

Step-by-step provisioning

  1. Compute the partition and materialise it. Run the spatial query that assigns scenes to cells and write the result — one row per cell with its scene list — to a manifest object. The batch job array then reads cell n from the manifest by index, which keeps the job submission small and makes the partition auditable after the fact.

  2. Define a job queue with two compute environments. A spot environment for the bulk of the work and an on-demand environment as fallback, with the queue preferring spot. Mosaicking is interruptible if each cell is independent, which is the second benefit of output partitioning: an interrupted cell costs one cell’s work.

  3. Size the job definition from the cell, not from the average. Memory and local storage must accommodate the largest cell, because one worker failing on a disk-full error fails the whole coverage. If cell sizes vary wildly, split the manifest into size classes and submit two arrays with different job definitions rather than sizing everything for the worst case.

  4. Set a retry policy that distinguishes interruption from failure. A spot reclamation should retry; a GDAL error on a corrupt input should not, because it will fail identically every time and three retries of a twenty-minute job is an hour wasted. Batch retry strategies can match on the exit reason, and using that is the difference between resilience and a cost multiplier.

  5. Write outputs atomically and record completion per cell. Each worker writes to a staging key and moves on success, then records the cell as complete. A re-run then processes only incomplete cells, which turns a partially failed coverage into a short catch-up rather than a full re-run.

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

variable "subnet_ids" { type = list(string) }
variable "gdal_image_digest" { type = string }

resource "aws_batch_compute_environment" "spot" {
  compute_environment_name = "raster-mosaic-spot"
  type                     = "MANAGED"
  service_role             = aws_iam_role.batch_service.arn

  compute_resources {
    type                = "SPOT"
    allocation_strategy = "SPOT_CAPACITY_OPTIMIZED"
    max_vcpus           = 512
    subnets             = var.subnet_ids
    security_group_ids  = [aws_security_group.batch.id]
    instance_role       = aws_iam_instance_profile.batch.arn
    # Compute-optimised families: mosaicking is CPU and I/O bound, not
    # memory bound, once the per-cell working set fits.
    instance_type = ["c6i", "c6a", "c7i"]
  }
}

resource "aws_batch_job_queue" "mosaic" {
  name     = "raster-mosaic"
  state    = "ENABLED"
  priority = 1

  # Spot first; on-demand absorbs the tail when spot capacity is short, so a
  # coverage does not stall overnight waiting for cheap instances.
  compute_environment_order {
    order               = 1
    compute_environment = aws_batch_compute_environment.spot.arn
  }
  compute_environment_order {
    order               = 2
    compute_environment = aws_batch_compute_environment.ondemand.arn
  }
}

resource "aws_batch_job_definition" "mosaic_cell" {
  name = "mosaic-cell"
  type = "container"
  platform_capabilities = ["EC2"]

  container_properties = jsonencode({
    # Digest-pinned: mosaic output depends on the GDAL and PROJ versions that
    # produced it, so a floating tag lets two runs disagree sub-pixel.
    image = "${var.registry}/gdal-mosaic@sha256:${var.gdal_image_digest}"
    resourceRequirements = [
      { type = "VCPU", value = "8" },
      # Sized for the LARGEST cell, not the average: one disk-full worker
      # fails the whole coverage.
      { type = "MEMORY", value = "32768" },
    ]
    # Intermediates are roughly three times the uncompressed input size for
    # one cell. The default ephemeral volume is not enough and fails with a
    # disk-full error that reads like a permissions problem.
    ephemeralStorage = { sizeInGiB = 200 }
    jobRoleArn       = aws_iam_role.mosaic_task.arn
  })

  retry_strategy {
    attempts = 3
    # Retry a spot reclamation; do not retry a deterministic GDAL failure,
    # which would fail identically three times and waste an hour.
    evaluate_on_exit {
      action           = "RETRY"
      on_status_reason = "Host EC2*"
    }
    evaluate_on_exit {
      action    = "EXIT"
      on_reason = "*"
    }
  }

  timeout { attempt_duration_seconds = 7200 }
}

variable "registry" { type = string }
Retrying interruptions but not deterministic failures A finished job attempt is classified by its exit reason. A host-level termination, which is what a spot reclamation produces, is retried because the same work will succeed on another host and only the partial progress is lost. Any other failure reason — a corrupt input, a missing projection definition, a disk-full condition — exits immediately without retrying, because the failure is deterministic and three attempts at a twenty-minute job multiply the cost of a certain failure. Both paths record the cell outcome, so a subsequent run processes only the cells that did not complete. Attempt ends read the exit reason Host termination spot reclaimed — RETRY Any other reason deterministic — EXIT Record the cell outcome a re-run covers only incomplete cells

Verification

Run one cell first, alone, and inspect the output before submitting four hundred. gdalinfo on the result should report the expected extent, projection, band count and nodata value, and the overview levels you asked for. Open it next to two adjacent cells and check the seams: visible edges usually mean the cells were rendered with inconsistent resampling or with a colour balance computed per cell rather than across the coverage.

Then check the resource envelope on that single run: peak local disk usage against the provisioned ephemeral storage, and peak memory against the job definition. A cell that used 90 per cent of its disk will fail on the largest cell in the set. Finally submit the full array and watch the completion record rather than the queue — a coverage is done when every cell is marked complete, not when the queue is empty, because a failed cell also leaves the queue.

Run one cell first, and what to look at when it finishes Before submitting a whole coverage, run a single cell and inspect three things. First the output itself: extent, projection, band count, nodata value and the overview levels requested must all be what the job intended. Second the seams: opening the cell beside two neighbours shows immediately whether a colour balance or a resampling choice was computed per cell rather than across the coverage. Third the resource envelope: peak local disk and peak memory measured against what the job definition provisions, because a cell that used ninety per cent of its disk will fail on the largest cell in the set. The output itself extent · projection · band count · nodata · overview levels The seams against two neighbours a visible edge means something was computed per cell, not across the coverage The resource envelope a cell using 90% of its disk will fail on the largest cell in the set

Preventing recurrence

  • Keep the manifest as an artifact. The partition is the reproducibility boundary; storing it alongside the outputs means a coverage can be regenerated exactly, including the scene list per cell.
  • Alarm on cells completed versus cells submitted. The queue emptying is not success. The ratio is.
  • Pin the image by digest and record it in the manifest. Two coverages produced by different GDAL builds may differ subtly, and knowing which build made which output is the only way to explain it later.
  • Re-measure the largest cell after every ingest cycle. Cell sizes grow as coverage densifies, and the sizing that fit last quarter is where next quarter’s disk-full failure comes from.

Frequently Asked Questions

Is spot really safe for a multi-hour mosaicking job?

Yes, provided cells are independent and outputs are written atomically at the end. An interruption then costs one cell’s partial work, which is minutes, not the coverage. It is unsafe only if a worker’s output is an input to another worker — which is precisely what output-cell partitioning eliminates.

Why not use a single very large instance?

Because the job is bursty. A large instance sized for the peak sits idle between coverages, and the same work on a spot fleet costs a fraction. The exception is a genuinely serial workload that cannot be partitioned, which mosaicking is not.

How do I choose the output cell size?

Large enough that per-cell overhead — container start, index query, GDAL initialisation — is small against the work, and small enough that the largest cell’s intermediates fit comfortably in local storage. In practice a few minutes of work per cell is a good target; sub-minute cells spend most of their time starting.

What causes visible seams between cells?

Nearly always something computed per cell that should be computed across the coverage: a colour balance, a histogram stretch, or a resampling choice that differs by cell because of how inputs were ordered. Compute those globally, pass them to each worker as parameters, and the seams disappear.