Spatial IaC Terraform & Pulumi for GIS

Spot Instance Strategies for Batch Geoprocessing Fleets

Batch geoprocessing is the ideal spot workload and the one teams most often get burned by. Ideal, because reprojecting a scene, building overviews or computing a zonal statistic is stateless, restartable and tolerant of a two-minute warning. Burned, because the naive configuration — one instance type, one availability zone, no checkpointing — turns a capacity reclamation into a failed coverage, and because the same interruption tolerance that makes spot safe for a five-minute task makes it treacherous for a four-hour one that writes nothing until the end. This guide extends Compute Node Orchestration within Geospatial Resource Provisioning, and its core claim is that spot safety is a property of your work unit, not of your instance configuration.

Making the work unit interruption-safe

Before any fleet configuration, size the work unit so that losing one is cheap. Three properties make a geoprocessing task safe to interrupt.

It is idempotent. Re-running it produces the same output. For raster work this usually means writing to a deterministic output key derived from the input identity and the processing parameters, so a re-run overwrites rather than duplicating.

It writes atomically at the end. Convert to a staging key and move on success. An interrupted task then leaves no partial output, and a consumer never opens a truncated GeoTIFF — the failure that produces the most confusing bug reports, because a partially written raster often opens successfully and returns wrong pixels.

It is short relative to the interruption rate. A task of five to fifteen minutes loses at most that much work. A four-hour task in a pool with a two per cent hourly interruption rate has a meaningful chance of never completing at all, because each attempt restarts from zero. When a task must be long, checkpoint it: write intermediate state at intervals and resume from the last checkpoint.

Task length, interruption, and the value of checkpointing Three timelines. A short task interrupted partway loses only the minutes already spent, and the retry runs to completion on another host. A long task with no checkpointing restarts from the beginning after every interruption, so in a pool with a meaningful interruption rate it can consume many hours of compute without ever producing an output. The same long task with periodic checkpoints resumes from the last checkpoint after an interruption, which converts it into a series of short segments and restores the safety property that made spot attractive. Short task — safe interrupted retry completes minutes lost Long task, no checkpoint — unsafe restarts from zero, again Long task, checkpointed — safe segment 1 segment 2 resume seg 2 segment 3 only one segment lost Spot safety is a property of the work unit, not of the fleet configuration.

Prerequisites and environment assumptions

Terraform 1.6 or later with hashicorp/aws pinned at ~> 5.60. A container image with the geoprocessing toolchain, digest-pinned so that a reclaimed task retried on another host runs identical code. A queue holding the work, since spot capacity arrives and disappears and a queue is what decouples submission from execution. Subnets in at least three availability zones — capacity is per zone, and a single-zone fleet is a fleet that stops when that zone is short.

Know the interruption characteristics of the instance families you intend to use. Interruption frequency varies enormously by family, size and zone, and the difference between two similar families can be an order of magnitude. Choosing families by price alone is how a fleet ends up thrashing.

Step-by-step fleet configuration

  1. Diversify across families, sizes and zones. The single most effective change. A fleet permitted to use eight instance types across three zones can almost always find capacity somewhere; a fleet pinned to one type in one zone is at the mercy of that one pool. For geoprocessing, compute-optimised families of similar vCPU-to-memory ratio are interchangeable, so diversification costs nothing in behaviour.

  2. Use a capacity-optimised allocation strategy. Selecting the pool with the deepest available capacity rather than the lowest price reduces interruptions substantially, and the price difference between the cheapest pool and the deepest one is usually small. Optimising purely for price maximises interruptions, which is a false economy once retries are counted.

  3. Keep a small on-demand base. A fleet that is entirely spot can reach zero capacity. A base of one or two on-demand instances guarantees the queue always drains, slowly, even during a capacity squeeze — which turns “the pipeline stopped overnight” into “the pipeline ran slowly overnight”.

  4. Handle the interruption notice. The two-minute warning is enough to finish or checkpoint a short task and to return the message to the queue rather than letting it time out. A handler that catches the notice, stops accepting new work and releases the current message makes reclamation nearly invisible.

  5. Set a maximum price only if you mean it. Capping the spot price below the on-demand rate is usually unnecessary — spot is already discounted — and a cap set too low silently removes pools from consideration, which reads as a capacity problem.

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.60" }
  }
}

variable "subnet_ids" {
  type        = list(string)
  description = "At least three zones — capacity is per zone."
}

resource "aws_ec2_fleet" "geoprocessing" {
  type = "maintain"

  launch_template_config {
    launch_template_specification {
      launch_template_id = aws_launch_template.geoproc.id
      version            = aws_launch_template.geoproc.latest_version
    }

    # Diversification is the single most effective interruption control.
    # These families are interchangeable for CPU-bound geoprocessing, so
    # accepting all of them costs nothing in behaviour.
    dynamic "override" {
      for_each = [
        "c6i.4xlarge", "c6a.4xlarge", "c7i.4xlarge",
        "c6i.8xlarge", "c6a.8xlarge", "c7i.8xlarge",
        "m6i.4xlarge", "m6a.4xlarge",
      ]
      content {
        instance_type = override.value
        subnet_id     = var.subnet_ids[0]
      }
    }
  }

  spot_options {
    # Deepest capacity, not lowest price. Optimising purely for price
    # maximises interruptions, which costs more once retries are counted.
    allocation_strategy         = "capacityOptimized"
    instance_interruption_behavior = "terminate"
  }

  # A small on-demand base means the queue always drains, slowly, even when
  # spot capacity is unavailable in every pool.
  target_capacity_specification {
    default_target_capacity_type = "spot"
    total_target_capacity        = 40
    on_demand_target_capacity    = 2
    spot_target_capacity         = 38
  }
}
#!/usr/bin/env bash
# Interruption handler, run on each worker. The two-minute notice is enough to
# release the in-flight message so another host picks it up immediately,
# instead of waiting for the visibility timeout to expire.
set -euo pipefail

TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 300")

while true; do
  code=$(curl -s -o /dev/null -w '%{http_code}' \
    -H "X-aws-ec2-metadata-token: $TOKEN" \
    http://169.254.169.254/latest/meta-data/spot/instance-action)

  if [ "$code" = "200" ]; then
    # Stop accepting new work, checkpoint the current task, and return the
    # message to the queue so it is redelivered now rather than in 15 minutes.
    touch /var/run/geoproc/drain
    /usr/local/bin/geoproc-checkpoint --flush
    /usr/local/bin/geoproc-release-message
    exit 0
  fi
  sleep 5
done
Using the interruption notice to release work immediately When the interruption notice appears the worker performs three actions within the two-minute window. It sets a drain flag so no new work is accepted. It flushes a checkpoint of the current task so a resumed attempt does not restart from zero. It explicitly returns the in-flight queue message, which makes the message immediately visible to another host. Without that last step the message stays invisible until its visibility timeout expires, which for a long-running geoprocessing task can be fifteen minutes of idle delay before anything retries it. Notice received two minutes left Drain accept no new work Checkpoint resume, do not restart Release the message visible again immediately Another host resumes seconds, not minutes Without the release, the message stays hidden until its visibility timeout expires — which for a long geoprocessing task is fifteen idle minutes.

Verification

Do not wait for a real interruption to find out whether the handler works. Simulate one — the fleet’s interruption simulation, or a manual invocation of the handler — and confirm three things: the drain flag stops new work, the checkpoint is written, and the message becomes visible on the queue immediately rather than after the visibility timeout.

Then measure the fleet under real load. The numbers worth tracking are interruptions per hundred task-hours, the ratio of tasks started to tasks completed, and the fraction of capacity currently served by on-demand. A completion ratio well below one means tasks are being lost rather than retried, and a persistently high on-demand fraction means the diversification list is too narrow for the capacity actually available.

Three numbers that describe a spot fleet honestly Interruptions per hundred task-hours characterises the pools the fleet actually selected, and is the number to compare when changing the instance-type list. Tasks started divided by tasks completed distinguishes a healthy interruption rate, where retries finish the work, from a fleet that is thrashing and losing tasks; a ratio well below one means work is being lost rather than retried. The fraction of capacity currently served by on-demand shows whether the diversification list is wide enough for the capacity that exists — a persistently high fraction means the fleet is falling back rather than finding spot. Interruptions per 100 task-hours describes the pools you selected compare it when the instance-type list changes Started / completed healthy interruption, or thrashing well below one means work is lost, not retried Fraction served on demand is the diversification list wide enough persistently high means falling back, not finding spot

Preventing recurrence

  • Alarm on the started-to-completed ratio. It is the metric that distinguishes a healthy interruption rate from a fleet that is thrashing, and it moves before the cost does.
  • Review the instance-type list quarterly. Families are added and retired, and a list written two years ago excludes the pools with the deepest capacity today.
  • Cap the retry count and route exhausted work to a dead-letter destination. A task failing for a deterministic reason will fail identically on every host, and without a cap it will consume spot capacity indefinitely, which is described further in Provisioning Batch Compute for Raster Mosaicking.
  • Tag spot capacity distinctly. Cost attribution and the anomaly detection in Budget Alerts and Anomaly Detection for Geospatial Spend both depend on being able to separate spot from on-demand.

Frequently Asked Questions

Is spot appropriate for anything user-facing?

Not for the request path. A tile renderer serving interactive maps should be on-demand or reserved capacity, because an interruption there is a user-visible error. Spot belongs to the batch tier, where the work is queued and a delayed task is invisible.

How long is too long for a single spot task?

Longer than about thirty minutes deserves checkpointing. The rule is that expected work lost per interruption should be small against the task’s total duration, and beyond half an hour without checkpoints that stops being true in most pools.

Does a low maximum price save money?

Rarely, and it usually costs. Spot is already discounted, and a low cap removes pools from consideration, producing what looks like a capacity shortage. Leave the price uncapped and control cost through instance selection and completion efficiency instead.

What if my geoprocessing needs GPUs?

The same principles apply with a narrower pool: GPU capacity is scarcer and interruptions are more frequent, so shorter work units and checkpointing matter more, and a larger on-demand base is usually justified because the alternative is a queue that does not drain at all.