Spatial IaC Terraform & Pulumi for GIS

Auto-Scaling EC2 Instances for WMS Endpoints: Incident Response & State Recovery

When a Web Map Service (WMS) auto-scaling group starts shedding 503 Service Unavailable responses during a scale-out, the root cause is almost always that the fleet scaled faster than GeoServer could become render-ready, or that the infrastructure-as-code state no longer matches the live cloud graph. This guide is a focused incident-response runbook for restoring a healthy, codified WMS tier and is part of the Compute Node Orchestration workflow within the broader Geospatial Resource Provisioning domain. WMS workloads carry distinct scaling signatures — JVM-heavy startup latency, raster tile cache initialization, and connection-pool saturation against the backing spatial database — so the remediation steps below differ sharply from those for a stateless web app, and they assume the data tier from PostGIS Cluster Provisioning is already healthy.

Symptom Identification and Triage

WMS auto-scaling incidents surface through three observable failure modes, each with a distinct signal you can confirm before touching the infrastructure code.

  • Target-group deregistration storms. New instances flap between initial and unhealthy in aws elbv2 describe-target-health, and clients see intermittent 503s. This means health checks validate TCP connectivity rather than an HTTP GetCapabilities response, so traffic routes to a JVM that has not finished warming up.
  • Connection-pool exhaustion on rapid scale-out. Instances pass health checks but tile requests stall with 504 gateway timeouts, and PostGIS logs show FATAL: remaining connection slots are reserved. The auto-scaling group expanded past the database connection ceiling because no pooling intermediary (PgBouncer or RDS Proxy) sits between the fleet and the PostGIS database.
  • Orchestration state drift. terraform plan reports changes you did not author, scaling policies reference a deprecated launch-template version, or instances hang in Pending:Wait. This is the signature of out-of-band console edits, ad-hoc security-group changes, or lifecycle hooks whose heartbeat timeout is shorter than the JVM-plus-cache hydration window.

Walk the triage path from the externally observable HTTP status, to the target-group health state, to the auto-scaling activity history, and finally to the IaC plan diff — only the last of these confirms drift.

WMS Auto-Scaling Incident Triage Decision Tree From the externally observable 503 or 504 status, three signals each map to one root cause and one remediation step. Unhealthy, flapping targets indicate a TCP health check rather than a GetCapabilities probe, resolved in Step 3. Healthy targets that still return 504 with PostGIS logging reserved connection slots indicate connection-pool exhaustion on scale-out, resolved by a pooled data path. Instances stuck in Pending:Wait or a non-empty plan indicate a lifecycle-hook timeout or out-of-band state drift, resolved in Steps 2 and 4. 503 / 504 at the WMS endpoint triage from the outside in Targets unhealthy / flapping describe-target-health · 503s Targets healthy, tiles 504 FATAL: connection slots reserved Stuck Pending:Wait / drift plan shows unauthored changes Health check is TCP, not GetCapabilities PostGIS connection-pool exhaustion on scale-out Lifecycle-hook timeout / out-of-band state drift Step 3 — ELB health check GetCapabilities + grace period Prevent — pooled path PgBouncer / RDS Proxy endpoint Steps 2 & 4 — refresh state extend hook heartbeat

Prerequisites and Environment Assumptions

This runbook assumes the following provider versions, backend, and permissions are in place before you begin:

  • Terraform >= 1.5 with the hashicorp/aws provider >= 5.0, or Pulumi with @pulumi/aws >= 6.0. Note that @pulumi/aws is used directly here; the higher-level @pulumi/awsx package does not expose an autoscaling.AutoScalingGroup construct.
  • A remote, lockable state backend (S3 + DynamoDB lock table, or Pulumi Cloud) so concurrent applies cannot race during recovery — see State Backend Selection for the locking rationale.
  • IAM permissions for autoscaling:*, elasticloadbalancing:*, ec2:DescribeInstances, ec2:*LaunchTemplate*, and read access to the state bucket. Scope these against the least-privilege boundaries defined in IAM Role Mapping for GIS.
  • A reachable WMS GetCapabilities endpoint on each node (default :8080/geoserver/ows) and CLI access to query CloudWatch and the load balancer.

Step-by-Step Remediation

Recover the fleet by prioritising infrastructure consistency over immediate capacity, so you do not amplify the incident by terminating healthy nodes that are still serving tiles.

  1. Freeze scale-in. Suspend the Terminate and ReplaceUnhealthy processes so reconciliation cannot kill nodes mid-request:

    aws autoscaling suspend-processes \
  --auto-scaling-group-name wms-prod-asg \
  --scaling-processes Terminate ReplaceUnhealthy

  2. Lock and refresh state. Reconcile the IaC backend with the live AWS inventory to expose drift before you change anything. A plain refresh-only plan never mutates resources:

    terraform plan -refresh-only -lock=true -out=drift.tfplan

  3. Re-converge the launch template and health check. Apply the corrected configuration below. The decisive geospatial fix is the ELB health-check type pointed at a real GetCapabilities request plus a health_check_grace_period long enough to absorb JVM warm-up and tile-cache hydration — a TCP check would mark a cold GeoServer “healthy” and route tiles to it prematurely.

    terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0"
    }
  }
}

resource "aws_autoscaling_group" "wms_asg" {
  name                      = "wms-prod-asg"
  vpc_zone_identifier       = var.private_subnet_ids
  min_size                  = 2
  max_size                  = 12
  desired_capacity          = 3
  health_check_type         = "ELB"
  health_check_grace_period = 300 # Absorbs JVM warmup & raster cache init
  wait_for_capacity_timeout = "10m"

  launch_template {
    id      = aws_launch_template.wms_lt.id
    version = "$Latest"
  }

  lifecycle {
    ignore_changes = [desired_capacity] # let scaling policy own capacity
  }

  # Rolling replace so active tile requests are never severed abruptly
  instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 75
      instance_warmup        = 300
    }
  }
}

resource "aws_lb_target_group" "wms_tg" {
  name     = "wms-prod-tg"
  port     = 8080
  protocol = "HTTP"
  vpc_id   = var.vpc_id

  health_check {
    path                = "/geoserver/ows?service=WMS&version=1.3.0&request=GetCapabilities"
    port                = "8080"
    protocol            = "HTTP"
    healthy_threshold   = 3
    unhealthy_threshold = 3
    timeout             = 10
    interval            = 30
    matcher             = "200"
  }

  deregistration_delay = 60 # let in-flight map requests drain
}
    terraform apply -target=aws_lb_target_group.wms_tg \
                -target=aws_autoscaling_group.wms_asg -lock=true

  4. Verify lifecycle-hook timeouts. Confirm every hook’s heartbeat exceeds the maximum JVM-plus-cache window so instances no longer strand in Pending:Wait; extend the hook rather than shortening the grace period.

  5. Resume normal scaling. Once targets report healthy, lift the suspension:

    aws autoscaling resume-processes --auto-scaling-group-name wms-prod-asg

For teams running Pulumi, the equivalent stack enforces identical constraints with stack-level type safety and integrates with the same Compute Node Orchestration workflow:

import * as aws from "@pulumi/aws"; // pin @pulumi/aws >= 6.0 in package.json
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();
const vpcId = config.require("vpcId");
const amiId = config.require("amiId");

const wmsTargetGroup = new aws.lb.TargetGroup("wms-tg", {
  port: 8080,
  protocol: "HTTP",
  vpcId: vpcId,
  healthCheck: {
    path: "/geoserver/ows?service=WMS&version=1.3.0&request=GetCapabilities",
    port: "8080",
    protocol: "HTTP",
    healthyThreshold: 3,
    unhealthyThreshold: 3,
    timeout: 10,
    interval: 30,
    matcher: "200",
  },
  deregistrationDelay: 60,
});

const wmsLaunchTemplate = new aws.ec2.LaunchTemplate("wms-lt", {
  imageId: amiId,
  instanceType: "c6i.2xlarge",
  userData: Buffer.from(`#!/bin/bash
systemctl enable geoserver
systemctl start geoserver`).toString("base64"),
  iamInstanceProfile: { name: wmsInstanceProfile.name },
  vpcSecurityGroupIds: [wmsSecurityGroup.id],
});

const wmsASG = new aws.autoscaling.Group("wms-asg", {
  vpcZoneIdentifiers: privateSubnetIds,
  desiredCapacity: 3,
  minSize: 2,
  maxSize: 12,
  healthCheckType: "ELB",
  healthCheckGracePeriod: 300,
  launchTemplate: {
    id: wmsLaunchTemplate.id,
    version: "$Latest",
  },
  targetGroupArns: [wmsTargetGroup.arn],
  instanceRefresh: {
    strategy: "Rolling",
    preferences: {
      minHealthyPercentage: 75,
      instanceWarmup: "300",
    },
  },
});

The pattern guarantees an instance only receives tile traffic once GeoServer can answer a real WMS request, not merely accept a TCP connection:

WMS Node Lifecycle: Scale-Out to Render-Ready A CloudWatch metric on queue depth and latency drives an ASG scale-out that launches an instance from the launch template. A 300-second health-check grace period absorbs JVM warmup and raster-cache hydration, after which a GetCapabilities probe gates entry to service: a 200 response promotes the node to InService where it receives tile traffic, while any other response fails the health check and replaces the node, looping back to a fresh launch so traffic only ever reaches a render-ready GeoServer. CloudWatch metric queue depth · latency ASG scale-out Launch from template Health-check grace 300s · JVM warmup GetCapabilities 200? yes InService receives tiles no Fail health check, replace replace cold node

Verification

Confirm the fix is live from both the orchestration layer and the map endpoint itself, not just the AWS console:

  • Target health is application-aware. Every registered target should report healthy, and the reason should reflect the HTTP probe, not a raw TCP accept:

    aws elbv2 describe-target-health \
  --target-group-arn "$(terraform output -raw wms_tg_arn)" \
  --query 'TargetHealthDescriptions[].TargetHealth.State'

  • The endpoint answers a real WMS request. A 200 with a WMS_Capabilities body — not just an open socket — proves the node is render-ready:

    curl -s -o /dev/null -w '%{http_code}\n' \
  "https://maps.example.org/geoserver/ows?service=WMS&version=1.3.0&request=GetCapabilities"

  • State is clean. A follow-up terraform plan -refresh-only reports No changes, confirming the live graph matches code and the drift is resolved.

  • The database is no longer saturated. PostGIS no longer logs remaining connection slots are reserved, confirming the pooled endpoint absorbed the scale-out.

Preventing Recurrence

Encode the fix so the failure modes cannot reappear through console edits or a future module change:

  • Policy-as-code gate. Add an OPA/Conftest or Sentinel rule that rejects any aws_autoscaling_group whose health_check_type is not ELB, or whose health_check_grace_period is below the measured JVM warm-up floor, and run it in the pull-request pipeline alongside the checks described in Module Design Patterns.
  • Scheduled drift detection. Run terraform plan -refresh-only -detailed-exitcode on a cron cadence and alert on a non-zero diff so out-of-band changes are caught before the next incident.
  • Pin the pooled data path. Require the launch template to point GeoServer at a PgBouncer or RDS Proxy endpoint rather than the raw cluster, so a scale-out can never exhaust PostGIS connection slots.
  • Lock down the runtime image. Build AMIs through automated Packer pipelines that validate JVM heap allocation and connection-pool limits against the GeoServer Deployment Patterns baseline before promotion, and keep style and dataset access scoped to Object Storage for Raster/Vector Workloads via least-privilege instance profiles.

For deeper background, the official AWS Auto Scaling health checks documentation covers ELB integration, and the GeoServer production hardening guide covers garbage-collection tuning, tile-cache eviction, and thread-pool sizing.

Frequently Asked Questions

Why do new WMS instances pass health checks but still return 503?

Because the health check validates TCP connectivity instead of an HTTP GetCapabilities response. Switch the target group to an HTTP check against /geoserver/ows?...request=GetCapabilities with a 200 matcher so a node only enters service once GeoServer can render.

How long should the health-check grace period be?

Long enough to cover the slowest observed JVM start plus raster tile-cache hydration — commonly 240-360 seconds. Measure it on a cold instance and set health_check_grace_period (and instance_warmup) above that floor rather than guessing.

How do I stop scale-out from exhausting PostGIS connections?

Route every GeoServer node through a PgBouncer or RDS Proxy endpoint and cap per-instance pool sizes, so the maximum fleet size multiplied by the pool size stays below the PostGIS connection ceiling.