Spatial IaC Terraform & Pulumi for GIS

Scheduled Drift Detection for Tile-Server Fleets

A fleet of tile servers spread across several autoscaling groups or ECS services is one of the hardest estates to keep honest, because the fleet legitimately rewrites its own launch templates, scaling numbers, and instance membership every day — and yet a single hand-edited security group or an out-of-band launch-template change can compromise the whole tier. This guide builds a scheduled job that runs refresh-only plans across every group in the fleet, distinguishes real drift from routine autoscaling churn, and routes only the dangerous signal to Slack or PagerDuty. It extends the classification and remediation model in Drift Detection and Remediation for Geospatial Estates and lives within CI/CD, Drift Detection & Policy Governance; the target scenario is the tile-server fleet whose scaling behaviour is defined in Auto-Scaling EC2 Instances for WMS Endpoints.

Symptom Identification and Triage

The signal that you need this job is either silence or noise, and both are failures. Silence is a fleet where a launch template was edited in the console three weeks ago, the change survived because no full instance replacement has happened since, and nobody knows — until an autoscaling event finally cycles instances onto the declared template and the manual fix vanishes mid-traffic. This is the insidious case for tile servers specifically: a heap or worker-count tweak applied to one node to relieve a hot WMS layer buys quiet for weeks, then evaporates the moment a spot interruption or scale-in replaces that instance, and the layer degrades again with no correlating deploy in the audit trail. Noise is a naive terraform plan scheduled against the fleet that exits 2 every single night because the ASG’s desired_capacity moved, the launch template’s $Latest version incremented, and the target group’s registered instance IDs rotated. A detector that pages on that noise is ignored within a week.

The triage question for any non-empty diff on a fleet is therefore always the same: is this a runtime-managed attribute the fleet is expected to move, or a configuration attribute only a human or an out-of-band process changes? The dangerous set is small and specific: a launch template edited outside code (a changed AMI, user-data, or instance profile), a security-group rule opened on the tile-server ingress or the PostGIS backend port, and a scaling policy or min/max bound altered in the console. The benign set is everything the scaling loop touches on its own.

A mixed fleet complicates the split because ASG-backed and ECS-backed tile servers churn on different attributes. On an autoscaling group the runtime-managed set is desired_capacity, the launch template $Latest pointer, and target-group instance membership. On an ECS service it is the running desired_count when application-autoscaling is attached, the deployment-controller’s rollout of a new task-set, and the platform version AWS resolves for a LATEST Fargate service. The dangerous ECS set mirrors the ASG one: a task definition revision registered by hand (a changed image tag, CPU/memory reservation, or environment block), a service’s assign_public_ip flipped on, or a security group swapped on the service’s network configuration. A single detector has to know, per resource type, which attributes to forgive.

Triage tree separating autoscaling churn from dangerous fleet drift A non-empty refresh-only plan across the tile-server fleet is triaged attribute by attribute. Decision: which attribute changed? Runtime-managed attributes — desired_capacity, the launch template Latest version pointer, and target-group registered instance IDs — are benign autoscaling churn, suppressed with ignore_changes and reconciled by refresh, and do not page. Configuration attributes — a launch template body edit such as AMI or user-data, a security-group ingress rule, or a scaling min/max bound — are dangerous drift that pages Slack or PagerDuty for human review. Non-empty plan exit code 2 Which attribute? Runtime-managed desired_capacity LT $Latest pointer target-group members Benign churn suppress + refresh, no page Config drift LT body: AMI / user-data security-group rule scaling min / max bound Page Slack / PagerDuty

Prerequisites and Environment Assumptions

  • Engine and providers pinned. Terraform >= 1.10.0 with hashicorp/aws pinned to ~> 5.60, matching the version the fleet’s deploy pipeline uses. The scheduled detector and the deploy must share provider versions or the detector reports phantom drift from shifting attribute defaults.
  • Read-only OIDC role. The scheduled job assumes an IAM role that can Describe* the ASGs, launch templates, ECS services, target groups, and security groups and read the state backend — and nothing that mutates. A drift detector must never hold apply permissions.
  • Isolated backend per environment. The job reads the same per-environment state path the fleet was deployed from, per State Backend Selection. A refresh-only plan takes no write lock, so it runs with -lock=false and never queues behind a live deploy.
  • ignore_changes already declared on runtime attributes. The fleet’s own Terraform should carry lifecycle { ignore_changes = [desired_capacity] } on the ASG so scaling never registers as drift. If it does not, add it first — the detector inherits whatever the resource declares.
  • An alert route. A Slack incoming-webhook URL or a PagerDuty Events API v2 routing key, stored as a CI secret, never in the repository.

Step-by-Step Remediation

  1. Suppress autoscaling churn at the resource, not the detector. Before scheduling anything, make the fleet’s Terraform ignore the attributes the scaling loop owns. This is the single change that turns the detector from noise into signal, because it removes benign drift from the diff entirely rather than filtering it after the fact.

    resource "aws_autoscaling_group" "tile_fleet" {
  name                = "wms-tile-fleet"
  min_size            = 3
  max_size            = 20
  desired_capacity    = 4
  vpc_zone_identifier = var.private_subnet_ids

  launch_template {
    id      = aws_launch_template.tile.id
    version = aws_launch_template.tile.latest_version # pin to the managed version, not "$Latest"
  }

  lifecycle {
    # The scaling loop owns desired_capacity; the detector must not treat it as drift.
    ignore_changes = [desired_capacity]
  }
}

  2. Schedule a refresh-only, detailed-exit-code plan across the fleet. Run the plan for each environment on a cadence matched to risk — hourly for the security-group-sensitive tier. The detailed exit code is what makes the result machine-routable: 0 clean, 2 drift, 1 error.

    # .github/workflows/fleet-drift.yml
name: fleet-drift-detection
on:
  schedule:
    - cron: "0 * * * *"       # hourly
  workflow_dispatch: {}
permissions:
  id-token: write             # OIDC — no static keys
  contents: read
jobs:
  detect:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/fleet-drift-readonly
          aws-region: us-east-1
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.10.5   # matches the fleet deploy pipeline
      - run: terraform init -input=false -backend-config=env/prod.s3.tfbackend
      - name: Refresh-only plan
        id: plan
        run: |
          set +e
          terraform plan -refresh-only -detailed-exitcode -lock=false \
            -target=aws_autoscaling_group.tile_fleet \
            -target=aws_launch_template.tile \
            -target=aws_security_group.tile_ingress \
            -out=fleet.plan
          echo "code=$?" >> "$GITHUB_OUTPUT"
          set -e

  3. Handle the exit code and route only real drift. Convert the plan to JSON, keep only resource changes (dropping no-op and read actions), and page only when a meaningful change remains. This is where a targeted plan and ignore_changes pay off: the JSON is empty on a churn-only night and populated only when a human-caused change slipped in.

    # exit 1 is a real error — surface it distinctly from drift
if [ "$CODE" = "1" ]; then echo "plan errored"; exit 1; fi
if [ "$CODE" = "2" ]; then
  terraform show -json fleet.plan \
    | jq '[.resource_changes[]
           | select(.change.actions != ["no-op"] and .change.actions != ["read"])
           | {addr: .address, actions: .change.actions}]' > drift.json
  if [ "$(jq 'length' drift.json)" -gt 0 ]; then
    curl -sS -X POST "$SLACK_WEBHOOK" -H 'content-type: application/json' \
      --data "$(jq -Rs --slurpfile d drift.json \
        '{text: ("Tile-fleet drift detected:\n" + ($d[0]|tostring))}' <<< "")"
    exit 1   # fail the run so the alert is also visible in CI history
  fi
fi

  4. Reconcile the classified drift. For a dangerous change, decide promote-or-revert first, then act with the narrowest tool. A launch-template body edited by hand that the team rejects is reverted with a targeted apply (terraform apply -target=aws_launch_template.tile); a widened security-group rule is restored the same way; a resource that exists but was never in state is adopted with terraform import rather than destroyed. Refresh-only reconciliation (terraform apply -refresh-only) is reserved for churn you have decided to accept — never for the security-group or launch-template changes, which must be resolved as configuration.

    The order matters most for launch templates and task definitions, because both are versioned and a blind apply can strand instances. If a launch-template body was edited in the console and you revert it, the ASG keeps running instances on whatever version they booted with until the next replacement, so pair the targeted apply with an instance refresh (aws autoscaling start-instance-refresh) once the declared version is restored, ensuring the fleet actually converges rather than merely recording that it should. For an ECS service, reverting a hand-registered task definition revision means pointing the service back at the code-managed revision and letting the deployment controller roll it out; forcing a new deployment (aws ecs update-service --force-new-deployment) confirms tasks relaunch on the declared definition. In both cases the detector should re-run immediately after remediation to prove the diff is genuinely empty, not merely suppressed.

Verification

Confirm the detector both catches real drift and stays quiet on churn:

  • Inject a benign change, expect silence. Manually set desired_capacity up by one (or let a scaling event fire) and confirm the next scheduled run exits 0 and posts nothing — proving ignore_changes suppresses churn.
  • Inject a dangerous change, expect a page. Add a throwaway ingress rule to the tile-server security group in the console, then trigger the workflow with workflow_dispatch. The run must exit 2, the drift.json must name aws_security_group.tile_ingress, and the alert must land in the channel. Remove the rule afterward.
  • Confirm the read-only boundary. Attempt any mutating call with the detector’s role (for example aws autoscaling set-desired-capacity) and confirm it is denied — the detector must not be able to change the fleet.
  • Confirm the tile endpoint is unaffected. After reconciliation, probe a live tile (curl -I https://tiles.example.com/wms/10/301/384.png) and assert a 200 with image/png, verifying remediation did not disturb serving.

Preventing Recurrence

The durable fix is to make the fleet unchangeable by hand and self-reporting when it changes anyway. Enforce a policy-as-code rule that fails any launch template or security group whose managed-by tag is absent, per Policy as Code for Spatial Resources, and gate all fleet changes through the OIDC-authenticated deploy pipeline in GitHub Actions OIDC for Terraform Spatial Deploys so console edits require an exception that itself trips the detector. Keep the scheduled detector permanently on as the backstop: it is the only thing that notices a manual change during the weeks between the edit and the instance replacement that would otherwise erase it silently.

Frequently Asked Questions

Why run a refresh-only plan instead of a normal plan?

A normal plan also proposes to enact any configuration change sitting on the branch but not yet applied, so its diff mixes real drift with pending deploys. -refresh-only compares state against live reality alone, isolating out-of-band changes, which is exactly what a drift detector must report.

How do I stop autoscaling from paging me every night?

Suppress the runtime-managed attributes at the resource with lifecycle { ignore_changes = [desired_capacity] }, pin the launch template version to the managed version rather than $Latest, and filter target-group membership out of the routed diff. Once the scaling loop’s own writes are excluded, only human-caused changes remain to alert on.

Should the drift job be allowed to apply the fix automatically?

No. The scheduled job assumes a read-only role and can never mutate the fleet. Dangerous drift needs a promote-or-revert decision before any apply, and auto-applying would silently reimpose whatever the repository last said — which may be stale — or launder a manual change into the accepted baseline.

Does the detector interfere with an in-progress deploy?

No. A refresh-only plan takes no write lock, so it runs with -lock=false and neither blocks nor is blocked by a concurrent apply. It only reads state and describes resources.