Drift Detection and Remediation for Geospatial Estates

Out-of-band changes are inevitable on a running geospatial platform: someone tunes a PostGIS index by hand during an incident, a colleague flips a CRS setting in the GeoServer admin console, autoscaling rewrites a launch template, and a lifecycle rule quietly transitions a tile bucket to a colder storage class. Each of these leaves the declared infrastructure and the live estate disagreeing, and the deployment ledger no longer describes reality. This discipline — detecting that divergence early, classifying it, and reconciling it deliberately — is the operational backbone of CI/CD, Drift Detection & Policy Governance, and it interlocks tightly with the deploy machinery in Pipeline Orchestration for Spatial Deployments and the preventive rules in Policy as Code for Spatial Resources. This guide builds a scheduled drift-detection system for spatial estates and a graded remediation playbook that distinguishes cosmetic noise from access-and-availability risk.

Why Geospatial Estates Drift

Every long-lived estate accumulates drift, but geospatial platforms accumulate it faster and in more damaging ways because the resources are operationally poked far more often than a typical web tier. Four sources dominate. The first is manual database tuning: a PostGIS cluster serving slow tile queries invites an on-call engineer to add a GiST index, bump work_mem for a heavy ST_Intersects join, or rewrite a parameter group in the console mid-incident. That change is real, it is load-bearing, and it exists nowhere in code. The second is console-level configuration of the map servers themselves — a coverage store repointed at a new raster path, a layer’s declared CRS edited from EPSG:3857 to EPSG:4326, a JVM heap setting changed on one GeoServer node but not its peers — none of which the IaC engine models unless GeoServer is fully provisioned declaratively.

The third source is automation you deliberately delegated. Autoscaling groups rewrite desired capacity, replace launch template versions, and cycle instances in a tile-server fleet; a blue/green deploy swaps a target group; a spot interruption forces a replacement. These are legitimate runtime behaviours, but they mutate exactly the attributes Terraform or Pulumi also manages, so a naive plan reports them as drift on every run. The fourth is lifecycle and policy machinery acting on object storage: an S3 lifecycle rule transitions a cold tile pyramid to Glacier, intelligent-tiering moves raster objects between access tiers, and a bucket policy is patched out-of-band to grant a new consumer. The estate that most needs drift detection is therefore also the one that generates the most benign, expected drift — which is precisely why classification, not raw detection, is the hard part.

Drift is dangerous in geospatial contexts for a specific reason: the resource graph is deep and the failure modes are silent. A CRS edited in the console does not throw an error; it renders tiles in the wrong place. A security-group rule widened by hand does not fail a health check; it exposes a PostGIS port. A launch template edited outside code survives until the next full replacement, at which point the fleet reverts to the declared image and the manual fix vanishes. Detection has to run on a schedule, independent of deploys, because the interval between a console edit and the next planned apply can be weeks — and by then nobody remembers the change.

Detection Mechanics: Exit Codes and Diff Surfaces

Scheduled detection rests on one primitive: a plan that reads live state, compares it to the declared configuration, and signals divergence through a machine-readable exit code rather than human-readable text. For Terraform the canonical form is terraform plan -detailed-exitcode, which returns 0 for no changes, 2 for a non-empty diff, and 1 for an error. Running it with -refresh-only restricts the comparison to state-versus-reality, so the job reports drift without proposing to enact configuration changes that a feature branch may have introduced but not yet merged. For Pulumi the equivalent is pulumi preview --diff --expect-no-changes, which exits non-zero when the live estate no longer matches the checkpoint. Both must run against the isolated, per-environment backend described in State Backend Selection; a drift job pointed at the wrong state path produces confident nonsense.

A third tool, driftctl, complements the engine-native checks by scanning the cloud account for resources that exist but are not tracked in state at all — the class of drift a plan cannot see because the resource was never in the graph. On a geospatial estate this catches the raster bucket someone created by hand for a one-off ingest, the orphaned EBS volume left by a replaced tile node, and the security group added out-of-band. Between them the two categories cover the full drift surface: plan/preview finds managed resources whose attributes diverged, and driftctl finds unmanaged resources that should be under code.

The detection cadence should match how fast a given tier can hurt you. Network and IAM resources — security groups fronting PostGIS, bucket policies on raster stores — warrant hourly checks because an exposed port is an active liability. Compute and tile-server configuration can run nightly. Object-storage lifecycle state, which drifts slowly and predictably, can run daily or weekly. Splitting the schedule by tier also keeps each run small and fast, so a slow raster-catalog refresh never delays the security-group scan.

Classifying Benign Versus Dangerous Drift

A drift job that pages on every non-zero exit is worse than no job at all, because the fleet’s own autoscaling guarantees a steady stream of alerts and the team learns to ignore them. The value is in the classifier that sits between detection and notification. The reliable dividing line is: does the drift change access, availability, or correctness, or does it only change a runtime-managed attribute the platform is expected to move on its own?

Benign drift is dominated by autoscaling churn — desired_capacity, the current launch template version an ASG points at, instance IDs in a target group — and by object-storage tier transitions that a lifecycle rule performed exactly as declared. The correct handling is to suppress these attributes from the diff (via ignore_changes in Terraform lifecycle blocks or Pulumi ignoreChanges) so they never register as drift in the first place, and to reconcile whatever remains with a refresh so state re-reads the current value. Dangerous drift is everything that a person changed with intent the code does not reflect: a security-group ingress opened to 0.0.0.0/0 on port 5432, a bucket policy that added a principal, a GeoServer CRS or datastore edit, a PostGIS parameter changed in the console, an IAM role whose trust policy was widened. These must page, must not auto-reconcile, and must be resolved by deciding whether the manual change becomes code or gets reverted.

Runnable Configuration

The job below runs a refresh-only, detailed-exit-code plan on a schedule from GitHub Actions, authenticates with short-lived OIDC credentials rather than static keys, and posts to an alerting webhook only when the exit code signals real drift. It pins the Terraform version and the provider so the scheduled run reproduces the same comparison the deploy pipeline uses; an unpinned provider is itself a source of phantom drift as attribute defaults shift between releases.

# .github/workflows/drift-detect.yml
name: scheduled-drift-detection
on:
  schedule:
    - cron: "0 * * * *"        # hourly for the network/IAM-sensitive estate
  workflow_dispatch: {}

permissions:
  id-token: write               # OIDC federation — no static AWS keys
  contents: read

jobs:
  drift:
    runs-on: ubuntu-latest
    env:
      TF_IN_AUTOMATION: "true"
    steps:
      - uses: actions/checkout@v4          # pinned major
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/drift-detect-readonly
          aws-region: us-east-1
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.10.5        # pin the engine for reproducible diffs
      - name: Init against the prod backend
        run: terraform init -input=false -backend-config=env/prod.s3.tfbackend
      - name: Refresh-only drift plan
        id: plan
        run: |
          set +e
          terraform plan -refresh-only -detailed-exitcode -lock=false -out=drift.plan
          echo "exitcode=$?" >> "$GITHUB_OUTPUT"
          set -e
      - name: Alert on drift (exit code 2 only)
        if: steps.plan.outputs.exitcode == '2'
        run: |
          terraform show -no-color drift.plan > drift.txt
          curl -sS -X POST "$DRIFT_WEBHOOK" \
            -H 'content-type: application/json' \
            --data "$(jq -Rs '{text: ("Drift detected in prod spatial estate:\n" + .)}' drift.txt)"
        env:
          DRIFT_WEBHOOK: ${{ secrets.DRIFT_WEBHOOK }}
      - name: Fail the run so it is visible
        if: steps.plan.outputs.exitcode == '2'
        run: exit 1

The read-only role is deliberately narrow: it can read state and describe resources but cannot mutate anything, so a scheduled job can never accidentally apply. The -lock=false flag is safe here because a refresh-only plan takes no write lock, and skipping the lock keeps the detector from queueing behind a legitimate apply. For estates that prefer serverless scheduling, the identical logic runs from an AWS Lambda on an EventBridge rule; the Lambda invokes the same pinned Terraform binary from a container image and publishes drift to SNS. The Terraform configuration whose defaults the job relies on pins its providers the same way every deploy does:

terraform {
  required_version = ">= 1.10.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60" # unpinned providers manufacture phantom drift as defaults change
    }
  }
}

When the same estate manages a PostGIS parameter group, drift on a memory parameter is common because operators tune it live. Declare those values in AWS 8 kB block units so the declared and live values compare cleanly, never as percentage strings that the API normalizes and that then read as perpetual drift.

Note: RDS/Aurora memory parameters such as shared_buffers are expressed in 8 kB blocks — for example {DBInstanceClassMemory/32768} or an explicit integer like 4194304 (32 GB) — not as a "25%" string. A percentage string is stored verbatim by the API and then diverges from the engine’s resolved integer on every refresh, so a memory parameter written that way will register as drift forever. See PostGIS Cluster Provisioning for the full parameter-group baseline.

Remediation: Import, Refresh, and Targeted Apply

Once drift is classified, remediation follows one of three deterministic paths, and choosing the wrong one is how a drift response becomes an outage. The gentlest is a refresh: terraform apply -refresh-only (or a clean pulumi refresh) updates state to match reality without changing any live resource. This is the correct move for benign drift and for the reconciliation step after you have decided a manual change is acceptable and want state to record it. It never touches infrastructure, so it cannot cause a regression.

The second path handles resources that exist in the cloud but not in state — the class driftctl surfaces. Here you bring the resource under management with terraform import (or pulumi import), which writes the live resource into state so future plans track it. Import is non-destructive and is the right response to the hand-created raster bucket or the out-of-band security group: rather than delete the resource and risk breaking a live consumer, you adopt it into code and then reconcile its configuration on the next reviewed change.

The third path is the sharp one: a targeted apply to force a divergent resource back to its declared configuration. When someone widened a security group by hand and the team decides the widening is wrong, terraform apply -target=aws_security_group_rule.postgis_ingress restores only that resource without dragging the rest of the estate through an apply. Targeting keeps the blast radius minimal, which matters when the estate contains long-running spatial resources whose full apply would take tens of minutes. The rule is to promote-or-revert first — decide whether the manual change becomes code — and only then apply, because a targeted apply against unreconciled code silently reimposes whatever the repository last said, which may itself be stale.

Troubleshooting and Failure Modes

Autoscaling churn floods the detector with false positives. A tile-server fleet whose ASG cycles instances nightly makes every drift run report changes to desired_capacity, launch template versions, and target-group membership. The fix is not to relax the schedule but to declare lifecycle { ignore_changes = [desired_capacity] } on the ASG and to exclude runtime-managed attributes from the diff, so the detector reports only configuration a human changed. The dedicated fleet-scale version of this problem is worked end to end in Scheduled Drift Detection for Tile-Server Fleets.

A refresh silently adopts a dangerous change. Running apply -refresh-only on a diff that includes a hand-widened security group makes state agree that 0.0.0.0/0 on port 5432 is correct, quietly laundering the exposure into the accepted baseline. Never refresh dangerous drift; classify first, and reserve refresh for churn and for changes you have consciously decided to keep. A refresh is a decision to accept reality, not a neutral cleanup.

GeoServer console edits are invisible to the plan. If GeoServer layers, coverage stores, and CRS declarations are configured through the admin UI rather than provisioned declaratively, terraform plan reports the estate as clean while the map service serves misprojected tiles. The estate is drifting in a dimension the engine cannot see. The durable fix is to bring GeoServer configuration under code — the patterns in GeoServer Deployment Patterns — and, until then, to add an external check that diffs the GeoServer REST catalog against a declared manifest.

The scheduled job runs against a stale or wrong backend. A drift job that points at a mismatched key or an out-of-date workspace compares production reality against a different environment’s declared state and reports either constant drift or a falsely clean estate. Pin the backend config per environment and assert the workspace at the start of the run; treat the backend block as a reviewed artifact per State Backend Selection.

Provider version skew manufactures drift. When the scheduled detector uses a newer provider than the last apply, changed attribute defaults and new computed fields surface as drift that no human caused. Pin the provider to the exact constraint the deploy pipeline uses so the comparison is apples-to-apples, and bump both together through a reviewed change rather than letting the detector float ahead.

Related

• Scheduled Drift Detection for Tile-Server Fleets — the fleet-scale detector and how to suppress autoscaling false positives.
• Pipeline Orchestration for Spatial Deployments — where the deploy machinery that competes with drift jobs for state lives.
• Policy as Code for Spatial Resources — preventive rules that stop drift-worthy changes before merge.
• State Backend Selection — the isolated per-environment backend a drift detector must read from.
• GeoServer Deployment Patterns — bringing console-configured map services under code so their drift is visible.
• CI/CD, Drift Detection & Policy Governance — the governance framework this detection loop sits within.