Spatial IaC Terraform & Pulumi for GIS

Managing Terraform State Locks for Spatial Data: Orphaned-Lock Recovery and Prevention

A spatial deployment that hangs on Error acquiring the state lock and then fails with context deadline exceeded is the single most common pipeline stall on geospatial estates, because provisioning routinely runs long enough to outlive a default lock timeout. This guide resolves that exact symptom — a held or orphaned lock blocking plan/apply on a PostGIS, tile-serving, or raster backend — and shows how to prevent it recurring. It sits within State Backend Selection, the part of Spatial IaC Architecture & Fundamentals that governs where the deployment ledger lives and how writes to it are serialized. Spatial applies are uniquely exposed: index builds, raster tile generation, and vector topology validation are long, resource-intensive operations whose execution windows frequently exceed the lock wait time a pipeline allows, leaving abandoned locks behind a preempted runner.

Symptom Identification and Triage

State lock failures in spatial workflows carry distinct error signatures. The primary indicator is Error acquiring the state lock or a state lock timeout during plan or apply. In CI/CD the same condition surfaces as a job that hangs and then exits with context deadline exceeded or lock acquisition timeout. Secondary signals include a ConditionalCheckFailedException returned by the DynamoDB lock table, concurrent operation detected warnings when two pull-request pipelines provision overlapping geospatial layers, and partial drift where a geodatabase schema or vector dataset was created on the cloud but never recorded in state because the lock cycle was interrupted mid-write.

Triage starts by identifying the lock backend, because S3-with-DynamoDB, native S3 lockfiles, Azure Blob lease, and GCS each behave differently under load. Query the backend directly for the lock record and read its Info payload — Terraform stamps it with ID, Operation, Who, Created, and Version. Cross-reference that Created timestamp and Who field against your pipeline run history to classify the lock:

  • Live lock — the originating runner or developer session is still executing. This is not a fault; a heavy raster import is simply holding the lock legitimately.
  • Orphaned lock — the Who/Created record maps to a run that has already terminated (runner preemption, network partition, or a provider timeout during a slow spatial operation). This is the case that requires recovery.
  • Contention — multiple queued runs competing for the same state path, typically from missing pipeline serialization across PRs.

Spatial state amplifies all three because the state payload carries coordinate reference system (CRS) metadata, bounding-box extents, and deep layer dependency graphs, and these files routinely exceed 10 MB. Large payloads push backend read/write capacity and lease-renewal latency, so what looks like contention is sometimes a backend simply throttling under the size of the state — a sizing concern shared with Cost Estimation Frameworks when capacity is provisioned, not on-demand.

Triage decision tree for spatial Terraform state-lock failures Lock acquisition fails. Read the lock Info record (Who and Created) and cross-reference against pipeline run history. Three classifications follow. If the Who/Created maps to a still-active run it is a live lock — wait for it to finish or cancel gracefully. If the originating run has already terminated it is an orphaned lock — capture a state snapshot then perform a controlled force-unlock. If multiple queued runs compete for the same state path it is contention — serialize the pipeline to single-flight per state path. Lock acquisition fails Read lock Info record Who · Created vs run history Live lock run still active Orphaned lock run terminated Contention multiple queued runs Wait for it to finish or cancel gracefully Snapshot state, then controlled force-unlock Serialize pipeline — single-flight per path

Prerequisites and Environment Assumptions

Before touching a lock, confirm the operating context this guide assumes:

  • Tooling versions. Terraform 1.11+ (native S3 locking via use_lockfile; the dynamodb_table backend argument is deprecated) or Terraform 1.5–1.10 with a DynamoDB lock table. Pulumi 3.x for the comparison sequence. Pin them in required_version so recovery runs on the same engine that wrote the state.
  • Backend access. Network reachability and valid short-lived credentials to the state backend (S3 bucket + DynamoDB table, Azure Blob container, or GCS bucket). Recovery commands must run against the same isolated backend path the failed run used — confirm the key/prefix matches the environment, per State Backend Selection.
  • IAM permissions. The operator needs dynamodb:GetItem and dynamodb:DeleteItem on the specific lock table to inspect and release a lock, plus s3:GetObject/s3:PutObject on the state object to snapshot and reconcile. Force-unlock should be a human-gated role, never the default CI/CD runner identity — a boundary defined in IAM Role Mapping for GIS.
  • A snapshot target. An immutable, encrypted object-storage location for the pre-recovery state backup, ideally in the same bucket family described in Object Storage for Raster/Vector.

Step-by-Step Remediation

Recovery prioritizes state integrity over speed. Forcing an unlock blindly can strand spatial resources — a half-created geodatabase schema or a populated tile cache that the state never recorded. Follow this deterministic, non-destructive sequence.

  1. Validate lock ownership and backend health. Read the lock record and confirm it is genuinely orphaned. If the originating runner or developer session is still active, let it finish or cancel it gracefully — do not unlock a live operation. Check backend latency and IAM reachability to rule out a transient connectivity stall masquerading as a held lock.
  2. Capture a state snapshot. Export current state before any mutation, then store it immutably. This is the rollback point if reconciliation goes wrong.
  3. Execute a controlled unlock. Only after confirming no active writes, release the lock by its exact ID and record the ID, timestamp, and operator in your incident system. Never automate force-unlock without a human-in-the-loop gate — a wrong unlock during a live PostGIS index build corrupts the resource graph.
  4. Reconcile drift and validate spatial assets. Run a refresh-only plan to detect divergence between the backend and real infrastructure, then inspect the geospatial resources the interrupted run may have touched: PostGIS extension versions, geodatabase schemas, and raster tile-cache integrity. If partial resources exist outside state, bring them back with terraform import rather than recreating them.
  5. Re-run with serialized execution. Replay the original deployment with an explicit -lock-timeout long enough for the spatial operation and with CI/CD queueing that prevents concurrent applies on the shared backend.

The recovery sequence is deliberately non-destructive — confirm the lock is genuinely orphaned and capture a snapshot before forcing anything:

Non-destructive recovery sequence for an orphaned spatial state lock Start at a lock timeout or orphaned lock. Decision: is the lock genuinely orphaned? If no, because the runner is still active, let it finish or cancel gracefully. If yes, step one is to export a state snapshot to an immutable store before any mutation. Step two is a controlled, human-approved force-unlock by exact lock ID. Step three is a refresh-only plan that reconciles drift and imports any partial spatial resources. Step four is to re-run the deployment with serialized execution and an explicit lock-timeout sized to the slowest spatial operation. Lock timeout / orphaned lock Lock genuinely orphaned? no Runner still active — finish or cancel gracefully yes 1 · Export state snapshot 2 · Controlled force-unlock by ID 3 · Refresh-only plan — reconcile drift 4 · Re-run: serialized execution + spatial-sized -lock-timeout

The minimal command sequence for the orphaned-lock case:

# 1. Inspect the lock record (DynamoDB-backed backend)
aws dynamodb get-item \
  --table-name gis-state-locks \
  --key '{"LockID":{"S":"gis-platform-state/prod/spatial-core/terraform.tfstate-md5"}}'

# 2. Snapshot state to an immutable, encrypted store BEFORE any mutation
terraform state pull > "state-backup-$(date +%s).tfstate"

# 3. Controlled, human-approved unlock by exact lock ID
terraform force-unlock 1a2b3c4d-5e6f-7a8b-9c0d-112233445566

# 4. Reconcile drift left by the interrupted spatial apply
terraform plan -refresh-only -out=refresh.plan
terraform apply refresh.plan

# 5. Re-run with a spatial-sized lock wait and serialized execution
terraform apply -auto-approve -lock-timeout=60m

Verification

Confirm the fix is live rather than assuming the unlock succeeded:

  • Lock cleared. Re-query the lock table — aws dynamodb get-item should return no Item for the LockID, or for native S3 locking the .tflock object should be absent from the state prefix.
  • State converged. A fresh terraform plan reports No changes. Your infrastructure matches the configuration. If the refresh-only step imported partial resources, the diff should now be empty rather than proposing to recreate a PostGIS instance or tile bucket.
  • Spatial assets intact. Validate the resources the interrupted run touched: ogrinfo or a SELECT postgis_full_version(); against the database confirms the extension matrix matches the PostGIS Cluster Provisioning baseline, and a request to a tile endpoint (e.g. a /{z}/{x}/{y}.pbf probe) confirms the cache survived.
  • Backend audit trail. The CloudTrail DeleteItem event on the lock table should record the force-unlock with the operator identity you expect — no anonymous or CI-runner principal.

Preventing Recurrence

Encode the fix so the same orphaned lock cannot recur. Start at the backend: pin the engine and prefer native locking, and treat the backend block as a reviewed, version-controlled artifact rather than terminal input. The trade-off between this approach and Pulumi’s checkpoint model is examined in Terraform vs Pulumi for GIS; Pulumi’s pulumi cancel rolls back pending operations without manual lock-ID resolution, while Terraform requires the CI/CD orchestration below.

terraform {
  required_version = ">= 1.11.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
  backend "s3" {
    bucket       = "gis-platform-state"
    key          = "prod/spatial-core/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true # native S3 locking (1.11+); no DynamoDB table needed
    # Lock wait is set per run, sized to the slowest spatial operation:
    #   terraform apply -lock-timeout=60m
  }
}

For existing estates still on a DynamoDB lock table, the table itself is the place to add a stale-lock safety net. TTL on an Expires attribute will reap truly abandoned records, but Terraform does not populate that attribute — stamp it from a scheduled job, and keep -lock-timeout as the primary mechanism rather than relying on expiry.

resource "aws_dynamodb_table" "state_lock" {
  name         = "gis-state-locks"
  billing_mode = "PAY_PER_REQUEST" # absorbs 10 MB+ spatial-state write spikes
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }

  # Safety net for orphaned locks only; a sidecar job stamps 'Expires'
  # on stale entries — Terraform never writes this attribute itself.
  ttl {
    attribute_name = "Expires"
    enabled        = true
  }

  server_side_encryption {
    enabled = true
  }
}

Beyond the backend, three durable guardrails close the loop:

  • Isolate heavy spatial work. Decouple raster processing, vector topology validation, and schema migrations into separate stacks or state paths so a long index build never holds the lock that a routine apply needs. This is the separate-backend-per-concern discipline from Module Design Patterns, and it shrinks the lock-contention window directly.

  • Serialize and retry in the pipeline. Enforce single-flight execution per state path and add exponential backoff with jitter so a transient contention does not become a thundering herd of competing applies.

    # GitHub Actions: spatial-sized lock wait with backoff
- name: Terraform Apply with Retry
  run: |
    MAX_RETRIES=3
    RETRY_COUNT=0
    while [ $RETRY_COUNT -lt $MAX_RETRIES ]; do
      terraform apply -auto-approve -lock-timeout=60m && break
      RETRY_COUNT=$((RETRY_COUNT + 1))
      sleep $(( 2 ** RETRY_COUNT * 60 ))
    done

  • Alert on lock age and lock down access. If a lock persists beyond ~45 minutes, raise an incident and halt downstream pipelines until backend health is confirmed. Scope CI/CD runners to least-privilege lock operations (PutItem, GetItem, DeleteItem, UpdateItem on the specific LockID pattern — never Scan/Query), route lock-table mutations to a central audit log, and reserve force-unlock for the human-gated role defined in IAM Role Mapping for GIS. Pair this with pre-apply spatial validation (ogrinfo, topology checks) so a provider timeout never abandons a lock mid-write in the first place.