Spatial IaC Terraform & Pulumi for GIS

Pulumi Programmatic Resource Mapping for GeoServer: Incident Response and State Recovery

When a Pulumi up reports pending creation for a workspace, datastore, or layer group that already exists on a live GeoServer instance, the stack has lost its mapping between declared resource URNs and the runtime objects in the GeoServer catalog. This guide is the recovery runbook for that exact failure: it isolates the divergence, reconciles Pulumi state against the GeoServer REST API without destroying live layers, and encodes a guardrail so the drift cannot recur. It applies the immutable-publishing-tier discipline established in GeoServer Deployment Patterns and sits inside the broader Geospatial Resource Provisioning practice, so the reconciliation steps assume the catalog is reconstructable from code rather than hand-edited in the admin console. Because GeoServer has no official Pulumi provider, every resource here is managed through the REST API wrapped in a command.local.Command, which is precisely why state can diverge from reality and why the recovery is procedural rather than a single refresh.

Symptom identification and triage

Triage starts by locating the failure boundary between the Pulumi state backend and the GeoServer REST API. The mapping is healthy when pulumi preview reports no changes and a GET against /rest/workspaces.json returns the same workspaces the stack declares. Divergence shows up as one of a small set of concrete signals:

  • Preview/apply divergence. pulumi preview lists + create for a command:local:Command resource whose target object already exists in the catalog. The create command then fails with an HTTP 500 or 409 Conflict because GeoServer rejects a duplicate workspace or datastore name.
  • 404 Not Found on update. The stack believes a datastore exists, but a curl to /rest/workspaces/{ws}/datastores/{name}.json returns 404. The resource was deleted out-of-band (console or a sibling stack) and Pulumi state is now a phantom.
  • State lock contention. pulumi up blocks with the stack is currently locked by ..., the residue of an interrupted CI job that left a partial apply and orphaned REST registrations.
  • Credential rejection. REST calls return 401 Unauthorized; GeoServer admin or PostGIS credentials were rotated in the cloud secret manager without updating the matching Pulumi config secret.

Classify the incident before touching state. A 409 on create means an orphaned live resource (exists in GeoServer, absent from state) that must be imported. A 404 on update means a phantom state resource (exists in state, absent from GeoServer) that must be removed from state. A 401 is a credential drift that is fixed by reconfiguring secrets, not by mutating state at all. Misreading the class is how a recovery turns into an outage — deleting a “phantom” that is actually live drops the layer for every downstream tile consumer.

Triage Decision Tree: Classifying Pulumi-to-GeoServer State Divergence From a failing pulumi preview or up, the error signal is classified into four mutually exclusive branches. A 409 is an orphaned live resource (in catalog, not in state) imported with pulumi import; a 404 is a phantom state entry (in state, not in catalog) cleared with pulumi state delete; a 401 is credential drift fixed by rotating the config secret without mutating state; a lock error is an interrupted apply released with pulumi cancel. Each leaf names the one safe action, so a live layer is never deleted as if it were a phantom. pulumi preview shows diff or pulumi up fails Classify the error signal 409 already exists create hits live object Orphaned live resource in catalog, not in state pulumi import adopt URN, no recreate 404 on update REST probe misses Phantom state entry in state, not in catalog pulumi state delete state only, no API call 401 unauthorized REST call rejected Credential drift secret rotated, not synced rotate config secret no state mutation stack locked operation blocked Interrupted apply stale advisory lock pulumi cancel release, then re-probe signal class safe action

Before any mutation, halt automated deploys, confirm the admin console is reachable, and capture a raw stack export so every subsequent step is reversible.

Prerequisites and environment assumptions

This runbook assumes a TypeScript Pulumi program that drives GeoServer through command.local.Command. Pin the toolchain so the recovery is reproducible against a known surface:

  • Pulumi CLI ≥ 3.130, with the stack’s state on a locking-capable backend. The reconciliation depends on the advisory lock guaranteed by the patterns in State Backend Selection; a backend without locking can interleave a concurrent apply mid-recovery.
  • Provider packages pinned in package.json: @pulumi/pulumi ~3.130 and @pulumi/command ~1.0. The local.Command resource schema is version-sensitive, so an unpinned upgrade can itself appear as a diff.
  • Network reachability from the execution context to the GeoServer REST endpoint over a private path — the routes governed by VPC Routing for Tile Servers — and curl available on the runner that executes the command resources.
  • Least-privilege identity. The execution role reads the GeoServer admin secret and the PostGIS connection secret from the cloud secret manager only, scoped per the IAM Role Mapping for GIS boundary. No literal credentials live in the stack.
  • Config secrets present. gsAdminPassword and postgisEndpoint must resolve via pulumi config; a missing secret produces the 401 symptom above rather than a recoverable diff.

Step-by-step remediation

Each step is annotated with why it matters for a spatial catalog specifically — workspace hierarchies, SRS-bearing datastores, and coverage stores cannot be regenerated blind without dropping published layers.

  1. Capture an immutable backup. Never mutate state without a rollback artifact; a botched state delete is otherwise unrecoverable.

    pulumi stack export --file geoserver-state-backup-$(date +%s).json

  2. Probe the live catalog. Compare the REST truth against what the stack declares. The workspace list is the cheapest authoritative signal of which resources actually exist.

    curl -s -u admin:"${GS_ADMIN_PASSWORD}" \
  -H "Accept: application/json" \
  https://geoserver.example.com/geoserver/rest/workspaces.json

  3. Clear a stale lock, if present. An interrupted CI job leaves a lock that blocks every subsequent operation; cancel it only after confirming no apply is genuinely in flight.

    pulumi cancel --yes   # releases the advisory lock from an interrupted run

  4. Reconcile orphaned live resources (the 409 class). Import the existing GeoServer object so its URN is adopted into state instead of recreated. Because there is no typed provider, import the command:local:Command resource by id and let the next create become a no-op against the now-known object.

    pulumi import command:local:Command register-postgis-datastore <existing-resource-id>

  5. Remove phantom state resources (the 404 class). Delete the state entry only — pulumi state delete does not call the GeoServer API, so a layer that was already removed out-of-band is cleared from state without a second destructive request.

    pulumi state delete "urn:pulumi:prod::geoserver-stack::command:local:Command::register-postgis-datastore"

  6. Re-declare the resource as a REST-managed command. With state aligned, the program below is the minimal reproducible mapping for a single PostGIS datastore. The REST API is the authoritative management interface; the delete command keeps teardown symmetric so a future pulumi destroy does not strand the store.

    import * as pulumi from "@pulumi/pulumi";   // pin: @pulumi/pulumi ~3.130
import * as command from "@pulumi/command"; // pin: @pulumi/command ~1.0

const config = new pulumi.Config();
const gsAdminPassword = config.requireSecret("gsAdminPassword");
const postgisEndpoint = config.require("postgisEndpoint");

// GeoServer rejects a duplicate name, so state must already be reconciled
// before this create runs — otherwise it re-triggers the 409 it just fixed.
const datastoreConfig = pulumi.interpolate`{
  "dataStore": {
    "name": "production_vector_store",
    "connectionParameters": { "entry": [
      {"@key": "host", "$": "${postgisEndpoint}"},
      {"@key": "port", "$": "5432"},
      {"@key": "database", "$": "spatial_prod"},
      {"@key": "user", "$": "gis_reader"},
      {"@key": "dbtype", "$": "postgis"},
      {"@key": "schema", "$": "public"}
    ] }
  }
}`;

new command.local.Command("register-postgis-datastore", {
  create: pulumi.interpolate`curl -sf -u admin:${gsAdminPassword} \
    -H "Content-Type: application/json" -X POST -d '${datastoreConfig}' \
    https://geoserver.example.com/geoserver/rest/workspaces/gis_core/datastores`,
  delete: pulumi.interpolate`curl -sf -u admin:${gsAdminPassword} -X DELETE \
    "https://geoserver.example.com/geoserver/rest/workspaces/gis_core/datastores/production_vector_store?recurse=true"`,
});

  7. Refresh against reality. pulumi refresh re-reads the live environment into state. This matters when the datastore backs raster coverages routed through Object Storage for Raster/Vector Workloads, because GeoWebCache holds local metadata that can reference stale prefixes until refreshed.

    pulumi refresh --yes

Verification

The fix is live only when state, the catalog, and a real spatial request all agree. Run three independent checks:

# 1. State now matches the live catalog — must report no pending changes.
pulumi preview --diff --expect-no-changes

# 2. The datastore is reachable and connected, not just registered.
curl -s -u admin:"${GS_ADMIN_PASSWORD}" \
  https://geoserver.example.com/geoserver/rest/workspaces/gis_core/datastores/production_vector_store.json

# 3. A published layer actually renders — a GetMap probe is the end-to-end proof.
curl -s -o /dev/null -w "%{http_code}\n" \
  "https://geoserver.example.com/geoserver/gis_core/wms?service=WMS&version=1.3.0&request=GetMap&layers=gis_core:parcels&bbox=-180,-90,180,90&width=256&height=256&crs=EPSG:4326&format=image/png"

A 200 from the GetMap probe and an empty --expect-no-changes diff together confirm the mapping is restored: the datastore is connected, the layer resolves to the expected EPSG code and axis order, and the stack no longer believes it must recreate anything. A GetCapabilities request that advertises every expected workspace is the broader sanity check before re-enabling automated deploys.

Preventing recurrence

State mapping drifts when GeoServer is mutable outside the pipeline, so the durable fix removes that possibility and detects any breach early.

  • Scheduled drift detection. A nightly pulumi preview --expect-no-changes in CI fails the job the moment the live catalog diverges from committed state, surfacing out-of-band console edits within 24 hours instead of at the next deploy.
  • Policy-as-code gates. A CrossGuard rule rejects any command resource whose create embeds a literal password rather than a config.requireSecret reference, and blocks datastore payloads that point at a public host — drawing the same boundaries as Security Group Hardening.
  • Read-only console. Disable admin-console write access in production so the REST-managed stack is the only mutation path; an unwritable catalog cannot drift.
  • Independent credential rotation. Rotate the GeoServer admin and PostGIS secrets in the secret manager and reload them into pulumi config on the same cadence, so a rotation never leaves the stack stranded on a stale 401.
  • Module-level import discipline. Whenever a layer is created by hand during an incident, follow it immediately with pulumi import so the URN-to-catalog mapping is captured before the next plan runs.

The decision to express this mapping in TypeScript rather than HCL — and the trade-offs of driving a provider-less service through command resources — is examined in Terraform vs Pulumi for Spatial Infrastructure as Code.