Spatial IaC Terraform & Pulumi for GIS

Observability for Spatial Infrastructure

A spatial platform fails in a shape that generic dashboards render invisible. Aggregate tile latency stays flat while the p99 at zoom 14 over a dense metropolitan extent triples, because ninety-five per cent of requests are cheap ocean and sparse rural tiles that the cache serves in milliseconds. Error rates stay near zero while a renderer returns structurally valid but empty tiles. Database CPU looks healthy while one ST_Intersects query against an un-generalized geometry column consumes an entire connection slot for eleven seconds. Observability for this estate therefore means instrumenting the dimensions that are specific to geospatial work — zoom level, layer, tile coordinate density, projection, and payload size — rather than inheriting a web-service dashboard and hoping. This topic sits within CI/CD Automation and Governance, because in an Infrastructure-as-Code estate the dashboards, alarms and service objectives are themselves provisioned artifacts that must be versioned, reviewed and tested alongside the resources they watch.

Treating telemetry as code has a second benefit that matters during incidents. When a dashboard is provisioned from the same module as the resource, it exists the moment the resource does — there is no window in which a newly deployed tile service is running unmonitored because nobody has got round to the console yet. The Drift Detection and Remediation machinery then applies to observability as much as to compute: a deleted alarm is drift, and should page someone the same way a deleted security rule does.

Environment parity and configuration drift mitigation

Parity in observability is mostly about thresholds, and thresholds are where copy-paste does the most damage. A staging environment running one renderer replica against a fixture dataset cannot share a latency objective with a production fleet of eight replicas against the full national extent, so a threshold cloned across environments is either permanently firing in one or permanently silent in the other. Both outcomes end the same way: the alert is muted, and the signal is gone.

Derive thresholds from the environment’s own capacity inputs rather than restating a number. If the module knows the replica count, the pool size and the instance class, it knows enough to compute a saturation threshold as a fraction of capacity. That turns a scaling change into an automatically corrected alarm rather than a stale one, and it removes the most common reason alerting stops matching reality — that somebody resized the fleet and nobody remembered the alarm existed.

The second parity axis is dimension cardinality, and it drifts in the opposite direction: toward too much. Emitting a metric per tile coordinate is irresistible and ruinous, because the coordinate space is unbounded and the metrics bill scales with it. The workable compromise is to keep unbounded identifiers in logs and traces, where they are searchable but not aggregated, and to keep metrics on bounded dimensions only — zoom level, layer name, cache result, status class. Zoom is bounded at roughly twenty values, layers at a few dozen; both are safe. Tile x and y are not.

Bounded dimensions for metrics, unbounded ones for logs Two columns. On the left, dimensions safe to attach to metrics because their value space is bounded: zoom level with about twenty values, layer name with a few dozen, cache result with three, and status class with five. On the right, dimensions that must stay in logs and traces because their value space is unbounded: tile x and y coordinates, request identifiers, client addresses and bounding boxes. Attaching an unbounded dimension to a metric makes the metrics bill scale with the coordinate space rather than with traffic. Metrics — bounded dimensions zoom level ~20 values layer name dozens cache result hit · miss · bypass status class 2xx · 4xx · 5xx Safe to aggregate — the series count stays finite. Logs and traces — unbounded tile x and y 4^zoom values request id one per request client address unbounded bounding box continuous Searchable during an incident, never aggregated.

CI/CD validation and operational guardrails

Alerting rules are code, and code that has never been executed does not work. The specific hazard with alarms is that the untested state is the one you need: an alarm is defined, deployed, and never fires until the night it should, at which point a wrong metric name, a missing dimension or a notification target that was decommissioned two quarters ago turns the outage into an unmonitored outage.

Test alarms the way you test anything else — by making them fire. A synthetic stage in the pipeline can push a value past the threshold on a test metric with the same shape as the real one, then assert that the alarm transitions to the alarm state and that the notification reaches its destination. That single test validates the metric name, the dimension set, the evaluation window, the threshold arithmetic and the delivery path, which is most of what silently breaks.

Two guardrails keep the alerting estate from decaying under its own weight. The first is an ownership requirement: every alarm carries a tag naming the team that receives it, and a plan that adds an alarm without one fails. Unowned alarms are the ones that fire into an empty channel for months. The second is a runbook link on every alarm description, so the first thing an on-call engineer sees at 3am is the document rather than a metric name they must interpret. Both are trivially enforced in the same Policy as Code for Spatial Resources gate that governs the rest of the estate.

The third guardrail is a budget on the telemetry itself. Metrics, log ingestion and trace sampling are metered, and a spatial platform emitting per-layer, per-zoom histograms across a large fleet can generate an observability bill that rivals its compute bill. Sample traces rather than collecting all of them, keep high-cardinality data in logs with a short retention and a longer-retention aggregate, and price the telemetry configuration in the same pre-merge estimate that prices the infrastructure, as described in Cost Estimation Frameworks.

Proving an alarm works by making it fire A pipeline stage publishes a synthetic data point past the threshold on a shadow metric carrying the same name shape and dimensions as the production metric. It then waits for the alarm to transition into the alarm state, and asserts that a notification was delivered to the owning team's channel. Finally it publishes a value below the threshold and confirms the alarm returns to normal. A failure at the transition step means the metric name or dimension set is wrong; a failure at the delivery step means the notification target is wrong. Both are failures that would otherwise only be discovered during a real incident. Publish over threshold shadow metric Assert transition state becomes ALARM Assert delivery owner channel received Reset back to normal Wrong metric or dimension the alarm could never fire Wrong notification target it fires into an empty channel An alarm's untested state is the one you need — the first real firing should not also be its first firing.

Resource architecture and service integration

The instrumentation points that matter follow the request path. At the edge, the CDN reports cache hit ratio, origin request rate and origin latency — the hit ratio being the single most informative number in a tile platform, because a fall from 95 per cent to 80 per cent quadruples origin load without any change in user-visible traffic. At the renderer, latency and error rate broken down by layer and zoom, plus connection-pool utilisation, which is the leading indicator that precedes every pool-exhaustion incident. At the database, the standard saturation signals plus pg_stat_statements ordered by total time, which is where a single unindexed spatial predicate reveals itself. And at the pipeline, the age of the most recent successful ingest, which is the only metric that catches a data staleness incident where every service is healthy and the content is a week old.

That last one deserves emphasis because it is the failure mode most often missing from a dashboard. Every component can be green while the platform is serving stale data: the ingest job failed silently three days ago, the tiles are still cached, the renderers still respond, and nobody notices until a user reports that a new development does not appear on the map. A freshness metric — the wall-clock age of the newest successfully processed source object, per dataset — turns that into an alarm. It integrates naturally with the pipelines described in Raster Pipeline Provisioning, which already know when each stage last succeeded.

Tracing ties the layers together and is worth the effort specifically because a slow tile has at least four plausible causes. A trace that follows a single request from the edge through the renderer into the database query attributes the latency definitively, replacing an argument between three teams with a span diagram. Propagate a trace identifier from the edge, attach the tile coordinate and layer as span attributes rather than as metric dimensions, and sample aggressively — a small percentage of traced requests is ample to characterise a latency distribution, and the full firehose is what makes tracing unaffordable.

Service level objectives for map endpoints

An objective is only useful if it describes something a user would recognise as working, and for a map that is not “the tile service returned 200”. A user experiences a map as working when the visible extent fills in quickly enough that panning feels continuous, which decomposes into three measurable claims: the tiles for the current viewport arrive within a budget, they contain the features they should, and the data they contain is not stale. An objective built on availability alone will report a healthy quarter through an incident in which every tile was empty.

Define the objective per layer class rather than per service, because the classes have genuinely different budgets. A cached basemap should be served from the edge in tens of milliseconds and any origin request is already a mild anomaly. A live operational layer rendered from PostGIS carries a much larger budget, because it is doing real work per request, and holding it to the basemap’s number would either force absurd over-provisioning or guarantee a permanently violated objective. Two objectives with honest numbers are more useful than one objective with a number nobody believes.

Choose the measurement point deliberately. Edge-measured latency includes the network path to the user and is what the user experiences; origin-measured latency excludes it and is what the platform team can actually act on. Both are legitimate and they answer different questions, so publish both and be explicit about which one the objective is written against. The common failure is an objective measured at the origin and reported as user experience, which quietly excludes the cache-miss path that hurts most.

Error budgets are what make an objective operational rather than decorative. A budget converts “we aim for 99.5 per cent” into a concrete quantity of failure per period that the team may spend, and spending it deliberately — on a risky migration, on a schema change during business hours — is a legitimate engineering decision. The rule worth encoding is that when the remaining budget falls below a threshold, deploys other than reliability fixes stop, which is enforceable in the same delivery pipeline described in Pipeline Orchestration for Spatial Deploys.

The three claims behind a map service objective A working map objective rests on three measurable claims rather than on availability alone. Latency: the tiles covering the current viewport arrive within a budget, measured separately for cached basemap layers and live rendered layers because their budgets differ by an order of magnitude. Correctness: a returned tile decodes and contains the expected layers and a non-zero feature count, which availability alone cannot detect. Freshness: the newest successfully ingested source object is younger than the dataset's staleness budget. An objective built on availability alone reports a healthy quarter through an incident in which every tile returned successfully and was empty. Latency viewport tiles within budget cached basemap: tens of ms live layer: a larger, honest number state the measurement point Correctness the tile decodes expected layers present feature count above zero invisible to availability alone Freshness age of newest ingest per dataset, not per service catches the silent failure where everything is green Error budget spent deliberately below the remaining-budget threshold, only reliability fixes ship enforced in the delivery pipeline, not in a quarterly review

Runnable configuration

The following module provisions a latency objective for a tile endpoint whose threshold is derived from the deployment’s own capacity inputs, with ownership and a runbook required on the alarm and a composite alarm that suppresses downstream noise when the database is the actual cause.

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

variable "replica_count" { type = number }
variable "pool_size" { type = number }
variable "owner_team" { type = string }
variable "runbook_url" { type = string }

locals {
  # The threshold is derived, not restated: resizing the fleet corrects the
  # alarm automatically instead of leaving a stale number behind.
  concurrent_capacity = var.replica_count * var.pool_size
  pool_warn_threshold = floor(local.concurrent_capacity * 0.75)
}

resource "aws_cloudwatch_metric_alarm" "tile_p99_latency" {
  alarm_name          = "tiles-p99-latency-zoom-high"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 3
  threshold           = 800 # milliseconds at the origin, not at the edge
  treat_missing_data  = "notBreaching"

  metric_query {
    id          = "p99"
    return_data = true
    metric {
      namespace   = "SpatialPlatform/Tiles"
      metric_name = "OriginLatency"
      period      = 60
      stat        = "p99"
      # Zoom is a bounded dimension and safe to aggregate on. Tile x and y are
      # not, and stay in the trace attributes instead.
      dimensions = {
        ZoomBand = "z12-z16"
        Layer    = "parcels"
      }
    }
  }

  alarm_description = "p99 origin latency for dense-zoom parcel tiles. Runbook: ${var.runbook_url}"
  alarm_actions     = [aws_sns_topic.oncall.arn]
  tags = {
    Owner   = var.owner_team
    Runbook = var.runbook_url
  }
}

resource "aws_cloudwatch_metric_alarm" "pool_saturation" {
  alarm_name          = "tiles-connection-pool-saturation"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  threshold           = local.pool_warn_threshold
  metric_name         = "ActiveConnections"
  namespace           = "SpatialPlatform/Tiles"
  period              = 60
  statistic           = "Maximum"
  alarm_description   = "Leading indicator of pool exhaustion. Runbook: ${var.runbook_url}"
  alarm_actions       = [aws_sns_topic.oncall.arn]
  tags = {
    Owner   = var.owner_team
    Runbook = var.runbook_url
  }
}

# When the database is saturated every downstream alarm fires at once. The
# composite alarm pages on the cause and suppresses the symptoms.
resource "aws_cloudwatch_composite_alarm" "tile_path_degraded" {
  alarm_name = "tile-path-degraded"
  alarm_rule = join(" ", [
    "ALARM(${aws_cloudwatch_metric_alarm.tile_p99_latency.alarm_name})",
    "AND NOT ALARM(${aws_cloudwatch_metric_alarm.pool_saturation.alarm_name})",
  ])
  alarm_actions = [aws_sns_topic.oncall.arn]
}

Guardrails embedded in the configuration

  • Thresholds are computed from capacity inputs. pool_warn_threshold is a fraction of replica_count × pool_size, so a scaling change updates the alarm in the same apply that changes the fleet. A literal number here is a guaranteed future mismatch.
  • Ownership and runbook are required tags. An alarm without a named owner fires into an empty channel; an alarm without a runbook link costs the responder fifteen minutes of orientation at the worst possible time.
  • Only bounded dimensions appear on metrics. ZoomBand and Layer are finite; tile coordinates are not and stay in traces. This is a cost control as much as a design preference.
  • treat_missing_data is explicit. The default behaviour differs by provider and version, and an alarm that silently treats absent data as breaching will page on every low-traffic night, which is how alerting gets muted.
  • A composite alarm encodes causality. Paging on the symptom and the cause simultaneously trains responders to ignore pages; suppressing the symptom when the cause is already firing keeps the signal meaningful.

One organisational note completes the picture. Dashboards provisioned from a module are only as useful as the review they receive, and the review that matters is not the pull request — it is the periodic look at whether the panels answered the questions the last three incidents actually asked. Fold that into the incident review: when a responder had to run an ad-hoc query to find something, that query is a missing panel, and adding it to the module is how the estate’s telemetry improves rather than merely accumulating.

Troubleshooting and failure modes

1. Flat aggregates hiding a real regression. The dashboard is green because cheap tiles dominate the average. Break every latency panel down by zoom band and layer before trusting it — the aggregate over a tile workload is nearly meaningless because the cost distribution across the tile space is extremely skewed.

2. Cardinality explosion in the metrics bill. A dimension containing tile coordinates, request identifiers or bounding boxes multiplies series count without bound. The symptom is a metrics bill growing faster than traffic; the fix is to move the dimension into logs and traces.

3. An alarm that never fired and never could. A renamed metric or a dimension set that does not match any published series produces an alarm permanently in insufficient-data state, which most consoles render as calm rather than as broken. The synthetic firing test is the only reliable detection.

4. Stale data with every component healthy. No component reports an error; the content is a week old. Only a freshness metric on the ingest pipeline catches this, and it is the metric most commonly absent.

5. Alert storms from a single upstream cause. Database saturation lights up every downstream alarm at once. Without composite alarms encoding causality, responders learn to ignore the page, and the next unrelated alert is ignored too.

6. Traces that are sampled away exactly when needed. Uniform low-rate sampling means the pathological slow requests — the ones worth studying — are almost never captured. Sample by latency where the backend supports it, so slow requests are retained preferentially and the trace store keeps the interesting tail rather than a representative middle.

A closing word on retention, because it decides what a post-incident review can actually establish. Metrics aggregated at one-minute resolution are cheap to keep for a year and are what you need to answer “was this happening before”; raw logs at full cardinality are expensive to keep for a month and are what you need to answer “which request”. Set the two retentions independently and deliberately rather than accepting one default for both — a platform that keeps everything for ninety days is paying for logs it will never read and discarding the trend line that would have shown the regression starting.